Skip to content

Run a virtual iPhone on an Apple Silicon Mac.

Swift

docs: the D2/D3 handover, and un-stale the ledger

Lakr4cb4349f2996
diff --git a/research/d2_d3_self_containment.md b/research/d2_d3_self_containment.md
new file mode 100644
index 00000000..d31f7cec
--- /dev/null
+++ b/research/d2_d3_self_containment.md
@@ -0,0 +1,222 @@
+# D2 / D3 — the three tiers, and a dist runtime with nothing outside macOS in it
+
+Status as of `a908f81` on `vphone-intg-update`. This is the handover for the
+self-containment work: what the rule actually is, what was removed and where
+each thing went, what is verified and how, and what is left.
+
+---
+
+## 1. The rule was never one rule
+
+`check_aux.sh` used to hold a single list of "programs we still reach for",
+applied to the whole repository. That made `xcrun` in a build script and
+`xcrun` in something the `.app` ships count the same, so the list shrinking
+said nothing about whether the product was any closer to standing on its own.
+
+Three environments run code here, and they have different rules:
+
+| tier | who runs it | may use |
+| --- | --- | --- |
+| **build** | the machine that builds the `.app` — a developer's Mac or CI | Xcode, `xcrun`, clang, swift, git, **and Homebrew**. Nothing in this tier ships. |
+| **dist** | the shipped `.app`, on a stranger's clean macOS | `/usr/lib`, `/System`, and what is inside the bundle. No Homebrew, no Xcode, **no `PATH` lookup at all**. |
+| **guest** | inside the VM | out of host self-containment scope. These ship as payload to be copied in, never run on the host. |
+
+Every script declares its tier on **line 2**:
+
+```sh
+#!/bin/zsh
+# vphone-tier: dist
+```
+
+`scripts/dist_manifest.sh` reads those declarations and prints the dist
+payload. `scripts/build.sh` and `make bundle` both stage from it, so the
+bundler is an **allowlist**. It used to be a list of exclusions, which is how
+the `.app` came to carry `build.sh`, `check_aux.sh` and `setup_tools.sh` — the
+last of which runs `brew install`. An undeclared script now ships nowhere and
+gate 0 says so.
+
+> `make bundle` stages `Contents/Resources` now. It did not, and `build.sh`
+> did, so `make check-aux` was inspecting a five-binary bundle while users got
+> twenty scripts and a Homebrew-linked `trustcache`. **The gate was green
+> because it was looking at the wrong artifact.** If you add a bundling step,
+> add it to both or the gate goes blind again.
+
+---
+
+## 2. What the dist tier reached for, and where each one went
+
+| was | now | checked by |
+| --- | --- | --- |
+| `xcrun` (five iOS binaries cross-compiled at CFW-install time) | `scripts/guest_binaries.mk` builds them at build time; they ship in `Contents/Resources/guest` | gate 2, and a byte comparison against the old invocations |
+| `ldid` | `vphone-cli sign` / `dump-entitlements` | same CodeDirectory and CDHash as `ldid` on the real `vphoned`; gate 3 signs with `env -i PATH=/usr/bin:/bin` |
+| `gtar`, `zstd`, `tar` | `vphone-archive` (`extract`, `create`, `decompress`) | gate 3 round-trips a `.tzst` with no `zstd` on `PATH` |
+| `ipsw fw aea --key` | `vphone-cli fw aea-key` | byte-identical to `ipsw` on a real 24A435 archive |
+| `ipsw img4 im4p create/extract` | `vphone-cli fw im4p-create` / `im4p-extract`, and `IM4P` directly in `CryptexFilesystemPatcherSealing` | byte-identical containers for `isys`/`trst`/`msys`; gate 3 round-trips one |
+| `ipsw download ipsw --urls` | `vphone-cli fw urls` | same 31 URLs for `iPhone17,3`, diffed against `ipsw` |
+| `ipsw download appledb` + partial zip extract | `vphone-cli fw seal-tool` over `VPhoneRemoteZip` | pulls `apfs_sealvolume` out of an 18 GB remote IPSW in ~7 s |
+| `aria2c`, `wget` | deleted; `/usr/bin/curl` resumes with `-C -` | gate 2 |
+| `trustcache` (bundled, linked `/opt/homebrew`'s `libcrypto.3`) | **deleted** — nothing ever invoked it; the trust cache comes from `cryptexctl` in `/System/Library/SecurityResearch` | gate 1 |
+
+`DIST_REMAINING` in `check_aux.sh` — the list a release requires to be empty —
+**is empty**. Keep it that way; adding a line is a decision to ship something
+that does not work on a clean Mac.
+
+### The compile/install split
+
+Five Mach-Os run inside the guest: `vphoned`, `TweakLoader.dylib`,
+`vpregister`, `libvcamcaptured.dylib`, `libcamfix.dylib`. Every one used to be
+compiled at CFW-install time — three by the installers, and `vphoned` twice
+more (again by `FirmwarePatcher.buildVphoned`). That made a full Xcode install
+a prerequisite for putting firmware on a VM.
+
+**Compiling moved to build time. Signing did not, and cannot**: it uses the
+target VM's own `cfw_input/signcert.p12`, which does not exist until a VM does.
+`cfw install` still runs in dist, as it must — it is something `vphone-cli`
+does on the user's machine.
+
+`scripts/guest_binaries.mk` keeps the installers' own argument order,
+framework list included. That looks arbitrary and is load-bearing:
+`-framework` order decides `LC_LOAD_DYLIB` order, which decides the indirect
+symbol table's numbering. With it, `vpregister` and `libvcamcaptured.dylib`
+come out **byte-identical** to what the installers used to produce.
+`TweakLoader.dylib` differs in exactly two fields per slice — `LC_UUID`
+(content-derived, differs between any two links) and `LC_ID_DYLIB` (it takes no
+`-install_name`, so this defaults to the output path; nothing reads it, the
+guest loads the file by absolute path from `/var/jb/usr/lib`).
+
+---
+
+## 3. Reading files: map, don't slurp
+
+This bit cost a machine, so it is worth stating plainly.
+
+`Data(contentsOf:)` with no options reads the whole file into resident memory.
+Fine for a plist; ruinous for what this project opens:
+
+* `DSCLocalSymbolTable` parsed `dyld_shared_cache_arm64e.symbols`, **1.17 GB**
+ on iOS 27, reading its string table and nlist array whole.
+* `DSCChunkSet.findStringVMAs` read each executable mapping whole — ~130 MB
+ each, two dozen of them — so one full-cache string search was **3.3 GB** of
+ `Malloc Large`.
+* `ManifestHashPatcher` hashes every component the build identity names, and
+ that list includes `OS`: a **ten gigabyte** filesystem image.
+
+All three map now. Measured on `DSCFlatAddressingTests`, footprint went
+**3419 MB → 13 MB**. RSS is still ~1.1 GB and that is the point: it is clean
+file-backed pages the kernel reclaims for free, not dirty heap it has to swap.
+
+Two details that are easy to get wrong:
+
+1. **Do not slice a mapping and hand it to `loadLE(_:at:)`.** That helper
+ passes its offset to `Data.copyBytes(to:from:)`, whose range is in the
+ collection's index space — correct only where `startIndex` is zero, which a
+ slice of a mapping never is. `DSCLocalSymbolTable` holds one mapping and
+ *offsets into it* for this reason.
+
+2. **Never map a buffer you will mutate and write back over its own file.**
+ `Data.write(to:)` replaces the file the buffer is mapped from; the mapping
+ is invalidated underneath the write that is reading through it, and the next
+ page fault is a **SIGBUS** — a killed process, no failed assertion, nothing
+ in the log. Those reads say so in the spelling:
+ `Data(contentsOfFileToRewrite:)`, defined once in
+ `sources/FirmwarePatcher/Binary/InPlaceRewrite.swift`.
+ `VPhoneSigner` keeps its mapping and is safe *because* it renames over the
+ file rather than truncating it; rename does not invalidate a mapping.
+
+`make check-aux` enforces the rule for `sources/`. **`tests/` is deliberately
+outside it**: a test opens a committed fixture of a few megabytes, so mapping
+buys nothing, and those fixtures are exactly what the patch tests mutate and
+write back over. Holding the tests to the production rule would trade a memory
+problem they do not have for a SIGBUS they would.
+
+---
+
+## 4. The gates
+
+`make check-aux`. None of them proves self-containment alone.
+
+| gate | what it does |
+| --- | --- |
+| 0 | every script declares a tier on line 2 |
+| 1 | recursive `otool -L` over the bundle; every absolute path fails, including one that resolves inside the bundle today |
+| 1b | the same, on a copy moved elsewhere and renamed |
+| 1c | the bundle holds the dist manifest and **nothing else** — a build-tier file inside it is a hard failure, and so is a stale file the manifest does not name |
+| 2 | per-tier source scan: `PATH` lookups, Homebrew paths, dist scripts exec'ing build-tier scripts, unmapped reads in `sources/`, and any interpreter anywhere |
+| 3 | each entry binary doing its smallest real job under `env -i PATH=/usr/bin:/bin` |
+
+**Gate 4 is still missing**: a machine with no Homebrew at all, running a
+matrix of real work. Gate 3 clears the environment but `/opt/homebrew` is still
+on disk, so it cannot see a hardcoded absolute path. Do not read a green run
+here as "it will work elsewhere".
+
+---
+
+## 5. Verified
+
+Against the real tools, not by inspection:
+
+* `fw urls --device iPhone17,3` — identical 31 URLs to `ipsw download ipsw --urls`.
+* `fw aea-key` — byte-identical to `ipsw fw aea --key`, and it decrypted both
+ 24A435 volumes (the 8.8 GB rootfs and the 2.3 GB SystemOS cryptex) with
+ `/usr/bin/aea`.
+* `fw im4p-create` — byte-identical containers for `isys`/`0`, `trst`/`1`,
+ `msys`/`0`; `im4p-extract` round-trips to the original payload.
+* `fw seal-tool --version 26.1` — a working arm64e `apfs_sealvolume` out of an
+ 18 GB remote macOS IPSW, in 7.4 s, from a **relocated** bundle under
+ `env -i PATH=/usr/bin:/bin`.
+* `fw_prepare.sh --list` — the full firmware matrix, same conditions.
+* `vphone-cli sign` — same CodeDirectory, same CDHash as `ldid` on the real
+ `vphoned`. (The CMS blob differs; it always has. The cdhash is what AMFI
+ checks.)
+* All gates green, `DIST_REMAINING` empty.
+* **641 tests in 113 suites, zero issues, exit 0.**
+
+### Not verified
+
+**A CFW install has not been run end to end since this change.** It needs root
+and a powered-off VM. The pieces it touches were each checked in isolation, but
+the flow — `cfw install --variant jb` staging the five prebuilt guest binaries,
+signing them with `vphone-cli sign`, unpacking the bootstrap with
+`vphone-archive` — has not been driven once. **That is the first thing to do.**
+
+---
+
+## 6. Test fixtures — where they came from
+
+`ipsws/` is gitignored; the fixtures are large and must be rebuilt locally.
+
+* `ipsws/ref_extract/macho_pristine/` and `ref_extract/dsc_pristine/` came out
+ of `iPhone17,3_27.0_24A435_Restore.ipsw`: the six Mach-Os from the rootfs
+ (`043-69462-653.dmg.aea`), the 6.7 GB shared cache from the SystemOS cryptex
+ (`043-70113-702.dmg.aea`). Both decrypted with `vphone-cli fw aea-key` plus
+ `/usr/bin/aea`.
+ **24A435 is the right build and that is checkable**: `watchdogd` hashes to
+ `0309b868…` and `mobileactivationd` to `89233513…`, which are the digests
+ frozen in `DSCHVVMMPatcherTests`. If those do not match, the fixture is from
+ the wrong firmware and nothing below it is a comparison.
+* `ipsws/patch_refactor_input/raw_payloads/` was extracted from a prepared
+ 26.1/23B85 restore tree with `vphone-cli fw im4p-extract`.
+* `ipsws/patch_refactor_input/reference_patches/` **cannot be regenerated.** It
+ is the output of the deleted Python patchers. `PatchComparisonTests` skips
+ when it is absent, with that written into the file. What still guards those
+ patchers is the frozen-digest work in the same directory, which carries its
+ reference values in the source rather than in a file nobody has.
+
+---
+
+## 7. Left open
+
+* **`cfw-kit` is build tier**, declared in writing in `cfw-kit/run.sh` with the
+ reason: it still reaches for `ldid`/`gtar`/`zstd`/`ipsw` the way
+ `scripts/cfw_install*.sh` did before they were cleaned out, so it does not
+ ship and a dist user cannot run it. Whether it gets the same treatment or
+ stays a development tool is a P3 question, left open on purpose. **Do not add
+ it to `dist_manifest.sh` until it is cleaned.**
+* **D4 — the host-side shell.** The dist tier is still ten `.sh` files. Turning
+ them into Swift is the next line of work and is deliberately not started
+ here; the tiers and the gates are what make it possible to do it one script
+ at a time without losing track of what ships.
+* **Gate 4**, above.
+* `git-lfs` is the one Homebrew formula left in `setup_machine.sh`, and it is
+ not a tool this project runs — it is what `git clone` needs for
+ `scripts/resources`.
diff --git a/research/intg_update_status.md b/research/intg_update_status.md
index 28d0a333..5b546226 100644
--- a/research/intg_update_status.md
+++ b/research/intg_update_status.md
@@ -10,15 +10,21 @@
> 7.5%; both were stale by several commits. Every number below was measured with
> a command, and the commands are in "How these were counted" at the end so the
> next person can disagree with the measurement rather than the prose.
+>
+> **Updated at `a908f81`**, after D2 and D3 landed. The D2/D3 rows below are
+> rewritten; the D4 row is not, and the detail is in
+> [`d2_d3_self_containment.md`](./d2_d3_self_containment.md), which is the
+> handover for that work — what the tiers are, what each removed program was
+> replaced by, what was verified against the real tool, and what is left open.
## Where the four delivery lines stand
| line | plan's completion bar | now |
| --- | --- | --- |
| **D1** Python → zero | hard gate, 100%, achieved at **P2.4** | **6,070 / 6,070 lines — done.** No `.py` tracked, no heredoc, no runtime `python3` |
-| **D2** self-contained admission rule | `make check-aux` green | gates 1 and 1b **green**; gate 2 has **38 registered** items (**35** once P2's deletions land) and 0 unregistered; gate 4 does not exist |
-| **D3** drop third-party programs | gtar/bsdtar/unzip/zstd/ldid/… | replacements all exist and the `.app` ships **3 binaries and no tools**; the installer shell still resolves `gtar` / `zstd` / `ldid` itself |
-| **D4** shell → zero | P3 required, P4 in scope | **0%**, and it grew: **7,173 lines** in 27 files, from 5,895 in 22 (**7,072 in 25** once P2's deletions land) |
+| **D2** self-contained admission rule | `make check-aux` green | **green, and the rule is now per tier.** Gates 0, 1, 1b, 1c, 2 and 3 all pass; the dist tier's registered-exception list is **empty**. Gate 4 (a machine with no Homebrew) still does not exist |
+| **D3** drop third-party programs | gtar/bsdtar/unzip/zstd/ldid/… | **done for the dist tier.** `ldid`, `gtar`, `zstd`, `tar`, `ipsw`, `aria2c`, `wget`, `xcrun` and the bundled `trustcache` are all gone from what ships; `setup_tools.sh` installs no Homebrew formula at all, and `setup_machine.sh` is down to `git-lfs`, which `git clone` needs rather than this project |
+| **D4** shell → zero | P3 required, P4 in scope | **0%.** The dist tier is ten `.sh` files; the tiers and gates are what make it possible to convert them one at a time without losing track of what ships |
D4 going up is not an accounting artifact. `cfw-kit/` (1,036 lines) and
`scripts/check_aux.sh` (327) are both new on this branch; everything else nets
@@ -37,9 +43,9 @@ is 2,396 lines against 2,410 at the branch base.
| — | AMFI bypass moved out of the project | ✅ docs in 6 languages |
| **P0** | 455 lines of Python | ✅ **complete** |
| P0.5 | `VPhoneArchive` + `vphone-archive` | ✅ library, binary, tests, fingerprint tool |
-| P0.5 | switch the archive call sites | ◐ **package side done, shell side not.** `FirmwarePatcher` has no `tar` calls left; `$TAR` in `cfw_install*.sh` and `cfw-kit/lib/common.sh` still finds `gtar` |
-| P0.5 | `VPhoneSign`, drop `ldid` | ◐ **Swift side done.** `--use-ldid` and `VPhoneSignLdid.swift` deleted at `a779f4e`; the tests compare against frozen ldid digests rather than a live one. `cfw-kit` and the Makefile still look `ldid` up on `PATH` |
-| P0.5 | admission gates 1–3 | ✅ `make check-aux`; gate 1 now passes |
+| P0.5 | switch the archive call sites | ✅ **done at `356bec6`.** `$TAR` is gone from `cfw_install*.sh`; only `cfw-kit` still finds `gtar`, and it is build tier and does not ship |
+| P0.5 | `VPhoneSign`, drop `ldid` | ✅ **done at `356bec6`.** The installers and the Makefile call `vphone-cli sign`; nothing looks `ldid` up any more except `cfw-kit` |
+| P0.5 | admission gates | ✅ **`make check-aux`, six gates, all green, dist list empty** — see [`d2_d3_self_containment.md`](./d2_d3_self_containment.md) |
| **P1.0–1.5** | CFW patchers | ✅ **complete** — `scripts/patchers/` deleted at `d90371a`, 26 files / 6,539 lines into 24 `vphone-cli cfw` verbs |
| **P2.0** | can libirecovery see the virtual DFU endpoint? | ✅ **yes** — `research/p2_dfu_spike.md` |
| **P2.1–2.2** | vendor libirecovery + idevicerestore | ✅ `sources/MobileRecoveryCore`, `sources/MobileRestoreCore` |
@@ -90,7 +96,18 @@ Two corrections to the earlier ledger's arithmetic:
## D2, counted
-`zsh scripts/check_aux.sh --fast`:
+> Superseded at `a908f81`. The counts below were taken when the gate had ONE
+> flat list applied to the whole repository — which is exactly the thing that
+> made them hard to read: `xcrun` in a build script and `xcrun` in something
+> the `.app` ships counted the same, so "38 registered items" did not say
+> whether the product was any closer to standing on its own. Kept because the
+> shape of the old list is what the tier split was a response to.
+>
+> **Now**: `zsh scripts/check_aux.sh` reports gates 0, 1, 1b, 1c, 2 and 3 all
+> green, with the **dist tier's registered list empty**. See
+> [`d2_d3_self_containment.md`](./d2_d3_self_containment.md).
+
+`zsh scripts/check_aux.sh --fast`, as of the previous revision:
- **Gate 1 (dependency closure)** and **gate 1b (relocation)** both report
nothing. This is the change since the last revision of this file, which
@@ -108,10 +125,12 @@ Two corrections to the earlier ledger's arithmetic:
- **Gate 3** was skipped here (`--fast`). CI must not skip it.
- **Gate 4 — a machine with no Homebrew — still does not exist**, and is still
the only thing that can support "it works elsewhere". The script says so
- itself. Gates 1–3 are necessary and not sufficient.
+ itself. The other gates are necessary and not sufficient. **This is still
+ true at `a908f81`.**
-`ipsw` and `aea` are deliberately out of scope this round and are not counted
-against D3.
+`ipsw` and `aea` were out of scope for that round. They are not now: `ipsw` is
+gone, and `aea` stayed because `/usr/bin/aea` is part of macOS — it is called
+by absolute path, so there is nothing left to look up.
## D4, counted