Run a virtual iPhone on an Apple Silicon Mac.
Swift
docs: the D2/D3 handover, and un-stale the ledger
Lakr4cb4349f2996
diff --git a/research/d2_d3_self_containment.md b/research/d2_d3_self_containment.mdnew file mode 100644index 00000000..d31f7cec--- /dev/null+++ b/research/d2_d3_self_containment.md@@ -0,0 +1,222 @@+# D2 / D3 — the three tiers, and a dist runtime with nothing outside macOS in it++Status as of `a908f81` on `vphone-intg-update`. This is the handover for the+self-containment work: what the rule actually is, what was removed and where+each thing went, what is verified and how, and what is left.++---++## 1. The rule was never one rule++`check_aux.sh` used to hold a single list of "programs we still reach for",+applied to the whole repository. That made `xcrun` in a build script and+`xcrun` in something the `.app` ships count the same, so the list shrinking+said nothing about whether the product was any closer to standing on its own.++Three environments run code here, and they have different rules:++| tier | who runs it | may use |+| --- | --- | --- |+| **build** | the machine that builds the `.app` — a developer's Mac or CI | Xcode, `xcrun`, clang, swift, git, **and Homebrew**. Nothing in this tier ships. |+| **dist** | the shipped `.app`, on a stranger's clean macOS | `/usr/lib`, `/System`, and what is inside the bundle. No Homebrew, no Xcode, **no `PATH` lookup at all**. |+| **guest** | inside the VM | out of host self-containment scope. These ship as payload to be copied in, never run on the host. |++Every script declares its tier on **line 2**:++```sh+#!/bin/zsh+# vphone-tier: dist+```++`scripts/dist_manifest.sh` reads those declarations and prints the dist+payload. `scripts/build.sh` and `make bundle` both stage from it, so the+bundler is an **allowlist**. It used to be a list of exclusions, which is how+the `.app` came to carry `build.sh`, `check_aux.sh` and `setup_tools.sh` — the+last of which runs `brew install`. An undeclared script now ships nowhere and+gate 0 says so.++> `make bundle` stages `Contents/Resources` now. It did not, and `build.sh`+> did, so `make check-aux` was inspecting a five-binary bundle while users got+> twenty scripts and a Homebrew-linked `trustcache`. **The gate was green+> because it was looking at the wrong artifact.** If you add a bundling step,+> add it to both or the gate goes blind again.++---++## 2. What the dist tier reached for, and where each one went++| was | now | checked by |+| --- | --- | --- |+| `xcrun` (five iOS binaries cross-compiled at CFW-install time) | `scripts/guest_binaries.mk` builds them at build time; they ship in `Contents/Resources/guest` | gate 2, and a byte comparison against the old invocations |+| `ldid` | `vphone-cli sign` / `dump-entitlements` | same CodeDirectory and CDHash as `ldid` on the real `vphoned`; gate 3 signs with `env -i PATH=/usr/bin:/bin` |+| `gtar`, `zstd`, `tar` | `vphone-archive` (`extract`, `create`, `decompress`) | gate 3 round-trips a `.tzst` with no `zstd` on `PATH` |+| `ipsw fw aea --key` | `vphone-cli fw aea-key` | byte-identical to `ipsw` on a real 24A435 archive |+| `ipsw img4 im4p create/extract` | `vphone-cli fw im4p-create` / `im4p-extract`, and `IM4P` directly in `CryptexFilesystemPatcherSealing` | byte-identical containers for `isys`/`trst`/`msys`; gate 3 round-trips one |+| `ipsw download ipsw --urls` | `vphone-cli fw urls` | same 31 URLs for `iPhone17,3`, diffed against `ipsw` |+| `ipsw download appledb` + partial zip extract | `vphone-cli fw seal-tool` over `VPhoneRemoteZip` | pulls `apfs_sealvolume` out of an 18 GB remote IPSW in ~7 s |+| `aria2c`, `wget` | deleted; `/usr/bin/curl` resumes with `-C -` | gate 2 |+| `trustcache` (bundled, linked `/opt/homebrew`'s `libcrypto.3`) | **deleted** — nothing ever invoked it; the trust cache comes from `cryptexctl` in `/System/Library/SecurityResearch` | gate 1 |++`DIST_REMAINING` in `check_aux.sh` — the list a release requires to be empty —+**is empty**. Keep it that way; adding a line is a decision to ship something+that does not work on a clean Mac.++### The compile/install split++Five Mach-Os run inside the guest: `vphoned`, `TweakLoader.dylib`,+`vpregister`, `libvcamcaptured.dylib`, `libcamfix.dylib`. Every one used to be+compiled at CFW-install time — three by the installers, and `vphoned` twice+more (again by `FirmwarePatcher.buildVphoned`). That made a full Xcode install+a prerequisite for putting firmware on a VM.++**Compiling moved to build time. Signing did not, and cannot**: it uses the+target VM's own `cfw_input/signcert.p12`, which does not exist until a VM does.+`cfw install` still runs in dist, as it must — it is something `vphone-cli`+does on the user's machine.++`scripts/guest_binaries.mk` keeps the installers' own argument order,+framework list included. That looks arbitrary and is load-bearing:+`-framework` order decides `LC_LOAD_DYLIB` order, which decides the indirect+symbol table's numbering. With it, `vpregister` and `libvcamcaptured.dylib`+come out **byte-identical** to what the installers used to produce.+`TweakLoader.dylib` differs in exactly two fields per slice — `LC_UUID`+(content-derived, differs between any two links) and `LC_ID_DYLIB` (it takes no+`-install_name`, so this defaults to the output path; nothing reads it, the+guest loads the file by absolute path from `/var/jb/usr/lib`).++---++## 3. Reading files: map, don't slurp++This bit cost a machine, so it is worth stating plainly.++`Data(contentsOf:)` with no options reads the whole file into resident memory.+Fine for a plist; ruinous for what this project opens:++* `DSCLocalSymbolTable` parsed `dyld_shared_cache_arm64e.symbols`, **1.17 GB**+ on iOS 27, reading its string table and nlist array whole.+* `DSCChunkSet.findStringVMAs` read each executable mapping whole — ~130 MB+ each, two dozen of them — so one full-cache string search was **3.3 GB** of+ `Malloc Large`.+* `ManifestHashPatcher` hashes every component the build identity names, and+ that list includes `OS`: a **ten gigabyte** filesystem image.++All three map now. Measured on `DSCFlatAddressingTests`, footprint went+**3419 MB → 13 MB**. RSS is still ~1.1 GB and that is the point: it is clean+file-backed pages the kernel reclaims for free, not dirty heap it has to swap.++Two details that are easy to get wrong:++1. **Do not slice a mapping and hand it to `loadLE(_:at:)`.** That helper+ passes its offset to `Data.copyBytes(to:from:)`, whose range is in the+ collection's index space — correct only where `startIndex` is zero, which a+ slice of a mapping never is. `DSCLocalSymbolTable` holds one mapping and+ *offsets into it* for this reason.++2. **Never map a buffer you will mutate and write back over its own file.**+ `Data.write(to:)` replaces the file the buffer is mapped from; the mapping+ is invalidated underneath the write that is reading through it, and the next+ page fault is a **SIGBUS** — a killed process, no failed assertion, nothing+ in the log. Those reads say so in the spelling:+ `Data(contentsOfFileToRewrite:)`, defined once in+ `sources/FirmwarePatcher/Binary/InPlaceRewrite.swift`.+ `VPhoneSigner` keeps its mapping and is safe *because* it renames over the+ file rather than truncating it; rename does not invalidate a mapping.++`make check-aux` enforces the rule for `sources/`. **`tests/` is deliberately+outside it**: a test opens a committed fixture of a few megabytes, so mapping+buys nothing, and those fixtures are exactly what the patch tests mutate and+write back over. Holding the tests to the production rule would trade a memory+problem they do not have for a SIGBUS they would.++---++## 4. The gates++`make check-aux`. None of them proves self-containment alone.++| gate | what it does |+| --- | --- |+| 0 | every script declares a tier on line 2 |+| 1 | recursive `otool -L` over the bundle; every absolute path fails, including one that resolves inside the bundle today |+| 1b | the same, on a copy moved elsewhere and renamed |+| 1c | the bundle holds the dist manifest and **nothing else** — a build-tier file inside it is a hard failure, and so is a stale file the manifest does not name |+| 2 | per-tier source scan: `PATH` lookups, Homebrew paths, dist scripts exec'ing build-tier scripts, unmapped reads in `sources/`, and any interpreter anywhere |+| 3 | each entry binary doing its smallest real job under `env -i PATH=/usr/bin:/bin` |++**Gate 4 is still missing**: a machine with no Homebrew at all, running a+matrix of real work. Gate 3 clears the environment but `/opt/homebrew` is still+on disk, so it cannot see a hardcoded absolute path. Do not read a green run+here as "it will work elsewhere".++---++## 5. Verified++Against the real tools, not by inspection:++* `fw urls --device iPhone17,3` — identical 31 URLs to `ipsw download ipsw --urls`.+* `fw aea-key` — byte-identical to `ipsw fw aea --key`, and it decrypted both+ 24A435 volumes (the 8.8 GB rootfs and the 2.3 GB SystemOS cryptex) with+ `/usr/bin/aea`.+* `fw im4p-create` — byte-identical containers for `isys`/`0`, `trst`/`1`,+ `msys`/`0`; `im4p-extract` round-trips to the original payload.+* `fw seal-tool --version 26.1` — a working arm64e `apfs_sealvolume` out of an+ 18 GB remote macOS IPSW, in 7.4 s, from a **relocated** bundle under+ `env -i PATH=/usr/bin:/bin`.+* `fw_prepare.sh --list` — the full firmware matrix, same conditions.+* `vphone-cli sign` — same CodeDirectory, same CDHash as `ldid` on the real+ `vphoned`. (The CMS blob differs; it always has. The cdhash is what AMFI+ checks.)+* All gates green, `DIST_REMAINING` empty.+* **641 tests in 113 suites, zero issues, exit 0.**++### Not verified++**A CFW install has not been run end to end since this change.** It needs root+and a powered-off VM. The pieces it touches were each checked in isolation, but+the flow — `cfw install --variant jb` staging the five prebuilt guest binaries,+signing them with `vphone-cli sign`, unpacking the bootstrap with+`vphone-archive` — has not been driven once. **That is the first thing to do.**++---++## 6. Test fixtures — where they came from++`ipsws/` is gitignored; the fixtures are large and must be rebuilt locally.++* `ipsws/ref_extract/macho_pristine/` and `ref_extract/dsc_pristine/` came out+ of `iPhone17,3_27.0_24A435_Restore.ipsw`: the six Mach-Os from the rootfs+ (`043-69462-653.dmg.aea`), the 6.7 GB shared cache from the SystemOS cryptex+ (`043-70113-702.dmg.aea`). Both decrypted with `vphone-cli fw aea-key` plus+ `/usr/bin/aea`.+ **24A435 is the right build and that is checkable**: `watchdogd` hashes to+ `0309b868…` and `mobileactivationd` to `89233513…`, which are the digests+ frozen in `DSCHVVMMPatcherTests`. If those do not match, the fixture is from+ the wrong firmware and nothing below it is a comparison.+* `ipsws/patch_refactor_input/raw_payloads/` was extracted from a prepared+ 26.1/23B85 restore tree with `vphone-cli fw im4p-extract`.+* `ipsws/patch_refactor_input/reference_patches/` **cannot be regenerated.** It+ is the output of the deleted Python patchers. `PatchComparisonTests` skips+ when it is absent, with that written into the file. What still guards those+ patchers is the frozen-digest work in the same directory, which carries its+ reference values in the source rather than in a file nobody has.++---++## 7. Left open++* **`cfw-kit` is build tier**, declared in writing in `cfw-kit/run.sh` with the+ reason: it still reaches for `ldid`/`gtar`/`zstd`/`ipsw` the way+ `scripts/cfw_install*.sh` did before they were cleaned out, so it does not+ ship and a dist user cannot run it. Whether it gets the same treatment or+ stays a development tool is a P3 question, left open on purpose. **Do not add+ it to `dist_manifest.sh` until it is cleaned.**+* **D4 — the host-side shell.** The dist tier is still ten `.sh` files. Turning+ them into Swift is the next line of work and is deliberately not started+ here; the tiers and the gates are what make it possible to do it one script+ at a time without losing track of what ships.+* **Gate 4**, above.+* `git-lfs` is the one Homebrew formula left in `setup_machine.sh`, and it is+ not a tool this project runs — it is what `git clone` needs for+ `scripts/resources`.diff --git a/research/intg_update_status.md b/research/intg_update_status.mdindex 28d0a333..5b546226 100644--- a/research/intg_update_status.md+++ b/research/intg_update_status.md@@ -10,15 +10,21 @@> 7.5%; both were stale by several commits. Every number below was measured with> a command, and the commands are in "How these were counted" at the end so the> next person can disagree with the measurement rather than the prose.+>+> **Updated at `a908f81`**, after D2 and D3 landed. The D2/D3 rows below are+> rewritten; the D4 row is not, and the detail is in+> [`d2_d3_self_containment.md`](./d2_d3_self_containment.md), which is the+> handover for that work — what the tiers are, what each removed program was+> replaced by, what was verified against the real tool, and what is left open.## Where the four delivery lines stand| line | plan's completion bar | now || --- | --- | --- || **D1** Python → zero | hard gate, 100%, achieved at **P2.4** | **6,070 / 6,070 lines — done.** No `.py` tracked, no heredoc, no runtime `python3` |-| **D2** self-contained admission rule | `make check-aux` green | gates 1 and 1b **green**; gate 2 has **38 registered** items (**35** once P2's deletions land) and 0 unregistered; gate 4 does not exist |-| **D3** drop third-party programs | gtar/bsdtar/unzip/zstd/ldid/… | replacements all exist and the `.app` ships **3 binaries and no tools**; the installer shell still resolves `gtar` / `zstd` / `ldid` itself |-| **D4** shell → zero | P3 required, P4 in scope | **0%**, and it grew: **7,173 lines** in 27 files, from 5,895 in 22 (**7,072 in 25** once P2's deletions land) |+| **D2** self-contained admission rule | `make check-aux` green | **green, and the rule is now per tier.** Gates 0, 1, 1b, 1c, 2 and 3 all pass; the dist tier's registered-exception list is **empty**. Gate 4 (a machine with no Homebrew) still does not exist |+| **D3** drop third-party programs | gtar/bsdtar/unzip/zstd/ldid/… | **done for the dist tier.** `ldid`, `gtar`, `zstd`, `tar`, `ipsw`, `aria2c`, `wget`, `xcrun` and the bundled `trustcache` are all gone from what ships; `setup_tools.sh` installs no Homebrew formula at all, and `setup_machine.sh` is down to `git-lfs`, which `git clone` needs rather than this project |+| **D4** shell → zero | P3 required, P4 in scope | **0%.** The dist tier is ten `.sh` files; the tiers and gates are what make it possible to convert them one at a time without losing track of what ships |D4 going up is not an accounting artifact. `cfw-kit/` (1,036 lines) and`scripts/check_aux.sh` (327) are both new on this branch; everything else nets@@ -37,9 +43,9 @@ is 2,396 lines against 2,410 at the branch base.| — | AMFI bypass moved out of the project | ✅ docs in 6 languages || **P0** | 455 lines of Python | ✅ **complete** || P0.5 | `VPhoneArchive` + `vphone-archive` | ✅ library, binary, tests, fingerprint tool |-| P0.5 | switch the archive call sites | ◐ **package side done, shell side not.** `FirmwarePatcher` has no `tar` calls left; `$TAR` in `cfw_install*.sh` and `cfw-kit/lib/common.sh` still finds `gtar` |-| P0.5 | `VPhoneSign`, drop `ldid` | ◐ **Swift side done.** `--use-ldid` and `VPhoneSignLdid.swift` deleted at `a779f4e`; the tests compare against frozen ldid digests rather than a live one. `cfw-kit` and the Makefile still look `ldid` up on `PATH` |-| P0.5 | admission gates 1–3 | ✅ `make check-aux`; gate 1 now passes |+| P0.5 | switch the archive call sites | ✅ **done at `356bec6`.** `$TAR` is gone from `cfw_install*.sh`; only `cfw-kit` still finds `gtar`, and it is build tier and does not ship |+| P0.5 | `VPhoneSign`, drop `ldid` | ✅ **done at `356bec6`.** The installers and the Makefile call `vphone-cli sign`; nothing looks `ldid` up any more except `cfw-kit` |+| P0.5 | admission gates | ✅ **`make check-aux`, six gates, all green, dist list empty** — see [`d2_d3_self_containment.md`](./d2_d3_self_containment.md) || **P1.0–1.5** | CFW patchers | ✅ **complete** — `scripts/patchers/` deleted at `d90371a`, 26 files / 6,539 lines into 24 `vphone-cli cfw` verbs || **P2.0** | can libirecovery see the virtual DFU endpoint? | ✅ **yes** — `research/p2_dfu_spike.md` || **P2.1–2.2** | vendor libirecovery + idevicerestore | ✅ `sources/MobileRecoveryCore`, `sources/MobileRestoreCore` |@@ -90,7 +96,18 @@ Two corrections to the earlier ledger's arithmetic:## D2, counted-`zsh scripts/check_aux.sh --fast`:+> Superseded at `a908f81`. The counts below were taken when the gate had ONE+> flat list applied to the whole repository — which is exactly the thing that+> made them hard to read: `xcrun` in a build script and `xcrun` in something+> the `.app` ships counted the same, so "38 registered items" did not say+> whether the product was any closer to standing on its own. Kept because the+> shape of the old list is what the tier split was a response to.+>+> **Now**: `zsh scripts/check_aux.sh` reports gates 0, 1, 1b, 1c, 2 and 3 all+> green, with the **dist tier's registered list empty**. See+> [`d2_d3_self_containment.md`](./d2_d3_self_containment.md).++`zsh scripts/check_aux.sh --fast`, as of the previous revision:- **Gate 1 (dependency closure)** and **gate 1b (relocation)** both reportnothing. This is the change since the last revision of this file, which@@ -108,10 +125,12 @@ Two corrections to the earlier ledger's arithmetic:- **Gate 3** was skipped here (`--fast`). CI must not skip it.- **Gate 4 — a machine with no Homebrew — still does not exist**, and is stillthe only thing that can support "it works elsewhere". The script says so- itself. Gates 1–3 are necessary and not sufficient.+ itself. The other gates are necessary and not sufficient. **This is still+ true at `a908f81`.**-`ipsw` and `aea` are deliberately out of scope this round and are not counted-against D3.+`ipsw` and `aea` were out of scope for that round. They are not now: `ipsw` is+gone, and `aea` stayed because `/usr/bin/aea` is part of macOS — it is called+by absolute path, so there is nothing left to look up.## D4, counted