Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

sysimage/unpack.sh: the IPSW is the user's own; no script downloads Apple's files any more

shimoku7a397f799b05
diff --git a/.gitignore b/.gitignore
index 9f5664a..51680ca 100644
--- a/.gitignore
+++ b/.gitignore
@@ -63,7 +63,7 @@ __pycache__/
!/userland/var/
/sysimage/*
-!/sysimage/fetch.sh
+!/sysimage/unpack.sh
!/sysimage/*.py
# of xnu-deps only what is ours; setup.sh fetches and builds the rest
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 87ca5b1..74128f5 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -4,8 +4,8 @@
kernel, loader and user space stay device-neutral and take their numbers from the device tree
and the phone's own Android.
- Nothing of Apple's in the tree: no iOS files, images, caches, screenshots or headers derived
- from the firmware. The build fetches what it needs from the user's own IPSW
- (`sysimage/fetch.sh`, `xnuloader/mkidentity.sh`). CI refuses pictures and iOS files.
+ from the firmware. The build takes what it needs from the IPSW the user got
+ themselves (`sysimage/unpack.sh`, `xnuloader/mkidentity.sh`); no script downloads Apple's files. CI refuses pictures and iOS files.
- Nothing of the vendor's either: firmware is pulled from the user's phone (`make firmware`).
- Kernel changes are made in `xnu-src/` and exported with `xnu/export.sh` before committing;
the patch must apply to Apple's `xnu-8796.121.2` tag (CI checks).
diff --git a/LEGAL.md b/LEGAL.md
index 45e4422..57ed30f 100644
--- a/LEGAL.md
+++ b/LEGAL.md
@@ -56,9 +56,9 @@ program that can write to flash, and CI fails if one appears that is not listed.
## What this repository does NOT contain and never distributes
- **No part of iOS**: no iOS binary, image, cache, font, picture or screenshot. The user-space
- half of the system image is built on your machine from the iOS 16.5 IPSW, which
- `sysimage/fetch.sh` downloads from Apple's public servers under Apple's license, to your
- machine only. Nothing of it is committed here.
+ half of the system image is built on your machine from the iOS 16.5 IPSW, which you
+ get yourself, under Apple's license, and hand to `sysimage/unpack.sh` on your machine; no script
+ here downloads it. Nothing of it is committed here.
- **No device firmware**: the GPU, Wi-Fi, audio DSP, modem, haptics and camera firmware is the
vendor's (Qualcomm/Xiaomi). It is pulled from your own phone (`make firmware`) and is git-ignored.
- **No keys, certificates, activation data, or anything to impersonate a genuine Apple device to
diff --git a/Makefile b/Makefile
index 3d05fce..d44af8f 100644
--- a/Makefile
+++ b/Makefile
@@ -2,7 +2,7 @@
# below it; run the script itself for its options.
#
# make help
-# make fetch the OS files from the iOS 16.5 IPSW into sysimage/ (sysimage/fetch.sh)
+# make unpack IPSW=F the OS files out of your own iOS 16.5 IPSW into sysimage/ (sysimage/unpack.sh)
# make images userland/root.apfs and userland/system.apfs (userland/build.sh)
# make kernel the kernel from source (xnu/apply.sh, xnu-deps/setup.sh, xnu-build.sh)
# make loader the boot loader, with identity.h made from your IPSW (xnuloader/build.sh)
@@ -50,7 +50,7 @@ IMAGE_OPTS ?= REST=1
ESP_DIR ?= build/esp
PAYLOAD ?= qemu/payload.bin
-.PHONY: devices help fetch images kernel loader loader-open hello qemu qemu-open phone prebuilt clean all doctor defconfig config savedefconfig esp
+.PHONY: devices help unpack images kernel loader loader-open hello qemu qemu-open phone prebuilt clean all doctor defconfig config savedefconfig esp
help:
@awk '{ if (!/^#/) exit; sub(/^# ?/, ""); print }' Makefile
@@ -60,8 +60,8 @@ doctor:
all: kernel loader images
-fetch:
- sysimage/fetch.sh
+unpack:
+ sysimage/unpack.sh $(IPSW)
images:
$(IMAGE_OPTS) userland/build.sh
diff --git a/NOTICE b/NOTICE
index 9bb9970..ca2335c 100644
--- a/NOTICE
+++ b/NOTICE
@@ -22,8 +22,8 @@ In the repository:
Fetched at build time, never committed:
-- the iOS 16.5 IPSW (Apple): the user space of the system image; sysimage/fetch.sh downloads it from
- Apple's public servers; its license is Apple's.
+- the iOS 16.5 IPSW (Apple): the user space of the system image; the user gets it and
+ sysimage/unpack.sh unpacks it on their machine; nothing downloads it; its license is Apple's.
- the iPhoneOS 16.5 SDK from github.com/theos/sdks, and theos's cctools (Apple's open-source cctools): the
kernel's and user space's build tools.
- the phone's firmware (GPU, Wi-Fi, audio DSP, modem, haptics, cameras): the vendor's, pulled from the
diff --git a/README.md b/README.md
index 4c794d0..b4ea8c2 100644
--- a/README.md
+++ b/README.md
@@ -62,9 +62,8 @@ make phone SECS=600 # a 10-minute session on the phone, in RAM
| Repository | What |
|---|---|
| [hakios](https://github.com/oosexgang/hakios) | **common**: the kernel port, every common driver, the loader, user space, the tools, this documentation, and `devices/qemu` |
-| [hakios-redwood](https://github.com/oosexgang/hakios-redwood) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script, and its SoC's layer (`soc/`: the Qualcomm SC7280 family) |
+| [hakios-sm7325](https://github.com/oosexgang/hakios-sm7325) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script, and its SoC's layer (`soc/`: the Qualcomm SC7280 family) |
| [hakios-uefi](https://github.com/oosexgang/hakios-uefi) | HakiOS's UEFI: Mu-Silicium with HakiOS's boot path, banner and name |
-| [hakios-web](https://github.com/oosexgang/hakios-web) | the website, with this documentation and the FAQ |
`devices/manifest.conf` in the common repository names each one and the revision it is at; `tools/hakios-device.py sync` clones them
into `devices/` and `uefi/`. One repository per device: it carries its SoC's layer, and `tools/hakios-device.py new` makes one. [How the repositories fit](docs/porting/repositories.md).
@@ -85,8 +84,8 @@ In it: only our own code and data: the patch to Apple's open-source XNU (`xnu/ha
loader, our programs and libraries, the tools, `prebuilt/` (built from this source; the loader **without** any Apple data:
`make loader` makes the one that starts iOS, on your machine, from your own IPSW), and the device and SoC directories.
-Never in it: any file of iOS (you build the system image on your machine from the IPSW, which `sysimage/fetch.sh` downloads from
-Apple's public servers without decrypting or circumventing anything); the phone's firmware (pulled from your own phone into
+Never in it: any file of iOS (you build the system image on your machine from an IPSW you get yourself and hand to
+`sysimage/unpack.sh`; nothing is downloaded for you, decrypted or circumvented); the phone's firmware (pulled from your own phone into
`fw-*/`, which git ignores); keys, certificates, activation, or anything that would make the phone pass as an iPhone to Apple.
## Independent, and Apple-free at the core
diff --git a/README.uk.md b/README.uk.md
index 8e27e68..c2fd560 100644
--- a/README.uk.md
+++ b/README.uk.md
@@ -46,8 +46,8 @@ HakiOS — це адаптація ядра Darwin (відкритий XNU ві
Чого немає і не буде:
- жодного файлу iOS: ні бінарників, ні образів, ні кешів, ні шрифтів, ні картинок, ні знімків
- екрана. Образ системи ви збираєте на своїй машині з IPSW iOS 16.5, який `sysimage/fetch.sh`
- завантажує з публічних серверів Apple, нічого не розшифровуючи й не обходячи;
+ екрана. Образ системи ви збираєте на своїй машині з IPSW iOS 16.5, який ви берете самі й
+ передаєте `sysimage/unpack.sh`; за вас нічого не завантажується, не розшифровується й не обходиться;
- прошивок телефона (GPU, Wi-Fi, аудіо-DSP, модем, вібро, камери): вони вендорські і знімаються
з вашого власного телефона в `fw-*/`, які git ігнорує;
- ключів, сертифікатів, активації чи будь-чого, що видавало б телефон за iPhone перед службами
@@ -79,7 +79,7 @@ Linux x86-64, близько 12 ГБ на диску, 7-Zip 23 або новіш
git clone https://github.com/oosexgang/hakios.git hakios
cd hakios
make doctor # чого бракує цьому комп'ютеру (дисплей QEMU, UEFI, пам'ять) і що робити
-make fetch # IPSW iOS 16.5 із серверів Apple, потрібне розпаковується (7,4 ГБ)
+make unpack IPSW=~/ваш.ipsw # ваш власний IPSW iOS 16.5, потрібне розпаковується (7,4 ГБ)
make images # образи APFS: userland/root.apfs і userland/system.apfs
make loader # завантажувач з ідентичністю iPhone з ВАШОЇ копії IPSW (потрібні clang і lld)
make qemu # сеанс на 20 хвилин
diff --git a/README.zh.md b/README.zh.md
index 57560cb..bb25f74 100644
--- a/README.zh.md
+++ b/README.zh.md
@@ -39,7 +39,7 @@ Linux 电脑上的 QEMU(`virt`,aarch64)里运行,不需要任何 Apple
永远不会有的东西:
- 任何 iOS 文件:没有二进制、镜像、缓存、字体、图片或截图。系统镜像由你在自己机器上用 iOS 16.5
- 的 IPSW 构建,`sysimage/fetch.sh` 从 Apple 的公开服务器为你下载它,不解密、不绕过任何东西;
+ 的 IPSW 构建,IPSW 由你自己获取并交给 `sysimage/unpack.sh`;项目不替你下载、不解密、不绕过任何东西;
- 手机的固件(GPU、Wi-Fi、音频 DSP、调制解调器、振动、摄像头):属于厂商,从你自己的手机取到
`fw-*/`,git 忽略这些目录;
- 密钥、证书、激活,或任何让手机在 Apple 服务面前冒充 iPhone 的东西。这里无法登录 Apple 账户,
@@ -71,7 +71,7 @@ Linux x86-64,约 12 GB 磁盘,7-Zip 23 或更新,`lz4`,`qemu-system-aarc
git clone https://github.com/oosexgang/hakios.git hakios
cd hakios
make doctor # 检查本机缺什么(QEMU 显示设备、UEFI 固件、内存)以及怎么办
-make fetch # 从 Apple 服务器取 iOS 16.5 IPSW,解包所需部分(7.4 GB)
+make unpack IPSW=~/your.ipsw # 你自己获取的 iOS 16.5 IPSW,解包所需部分(7.4 GB)
make images # APFS 镜像:userland/root.apfs 和 userland/system.apfs
make loader # 引导程序,iPhone 的标识取自你自己的 IPSW 副本(需要 clang 和 lld)
make qemu # 20 分钟的会话
diff --git a/THIRD_PARTY_LICENSES.md b/THIRD_PARTY_LICENSES.md
index f723705..66ab459 100644
--- a/THIRD_PARTY_LICENSES.md
+++ b/THIRD_PARTY_LICENSES.md
@@ -27,7 +27,7 @@ This supersedes the summary in `NOTICE`, which stays as the short, canonical lis
| Component | Fetched by | Origin | License |
|---|---|---|---|
-| iOS 16.5 IPSW (user space of the system image) | `sysimage/fetch.sh` | Apple | Apple's — user supplies, user's own copy |
+| iOS 16.5 IPSW (user space of the system image) | `sysimage/unpack.sh` | Apple | Apple's — user supplies, user's own copy |
| iPhoneOS 16.5 SDK | `xnu-deps/setup.sh` (github.com/theos/sdks) | Apple (headers) via theos | Apple's / theos |
| Apple open sources (AvailabilityVersions, libdispatch, libpthread, cctools, bootstrap_cmds) | `xnu-deps/setup.sh` | Apple | APSL 2.0 |
| MacOSX13.3 SDK (Kernel.framework DriverKit headers only) | `xnu-deps/setup.sh` | Apple via joseluisq/macosx-sdks | Apple's |
diff --git a/docs/getting-started/building.md b/docs/getting-started/building.md
index 8dbfcb0..e95c827 100644
--- a/docs/getting-started/building.md
+++ b/docs/getting-started/building.md
@@ -42,7 +42,7 @@ make loader # xnuloader/build.sh: clang and lld-link, an EFI applicatio
```
The header `xnuloader/identity.h`, the properties by which iOS tells which iPhone it is, is not kept in the repository:
-`xnuloader/mkidentity.sh` makes it at build time from the device tree in your own IPSW. `make loader-open` makes the placeholder
+`xnuloader/mkidentity.sh` makes it at build time from the device tree in your own IPSW (`sysimage/unpack.sh` leaves it in `sysimage/`). `make loader-open` makes the placeholder
loader that boots the open kernel only.
## User space and the images
diff --git a/docs/getting-started/qemu.md b/docs/getting-started/qemu.md
index f73a13c..25dc8c8 100644
--- a/docs/getting-started/qemu.md
+++ b/docs/getting-started/qemu.md
@@ -22,7 +22,7 @@ Linux x86-64, about 12 GB of disk, 7-Zip 23 or newer, `lz4`, `qemu-system-aarch6
```sh
make doctor # QEMU's display device, UEFI firmware, memory: what is missing, and what to do
-sysimage/fetch.sh # the iOS 16.5 IPSW from Apple's servers, the needed parts unpacked (7.4 GB)
+sysimage/unpack.sh ~/iPhone10,3,iPhone10,6_16.5_20F66_Restore.ipsw # YOUR IPSW: the needed parts unpacked (7.4 GB)
REST=1 userland/build.sh # the APFS images: userland/root.apfs and userland/system.apfs
make loader # the boot loader, with the iPhone's identity taken from YOUR copy of the IPSW
mkdir -p logs
diff --git a/docs/legal-review.md b/docs/legal-review.md
index 8f31457..5573cac 100644
--- a/docs/legal-review.md
+++ b/docs/legal-review.md
@@ -42,7 +42,7 @@ gate anything that contains Apple's code behind it.**
### What the build does that touches Apple's code (the exposure that is yours, not the user's)
-- `sysimage/fetch.sh` downloads parts of the IPSW from Apple's public CDN and `userland/build.sh`
+- `sysimage/unpack.sh` unpacks parts of an IPSW the user supplies (nothing is downloaded) and `userland/build.sh`
assembles a modified root image: it selects parts of iOS, replaces launchd with `stage0`, injects
our dylibs into SpringBoard and other Apple processes (`userland/src/inject*.c`), and
`xnuloader/mkidentity.py` copies the iPhone's device-tree identity so the system "believes it is
diff --git a/docs/port-pkg110.md b/docs/port-pkg110.md
index b47951d..97a9e97 100644
--- a/docs/port-pkg110.md
+++ b/docs/port-pkg110.md
@@ -81,9 +81,9 @@ the boot test itself: `./xl enter` does a `fastboot boot`, which a truly locked
from `/fw` of the system image, which `MODEM=1 userland/build.sh` fills from `fw-modem/`
(`hakios.mpss.c:1045-1110`). The modem's own file systems (modemst1/2, fsc, fsg) do come from the
phone's UFS (`ufs=2`), but in addition to the image, not instead of it. So the project's standard
- path applies: `sysimage/fetch.sh` (iOS 16.5 from Apple's public CDN, ~7.4 GB) and
+ path applies: `sysimage/unpack.sh YOUR.ipsw` (iOS 16.5, which you supply; ~7.4 GB unpacked) and
`PREBUILT=1 REST=1 MODEM=1 SYSTEM=1 userland/build.sh`.
-- **Build in WSL's own ext4, not on `/mnt/d`.** `fetch.sh` relies on 7z leaving files named
+- **Build in WSL's own ext4, not on `/mnt/d`.** `unpack.sh` relies on 7z leaving files named
`NAME:com.apple.decmpfs` (finished by `unzbm`); `:` is not a legal NTFS file-name character, and
drvfs is slow on tens of thousands of small files. A local clone in `~/hakios` plus a copy of
`fw-*` is used; the finished `root.apfs`/`system.apfs` are copied back to the D: checkout for `xl`.
@@ -104,7 +104,7 @@ the boot test itself: `./xl enter` does a `fastboot boot`, which a truly locked
In WSL's ext4 (`~/hakios`, a local clone plus `fw-*`):
-- `sysimage/fetch.sh`: ~9 min. `done: 2.9G rootfs, 26M cryptex, 11M App cryptex, 2.6G dyld shared
+- `sysimage/unpack.sh`: ~9 min. `done: 2.9G rootfs, 26M cryptex, 11M App cryptex, 2.6G dyld shared
cache` (5.7 GB in `sysimage/`); `unzbm` built by `cc`, the LZBITMAP files finished, PingFang
trimmed 74 -> 26 MB.
- `PREBUILT=1 REST=1 MODEM=1 SYSTEM=1 userland/build.sh`: ~4 min, exit 0. The small image got
diff --git a/docs/reference/tools.md b/docs/reference/tools.md
index 102da0a..92489b2 100644
--- a/docs/reference/tools.md
+++ b/docs/reference/tools.md
@@ -8,7 +8,7 @@ Every `make` target is a script below it; run the script itself for its options
|---|---|
| `make` | the list below |
| `make doctor` | checks this machine (and a phone on `adb`) for what HakiOS needs |
-| `make fetch` | the OS files from the iOS 16.5 IPSW into `sysimage/` |
+| `make unpack IPSW=F` | the OS files out of your own iOS 16.5 IPSW into `sysimage/` (nothing is downloaded) |
| `make images` | `userland/root.apfs` and `userland/system.apfs` |
| `make kernel` | the kernel from source, with the `.config`'s drivers |
| `make loader` / `make loader-open` | the boot loader with your IPSW's identity / the placeholder that boots the open kernel only |
diff --git a/docs/ru/notes.md b/docs/ru/notes.md
index c0fac64..caff9ae 100644
--- a/docs/ru/notes.md
+++ b/docs/ru/notes.md
@@ -42,8 +42,8 @@
## Быстрый старт: QEMU, без сборки ядра
В `prebuilt/` лежат собранные из этого же репозитория ядро, загрузчик и наши программы, так что нужен только
-образ системы. **Файлов iOS в репозитории нет** — скрипт сам скачивает их с серверов Apple (из IPSW iOS 16.5
-для iPhone X) и раскладывает.
+образ системы. **Файлов iOS в репозитории нет** — IPSW iOS 16.5 для iPhone X вы берёте сами,
+скрипт только распаковывает из него нужное.
Нужно: Linux x86-64, ~12 ГБ свободного места (готовое занимает ~8 ГБ), от 4 ГБ свободной памяти. Пакеты (Arch):
@@ -56,7 +56,7 @@ sudo pacman -S --needed git python 7zip lz4 gcc qemu-system-aarch64 qemu-ui-gtk
```sh
git clone [email protected]:vcvkk/darwin-redwood.git
cd darwin-redwood
-sysimage/fetch.sh # скачивает 7,4 ГБ с CDN Apple и распаковывает нужное (минут 5–15)
+sysimage/unpack.sh ~/iPhone10,3,iPhone10,6_16.5_20F66_Restore.ipsw # ваш IPSW: распаковывает нужное
REST=1 userland/build.sh # собирает образы APFS: userland/root.apfs и userland/system.apfs (2,5 ГБ)
mkdir -p logs
qemu/run-ios.sh 1200 -display gtk > logs/qemu.log # сеанс на 20 минут
@@ -65,9 +65,8 @@ qemu/run-ios.sh 1200 -display gtk > logs/qemu.log # сеанс на 20 ми
Через 2–3 минуты в окне появится экран блокировки.
**Сколько это занимает.** Измерено 20 октября на этой машине (Linux x86-64, 12 потоков, NVMe), проходом
-по этим самым командам из чистого клона: `sysimage/fetch.sh` — 1 мин 45 с, из них скачивание криптексов
-2,7 ГБ (образ корня, 4,7 ГБ, в этом прогоне уже лежал рядом; с нуля к этому времени добавится его
-скачивание — на том же канале около трёх минут); `REST=1 userland/build.sh` — 1 мин 52 с; QEMU до
+по этим самым командам из чистого клона: `sysimage/unpack.sh` — 1 мин 45 с
+(тогда скрипт ещё сам качал образы; теперь IPSW вы кладёте сами, и остаётся только распаковка); `REST=1 userland/build.sh` — 1 мин 52 с; QEMU до
экрана блокировки — ещё 2–3 минуты. То есть около восьми минут от пустого каталога до iOS, если канал
отдаёт порядка 25 МБ/с; сборка ядра и загрузчика в это не входит и не нужна — они в `prebuilt/`.
@@ -197,7 +196,7 @@ codec DMA TX_CDC_DMA_TX_3 и звуковой процессор в поток,
CoreGraphics в них не рисовал — плитки оставались нулями. Теперь загрузчик пишет в дерево sRGB, как и есть у
нашего кадрового буфера, и WebKit рисует в BGRA (`xnuloader/loader.c`, `dt_identity()`; проверка —
`RC_JOBS=rc.jobs.safari`). Цветные эмодзи и китайский шрифт PingFang (130 и 74 МБ) целиком в образ системы не
- помещаются, и их знаки выходили квадратиками с вопросом; теперь `sysimage/fetch.sh` кладёт их в образ уменьшенными
+ помещаются, и их знаки выходили квадратиками с вопросом; теперь `sysimage/unpack.sh` кладёт их в образ уменьшенными
(`tools/font_trim.py`): эмодзи — без самых крупных картинок (остаются 40, 64 и 96 точек; 59 МБ), PingFang — с
обычным и полужирным начертаниями упрощённого, традиционного и гонконгского письма (26 МБ). Проверка —
`RC_JOBS=rc.jobs.fonts` (`font_test`: к какому шрифту система идёт за эмодзи, китайским, японским и корейским и
@@ -234,7 +233,7 @@ codec DMA TX_CDC_DMA_TX_3 и звуковой процессор в поток,
## На телефоне
Нужен именно **POCO X5 Pro 5G (redwood)** с разблокированным загрузчиком, `adb` и `fastboot` на компьютере
-(`sudo pacman -S android-tools`), и образы из предыдущего раздела (`sysimage/fetch.sh`, `REST=1 userland/build.sh`).
+(`sudo pacman -S android-tools`), и образы из предыдущего раздела (`sysimage/unpack.sh`, `REST=1 userland/build.sh`).
**Прошивка самого телефона.** В репозитории её нет и быть не может: она вендорская и берётся с того самого
аппарата, на котором всё будет работать, из его Android (подробности — в разделах по каждому устройству):
@@ -1710,7 +1709,7 @@ xnuloader/build.sh # clang + lld-link → xnuloader/loader.efi (EFI-пр
```
`xnuloader/identity.h` генерируется при сборке `xnuloader/mkidentity.sh` (через `mkidentity.py`) из дерева устройств
-iPhone X в вашем IPSW (так система узнаёт, «какой она iPhone»); в репозитории его нет. Вручную: `python3 sysimage/fetch-from-ipsw.py Firmware/all_flash/DeviceTree.d22ap.im4p dt.im4p`,
+iPhone X в вашем IPSW (так система узнаёт, «какой она iPhone»); в репозитории его нет. Вручную: `python3 sysimage/ipsw-member.py ВАШ.ipsw Firmware/all_flash/DeviceTree.d22ap.im4p dt.im4p`,
`ipsw img4 im4p extract dt.im4p -o dt.bin`, `xnuloader/mkidentity.py dt.bin > xnuloader/identity.h`. Кроме свойств
корня и `/product` оттуда же берётся узел `/product/haptics` (`closed-loop`, `supports-3rd-party-haptics`): по нему
MobileGestalt отвечает `DeviceSupportsClosedLoopHaptics`, и iOS играет вибрацию и отклики волнами на устройстве
@@ -3235,7 +3234,7 @@ iOS, которого она ждёт, — это уже другое: созд
## Про файлы Apple и лицензии
-- Файлов iOS в репозитории нет: `sysimage/fetch.sh` скачивает их из IPSW с CDN Apple на вашу машину.
+- Файлов iOS в репозитории нет: IPSW вы берёте сами, `sysimage/unpack.sh` распаковывает его на вашей машине; ничего не скачивается.
- `xnu/` — производная работа от XNU, распространяется на условиях APSL 2.0, как и сам XNU; `prebuilt/mach.*` —
ядро, собранное из этих исходников.
- `xnu/files/bsd/miscfs/apfs/libzbitmap/` — декодер LZBITMAP libzbitmap (Corellium LLC, GPL-2.0+ OR MIT; здесь —
diff --git a/docs/windows.md b/docs/windows.md
index 31234b7..0c9783c 100644
--- a/docs/windows.md
+++ b/docs/windows.md
@@ -7,7 +7,7 @@ Everything here was done on a real machine and phone; `tools/doctor.sh` checks t
| Job | Where | Why |
|---|---|---|
-| Build the images (`sysimage/fetch.sh`, `userland/build.sh`) | **WSL 2, in its own filesystem** (`~/hakios`) | `fetch.sh` makes files named `NAME:com.apple.decmpfs`; NTFS forbids `:`. The Windows drive is also slow for tens of thousands of small files. |
+| Build the images (`sysimage/unpack.sh`, `userland/build.sh`) | **WSL 2, in its own filesystem** (`~/hakios`) | `unpack.sh` makes files named `NAME:com.apple.decmpfs`; NTFS forbids `:`. The Windows drive is also slow for tens of thousands of small files. |
| Drive the phone (`./xl`, `tools/modem-cycle-win.sh`) | **Git Bash**, in your Windows checkout | `fastboot` and `adb` are Windows programs that see the phone's USB; WSL 2 does not, without usbipd. |
So: build in WSL, copy `userland/root.apfs` and `userland/system.apfs` to the Windows checkout, run `xl`
@@ -41,7 +41,7 @@ from Git Bash.
```sh
# WSL, in ~/hakios (a clone in WSL's own filesystem, with fw-*/ copied in):
-sysimage/fetch.sh # iOS 16.5 from Apple's servers, ~9 min
+sysimage/unpack.sh ~/iPhone10,3,iPhone10,6_16.5_20F66_Restore.ipsw # your own IPSW, ~5 min
PREBUILT=1 REST=1 MODEM=1 SYSTEM=1 userland/build.sh # root.apfs + system.apfs, ~4 min, no theos needed
cp userland/root.apfs userland/system.apfs /mnt/d/.../hakios/userland/
diff --git a/sysimage/fetch-from-ipsw.py b/sysimage/fetch-from-ipsw.py
deleted file mode 100755
index b5648f3..0000000
--- a/sysimage/fetch-from-ipsw.py
+++ /dev/null
@@ -1,109 +0,0 @@
-#!/usr/bin/env python3
-"""Fetch one member of the iOS 16.5 iPhone X firmware archive without downloading the rest.
-
-usage: fetch-from-ipsw.py MEMBER OUTPUT
-
-The IPSW is a ZIP64 archive on Apple's CDN; its directory is read with HTTP range requests and
-the member is streamed (and inflated, if it is compressed) straight to OUTPUT. A download that
-breaks off is taken up again: a member stored without compression (the big disk images are)
-continues where it stopped, also when OUTPUT is the partial file of an earlier run; a compressed
-one starts over.
-"""
-import http.client
-import os
-import struct
-import sys
-import time
-import urllib.request
-import zlib
-
-ATTEMPTS = 20
-
-URL = ('https://updates.cdn-apple.com/2023SpringFCS/fullrestores/032-85130/6AB0D10A-8AF9-4F6C-8CF5-A1B315CABE88/'
- 'iPhone10,3,iPhone10,6_16.5_20F66_Restore.ipsw')
-SIZE = 6165698198
-
-
-def fetch(start, end):
- request = urllib.request.Request(URL, headers={'Range': f'bytes={start}-{end}'})
- return urllib.request.urlopen(request, timeout=120)
-
-
-def directory():
- tail = fetch(SIZE - 65536, SIZE - 1).read()
- at = tail.rfind(b'PK\x06\x06')
- cd_size, cd_offset = struct.unpack('<QQ', tail[at + 40:at + 56])
- cd = fetch(cd_offset, cd_offset + cd_size - 1).read()
- p = 0
- while cd[p:p + 4] == b'PK\x01\x02':
- method, = struct.unpack('<H', cd[p + 10:p + 12])
- csize, usize, nl, el, cl = struct.unpack('<IIHHH', cd[p + 20:p + 34])
- offset, = struct.unpack('<I', cd[p + 42:p + 46])
- name = cd[p + 46:p + 46 + nl].decode()
- extra = cd[p + 46 + nl:p + 46 + nl + el]
- q = 0
- while q + 4 <= len(extra):
- tag, length = struct.unpack('<HH', extra[q:q + 4])
- if tag == 1:
- values = list(struct.unpack('<' + 'Q' * (length // 8), extra[q + 4:q + 4 + length // 8 * 8]))
- if usize == 0xFFFFFFFF:
- usize = values.pop(0)
- if csize == 0xFFFFFFFF:
- csize = values.pop(0)
- if offset == 0xFFFFFFFF:
- offset = values.pop(0)
- q += 4 + length
- yield name, method, csize, usize, offset
- p += 46 + nl + el + cl
-
-
-def download(data, method, csize, output):
- """One attempt at the member whose bytes start at DATA in the archive; returns the size of OUTPUT."""
- if method == 0:
- have = os.path.getsize(output) if os.path.exists(output) else 0
- if have > csize:
- have = 0
- if have < csize:
- with fetch(data + have, data + csize - 1) as stream, open(output, 'ab' if have else 'wb') as out:
- while chunk := stream.read(1 << 20):
- out.write(chunk)
- return os.path.getsize(output)
- inflater = zlib.decompressobj(-15)
- written = 0
- with fetch(data, data + csize - 1) as stream, open(output, 'wb') as out:
- while chunk := stream.read(1 << 20):
- chunk = inflater.decompress(chunk)
- out.write(chunk)
- written += len(chunk)
- tail = inflater.flush()
- out.write(tail)
- return written + len(tail)
-
-
-def main():
- member, output = sys.argv[1], sys.argv[2]
- for name, method, csize, usize, offset in directory():
- if name != member:
- continue
- if method not in (0, 8):
- sys.exit(f'{member}: compression method {method} is not supported')
- header = fetch(offset, offset + 29).read()
- nl, el = struct.unpack('<HH', header[26:30])
- data = offset + 30 + nl + el
- written = -1
- for attempt in range(1, ATTEMPTS + 1):
- try:
- written = download(data, method, csize, output)
- except (OSError, http.client.HTTPException, zlib.error) as error:
- print(f'{member}: {error}', file=sys.stderr)
- if written == usize:
- break
- print(f'{member}: the download broke off, trying again ({attempt} of {ATTEMPTS})', file=sys.stderr)
- time.sleep(5)
- print(f'{member}: {written} bytes (expected {usize})')
- sys.exit(0 if written == usize else 1)
- sys.exit(f'{member}: not in the archive')
-
-
-if __name__ == '__main__':
- main()
diff --git a/sysimage/ipsw-member.py b/sysimage/ipsw-member.py
new file mode 100755
index 0000000..069bbc7
--- /dev/null
+++ b/sysimage/ipsw-member.py
@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+"""Copy one member out of an IPSW the user has on their own disk.
+
+usage: ipsw-member.py IPSW MEMBER OUTPUT
+
+An IPSW is a ZIP64 archive; Python's zipfile reads it as it is. The member is streamed to OUTPUT
+(inflated if it is compressed), so the 4.7 GB root filesystem image never has to fit in memory.
+Nothing is downloaded: the project ships no file of Apple's and fetches none. The IPSW is the
+user's, got by the user from wherever they get their firmware.
+"""
+import shutil
+import sys
+import zipfile
+
+
+def main():
+ if len(sys.argv) != 4:
+ sys.exit(__doc__.split('\n\n')[1])
+ ipsw, member, output = sys.argv[1:]
+ try:
+ archive = zipfile.ZipFile(ipsw)
+ except (OSError, zipfile.BadZipFile) as error:
+ sys.exit(f'ipsw-member.py: {ipsw}: {error}')
+ with archive:
+ try:
+ info = archive.getinfo(member)
+ except KeyError:
+ sys.exit(f'ipsw-member.py: {member} is not in {ipsw} (is it the iOS 16.5 20F66 IPSW for iPhone10,3/iPhone10,6?)')
+ with archive.open(info) as stream, open(output, 'wb') as out:
+ shutil.copyfileobj(stream, out, 1 << 20)
+ print(f'{member}: {info.file_size} bytes')
+
+
+if __name__ == '__main__':
+ main()
diff --git a/sysimage/select-system.py b/sysimage/select-system.py
index 99a2229..e5cda12 100755
--- a/sysimage/select-system.py
+++ b/sysimage/select-system.py
@@ -25,7 +25,7 @@ The whole root is 7.9 GB and the kernel on the phone has about 4 GB of RAM for e
are left out, and so are a few very big daemons that have nothing to do with the display;
- /Library (the wallpapers, regional tables) without its sounds;
- of pictures, what the user interface itself is drawn with: small images, asset catalogs of
- moderate size; the fonts, the two biggest (colour emoji and Chinese) made smaller by fetch.sh;
+ moderate size; the fonts, the two biggest (colour emoji and Chinese) made smaller by unpack.sh;
- the empty LANGUAGE.lproj directories of the bundles whose strings are in one .loctable file
for all languages: they are how CoreFoundation knows which languages a bundle has, and
without them every string comes out as its key.
@@ -64,7 +64,7 @@ SKIP_APP_EXT = ('.mov', '.mp4', '.m4a', '.caf', '.wav', '.heic', '.aiff', '.meta
# (the one with all the SF Symbols alone is 170 MB and is left out)
MAX_ASSET_CATALOG = 4 << 20
# fonts: all but the very big ones; those two -- the colour emoji, 130 MB, and PingFang, the Chinese faces, 74 MB --
-# are taken and made smaller after extraction (fetch.sh, ../tools/font_trim.py): the emoji without their biggest
+# are taken and made smaller after extraction (unpack.sh, ../tools/font_trim.py): the emoji without their biggest
# bitmaps (59 MB), PingFang with the regular and semibold faces for simplified, traditional and Hong Kong Chinese,
# both those programs name and the hidden ones (".PingFangSC-Regular") the system font falls back to
# (26 MB). Without them Safari and the rest drew emoji and Chinese characters as boxes with a question mark.
diff --git a/sysimage/fetch.sh b/sysimage/unpack.sh
similarity index 55%
rename from sysimage/fetch.sh
rename to sysimage/unpack.sh
index ee15b25..f19de4c 100755
--- a/sysimage/fetch.sh
+++ b/sysimage/unpack.sh
@@ -1,25 +1,44 @@
#!/bin/bash
-# Get the files of iOS 16.5 (20F66, iPhone X) that the system image is made of, straight from
-# Apple's servers, and lay them out the way ../userland/build.sh expects them:
+# Unpack the files of iOS 16.5 (20F66, iPhone X) that the system image is made of, out of an IPSW
+# you have on your own disk, and lay them out the way ../userland/build.sh expects them:
#
# rootfs/ the chosen part of the root filesystem (see select-system.py)
# system.files/.manifest what was chosen, with owners, modes and symbolic links
# var.skel the skeleton of /private/var (see mkvarskel.py)
# cryptex/ the OS cryptex without the dyld shared cache
-# cryptex-app.dmg the App cryptex as Apple ships it (10 MB, an APFS container: Safari's
+# cryptex-app.dmg the App cryptex as it is in the IPSW (10 MB, an APFS container: Safari's
# daemons, its settings and the bundles around it, which the root volume's
# launchd jobs and links reach through /System/Cryptexes/App)
# cryptex-app/ ... and its files
# 20F66__iPhone10,3_6/ the dyld shared cache (arm64)
+# DeviceTree.d22ap.im4p the device tree, for ../xnuloader/mkidentity.sh
#
-# Nothing of this is in the repository: the two disk images are fetched out of the IPSW on
-# Apple's CDN (fetch-from-ipsw.py reads single members of the archive, 4.7 GB + 2.7 GB instead of
-# all 6.2 GB) and unpacked with 7z. Needs: python3, 7z (7-Zip 23 or newer: it reads APFS), cc.
-# About 8 GB of disk while it runs, 5.3 GB afterwards.
+# The project ships no file of Apple's and downloads none. The IPSW is yours: get
+# iPhone10,3,iPhone10,6_16.5_20F66_Restore.ipsw yourself (6 165 698 198 bytes) and give its path:
#
-# KEEP=1 keep the downloaded disk images (rootfs.dmg is needed again only after changing
-# select-system.py; with it in place a second run downloads nothing)
+# sysimage/unpack.sh ~/Downloads/iPhone10,3,iPhone10,6_16.5_20F66_Restore.ipsw
+# IPSW=... sysimage/unpack.sh the same, from the environment (make unpack IPSW=...)
+#
+# The disk images are copied out of the IPSW (ipsw-member.py) and unpacked with 7z. Needs: python3,
+# 7z (7-Zip 23 or newer: it reads APFS), cc. About 8 GB of disk while it runs, 5.3 GB afterwards.
+#
+# KEEP=1 keep the copied disk images (rootfs.dmg is needed again only after changing
+# select-system.py; with it in place a second run does not read the IPSW for it)
set -e
+IPSW=${1:-${IPSW:-}}
+if [ -z "$IPSW" ]; then
+ sed -n '16,20p' "$0" | sed 's/^# \{0,1\}//' >&2
+ echo "unpack.sh: give the path of your iOS 16.5 IPSW" >&2
+ exit 1
+fi
+[ -f "$IPSW" ] || { echo "unpack.sh: $IPSW: no such file" >&2; exit 1; }
+IPSW=$(cd "$(dirname "$IPSW")" && pwd)/$(basename "$IPSW")
+size=$(wc -c < "$IPSW" | tr -d ' ')
+# select-system.py and the checks below are made for this one build; another IPSW would unpack, then fail somewhere later
+if [ "$size" != 6165698198 ] && [ -z "${ANY_IPSW:-}" ]; then
+ echo "unpack.sh: $IPSW is $size bytes, not the 6165698198 of iOS 16.5 20F66 for iPhone10,3/10,6 (ANY_IPSW=1 tries it anyway)" >&2
+ exit 1
+fi
cd "$(dirname "$0")"
ROOTFS_DMG=096-04088-077.dmg # the root filesystem, APFS
CRYPTEX_DMG=096-04666-078.dmg # the OS cryptex: dyld shared cache and a few libraries
@@ -28,7 +47,7 @@ CACHE=20F66__iPhone10,3_6
UNZBM=../xnu-deps/host/bin/unzbm
for tool in python3 7z cc; do
- command -v $tool >/dev/null || { echo "fetch.sh: $tool is not installed"; exit 1; }
+ command -v $tool >/dev/null || { echo "unpack.sh: $tool is not installed"; exit 1; }
done
# 7z leaves files compressed with LZBITMAP as an empty file and two attribute files; unzbm finishes them
if [ ! -x $UNZBM ]; then
@@ -39,21 +58,21 @@ unzbm_all() { # unzbm_all DIR: expand every LZBITMAP file below DIR
find "$1" -type f -name '*:com.apple.decmpfs' | sed 's/:com\.apple\.decmpfs$//' | tr '\n' '\0' | xargs -0 -r -n 200 $UNZBM
local left
left=$(find "$1" -name '*:com.apple.*' | wc -l)
- [ "$left" -eq 0 ] || { echo "fetch.sh: $left files in $1 are still compressed"; exit 1; }
+ [ "$left" -eq 0 ] || { echo "unpack.sh: $left files in $1 are still compressed"; exit 1; }
}
# --- the root filesystem --------------------------------------------------------------------------
if [ ! -f .done-rootfs ] || [ select-system.py -nt .done-rootfs ]; then
if [ ! -f rootfs.dmg ]; then
- echo "fetch.sh: downloading the root filesystem image (4.7 GB)"
- python3 fetch-from-ipsw.py $ROOTFS_DMG rootfs.dmg.part
+ echo "unpack.sh: copying the root filesystem image out of the IPSW (4.7 GB)"
+ python3 ipsw-member.py "$IPSW" $ROOTFS_DMG rootfs.dmg.part
mv rootfs.dmg.part rootfs.dmg
fi
- echo "fetch.sh: listing the image"
+ echo "unpack.sh: listing the image"
7z l -slt rootfs.dmg > rootfs.slt
python3 select-system.py rootfs.slt system.files system.manifest
python3 mkvarskel.py rootfs.slt > var.skel
- echo "fetch.sh: extracting $(wc -l < system.files) files"
+ echo "unpack.sh: extracting $(wc -l < system.files) files"
rm -rf rootfs
7z x -y -orootfs rootfs.dmg @system.files > extract-system.log
# the two fonts too big for the image as they are, made smaller in place (see select-system.py)
@@ -68,11 +87,11 @@ fi
# --- the OS cryptex -------------------------------------------------------------------------------
if [ ! -f .done-cryptex ]; then
if [ ! -f cryptex-os.dmg ]; then
- echo "fetch.sh: downloading the OS cryptex image (2.7 GB)"
- python3 fetch-from-ipsw.py $CRYPTEX_DMG cryptex-os.dmg.part
+ echo "unpack.sh: copying the OS cryptex image out of the IPSW (2.7 GB)"
+ python3 ipsw-member.py "$IPSW" $CRYPTEX_DMG cryptex-os.dmg.part
mv cryptex-os.dmg.part cryptex-os.dmg
fi
- echo "fetch.sh: extracting the cryptex"
+ echo "unpack.sh: extracting the cryptex"
rm -rf cryptex cryptex.cache $CACHE
# 7z reports every LZBITMAP file as an error ("Unsupported Method"): expected, unzbm_all checks the result
7z x -y -ocryptex cryptex-os.dmg '-xr!com.apple.dyld' '-x!System/DriverKit' > extract-cryptex.log 2>&1 || true
@@ -82,7 +101,7 @@ if [ ! -f .done-cryptex ]; then
unzbm_all cryptex.cache
mv cryptex.cache/System/Library/Caches/com.apple.dyld $CACHE
rm -rf cryptex.cache
- [ "$(ls $CACHE | wc -l)" -eq 43 ] || { echo "fetch.sh: the dyld shared cache is incomplete"; exit 1; }
+ [ "$(ls $CACHE | wc -l)" -eq 43 ] || { echo "unpack.sh: the dyld shared cache is incomplete"; exit 1; }
touch .done-cryptex
[ -n "$KEEP" ] || rm -f cryptex-os.dmg
fi
@@ -92,15 +111,17 @@ fi
# and unpacked (the system image has its files at /System/Cryptexes/App)
if [ ! -f .done-cryptex-app ]; then
if [ ! -f cryptex-app.dmg ]; then
- echo "fetch.sh: downloading the App cryptex image (10 MB)"
- python3 fetch-from-ipsw.py $APP_CRYPTEX_DMG cryptex-app.dmg.part
+ echo "unpack.sh: copying the App cryptex image out of the IPSW (10 MB)"
+ python3 ipsw-member.py "$IPSW" $APP_CRYPTEX_DMG cryptex-app.dmg.part
mv cryptex-app.dmg.part cryptex-app.dmg
fi
- echo "fetch.sh: extracting the App cryptex"
+ echo "unpack.sh: extracting the App cryptex"
rm -rf cryptex-app
7z x -y -ocryptex-app cryptex-app.dmg > extract-cryptex-app.log 2>&1 || true
unzbm_all cryptex-app
- [ -f cryptex-app/usr/libexec/webbookmarksd ] || { echo "fetch.sh: the App cryptex is incomplete"; exit 1; }
+ [ -f cryptex-app/usr/libexec/webbookmarksd ] || { echo "unpack.sh: the App cryptex is incomplete"; exit 1; }
touch .done-cryptex-app
fi
-echo "fetch.sh: done: $(du -sh rootfs | cut -f1) rootfs, $(du -sh cryptex | cut -f1) cryptex, $(du -sh cryptex-app | cut -f1) App cryptex, $(du -sh $CACHE | cut -f1) dyld shared cache"
+# --- the device tree, for xnuloader/mkidentity.sh -----------------------------------------------
+[ -f DeviceTree.d22ap.im4p ] || python3 ipsw-member.py "$IPSW" Firmware/all_flash/DeviceTree.d22ap.im4p DeviceTree.d22ap.im4p
+echo "unpack.sh: done: $(du -sh rootfs | cut -f1) rootfs, $(du -sh cryptex | cut -f1) cryptex, $(du -sh cryptex-app | cut -f1) App cryptex, $(du -sh $CACHE | cut -f1) dyld shared cache"
diff --git a/tools/apfs/run-tests.sh b/tools/apfs/run-tests.sh
index 7605c0c..fa9ac46 100755
--- a/tools/apfs/run-tests.sh
+++ b/tools/apfs/run-tests.sh
@@ -61,7 +61,7 @@ echo "== 2. the UDIF reader on sysimage/rootfs.dmg"
if [ -f "$ROOT/sysimage/rootfs.dmg" ]; then
python3 "$HERE/dmg.py" info "$ROOT/sysimage/rootfs.dmg" | tail -n 14 || fail=1
else
- echo " skipped: sysimage/rootfs.dmg is not here (run sysimage/fetch.sh with KEEP=1)"
+ echo " skipped: sysimage/rootfs.dmg is not here (run sysimage/unpack.sh with KEEP=1)"
fi
echo
diff --git a/tools/apfs/verify.py b/tools/apfs/verify.py
index 175c87d..1651385 100755
--- a/tools/apfs/verify.py
+++ b/tools/apfs/verify.py
@@ -4,7 +4,7 @@
# Two independent references, both already in sysimage/:
#
# listing sysimage/rootfs.slt -- what 7-Zip's own APFS reader saw in the same image
-# (`7z l -slt rootfs.dmg`, made by sysimage/fetch.sh). Covers every entry in
+# (`7z l -slt rootfs.dmg`, made by sysimage/unpack.sh). Covers every entry in
# the volume: path, mode, owner, size, symbolic-link target, inode number.
#
# content sysimage/rootfs/ -- the files 7-Zip actually extracted, compared byte for
@@ -36,7 +36,7 @@ import apfs
SYSTEM_OBJ_ID_MARK = 0x0FFFFFFF00000000
-# sysimage/fetch.sh cuts these two fonts down after extracting them, so what is on disk in
+# sysimage/unpack.sh cuts these two fonts down after extracting them, so what is on disk in
# the reference tree is deliberately not what is in the image
TRIMMED = {
"System/Library/Fonts/CoreAddition/AppleColorEmoji-160px.ttc",
@@ -241,7 +241,7 @@ def cmd_content(image, refdir, limit, do_all, stratified):
print("volume %r, reference tree %s" % (v.volname, refdir))
mine = build_tree(v, note=True)
- # every regular file that exists in both, minus the two fonts fetch.sh rewrites
+ # every regular file that exists in both, minus the two fonts unpack.sh rewrites
cands = []
for root, _dirs, files in os.walk(refdir):
for fn in files:
diff --git a/tools/apfs/write-tests.sh b/tools/apfs/write-tests.sh
index 54bec97..637b737 100755
--- a/tools/apfs/write-tests.sh
+++ b/tools/apfs/write-tests.sh
@@ -8,7 +8,7 @@
# check-image.sh pins, cloned into xnu-deps/build/apfsprogs and built there: the
# reader that has nothing to do with us and checks the space manager's bitmaps
# against every block anything refers to
-# 7-Zip a third APFS reader, already needed by sysimage/fetch.sh
+# 7-Zip a third APFS reader, already needed by sysimage/unpack.sh
#
# usage: tools/apfs/write-tests.sh [-k] [TEST ...] (-k: keep the images in apfs-work)
# The tests: baseline writes refusals volume-gates extent-gates power-failure real-images
diff --git a/tools/doctor.sh b/tools/doctor.sh
index 5f1bacd..2f470f9 100755
--- a/tools/doctor.sh
+++ b/tools/doctor.sh
@@ -37,7 +37,7 @@ else
bad "python3 does not run" "install Python 3 (on Windows: python.org, not the Microsoft Store stub)"
fi
-# 7-Zip 23 or newer: sysimage/fetch.sh unpacks APFS with it
+# 7-Zip 23 or newer: sysimage/unpack.sh unpacks APFS with it
z=""; for t in 7zz 7z 7za; do have $t && { z=$t; break; }; done
if [ -n "$z" ]; then
zv=$($z 2>&1 | head -3 | grep -oE '[0-9]+\.[0-9]+' | head -1)
@@ -46,7 +46,7 @@ else
need "7-Zip not found" "install 7-Zip 23 or newer (apt: 7zip, brew: sevenzip)"
fi
-if have cc || have clang; then ok "C compiler ($(have cc && echo cc || echo clang)): builds unzbm for sysimage/fetch.sh"; else need "no C compiler" "install clang or gcc"; fi
+if have cc || have clang; then ok "C compiler ($(have cc && echo cc || echo clang)): builds unzbm for sysimage/unpack.sh"; else need "no C compiler" "install clang or gcc"; fi
have git && ok "git" || bad "git not found" "install git"
if have lld-link; then ok "lld-link: the boot loader can be built (xnuloader/build.sh)"; else warn "lld-link not found" "needed to build the boot loader with the iPhone identity from your own IPSW (make loader), which the iOS run needs (apt: lld; brew: lld); the open kernel (make qemu-open) runs with prebuilt/loader.efi, which has no Apple data"; fi
have lz4 && ok "lz4" || warn "lz4 not found" "needed by some image steps (apt: lz4)"
@@ -63,10 +63,10 @@ fi
kb=$(df -Pk . 2>/dev/null | awk 'NR == 2 { print $4 }')
if [ -n "$kb" ] && [ "$kb" -ge 12582912 ]; then ok "disk: $((kb / 1048576)) GB free here"; else warn "disk: ${kb:+$((kb / 1048576)) GB }free here, 12 GB wanted" "free some space, or work in another directory"; fi
-# the trap: fetch.sh makes files named NAME:com.apple.decmpfs, and ':' is not a legal NTFS name character
+# the trap: unpack.sh makes files named NAME:com.apple.decmpfs, and ':' is not a legal NTFS name character
case "$host:$PWD" in
wsl:/mnt/*) warn "this checkout is on a Windows drive (/mnt/...)" "build in WSL's own filesystem: git clone it to ~/hakios and copy fw-* there (docs/windows.md)" ;;
-gitbash:*) warn "Git Bash cannot run sysimage/fetch.sh (NTFS forbids ':' in file names)" "build the images in WSL, copy them back; run xl from here (docs/windows.md)" ;;
+gitbash:*) warn "Git Bash cannot run sysimage/unpack.sh (NTFS forbids ':' in file names)" "build the images in WSL, copy them back; run xl from here (docs/windows.md)" ;;
esac
# rebuilding the kernel or userland needs theos's SDK and its Linux toolchain's cctools linker (xnu-deps/setup.sh)
@@ -101,7 +101,7 @@ fi
head1 "What is already built here"
if [ -f fw-modem/modem.mdt ]; then ok "phone firmware pulled (fw-*/)"; else warn "no fw-modem/ yet" "with the phone on adb: devices/redwood/pull-firmware.sh, or pull-firmware-root.sh if that gets nothing"; fi
-if [ -f userland/system.apfs ] && [ -f userland/root.apfs ]; then ok "userland/system.apfs and root.apfs"; else warn "no iOS images yet" "make fetch, then PREBUILT=1 REST=1 MODEM=1 SYSTEM=1 userland/build.sh (needs iOS 16.5 from Apple's servers, ~8 GB)"; fi
+if [ -f userland/system.apfs ] && [ -f userland/root.apfs ]; then ok "userland/system.apfs and root.apfs"; else warn "no iOS images yet" "make unpack IPSW=your.ipsw (iOS 16.5 20F66 for iPhone X, which you get yourself), then PREBUILT=1 REST=1 MODEM=1 SYSTEM=1 userland/build.sh (~8 GB)"; fi
head1 "Phone"
adb=""
diff --git a/tools/font_trim.py b/tools/font_trim.py
index 1a57846..5272e44 100755
--- a/tools/font_trim.py
+++ b/tools/font_trim.py
@@ -14,7 +14,7 @@ the phone's memory, made from the files themselves (nothing of them is in the re
matches REGEX, and only the tables they use.
Both write a TrueType collection that CoreText reads as it reads the original; DST may be SRC.
-sysimage/fetch.sh makes both for the system image this way.
+sysimage/unpack.sh makes both for the system image this way.
"""
import os
import re
diff --git a/userland/mkjobs.py b/userland/mkjobs.py
index cc41251..7e1f42f 100755
--- a/userland/mkjobs.py
+++ b/userland/mkjobs.py
@@ -239,7 +239,7 @@ src_root, dest, labels = args[0], args[1], args[2:]
oslog = os.environ.get('OSLOG', '').split()
src = f'{src_root}/System/Library/LaunchDaemons'
# A program in a cryptex -- on a device a volume of its own, grafted at /System/Cryptexes/NAME (eight of
-# Safari's daemons are in the App cryptex) -- is looked for where fetch.sh unpacked that cryptex, beside
+# Safari's daemons are in the App cryptex) -- is looked for where unpack.sh unpacked that cryptex, beside
# the root filesystem; the images have it at the same place (build.sh).
CRYPTEXES = {'/System/Cryptexes/OS/': 'cryptex', '/System/Cryptexes/App/': 'cryptex-app'}
diff --git a/xnuloader/mkidentity.sh b/xnuloader/mkidentity.sh
index 1b22100..5f06014 100755
--- a/xnuloader/mkidentity.sh
+++ b/xnuloader/mkidentity.sh
@@ -3,13 +3,15 @@
# from the device tree in the IPSW (Firmware/all_flash/DeviceTree.d22ap.im4p) at build time, on
# the builder's machine, from their own copy of the firmware. The header is not in the repository.
#
-# sysimage/fetch-from-ipsw.py the one file out of the IPSW on Apple's CDN (34 KB, HTTP ranges)
+# sysimage/ipsw-member.py the one file out of your IPSW (34 KB); sysimage/unpack.sh leaves it
+# in sysimage/ too
# xnuloader/im4p.py its payload: an LZFSE stream
# lzfse(1) / compression_tool(1) / python3 -m lzfse / xnu-deps/host/bin/lzfse (built from the
# BSD-licensed github.com/lzfse/lzfse when nothing else is there)
# xnuloader/mkidentity.py the header from the decoded tree
#
-# usage: xnuloader/mkidentity.sh [DeviceTree.im4p] (an im4p already at hand skips the download)
+# usage: xnuloader/mkidentity.sh [DeviceTree.im4p | your.ipsw]
+# (with no argument: sysimage/DeviceTree.d22ap.im4p, or $IPSW)
# xnuloader/mkidentity.sh --placeholder (no IPSW: an identity.h with no Apple data,
# enough to build and boot the open kernel under
# QEMU with an Apple-free user space, not iOS)
@@ -24,10 +26,18 @@ if [ "$1" = --placeholder ] || [ "${PLACEHOLDER:-}" = 1 ]; then
fi
WORK=$(mktemp -d "${TMPDIR:-/tmp}/mkidentity.XXXXXX")
trap 'rm -rf "$WORK"' EXIT
-IM4P=${1:-$WORK/dt.im4p}
-if [ ! -f "$IM4P" ]; then
- python3 "$TOP/sysimage/fetch-from-ipsw.py" Firmware/all_flash/DeviceTree.d22ap.im4p "$IM4P"
+# nothing is downloaded: the device tree comes from the IPSW you have
+SRC=${1:-}
+[ -n "$SRC" ] || { [ -f "$TOP/sysimage/DeviceTree.d22ap.im4p" ] && SRC=$TOP/sysimage/DeviceTree.d22ap.im4p; } || SRC=${IPSW:-}
+if [ -z "$SRC" ] || [ ! -f "$SRC" ]; then
+ echo "mkidentity.sh: no device tree. Run sysimage/unpack.sh YOUR.ipsw first, or give the IPSW or" >&2
+ echo " DeviceTree.d22ap.im4p as the argument (or --placeholder for the open kernel without iOS)" >&2
+ exit 1
fi
+case $SRC in
+*.ipsw) python3 "$TOP/sysimage/ipsw-member.py" "$SRC" Firmware/all_flash/DeviceTree.d22ap.im4p "$WORK/dt.im4p"; IM4P=$WORK/dt.im4p ;;
+*) IM4P=$SRC ;;
+esac
python3 "$HERE/im4p.py" "$IM4P" "$WORK/dt.lzfse"
decode() {
if command -v lzfse >/dev/null; then lzfse -decode -i "$1" -o "$2"