Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
One repository per device: it carries its SoC's layer (soc/); new copies or makes it, and --repo makes a git repository to push
shimoku74a6e3cfdf6164
diff --git a/Makefile b/Makefileindex af21276..3d05fce 100644--- a/Makefile+++ b/Makefile@@ -19,7 +19,7 @@## Configuration, the way Linux's make does it (configs/, tools/hakios-config.py; docs/configs.md):# make redwood_defconfig .config from configs/defconfig + redwood_defconfig (devices/redwood/configs/), which is-# common_defconfig + sc7280.config (socs/sc7280/) + redwood.config+# common_defconfig + sc7280.config (devices/redwood/soc/) + redwood.config# (any NAME_defconfig; make common_defconfig is the common kernel alone; docs/configs.md)# make devices the device and SoC repositories (tools/hakios-device.py list; docs/porting/repositories.md)# make virt.config lay a fragment (configs/virt.config) over the .config there isdiff --git a/README.md b/README.mdindex 77c8a1c..4c794d0 100644--- a/README.md+++ b/README.md@@ -62,13 +62,12 @@ make phone SECS=600 # a 10-minute session on the phone, in RAM| Repository | What ||---|---|| [hakios](https://github.com/oosexgang/hakios) | **common**: the kernel port, every common driver, the loader, user space, the tools, this documentation, and `devices/qemu` |-| [hakios-sc7280-common](https://github.com/oosexgang/hakios-sc7280-common) | the Qualcomm SC7280 family (Snapdragon 778G): its configuration fragment and the drivers it builds |-| [hakios-redwood](https://github.com/oosexgang/hakios-redwood) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script |+| [hakios-redwood](https://github.com/oosexgang/hakios-redwood) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script, and its SoC's layer (`soc/`: the Qualcomm SC7280 family) || [hakios-uefi](https://github.com/oosexgang/hakios-uefi) | HakiOS's UEFI: Mu-Silicium with HakiOS's boot path, banner and name || [hakios-web](https://github.com/oosexgang/hakios-web) | the website, with this documentation and the FAQ |`devices/manifest.conf` in the common repository names each one and the revision it is at; `tools/hakios-device.py sync` clones them-into `socs/`, `devices/` and `uefi/`. [How the repositories fit](docs/porting/repositories.md).+into `devices/` and `uefi/`. One repository per device: it carries its SoC's layer, and `tools/hakios-device.py new` makes one. [How the repositories fit](docs/porting/repositories.md).```shgit clone https://github.com/oosexgang/hakios.git && cd hakiosdiff --git a/devices/manifest.conf b/devices/manifest.confindex ae575bd..ad3c578 100644--- a/devices/manifest.conf+++ b/devices/manifest.conf@@ -1,6 +1,7 @@# devices/manifest.conf: every SoC and device repository HakiOS knows of (Android's repo manifest, UniXNU's devices.toml).#-# soc|device|uefi <name> (the UEFI is one for every device: it goes to uefi/)+# device|uefi <name> (a device carries its SoC's layer in soc/; the UEFI is one, in uefi/)+# soc <name> (a SoC layer of its own, in socs/<name>: for one many devices share)# path <directory in this tree> kept here, for now: tools/hakios-device.py split makes it a repository# repository <git URL> a repository of its own, cloned into socs/<name> or devices/<name># revision <commit> ... at this commit (tools/hakios-device.py lock writes it)@@ -8,13 +9,9 @@# `tools/hakios-device.py sync` clones and updates the ones with a repository; `list` shows them all; docs/porting/repositories.md# is the guide. A device of yours: open a pull request that adds its block here, with its repository and revision.-soc sc7280- repository https://github.com/oosexgang/hakios-sc7280-common.git- revision 1ef455525b734ba2066ff294aa11f7e5be3c20fc-device redwoodrepository https://github.com/oosexgang/hakios-redwood.git- revision da8a0b307f17f43a79929c1b2d49d1fed4ec1a75+ revision 84c2cb861bccfb4d76fba9976b2d7810a95ed662device qemupath devices/qemudiff --git a/docs/concepts/architecture.md b/docs/concepts/architecture.mdindex 21a111e..ccbf840 100644--- a/docs/concepts/architecture.md+++ b/docs/concepts/architecture.md@@ -26,8 +26,8 @@ yours (the user space you lay out).```configs/common_defconfig (common)- socs/sc7280/configs/sc7280.config (SoC) make redwood_defconfig- devices/redwood/configs/redwood.config (device) -------------------------> .config+ devices/redwood/soc/configs/sc7280.config (SoC) make redwood_defconfig+ devices/redwood/configs/redwood.config (device) -------------------------> .config|CONFIG_HAKIOS_* -> tools/hakios-config.py kernel -> XNU's options (config/MASTER.hakios, conf/files) -> make kernelevery other CONFIG_* -> tools/hakios-config.py args -> the kernel's boot-arguments -> make phone, ./xl run@@ -46,7 +46,7 @@ device's. A device's defconfig is three lines: the common kernel, its SoC's frag| Layer | Has | Example ||---|---|---|| common | the kernel port, every driver, the loader, user space, the tools, the documentation | this repository |-| SoC | the SoC's facts (cores, addresses that are the family's), its drivers | `socs/sc7280` |+| SoC | the SoC's facts (cores, addresses that are the family's), its drivers; carried by each device that has the SoC | `devices/redwood/soc` || device | its defconfig, its status, its UEFI platform, its firmware script, drivers of its own | `devices/redwood` || UEFI | Mu-Silicium with HakiOS's boot path, banner and name | `uefi` ([its README](https://github.com/oosexgang/hakios/blob/main/uefi/README.md)) |@@ -69,7 +69,7 @@ device's. A device's defconfig is three lines: the common kernel, its SoC's frag| `xnuloader/` | the boot loader (a UEFI application, no EDK2) || `userland/` | programs and libraries of our own (`src/`), the Metal plug-in, the image build, launchd jobs || `configs/` | the common kernel's configuration and the generic fragments |-| `socs/`, `devices/` | the SoC and device repositories, and `devices/manifest.conf` |+| `devices/` | the device repositories (each with its SoC's layer in `soc/`), and `devices/manifest.conf` || `sysimage/` | scripts that fetch and unpack the OS files from the IPSW (none of the files) || `tools/` | configuration, devices, checks, APFS, firmware, QMI, the run scripts || `qemu/` | running in QEMU |diff --git a/docs/getting-started/phone.md b/docs/getting-started/phone.mdindex fd1291f..702aa9a 100644--- a/docs/getting-started/phone.md+++ b/docs/getting-started/phone.md@@ -20,7 +20,7 @@ that you create yourself, and nowhere else. [What is written, exactly](../concep## Step by step```sh-tools/hakios-device.py sync # hakios-redwood, hakios-sc7280-common, hakios-uefi into devices/, socs/, uefi/+tools/hakios-device.py sync # hakios-redwood (with its SoC layer) and hakios-uefi into devices/ and uefi/devices/redwood/pull-firmware.sh # Android running, on adb: GPU, Wi-Fi, audio, haptics, modem, cameras -> fw-*/make redwood_defconfig # the phone's configuration: the common kernel, the SC7280's, the phone's ownmake config # what it is: the drivers built in, the kernel's arguments, the firmware it wantsdiff --git a/docs/index.md b/docs/index.mdindex bb47780..dec3ee0 100644--- a/docs/index.md+++ b/docs/index.md@@ -45,10 +45,9 @@ Android. What is and is not written to a phone: [Architecture](concepts/architec| Repository | What ||---|---|| [hakios](https://github.com/oosexgang/hakios) | **common**: the kernel port, every common driver, the loader, user space, the tools, this documentation, and `devices/qemu` |-| [hakios-sc7280-common](https://github.com/oosexgang/hakios-sc7280-common) | the Qualcomm SC7280 family (Snapdragon 778G): its configuration fragment and the drivers it builds |-| [hakios-redwood](https://github.com/oosexgang/hakios-redwood) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script |+| [hakios-redwood](https://github.com/oosexgang/hakios-redwood) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script, and its SoC's layer (`soc/`: the Qualcomm SC7280 family) || [hakios-uefi](https://github.com/oosexgang/hakios-uefi) | HakiOS's UEFI: Mu-Silicium with HakiOS's boot path, banner and name || [hakios-web](https://github.com/oosexgang/hakios-web) | the website, with this documentation and the FAQ |`devices/manifest.conf` in the common repository names each one and the revision it is at; `tools/hakios-device.py sync` clones them-into `socs/`, `devices/` and `uefi/`. [How the repositories fit](porting/repositories.md).+into `devices/` and `uefi/`. One repository per device: it carries its SoC's layer, and `tools/hakios-device.py new` makes one. [How the repositories fit](porting/repositories.md).diff --git a/docs/porting/drivers.md b/docs/porting/drivers.mdindex 6265e4d..43b2703 100644--- a/docs/porting/drivers.md+++ b/docs/porting/drivers.md@@ -10,7 +10,7 @@ reference for the mechanism.|---|---|| for hardware many devices share, or a SoC family's | the common repository: `drivers/<subsystem>/` || one device's own (a touch controller only it has) | that device's repository: `devices/<name>/drivers/<subsystem>/` |-| a SoC family's own | `socs/<soc>/drivers/<subsystem>/` |+| a SoC family's own | the SoC's layer: `devices/<name>/soc/drivers/<subsystem>/` |Subsystems: input, storage, gpu, camera, haptic, audio, power, wifi, bluetooth, modem, net, core.diff --git a/docs/porting/index.md b/docs/porting/index.mdindex a403d30..6ebe69b 100644--- a/docs/porting/index.md+++ b/docs/porting/index.md@@ -31,8 +31,9 @@ That makes `devices/mydevice/` with:left out**: their addresses, pins and rails are the phone they were written for, and a driver of another phone touching yoursis the last thing you want. The drivers of your SoC (from its fragment) and QEMU's stay in.-A SoC without a repository yet: make `socs/<soc>/` the same way by hand (`soc.conf`, `configs/<soc>.config`), from-[socs/sc7280](https://github.com/oosexgang/hakios/tree/main/socs/sc7280).+The SoC's layer comes with the device, in `devices/mydevice/soc/`: a copy of the one another device carries when the SoC is one+HakiOS knows (`--soc sc7280` copies redwood's), or a new one to fill in for a SoC nobody has ported yet (`--soc sm8250+--soc-title "Snapdragon 865"`). `--repo` makes the directory a git repository, ready to push as `hakios-mydevice`.**Done when** `tools/hakios-device.py check mydevice` says it holds together, and the README says which SoC values differ and youknow why.diff --git a/docs/porting/repositories.md b/docs/porting/repositories.mdindex 4410f4b..9e8b5f8 100644--- a/docs/porting/repositories.md+++ b/docs/porting/repositories.md@@ -4,10 +4,11 @@ HakiOS is put together from layers, the way an Android device's source is (`devithat names them):```-hakios/ common: the kernel port, every common driver, the loader, user space, the tools, the docs-├── socs/<soc>/ a SoC family: soc.conf, configs/<soc>.config, drivers/ of its own-├── devices/<name>/ a device: device.conf, configs/<name>_defconfig and <name>.config, drivers/, its UEFI, scripts-└── devices/manifest.conf every SoC and device repository: kept in this tree, or a repository of its own at a revision+hakios/ common: the kernel port, every common driver, the loader, user space, the tools, the docs+├── devices/<name>/ a device, one repository: device.conf, configs/, drivers/, its UEFI platform, scripts+│ └── soc/ ... and its SoC family's layer: soc.conf, configs/<soc>.config, the SoC's drivers+├── uefi/ HakiOS's UEFI (Mu-Silicium with HakiOS's changes), one for every device+└── devices/manifest.conf every device and the UEFI: kept in this tree, or a repository of its own at a revision```A SoC or device directory is the same whether it is kept in the common tree (`devices/qemu` is) or is a git repository of its own@@ -22,6 +23,7 @@ that `tools/hakios-device.py sync` cloned into place (the others are): the tools| `configs/<name>_defconfig` | `#include common_defconfig`, `#include <soc>.config`, `#include <name>.config` | yes || `configs/<name>.config` | what is this device's own: drivers left out or in, arguments, `CONFIG_DEVICE="<name>"` | yes || `drivers/<subsystem>/conf` and the files it names | drivers of its own ([Writing a driver](drivers.md)) | no |+| `soc/` | its SoC family's layer: `soc.conf`, `configs/<soc>.config`, the SoC's drivers ([below](#the-socs-layer)) | yes, unless `soc none` || `pull-firmware.sh` | the device's firmware, from your own device, into `fw-*/` | for a phone || `uefi/`, `prebuilt/` | the device's UEFI port and the images made of it | for a phone || `README.md` | anything else a person needs to know | yes |@@ -53,7 +55,9 @@ device redwoodThe device's page in this documentation ([redwood](../devices/redwood.md)) and the table of [all devices](../devices/index.md)are made from it by `tools/docs-gen.py`: the status is written once, next to the code it describes.-## A SoC repository+## The SoC's layer++A device carries its SoC family's layer with it, in `soc/`, so that one repository is all a device needs:```soc sc7280@@ -64,17 +68,21 @@ soc sc7280maintainer the HakiOS developers (README.md)```-and `configs/<soc>.config`, the SoC's facts: `CONFIG_SOC`, its cores, and **the drivers of the SoC family**-(`CONFIG_HAKIOS_CPUFREQ=y`, `CONFIG_HAKIOS_THERMAL=y`, ...), so that every device with the SoC has them, whatever it leaves out of-its own.+and `soc/configs/<soc>.config`, the SoC's facts: `CONFIG_SOC`, its cores, and **the drivers of the SoC family**+(`CONFIG_HAKIOS_CPUFREQ=y`, `CONFIG_HAKIOS_THERMAL=y`, ...), so that the device has them whatever it leaves out of its own.++Two devices with the same SoC each carry a copy. The copies must be the same: the tools take the SoC once, and refuse two copies that+differ, naming both, because a build would otherwise depend on which device was checked out. `tools/hakios-device.py new --soc sc7280`+copies the layer from a device that has it; a SoC nobody has yet gets a new layer to fill in. A SoC many devices share can also be a+repository of its own in `socs/<soc>/` (a `soc` block in the manifest); the tools read both.## The manifest`devices/manifest.conf` names every SoC and device:```-soc sc7280- repository https://github.com/oosexgang/hakios-sc7280-common.git+device redwood+ repository https://github.com/oosexgang/hakios-redwood.gitrevisiondevice qemu@@ -89,7 +97,7 @@ device sheng|---|---|| `tools/hakios-device.py list` (`make devices`) | every device and SoC, where it is, how much of it works || `tools/hakios-device.py check [NAME...]` | a repository holds together: its conf reads, its defconfig includes the common kernel and its SoC's fragment and passes `hakios-config.py check`, its drivers' files are there, no two repositories have a configuration of the same name |-| `tools/hakios-device.py new NAME --soc SOC --title ...` | a new device repository ([Port a device](index.md)) |+| `tools/hakios-device.py new NAME --soc SOC --title ... [--fdt FILE] [--repo]` | a whole new device repository, its SoC layer with it; `--repo` makes it a git repository ready to push ([Port a device](index.md)) || `tools/hakios-device.py sync [NAME...]` | clone or update the repositories that live elsewhere, at their revisions || `tools/hakios-device.py lock NAME` | write the revision checked out into the manifest || `tools/hakios-device.py split NAME DIR` | a device kept in this tree, as a git repository of its own with its history, ready to push |@@ -114,7 +122,7 @@ cannot declare `standard` files: what every kernel has is the common repository'| | Android | UniXNU | HakiOS ||---|---|---|---|| the shared part | AOSP, the GKI kernel | `UniXNU` (tools, components, ARM64 support) | `hakios` (the kernel port, drivers, loader, tools) |-| a SoC | `<soc>-common` | `UniXNU-<soc>-common` (`soc.toml`) | `socs/<soc>` (`soc.conf`) |+| a SoC | `<soc>-common` | `UniXNU-<soc>-common` (`soc.toml`) | `soc/` inside each device repository (`soc.conf`); `socs/<soc>` for a shared one || a device | `device/<vendor>/<name>` | `UniXNU-<name>` (`board.toml`) | `devices/<name>` (`device.conf`) || what names them | `repo` manifest | `devices.toml` | `devices/manifest.conf` || a driver | kernel module, `Kconfig` | a component repository | `drivers/<sub>/conf`, an XNU `option` |diff --git a/tools/docs-gen.py b/tools/docs-gen.pyindex 5463fcf..ea8cb11 100755--- a/tools/docs-gen.py+++ b/tools/docs-gen.py@@ -104,7 +104,10 @@ def devices_page(hd):" | ".join(short.get(states.get(a), "") for a in areas)))out += ["", "An empty cell is a part nobody has looked at on that device yet.", "", "## SoC families", "", "| SoC | What | |", "|---|---|---|"]for name, fields in socs:- out.append("| `%s` | %s | [`socs/%s/`](%ssocs/%s) |" % (name, hd.one(fields, "title"), name, REPO.replace("/blob/", "/tree/"), name))+ carried = [d for k, d in hd.present() if k == "device" and os.path.isdir(os.path.join(ROOT, "devices", d, "soc"))+ and hd.config_tool().soc_name(os.path.join(ROOT, "devices", d, "soc")) == name]+ where = "carried by %s (`devices/%s/soc/`)" % (", ".join(carried), carried[0]) if carried else "`socs/%s/`" % name+ out.append("| `%s` | %s | %s |" % (name, hd.one(fields, "title"), where))out.append("")return "\n".join(out)diff --git a/tools/hakios-config-test.py b/tools/hakios-config-test.pyindex ea53a2b..159cea7 100755--- a/tools/hakios-config-test.py+++ b/tools/hakios-config-test.py@@ -45,11 +45,11 @@ def args_of(out):return out.stdout.split()-# The reference phone's configuration is what most of this checks against: its repositories (devices/redwood, socs/sc7280) are+# The reference phone's configuration is what most of this checks against: its repositories (devices/redwood, with its SoC layer) are# their own, and a checkout of the common repository alone does not have them until `tools/hakios-device.py sync` clones them.-if not (os.path.isdir(os.path.join(ROOT, "devices", "redwood", "configs")) and os.path.isdir(os.path.join(ROOT, "socs", "sc7280", "configs"))):- print("hakios-config-test: SKIPPED: it checks the reference phone's configuration, and devices/redwood and socs/sc7280 are not "- "checked out (tools/hakios-device.py sync redwood sc7280)")+if not (os.path.isdir(os.path.join(ROOT, "devices", "redwood", "configs")) and os.path.isdir(os.path.join(ROOT, "devices", "redwood", "soc", "configs"))):+ print("hakios-config-test: SKIPPED: it checks the reference phone's configuration, and devices/redwood is not "+ "checked out (tools/hakios-device.py sync redwood)")sys.exit(0)work = tempfile.mkdtemp(prefix="hakios-config-test-")diff --git a/tools/hakios-config.py b/tools/hakios-config.pyindex 13f825a..68decaa 100755--- a/tools/hakios-config.py+++ b/tools/hakios-config.py@@ -147,9 +147,52 @@ def layers(root):for name in sorted(os.listdir(base)) if os.path.isdir(base) else []:if os.path.isfile(os.path.join(base, name, marker)):out.append((kind, name, os.path.join(base, name)))+ # A device repository carries its SoC's layer with it (devices/<name>/soc/, with a soc.conf that names the SoC), so that one+ # repository is all a device needs. Two devices with the same SoC each carry a copy: the copies must be the same, or a build+ # would depend on which device is checked out.+ seen = {name: (top, soc_digest(top)) for kind, name, top in out if kind == "soc"}+ for kind, name, top in list(out):+ if kind != "device":+ continue+ soc = os.path.join(top, "soc")+ named = soc_name(soc)+ if named is None:+ continue+ digest = soc_digest(soc)+ if named in seen:+ if seen[named][1] != digest:+ raise ConfigError("the SoC %s is in %s and in %s, and the two differ: make them the same (one SoC, one layer)" %+ (named, os.path.relpath(seen[named][0], root), os.path.relpath(soc, root)))+ continue+ seen[named] = (soc, digest)+ out.insert(1 + sum(1 for k, _n, _t in out if k == "soc"), ("soc", named, soc))return out+def soc_name(top):+ """The name a soc.conf in TOP gives its SoC (`soc <name>`), or None if there is none."""+ path = os.path.join(top, "soc.conf")+ if not os.path.isfile(path):+ return None+ for line in open(path, encoding="utf-8"):+ m = re.match(r"soc\s+([a-z0-9][a-z0-9_-]*)\s*$", line)+ if m:+ return m.group(1)+ return None+++def soc_digest(top):+ """What a SoC layer holds, as one value: two copies of the same SoC compare equal when every file is the same."""+ import hashlib+ h = hashlib.sha256()+ for d, dirs, files in sorted(os.walk(top)):+ dirs.sort()+ for f in sorted(files):+ path = os.path.join(d, f)+ h.update(os.path.relpath(path, top).encode() + b"\0" + open(path, "rb").read() + b"\0")+ return h.hexdigest()++class Driver:def __init__(self, name, sub, where, origin=None):self.name, self.sub, self.where = name, sub, wherediff --git a/tools/hakios-device-test.py b/tools/hakios-device-test.pyindex 74a3a64..ef18422 100755--- a/tools/hakios-device-test.py+++ b/tools/hakios-device-test.py@@ -57,8 +57,24 @@ try:"# CONFIG_HAKIOS_GOODIX is not set" in conf and "HAKIOS_THERMAL" not in conf and "HAKIOS_VIRTIO" not in conf, conf)r = tool(root, "hakios-device.py", "new", "fakephone", "--soc", "fakesoc", "--title", "x")check("new: refuses a device that exists", r.returncode != 0 and "already exists" in r.stderr, r.stderr)- r = tool(root, "hakios-device.py", "new", "other", "--soc", "nosuchsoc", "--title", "x")- check("new: refuses a SoC with no repository", r.returncode != 0 and "no socs/nosuchsoc" in r.stderr, r.stderr)+ check("new: the device carries its SoC's layer, a copy of the one there is",+ open(os.path.join(root, "devices", "fakephone", "soc", "soc.conf")).read() == open(os.path.join(root, "socs", "fakesoc", "soc.conf")).read())+ r = tool(root, "hakios-device.py", "new", "otherphone", "--soc", "newsoc", "--title", "Other Phone", "--repo")+ check("new: a SoC nobody has yet gets a new layer in the device, and --repo makes the device a git repository",+ r.returncode == 0 and os.path.isfile(os.path.join(root, "devices", "otherphone", "soc", "configs", "newsoc.config")) and+ os.path.isdir(os.path.join(root, "devices", "otherphone", ".git")) and "git -C devices/otherphone remote add origin" in r.stdout,+ r.stdout + r.stderr)+ shutil.rmtree(os.path.join(root, "devices", "otherphone"))+ # two copies of one SoC: the same is fine, different is refused+ with open(os.path.join(root, "devices", "fakephone", "soc", "configs", "fakesoc.config"), "a") as f:+ f.write("CONFIG_CPUS=4\n")+ r = tool(root, "hakios-config.py", "list")+ check("two different copies of one SoC are refused, with both named", r.returncode != 0 and "the SoC fakesoc is in" in r.stderr and+ "differ" in r.stderr, r.stdout + r.stderr)+ shutil.copy(os.path.join(root, "socs", "fakesoc", "configs", "fakesoc.config"), os.path.join(root, "devices", "fakephone", "soc", "configs"))+ shutil.rmtree(os.path.join(root, "socs"))+ r = tool(root, "hakios-config.py", "defconfig", "fakephone")+ check("a device that carries its SoC needs nothing else: no socs/ at all", r.returncode == 0, r.stdout + r.stderr)# a driver of the device's ownd = os.path.join(root, "devices", "fakephone", "drivers", "input")@@ -97,7 +113,7 @@ try:check("a device repository cannot declare `standard` files", "a SoC or device repository has drivers, not `standard`" in r.stdout + r.stderr,r.stdout + r.stderr)os.remove(os.path.join(d, "conf"))- shutil.copy(os.path.join(root, "socs", "fakesoc", "configs", "fakesoc.config"),+ shutil.copy(os.path.join(root, "devices", "fakephone", "soc", "configs", "fakesoc.config"),os.path.join(root, "devices", "fakephone", "configs", "fakesoc.config"))r = tool(root, "hakios-config.py", "defconfig", "fakephone")check("the same configuration name in two repositories is refused", r.returncode != 0 and "more than one repository" in r.stderr, r.stderr)diff --git a/tools/hakios-device.py b/tools/hakios-device.pyindex d8c609b..bd4200a 100755--- a/tools/hakios-device.py+++ b/tools/hakios-device.py@@ -44,10 +44,16 @@ class Problem(Exception):pass+_CONFIG = []++def config_tool():+ if _CONFIG:+ return _CONFIG[0]spec = importlib.util.spec_from_file_location("hakios_config", os.path.join(HERE, "hakios-config.py"))module = importlib.util.module_from_spec(spec)spec.loader.exec_module(module)+ _CONFIG.append(module)return module@@ -88,10 +94,28 @@ def manifest():return read_blocks(path, ("device", "soc", "uefi"), {"device": DEVICE_KEYS, "soc": SOC_KEYS, "uefi": {"path", "repository", "revision"}})+def soc_homes():+ """{SoC name: its directory}: socs/<name>/, then the SoC layers devices carry (devices/<device>/soc/), the first of each."""+ out = {}+ base = os.path.join(ROOT, "socs")+ for name in sorted(os.listdir(base)) if os.path.isdir(base) else []:+ if os.path.isfile(os.path.join(base, name, "soc.conf")):+ out.setdefault(name, os.path.join(base, name))+ base = os.path.join(ROOT, "devices")+ for dev in sorted(os.listdir(base)) if os.path.isdir(base) else []:+ top = os.path.join(base, dev, "soc")+ named = config_tool().soc_name(top)+ if named:+ out.setdefault(named, top)+ return out++def home(kind, name):if kind == "uefi":return os.path.join(ROOT, "uefi") # one UEFI for every device: Mu-Silicium with HakiOS's changes (uefi/README.md)- return os.path.join(ROOT, "devices" if kind == "device" else "socs", name)+ if kind == "soc":+ return soc_homes().get(name, os.path.join(ROOT, "socs", name))+ return os.path.join(ROOT, "devices", name)def describe(kind, name):@@ -107,12 +131,11 @@ def describe(kind, name):def present():"""[(kind, name)] of every device and SoC checked out, whether or not the manifest names it."""- out = []- for kind, top, marker in (("soc", "socs", "soc.conf"), ("device", "devices", "device.conf")):- base = os.path.join(ROOT, top)- for name in sorted(os.listdir(base)) if os.path.isdir(base) else []:- if os.path.isfile(os.path.join(base, name, marker)):- out.append((kind, name))+ out = [("soc", name) for name in sorted(soc_homes())]+ base = os.path.join(ROOT, "devices")+ for name in sorted(os.listdir(base)) if os.path.isdir(base) else []:+ if os.path.isfile(os.path.join(base, name, "device.conf")):+ out.append(("device", name))return out@@ -178,8 +201,8 @@ def check_one(kind, name, hc):soc = one(fields, "soc")if soc is None:problems.append("%s/device.conf: no soc (`none` for a machine with no SoC repository)" % shown)- elif soc != "none" and not os.path.isfile(os.path.join(ROOT, "socs", soc, "soc.conf")):- problems.append("%s/device.conf: soc %s, and there is no socs/%s/soc.conf (tools/hakios-device.py sync %s)" % (shown, soc, soc, soc))+ elif soc != "none" and soc not in soc_homes():+ problems.append("%s/device.conf: soc %s, and neither %s/soc/ nor socs/%s/ has its soc.conf" % (shown, soc, shown, soc))defconfig = one(fields, "defconfig")path = os.path.join(top, "configs", defconfig or "")if not defconfig or not os.path.isfile(path):@@ -236,7 +259,7 @@ def cmd_check(a):NEW_README = """# {title} ({name})A HakiOS device repository: what is true of this device and of no other. The kernel, the loader, the drivers and the tools are-the common repository's; the {soc} family's facts are `socs/{soc}/`'s. How the three are put together:+the common repository's; the {soc} family's facts are in `soc/`, carried with the device. How the three are put together:[docs/porting/repositories.md](https://github.com/oosexgang/hakios/blob/main/docs/porting/repositories.md).| File | What |@@ -244,6 +267,7 @@ the common repository's; the {soc} family's facts are `socs/{soc}/`'s. How the t| `device.conf` | what the device is, and how far each part of it works: its page in the documentation is made from it || `configs/{name}_defconfig` | the common kernel, the SoC's fragment, this device's: `make {name}_defconfig` || `configs/{name}.config` | what is this device's own: the drivers built in, the arguments it needs |+| `soc/` | its SoC family's layer ({soc}): `soc.conf`, `configs/{soc}.config`, the SoC's drivers; carried here so that this one repository is all the device needs || `drivers/` | drivers of its own, if it needs any (`drivers/<subsystem>/conf`, as in the common repository) |Every phone driver of the common repository is left out to begin with: their addresses and pins are the phone they were written@@ -267,12 +291,25 @@ def cmd_new(a):print(out.stdout.rstrip())if out.returncode != 0:raise Problem("tools/new-device.py: %s" % (out.stderr.strip() or "failed"))- if a.soc != "none" and not os.path.isfile(os.path.join(ROOT, "socs", a.soc, "soc.conf")):- raise Problem("no socs/%s/soc.conf: make the SoC's repository first, or sync it (known: %s)" %- (a.soc, ", ".join(n for k, n in present() if k == "soc") or "none"))+ if a.soc != "none":+ # the device carries its SoC's layer: a copy of one there is (another device's, synced), or a new one to fill in+ socs = soc_homes()+ os.makedirs(top, exist_ok=True)+ if a.soc in socs:+ shutil.copytree(socs[a.soc], os.path.join(top, "soc"), ignore=shutil.ignore_patterns(".git"))+ print("hakios-device: devices/%s/soc/: the %s layer, copied from %s" % (name, a.soc, os.path.relpath(socs[a.soc], ROOT)))+ else:+ os.makedirs(os.path.join(top, "soc", "configs"))+ with open(os.path.join(top, "soc", "soc.conf"), "w", encoding="utf-8") as f:+ f.write("# the %s SoC family, carried by this device's repository (docs/porting/repositories.md)\n\nsoc %s\n"+ "\ttitle %s\n\tvendor ?\n\tmaintainer %s\n" % (a.soc, a.soc, a.soc_title or a.soc, a.maintainer))+ with open(os.path.join(top, "soc", "configs", a.soc + ".config"), "w", encoding="utf-8") as f:+ f.write('# HakiOS fragment: the %s family -- what is true of the SoC, whatever device it is in.\nCONFIG_SOC="%s"\n' % (a.soc, a.soc))+ print("hakios-device: devices/%s/soc/: a new SoC layer for %s: its cores, addresses and drivers go there" % (name, a.soc))hc = config_tool()# what a new device begins without: every driver written for one machine (not QEMU's), but the ones its SoC's fragment builds- soc_has = hc.parse(hc.find_config(ROOT, a.soc + ".config"))[0] if a.soc != "none" else {}+ soc_fragment = os.path.join(top, "soc", "configs", a.soc + ".config") # the copy the device now carries+ soc_has = hc.parse(soc_fragment)[0] if a.soc != "none" else {}left_out = [d.symbol for d in hc.DRIVERS if not d.name.startswith("VIRTIO_") and soc_has.get(d.symbol) != "y"]os.makedirs(os.path.join(top, "configs"), exist_ok=True)with open(os.path.join(top, "device.conf"), "w", encoding="utf-8") as f:@@ -290,7 +327,7 @@ def cmd_new(a):f.write('CONFIG_DEVICE="%s"\n' % name)for sym in left_out:f.write("# CONFIG_%s is not set\n" % sym)- common = hc.overlay({}, [hc.find_config(ROOT, "common_defconfig")] + ([hc.find_config(ROOT, a.soc + ".config")] if soc_has else []))+ common = hc.overlay({}, [hc.find_config(ROOT, "common_defconfig")] + ([soc_fragment] if a.soc != "none" else []))problems, _ = hc.check(dict(common, **{s: "n" for s in left_out}))for p in problems:m = re.match(r"CONFIG_([A-Z0-9_]+) gives ", p)@@ -307,6 +344,13 @@ def cmd_new(a):print("hakios-device: devices/%s made: device.conf, configs/%s_defconfig, configs/%s.config, README.md" % (name, name, name))print("hakios-device: %d phone drivers left out to begin with; `make %s_defconfig && make config` shows what is built" % (len(left_out), name))print("hakios-device: next: docs/porting/index.md, step by step; tools/new-device.py reads the phone's own device tree")+ if a.repo:+ git("init", "-q", "-b", "main", cwd=top)+ git("add", "-A", cwd=top)+ git("commit", "-q", "-m", "%s: a HakiOS device repository, made by tools/hakios-device.py new" % name, cwd=top)+ print("hakios-device: devices/%s is a git repository; to publish it: make an empty repository (hakios-%s), then\n"+ " git -C devices/%s remote add origin <its URL> && git -C devices/%s push -u origin main\n"+ "and add its block to devices/manifest.conf (repository, revision) in a pull request" % (name, name, name, name))return cmd_check(argparse.Namespace(names=[name]))@@ -400,7 +444,9 @@ def main():p.set_defaults(fn=cmd_check)p = sub.add_parser("new", help="a new device repository")p.add_argument("name")- p.add_argument("--soc", required=True, help="its SoC family (socs/<soc>/), or none")+ p.add_argument("--soc", required=True, help="its SoC family: copied from a device or socs/ that has it, else a new layer; or none")+ p.add_argument("--soc-title", help="for a new SoC layer: what the family is called")+ p.add_argument("--repo", action="store_true", help="make the device's directory a git repository, ready to push")p.add_argument("--title", required=True, help='what it is called ("POCO X5 Pro 5G")')p.add_argument("--vendor", default="?")p.add_argument("--maintainer", default="?")diff --git a/tools/publish-repos.sh b/tools/publish-repos.shindex eb3a43b..ea65647 100755--- a/tools/publish-repos.sh+++ b/tools/publish-repos.sh@@ -9,7 +9,7 @@set -eDIR=${1:?usage: tools/publish-repos.sh DIR [OWNER]}OWNER=${2:-oosexgang}-REPOS="hakios hakios-sc7280-common hakios-redwood hakios-uefi hakios-web"+REPOS=${REPOS:-"hakios hakios-redwood hakios-uefi hakios-web"}for r in $REPOS; do[ -d "$DIR/$r/.git" ] || { echo "publish-repos: no $DIR/$r (tools/split-repos.sh makes the first four; the website is hakios-web)"; continue; }url="https://github.com/$OWNER/$r.git"diff --git a/tools/split-repos.sh b/tools/split-repos.shdeleted file mode 100755index 9c725d4..0000000--- a/tools/split-repos.sh+++ /dev/null@@ -1,103 +0,0 @@-#!/bin/bash-# Make the HakiOS repositories out of this one, the way UniXNU keeps its parts apart: the common repository, a SoC's, a device's,-# the UEFI. Each one keeps the history of its own files (a clone with git filter-branch --subdirectory-filter: git subtree, as-# Xcode ships it, dies on a history this long). This repository is not changed: the results are new clones in OUT, ready to push.-#-# tools/split-repos.sh OUT [OWNER] OWNER: where they will live on GitHub (default oosexgang, the organization)-#-# OUT/hakios common: everything but the parts below; devices/manifest.conf names those by repository-# OUT/hakios-sc7280-common socs/sc7280-# OUT/hakios-redwood devices/redwood-# OUT/hakios-uefi uefi-#-# Commit messages lose their "Co-Authored-By: Claude ..." and "Claude-Session: ..." lines (the project's contributors are its people); authors and dates-# are kept as they are. Pushing is tools/publish-repos.sh's, with your own GitHub login.-set -e-SRC=$(cd "$(dirname "$(readlink -f "$0")")/.." && pwd)-OUT=${1:?usage: tools/split-repos.sh OUT [OWNER]}-OWNER=${2:-oosexgang}-mkdir -p "$OUT"-OUT=$(cd "$OUT" && pwd)-[ -z "$(git -C "$SRC" status --porcelain)" ] || { echo "split-repos: commit first: the parts are made from what is committed" >&2; exit 1; }--PARTS="socs/sc7280:hakios-sc7280-common:soc:sc7280 devices/redwood:hakios-redwood:device:redwood uefi:hakios-uefi:uefi:hakios-uefi"--MSG="sed -E '/^[Cc]o-[Aa]uthored-[Bb]y: Claude/d; /^Claude-Session: /d' | sed -e :a -e '/^\n*\$/{\$d;N;ba' -e '}'"--# a clone of this repository's current branch as main, with nothing else in it-fresh_clone() {- git clone -q --no-local --single-branch --branch "$(git -C "$SRC" rev-parse --abbrev-ref HEAD)" "$SRC" "$1"- git -C "$1" branch -q -m main- git -C "$1" remote remove origin- for t in $(git -C "$1" tag); do git -C "$1" tag -d "$t" >/dev/null; done-}--# the messages without Claude's trailer lines (and no blank line left at the end); with a directory, only its history, at the top-rewrite() {- FILTER_BRANCH_SQUELCH_WARNING=1 git -C "$1" filter-branch -f --msg-filter "$MSG" ${2:+--subdirectory-filter "$2"} -- main >/dev/null- for ref in $(git -C "$1" for-each-ref --format='%(refname)' refs/original); do git -C "$1" update-ref -d "$ref"; done- git -C "$1" reflog expire --expire=now --all- git -C "$1" gc -q --prune=now-}--for part in $PARTS; do- IFS=: read -r prefix name _kind _id <<<"$part"- [ -e "$OUT/$name" ] && { echo "split-repos: $OUT/$name exists" >&2; exit 1; }- echo "split-repos: $prefix -> $name"- fresh_clone "$OUT/$name"- rewrite "$OUT/$name" "$prefix"-done--echo "split-repos: the common repository -> hakios"-[ -e "$OUT/hakios" ] && { echo "split-repos: $OUT/hakios exists" >&2; exit 1; }-C=$OUT/hakios-fresh_clone "$C"-rewrite "$C"-for part in $PARTS; do- IFS=: read -r prefix name kind id <<<"$part"- git -C "$C" rm -q -r "$prefix"- rev=$(git -C "$OUT/$name" rev-parse HEAD)- python3 - "$C/devices/manifest.conf" "$kind" "$id" "https://github.com/$OWNER/$name.git" "$rev" <<'PY'-import re, sys-path, kind, name, url, rev = sys.argv[1:]-text = open(path).read()-block = re.compile(r"^%s %s\n(?:\t.*\n?)*" % (kind, re.escape(name)), re.M)-new = "%s %s\n\trepository %s\n\trevision %s\n" % (kind, name, url, rev)-text, n = block.subn(new, text)-assert n == 1, (kind, name)-open(path, "w").write(text)-PY-done-# what sync clones into the tree is not the common repository's-python3 - "$C/.gitignore" <<'PY'-import sys-path = sys.argv[1]-text = open(path).read()-text = text.replace("!/uefi/\n", "")-text += ("\n# The SoC, device and UEFI repositories that tools/hakios-device.py sync clones into place (devices/manifest.conf) are their own:\n"- "/socs/*\n!/socs/README.md\n/devices/*\n!/devices/manifest.conf\n!/devices/README.md\n!/devices/qemu/\n")-open(path, "w").write(text)-PY-mkdir -p "$C/socs"-cat > "$C/socs/README.md" <<'EOF'-# SoC repositories--`tools/hakios-device.py sync` clones each SoC repository `devices/manifest.conf` names into `socs/<name>/`. A new one:-[How the repositories fit](../docs/porting/repositories.md).-EOF-cat > "$C/devices/README.md" <<'EOF'-# Device repositories--`devices/manifest.conf` names every device; `tools/hakios-device.py sync` clones each into `devices/<name>/`. `devices/qemu/` is-kept here: QEMU's machine, with none of a phone's drivers, the device every checkout of the common repository can build and run.-A device of your own: `tools/hakios-device.py new` ([Port a device](../docs/porting/index.md)).-EOF-git -C "$C" add -A-git -C "$C" commit -q -m "The SoC, device and UEFI repositories are their own: hakios-sc7280-common, hakios-redwood, hakios-uefi--devices/manifest.conf names them by repository and revision; tools/hakios-device.py sync clones them into socs/, devices/ and uefi/."-echo "split-repos: done in $OUT"-for r in hakios $(for p in $PARTS; do echo "$p" | cut -d: -f2; done); do- printf ' %-22s %5s commits, %s\n' "$r" "$(git -C "$OUT/$r" rev-list --count HEAD)" "$(git -C "$OUT/$r" log -1 --format=%h)"-done-echo "split-repos: next: tools/publish-repos.sh $OUT"