Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

One repository per device: it carries its SoC's layer (soc/); new copies or makes it, and --repo makes a git repository to push

shimoku74a6e3cfdf6164
diff --git a/Makefile b/Makefile
index af21276..3d05fce 100644
--- a/Makefile
+++ b/Makefile
@@ -19,7 +19,7 @@
#
# Configuration, the way Linux's make does it (configs/, tools/hakios-config.py; docs/configs.md):
# make redwood_defconfig .config from configs/defconfig + redwood_defconfig (devices/redwood/configs/), which is
-# common_defconfig + sc7280.config (socs/sc7280/) + redwood.config
+# common_defconfig + sc7280.config (devices/redwood/soc/) + redwood.config
# (any NAME_defconfig; make common_defconfig is the common kernel alone; docs/configs.md)
# make devices the device and SoC repositories (tools/hakios-device.py list; docs/porting/repositories.md)
# make virt.config lay a fragment (configs/virt.config) over the .config there is
diff --git a/README.md b/README.md
index 77c8a1c..4c794d0 100644
--- a/README.md
+++ b/README.md
@@ -62,13 +62,12 @@ make phone SECS=600 # a 10-minute session on the phone, in RAM
| Repository | What |
|---|---|
| [hakios](https://github.com/oosexgang/hakios) | **common**: the kernel port, every common driver, the loader, user space, the tools, this documentation, and `devices/qemu` |
-| [hakios-sc7280-common](https://github.com/oosexgang/hakios-sc7280-common) | the Qualcomm SC7280 family (Snapdragon 778G): its configuration fragment and the drivers it builds |
-| [hakios-redwood](https://github.com/oosexgang/hakios-redwood) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script |
+| [hakios-redwood](https://github.com/oosexgang/hakios-redwood) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script, and its SoC's layer (`soc/`: the Qualcomm SC7280 family) |
| [hakios-uefi](https://github.com/oosexgang/hakios-uefi) | HakiOS's UEFI: Mu-Silicium with HakiOS's boot path, banner and name |
| [hakios-web](https://github.com/oosexgang/hakios-web) | the website, with this documentation and the FAQ |
`devices/manifest.conf` in the common repository names each one and the revision it is at; `tools/hakios-device.py sync` clones them
-into `socs/`, `devices/` and `uefi/`. [How the repositories fit](docs/porting/repositories.md).
+into `devices/` and `uefi/`. One repository per device: it carries its SoC's layer, and `tools/hakios-device.py new` makes one. [How the repositories fit](docs/porting/repositories.md).
```sh
git clone https://github.com/oosexgang/hakios.git && cd hakios
diff --git a/devices/manifest.conf b/devices/manifest.conf
index ae575bd..ad3c578 100644
--- a/devices/manifest.conf
+++ b/devices/manifest.conf
@@ -1,6 +1,7 @@
# devices/manifest.conf: every SoC and device repository HakiOS knows of (Android's repo manifest, UniXNU's devices.toml).
#
-# soc|device|uefi <name> (the UEFI is one for every device: it goes to uefi/)
+# device|uefi <name> (a device carries its SoC's layer in soc/; the UEFI is one, in uefi/)
+# soc <name> (a SoC layer of its own, in socs/<name>: for one many devices share)
# path <directory in this tree> kept here, for now: tools/hakios-device.py split makes it a repository
# repository <git URL> a repository of its own, cloned into socs/<name> or devices/<name>
# revision <commit> ... at this commit (tools/hakios-device.py lock writes it)
@@ -8,13 +9,9 @@
# `tools/hakios-device.py sync` clones and updates the ones with a repository; `list` shows them all; docs/porting/repositories.md
# is the guide. A device of yours: open a pull request that adds its block here, with its repository and revision.
-soc sc7280
- repository https://github.com/oosexgang/hakios-sc7280-common.git
- revision 1ef455525b734ba2066ff294aa11f7e5be3c20fc
-
device redwood
repository https://github.com/oosexgang/hakios-redwood.git
- revision da8a0b307f17f43a79929c1b2d49d1fed4ec1a75
+ revision 84c2cb861bccfb4d76fba9976b2d7810a95ed662
device qemu
path devices/qemu
diff --git a/docs/concepts/architecture.md b/docs/concepts/architecture.md
index 21a111e..ccbf840 100644
--- a/docs/concepts/architecture.md
+++ b/docs/concepts/architecture.md
@@ -26,8 +26,8 @@ yours (the user space you lay out).
```
configs/common_defconfig (common)
- socs/sc7280/configs/sc7280.config (SoC) make redwood_defconfig
- devices/redwood/configs/redwood.config (device) -------------------------> .config
+ devices/redwood/soc/configs/sc7280.config (SoC) make redwood_defconfig
+ devices/redwood/configs/redwood.config (device) -------------------------> .config
|
CONFIG_HAKIOS_* -> tools/hakios-config.py kernel -> XNU's options (config/MASTER.hakios, conf/files) -> make kernel
every other CONFIG_* -> tools/hakios-config.py args -> the kernel's boot-arguments -> make phone, ./xl run
@@ -46,7 +46,7 @@ device's. A device's defconfig is three lines: the common kernel, its SoC's frag
| Layer | Has | Example |
|---|---|---|
| common | the kernel port, every driver, the loader, user space, the tools, the documentation | this repository |
-| SoC | the SoC's facts (cores, addresses that are the family's), its drivers | `socs/sc7280` |
+| SoC | the SoC's facts (cores, addresses that are the family's), its drivers; carried by each device that has the SoC | `devices/redwood/soc` |
| device | its defconfig, its status, its UEFI platform, its firmware script, drivers of its own | `devices/redwood` |
| UEFI | Mu-Silicium with HakiOS's boot path, banner and name | `uefi` ([its README](https://github.com/oosexgang/hakios/blob/main/uefi/README.md)) |
@@ -69,7 +69,7 @@ device's. A device's defconfig is three lines: the common kernel, its SoC's frag
| `xnuloader/` | the boot loader (a UEFI application, no EDK2) |
| `userland/` | programs and libraries of our own (`src/`), the Metal plug-in, the image build, launchd jobs |
| `configs/` | the common kernel's configuration and the generic fragments |
-| `socs/`, `devices/` | the SoC and device repositories, and `devices/manifest.conf` |
+| `devices/` | the device repositories (each with its SoC's layer in `soc/`), and `devices/manifest.conf` |
| `sysimage/` | scripts that fetch and unpack the OS files from the IPSW (none of the files) |
| `tools/` | configuration, devices, checks, APFS, firmware, QMI, the run scripts |
| `qemu/` | running in QEMU |
diff --git a/docs/getting-started/phone.md b/docs/getting-started/phone.md
index fd1291f..702aa9a 100644
--- a/docs/getting-started/phone.md
+++ b/docs/getting-started/phone.md
@@ -20,7 +20,7 @@ that you create yourself, and nowhere else. [What is written, exactly](../concep
## Step by step
```sh
-tools/hakios-device.py sync # hakios-redwood, hakios-sc7280-common, hakios-uefi into devices/, socs/, uefi/
+tools/hakios-device.py sync # hakios-redwood (with its SoC layer) and hakios-uefi into devices/ and uefi/
devices/redwood/pull-firmware.sh # Android running, on adb: GPU, Wi-Fi, audio, haptics, modem, cameras -> fw-*/
make redwood_defconfig # the phone's configuration: the common kernel, the SC7280's, the phone's own
make config # what it is: the drivers built in, the kernel's arguments, the firmware it wants
diff --git a/docs/index.md b/docs/index.md
index bb47780..dec3ee0 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -45,10 +45,9 @@ Android. What is and is not written to a phone: [Architecture](concepts/architec
| Repository | What |
|---|---|
| [hakios](https://github.com/oosexgang/hakios) | **common**: the kernel port, every common driver, the loader, user space, the tools, this documentation, and `devices/qemu` |
-| [hakios-sc7280-common](https://github.com/oosexgang/hakios-sc7280-common) | the Qualcomm SC7280 family (Snapdragon 778G): its configuration fragment and the drivers it builds |
-| [hakios-redwood](https://github.com/oosexgang/hakios-redwood) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script |
+| [hakios-redwood](https://github.com/oosexgang/hakios-redwood) | POCO X5 Pro 5G: its defconfig, status, UEFI platform, firmware script, and its SoC's layer (`soc/`: the Qualcomm SC7280 family) |
| [hakios-uefi](https://github.com/oosexgang/hakios-uefi) | HakiOS's UEFI: Mu-Silicium with HakiOS's boot path, banner and name |
| [hakios-web](https://github.com/oosexgang/hakios-web) | the website, with this documentation and the FAQ |
`devices/manifest.conf` in the common repository names each one and the revision it is at; `tools/hakios-device.py sync` clones them
-into `socs/`, `devices/` and `uefi/`. [How the repositories fit](porting/repositories.md).
+into `devices/` and `uefi/`. One repository per device: it carries its SoC's layer, and `tools/hakios-device.py new` makes one. [How the repositories fit](porting/repositories.md).
diff --git a/docs/porting/drivers.md b/docs/porting/drivers.md
index 6265e4d..43b2703 100644
--- a/docs/porting/drivers.md
+++ b/docs/porting/drivers.md
@@ -10,7 +10,7 @@ reference for the mechanism.
|---|---|
| for hardware many devices share, or a SoC family's | the common repository: `drivers/<subsystem>/` |
| one device's own (a touch controller only it has) | that device's repository: `devices/<name>/drivers/<subsystem>/` |
-| a SoC family's own | `socs/<soc>/drivers/<subsystem>/` |
+| a SoC family's own | the SoC's layer: `devices/<name>/soc/drivers/<subsystem>/` |
Subsystems: input, storage, gpu, camera, haptic, audio, power, wifi, bluetooth, modem, net, core.
diff --git a/docs/porting/index.md b/docs/porting/index.md
index a403d30..6ebe69b 100644
--- a/docs/porting/index.md
+++ b/docs/porting/index.md
@@ -31,8 +31,9 @@ That makes `devices/mydevice/` with:
left out**: their addresses, pins and rails are the phone they were written for, and a driver of another phone touching yours
is the last thing you want. The drivers of your SoC (from its fragment) and QEMU's stay in.
-A SoC without a repository yet: make `socs/<soc>/` the same way by hand (`soc.conf`, `configs/<soc>.config`), from
-[socs/sc7280](https://github.com/oosexgang/hakios/tree/main/socs/sc7280).
+The SoC's layer comes with the device, in `devices/mydevice/soc/`: a copy of the one another device carries when the SoC is one
+HakiOS knows (`--soc sc7280` copies redwood's), or a new one to fill in for a SoC nobody has ported yet (`--soc sm8250
+--soc-title "Snapdragon 865"`). `--repo` makes the directory a git repository, ready to push as `hakios-mydevice`.
**Done when** `tools/hakios-device.py check mydevice` says it holds together, and the README says which SoC values differ and you
know why.
diff --git a/docs/porting/repositories.md b/docs/porting/repositories.md
index 4410f4b..9e8b5f8 100644
--- a/docs/porting/repositories.md
+++ b/docs/porting/repositories.md
@@ -4,10 +4,11 @@ HakiOS is put together from layers, the way an Android device's source is (`devi
that names them):
```
-hakios/ common: the kernel port, every common driver, the loader, user space, the tools, the docs
-├── socs/<soc>/ a SoC family: soc.conf, configs/<soc>.config, drivers/ of its own
-├── devices/<name>/ a device: device.conf, configs/<name>_defconfig and <name>.config, drivers/, its UEFI, scripts
-└── devices/manifest.conf every SoC and device repository: kept in this tree, or a repository of its own at a revision
+hakios/ common: the kernel port, every common driver, the loader, user space, the tools, the docs
+├── devices/<name>/ a device, one repository: device.conf, configs/, drivers/, its UEFI platform, scripts
+│ └── soc/ ... and its SoC family's layer: soc.conf, configs/<soc>.config, the SoC's drivers
+├── uefi/ HakiOS's UEFI (Mu-Silicium with HakiOS's changes), one for every device
+└── devices/manifest.conf every device and the UEFI: kept in this tree, or a repository of its own at a revision
```
A SoC or device directory is the same whether it is kept in the common tree (`devices/qemu` is) or is a git repository of its own
@@ -22,6 +23,7 @@ that `tools/hakios-device.py sync` cloned into place (the others are): the tools
| `configs/<name>_defconfig` | `#include common_defconfig`, `#include <soc>.config`, `#include <name>.config` | yes |
| `configs/<name>.config` | what is this device's own: drivers left out or in, arguments, `CONFIG_DEVICE="<name>"` | yes |
| `drivers/<subsystem>/conf` and the files it names | drivers of its own ([Writing a driver](drivers.md)) | no |
+| `soc/` | its SoC family's layer: `soc.conf`, `configs/<soc>.config`, the SoC's drivers ([below](#the-socs-layer)) | yes, unless `soc none` |
| `pull-firmware.sh` | the device's firmware, from your own device, into `fw-*/` | for a phone |
| `uefi/`, `prebuilt/` | the device's UEFI port and the images made of it | for a phone |
| `README.md` | anything else a person needs to know | yes |
@@ -53,7 +55,9 @@ device redwood
The device's page in this documentation ([redwood](../devices/redwood.md)) and the table of [all devices](../devices/index.md)
are made from it by `tools/docs-gen.py`: the status is written once, next to the code it describes.
-## A SoC repository
+## The SoC's layer
+
+A device carries its SoC family's layer with it, in `soc/`, so that one repository is all a device needs:
```
soc sc7280
@@ -64,17 +68,21 @@ soc sc7280
maintainer the HakiOS developers (README.md)
```
-and `configs/<soc>.config`, the SoC's facts: `CONFIG_SOC`, its cores, and **the drivers of the SoC family**
-(`CONFIG_HAKIOS_CPUFREQ=y`, `CONFIG_HAKIOS_THERMAL=y`, ...), so that every device with the SoC has them, whatever it leaves out of
-its own.
+and `soc/configs/<soc>.config`, the SoC's facts: `CONFIG_SOC`, its cores, and **the drivers of the SoC family**
+(`CONFIG_HAKIOS_CPUFREQ=y`, `CONFIG_HAKIOS_THERMAL=y`, ...), so that the device has them whatever it leaves out of its own.
+
+Two devices with the same SoC each carry a copy. The copies must be the same: the tools take the SoC once, and refuse two copies that
+differ, naming both, because a build would otherwise depend on which device was checked out. `tools/hakios-device.py new --soc sc7280`
+copies the layer from a device that has it; a SoC nobody has yet gets a new layer to fill in. A SoC many devices share can also be a
+repository of its own in `socs/<soc>/` (a `soc` block in the manifest); the tools read both.
## The manifest
`devices/manifest.conf` names every SoC and device:
```
-soc sc7280
- repository https://github.com/oosexgang/hakios-sc7280-common.git
+device redwood
+ repository https://github.com/oosexgang/hakios-redwood.git
revision
device qemu
@@ -89,7 +97,7 @@ device sheng
|---|---|
| `tools/hakios-device.py list` (`make devices`) | every device and SoC, where it is, how much of it works |
| `tools/hakios-device.py check [NAME...]` | a repository holds together: its conf reads, its defconfig includes the common kernel and its SoC's fragment and passes `hakios-config.py check`, its drivers' files are there, no two repositories have a configuration of the same name |
-| `tools/hakios-device.py new NAME --soc SOC --title ...` | a new device repository ([Port a device](index.md)) |
+| `tools/hakios-device.py new NAME --soc SOC --title ... [--fdt FILE] [--repo]` | a whole new device repository, its SoC layer with it; `--repo` makes it a git repository ready to push ([Port a device](index.md)) |
| `tools/hakios-device.py sync [NAME...]` | clone or update the repositories that live elsewhere, at their revisions |
| `tools/hakios-device.py lock NAME` | write the revision checked out into the manifest |
| `tools/hakios-device.py split NAME DIR` | a device kept in this tree, as a git repository of its own with its history, ready to push |
@@ -114,7 +122,7 @@ cannot declare `standard` files: what every kernel has is the common repository'
| | Android | UniXNU | HakiOS |
|---|---|---|---|
| the shared part | AOSP, the GKI kernel | `UniXNU` (tools, components, ARM64 support) | `hakios` (the kernel port, drivers, loader, tools) |
-| a SoC | `<soc>-common` | `UniXNU-<soc>-common` (`soc.toml`) | `socs/<soc>` (`soc.conf`) |
+| a SoC | `<soc>-common` | `UniXNU-<soc>-common` (`soc.toml`) | `soc/` inside each device repository (`soc.conf`); `socs/<soc>` for a shared one |
| a device | `device/<vendor>/<name>` | `UniXNU-<name>` (`board.toml`) | `devices/<name>` (`device.conf`) |
| what names them | `repo` manifest | `devices.toml` | `devices/manifest.conf` |
| a driver | kernel module, `Kconfig` | a component repository | `drivers/<sub>/conf`, an XNU `option` |
diff --git a/tools/docs-gen.py b/tools/docs-gen.py
index 5463fcf..ea8cb11 100755
--- a/tools/docs-gen.py
+++ b/tools/docs-gen.py
@@ -104,7 +104,10 @@ def devices_page(hd):
" | ".join(short.get(states.get(a), "") for a in areas)))
out += ["", "An empty cell is a part nobody has looked at on that device yet.", "", "## SoC families", "", "| SoC | What | |", "|---|---|---|"]
for name, fields in socs:
- out.append("| `%s` | %s | [`socs/%s/`](%ssocs/%s) |" % (name, hd.one(fields, "title"), name, REPO.replace("/blob/", "/tree/"), name))
+ carried = [d for k, d in hd.present() if k == "device" and os.path.isdir(os.path.join(ROOT, "devices", d, "soc"))
+ and hd.config_tool().soc_name(os.path.join(ROOT, "devices", d, "soc")) == name]
+ where = "carried by %s (`devices/%s/soc/`)" % (", ".join(carried), carried[0]) if carried else "`socs/%s/`" % name
+ out.append("| `%s` | %s | %s |" % (name, hd.one(fields, "title"), where))
out.append("")
return "\n".join(out)
diff --git a/tools/hakios-config-test.py b/tools/hakios-config-test.py
index ea53a2b..159cea7 100755
--- a/tools/hakios-config-test.py
+++ b/tools/hakios-config-test.py
@@ -45,11 +45,11 @@ def args_of(out):
return out.stdout.split()
-# The reference phone's configuration is what most of this checks against: its repositories (devices/redwood, socs/sc7280) are
+# The reference phone's configuration is what most of this checks against: its repositories (devices/redwood, with its SoC layer) are
# their own, and a checkout of the common repository alone does not have them until `tools/hakios-device.py sync` clones them.
-if not (os.path.isdir(os.path.join(ROOT, "devices", "redwood", "configs")) and os.path.isdir(os.path.join(ROOT, "socs", "sc7280", "configs"))):
- print("hakios-config-test: SKIPPED: it checks the reference phone's configuration, and devices/redwood and socs/sc7280 are not "
- "checked out (tools/hakios-device.py sync redwood sc7280)")
+if not (os.path.isdir(os.path.join(ROOT, "devices", "redwood", "configs")) and os.path.isdir(os.path.join(ROOT, "devices", "redwood", "soc", "configs"))):
+ print("hakios-config-test: SKIPPED: it checks the reference phone's configuration, and devices/redwood is not "
+ "checked out (tools/hakios-device.py sync redwood)")
sys.exit(0)
work = tempfile.mkdtemp(prefix="hakios-config-test-")
diff --git a/tools/hakios-config.py b/tools/hakios-config.py
index 13f825a..68decaa 100755
--- a/tools/hakios-config.py
+++ b/tools/hakios-config.py
@@ -147,9 +147,52 @@ def layers(root):
for name in sorted(os.listdir(base)) if os.path.isdir(base) else []:
if os.path.isfile(os.path.join(base, name, marker)):
out.append((kind, name, os.path.join(base, name)))
+ # A device repository carries its SoC's layer with it (devices/<name>/soc/, with a soc.conf that names the SoC), so that one
+ # repository is all a device needs. Two devices with the same SoC each carry a copy: the copies must be the same, or a build
+ # would depend on which device is checked out.
+ seen = {name: (top, soc_digest(top)) for kind, name, top in out if kind == "soc"}
+ for kind, name, top in list(out):
+ if kind != "device":
+ continue
+ soc = os.path.join(top, "soc")
+ named = soc_name(soc)
+ if named is None:
+ continue
+ digest = soc_digest(soc)
+ if named in seen:
+ if seen[named][1] != digest:
+ raise ConfigError("the SoC %s is in %s and in %s, and the two differ: make them the same (one SoC, one layer)" %
+ (named, os.path.relpath(seen[named][0], root), os.path.relpath(soc, root)))
+ continue
+ seen[named] = (soc, digest)
+ out.insert(1 + sum(1 for k, _n, _t in out if k == "soc"), ("soc", named, soc))
return out
+def soc_name(top):
+ """The name a soc.conf in TOP gives its SoC (`soc <name>`), or None if there is none."""
+ path = os.path.join(top, "soc.conf")
+ if not os.path.isfile(path):
+ return None
+ for line in open(path, encoding="utf-8"):
+ m = re.match(r"soc\s+([a-z0-9][a-z0-9_-]*)\s*$", line)
+ if m:
+ return m.group(1)
+ return None
+
+
+def soc_digest(top):
+ """What a SoC layer holds, as one value: two copies of the same SoC compare equal when every file is the same."""
+ import hashlib
+ h = hashlib.sha256()
+ for d, dirs, files in sorted(os.walk(top)):
+ dirs.sort()
+ for f in sorted(files):
+ path = os.path.join(d, f)
+ h.update(os.path.relpath(path, top).encode() + b"\0" + open(path, "rb").read() + b"\0")
+ return h.hexdigest()
+
+
class Driver:
def __init__(self, name, sub, where, origin=None):
self.name, self.sub, self.where = name, sub, where
diff --git a/tools/hakios-device-test.py b/tools/hakios-device-test.py
index 74a3a64..ef18422 100755
--- a/tools/hakios-device-test.py
+++ b/tools/hakios-device-test.py
@@ -57,8 +57,24 @@ try:
"# CONFIG_HAKIOS_GOODIX is not set" in conf and "HAKIOS_THERMAL" not in conf and "HAKIOS_VIRTIO" not in conf, conf)
r = tool(root, "hakios-device.py", "new", "fakephone", "--soc", "fakesoc", "--title", "x")
check("new: refuses a device that exists", r.returncode != 0 and "already exists" in r.stderr, r.stderr)
- r = tool(root, "hakios-device.py", "new", "other", "--soc", "nosuchsoc", "--title", "x")
- check("new: refuses a SoC with no repository", r.returncode != 0 and "no socs/nosuchsoc" in r.stderr, r.stderr)
+ check("new: the device carries its SoC's layer, a copy of the one there is",
+ open(os.path.join(root, "devices", "fakephone", "soc", "soc.conf")).read() == open(os.path.join(root, "socs", "fakesoc", "soc.conf")).read())
+ r = tool(root, "hakios-device.py", "new", "otherphone", "--soc", "newsoc", "--title", "Other Phone", "--repo")
+ check("new: a SoC nobody has yet gets a new layer in the device, and --repo makes the device a git repository",
+ r.returncode == 0 and os.path.isfile(os.path.join(root, "devices", "otherphone", "soc", "configs", "newsoc.config")) and
+ os.path.isdir(os.path.join(root, "devices", "otherphone", ".git")) and "git -C devices/otherphone remote add origin" in r.stdout,
+ r.stdout + r.stderr)
+ shutil.rmtree(os.path.join(root, "devices", "otherphone"))
+ # two copies of one SoC: the same is fine, different is refused
+ with open(os.path.join(root, "devices", "fakephone", "soc", "configs", "fakesoc.config"), "a") as f:
+ f.write("CONFIG_CPUS=4\n")
+ r = tool(root, "hakios-config.py", "list")
+ check("two different copies of one SoC are refused, with both named", r.returncode != 0 and "the SoC fakesoc is in" in r.stderr and
+ "differ" in r.stderr, r.stdout + r.stderr)
+ shutil.copy(os.path.join(root, "socs", "fakesoc", "configs", "fakesoc.config"), os.path.join(root, "devices", "fakephone", "soc", "configs"))
+ shutil.rmtree(os.path.join(root, "socs"))
+ r = tool(root, "hakios-config.py", "defconfig", "fakephone")
+ check("a device that carries its SoC needs nothing else: no socs/ at all", r.returncode == 0, r.stdout + r.stderr)
# a driver of the device's own
d = os.path.join(root, "devices", "fakephone", "drivers", "input")
@@ -97,7 +113,7 @@ try:
check("a device repository cannot declare `standard` files", "a SoC or device repository has drivers, not `standard`" in r.stdout + r.stderr,
r.stdout + r.stderr)
os.remove(os.path.join(d, "conf"))
- shutil.copy(os.path.join(root, "socs", "fakesoc", "configs", "fakesoc.config"),
+ shutil.copy(os.path.join(root, "devices", "fakephone", "soc", "configs", "fakesoc.config"),
os.path.join(root, "devices", "fakephone", "configs", "fakesoc.config"))
r = tool(root, "hakios-config.py", "defconfig", "fakephone")
check("the same configuration name in two repositories is refused", r.returncode != 0 and "more than one repository" in r.stderr, r.stderr)
diff --git a/tools/hakios-device.py b/tools/hakios-device.py
index d8c609b..bd4200a 100755
--- a/tools/hakios-device.py
+++ b/tools/hakios-device.py
@@ -44,10 +44,16 @@ class Problem(Exception):
pass
+_CONFIG = []
+
+
def config_tool():
+ if _CONFIG:
+ return _CONFIG[0]
spec = importlib.util.spec_from_file_location("hakios_config", os.path.join(HERE, "hakios-config.py"))
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
+ _CONFIG.append(module)
return module
@@ -88,10 +94,28 @@ def manifest():
return read_blocks(path, ("device", "soc", "uefi"), {"device": DEVICE_KEYS, "soc": SOC_KEYS, "uefi": {"path", "repository", "revision"}})
+def soc_homes():
+ """{SoC name: its directory}: socs/<name>/, then the SoC layers devices carry (devices/<device>/soc/), the first of each."""
+ out = {}
+ base = os.path.join(ROOT, "socs")
+ for name in sorted(os.listdir(base)) if os.path.isdir(base) else []:
+ if os.path.isfile(os.path.join(base, name, "soc.conf")):
+ out.setdefault(name, os.path.join(base, name))
+ base = os.path.join(ROOT, "devices")
+ for dev in sorted(os.listdir(base)) if os.path.isdir(base) else []:
+ top = os.path.join(base, dev, "soc")
+ named = config_tool().soc_name(top)
+ if named:
+ out.setdefault(named, top)
+ return out
+
+
def home(kind, name):
if kind == "uefi":
return os.path.join(ROOT, "uefi") # one UEFI for every device: Mu-Silicium with HakiOS's changes (uefi/README.md)
- return os.path.join(ROOT, "devices" if kind == "device" else "socs", name)
+ if kind == "soc":
+ return soc_homes().get(name, os.path.join(ROOT, "socs", name))
+ return os.path.join(ROOT, "devices", name)
def describe(kind, name):
@@ -107,12 +131,11 @@ def describe(kind, name):
def present():
"""[(kind, name)] of every device and SoC checked out, whether or not the manifest names it."""
- out = []
- for kind, top, marker in (("soc", "socs", "soc.conf"), ("device", "devices", "device.conf")):
- base = os.path.join(ROOT, top)
- for name in sorted(os.listdir(base)) if os.path.isdir(base) else []:
- if os.path.isfile(os.path.join(base, name, marker)):
- out.append((kind, name))
+ out = [("soc", name) for name in sorted(soc_homes())]
+ base = os.path.join(ROOT, "devices")
+ for name in sorted(os.listdir(base)) if os.path.isdir(base) else []:
+ if os.path.isfile(os.path.join(base, name, "device.conf")):
+ out.append(("device", name))
return out
@@ -178,8 +201,8 @@ def check_one(kind, name, hc):
soc = one(fields, "soc")
if soc is None:
problems.append("%s/device.conf: no soc (`none` for a machine with no SoC repository)" % shown)
- elif soc != "none" and not os.path.isfile(os.path.join(ROOT, "socs", soc, "soc.conf")):
- problems.append("%s/device.conf: soc %s, and there is no socs/%s/soc.conf (tools/hakios-device.py sync %s)" % (shown, soc, soc, soc))
+ elif soc != "none" and soc not in soc_homes():
+ problems.append("%s/device.conf: soc %s, and neither %s/soc/ nor socs/%s/ has its soc.conf" % (shown, soc, shown, soc))
defconfig = one(fields, "defconfig")
path = os.path.join(top, "configs", defconfig or "")
if not defconfig or not os.path.isfile(path):
@@ -236,7 +259,7 @@ def cmd_check(a):
NEW_README = """# {title} ({name})
A HakiOS device repository: what is true of this device and of no other. The kernel, the loader, the drivers and the tools are
-the common repository's; the {soc} family's facts are `socs/{soc}/`'s. How the three are put together:
+the common repository's; the {soc} family's facts are in `soc/`, carried with the device. How the three are put together:
[docs/porting/repositories.md](https://github.com/oosexgang/hakios/blob/main/docs/porting/repositories.md).
| File | What |
@@ -244,6 +267,7 @@ the common repository's; the {soc} family's facts are `socs/{soc}/`'s. How the t
| `device.conf` | what the device is, and how far each part of it works: its page in the documentation is made from it |
| `configs/{name}_defconfig` | the common kernel, the SoC's fragment, this device's: `make {name}_defconfig` |
| `configs/{name}.config` | what is this device's own: the drivers built in, the arguments it needs |
+| `soc/` | its SoC family's layer ({soc}): `soc.conf`, `configs/{soc}.config`, the SoC's drivers; carried here so that this one repository is all the device needs |
| `drivers/` | drivers of its own, if it needs any (`drivers/<subsystem>/conf`, as in the common repository) |
Every phone driver of the common repository is left out to begin with: their addresses and pins are the phone they were written
@@ -267,12 +291,25 @@ def cmd_new(a):
print(out.stdout.rstrip())
if out.returncode != 0:
raise Problem("tools/new-device.py: %s" % (out.stderr.strip() or "failed"))
- if a.soc != "none" and not os.path.isfile(os.path.join(ROOT, "socs", a.soc, "soc.conf")):
- raise Problem("no socs/%s/soc.conf: make the SoC's repository first, or sync it (known: %s)" %
- (a.soc, ", ".join(n for k, n in present() if k == "soc") or "none"))
+ if a.soc != "none":
+ # the device carries its SoC's layer: a copy of one there is (another device's, synced), or a new one to fill in
+ socs = soc_homes()
+ os.makedirs(top, exist_ok=True)
+ if a.soc in socs:
+ shutil.copytree(socs[a.soc], os.path.join(top, "soc"), ignore=shutil.ignore_patterns(".git"))
+ print("hakios-device: devices/%s/soc/: the %s layer, copied from %s" % (name, a.soc, os.path.relpath(socs[a.soc], ROOT)))
+ else:
+ os.makedirs(os.path.join(top, "soc", "configs"))
+ with open(os.path.join(top, "soc", "soc.conf"), "w", encoding="utf-8") as f:
+ f.write("# the %s SoC family, carried by this device's repository (docs/porting/repositories.md)\n\nsoc %s\n"
+ "\ttitle %s\n\tvendor ?\n\tmaintainer %s\n" % (a.soc, a.soc, a.soc_title or a.soc, a.maintainer))
+ with open(os.path.join(top, "soc", "configs", a.soc + ".config"), "w", encoding="utf-8") as f:
+ f.write('# HakiOS fragment: the %s family -- what is true of the SoC, whatever device it is in.\nCONFIG_SOC="%s"\n' % (a.soc, a.soc))
+ print("hakios-device: devices/%s/soc/: a new SoC layer for %s: its cores, addresses and drivers go there" % (name, a.soc))
hc = config_tool()
# what a new device begins without: every driver written for one machine (not QEMU's), but the ones its SoC's fragment builds
- soc_has = hc.parse(hc.find_config(ROOT, a.soc + ".config"))[0] if a.soc != "none" else {}
+ soc_fragment = os.path.join(top, "soc", "configs", a.soc + ".config") # the copy the device now carries
+ soc_has = hc.parse(soc_fragment)[0] if a.soc != "none" else {}
left_out = [d.symbol for d in hc.DRIVERS if not d.name.startswith("VIRTIO_") and soc_has.get(d.symbol) != "y"]
os.makedirs(os.path.join(top, "configs"), exist_ok=True)
with open(os.path.join(top, "device.conf"), "w", encoding="utf-8") as f:
@@ -290,7 +327,7 @@ def cmd_new(a):
f.write('CONFIG_DEVICE="%s"\n' % name)
for sym in left_out:
f.write("# CONFIG_%s is not set\n" % sym)
- common = hc.overlay({}, [hc.find_config(ROOT, "common_defconfig")] + ([hc.find_config(ROOT, a.soc + ".config")] if soc_has else []))
+ common = hc.overlay({}, [hc.find_config(ROOT, "common_defconfig")] + ([soc_fragment] if a.soc != "none" else []))
problems, _ = hc.check(dict(common, **{s: "n" for s in left_out}))
for p in problems:
m = re.match(r"CONFIG_([A-Z0-9_]+) gives ", p)
@@ -307,6 +344,13 @@ def cmd_new(a):
print("hakios-device: devices/%s made: device.conf, configs/%s_defconfig, configs/%s.config, README.md" % (name, name, name))
print("hakios-device: %d phone drivers left out to begin with; `make %s_defconfig && make config` shows what is built" % (len(left_out), name))
print("hakios-device: next: docs/porting/index.md, step by step; tools/new-device.py reads the phone's own device tree")
+ if a.repo:
+ git("init", "-q", "-b", "main", cwd=top)
+ git("add", "-A", cwd=top)
+ git("commit", "-q", "-m", "%s: a HakiOS device repository, made by tools/hakios-device.py new" % name, cwd=top)
+ print("hakios-device: devices/%s is a git repository; to publish it: make an empty repository (hakios-%s), then\n"
+ " git -C devices/%s remote add origin <its URL> && git -C devices/%s push -u origin main\n"
+ "and add its block to devices/manifest.conf (repository, revision) in a pull request" % (name, name, name, name))
return cmd_check(argparse.Namespace(names=[name]))
@@ -400,7 +444,9 @@ def main():
p.set_defaults(fn=cmd_check)
p = sub.add_parser("new", help="a new device repository")
p.add_argument("name")
- p.add_argument("--soc", required=True, help="its SoC family (socs/<soc>/), or none")
+ p.add_argument("--soc", required=True, help="its SoC family: copied from a device or socs/ that has it, else a new layer; or none")
+ p.add_argument("--soc-title", help="for a new SoC layer: what the family is called")
+ p.add_argument("--repo", action="store_true", help="make the device's directory a git repository, ready to push")
p.add_argument("--title", required=True, help='what it is called ("POCO X5 Pro 5G")')
p.add_argument("--vendor", default="?")
p.add_argument("--maintainer", default="?")
diff --git a/tools/publish-repos.sh b/tools/publish-repos.sh
index eb3a43b..ea65647 100755
--- a/tools/publish-repos.sh
+++ b/tools/publish-repos.sh
@@ -9,7 +9,7 @@
set -e
DIR=${1:?usage: tools/publish-repos.sh DIR [OWNER]}
OWNER=${2:-oosexgang}
-REPOS="hakios hakios-sc7280-common hakios-redwood hakios-uefi hakios-web"
+REPOS=${REPOS:-"hakios hakios-redwood hakios-uefi hakios-web"}
for r in $REPOS; do
[ -d "$DIR/$r/.git" ] || { echo "publish-repos: no $DIR/$r (tools/split-repos.sh makes the first four; the website is hakios-web)"; continue; }
url="https://github.com/$OWNER/$r.git"
diff --git a/tools/split-repos.sh b/tools/split-repos.sh
deleted file mode 100755
index 9c725d4..0000000
--- a/tools/split-repos.sh
+++ /dev/null
@@ -1,103 +0,0 @@
-#!/bin/bash
-# Make the HakiOS repositories out of this one, the way UniXNU keeps its parts apart: the common repository, a SoC's, a device's,
-# the UEFI. Each one keeps the history of its own files (a clone with git filter-branch --subdirectory-filter: git subtree, as
-# Xcode ships it, dies on a history this long). This repository is not changed: the results are new clones in OUT, ready to push.
-#
-# tools/split-repos.sh OUT [OWNER] OWNER: where they will live on GitHub (default oosexgang, the organization)
-#
-# OUT/hakios common: everything but the parts below; devices/manifest.conf names those by repository
-# OUT/hakios-sc7280-common socs/sc7280
-# OUT/hakios-redwood devices/redwood
-# OUT/hakios-uefi uefi
-#
-# Commit messages lose their "Co-Authored-By: Claude ..." and "Claude-Session: ..." lines (the project's contributors are its people); authors and dates
-# are kept as they are. Pushing is tools/publish-repos.sh's, with your own GitHub login.
-set -e
-SRC=$(cd "$(dirname "$(readlink -f "$0")")/.." && pwd)
-OUT=${1:?usage: tools/split-repos.sh OUT [OWNER]}
-OWNER=${2:-oosexgang}
-mkdir -p "$OUT"
-OUT=$(cd "$OUT" && pwd)
-[ -z "$(git -C "$SRC" status --porcelain)" ] || { echo "split-repos: commit first: the parts are made from what is committed" >&2; exit 1; }
-
-PARTS="socs/sc7280:hakios-sc7280-common:soc:sc7280 devices/redwood:hakios-redwood:device:redwood uefi:hakios-uefi:uefi:hakios-uefi"
-
-MSG="sed -E '/^[Cc]o-[Aa]uthored-[Bb]y: Claude/d; /^Claude-Session: /d' | sed -e :a -e '/^\n*\$/{\$d;N;ba' -e '}'"
-
-# a clone of this repository's current branch as main, with nothing else in it
-fresh_clone() {
- git clone -q --no-local --single-branch --branch "$(git -C "$SRC" rev-parse --abbrev-ref HEAD)" "$SRC" "$1"
- git -C "$1" branch -q -m main
- git -C "$1" remote remove origin
- for t in $(git -C "$1" tag); do git -C "$1" tag -d "$t" >/dev/null; done
-}
-
-# the messages without Claude's trailer lines (and no blank line left at the end); with a directory, only its history, at the top
-rewrite() {
- FILTER_BRANCH_SQUELCH_WARNING=1 git -C "$1" filter-branch -f --msg-filter "$MSG" ${2:+--subdirectory-filter "$2"} -- main >/dev/null
- for ref in $(git -C "$1" for-each-ref --format='%(refname)' refs/original); do git -C "$1" update-ref -d "$ref"; done
- git -C "$1" reflog expire --expire=now --all
- git -C "$1" gc -q --prune=now
-}
-
-for part in $PARTS; do
- IFS=: read -r prefix name _kind _id <<<"$part"
- [ -e "$OUT/$name" ] && { echo "split-repos: $OUT/$name exists" >&2; exit 1; }
- echo "split-repos: $prefix -> $name"
- fresh_clone "$OUT/$name"
- rewrite "$OUT/$name" "$prefix"
-done
-
-echo "split-repos: the common repository -> hakios"
-[ -e "$OUT/hakios" ] && { echo "split-repos: $OUT/hakios exists" >&2; exit 1; }
-C=$OUT/hakios
-fresh_clone "$C"
-rewrite "$C"
-for part in $PARTS; do
- IFS=: read -r prefix name kind id <<<"$part"
- git -C "$C" rm -q -r "$prefix"
- rev=$(git -C "$OUT/$name" rev-parse HEAD)
- python3 - "$C/devices/manifest.conf" "$kind" "$id" "https://github.com/$OWNER/$name.git" "$rev" <<'PY'
-import re, sys
-path, kind, name, url, rev = sys.argv[1:]
-text = open(path).read()
-block = re.compile(r"^%s %s\n(?:\t.*\n?)*" % (kind, re.escape(name)), re.M)
-new = "%s %s\n\trepository %s\n\trevision %s\n" % (kind, name, url, rev)
-text, n = block.subn(new, text)
-assert n == 1, (kind, name)
-open(path, "w").write(text)
-PY
-done
-# what sync clones into the tree is not the common repository's
-python3 - "$C/.gitignore" <<'PY'
-import sys
-path = sys.argv[1]
-text = open(path).read()
-text = text.replace("!/uefi/\n", "")
-text += ("\n# The SoC, device and UEFI repositories that tools/hakios-device.py sync clones into place (devices/manifest.conf) are their own:\n"
- "/socs/*\n!/socs/README.md\n/devices/*\n!/devices/manifest.conf\n!/devices/README.md\n!/devices/qemu/\n")
-open(path, "w").write(text)
-PY
-mkdir -p "$C/socs"
-cat > "$C/socs/README.md" <<'EOF'
-# SoC repositories
-
-`tools/hakios-device.py sync` clones each SoC repository `devices/manifest.conf` names into `socs/<name>/`. A new one:
-[How the repositories fit](../docs/porting/repositories.md).
-EOF
-cat > "$C/devices/README.md" <<'EOF'
-# Device repositories
-
-`devices/manifest.conf` names every device; `tools/hakios-device.py sync` clones each into `devices/<name>/`. `devices/qemu/` is
-kept here: QEMU's machine, with none of a phone's drivers, the device every checkout of the common repository can build and run.
-A device of your own: `tools/hakios-device.py new` ([Port a device](../docs/porting/index.md)).
-EOF
-git -C "$C" add -A
-git -C "$C" commit -q -m "The SoC, device and UEFI repositories are their own: hakios-sc7280-common, hakios-redwood, hakios-uefi
-
-devices/manifest.conf names them by repository and revision; tools/hakios-device.py sync clones them into socs/, devices/ and uefi/."
-echo "split-repos: done in $OUT"
-for r in hakios $(for p in $PARTS; do echo "$p" | cut -d: -f2; done); do
- printf ' %-22s %5s commits, %s\n' "$r" "$(git -C "$OUT/$r" rev-list --count HEAD)" "$(git -C "$OUT/$r" log -1 --format=%h)"
-done
-echo "split-repos: next: tools/publish-repos.sh $OUT"