Skip to content

Debloat your Android TV, install apps, browse and transfer files on it, all from your phone. Just 3 MB, no tracking, no root, no PC.

data/index.json carries its signature on line 2

tellyscope-agent21015d2735da
diff --git a/.github/workflows/data.yml b/.github/workflows/data.yml
index ffec904..6755191 100644
--- a/.github/workflows/data.yml
+++ b/.github/workflows/data.yml
@@ -19,5 +19,5 @@ jobs:
- uses: actions/checkout@v7
- name: Tests
run: python3 -m unittest discover -s tools -p 'test_data.py' -v
- - name: data/index.json is fresh and data/index.json.sig verifies
+ - name: data/index.json is fresh and its signature verifies
run: python3 tools/data.py check
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 63cece5..58745b2 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -42,8 +42,20 @@ The tests use small fixtures in [`app/src/test/resources/fixtures/`](app/src/tes
### Signing `data/`
-The app trusts `data/` only when `data/index.json.sig` verifies: an ECDSA P-256 / SHA-256 signature over
-the exact bytes of `data/index.json`, which holds the sha256 of every file in `data/`. The public keys
+The app trusts `data/` only when `data/index.json` is signed. It holds the sha256 of every file in
+`data/`, and its line 2 holds the signatures:
+
+```json
+{
+ "sigs": ["p256:<base64>"],
+ "schema": 1,
+ …
+```
+
+Each entry is an ECDSA P-256 / SHA-256 signature (DER, base64) over the exact bytes of the file with
+line 2 cut out. Nothing parses and rewrites the JSON to check it, so a file reformatted after signing
+fails, and `index` puts it right. A checker skips tags it doesn't know and passes if any `p256` entry
+verifies against any key, so a new key or algorithm can sign next to the old one. The public keys
are built into the app and kept in [`tools/data-keys/`](tools/data-keys/) for the `Data` check: a primary,
used for every update, and a backup kept offline, so losing the primary doesn't lock apps out. The
private keys stay on the maintainer's machine, never in this repo or in CI. Make each one once:
@@ -54,7 +66,8 @@ openssl pkey -in tellyscope-data-primary.pem -pubout -outform DER | openssl base
# the same with tellyscope-data-backup.pem > tools/data-keys/backup.pub, then store the backup offline
```
-Changing a public key needs an app release.
+Changing a public key needs an app release. To rotate, sign with both private keys
+(`sign --key old.pem --key new.pem`) until no app that knows only the old key is in use.
**Sign last, after review, and only a branch you'll merge.** Once a signed `data/index.json` is on
GitHub it stays fetchable by its commit forever, and anyone serving it gets it treated as signed, even
@@ -67,10 +80,18 @@ Every change to `data/`:
2. After review, on the pull request's branch, the maintainer runs
`python3 tools/data.py sign --key ~/path/to/tellyscope-data-primary.pem`. It refuses a key that
`tools/data-keys/` doesn't verify.
-3. Commit `data/index.json.sig`, let `Data` pass, merge.
+3. Commit `data/index.json`, let `Data` pass, merge.
+
+`index` leaves `data/index.json` alone, signature included, while nothing changes. Otherwise it raises
+`version` (`YYYYMMDDNN`, UTC) and writes it unsigned; the app refuses an index older than the one it has.
+
+**The format is fixed once an app ships with it**, so it changes only by adding:
-`index` keeps `version` (`YYYYMMDDNN`, UTC) while nothing else changes and raises it otherwise; the app
-refuses an index older than the one it has.
+- A new field, in any file, keeps `schema: 1`. Apps ignore keys they don't know, so add a field only
+ when an app that ignores it still behaves correctly.
+- A breaking change goes in a new folder, `data/v2/`, with its own signed `index.json`, read by the app
+ versions that understand it. `data/` stays as it is and keeps serving older apps until they're gone.
+- A new list is a new entry in `lists`, and older apps skip it.
## README screenshots
diff --git a/data/index.json.sig b/data/index.json.sig
deleted file mode 100644
index 7845171..0000000
--- a/data/index.json.sig
+++ /dev/null
@@ -1 +0,0 @@
-MEYCIQDq2aXgn1jiRqokCTHszRIEKHm01+98Q2nFCTgq3daONAIhAKE/NwR0vwF5ZacHwJnZk2jLQMCZpycVh/qadGq7UZQI
diff --git a/tools/data.py b/tools/data.py
index 2749eb3..2d5db68 100644
--- a/tools/data.py
+++ b/tools/data.py
@@ -1,10 +1,14 @@
#!/usr/bin/env python3
"""Index and sign data/, the lists the app downloads from this repo. Python 3.9+, standard library only.
- tools/data.py index rebuild data/index.json from data/
- tools/data.py sign --key <private key.pem> write data/index.json.sig, signed with that key
- tools/data.py check [--unsigned] fail if data/index.json is stale or data/index.json.sig
- doesn't verify against tools/data-keys/ (--unsigned skips that)
+ tools/data.py index rebuild data/index.json from data/, unsigned; left alone
+ while nothing changed
+ tools/data.py sign --key <private key.pem>… sign data/index.json with each key, on its line 2
+ tools/data.py check [--unsigned] fail if data/index.json is stale or its signature doesn't
+ verify against tools/data-keys/ (--unsigned skips that)
+
+The signature sits on line 2 of data/index.json, ` "sigs": ["p256:<base64>", …],`. What is signed is
+the file without that line, cut out as bytes: never re-serialise the JSON to check it.
Signing and checking signatures need the openssl command. How and when to sign: CONTRIBUTING.md.
"""
@@ -22,6 +26,7 @@ REPO = Path(__file__).resolve().parent.parent
DATA = REPO / "data"
KEYS = REPO / "tools" / "data-keys"
LISTS = ("launchers", "labels", "streaming")
+SIGS = b' "sigs": '
# Every P-256 public key in SubjectPublicKeyInfo DER starts with these bytes, then 64 bytes of point.
P256_SPKI = bytes.fromhex("3059301306072a8648ce3d020106082a8648ce3d030107034200" "04")
@@ -43,6 +48,7 @@ def version(index):
old one. The app refuses an index older than the one it has."""
path = DATA / "index.json"
old = json.loads(path.read_text(encoding="utf-8")) if path.is_file() else {}
+ old.pop("sigs", None)
old.pop("schema", None)
previous = old.pop("version", 0)
if old == index and previous:
@@ -68,7 +74,7 @@ def public_keys():
else:
keys.append(der)
if not keys and not errors:
- errors.append("tools/data-keys/: no public key to check data/index.json.sig against")
+ errors.append("tools/data-keys/: no public key to check data/index.json's signature against")
return keys, errors
@@ -87,45 +93,87 @@ def verifies(data, signature, key):
"-signature", str(tmp / "sig"), str(tmp / "data")).returncode == 0
+def split(raw):
+ """(unsigned bytes, sigs line or None): line 2 is the signature line when it starts with "sigs",
+ and the unsigned bytes are the file with exactly that line cut out."""
+ first = raw.find(b"\n") + 1
+ second = raw.find(b"\n", first) + 1
+ if first and second and raw[first:second].startswith(SIGS):
+ return raw[:first] + raw[second:], raw[first:second]
+ return raw, None
+
+
+def signatures(line):
+ """The p256 signatures on a sigs line, as DER; entries with other tags are skipped. None if the
+ line isn't ` "sigs": [<strings>],`."""
+ body = line[len(SIGS):].rstrip(b"\n")
+ try:
+ entries = json.loads(body[:-1]) if body.endswith(b",") else None
+ except ValueError:
+ entries = None
+ if not isinstance(entries, list) or not all(isinstance(e, str) for e in entries):
+ return None
+ found = []
+ for tag, _, value in (e.partition(":") for e in entries):
+ if tag == "p256":
+ try:
+ found.append(base64.b64decode(value, validate=True))
+ except ValueError:
+ pass
+ return found
+
+
def check_signature():
keys, errors = public_keys()
- sig = DATA / "index.json.sig"
if errors:
return errors
- if not sig.is_file():
- return ["data/index.json.sig is missing: run tools/data.py sign --key <private key>"]
- try:
- signature = base64.b64decode(sig.read_text(encoding="utf-8").strip(), validate=True)
- except ValueError:
- return ["data/index.json.sig: not base64"]
- if not any(verifies((DATA / "index.json").read_bytes(), signature, k) for k in keys):
- return ["data/index.json.sig doesn't verify against tools/data-keys/: data/index.json changed "
+ unsigned, line = split((DATA / "index.json").read_bytes())
+ if line is None:
+ return ["data/index.json is unsigned: run tools/data.py sign --key <private key>"]
+ sigs = signatures(line)
+ if sigs is None:
+ return ['data/index.json: line 2 isn\'t ` "sigs": ["p256:<base64>", …],`']
+ if not any(verifies(unsigned, sig, k) for sig in sigs for k in keys):
+ return ["data/index.json's signature doesn't verify against tools/data-keys/: the file changed "
"since it was signed, run tools/data.py sign --key <private key>"]
return []
-def sign(key):
- """Sign data/index.json's exact bytes with key and write data/index.json.sig, if a key in
- tools/data-keys/ verifies it."""
- data = (DATA / "index.json").read_bytes()
+def sign(keys):
+ """Sign data/index.json without its sigs line with each key, and write the signatures on line 2,
+ if a key in tools/data-keys/ verifies every one of them."""
+ path = DATA / "index.json"
+ unsigned, _ = split(path.read_bytes())
+ public, errors = public_keys()
+ entries = []
with tempfile.TemporaryDirectory() as tmp:
- (Path(tmp) / "data").write_bytes(data)
- run = openssl("dgst", "-sha256", "-sign", str(key), str(Path(tmp) / "data"))
- if run.returncode != 0:
- print(run.stderr.decode(errors="replace"), file=sys.stderr)
- return 1
- keys, errors = public_keys()
- if errors or not any(verifies(data, run.stdout, k) for k in keys):
- print(*errors, "the signature doesn't verify against any key in tools/data-keys/: wrong key?",
- sep="\n", file=sys.stderr)
- return 1
- (DATA / "index.json.sig").write_text(base64.b64encode(run.stdout).decode() + "\n", encoding="utf-8")
+ (Path(tmp) / "data").write_bytes(unsigned)
+ for key in keys:
+ run = openssl("dgst", "-sha256", "-sign", str(key), str(Path(tmp) / "data"))
+ if run.returncode != 0:
+ print(run.stderr.decode(errors="replace"), file=sys.stderr)
+ return 1
+ if errors or not any(verifies(unsigned, run.stdout, k) for k in public):
+ print(*errors, f"{key}: the signature doesn't verify against any key in tools/data-keys/: "
+ "wrong key?", sep="\n", file=sys.stderr)
+ return 1
+ entries.append("p256:" + base64.b64encode(run.stdout).decode())
+ first = unsigned.find(b"\n") + 1
+ path.write_bytes(unsigned[:first] + SIGS + json.dumps(entries).encode() + b",\n" + unsigned[first:])
return 0
+def index():
+ """Write data/index.json unsigned, unless it already says the same, so a signature stays put."""
+ path = DATA / "index.json"
+ text = dump(build_index())
+ if not path.is_file() or split(path.read_bytes())[0] != text.encode():
+ path.write_text(text, encoding="utf-8")
+
+
def check(signed=True):
- index = DATA / "index.json"
- if not index.is_file() or index.read_text(encoding="utf-8") != dump(build_index()):
+ path = DATA / "index.json"
+ if not path.is_file() or split(path.read_bytes())[0] != dump(build_index()).encode():
errors = ["data/index.json is stale: run tools/data.py index"]
else:
errors = check_signature() if signed else []
@@ -137,10 +185,10 @@ def check(signed=True):
def main(argv):
if argv[:1] == ["check"] and argv[1:] in ([], ["--unsigned"]):
return check(signed=not argv[1:])
- if argv[:2] == ["sign", "--key"] and len(argv) == 3:
- return sign(argv[2])
+ if argv[:1] == ["sign"] and len(argv) > 1 and argv[1::2] == ["--key"] * (len(argv) // 2) and len(argv) % 2:
+ return sign(argv[2::2])
if argv == ["index"]:
- (DATA / "index.json").write_text(dump(build_index()), encoding="utf-8")
+ index()
return 0
print(__doc__, file=sys.stderr)
return 2
diff --git a/tools/test_data.py b/tools/test_data.py
index 54847e6..af538dd 100644
--- a/tools/test_data.py
+++ b/tools/test_data.py
@@ -11,7 +11,7 @@ import data # noqa: E402
class Signing(unittest.TestCase):
- """data/index.json.sig is what the app trusts, so check must fail whenever it doesn't cover the index."""
+ """Line 2 of data/index.json is what the app trusts, so check must fail whenever it doesn't cover the rest."""
def setUp(self):
self.root = Path(tempfile.mkdtemp())
@@ -26,34 +26,97 @@ class Signing(unittest.TestCase):
"-out", str(self.root / f"{name}.pem")], check=True, capture_output=True)
(self.root / "keys" / "primary.pub").write_text("# test\n" + self.public("primary") + "\n")
(self.root / "keys" / "backup.pub").write_text("# placeholder\n")
- (self.root / "data" / "index.json").write_text('{"schema": 1, "version": 2026110301}\n')
+ self.index().write_text(data.dump({"schema": 1, "version": 2026110301}))
+
+ def index(self):
+ return self.root / "data" / "index.json"
def public(self, name):
der = subprocess.run(["openssl", "pkey", "-in", str(self.root / f"{name}.pem"), "-pubout", "-outform", "DER"],
check=True, capture_output=True).stdout
return base64.b64encode(der).decode()
- def test_a_signed_index_passes(self):
- self.assertEqual(data.sign(self.root / "primary.pem"), 0)
+ def sigs(self, *entries):
+ """Puts a sigs line with these entries on line 2."""
+ unsigned, _ = data.split(self.index().read_bytes())
+ line = data.SIGS + data.json.dumps(list(entries)).encode() + b",\n"
+ self.index().write_bytes(unsigned[:2] + line + unsigned[2:])
+
+ def entries(self):
+ return data.json.loads(self.index().read_text())["sigs"]
+
+ def test_a_signed_index_passes_and_is_still_json(self):
+ before = self.index().read_bytes()
+ self.assertEqual(data.sign([self.root / "primary.pem"]), 0)
self.assertEqual(data.check_signature(), [])
+ self.assertEqual(data.split(self.index().read_bytes())[0], before)
+ self.assertEqual(self.index().read_text().splitlines()[1][:len(' "sigs": ["p256:')], ' "sigs": ["p256:')
+ self.assertEqual(data.json.loads(self.index().read_text())["version"], 2026110301)
def test_an_edit_after_signing_fails(self):
- data.sign(self.root / "primary.pem")
- (self.root / "data" / "index.json").write_text('{"schema": 1, "version": 2026110302}\n')
+ data.sign([self.root / "primary.pem"])
+ self.index().write_text(self.index().read_text().replace("2026110301", "2026110302"))
self.assertIn("doesn't verify", "\n".join(data.check_signature()))
- def test_a_missing_signature_fails(self):
- self.assertIn("index.json.sig is missing", "\n".join(data.check_signature()))
+ def test_a_reformatted_index_fails(self):
+ data.sign([self.root / "primary.pem"])
+ self.index().write_text(self.index().read_text().replace("\n ", "\n "))
+ self.assertNotEqual(data.check_signature(), [])
+
+ def test_an_unsigned_index_fails(self):
+ self.assertIn("is unsigned", "\n".join(data.check_signature()))
+
+ def test_a_sigs_line_that_is_not_a_list_of_strings_fails(self):
+ self.sigs(1)
+ self.assertIn("line 2 isn't", "\n".join(data.check_signature()))
def test_sign_refuses_a_key_not_in_keys(self):
- self.assertEqual(data.sign(self.root / "other.pem"), 1)
- self.assertFalse((self.root / "data" / "index.json.sig").exists())
+ before = self.index().read_bytes()
+ self.assertEqual(data.sign([self.root / "other.pem"]), 1)
+ self.assertEqual(self.index().read_bytes(), before)
+
+ def test_signing_again_replaces_the_signatures(self):
+ data.sign([self.root / "primary.pem"])
+ data.sign([self.root / "primary.pem"])
+ self.assertEqual(len(self.entries()), 1)
+ self.assertEqual(data.check_signature(), [])
def test_the_backup_key_verifies_too(self):
(self.root / "keys" / "backup.pub").write_text(self.public("other") + "\n")
- data.sign(self.root / "other.pem")
+ data.sign([self.root / "other.pem"])
self.assertEqual(data.check_signature(), [])
+ def test_two_keys_sign_side_by_side_and_either_passes(self):
+ (self.root / "keys" / "backup.pub").write_text(self.public("other") + "\n")
+ self.assertEqual(data.sign([self.root / "primary.pem", self.root / "other.pem"]), 0)
+ first, second = self.entries()
+ self.sigs("p256:" + base64.b64encode(b"junk").decode(), second)
+ self.assertEqual(data.check_signature(), [])
+ self.sigs(first)
+ self.assertEqual(data.check_signature(), [])
+
+ def test_an_unknown_tag_is_skipped(self):
+ data.sign([self.root / "primary.pem"])
+ (good,) = self.entries()
+ self.sigs("ed25519:" + good[len("p256:"):], good)
+ self.assertEqual(data.check_signature(), [])
+ self.sigs("ed25519:" + good[len("p256:"):])
+ self.assertIn("doesn't verify", "\n".join(data.check_signature()))
+
+ def test_index_leaves_a_signed_file_alone_and_check_sees_through_the_signature(self):
+ for name in data.LISTS:
+ (self.root / "data" / f"{name}.json").write_text("[]\n")
+ data.index()
+ data.sign([self.root / "primary.pem"])
+ signed = self.index().read_bytes()
+ data.index()
+ self.assertEqual(self.index().read_bytes(), signed)
+ self.assertEqual(data.check(), 0)
+ (self.root / "data" / "labels.json").write_text("{}\n")
+ self.assertEqual(data.check(signed=False), 1)
+ data.index()
+ self.assertIsNone(data.split(self.index().read_bytes())[1])
+
def test_a_key_that_is_not_p256_is_caught(self):
(self.root / "keys" / "backup.pub").write_text("bm90IGEga2V5\n")
self.assertIn("not a base64 P-256 public key", "\n".join(data.public_keys()[1]))