Skip to content

A self-hostable bookmark-everything app (links, notes and images) with AI-based automatic tagging and full text search

chore: Move away from weblate for i18n

Mohamed Bassem0a7fbf467b0c
diff --git a/.agents/skills/fill-translations/SKILL.md b/.agents/skills/fill-translations/SKILL.md
new file mode 100644
index 00000000..d4f6b77b
--- /dev/null
+++ b/.agents/skills/fill-translations/SKILL.md
@@ -0,0 +1,73 @@
+---
+name: fill-translations
+description: Find and fill missing or outdated translation strings in the web app's locales, using the English (en) strings as the source of truth. Use when asked to update, sync, fill or fix translations/i18n, or after adding or changing English strings in apps/web/lib/i18n/locales/en/translation.json.
+---
+
+# Filling missing and outdated translations
+
+The web app's strings live in `apps/web/lib/i18n/locales/<locale>/translation.json` (nested i18next JSON). `en` is the source of truth. Features usually add keys to `en` only, so the other locales drift.
+
+`i18n.mjs` in this directory compares every locale against `en` and reports three kinds of problem per key:
+
+- **MISSING**: the key exists in `en` but is absent or empty in the locale.
+- **OUTDATED**: the `en` value changed after the locale's value last changed. The script works this out from the git history of both files plus the working tree, so it also catches `en` edits you haven't committed yet.
+- **OBSOLETE**: the key exists in the locale but no longer in `en` (usually a removed or renamed key).
+
+Plurals follow i18next's `_one`/`_other` suffixes. For each plural group in `en`, the script expects the locale's own CLDR categories (from `Intl.PluralRules`). Arabic, for example, needs `_zero`, `_one`, `_two`, `_few`, `_many` and `_other`, and Japanese needs only `_other`.
+
+## 1. Report
+
+From the repo root:
+
+```sh
+node .agents/skills/fill-translations/i18n.mjs report # summary table + details
+node .agents/skills/fill-translations/i18n.mjs report --locales de,fr # subset
+node .agents/skills/fill-translations/i18n.mjs report --json # machine-readable
+```
+
+Each OUTDATED entry prints the current `en` value, the `en` value at the time the locale was last translated, and the current translation. Use the diff between the two `en` values to see what to change in the translation.
+
+## 2. Translate
+
+For each locale with work to do, write a flat JSON file of `"dotted.key": "translation"` to `$TMPDIR`, for example `$TMPDIR/i18n-de.json`. Include every MISSING and OUTDATED key. There are about 30 locales. When many need work, split them across parallel subagents, giving each a few locales and these rules:
+
+- Translate from the **current** `en` value. Read the surrounding keys in the locale file and reuse the terms it already uses (e.g. its word for "bookmark", "list", "tag", "highlight"), its formality level (du/Sie, tu/vous) and its capitalization style.
+- Keep `{{placeholders}}` exactly as they are, untranslated. Keep numbered tags like `<1>…</1>` and keep their contents translated. Keep product names (Karakeep), keyboard shortcuts and technical tokens (URL, API, RSS, OAuth) unchanged.
+- Match the `en` punctuation style, e.g. `…` rather than `...` when `en` uses it.
+- For OUTDATED keys, always write a value, even when the change is only punctuation or wording. A key whose value doesn't change stays flagged.
+- For plural groups, write every form the report lists as missing for that locale, using the language's grammar for each category (not a copy of `_other`).
+- `en_US` is an English variant. Copy the `en` values (with US spelling).
+- `zhtw` is Traditional Chinese, `zh` is Simplified Chinese, and `nb_NO` is Norwegian Bokmål.
+- For an OBSOLETE key, check whether it's a renamed key whose new name is MISSING. If the meaning didn't change, reuse the old translation for the new key.
+
+## 3. Apply
+
+```sh
+node .agents/skills/fill-translations/i18n.mjs apply de $TMPDIR/i18n-de.json --prune
+```
+
+`apply` merges the values into the locale file and puts each new key next to its `en` sibling, so the diff stays small. It writes the file in the repo's format (2-space JSON with a trailing newline). `--prune` removes OBSOLETE keys. Before pruning, check that a key really is unused. A key built dynamically in code (e.g. ``t(`admin.background_jobs.${name}`)``) shows up as obsolete only if it was also removed from `en`, so `en` is the reference. If in doubt, grep `apps/web` for the key's last segment.
+
+## 4. Verify
+
+```sh
+node .agents/skills/fill-translations/i18n.mjs report
+git diff --stat apps/web/lib/i18n/locales
+```
+
+Every locale should report 0 missing and 0 obsolete. The only OUTDATED entries left should be ones where you deliberately kept the old value. Skim the diff for keys that were copied without being translated, and for broken placeholders. To list the placeholder mismatches:
+
+```sh
+node -e '
+const fs=require("fs"),d="apps/web/lib/i18n/locales";
+const flat=(o,p="",r={})=>{for(const[k,v]of Object.entries(o)){const q=p?p+"."+k:k;typeof v=="object"?flat(v,q,r):r[q]=v}return r};
+const ph=s=>(s.match(/\{\{[^}]+\}\}|<\/?\d+>/g)||[]).sort().join(" ");
+const en=flat(JSON.parse(fs.readFileSync(d+"/en/translation.json")));
+for(const l of fs.readdirSync(d)){const f=d+"/"+l+"/translation.json";if(!fs.existsSync(f))continue;
+ for(const[k,v]of Object.entries(flat(JSON.parse(fs.readFileSync(f))))){const e=en[k]??en[k.replace(/_(zero|one|two|few|many)$/,"_other")];
+ if(e!==undefined&&ph(e)!==ph(v)&&!/_(zero|one)$/.test(k))console.log(l,k,JSON.stringify(v))}}'
+```
+
+(`_zero`/`_one` forms are skipped because they may legitimately drop `{{count}}`.)
+
+Commit with an `i18n:` prefix, e.g. `i18n: fill missing translations`.
diff --git a/.agents/skills/fill-translations/i18n.mjs b/.agents/skills/fill-translations/i18n.mjs
new file mode 100644
index 00000000..8ea48b2e
--- /dev/null
+++ b/.agents/skills/fill-translations/i18n.mjs
@@ -0,0 +1,346 @@
+#!/usr/bin/env node
+// Finds missing, outdated and obsolete translation keys in the web app's
+// locales (using `en` as the source of truth), and merges translations back.
+//
+// node .agents/skills/fill-translations/i18n.mjs report [--locales de,fr] [--json]
+// node .agents/skills/fill-translations/i18n.mjs apply <locale> <file.json> [--prune]
+//
+// "Outdated" means the English value of a key changed (in git history, or in
+// the working tree) after the last time the locale's value for that key
+// changed.
+
+import { execFileSync, spawnSync } from "node:child_process";
+import fs from "node:fs";
+import path from "node:path";
+
+const REPO = execFileSync("git", ["rev-parse", "--show-toplevel"], {
+ encoding: "utf8",
+}).trim();
+const LOCALES_REL = "apps/web/lib/i18n/locales";
+const LOCALES_DIR = path.join(REPO, LOCALES_REL);
+const SOURCE = "en";
+const PLURAL_SUFFIX = /_(zero|one|two|few|many|other)$/;
+const PLURAL_ORDER = ["zero", "one", "two", "few", "many", "other"];
+
+const fileRel = (locale) => `${LOCALES_REL}/${locale}/translation.json`;
+const fileAbs = (locale) => path.join(REPO, fileRel(locale));
+
+function flatten(obj, prefix = "", out = {}) {
+ for (const [k, v] of Object.entries(obj)) {
+ const key = prefix ? `${prefix}.${k}` : k;
+ if (v && typeof v === "object") flatten(v, key, out);
+ else out[key] = v;
+ }
+ return out;
+}
+
+// Returns [{ ts, flat }] from oldest to newest, ending with the working tree.
+function history(locale) {
+ const rel = fileRel(locale);
+ const log = execFileSync(
+ "git",
+ ["log", "--reverse", "--format=%H %ct", "--", rel],
+ { cwd: REPO, encoding: "utf8" },
+ )
+ .trim()
+ .split("\n")
+ .filter(Boolean)
+ .map((l) => {
+ const [sha, ts] = l.split(" ");
+ return { sha, ts: Number(ts) };
+ });
+
+ const res = spawnSync("git", ["cat-file", "--batch"], {
+ cwd: REPO,
+ input: log.map((c) => `${c.sha}:${rel}\n`).join(""),
+ maxBuffer: 1024 * 1024 * 1024,
+ });
+ const buf = res.stdout;
+ const snapshots = [];
+ let pos = 0;
+ for (const c of log) {
+ const nl = buf.indexOf(10, pos);
+ const header = buf.subarray(pos, nl).toString();
+ pos = nl + 1;
+ if (header.endsWith("missing")) continue; // file deleted in this commit
+ const size = Number(header.split(" ")[2]);
+ const content = buf.subarray(pos, pos + size).toString();
+ pos += size + 1;
+ try {
+ snapshots.push({ ts: c.ts, flat: flatten(JSON.parse(content)) });
+ } catch {
+ // Unparseable historical version; skip it.
+ }
+ }
+ if (fs.existsSync(fileAbs(locale))) {
+ snapshots.push({
+ ts: Math.floor(Date.now() / 1000),
+ flat: flatten(JSON.parse(fs.readFileSync(fileAbs(locale), "utf8"))),
+ });
+ }
+ return snapshots;
+}
+
+// For every key in the latest snapshot: its current value, and the full list
+// of (ts, value) changes.
+function keyTimeline(snapshots) {
+ const changes = {};
+ let prev = {};
+ for (const { ts, flat } of snapshots) {
+ for (const [k, v] of Object.entries(flat)) {
+ if (prev[k] !== v) (changes[k] ??= []).push({ ts, value: v });
+ }
+ prev = flat;
+ }
+ return { current: prev, changes };
+}
+
+const lastChange = (tl, key) => tl.changes[key]?.at(-1)?.ts ?? 0;
+const valueAt = (tl, key, ts) => {
+ let v;
+ for (const c of tl.changes[key] ?? []) if (c.ts <= ts) v = c.value;
+ return v;
+};
+
+function bcp47(locale) {
+ if (locale === "zhtw") return "zh-TW";
+ return locale.replace("_", "-");
+}
+
+function pluralCategories(locale) {
+ try {
+ return new Intl.PluralRules(bcp47(locale)).resolvedOptions()
+ .pluralCategories;
+ } catch {
+ return ["one", "other"];
+ }
+}
+
+function listLocales() {
+ return fs
+ .readdirSync(LOCALES_DIR)
+ .filter(
+ (d) =>
+ d !== SOURCE &&
+ fs.existsSync(path.join(LOCALES_DIR, d, "translation.json")),
+ )
+ .sort();
+}
+
+// The keys a locale is expected to have, mapped to the en key they mirror.
+// Plural groups (foo_one/foo_other in en) expand to the locale's own CLDR
+// plural categories.
+function expectedKeys(enFlat, locale) {
+ const expected = {};
+ const pluralBases = new Set();
+ for (const k of Object.keys(enFlat)) {
+ const m = k.match(PLURAL_SUFFIX);
+ if (m && `${k.slice(0, -m[0].length)}_other` in enFlat) {
+ pluralBases.add(k.slice(0, -m[0].length));
+ } else {
+ expected[k] = k;
+ }
+ }
+ const cats = pluralCategories(locale);
+ for (const base of pluralBases) {
+ for (const cat of cats) {
+ const k = `${base}_${cat}`;
+ expected[k] = k in enFlat ? k : `${base}_other`;
+ }
+ }
+ return { expected, pluralBases };
+}
+
+function analyze(locale, en) {
+ const tl = keyTimeline(history(locale));
+ const { expected, pluralBases } = expectedKeys(en.current, locale);
+ const missing = [];
+ const outdated = [];
+ const obsolete = [];
+
+ for (const [key, enKey] of Object.entries(expected)) {
+ const value = tl.current[key];
+ if (value === undefined || value === "") {
+ missing.push({ key, en: en.current[enKey] });
+ continue;
+ }
+ const enTs = lastChange(en, enKey);
+ const locTs = lastChange(tl, key);
+ if (enTs > locTs) {
+ outdated.push({
+ key,
+ en: en.current[enKey],
+ enWhenTranslated: valueAt(en, enKey, locTs) ?? null,
+ current: value,
+ });
+ }
+ }
+
+ for (const [key, value] of Object.entries(tl.current)) {
+ if (key in expected) continue;
+ const m = key.match(PLURAL_SUFFIX);
+ // Extra plural forms of a known plural group are harmless.
+ if (m && pluralBases.has(key.slice(0, -m[0].length))) continue;
+ obsolete.push({ key, value });
+ }
+
+ return { locale, missing, outdated, obsolete };
+}
+
+function report(args) {
+ const json = args.includes("--json");
+ const li = args.indexOf("--locales");
+ const locales =
+ li >= 0 ? args[li + 1].split(",").map((s) => s.trim()) : listLocales();
+
+ const en = keyTimeline(history(SOURCE));
+ const results = locales.map((l) => analyze(l, en));
+
+ if (json) {
+ console.log(JSON.stringify(results, null, 2));
+ return;
+ }
+
+ console.log("locale missing outdated obsolete");
+ for (const r of results) {
+ console.log(
+ `${r.locale.padEnd(8)} ${String(r.missing.length).padStart(7)} ${String(r.outdated.length).padStart(9)} ${String(r.obsolete.length).padStart(9)}`,
+ );
+ }
+ for (const r of results) {
+ if (!r.missing.length && !r.outdated.length && !r.obsolete.length) continue;
+ console.log(`\n## ${r.locale}`);
+ for (const m of r.missing)
+ console.log(`MISSING ${m.key}: ${JSON.stringify(m.en)}`);
+ for (const o of r.outdated) {
+ console.log(`OUTDATED ${o.key}`);
+ console.log(` en now: ${JSON.stringify(o.en)}`);
+ console.log(
+ ` en when translated: ${JSON.stringify(o.enWhenTranslated)}`,
+ );
+ console.log(` current: ${JSON.stringify(o.current)}`);
+ }
+ for (const o of r.obsolete)
+ console.log(`OBSOLETE ${o.key}: ${JSON.stringify(o.value)}`);
+ }
+}
+
+// Sets `value` at `parts` in `obj`, inserting new keys right after their
+// closest preceding sibling in the en file so diffs stay local.
+function setOrdered(obj, enObj, parts, value) {
+ const [head, ...rest] = parts;
+ const child = rest.length
+ ? obj[head] && typeof obj[head] === "object"
+ ? obj[head]
+ : {}
+ : value;
+ if (rest.length) setOrdered(child, enObj?.[head] ?? {}, rest, value);
+
+ if (head in obj) {
+ obj[head] = child;
+ return obj;
+ }
+ const enKeys = enObj ? Object.keys(enObj) : [];
+ const idx = enKeys.indexOf(head);
+ let after = null;
+ for (let i = idx - 1; i >= 0; i--) {
+ if (enKeys[i] in obj) {
+ after = enKeys[i];
+ break;
+ }
+ }
+ const entries = Object.entries(obj);
+ let pos =
+ after === null
+ ? idx < 0
+ ? entries.length
+ : 0
+ : entries.findIndex(([k]) => k === after) + 1;
+ // For plural forms that en doesn't have (e.g. foo_few), keep CLDR order
+ // (zero, one, two, few, many, other) among the sibling forms.
+ const m = idx < 0 && head.match(PLURAL_SUFFIX);
+ if (m) {
+ const base = head.slice(0, -m[0].length);
+ const order = PLURAL_ORDER.indexOf(m[1]);
+ const siblings = entries
+ .map(([k], i) => ({
+ i,
+ form: k.startsWith(`${base}_`)
+ ? PLURAL_ORDER.indexOf(k.slice(base.length + 1))
+ : -1,
+ }))
+ .filter((s) => s.form >= 0);
+ const next = siblings.find((s) => s.form > order);
+ if (next) pos = next.i;
+ else if (siblings.length) pos = siblings.at(-1).i + 1;
+ }
+ entries.splice(pos, 0, [head, child]);
+ for (const k of Object.keys(obj)) delete obj[k];
+ for (const [k, v] of entries) obj[k] = v;
+ return obj;
+}
+
+function deletePath(obj, parts) {
+ const [head, ...rest] = parts;
+ if (!(head in obj)) return;
+ if (!rest.length) {
+ delete obj[head];
+ return;
+ }
+ deletePath(obj[head], rest);
+ if (
+ obj[head] &&
+ typeof obj[head] === "object" &&
+ !Object.keys(obj[head]).length
+ )
+ delete obj[head];
+}
+
+function apply(args) {
+ const [locale, input] = args;
+ if (!locale || !input) {
+ console.error("usage: apply <locale> <translations.json> [--prune]");
+ process.exit(1);
+ }
+ const enObj = JSON.parse(fs.readFileSync(fileAbs(SOURCE), "utf8"));
+ const target = fs.existsSync(fileAbs(locale))
+ ? JSON.parse(fs.readFileSync(fileAbs(locale), "utf8"))
+ : {};
+ const translations = flatten(JSON.parse(fs.readFileSync(input, "utf8")));
+
+ for (const [key, value] of Object.entries(translations)) {
+ if (typeof value !== "string" || !value) {
+ console.error(`skipping ${key}: empty or non-string value`);
+ continue;
+ }
+ setOrdered(target, enObj, key.split("."), value);
+ }
+
+ let pruned = 0;
+ if (args.includes("--prune")) {
+ const { expected, pluralBases } = expectedKeys(flatten(enObj), locale);
+ for (const key of Object.keys(flatten(target))) {
+ if (key in expected) continue;
+ const m = key.match(PLURAL_SUFFIX);
+ if (m && pluralBases.has(key.slice(0, -m[0].length))) continue;
+ deletePath(target, key.split("."));
+ pruned++;
+ }
+ }
+
+ fs.mkdirSync(path.dirname(fileAbs(locale)), { recursive: true });
+ fs.writeFileSync(fileAbs(locale), JSON.stringify(target, null, 2) + "\n");
+ console.log(
+ `${locale}: wrote ${Object.keys(translations).length} keys${pruned ? `, pruned ${pruned} obsolete keys` : ""}`,
+ );
+}
+
+const [cmd, ...rest] = process.argv.slice(2);
+if (cmd === "report") report(rest);
+else if (cmd === "apply") apply(rest);
+else {
+ console.error(
+ "usage:\n i18n.mjs report [--locales de,fr] [--json]\n i18n.mjs apply <locale> <translations.json> [--prune]",
+ );
+ process.exit(1);
+}
diff --git a/README.md b/README.md
index 737b0563..23e5c8d0 100644
--- a/README.md
+++ b/README.md
@@ -8,9 +8,6 @@
<a href="https://discord.gg/NrgeYywsFh">
<img alt="Discord" src="https://img.shields.io/discord/1223681308962721802?label=chat%20on%20discord" />
</a>
- <a href="https://hosted.weblate.org/engage/hoarder/">
- <img src="https://hosted.weblate.org/widget/hoarder/hoarder/svg-badge.svg" alt="Translation status" />
- </a>
</div>
# <img height="50px" src="./screenshots/logo.png" />
@@ -98,7 +95,7 @@ I'm a systems engineer in my day job (and have been for the past 7 years). I did
## Translations
-Karakeep uses Weblate for managing translations. If you want to help translate Karakeep, you can do so [here](https://hosted.weblate.org/engage/hoarder/).
+Translations live in [`apps/web/lib/i18n/locales`](apps/web/lib/i18n/locales), one `translation.json` per language, with English (`en`) as the source. To fix or improve a translation, open a pull request that edits the relevant file.
## Karakeep Cloud ☁️