Skip to content

A self-hostable bookmark-everything app (links, notes and images) with AI-based automatic tagging and full text search

feat(dev): add `pnpm dev:worktree` to run the dev env per worktree

Mohamed Bassem2ef23431edca
diff --git a/.agents/skills/dev-env/SKILL.md b/.agents/skills/dev-env/SKILL.md
new file mode 100644
index 00000000..bc651732
--- /dev/null
+++ b/.agents/skills/dev-env/SKILL.md
@@ -0,0 +1,77 @@
+---
+name: dev-env
+description: Run the Karakeep dev environment (web + workers) inside a git worktree and log in as a seeded user. Use when asked to start/run the app, verify a change in the browser, take a screenshot of the web UI, or when the dev server fails to boot (port conflicts, login redirects to another worktree, stale migrations, EMFILE watcher loops, tsx EPERM).
+---
+
+# Running the Karakeep dev env in a worktree
+
+Many worktrees run on this machine at the same time, often each with its own dev server. `pnpm dev:worktree` (`scripts/dev-worktree.sh`) handles the resulting conflicts. Use it rather than `pnpm web` / `pnpm workers`.
+
+## 1. Start it
+
+Run from the worktree root, in the background (it doesn't return until stopped):
+
+```sh
+# run_in_background: true
+pnpm dev:worktree >| $TMPDIR/karakeep-$(basename $PWD)-dev.out 2>&1
+```
+
+Then wait for it to report readiness:
+
+```sh
+for i in $(seq 1 150); do grep -qE "is ready|exited|didn't become" $TMPDIR/karakeep-$(basename $PWD)-dev.out && break; sleep 2; done; tail -20 $TMPDIR/karakeep-$(basename $PWD)-dev.out
+```
+
+The script:
+
+1. Sets up only the pieces that are missing: runs `pnpm install`, writes `.env` (random `NEXTAUTH_SECRET`, absolute `DATA_DIR`), symlinks `.env` into `apps/web`, `apps/workers` and `packages/db`, and applies the seed snapshot into `data/`.
+2. Migrates the main DB and the queue DB. The seed snapshot is older than HEAD, so this always runs.
+3. Picks the first free port in 3100–3199 and sets `NEXTAUTH_URL` to match.
+4. Starts the workers, then `next dev` with `WATCHPACK_POLLING=true`, and waits until `/signin` returns 200.
+5. Prints the URL, the seed users, and the log paths (`$TMPDIR/karakeep-dev-<worktree>/{web,workers}.log`).
+
+Overrides: `PORT=3105`, `DEV_HOST=<host>` (see step 3), and `WATCHPACK_POLLING=false`. `pnpm dev:worktree setup` does steps 1–2 without starting anything.
+
+Web changes hot-reload. Worker changes don't: restart the script.
+
+## 2. Stop it
+
+Stop the background task (TaskStop). The script stops both services on exit. Don't touch processes from other worktrees; to check which worktree owns a process, run `lsof -a -p <pid> -d cwd`.
+
+## 3. Log in
+
+Seed users (all share the password `test1234`):
+
+| Email | Role | Data |
+| --- | --- | --- |
+| `[email protected]` | admin | 20 bookmarks, 3 tags, 2 manual + 2 smart lists |
+| `[email protected]` | user | 4 bookmarks, 3 tags, 2 manual + 2 smart lists |
+| `[email protected]` | user | empty |
+
+Login flow with any browser automation tool:
+
+1. Navigate to `http://localhost:$PORT/signin`.
+2. Type the email into `input[type='email']` and the password into `input[type='password']`, then press Enter.
+3. Wait for the URL to contain `/dashboard` and the page to show `Seed snapshot bookmark`.
+
+Browser gotchas:
+
+- **The browser must use the host in `NEXTAUTH_URL`.** If your browser tool reaches the app through something other than `localhost`, such as a LAN IP like `http://10.0.10.13:3100`, restart with `DEV_HOST=<that host>`. Otherwise sign-in goes to the wrong place, and Next blocks HMR and fonts for that origin. The same applies to `127.0.0.1`.
+- **Cookies are per host, not per port.** Worktrees share cookies. With the per-worktree random secret, another worktree's session cookie is rejected: you land on `/signin`, and the web log shows `JWT_SESSION_ERROR` until you log in. That's expected. If `.env` has a shared secret such as `test1234`, another worktree's session is valid on yours, so open `/logout` first.
+- The dashboard's accessibility tree is large. If your screenshot tool returns it with the image, save the snapshot to a file and read only the PNG.
+
+## 4. Not running in dev
+
+- **Meilisearch**: without `MEILI_ADDR`, search is disabled. Start it with `docker run -p 7700:7700 getmeili/meilisearch:v1.41.0` and set `MEILI_ADDR=http://127.0.0.1:7700` only if you need search. Port 7700 may already belong to another worktree's container.
+- **Crawling**: the workers run in "browserless mode" with no `BROWSER_WEB_URL`. The seed data is already crawled, so this only matters for newly added links.
+- **Inference**: no `OPENAI_API_KEY`, so no auto-tagging.
+
+## 5. Troubleshooting and running pieces by hand
+
+- **`listen EPERM ... tsx-*/*.pipe`**: the tsx CLI (`pnpm db:migrate`, `pnpm workers`, `pnpm seed:apply`) opens an IPC socket that the Claude Code sandbox blocks. Run the file through node's loader instead, e.g. `(cd packages/db && node --import tsx migrate.ts)`. The script already does this.
+- **`EMFILE: too many open files, watch`, then `.next/dev` "was deleted" and endless restarts**: native file watching is blocked in the sandbox. Keep `WATCHPACK_POLLING=true`.
+- **A stray `packages/db/db.db`**: migrations ran with an empty `DATA_DIR`. Scripts must `import "dotenv/config"` before `@karakeep/shared/config`, which parses `process.env` at import time. Delete the stray file and re-run.
+- **Jobs never run / queue errors**: `queue.db` needs `pnpm db:migrate:queue` (or `(cd apps/workers && node --import tsx scripts/migrateQueue.ts)`). The workers also migrate it on startup; the web app never does.
+- **Reset the data**: stop the script, then run `(cd tools/seed-snapshot && node --import tsx src/apply.ts --force)`.
+- **Never copy `.env` or `data/` from the main checkout**: the seed gives every worktree the same known state.
+- **`$TMPDIR` is shared by all agent sessions**: generic names like `web.log` collide with other worktrees, and zsh `noclobber` refuses to overwrite them. Put the worktree name in log file names and use `>|`.
diff --git a/.claude/skills b/.claude/skills
new file mode 120000
index 00000000..2b7a412b
--- /dev/null
+++ b/.claude/skills
@@ -0,0 +1 @@
+../.agents/skills
\ No newline at end of file
diff --git a/apps/workers/package.json b/apps/workers/package.json
index 7faed068..e9f3a81a 100644
--- a/apps/workers/package.json
+++ b/apps/workers/package.json
@@ -71,6 +71,7 @@
"scripts": {
"start": "tsx watch index.ts",
"start:prod": "tsx index.ts",
+ "migrate:queue": "tsx scripts/migrateQueue.ts",
"build": "tsdown",
"build:watch": "tsdown --watch",
"lint": "oxlint --ignore-path=../../.gitignore .",
diff --git a/apps/workers/scripts/migrateQueue.ts b/apps/workers/scripts/migrateQueue.ts
new file mode 100644
index 00000000..6e84cd67
--- /dev/null
+++ b/apps/workers/scripts/migrateQueue.ts
@@ -0,0 +1,17 @@
+// Must load before @karakeep/shared/config, which parses process.env at import time.
+import "dotenv/config";
+
+import { loadAllPlugins, prepareQueue } from "@karakeep/shared-server";
+
+/**
+ * Runs the queue provider's migrations (e.g. liteque's queue.db) without
+ * starting the workers. The workers also do this on startup.
+ */
+async function main() {
+ await loadAllPlugins();
+ await prepareQueue();
+ // Plugins may leave connections open; exit explicitly.
+ process.exit(0);
+}
+
+void main();
diff --git a/package.json b/package.json
index 1607d836..e7238217 100644
--- a/package.json
+++ b/package.json
@@ -10,12 +10,14 @@
"clean:workspaces": "turbo clean",
"db:generate": "pnpm --filter @karakeep/db run generate",
"db:migrate": "pnpm --filter @karakeep/db run migrate",
+ "db:migrate:queue": "pnpm --filter @karakeep/workers run migrate:queue",
"db:studio": "pnpm --filter @karakeep/db studio",
"seed:apply": "pnpm --filter @karakeep/seed-snapshot run apply",
"seed:snapshot": "pnpm --filter @karakeep/seed-snapshot run snapshot",
"workers": "pnpm --filter @karakeep/workers run start",
"web": "pnpm --filter @karakeep/web run dev",
"web:portless": "pnpm --filter @karakeep/web run dev:portless",
+ "dev:worktree": "bash scripts/dev-worktree.sh",
"android": "pnpm --filter @karakeep/mobile android",
"ios": "pnpm --filter @karakeep/mobile ios",
"prepare": "husky",
diff --git a/scripts/dev-worktree.sh b/scripts/dev-worktree.sh
new file mode 100644
index 00000000..c07780db
--- /dev/null
+++ b/scripts/dev-worktree.sh
@@ -0,0 +1,192 @@
+#!/usr/bin/env bash
+
+# Sets up and runs the dev env (web + workers) for a single git worktree, without
+# clashing with dev envs running from other worktrees on the same machine.
+#
+# Usage:
+# pnpm dev:worktree # set up, migrate, then run web + workers until stopped
+# pnpm dev:worktree setup # set up and migrate only
+#
+# Env overrides:
+# PORT web port (default: first free port in 3100-3199)
+# DEV_HOST hostname the browser uses to reach the app (default: localhost).
+# Set it when the browser can't use localhost (e.g. a LAN IP).
+# WATCHPACK_POLLING poll for file changes instead of native watching (default: true,
+# because native watching fails inside the Claude Code sandbox)
+#
+# Scripts run through `node --import tsx` rather than the tsx CLI, because the tsx CLI
+# opens an IPC socket that sandboxes can block.
+
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT"
+
+WORKTREE_ID="$(basename "$ROOT")"
+LOG_DIR="${TMPDIR:-/tmp}/karakeep-dev-$WORKTREE_ID"
+
+log() {
+ printf '==> %s\n' "$*"
+}
+
+setup() {
+ if [ ! -d node_modules ]; then
+ log "Installing dependencies"
+ pnpm install
+ fi
+
+ # A random secret per worktree means session cookies left behind by other
+ # worktrees (cookies are shared across localhost ports) are rejected instead
+ # of silently logging you in.
+ if ! grep -qs "^NEXTAUTH_SECRET=" .env; then
+ log "Adding a random NEXTAUTH_SECRET to .env"
+ echo "NEXTAUTH_SECRET=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')" >>.env
+ fi
+ if ! grep -qs "^DATA_DIR=" .env; then
+ log "Adding DATA_DIR to .env"
+ echo "DATA_DIR=$ROOT/data" >>.env
+ fi
+ for dir in apps/web apps/workers packages/db; do
+ if [ ! -e "$dir/.env" ]; then
+ ln -s "$ROOT/.env" "$dir/.env"
+ fi
+ done
+
+ if [ ! -d data ]; then
+ log "Applying the seed snapshot"
+ (cd tools/seed-snapshot && node --import tsx src/apply.ts)
+ fi
+
+ log "Migrating the database"
+ (cd packages/db && node --import tsx migrate.ts)
+ log "Migrating the queue"
+ (cd apps/workers && node --import tsx scripts/migrateQueue.ts)
+}
+
+port_in_use() {
+ lsof -nP -iTCP:"$1" -sTCP:LISTEN >/dev/null 2>&1
+}
+
+pick_port() {
+ local port
+ for port in $(seq 3100 3199); do
+ if ! port_in_use "$port"; then
+ echo "$port"
+ return
+ fi
+ done
+}
+
+WEB_PID=""
+WORKERS_PID=""
+
+cleanup() {
+ trap - EXIT INT TERM
+ log "Stopping web and workers"
+ # Each service runs in its own process group (set -m), so this also stops
+ # the processes they spawn (pnpm -> next -> next-server).
+ if [ -n "$WEB_PID" ]; then kill -- "-$WEB_PID" 2>/dev/null || true; fi
+ if [ -n "$WORKERS_PID" ]; then kill -- "-$WORKERS_PID" 2>/dev/null || true; fi
+ wait 2>/dev/null || true
+}
+
+# Prints the tail of the log of whichever process died and exits.
+fail_if_dead() {
+ local name pid
+ for name in web workers; do
+ if [ "$name" = web ]; then pid=$WEB_PID; else pid=$WORKERS_PID; fi
+ if ! kill -0 "$pid" 2>/dev/null; then
+ log "The $name process exited. Last lines of $LOG_DIR/$name.log:"
+ tail -n 30 "$LOG_DIR/$name.log"
+ exit 1
+ fi
+ done
+}
+
+start() {
+ if [ -z "${PORT:-}" ]; then
+ PORT="$(pick_port)"
+ if [ -z "$PORT" ]; then
+ log "No free port in 3100-3199"
+ exit 1
+ fi
+ elif port_in_use "$PORT"; then
+ log "Port $PORT is already in use"
+ exit 1
+ fi
+
+ # NEXTAUTH_URL must match the URL the browser uses, or sign-in and sign-out
+ # requests go to whatever is running on the default (localhost:3000).
+ local host="${DEV_HOST:-localhost}"
+ export PORT
+ export NEXTAUTH_URL="http://$host:$PORT"
+ export WATCHPACK_POLLING="${WATCHPACK_POLLING:-true}"
+ if [ "$host" != localhost ]; then
+ # Next blocks dev resources (HMR, fonts) for origins other than localhost.
+ export ALLOWED_DEV_ORIGINS="${ALLOWED_DEV_ORIGINS:+$ALLOWED_DEV_ORIGINS,}$host"
+ fi
+
+ mkdir -p "$LOG_DIR"
+ trap cleanup EXIT
+ trap 'exit 130' INT
+ trap 'exit 143' TERM
+ set -m
+
+ log "Starting workers (log: $LOG_DIR/workers.log)"
+ (cd apps/workers && exec node --import tsx index.ts) >"$LOG_DIR/workers.log" 2>&1 &
+ WORKERS_PID=$!
+
+ log "Starting web on port $PORT (log: $LOG_DIR/web.log)"
+ (cd apps/web && exec pnpm exec next dev) >"$LOG_DIR/web.log" 2>&1 &
+ WEB_PID=$!
+
+ log "Waiting for the web app (the first compile can take a minute)"
+ local deadline=$((SECONDS + 300)) code
+ while :; do
+ fail_if_dead
+ code="$(curl -s -o /dev/null -m 60 -w '%{http_code}' "http://localhost:$PORT/signin" || true)"
+ if [ "$code" = 200 ]; then
+ break
+ fi
+ if [ "$SECONDS" -ge "$deadline" ]; then
+ log "The web app didn't become ready within 5 minutes (last status: $code)"
+ exit 1
+ fi
+ sleep 2
+ done
+
+ cat <<EOF
+
+Karakeep is ready at $NEXTAUTH_URL
+
+Seed users (password: test1234):
+ [email protected] admin, 20 bookmarks
+ [email protected] user, 4 bookmarks
+ [email protected] user, no data
+
+Logs:
+ $LOG_DIR/web.log
+ $LOG_DIR/workers.log
+
+Press Ctrl+C to stop.
+EOF
+
+ while :; do
+ fail_if_dead
+ sleep 2
+ done
+}
+
+case "${1:-start}" in
+ setup)
+ setup
+ ;;
+ start)
+ setup
+ start
+ ;;
+ *)
+ echo "Usage: $0 [setup|start]" >&2
+ exit 1
+ ;;
+esac