A self-hostable bookmark-everything app (links, notes and images) with AI-based automatic tagging and full text search
feat(dev): add `pnpm dev:worktree` to run the dev env per worktree
Mohamed Bassem2ef23431edca
diff --git a/.agents/skills/dev-env/SKILL.md b/.agents/skills/dev-env/SKILL.mdnew file mode 100644index 00000000..bc651732--- /dev/null+++ b/.agents/skills/dev-env/SKILL.md@@ -0,0 +1,77 @@+---+name: dev-env+description: Run the Karakeep dev environment (web + workers) inside a git worktree and log in as a seeded user. Use when asked to start/run the app, verify a change in the browser, take a screenshot of the web UI, or when the dev server fails to boot (port conflicts, login redirects to another worktree, stale migrations, EMFILE watcher loops, tsx EPERM).+---++# Running the Karakeep dev env in a worktree++Many worktrees run on this machine at the same time, often each with its own dev server. `pnpm dev:worktree` (`scripts/dev-worktree.sh`) handles the resulting conflicts. Use it rather than `pnpm web` / `pnpm workers`.++## 1. Start it++Run from the worktree root, in the background (it doesn't return until stopped):++```sh+# run_in_background: true+pnpm dev:worktree >| $TMPDIR/karakeep-$(basename $PWD)-dev.out 2>&1+```++Then wait for it to report readiness:++```sh+for i in $(seq 1 150); do grep -qE "is ready|exited|didn't become" $TMPDIR/karakeep-$(basename $PWD)-dev.out && break; sleep 2; done; tail -20 $TMPDIR/karakeep-$(basename $PWD)-dev.out+```++The script:++1. Sets up only the pieces that are missing: runs `pnpm install`, writes `.env` (random `NEXTAUTH_SECRET`, absolute `DATA_DIR`), symlinks `.env` into `apps/web`, `apps/workers` and `packages/db`, and applies the seed snapshot into `data/`.+2. Migrates the main DB and the queue DB. The seed snapshot is older than HEAD, so this always runs.+3. Picks the first free port in 3100–3199 and sets `NEXTAUTH_URL` to match.+4. Starts the workers, then `next dev` with `WATCHPACK_POLLING=true`, and waits until `/signin` returns 200.+5. Prints the URL, the seed users, and the log paths (`$TMPDIR/karakeep-dev-<worktree>/{web,workers}.log`).++Overrides: `PORT=3105`, `DEV_HOST=<host>` (see step 3), and `WATCHPACK_POLLING=false`. `pnpm dev:worktree setup` does steps 1–2 without starting anything.++Web changes hot-reload. Worker changes don't: restart the script.++## 2. Stop it++Stop the background task (TaskStop). The script stops both services on exit. Don't touch processes from other worktrees; to check which worktree owns a process, run `lsof -a -p <pid> -d cwd`.++## 3. Log in++Seed users (all share the password `test1234`):++| Email | Role | Data |+| --- | --- | --- |+| `[email protected]` | admin | 20 bookmarks, 3 tags, 2 manual + 2 smart lists |+| `[email protected]` | user | 4 bookmarks, 3 tags, 2 manual + 2 smart lists |+| `[email protected]` | user | empty |++Login flow with any browser automation tool:++1. Navigate to `http://localhost:$PORT/signin`.+2. Type the email into `input[type='email']` and the password into `input[type='password']`, then press Enter.+3. Wait for the URL to contain `/dashboard` and the page to show `Seed snapshot bookmark`.++Browser gotchas:++- **The browser must use the host in `NEXTAUTH_URL`.** If your browser tool reaches the app through something other than `localhost`, such as a LAN IP like `http://10.0.10.13:3100`, restart with `DEV_HOST=<that host>`. Otherwise sign-in goes to the wrong place, and Next blocks HMR and fonts for that origin. The same applies to `127.0.0.1`.+- **Cookies are per host, not per port.** Worktrees share cookies. With the per-worktree random secret, another worktree's session cookie is rejected: you land on `/signin`, and the web log shows `JWT_SESSION_ERROR` until you log in. That's expected. If `.env` has a shared secret such as `test1234`, another worktree's session is valid on yours, so open `/logout` first.+- The dashboard's accessibility tree is large. If your screenshot tool returns it with the image, save the snapshot to a file and read only the PNG.++## 4. Not running in dev++- **Meilisearch**: without `MEILI_ADDR`, search is disabled. Start it with `docker run -p 7700:7700 getmeili/meilisearch:v1.41.0` and set `MEILI_ADDR=http://127.0.0.1:7700` only if you need search. Port 7700 may already belong to another worktree's container.+- **Crawling**: the workers run in "browserless mode" with no `BROWSER_WEB_URL`. The seed data is already crawled, so this only matters for newly added links.+- **Inference**: no `OPENAI_API_KEY`, so no auto-tagging.++## 5. Troubleshooting and running pieces by hand++- **`listen EPERM ... tsx-*/*.pipe`**: the tsx CLI (`pnpm db:migrate`, `pnpm workers`, `pnpm seed:apply`) opens an IPC socket that the Claude Code sandbox blocks. Run the file through node's loader instead, e.g. `(cd packages/db && node --import tsx migrate.ts)`. The script already does this.+- **`EMFILE: too many open files, watch`, then `.next/dev` "was deleted" and endless restarts**: native file watching is blocked in the sandbox. Keep `WATCHPACK_POLLING=true`.+- **A stray `packages/db/db.db`**: migrations ran with an empty `DATA_DIR`. Scripts must `import "dotenv/config"` before `@karakeep/shared/config`, which parses `process.env` at import time. Delete the stray file and re-run.+- **Jobs never run / queue errors**: `queue.db` needs `pnpm db:migrate:queue` (or `(cd apps/workers && node --import tsx scripts/migrateQueue.ts)`). The workers also migrate it on startup; the web app never does.+- **Reset the data**: stop the script, then run `(cd tools/seed-snapshot && node --import tsx src/apply.ts --force)`.+- **Never copy `.env` or `data/` from the main checkout**: the seed gives every worktree the same known state.+- **`$TMPDIR` is shared by all agent sessions**: generic names like `web.log` collide with other worktrees, and zsh `noclobber` refuses to overwrite them. Put the worktree name in log file names and use `>|`.diff --git a/.claude/skills b/.claude/skillsnew file mode 120000index 00000000..2b7a412b--- /dev/null+++ b/.claude/skills@@ -0,0 +1 @@+../.agents/skills\ No newline at end of filediff --git a/apps/workers/package.json b/apps/workers/package.jsonindex 7faed068..e9f3a81a 100644--- a/apps/workers/package.json+++ b/apps/workers/package.json@@ -71,6 +71,7 @@"scripts": {"start": "tsx watch index.ts","start:prod": "tsx index.ts",+ "migrate:queue": "tsx scripts/migrateQueue.ts","build": "tsdown","build:watch": "tsdown --watch","lint": "oxlint --ignore-path=../../.gitignore .",diff --git a/apps/workers/scripts/migrateQueue.ts b/apps/workers/scripts/migrateQueue.tsnew file mode 100644index 00000000..6e84cd67--- /dev/null+++ b/apps/workers/scripts/migrateQueue.ts@@ -0,0 +1,17 @@+// Must load before @karakeep/shared/config, which parses process.env at import time.+import "dotenv/config";++import { loadAllPlugins, prepareQueue } from "@karakeep/shared-server";++/**+ * Runs the queue provider's migrations (e.g. liteque's queue.db) without+ * starting the workers. The workers also do this on startup.+ */+async function main() {+ await loadAllPlugins();+ await prepareQueue();+ // Plugins may leave connections open; exit explicitly.+ process.exit(0);+}++void main();diff --git a/package.json b/package.jsonindex 1607d836..e7238217 100644--- a/package.json+++ b/package.json@@ -10,12 +10,14 @@"clean:workspaces": "turbo clean","db:generate": "pnpm --filter @karakeep/db run generate","db:migrate": "pnpm --filter @karakeep/db run migrate",+ "db:migrate:queue": "pnpm --filter @karakeep/workers run migrate:queue","db:studio": "pnpm --filter @karakeep/db studio","seed:apply": "pnpm --filter @karakeep/seed-snapshot run apply","seed:snapshot": "pnpm --filter @karakeep/seed-snapshot run snapshot","workers": "pnpm --filter @karakeep/workers run start","web": "pnpm --filter @karakeep/web run dev","web:portless": "pnpm --filter @karakeep/web run dev:portless",+ "dev:worktree": "bash scripts/dev-worktree.sh","android": "pnpm --filter @karakeep/mobile android","ios": "pnpm --filter @karakeep/mobile ios","prepare": "husky",diff --git a/scripts/dev-worktree.sh b/scripts/dev-worktree.shnew file mode 100644index 00000000..c07780db--- /dev/null+++ b/scripts/dev-worktree.sh@@ -0,0 +1,192 @@+#!/usr/bin/env bash++# Sets up and runs the dev env (web + workers) for a single git worktree, without+# clashing with dev envs running from other worktrees on the same machine.+#+# Usage:+# pnpm dev:worktree # set up, migrate, then run web + workers until stopped+# pnpm dev:worktree setup # set up and migrate only+#+# Env overrides:+# PORT web port (default: first free port in 3100-3199)+# DEV_HOST hostname the browser uses to reach the app (default: localhost).+# Set it when the browser can't use localhost (e.g. a LAN IP).+# WATCHPACK_POLLING poll for file changes instead of native watching (default: true,+# because native watching fails inside the Claude Code sandbox)+#+# Scripts run through `node --import tsx` rather than the tsx CLI, because the tsx CLI+# opens an IPC socket that sandboxes can block.++set -euo pipefail++ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+cd "$ROOT"++WORKTREE_ID="$(basename "$ROOT")"+LOG_DIR="${TMPDIR:-/tmp}/karakeep-dev-$WORKTREE_ID"++log() {+ printf '==> %s\n' "$*"+}++setup() {+ if [ ! -d node_modules ]; then+ log "Installing dependencies"+ pnpm install+ fi++ # A random secret per worktree means session cookies left behind by other+ # worktrees (cookies are shared across localhost ports) are rejected instead+ # of silently logging you in.+ if ! grep -qs "^NEXTAUTH_SECRET=" .env; then+ log "Adding a random NEXTAUTH_SECRET to .env"+ echo "NEXTAUTH_SECRET=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')" >>.env+ fi+ if ! grep -qs "^DATA_DIR=" .env; then+ log "Adding DATA_DIR to .env"+ echo "DATA_DIR=$ROOT/data" >>.env+ fi+ for dir in apps/web apps/workers packages/db; do+ if [ ! -e "$dir/.env" ]; then+ ln -s "$ROOT/.env" "$dir/.env"+ fi+ done++ if [ ! -d data ]; then+ log "Applying the seed snapshot"+ (cd tools/seed-snapshot && node --import tsx src/apply.ts)+ fi++ log "Migrating the database"+ (cd packages/db && node --import tsx migrate.ts)+ log "Migrating the queue"+ (cd apps/workers && node --import tsx scripts/migrateQueue.ts)+}++port_in_use() {+ lsof -nP -iTCP:"$1" -sTCP:LISTEN >/dev/null 2>&1+}++pick_port() {+ local port+ for port in $(seq 3100 3199); do+ if ! port_in_use "$port"; then+ echo "$port"+ return+ fi+ done+}++WEB_PID=""+WORKERS_PID=""++cleanup() {+ trap - EXIT INT TERM+ log "Stopping web and workers"+ # Each service runs in its own process group (set -m), so this also stops+ # the processes they spawn (pnpm -> next -> next-server).+ if [ -n "$WEB_PID" ]; then kill -- "-$WEB_PID" 2>/dev/null || true; fi+ if [ -n "$WORKERS_PID" ]; then kill -- "-$WORKERS_PID" 2>/dev/null || true; fi+ wait 2>/dev/null || true+}++# Prints the tail of the log of whichever process died and exits.+fail_if_dead() {+ local name pid+ for name in web workers; do+ if [ "$name" = web ]; then pid=$WEB_PID; else pid=$WORKERS_PID; fi+ if ! kill -0 "$pid" 2>/dev/null; then+ log "The $name process exited. Last lines of $LOG_DIR/$name.log:"+ tail -n 30 "$LOG_DIR/$name.log"+ exit 1+ fi+ done+}++start() {+ if [ -z "${PORT:-}" ]; then+ PORT="$(pick_port)"+ if [ -z "$PORT" ]; then+ log "No free port in 3100-3199"+ exit 1+ fi+ elif port_in_use "$PORT"; then+ log "Port $PORT is already in use"+ exit 1+ fi++ # NEXTAUTH_URL must match the URL the browser uses, or sign-in and sign-out+ # requests go to whatever is running on the default (localhost:3000).+ local host="${DEV_HOST:-localhost}"+ export PORT+ export NEXTAUTH_URL="http://$host:$PORT"+ export WATCHPACK_POLLING="${WATCHPACK_POLLING:-true}"+ if [ "$host" != localhost ]; then+ # Next blocks dev resources (HMR, fonts) for origins other than localhost.+ export ALLOWED_DEV_ORIGINS="${ALLOWED_DEV_ORIGINS:+$ALLOWED_DEV_ORIGINS,}$host"+ fi++ mkdir -p "$LOG_DIR"+ trap cleanup EXIT+ trap 'exit 130' INT+ trap 'exit 143' TERM+ set -m++ log "Starting workers (log: $LOG_DIR/workers.log)"+ (cd apps/workers && exec node --import tsx index.ts) >"$LOG_DIR/workers.log" 2>&1 &+ WORKERS_PID=$!++ log "Starting web on port $PORT (log: $LOG_DIR/web.log)"+ (cd apps/web && exec pnpm exec next dev) >"$LOG_DIR/web.log" 2>&1 &+ WEB_PID=$!++ log "Waiting for the web app (the first compile can take a minute)"+ local deadline=$((SECONDS + 300)) code+ while :; do+ fail_if_dead+ code="$(curl -s -o /dev/null -m 60 -w '%{http_code}' "http://localhost:$PORT/signin" || true)"+ if [ "$code" = 200 ]; then+ break+ fi+ if [ "$SECONDS" -ge "$deadline" ]; then+ log "The web app didn't become ready within 5 minutes (last status: $code)"+ exit 1+ fi+ sleep 2+ done++ cat <<EOF++Karakeep is ready at $NEXTAUTH_URL++Seed users (password: test1234):+ [email protected] admin, 20 bookmarks+ [email protected] user, 4 bookmarks+ [email protected] user, no data++Logs:+ $LOG_DIR/web.log+ $LOG_DIR/workers.log++Press Ctrl+C to stop.+EOF++ while :; do+ fail_if_dead+ sleep 2+ done+}++case "${1:-start}" in+ setup)+ setup+ ;;+ start)+ setup+ start+ ;;+ *)+ echo "Usage: $0 [setup|start]" >&2+ exit 1+ ;;+esac