Skip to content

Self-hosted workflows for coding-agent delivery. Orchestrate CodeBuddy, Codex and Claude Code with scripts and human approval gates. Includes Local workflows for coding assistants.

fix(install): reapply schema extensions when upgrading (#225)

fx-kurumi47261cc67a6a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2e7094f3..f79c170d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,8 @@ Platform releases use `vX.Y.Z` tags and publish `ghcr.io/tencent/loopforge`. Loo
- A DeepSeek Harness reply longer than 8 KB (about 2,700 Chinese characters) is shown in full. `dsh --json` cuts every event string at 8 KiB and only its `final` event carries the whole answer, so the chat stopped mid-sentence while the saved reply was complete.
- SOP sandboxes of the published image install OpenCode, Kimi Code and Pi on first use, as they already did for Claude Code. An SOP role on one of these CLIs used to fail with `exec: <cli>: not found` (exit 127) even after a conversation installed it.
- Run logs and telemetry redact every environment variable ending in `_API_KEY`, `_TOKEN`, `_SECRET` or `_PASSWORD`. Agent CLI keys such as `DEEPSEEK_API_KEY`, and `AGENTFLOW_CLI_API_KEY` used with a custom Base URL, used to appear in clear text in the `docker exec` line of an SOP run log, readable by project members. Rotate a key that ran an SOP role before this release.
+- Upgrading with `./start.sh pull` now re-applies the schema pieces gorm AutoMigrate cannot create: the `run_telemetry.adapter` and `run_logs.level` generated columns from `migrations/001_run_obs_generated_columns.sql`, and the `INSERT IGNORE` starter catalog. A data volume from an older release no longer fails with `Error 1054 Unknown column 'adapter'` the first time someone opens a run detail. The installer and the upgrade command share one script, `scripts/install-schema-extensions.sh`, and a failed step stops the command with the migration file an operator has to run by hand.
+- `docs/deployment.md` splits `migrations/` into scripts the installer already covers, scripts an upgrade must run once, and scripts that only backfill existing rows.
- Repository clone URLs no longer carry credentials. Saving a `cloneUrl` with a user name or password, such as `https://user:[email protected]/acme/app.git`, is rejected with a pointer to the Git token in personal settings; the repositories API, bootstrap payload, run, task and requirement views, and the runner task management list strip credentials from `cloneUrl` and `webUrl` of rows saved earlier; a `webUrl` with credentials is rejected on save; runner clone errors and redacted Git output mask them. `ssh://git@host/...` and `git@host:path` remotes are still accepted.
diff --git a/docs/deployment.md b/docs/deployment.md
index a133f19c..678659cd 100644
--- a/docs/deployment.md
+++ b/docs/deployment.md
@@ -126,10 +126,20 @@ Configure provider credentials in the platform UI, not the host's global CLI con
## Observability schema
-After the base tables have been created, explicitly run `bash scripts/install-observability-schema.sh` for the Compose database. It installs absent `run_telemetry.adapter` and `run_logs.level` columns and their indexes from upstream migration `001_run_obs_generated_columns.sql`. The local development installer calls this step before installing templates. It can be repeated without reapplying existing columns. Application startup does not run data migrations.
+After the base tables have been created, run `bash scripts/install-schema-extensions.sh` for the Compose database. It installs absent `run_telemetry.adapter` and `run_logs.level` columns and their indexes from upstream migration `001_run_obs_generated_columns.sql`, then replays the `INSERT IGNORE` starter catalog. Both steps can be repeated without touching existing rows. The installer runs it before printing the address, and `./start.sh pull` runs it again after recreating containers, so an older data volume upgraded to a newer image picks up the columns the new code reads. Application startup does not run data migrations.
For an external MySQL deployment, review that migration and check column existence before applying its corresponding statements. Stored generated columns can rebuild large existing tables; schedule the operation accordingly.
+### Which migrations an upgrade needs
+
+`migrations/` holds three kinds of script. Check the release notes for the files a given version adds, then read the script's own header before running it.
+
+| Kind | Example | What an operator does |
+| --- | --- | --- |
+| Already applied by the installer and `./start.sh pull` | `001_run_obs_generated_columns.sql`, `migrations/seed/*.sql` | Nothing; the script above covers it |
+| Must run once when upgrading | `009_bot_interactions_utf8mb4.sql` (`CONVERT TO` an existing table), new tables and columns added by hand | Run it on the database after reviewing prerequisites |
+| Backfills and one-time rewrites of existing rows | `016_public_optional_repositories.sql`, `021_assistant_session_sources.sql` | Run it if the history matters; skipping it changes no schema but leaves old rows unconverted |
+
## Updating existing public SOPs
Repository binding is optional for all built-in generic SOPs. To update the original catalog instructions on an existing installation, explicitly apply:
diff --git a/scripts/install-schema-extensions.sh b/scripts/install-schema-extensions.sh
new file mode 100755
index 00000000..7ae072e1
--- /dev/null
+++ b/scripts/install-schema-extensions.sh
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+# Bring the Compose database up to what this release reads but gorm
+# AutoMigrate cannot create: the generated columns from migration 001 and
+# the INSERT IGNORE starter catalog. Both steps are safe to repeat, so the
+# installer and `./start.sh pull` run it after the platform is healthy.
+set -euo pipefail
+cd "$(dirname "$0")/.."
+source scripts/docker-access.sh
+
+if ! bash scripts/install-observability-schema.sh; then
+ printf '\n✗ Could not add the generated columns run details need.\n' >&2
+ printf ' Run migrations/001_run_obs_generated_columns.sql against the LoopForge database, then retry.\n' >&2
+ exit 1
+fi
+cat migrations/seed/*.sql | docker_access compose exec -T mysql sh -c 'MYSQL_PWD="$MYSQL_PASSWORD" mysql -u loopforge loopforge'
diff --git a/scripts/install.sh b/scripts/install.sh
index f754432c..c970ad09 100755
--- a/scripts/install.sh
+++ b/scripts/install.sh
@@ -192,8 +192,7 @@ else
docker_access compose --profile platform up -d --no-build --wait --wait-timeout 180
fi
step 'Installing the schema extensions and the starter catalog'
-bash scripts/install-observability-schema.sh
-cat migrations/seed/*.sql | docker_access compose exec -T mysql sh -c 'MYSQL_PWD="$MYSQL_PASSWORD" mysql -u loopforge loopforge'
+bash scripts/install-schema-extensions.sh
python3 scripts/install-address.py "$port"
users=$(docker_access compose exec -T mysql sh -c 'MYSQL_PWD="$MYSQL_PASSWORD" mysql -N -u loopforge loopforge -e "select count(*) from users"' 2>/dev/null || echo 0)
diff --git a/scripts/test-install.py b/scripts/test-install.py
index 131fddf5..0fae7ccd 100644
--- a/scripts/test-install.py
+++ b/scripts/test-install.py
@@ -32,10 +32,11 @@ class InstallerTests(unittest.TestCase):
self.root = Path(self.temp.name)
(self.root / 'scripts').mkdir()
(self.root / 'migrations/seed').mkdir(parents=True)
- for name in ('install.sh', 'configure.sh', 'docker-access.sh', 'ghcr-mirror.sh', 'install-address.py', 'install-compose.py', 'install-observability-schema.sh'):
+ for name in ('install.sh', 'configure.sh', 'docker-access.sh', 'ghcr-mirror.sh', 'install-address.py', 'install-compose.py', 'install-observability-schema.sh', 'install-schema-extensions.sh'):
shutil.copyfile(ROOT / 'scripts' / name, self.root / 'scripts' / name)
shutil.copyfile(ROOT / 'start.sh', self.root / 'start.sh')
- (self.root / 'migrations/seed/001_test.sql').write_text('SELECT 1;\n')
+ (self.root / 'migrations/001_run_obs_generated_columns.sql').write_text((ROOT / 'migrations/001_run_obs_generated_columns.sql').read_text())
+ (self.root / 'migrations/seed/001_test.sql').write_text('INSERT IGNORE INTO agents (id) VALUES (1);\n')
(self.root / 'mock').mkdir()
self.executable('docker', '''#!/usr/bin/env bash
printf '%s %s\\n' "${MOCK_PRIVILEGED:-0}" "$*" >> "$MOCK_LOG"
@@ -52,8 +53,8 @@ if [[ "${MOCK_EXPIRE_AFTER_BUILD:-0}" == 1 && "$*" == *'build platform'* ]]; the
if [[ "$*" == *'exec -T mysql'* ]]; then
if [[ "$*" == *'count(*) from users'* ]]; then echo 0; exit 0; fi
query=$(cat)
- if [[ "$query" == *'information_schema.columns'* ]]; then echo 1; fi
- if [[ "$*" == *'count(*) from users'* ]]; then echo 0; fi
+ printf '%s\n' "$query" >> "$MOCK_SQL_LOG"
+ if [[ "$query" == *'information_schema.columns'* ]]; then printf '%s\n' "${MOCK_COLUMN_COUNT:-1}"; fi
fi
''')
self.executable('id', '#!/usr/bin/env bash\nif [[ "$1" == -u ]]; then echo "${MOCK_USER:-0}"; else /usr/bin/id "$@"; fi\n')
@@ -64,9 +65,10 @@ fi
export MOCK_PRIVILEGED=1
exec "$@"
''')
- self.environ = dict(os.environ, PATH=str(self.root / 'mock') + ':' + os.environ['PATH'], MOCK_LOG=str(self.root / 'docker.log'), AGENTFLOW_HTTP_PORT='18765', LOOPFORGE_PUBLIC_URL='https://team.example.test', LOOPFORGE_BIND='0.0.0.0')
+ self.environ = dict(os.environ, PATH=str(self.root / 'mock') + ':' + os.environ['PATH'], MOCK_LOG=str(self.root / 'docker.log'), MOCK_SQL_LOG=str(self.root / 'mysql.log'), AGENTFLOW_HTTP_PORT='18765', LOOPFORGE_PUBLIC_URL='https://team.example.test', LOOPFORGE_BIND='0.0.0.0')
for name in ('LOOPFORGE_DOCKER_USE_SUDO', 'LOOPFORGE_DATA_DIR', 'LOOPFORGE_IMAGE', 'COMPOSE_FILE', 'LOOPFORGE_BUILD_FROM_SOURCE', 'LOOPFORGE_PULL_POLICY', 'MIRROR', 'MOCK_USER', 'MOCK_DIRECT', 'MOCK_SUDO_DENIED', 'MOCK_IMAGE_PRESENT', 'MOCK_PULL_FAIL', 'MOCK_TAG_FAIL'):
self.environ.pop(name, None)
+ self.environ.pop('MOCK_COLUMN_COUNT', None)
def executable(self, name, contents):
target = self.root / 'mock' / name
@@ -189,6 +191,28 @@ exec "$@"
self.assertIn('pull --ignore-buildable --policy always', log)
self.assertNotIn('ghcr.nju.edu.cn', log)
+ def test_start_pull_reapplies_schema_extensions_after_containers_are_up(self):
+ self.assert_success(self.install())
+ (self.root / 'docker.log').unlink()
+ result = subprocess.run(['bash', 'start.sh', 'pull'], cwd=self.root, env=dict(self.environ, MOCK_COLUMN_COUNT='0'), text=True, capture_output=True, stdin=subprocess.DEVNULL, timeout=15)
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+ log = (self.root / 'docker.log').read_text()
+ sql = (self.root / 'mysql.log').read_text()
+ self.assertIn('Installed: run_telemetry.adapter', result.stdout)
+ self.assertIn('Installed: run_logs.level', result.stdout)
+ self.assertIn('ALTER TABLE run_telemetry', sql)
+ self.assertIn('ALTER TABLE run_logs', sql)
+ self.assertIn('INSERT IGNORE', sql)
+ # Recreated containers come first; the old volume then gets the columns.
+ self.assertLess(log.index('up -d --no-build --wait'), log.index('exec -T mysql'))
+
+ def test_start_pull_names_the_migration_when_columns_cannot_be_checked(self):
+ self.assert_success(self.install())
+ result = subprocess.run(['bash', 'start.sh', 'pull'], cwd=self.root, env=dict(self.environ, MOCK_COLUMN_COUNT='?'), text=True, capture_output=True, stdin=subprocess.DEVNULL, timeout=15)
+ self.assertNotEqual(result.returncode, 0)
+ self.assertIn('Could not inspect run_telemetry.adapter', result.stderr)
+ self.assertIn('migrations/001_run_obs_generated_columns.sql', result.stderr)
+
def test_start_pull_with_nju_mirror_does_not_contact_ghcr_for_platform(self):
self.use_compose_file()
self.assert_success(self.install())
diff --git a/start.sh b/start.sh
index 27e37eff..45929f2b 100755
--- a/start.sh
+++ b/start.sh
@@ -51,6 +51,8 @@ PY
compose pull --ignore-buildable --policy always
fi
compose up -d --no-build --wait --wait-timeout 180
+ # A newer image may read generated columns the old volume never got.
+ bash scripts/install-schema-extensions.sh
;;
logs) compose logs -f "${@:-platform}" ;;
stop) compose stop ;;