Database governance built for humans and agents — controlling changes and access across every major database.
Go cicddata-governancedata-maskingdata-securitydatabase-accessdatabase-governancedevopsdevsecopsflywaygitopsliquibasemongodbmysqloraclepostgresqlschema-migrationssecuritysql-clientsqlservertidb
refactor: drop id in worksheet_organizer (#19605)
ecmadao18476f68fd59
diff --git a/backend/migrator/migration/3.16/0004##drop_worksheet_organizer_id.sql b/backend/migrator/migration/3.16/0004##drop_worksheet_organizer_id.sqlnew file mode 100644index 0000000000..13965b3793--- /dev/null+++ b/backend/migrator/migration/3.16/0004##drop_worksheet_organizer_id.sql@@ -0,0 +1,5 @@+-- Drop unused id column from worksheet_organizer.+-- All queries use (worksheet_id, principal) which already has a unique index.+ALTER TABLE worksheet_organizer DROP CONSTRAINT worksheet_organizer_pkey;+ALTER TABLE worksheet_organizer DROP COLUMN id;+ALTER TABLE worksheet_organizer ADD CONSTRAINT worksheet_organizer_pkey PRIMARY KEY USING INDEX idx_worksheet_organizer_unique_sheet_id_principal;diff --git a/backend/migrator/migration/LATEST.sql b/backend/migrator/migration/LATEST.sqlindex 0adc58e34e..0a6873f771 100644--- a/backend/migrator/migration/LATEST.sql+++ b/backend/migrator/migration/LATEST.sql@@ -400,14 +400,12 @@ ALTER SEQUENCE worksheet_id_seq RESTART WITH 101;-- worksheet_organizer table stores the sheet status for a principal.CREATE TABLE worksheet_organizer (- id serial PRIMARY KEY,worksheet_id integer NOT NULL REFERENCES worksheet(id) ON DELETE CASCADE,principal text NOT NULL,- payload jsonb NOT NULL DEFAULT '{}'+ payload jsonb NOT NULL DEFAULT '{}',+ PRIMARY KEY (worksheet_id, principal));-CREATE UNIQUE INDEX idx_worksheet_organizer_unique_sheet_id_principal ON worksheet_organizer(worksheet_id, principal);-CREATE INDEX idx_worksheet_organizer_principal ON worksheet_organizer(principal);CREATE INDEX idx_worksheet_organizer_payload ON worksheet_organizer USING GIN(payload);diff --git a/backend/migrator/migrator_test.go b/backend/migrator/migrator_test.goindex 8e99aae3cb..43db12c665 100644--- a/backend/migrator/migrator_test.go+++ b/backend/migrator/migrator_test.go@@ -15,8 +15,8 @@ import (func TestLatestVersion(t *testing.T) {files, err := getSortedVersionedFiles()require.NoError(t, err)- require.Equal(t, semver.MustParse("3.16.3"), *files[len(files)-1].version)- require.Equal(t, "migration/3.16/0003##add_resource_id_to_api_tables.sql", files[len(files)-1].path)+ require.Equal(t, semver.MustParse("3.16.4"), *files[len(files)-1].version)+ require.Equal(t, "migration/3.16/0004##drop_worksheet_organizer_id.sql", files[len(files)-1].path)}func TestVersionUnique(t *testing.T) {diff --git a/backend/store/worksheet.go b/backend/store/worksheet.goindex ef490002ef..067e9fe31a 100644--- a/backend/store/worksheet.go+++ b/backend/store/worksheet.go@@ -285,8 +285,6 @@ func (s *Store) DeleteWorkSheet(ctx context.Context, sheetUID int) error {// WorksheetOrganizerMessage is the store message for worksheet organizer.type WorksheetOrganizerMessage struct {- UID int-// Related fieldsWorksheetUID intPrincipal string@@ -296,7 +294,6 @@ type WorksheetOrganizerMessage struct {func (s *Store) GetWorksheetOrganizer(ctx context.Context, worksheetUID int, principal string) (*WorksheetOrganizerMessage, error) {q := qb.Q().Space(`SELECT- id,payloadFROM worksheet_organizerWHERE worksheet_id = ? AND principal = ?@@ -314,7 +311,6 @@ func (s *Store) GetWorksheetOrganizer(ctx context.Context, worksheetUID int, pri}var payload []byteif err := s.GetDB().QueryRowContext(ctx, query, args...).Scan(- &worksheetOrganizer.UID,&payload,); err != nil {if err == sql.ErrNoRows {@@ -347,7 +343,6 @@ func (s *Store) UpsertWorksheetOrganizer(ctx context.Context, patch *WorksheetOrON CONFLICT(worksheet_id, principal) DO UPDATE SETpayload = EXCLUDED.payloadRETURNING- id,worksheet_id,principal,payload@@ -361,7 +356,6 @@ func (s *Store) UpsertWorksheetOrganizer(ctx context.Context, patch *WorksheetOrvar worksheetOrganizer WorksheetOrganizerMessagevar payload []byteif err := s.GetDB().QueryRowContext(ctx, query, args...).Scan(- &worksheetOrganizer.UID,&worksheetOrganizer.WorksheetUID,&worksheetOrganizer.Principal,&payload,diff --git a/docs/plans/saas/01.prepare-for-saas.md b/docs/plans/saas/01.prepare-for-saas.mdindex 5732554721..d16e6216e8 100644--- a/docs/plans/saas/01.prepare-for-saas.md+++ b/docs/plans/saas/01.prepare-for-saas.md@@ -223,366 +223,216 @@ Update the store/service layer where workspace policies are queried or created:----## 3. Migrate Integer IDs to String IDs+## 3. ID Migration for SaaS Multi-Tenancy### Why-Sequential integer IDs (`serial`/`bigserial`) have two problems for SaaS:+For SaaS, tables need different ID strategies depending on their scope:++1. **Project-scoped tables**: Keep `id` (convert to `bigserial`), auto-increment per project so IDs are human-readable within each project (e.g., project A: issue #1, #2, #3; project B: issue #1, #2, #3).+2. **Instance-scoped tables**: Keep `id`, auto-increment per instance.+3. **Cross-workspace tables**: Change `id` to `TEXT PRIMARY KEY` (uuid) since these have no natural scope boundary.+4. **Workspace-scoped tables**: Add `workspace_id` column for multi-tenancy.++### Complete Table Migration Summary++| Table | Current PK | Scope | Migration |+|---|---|---|---|+| **Top-level (no workspace_id)** | | | |+| `principal` | `serial` | Global | Deprecate `id`, use `email` as unique key |+| `web_refresh_token` | `token_hash text` | User-scoped | None |+| `sheet_blob` | `sha256 bytea` | Content-addressed | None |+| `replica_heartbeat` | `replica_id text` | Infra | None |+| **Instance-scoped (globally unique instance)** | | | |+| `instance` | `resource_id text` | Global | None |+| `db` | `(instance, name)` | Instance | None |+| `db_schema` | `(instance, db_name)` | Instance | None |+| `revision` | `bigserial` | Instance | Per-instance auto-increment id, add FK to `instance(resource_id)` |+| `sync_history` | `bigserial` | Instance | Per-instance auto-increment id |+| `changelog` | `bigserial` | Instance | Per-instance auto-increment id |+| **Workspace-scoped (need `workspace_id`)** | | | |+| `idp` | `resource_id text` | Workspace | Add `workspace_id` |+| `project` | `resource_id text` | Workspace | Add `workspace_id` |+| `service_account` | `email text` | Workspace | Add `workspace_id` |+| `workload_identity` | `email text` | Workspace | Add `workspace_id` |+| `setting` | `name text` | Workspace | Add `workspace_id` |+| `role` | `resource_id text` | Workspace | Add `workspace_id` |+| `policy` | `(resource_type, resource, type)` | Workspace | Add `workspace_id` |+| `user_group` | `id text` | Workspace | Add `workspace_id` |+| `review_config` | `id text` | Workspace | Add `workspace_id` |+| `oauth2_client` | `client_id text` | Workspace | Add `workspace_id` |+| `oauth2_authorization_code` | `code text` | Workspace (via client) | Add `workspace_id` |+| `oauth2_refresh_token` | `token_hash text` | Workspace (via client) | Add `workspace_id` |+| **Project-scoped (per-project auto-increment id)** | | | |+| `project_webhook` | `serial` | Project | Per-project auto-increment id |+| `plan` | `bigserial` | Project | Per-project auto-increment id |+| `issue` | `serial` | Project | Per-project auto-increment id |+| `worksheet` | `serial` | Project | Per-project auto-increment id |+| `plan_check_run` | `serial` | Project (via plan) | Add project ref |+| `plan_webhook_delivery` | `plan_id bigint` | Project (via plan) | Add project ref |+| `task` | `serial` | Project (via plan) | Add project ref |+| `task_run` | `serial` | Project (via task) | Add project ref |+| `task_run_log` | `(task_run_id, created_at)` | Project (via task_run) | Add project ref |+| `issue_comment` | `bigserial` | Project (via issue) | Add project ref |+| `worksheet_organizer` | `serial` | Project (via worksheet) | Add project ref |+| `db_group` | `(project, resource_id)` | Project | None (already has project) |+| `release` | `(project, train, iteration)` | Project | None (already has project) |+| **Cross-workspace (TEXT PK with uuid)** | | | |+| `audit_log` | `bigserial` | Cross-workspace | Change to `id text` PK (uuid) |+| `query_history` | `bigserial` | Cross-workspace | Change to `id text` PK (uuid), fix `project_id` FK reference |+| `instance_change_history` | `bigserial` | Cross-workspace | Change to `id text` PK (uuid) |+| `export_archive` | `serial` | Cross-workspace | Change to `id text` PK (uuid) |+| **Skip** | | | |+| `access_grant` | `id text` | Project | None (already text PK) |-1. **Security** -- Sequential IDs are guessable and leak information (total count, creation rate, ordering). An attacker can enumerate resources by incrementing IDs: `plans/101`, `plans/102`, etc.-2. **Multi-tenancy** -- In a shared database, sequential IDs leak cross-tenant information. A customer creating `issues/5000` can infer that ~5000 issues exist across all tenants.--String IDs (e.g., UUIDs or nanoids) are opaque, non-sequential, and safe to expose in URLs.--### ID Strategy Investigation--We evaluated four approaches to replace sequential integer IDs for SaaS. Key considerations:--- **Security**: IDs must be non-enumerable and leak no cross-tenant information.-- **Performance**: Indexed lookup speed matters for high-frequency queries.-- **Migration cost**: FK chains (`plan → task → task_run → task_run_log`) make schema migration expensive.-- **Consistency**: The codebase already has tables using text PKs (`project`, `instance`, `role`, `idp`, `user_group`, `review_config`, `access_grant`).--#### Plan 1: Keep Integer IDs (`bigserial`)--Change `serial` to `bigserial` where needed. IDs remain globally unique via PostgreSQL sequences (sequences are database-level, so even in a shared multi-tenant database, no two rows get the same ID).--| Pros | Cons |-|------|------|-| Zero migration cost -- no schema or FK changes | Sequential IDs are guessable (`plans/101`, `plans/102`) |-| Best query performance (8-byte integer index) | Leaks information: total count, creation rate, cross-tenant ordering |-| Simplest implementation | Inconsistent with existing text-PK tables |--**Verdict**: Fails the security requirement. Not suitable for SaaS.--#### Plan 2: Text UUID Replacing Integer ID (Chosen)--Add a `resource_id text NOT NULL DEFAULT gen_random_uuid()::text` column, migrate all store/API references from `id` to `resource_id`, migrate FK columns from integer to text, then drop the old `id` column. The `resource_id` becomes the new PK.--| Pros | Cons |-|------|------|-| Opaque, non-sequential, non-enumerable | ~30-40% slower indexed lookups vs bigint |-| Consistent with existing text-PK tables | Large migration: FK chain is 4 levels deep |-| Clean single-identity-per-row design | One-time cost to migrate all FK references |-| Shard-safe -- no coordination needed for ID generation | Larger storage (36 bytes vs 8 bytes per ID) |-| No custom ID generator required (`gen_random_uuid()` built-in) | |--**Verdict**: Best long-term design. One-time migration cost is worth the architectural simplicity.--#### Plan 3: Dual ID (Keep Integer `id` + Add Text `resource_id`)--Add `resource_id` for API exposure but keep integer `id` as PK and for all internal FK references. API layer maps between the two.--| Pros | Cons |-|------|------|-| No FK migration needed | Permanent dual-identity complexity |-| Internal queries stay on fast integer indexes | Every table has two identity columns forever |-| Incremental migration possible | API layer needs constant mapping between `id` and `resource_id` |-| | If you already have a globally unique UUID, the integer `id` serves no purpose |-| | Adds confusion about which ID to use in which context |--**Verdict**: Avoids migration pain but creates permanent schema complexity. If the UUID is already globally unique, keeping the integer `id` alongside it is unnecessary overhead.--#### Plan 4: Random `bigint`--Replace `serial`/`bigserial` with a random 64-bit integer generator (e.g., a custom PL/pgSQL function or application-level snowflake/KSUID). Keeps integer type but makes values non-sequential.--| Pros | Cons |-|------|------|-| Same performance as integer indexes | Needs custom ID generator (collision handling, uniqueness guarantees) |-| Non-sequential, harder to enumerate | Still leaks that IDs are integers (narrower space than UUID) |-| No type change -- FK columns stay integer | Inconsistent with existing text-PK tables |-| | 64-bit space has non-trivial collision probability at scale |--**Verdict**: Good performance but requires custom infrastructure and remains inconsistent with the existing text-PK pattern.--#### Decision: Plan 2--We chose Plan 2 (text UUID replacing integer `id`) because:+### Migration Plan-1. **Consistency** -- Aligns with the 10+ tables that already use text PKs. One ID pattern across the entire schema.-2. **Simplicity** -- Single identity per row. No mapping layer, no "which ID do I use?" confusion.-3. **Security** -- UUIDs are cryptographically random, non-enumerable, and leak no information.-4. **No custom infrastructure** -- PostgreSQL's built-in `gen_random_uuid()` handles generation.-5. **Future-proof** -- Text UUIDs are inherently shard-safe if we ever split into multiple databases.+#### Step 1: Remove Unused `id` Columns-The ~30-40% slower indexed lookups are acceptable given our query patterns (most hot paths filter by project/instance first, not by ID alone). The one-time FK migration cost is significant but bounded -- once complete, the schema is permanently simpler.+Drop `id` columns from tables that never use them — all lookups, JOINs, and FK references use natural keys.-### Tables That Expose Integer IDs in API+**Done in PR #19582** for: `idp`, `project`, `setting`, `role`, `policy`, `instance`, `db`, `db_schema`, `task_run_log`, `db_group`, `release`.-These 9 tables currently expose their integer `id` in API resource names:+**TODO:** `worksheet_organizer` — the `id serial PRIMARY KEY` is unused. All queries use `(worksheet_id, principal)` which already has a unique index. Drop `id` and promote `(worksheet_id, principal)` to PK.-| Table | API format | Current type |-|-------|-----------|-------------|-| `plan` | `projects/{project}/plans/{plan}` | bigserial |-| `task` | `.../tasks/{task}` | serial |-| `task_run` | `.../taskRuns/{taskRun}` | serial |-| `issue` | `projects/{project}/issues/{issue}` | serial |-| `issue_comment` | `.../issueComments/{issueComment}` | bigserial |-| `revision` | `.../revisions/{revision}` | bigserial |-| `changelog` | `.../changelogs/{changelog}` | bigserial |-| `worksheet` | `worksheets/{worksheet}` | serial |-| `project_webhook` | `projects/{project}/webhooks/{webhook}` | serial |+#### Step 2: Add Project/Instance Reference and Composite PKs-### Tables with Internal-Only Integer IDs+Add the scope column (`project` or `instance`) to child tables, change PKs from `(id)` to `(project, id)` or `(instance, id)`, and update all FK references to be composite. This ensures every query includes the scope, which is required for per-scope auto-increment in Step 3 and for tenant isolation in SaaS.-These 4 tables use integer IDs internally but do NOT expose them in API paths:+Split by FK chain:-| Table | Usage |-|-------|-------|-| `sync_history` | Referenced via `changelog.sync_history_id` FK |-| `query_history` | Internal query audit trail |-| `export_archive` | Internal export tracking |-| `worksheet_organizer` | Internal worksheet pinning |+**Step 2a: `plan` chain**-### Tables Already Using String IDs+Tables: `plan`, `task`, `task_run`, `task_run_log`, `plan_check_run`, `plan_webhook_delivery`-These tables already use string/email identifiers (no change needed):+Schema changes:+```+plan(id) → plan(project, id)+ -- plan already has `project` column, just change PK-- `principal` -- `users/{email}`-- `service_account` -- `serviceAccounts/{email}`-- `workload_identity` -- `workloadIdentities/{email}`-- `project` -- `projects/{resource_id}`-- `instance` -- `instances/{resource_id}`-- `idp` -- `idps/{resource_id}`-- `role` -- `roles/{resource_id}`-- `user_group` -- UUID text PK-- `review_config` -- text PK-- `access_grant` -- UUID text PK+plan_check_run(id) → plan_check_run(project, id)+ -- add `project` column, backfill from plan+ -- plan_check_run.plan_id: update UNIQUE index to (project, plan_id)-### Current Store Layer Details+plan_webhook_delivery(plan_id) → plan_webhook_delivery(project, plan_id)+ -- add `project` column, backfill from plan+ -- PK becomes (project, plan_id)-| Table | Message Field | Parsed Return Type | FK Dependents |-|-------|--------------|-------------------|---------------|-| `plan` | `UID int64` | `int64` via `GetProjectIDPlanID()` | `task.plan_id`, `issue.plan_id`, `plan_check_run.plan_id`, `plan_webhook_delivery.plan_id` |-| `task` | `ID int` | `int` via `GetProjectIDPlanIDStageIDTaskID()` | `task_run.task_id` |-| `task_run` | `ID int` | `int` via `GetProjectIDPlanIDStageIDTaskIDTaskRunID()` | `task_run_log.task_run_id` |-| `issue` | `UID int` | `int` via `GetProjectIDIssueUID()` | `issue_comment.issue_id` |-| `issue_comment` | `UID int` | `int` via `GetProjectIDIssueUIDIssueCommentUID()` | None |-| `revision` | `UID int64` | `int64` via `GetInstanceDatabaseRevisionID()` | None |-| `changelog` | `UID int64` | `int64` via `GetInstanceDatabaseChangelogUID()` | None (FK to `sync_history` is internal) |-| `worksheet` | `UID int` | `int` via `GetWorksheetUID()` | `worksheet_organizer.worksheet_id` |-| `project_webhook` | `ID int` | `int` (no dedicated parser yet) | None |+task(id) → task(project, id)+ -- add `project` column, backfill from plan+ -- task.plan_id FK → (project, plan_id) REFERENCES plan(project, id)-### JSONB / Proto Fields Storing Integer IDs+task_run(id) → task_run(project, id)+ -- add `project` column, backfill from task+ -- task_run.task_id FK → (project, task_id) REFERENCES task(project, id)+ -- UNIQUE(task_id, attempt) → UNIQUE(project, task_id, attempt)-These serialized fields embed integer IDs and need migration alongside the tables:+task_run_log(task_run_id, created_at) → task_run_log(project, task_run_id, created_at)+ -- add `project` column, backfill from task_run+ -- task_run_log.task_run_id FK → (project, task_run_id) REFERENCES task_run(project, id)+```-| JSONB Location | Proto Field | Type | References |-|---------------|-------------|------|-----------|-| `access_grant.payload` | `AccessGrantPayload.issue_id` | `int64` | `issue.id` |-| `task_run.result` | `TaskRunResult.export_archive_uid` | `int32` | `export_archive.id` (internal) |-| PG NOTIFY channel | `Signal.uid` | `int32` | `task_run.id` or `plan_check_run.uid` |+Code changes: **all SQL queries AND application logic** must use `(project, id)` together to identify a row — not `id` alone. This includes:+- Store queries: WHERE clauses, FK lookups, JOINs+- API layer: resource name parsing must extract and pass both project and id+- Runners/schedulers: any code that passes IDs via channels, signals, or caches must include project+- Proto messages: any message carrying an id (e.g., `Signal.uid`) must also carry the project-### FK Dependency Graph+**Step 2b: `issue` chain**-Migration must proceed bottom-up through the FK chain:+Tables: `issue`, `issue_comment`+Schema changes:```-plan ← task ← task_run ← task_run_log- ↑ ↑- │ └─ (task_run has UNIQUE(task_id, attempt))- ├─ issue ← issue_comment- ├─ plan_check_run (UNIQUE on plan_id, singleton)- └─ plan_webhook_delivery (plan_id is PK)--worksheet ← worksheet_organizer (ON DELETE CASCADE)--revision (standalone)-changelog (standalone, FK to sync_history is internal)-project_webhook (standalone)+issue(id) → issue(project, id)+ -- issue already has `project` column, just change PK+ -- issue.plan_id FK → UNIQUE index on plan(project, plan_id)... but issue.plan_id+ -- is nullable and references plan(id). After Step 2a, need composite FK.+ -- UNIQUE(plan_id) → UNIQUE(project, plan_id)++issue_comment(id) → issue_comment(project, id)+ -- add `project` column, backfill from issue+ -- issue_comment.issue_id FK → (project, issue_id) REFERENCES issue(project, id)```-### Migration Plan--**Phase 0: Drop unused `id` columns from tables that already use natural keys**--Many tables have a `serial`/`bigserial` `id` column as PRIMARY KEY but never use it -- all lookups, JOINs, and foreign keys use natural keys (`resource_id`, `email`, composite unique constraints). No other tables have FK references to these `id` columns. Dropping them simplifies the schema before adding `resource_id` to the API-exposed tables.+**Step 2c: `worksheet` chain**-11 tables have completely unused `id` columns:+Tables: `worksheet`, `worksheet_organizer`-| Table | Natural Key | `id` usage in store | FK references to `id`? |-|-------|------------|-------------------|----------------------|-| `project` | `resource_id` (unique) | None | No (FKs use `resource_id`) |-| `instance` | `resource_id` (unique) | None | No (FKs use `resource_id`) |-| `db` | `(instance, name)` (unique) | `RETURNING id` (scanned but discarded) | No (FKs use `(instance, name)`) |-| `setting` | `name` (unique) | None | No |-| `policy` | `(resource_type, resource, type)` (unique) | None | No |-| `idp` | `resource_id` (unique) | `ORDER BY id` only | No |-| `role` | `resource_id` (unique) | None | No |-| `db_schema` | `(instance, db_name)` (unique) | None | No |-| `db_group` | `(project, resource_id)` (unique) | None | No |-| `task_run_log` | `task_run_id` (FK) | `ORDER BY id` only | No |-| `release` | `(project, train, iteration)` (unique) | `RETURNING id` (discarded), `ORDER BY id` | No |+Schema changes:+```+worksheet(id) → worksheet(project, id)+ -- worksheet already has `project` column, just change PK-**Excluded:** `instance_change_history` -- the migrator uses `ORDER BY id DESC` to find the latest applied version, and since `version` is a semver text column, lexicographic sort does not work. Keep `id` here.+worksheet_organizer(worksheet_id, principal) → worksheet_organizer(project, worksheet_id, principal)+ -- add `project` column, backfill from worksheet+ -- worksheet_organizer.worksheet_id FK → (project, worksheet_id) REFERENCES worksheet(project, id)+```-Migration SQL:+**Step 2d: `project_webhook`**-```sql--- project: natural key = resource_id-ALTER TABLE project DROP COLUMN id;-ALTER TABLE project ADD PRIMARY KEY (resource_id);---- instance: natural key = resource_id-ALTER TABLE instance DROP COLUMN id;-ALTER TABLE instance ADD PRIMARY KEY (resource_id);---- db: natural key = (instance, name)-ALTER TABLE db DROP COLUMN id;-ALTER TABLE db ADD PRIMARY KEY (instance, name);-DROP INDEX IF EXISTS idx_db_unique_instance_name;---- setting: natural key = name-ALTER TABLE setting DROP COLUMN id;-ALTER TABLE setting ADD PRIMARY KEY (name);---- policy: natural key = (resource_type, resource, type)-ALTER TABLE policy DROP COLUMN id;-ALTER TABLE policy ADD PRIMARY KEY (resource_type, resource, type);---- idp: natural key = resource_id-ALTER TABLE idp DROP COLUMN id;-ALTER TABLE idp ADD PRIMARY KEY (resource_id);---- role: natural key = resource_id-ALTER TABLE role DROP COLUMN id;-ALTER TABLE role ADD PRIMARY KEY (resource_id);---- db_schema: natural key = (instance, db_name)-ALTER TABLE db_schema DROP COLUMN id;-ALTER TABLE db_schema ADD PRIMARY KEY (instance, db_name);-DROP INDEX IF EXISTS idx_db_schema_unique_instance_db_name;---- db_group: natural key = (project, resource_id)-ALTER TABLE db_group DROP COLUMN id;-ALTER TABLE db_group ADD PRIMARY KEY (project, resource_id);-DROP INDEX IF EXISTS idx_db_group_unique_project_resource_id;---- task_run_log: no natural PK, use task_run_id + created_at-ALTER TABLE task_run_log DROP COLUMN id;-ALTER TABLE task_run_log ADD PRIMARY KEY (task_run_id, created_at);---- release: natural key = (project, train, iteration)-ALTER TABLE release DROP COLUMN id;-ALTER TABLE release ADD PRIMARY KEY (project, train, iteration);-DROP INDEX IF EXISTS idx_release_project_train_iteration;+```+project_webhook(id) → project_webhook(project, id)+ -- already has `project` column, just change PK```-Code changes:--- **database.go**: Replace `RETURNING id` with `RETURNING instance, name` (or just remove it and use `ExecContext` instead of `QueryRowContext` since the returned value is discarded). 3 places: `CreateDatabase`, `UpsertDatabase`, `UpdateDatabase`.-- **release.go**: Change `RETURNING id, created_at` to `RETURNING created_at`. Remove `var id int64` scan variable.-- **idp.go**: Replace `ORDER BY id ASC` with `ORDER BY resource_id ASC`.-- **task_run_log.go**: Replace `ORDER BY id` with `ORDER BY created_at`.-- **release.go**: Replace `ORDER BY release.id DESC` with `ORDER BY release.created_at DESC`.-- **LATEST.sql**: Update table definitions to remove `id` columns, remove `ALTER SEQUENCE ... RESTART` statements, promote natural keys to PRIMARY KEY, drop redundant unique indexes.-- **migrator_test.go**: Update `TestLatestVersion`.+**Step 2e: Instance-scoped tables**-**Phase 1: Add `resource_id` column to API-exposed tables (DB-only, no code changes)**+Tables: `revision`, `sync_history`, `changelog`-```sql--- For each API-exposed table, add resource_id column.--- Use gen_random_uuid() for simplicity; can switch to nanoid if preferred.-ALTER TABLE plan ADD COLUMN resource_id text NOT NULL DEFAULT gen_random_uuid()::text;-ALTER TABLE task ADD COLUMN resource_id text NOT NULL DEFAULT gen_random_uuid()::text;-ALTER TABLE task_run ADD COLUMN resource_id text NOT NULL DEFAULT gen_random_uuid()::text;-ALTER TABLE issue ADD COLUMN resource_id text NOT NULL DEFAULT gen_random_uuid()::text;-ALTER TABLE issue_comment ADD COLUMN resource_id text NOT NULL DEFAULT gen_random_uuid()::text;-ALTER TABLE revision ADD COLUMN resource_id text NOT NULL DEFAULT gen_random_uuid()::text;-ALTER TABLE changelog ADD COLUMN resource_id text NOT NULL DEFAULT gen_random_uuid()::text;-ALTER TABLE worksheet ADD COLUMN resource_id text NOT NULL DEFAULT gen_random_uuid()::text;-ALTER TABLE project_webhook ADD COLUMN resource_id text NOT NULL DEFAULT gen_random_uuid()::text;---- Create unique indexes-CREATE UNIQUE INDEX idx_plan_unique_resource_id ON plan(resource_id);-CREATE UNIQUE INDEX idx_task_unique_resource_id ON task(resource_id);-CREATE UNIQUE INDEX idx_task_run_unique_resource_id ON task_run(resource_id);-CREATE UNIQUE INDEX idx_issue_unique_resource_id ON issue(resource_id);-CREATE UNIQUE INDEX idx_issue_comment_unique_resource_id ON issue_comment(resource_id);-CREATE UNIQUE INDEX idx_revision_unique_resource_id ON revision(resource_id);-CREATE UNIQUE INDEX idx_changelog_unique_resource_id ON changelog(resource_id);-CREATE UNIQUE INDEX idx_worksheet_unique_resource_id ON worksheet(resource_id);-CREATE UNIQUE INDEX idx_project_webhook_unique_resource_id ON project_webhook(resource_id);+Schema changes:```+revision(id) → revision(instance, id)+ -- revision already has `instance` column, just change PK-**Phase 2: Switch store/API layer to use `resource_id`**--Migrate tables in dependency order -- leaf tables first, root tables last.+sync_history(id) → sync_history(instance, id)+ -- sync_history already has `instance` column, just change PK-*Group A -- Standalone tables (no FK dependents):*-- `revision`, `changelog`, `project_webhook`, `issue_comment`-- Simplest to migrate: change message struct, SQL queries, and resource name parsers.+changelog(id) → changelog(instance, id)+ -- changelog already has `instance` column, just change PK+ -- changelog.sync_history_id FK → (instance, sync_history_id) REFERENCES sync_history(instance, id)+```-*Group B -- Tables with FK dependents:*-- `worksheet` (depended on by `worksheet_organizer.worksheet_id`)-- `task_run` (depended on by `task_run_log.task_run_id`)-- `task` (depended on by `task_run.task_id`)-- `issue` (depended on by `issue_comment.issue_id`)+#### Step 3: Per-Project / Per-Instance Auto-Increment IDs-*Group C -- Root table:*-- `plan` (depended on by `task.plan_id`, `issue.plan_id`, `plan_check_run.plan_id`, `plan_webhook_delivery.plan_id`)+After Step 2, all scoped tables have composite PKs `(project, id)` or `(instance, id)`. Now switch from global sequences to per-scope auto-increment.-For each table:-1. Change message struct field from `UID int`/`ID int` to `ResourceID string`-2. Update SQL: `SELECT resource_id` instead of `SELECT id`, `WHERE resource_id = $1` instead of `WHERE id = $1`, `RETURNING resource_id` instead of `RETURNING id`-3. Update `resource_name.go` parsing functions to return `string` instead of `int`/`int64`. Remove `strconv.Atoi()`/`strconv.ParseInt()` calls.-4. Update all callsites that use the parsed ID+Convert `serial`/`bigserial` to `bigint` (drop the global sequence default). Implement per-scope ID generation — options:-**Phase 3: Migrate FK columns from integer to string**+1. **Counter table**: e.g., `id_sequence(scope_type, scope_id, table_name, next_val)` with atomic `UPDATE ... RETURNING`.+2. **`SELECT MAX(id) + 1`** within the insert transaction with row-level locking on the scope.+3. **`SELECT COALESCE(MAX(id), 0) + 1`** with advisory lock on `(project, table_name)`.-After each table's own `resource_id` is live, update FK columns in dependent tables to reference the new string column. This requires a SQL migration per FK:+Tables affected:+- Project-scoped: `plan`, `issue`, `project_webhook`, `worksheet`, `plan_check_run`, `task`, `task_run`, `issue_comment`+- Instance-scoped: `revision`, `sync_history`, `changelog`-| FK column | Current reference | New reference |-|-----------|------------------|--------------|-| `task.plan_id` | `bigint → plan(id)` | `text → plan(resource_id)` |-| `issue.plan_id` | `bigint → plan(id)` | `text → plan(resource_id)` |-| `plan_check_run.plan_id` | `bigint → plan(id)` | `text → plan(resource_id)` |-| `plan_webhook_delivery.plan_id` | `bigint → plan(id)` | `text → plan(resource_id)` |-| `task_run.task_id` | `int → task(id)` | `text → task(resource_id)` |-| `task_run_log.task_run_id` | `int → task_run(id)` | `text → task_run(resource_id)` |-| `issue_comment.issue_id` | `int → issue(id)` | `text → issue(resource_id)` |-| `worksheet_organizer.worksheet_id` | `int → worksheet(id)` | `text → worksheet(resource_id)` |-| `changelog.sync_history_id` | `bigint → sync_history(id)` | Keep as-is (internal) |+#### Step 4: Cross-Workspace Tables — Text UUID PKs-Note: `task_run` has `UNIQUE(task_id, attempt)` -- this constraint must be recreated with the new text `task_id`.+For tables with no natural scope, replace `serial`/`bigserial` with `text` PK using `gen_random_uuid()`:-**Phase 4: Migrate JSONB/proto fields storing integer IDs**+| Table | Current | After |+|---|---|---|+| `audit_log` | `id bigserial PRIMARY KEY` | `id text PRIMARY KEY DEFAULT gen_random_uuid()::text` |+| `query_history` | `id bigserial PRIMARY KEY` | `id text PRIMARY KEY DEFAULT gen_random_uuid()::text`, fix `project_id` to proper FK |+| `instance_change_history` | `id bigserial PRIMARY KEY` | `id text PRIMARY KEY DEFAULT gen_random_uuid()::text` |+| `export_archive` | `id serial PRIMARY KEY` | `id text PRIMARY KEY DEFAULT gen_random_uuid()::text` |-| Proto field | Change | Data migration |-|------------|--------|---------------|-| `AccessGrantPayload.issue_id` | `int64` → `string` | Update `access_grant.payload` JSONB: convert `issueId` integer to `issue.resource_id` string |-| `TaskRunResult.export_archive_uid` | `int32` → `string` | Update `task_run.result` JSONB: convert `exportArchiveUid` to string |-| `Signal.uid` | `int32` → `string` | Update PG NOTIFY signal format (transient, no data migration) |+Also update any FK references to these tables (e.g., `changelog.sync_history_id` stays internal after Step 2e, `task_run.result` JSONB field `exportArchiveUid` needs proto/data migration).-**Phase 5: Drop old integer `id` columns and promote `resource_id` to PK**+### JSONB / Proto Fields Storing Integer IDs-After all FK references are migrated, drop the old integer `id` columns from the 9 API-exposed tables.+| JSONB Location | Proto Field | Type | References | Needs migration? |+|---|---|---|---|---|+| `task_run.result` | `TaskRunResult.export_archive_uid` | `int32` | `export_archive.id` | **Yes** — `export_archive.id` changes to text uuid in Step 4 |+| `access_grant.payload` | `AccessGrantPayload.issue_id` | `int64` | `issue.id` | No — `issue.id` stays integer (per-project bigint) |+| PG NOTIFY channel | `Signal.uid` | `int32` | `task_run.id` or `plan_check_run.id` | **Yes (Step 3)** — IDs are only unique per project after Step 3, so Signal must include `project` alongside `uid` to uniquely identify the resource |### Risks & Considerations-1. **Massive changeset** -- Touches virtually every store file, every API service file, and every resource name parsing function. Must be split into multiple PRs.-2. **FK chain depth** -- `plan → task → task_run → task_run_log` is 4 levels deep. Must migrate bottom-up.-3. **Unique constraint** -- `task_run` has `UNIQUE(task_id, attempt)` that needs recreating when `task_id` becomes text.-4. **`plan_check_run`** is a singleton per plan (unique on `plan_id`) -- also needs FK migration.-5. **Backward compatibility** -- Existing URLs with integer IDs in bookmarks/webhooks will break. Consider keeping integer-to-resource_id lookup during a transition period.-6. **Frontend** -- No change needed (the frontend already treats resource IDs as opaque strings in URL paths).-7. **Proto** -- No v1 API proto change needed (resource names are already strings).--### Recommended PR Split--| PR | Scope | Description |-|----|-------|-------------|-| 0 | Phase 0 | Drop unused `id` columns from 11 tables, promote natural keys to PK |-| 1 | Phase 1 | Add `resource_id` columns + unique indexes to all 9 API-exposed tables |-| 2 | Phase 2 Group A | `revision`, `changelog`, `project_webhook`, `issue_comment` -- store/API switch to `resource_id` |-| 3 | Phase 2 `worksheet` | Store/API switch + `worksheet_organizer` FK migration |-| 4 | Phase 2 `issue` | Store/API switch + `issue_comment.issue_id` FK migration |-| 5 | Phase 2 `task_run` | Store/API switch + `task_run_log.task_run_id` FK migration |-| 6 | Phase 2 `task` | Store/API switch + `task_run.task_id` FK migration (recreate UNIQUE constraint) |-| 7 | Phase 2 `plan` | Store/API switch + all plan FK dependents (`task`, `issue`, `plan_check_run`, `plan_webhook_delivery`) |-| 8 | Phase 3-5 | FK column migrations, JSONB/proto field migrations, drop old `id` columns |--### Internal-Only Tables--The 4 tables with internal-only integer IDs (`sync_history`, `query_history`, `export_archive`, `worksheet_organizer`) can keep integer IDs since they are never exposed in API paths. However, for consistency, consider migrating them in a follow-up.+1. **Step 2 is the largest change** — touches every store file in the FK chains, every API service, and every resource name parser. Split by FK chain (2a-2e) to keep PRs manageable.+2. **Composite FK rewiring** — changing PK from `(id)` to `(project, id)` requires dropping and recreating all dependent FKs, unique indexes, and constraints (e.g., `UNIQUE(task_id, attempt)` → `UNIQUE(project, task_id, attempt)`).+3. **`plan_check_run`** is a singleton per plan — unique index on `plan_id` becomes `(project, plan_id)`.+4. **`issue.plan_id`** is nullable — composite FK `(project, plan_id)` referencing `plan(project, id)` needs special handling for NULL `plan_id`.+5. **Backward compatibility** — existing API URLs with integer IDs will still work since the IDs don't change, only the PK structure does. Per-project IDs in Step 3 may cause ID reuse across projects.+6. **Per-scope auto-increment mechanism** (Step 3) needs careful concurrency handling to avoid duplicate IDs under concurrent inserts within the same project.