Database governance built for humans and agents — controlling changes and access across every major database.
Go cicddata-governancedata-maskingdata-securitydatabase-accessdatabase-governancedevopsdevsecopsflywaygitopsliquibasemongodbmysqloraclepostgresqlschema-migrationssecuritysql-clientsqlservertidb
docs: corroborate revision provenance, land ref writers before backfill
Danny9c1f7e4378ed
diff --git a/docs/design/sheet-blob-scoping.md b/docs/design/sheet-blob-scoping.mdindex 3897d7a43c..93e445960a 100644--- a/docs/design/sheet-blob-scoping.md+++ b/docs/design/sheet-blob-scoping.md@@ -164,24 +164,37 @@ Derive the authoring project from the revision's own provenance instead. `Revisiname. Parse it out, preferring `release` and falling back to `taskRun`; `backend/store/changelog.go:163-168`already does exactly this with `regexp_match(payload->>'taskRun', 'projects/([^/]+)/')`.-The disposition is three-way, not a fallback chain, and conflating the middle case with the last one-is the trap:+**Provenance is usable only if it is corroborated**: the `release` or `task_run` row it names must+still exist under the project it names. Existence of the *project* is not enough. The disposition is+three-way, not a fallback chain, and conflating the middle case with the last one is the trap:| Provenance | Action ||---|---|-| Names a project that still exists | Insert `(that project, sha)` |-| Names a project that has been purged | **Insert nothing.** The string stays for attribution |+| Corroborated — the named `release` or `task_run` row still exists under that project | Insert `(that project, sha)` |+| Present but not corroborated | **Insert nothing.** The string stays for attribution || Absent — both fields empty | Fall back to `db.project`, and add to the review list |-**Purged provenance must not fall through to `db.project`.** `revision` is the only source that-survives a purge of its authoring project: `plan`, `task`, `release` and `plan_check_run` rows are-all deleted with the project (`backend/store/project.go:578`, `:641`, `:651`, `:671`), but a+**Uncorroborated provenance must not fall through to `db.project`.** `revision` is the only source+that survives a purge of its authoring project: `plan`, `task`, `release` and `plan_check_run` rows+are all deleted with the project (`backend/store/project.go:578`, `:641`, `:651`, `:671`), but aworkspace-instance database is reassigned to the default project and keeps its revisions, whose`release` and `taskRun` strings still name the deleted project. Inserting that ref would violate`sheet_blob_ref`'s foreign key and abort the migration; joining only to live projects and thenfalling back would hand the default project SQL the purged project authored — the grant this design-refuses. Guard the provenance branch with `EXISTS (SELECT 1 FROM project …)` and let those rows-produce no ref at all.+refuses.++**Checking that the project exists is not sufficient**, which is why the guard is on the referenced+row. `DeleteProject` hard-deletes the `project` row and `CreateProject` accepts a caller-supplied+`ProjectId` (`backend/api/v1/project_service.go:210`), so a purged ID can be reused. A surviving+revision naming the *old* project would then resolve to an unrelated *new* project of the same name+and grant it the purged project's SQL — a worse outcome than the foreign-key abort, because it is+silent. Corroborating the referenced row distinguishes the cases: ordinary release deletion is soft+(`backend/store/release.go:223`), so the row persists, while purge hard-deletes it.++Residual risk: a reused project ID whose new releases happen to collide with the old release or task+IDs would still corroborate. Release IDs are deterministic (`train` plus iteration) and task IDs+restart per project, so this is possible rather than impossible. It is not detectable from the data,+so it goes on the review list alongside the other ambiguous rows rather than being silently trusted.Their content then becomes unreadable, which is the intended outcome: the authoring project is gone,so there is nobody left to ask. The immutable `release`/`taskRun` string still identifies it, so the@@ -383,9 +396,9 @@ A long list means cross-project references are already widespread and the plan nsince the backfill would make them permanent. Capture it at a known-good point: this design ispublic, including the rule that turns a reference into an ownership edge.-**Review provenance-less revisions too.** These are the rows that fell back to `db.project`, so a-database transferred before the migration would have granted its destination project a source-project's SQL:+**Review ambiguous revision provenance too.** Two populations, both of which the backfill had to+guess at. Rows with no provenance fell back to `db.project`, so a database transferred before the+migration would have granted its destination project a source project's SQL:```sqlSELECT r.instance, r.db_name, d.project@@ -395,6 +408,20 @@ WHERE r.payload->>'sheetSha256' IS NOT NULLAND COALESCE(r.payload->>'taskRun', '') = '';```+Rows whose provenance named a project that no longer corroborates it produced no ref, so their+content is now unreadable. A large count here means either widespread project purges or reused+project IDs, and is worth understanding before enforcing:++```sql+SELECT r.instance, r.db_name,+ (regexp_match(r.payload->>'release', 'projects/([^/]+)/'))[1] AS named_project+FROM revision r+WHERE r.payload->>'sheetSha256' IS NOT NULL+ AND COALESCE(r.payload->>'release', '') <> ''+ AND NOT EXISTS (SELECT 1 FROM sheet_blob_ref b+ WHERE b.sha256 = decode(r.payload->>'sheetSha256', 'hex'));+```+## Transaction lock ordering`sheet_blob_ref` is a project-owned branch and takes a position in the canonical sibling order in@@ -478,6 +505,21 @@ No schema needed; worth landing separately and first.## Order1. The five independent fixes, T6 first.-2. Migration and backfill; run both verification queries and review the multi-project list.-3. Store API, gate, call sites, purge delete, tests — gate shipping in shadow mode.-4. Flip to enforcing one release later, once shadow logs are clean.+2. Create `sheet_blob_ref` and land the ref *writers* — the transactional `CreateSheets` and the+ purge delete — **before** any backfill.+3. Store API, gate, call sites, tests — gate shipping in shadow mode.+4. Run the backfill once the writers are live everywhere, then the verification queries; review the+ multi-project list and the ambiguous-provenance list.+5. Flip to enforcing one release later, once shadow logs are clean.++**Writers must precede the backfill, or new sheets fall into the gap.** A one-shot backfill scans+history; it cannot cover a sheet created after it runs. If the migration ships before `CreateSheets`+writes refs, every sheet created in between lands in `sheet_blob` with no ref and appears in no+historical source, so it becomes a zero-ref miss and a NotFound at enforcement. A rolling upgrade+widens the same gap even when both ship together, since old replicas keep serving the old+`CreateSheets` against the new schema until the rollout completes.++The backfill is idempotent (`ON CONFLICT DO NOTHING`), so the safe pattern is to run it after the+rollout has fully converged and to re-run it immediately before flipping to enforcing. Shadow-mode+misses are the backstop: anything the backfill missed shows up there as a log line rather than as a+production NotFound.