Database governance built for humans and agents — controlling changes and access across every major database.
Audit payload redaction from proto annotations
Status: implemented · 2026-08-21 · addresses BYT-10090
Background
Audited RPCs write their request and response payloads to audit_log and to stdout.
getRequestString and getResponseString (backend/api/v1/audit.go:564, :748) switch on the
top-level message type and dispatch to one of 35 hand-written redactors, so a secret is
protected only on the RPCs someone remembered and is exposed the moment the same field appears
under a new parent.
Three credentials leaked that way — an OIDC client secret, a service-account key, and the SCIM
directory-sync token — fixed on main by #21110 and #21109. Three more are still unprotected, and
two of them are live today.
Instance.roles[].password is INPUT_ONLY (instance_role_service.proto:82) — only its container
Instance.roles is OUTPUT_ONLY — so clients are meant to send it, CreateInstance,
UpdateInstance and BatchUpdateInstances are all audited, and redactInstance passes roles
through untouched: {"name":"instances/i","roles":[{"roleName":"r","password":"secret"}]}.
UpdateUserRequest.otp_code is the second — redactUpdateUserRequest copies it verbatim
(audit.go:1177) while the handler validates it as an MFA proof (user_service.go:442-449), and a
failed validation neither consumes the code nor suppresses the audit row, so a still-live OTP
reaches audit_log and stdout.
database.instance_resource.data_sources[] is the third and the one that is not live. The
OUTPUT_ONLY sits on Database.instance_resource, not on the credentials; reads do populate
data_sources[], and what keeps them safe is leaf-level blanking in convertDataSources. It
survives on BatchUpdateDatabases as well as UpdateDatabase. Separately, QueryResult has two
redactors that disagree on whether rows_count survives.
Goals
- A field is marked once in the
.protoand redacted everywhere it appears, at any depth, on every RPC. - Reviewing what the audit log records means reading
.proto, not Go. - Cost does not scale with payload size. A
QueryResponsecan carry 100k+ rows. - A new secret field, or an existing one under a new parent, fails CI rather than shipping.
- A credential is never returned on a read path either (#21205,
66fba49268were read-path leaks of these same secrets). - Redaction does not mutate the caller's message — it runs on the live objects the handler returns.
Non-goals
- Sanitizing existing
audit_logrows. Treat them as disclosed and rotate. No purge path exists for that table; providing one is separate work. - Proving goal 5 outside the declared surfaces. The read-path assertion covers the population
defined under Enforcement below. What it cannot see is a function that populates a
SENSITIVEfield while sitting outside that population and outside the declared minting pairs. That residue is a review gate, not a proof. The rule itself lives in one place, under Enforcement; restating it here is how it drifted before. - Secrets inside free-form text.
redactRoleAttribute(read_redaction.go) masks a password hash within MariaDBSHOW GRANTSoutput. No field annotation expresses that; it stays hand-written on the read path. - Audit rows written outside the interceptor.
backend/component/recovery/service.go:420-429callsstore.CreateAuditLogdirectly, building itsRequeststring withencoding/jsonover anonymous Go structs (:155,:310,:370) — no proto message and no descriptor, so no annotation, plan, sweep, inventory or registry can observe it.resetUserPasswordmarshals that JSON withrequest.Passwordin scope. Goal 4 does not reach this writer; bringing it in means giving it a proto message, which is separate work. - Runtime enforcement. The redactor is a denylist, so at runtime an unannotated field is logged. The inventory below moves that failure to CI; nothing catches it in a binary built past a skipped lint, where today's allowlist rebuilds fail closed.
Design
Annotations
One extension, an enum, declared in proto/v1/v1/annotation.proto:
enum AuditBehavior {
AUDIT_BEHAVIOR_UNSPECIFIED = 0; // recorded normally
SENSITIVE = 1;
OMIT = 2;
}
extend google.protobuf.FieldOptions {
AuditBehavior audit_behavior = 100010;
}
string client_secret = 5 [(bytebase.v1.audit_behavior) = SENSITIVE];
repeated QueryRow rows = 3 [(bytebase.v1.audit_behavior) = OMIT];
| must not reach an audit payload | may the API return it | |
|---|---|---|
SENSITIVE |
yes | only on the response that mints it; never on a read path |
OMIT |
yes | yes — returning it is the point of the RPC |
ServiceAccount.service_key forces that distinction: CreateServiceAccount and key rotation
return it exactly once (service_account_service.go:117, :270), while convertToServiceAccount
never populates it. RotateDirectorySyncTokenResponse has the same shape.
The boundary is not "handler versus converter". UpdateUser mints through a converter, choosing
convertToUserMintingMFAEnrollment over convertToUser when an enrollment is in flight
(user_service.go:498-500), and that one deliberately sets temp_otp_secret and
temp_recovery_codes (:786-787).
So minting surfaces are declared, as (function, field) pairs rather than whole functions:
convertToUserMintingMFAEnrollment may populate temp_otp_secret and temp_recovery_codes, and
nothing else. User also carries password and service_key; exempting the function wholesale
would let a later line inside it return either one and still pass CI. The assertion runs on every
converter, minting ones included, and permits only what the pair declares — the shape
mcpDenialRequestsUnderReview already uses, for the reason its own comment gives: an exemption
granted broadly is an exemption for everything that later lands inside it.
Binding the pair to the producing function is necessary but not sufficient, because a caller that
merely returns another function's result assigns nothing itself. A GetUser written as
return convertToUserMintingMFAEnrollment(...) would satisfy both entries — the converter's pair
allows the MFA fields, and GetUser's own body assigns no SENSITIVE field — while a read RPC
handed back an enrollment secret. So the allowance is declared a second time, per RPC output:
UpdateUser may emit temp_otp_secret, GetUser may not. A value reaches a client only when the
producing function is allowed to assign it and the emitting RPC is allowed to return it. That
second half is where the table's "only on the response that mints it" is actually enforced; the
function pairs alone say only who may write it, never who may ship it.
The list is not one entry. LoginResponse.token and .mfa_temp_token (auth_service.proto:254,
:257) and ExchangeTokenResponse.access_token (:280) are the product's primary access tokens,
minted and returned by design — redactLoginResponse and redactExchangeTokenResponse exist for
exactly that reason. Neither is set through a convertTo* function, so the completeness lint below
never reaches them; they have to be named here, alongside ServiceAccount.service_key and the SCIM
token.
Declared rather than inferred, because an assertion applied blindly to every converter would reject a legitimate enrollment response, and the cheap way out would be to leave those two fields unannotated to keep CI green.
Not audit and not number 100000: bytebase.v1.audit already exists as a bool on
MethodOptions, and 100000 is allow_without_credential. Different extend scopes make reuse
legal, but grep is how a reviewer audits this, so the identifier has to be unambiguous.
Why one enum rather than two bools
The alternative is a bool per behavior — sensitive = true on credentials, audit_omit = true on
bulk. Both encode the same two classes and both read from the descriptor, so the choice rests on
three things.
Exclusivity becomes structural. Two bools permit [(sensitive) = true, (audit_omit) = true],
which is meaningless and which nothing rejects. One classification per field is unviolatable in the
enum.
One vocabulary at the field. = SENSITIVE and = OMIT read in parallel and answer to one
grep. Two bools mean two identifiers to know and two greps to audit a proto file.
One read in the redactor. A single GetExtension plus a switch, rather than two extension
reads with separate absent-value handling. A third behavior, if one is ever needed, is an enum
value rather than a third extension.
google.protobuf.FieldOptions.debug_redact was considered as the credential marker and rejected.
It is inert in Go — nothing in protojson, prototext, or the runtime reads it, at field or
enum-value level — so it would add no enforcement to a Go backend while reading like it did. The
lints are the whole enforcement story, and an annotation implying otherwise is worse than none.
Redaction
The hook point is marshalAuditPayload (originally two functions, getRequestString and
getResponseString), not WrapUnary. Unary and
streaming rows reach redaction only through those two: auditConnectStreamingConn.Send
(audit.go:171-193) builds its own auditEntry and calls createAuditLog directly. AdminExecute
is the only streaming RPC, is audited, and its response carries every row of an admin-mode query, so
moving the walk up into WrapUnary — where the service_data and permission-refusal plumbing
lives — silently drops streaming redaction. One end-to-end assertion on the Send path is
required: streaming persistence is otherwise exercised through a fake audit.LogWriter,
which bypasses the real insert.
A plan per message type, built from the descriptor on first use and cached by
protoreflect.FullName: the fields to drop, and the submessage fields that lead to one. An
annotated field is recorded and never descended into. A type needing nothing caches a nil plan —
343 of the 418 (method, direction) pairs. The surface has 209 methods and only 320 distinct
request/response types, so pairs and types are not interchangeable, and the cache is keyed on types.
Copy-and-share, one mechanism. Every message on the path to a redacted field is copied by
Range — never by iterating Descriptor().Fields() and calling Set, which panics on an unset
message, list or map field ("cannot be set with read-only value", "has invalid nil pointer") inside
the request path. Range yields only populated fields, which is also how an omitted field is never
copied: it is skipped during the copy, not cleared afterwards on a message that never held it. Every
subtree no redacted field sits under is shared by pointer and only read. No second strategy, no
per-type branch. Go has no filtered clone to reuse — proto.Clone, CloneOf and Merge take no
options, and field-mask libraries prune after a full clone — so this is that primitive.
Constraints:
A shared subtree is reachable from two messages, so the redacted copy is write-once: marshaled and discarded.
TestAuditRedactionDoesNotMutateInputpins this.Where a field must be dropped from a message that already holds it, that is
Clear(), not assignment to""— they differ foroptionalfields, where""leaves{"password":""}.InstanceRole.passwordisoptional. On the copy path the field is simply never copied.A
SENSITIVEoneof member is blanked; anOMITone is cleared. Clearing a scalar arm unsets the oneof and erases which arm was supplied:DataSourceExternalSecret.token(instance_service.proto:644) is a string arm whose siblingapp_roleis a message and would survive as{}, so clearing would make token auth indistinguishable from unconfigured while AppRole stayed legible. Blanking keeps the arm present with no value —{"token":""}— which is already the read path's convention (instance_service_converter_test.go:457: "Oneof members may stay present as an is-configured signal ... require blank content, not absence").That rationale is
SENSITIVE-only. ForOMITthe arm's presence is not audit metadata, and blanking would leave{"field":""}in a payload the annotation says to drop, soOMITclears. No currentOMITfield is a oneof arm; the rule exists so the first one does not silently contradict its own annotation.Otherwise oneofs need no special case: only the set arm reports
Has(), and clearing a field inside a message arm leaves the arm set. That coversredactIAMExtension's three current arms but not the function. Itsdefault:arm nils an unrecognized variant (audit.go:1308-1312, "A variant added to the oneof after this function was written is dropped whole rather than logged") — fail-closed, with no denylist equivalent, so a fourthiam_extensionarm on the one message whose arms are wall-to-wall credentials is logged until somebody annotates it, and the coverage sweep cannot see it because it fills only annotated fields. It also rebuildsAwsCredentialandGcpCredentialas empty messages, sorole_arnandexternal_id— not credentials, and so plausibly recorded as such in the inventory — would begin appearing. Deleting this one is a deliberate fail-open trade, not a no-op.The descriptor graph has four cyclic components, so the walk is guarded.
TablePartitionMetadata.subpartitions(database_service.proto:936) is the only self-recursive field. The other three are mutual, and a guard keyed onf.Message().FullName() == parent— which is what "self-recursive" invites — misses every one:ObjectSchema↔StructKind↔ArrayKind(database_catalog_service.proto:136,:158,:163).StructKind.propertiesis amap<string, ObjectSchema>, so this cycle runs through the map rule below. Reached fromUpdateDatabaseCatalog, which is audited.google.protobuf.Value↔Struct↔ListValue, reached fromQueryResponse.results[].rows[].values[].value_value— every auditedQueryandAdminExecute.google.api.expr.v1alpha1.Exprand its nested types, reached fromSetIamPolicyRequest.policy.bindings[].parsed_expr.
Three of the four sit on audited paths;
TablePartitionMetadatais the one that does not. An unguarded descend runs until the stack does, inside the interceptor, taking the RPC with it.Termination is the easy half. The harder one is that breaking a cycle by answering "no plan" is provisional: the resulting plan has no descend arm at all for the fields that resolved provisionally, so an annotation inside the cycle is applied at depth 0 and nowhere else, and the coverage sweep still passes because the top-level copy is clean. Declining to cache that answer fixes the poisoning but not the plan, and leaves the cycle head permanently uncacheable. The builder iterates to a fixed point — a requirement, not one of two options. The plan-construction test pins one root per component, not
DiffMetadataRequestalone.Maps do, and they are rebuilt, never shared.
dst.Set(fd, src.Get(fd))on a map field shares the Go map itself and its message values, so clearing a field in an entry writes through to the caller's live message — measured, the source lost the field. The correct form isdst.Mutable(fd).Map()plus a per-entrySetof a redacted copy. An annotated map is dropped whole; a map whose value type has a plan is rebuilt entry by entry.TestAuditRedactionDoesNotMutateInputneeds a fixture holding a map whose value type has a plan — none exists today, so this violation of goal 6 would not be caught.Repeated message fields are rebuilt per element, for the same reason.
Rangeyields a read-only list: assigning it panics, and sharing it cannot produce redacted elements without writing through to the caller's. So a repeated field whose element type has a plan is rebuilt viadst.Mutable(fd).List(), appending a redacted copy per element; one whose element type has no plan is shared whole. This is not a corner: the case that motivates the whole design,Instance.roles[].password, crosses a repeated message field, as doesinstance_resource.data_sources[]. The non-mutation test needs a repeated-message fixture on a redacted path alongside the map one.Every
Anythat reaches the row is registered and redacted. The descriptor walk seestype_urlandvalue, never the packed message, so an annotated field inside one is invisible to the redactor, the coverage lint and the inventory alike. Three paths put anAnyon an audit row and none passesmarshalAuditPayload:source call sites today packed types service_data(audit.go:378)4, via GetSetServiceDataFromContextbefore-image Setting, IAM policy deltasStatus.details(audit.go:376,convertErrToStatus:1482)6, via connect.NewErrorDetailPermissionDeniedDetail,PlanCheckRun_Resultordinary Anyfields2, both on SearchAuditLogsResponse.audit_logsrows already redacted when written Each is unpacked, redacted against the plan for its own descriptor, and re-packed — unless the packed type's plan is nil, in which case it is left exactly as found.
connect'sErrorDetail.Type()returns a bare full name, soStatus.details[].type_urlis stored today asbytebase.v1.PermissionDeniedDetail; an unconditional round-trip throughanypb.Newrewrites it totype.googleapis.com/bytebase.v1.PermissionDeniedDetail, silently changing whatSearchAuditLogsemits as@typefor a message that had nothing to redact. That alone is not enough: the packed type is chosen in Go rather than declared in a descriptor, so a handler that starts packing a new one would neither change the inventory nor fail it, and an unannotated credential inside it would still be written. So permitted types are a checked-in registry, enforced two ways, split by what a descriptor can see.The packed type is chosen in Go on all three paths, so all three need a call-site lint: the two setters' ten sites, plus every site assigning into an ordinary
Anyfield. It fails the build when a site packs a type the registry does not name for that destination.The descriptor adds a second trigger, not a substitute. An ordinary
Anyfield is descriptor-visible even though its packed type is not, so the walk also requires a registry entry perAny-typed field in the audited surface. The two catch different things — the descriptor half catches a new field, the call-site half a new packed type in an existing one — and either alone leaves a hole. Without the descriptor half a new field is governed by nothing; without the call-site half a producer switches an existing field's packed type with descriptor and registry untouched, CI green, and the runtime drops the payload silently. Both are the fail-silent record loss the next paragraph calls a defect.The interceptor also drops an unregistered
Anyrather than logging it — load-bearing rather than a backstop, sinceprotojson.Marshalfails the entire message on an unresolvableAny, so without the drop the whole audit row is lost.The lint is the half that matters. Dropping alone is fail-closed for secrecy but fail-silent for the record:
SetIamPolicy's policy deltas are the interesting part of that audit row, and losing them with no error is its own defect. The registry is also the derivation source for the inventory'sAnyhalf — registering a type is what pulls its fields in for classification.Nothing leaks today, but not for one reason. Two of the four
service_datasites (setting_service.go:174,:658) packconvertToSettingMessageoutput, which blanks its secrets —convertToAISettingandconvertToEmailSettingexplicitly,convertToAppIMSettingby building empty payloads. The other two (project_service.go:621,workspace_service.go:509) packanypb.New(&v1pb.AuditData{PolicyDelta: ...})fromfindIamPolicyDeltas, which is not a converter and blanks nothing; they are safe only becauseBindingDeltahappens to holdaction/role/member/condition. The read-path assertion cannot reach them at all, sinceconvertToProtoAnyreturns an*anypb.Any— soAuditDataandBindingDeltaget classified through the registry rather than assumed clean.PermissionDeniedDetailcarries a method, permissions and resource names;PlanCheckRun_Resultcarries advisory text. All of it is a property of the current call sites rather than an enforced one, andStatus.detailsfires on failed RPCs — exactly when a handler attaches context about what went wrong.Status.messageis a fourth payload, and no annotation can reach it.convertErrToStatussets it fromconnectErr.Message(), orerr.Error()for a non-connect error, onto the row ataudit.go:376, andlogAuditToStdoutprints it asstatus_message. It is a Go string with no descriptor, so goal 1 and the inventory are structurally blind to it. Not hypothetical:idp_service.go:283formats"failed to exchange access token, error: %s"from the oauth2 exchange error, whoseError()embeds the IdP token endpoint's raw response body, and oauth2 sends the client secret withAuthStyleInParams— on the same method whose request is redacted precisely because it carries that secret. Error strings that interpolate a remote response have to be handled where they are built; nothing in this design covers them.Directly assigned structured fields are in the population too.
createAuditLogsetsRequestMetadataandMcpDelegationonto the row besideServiceData(audit.go:379-380); neither passes an entry point nor is anAny. They are ordinarystorepbmessages with descriptors, so the plan runs on them unchanged and they join the inventory rather than needing an exception. This is not theoretical housekeeping:RequestMetadata.caller_supplied_user_agentis caller-controlled, and the store proto says so itself — "not authenticated and should be treated accordingly".
All 35 redactors are deleted, allowlist rebuilds included, and AdminExecute rows regain
rows_count. But "audit rows carry the non-secret remainder" is not uniformly benign, and two
values do not sort all of it.
OMIT has to cover more than bulk as the term is used above. redactMaskingReasons drops
semantic_type_icon "to avoid polluting audit logs with base64 data", and redactAIChatRequest
drops messages and tool_definitions as "an unbounded body". Neither field is in the OMIT set,
so deleting those two rebuilds restores base64 blobs to every Query row and the full AI transcript
to every AIService/Chat denial. Both go in.
The harder residue is content that is neither a credential nor bulk. redactUser returns
{name, email, title} while convertToUser populates phone, groups[] and profile timestamps.
SENSITIVE is unavailable — the read-path assertion would then fail on convertToUser itself — and
OMIT reads as "returning it is the point of the RPC", so unannotated is the path of least
resistance and every audited Login, CreateUser and UpdateUser starts writing a phone number
and group memberships into audit_log and stdout, which have different retention and no purge path.
redactPurchaseResponse drops a Stripe checkout URL, a bearer capability no reviewer classifies as
a credential. Either OMIT is redefined as "must not be recorded, for any reason" — the smaller
change, and what the rest of this design assumes — or a third value is needed. Settling that, and
walking each of the ten rebuilds for what it newly admits, is a precondition for deleting them.
Enforcement
Coverage. For every RPC in the population below, fill every annotated field in the request and response tree with a unique sentinel, redact, assert no sentinel survives. Catches a redactor that stops covering a field. Subsumes
TestAuditRedactsEveryInputOnlyDataSourceField.Oneofs need one message per arm, not one message with every arm set: setting a second arm clears the first, so a single-message sweep silently exercises only the last-numbered arm while every other assertion passes vacuously.
DataSourceExternalSecret.auth_optionwould be tested ontoken(5) and never onapp_role(4), whoserole_idandsecret_idare bothINPUT_ONLY— andapp_roleis the arm the blanking rule above is argued from. This is a cross-product over oneofs, materially more expensive than one pass.Inventory — the half that makes goal 4 real. A sentinel sweep cannot distinguish an unannotated credential from an ordinary field the audit row intentionally keeps; both survive redaction identically, so the sweep alone has no oracle. The oracle is a checked-in inventory of every string and bytes field reachable from a message that can reach
marshalAuditPayload. A field missing from it fails the build; clearing that failure means annotating the field or recording that it is not a credential. Same shape asmcpDenialRequestsUnderReview(mcp_gate_test.go:1322), which it replaces once the population matches.That population is wider than the audited RPCs.
WrapUnarystreams every call a permission check refused, whatever its annotation (audit-what-produces-a-row.md), so the refused methods carrying no audit annotation —ListInstanceDatabaseRequestandSwitchWorkspaceRequestamong them — are in scope; deriving the inventory from audited methods alone would omit exactly the populationTestLintDenialRequestsAreReviewedForRedactionexists to cover. Every registeredAnytype is in scope too, from all three paths in the table above, since those reach the row without passing either entry point. That half comes from the registry rather than from a descriptor walk, which is why the registry has to be enforced at its call sites for the inventory to mean anything. The directly assignedRequestMetadataandMcpDelegationare in scope on the same grounds, and those two do come from a descriptor walk.Read-path assertion.
assertNoInputOnlyValues(instance_service_converter_test.go:449) already requires everyINPUT_ONLYfield to come back blank from a converter; generalize it toSENSITIVE. Runs on every converter in the declared list, minting ones included: a minting converter is not skipped, it is allowed exactly the(function, field)pairs declared for it above, so a later line inside it that populatedpasswordorservice_keystill fails.The completeness lint keys on return type — a function in
backend/api/v1whose returns include av1pbmessage, directly or inside a slice, a map, a*connect.Response[T]wrapper, or any arm of a multi-value return, plus the response type parameter of a streaming handler — not on theconvertTo*name prefix. Streaming needs naming separately becauseAdminExecutereturns onlyerrorand emits throughSend(stream *connect.BidiStream[v1pb.AdminExecuteRequest, v1pb.AdminExecuteResponse],sql_service.go:74), so no return shape selects it — the one handler whose responses carry every row of an admin-mode query would otherwise sit outside the population that the Redaction section above goes out of its way to discuss. The wrapper matters because it is what every RPC handler returns (GetUseris(*connect.Response[v1pb.User], error),user_service.go:53), and a handler that builds its response inline rather than through a converter is otherwise outside the population entirely. Handlers mostly become statically checked entries — they need a store — but that is a checked decision per handler instead of a silent omission, and it is where the minting assignments named above stop being a separate hand-kept list and become ordinary members with declared(function, field)pairs. Containers are not an edge case here:convertDataSourcesreturns[]*v1pb.DataSourceandconvertInstanceRolesreturns[]*v1pb.InstanceRole(instance_service_converter.go:232,:49), and those two are precisely what the Background leans on to callinstance_resource.data_sources[]androles[].passwordsafe on reads. A bare-message predicate omits both, plusconvertToAuditLogsand 17 others — 20 of the 138v1pb-producing functions, including the most load-bearing ones. The prefix also matches 19convertToStore*functions going the other way, which carry credentials by design (convertToStoreInstanceholds every data-source password;convertToStoreProjectWebhookMessagecopiesWebhook.url, the fieldredactWebhookmasks as a bearer credential) and whichassertNoInputOnlyValuescannot run on at all, since it takes aprotoreflect.Messageasserting the v1 contract. Keying on the prefix produces 19 build failures whose only cheap fix is 19 whole-function exemptions — the blanket-exemption failure this design rejects.Return type alone still selects more than converters:
chatWithProvider(ai_service.go:55) makes an outbound provider call,getSession(rollout_service.go:647) takes a live*sql.DB,validateSpecs(plan_service.go:572) takes a*store.Store. A test cannot construct inputs for those generically, and hand-trimming the list back to the ones it can rebuilds the hole.So the lint checks membership, not executability. Every function in the population needs an entry, and an entry is one of two kinds: asserted, where the test seeds a non-empty sentinel into every
SENSITIVEfield the converter can source, then runs the generalizedassertNoInputOnlyValueson the output and requires none to survive; or statically checked, for a function a test cannot execute — needs a store, a live connection, an outbound call.A static entry is not a prose exemption. It carries an AST check that the function body assigns to no
SENSITIVEfield beyond the(function, field)pairs declared for it — the same shape as the minting pairs, and it needs no runtime. A recorded reason accompanies the check; it never stands alone. Prose-only was the first draft of this rule and is wrong for the reason the design gives everywhere else: a whole-function pass is an exemption for everything that later lands inside it. Adding a directSENSITIVEassignment to an already-exempted handler changes neither its return type nor its entry, so the lint would stay green while the reason went stale — most sharply for the inline handlers the wrapper rule above pulls into the population. A newv1pb-producing function fails the build until it is classified as one or the other. That is the shapemcpDenialRequestsUnderReviewalready uses, prose reason per entry included, and it is what makes an exemption here a checked per-field decision rather than a blanket skip.The seeding is what makes an asserted entry fail closed.
assertNoInputOnlyValueswalks populated fields, so a fixture leaving the source at its zero value passes trivially — and would keep passing after someone adds a line copyingpasswordorservice_keyfrom that same zero-valued input, which is the failure this section promises to catch. Unseeded, it is the same vacuity as setting every oneof arm in one message, on the read side.The generalized assertion must also traverse map values, which
assertNoInputOnlyValuesskips today (:485). This is inside the declared converter surface, so leaving it would contradict goal 5 outright rather than sit under a non-goal. Four converter-reachable maps carry message values —ObjectSchema.StructKind.properties,DataClassificationSetting.DataClassificationConfig.classification,SQLEditorThemeSetting.tokensandgoogle.protobuf.Struct.fields— and aSENSITIVEfield inside any of them would reach a read response with CI green. The redactor closes this above; the assertion closes it here.
Performance
| current | proposed | |
|---|---|---|
Instance, 1 data source |
4.1 µs | 5.9 µs |
Instance, 20 data sources |
38 µs | 51 µs |
QueryResponse, 100k rows |
1.3 µs | 1.6 µs |
Sheet, 5 MB |
0.16 ms · 5 MB | 0.001 ms · 0 MB |
BatchCreateSheets, 20 × 1 MB |
0.59 ms · 20 MB | 0.001 ms · 0 MB |
BatchUpdateInstances, 100 × 5 ds |
1.07 ms · 2.0 MB | 1.29 ms · 1.2 MB |
BatchUpdateInstances, 100 × 20 ds |
3.93 ms · 7.8 MB | 4.74 ms · 4.5 MB |
The only cost above a millisecond is the batch pair — +0.22 ms and +0.81 ms on requests carrying 500 and 2000 data sources — against about 40% less memory on the same requests.
The sheet rows are a fix. redactSheet and the two sheet request arms proto.CloneOf the whole
sheet and then null the content, so an audited 20-sheet batch create copies 20 MB to discard it.
Sharing never copies a dropped subtree.
Rejected on measurement: clone-then-clear, which is what a field-mask library gives, costs
114 ms and 87 MB on a 100k-row QueryResponse to produce a 215-byte row. Redacting during
marshal needs a protojson-compatible emitter of our own — protojson.MarshalOptions has no
field-filter hook — and would save only microseconds over sharing.
Plan construction for the whole v1 surface costs under a millisecond, once. Measured on
darwin/arm64, one run, with INPUT_ONLY standing in for the annotation set.
Alternatives considered
field_behavior = INPUT_ONLYanswers an API-contract question, not a secrecy one.ServiceAccount.service_keyisOUTPUT_ONLYand is a live credential; IdPclient_secret, AIapi_keyand Loginpasswordcarry no annotation at all.udpa.annotations.sensitive(CNCF, used by Envoy) merges PII with secrets, and an audit row often should carry identity. Also a newbufdependency for one boolean.- A
MASKvalue alongsideOMIT— there is no mask to model.maskedStringis""and protojson drops it, so every existing "mask" is already an omit.SENSITIVEandOMITdiffer in what the field is, not in how much of it survives; both drop it. - Two bools —
sensitiveon credentials plusaudit_omiton bulk. Rejected; see "Why one enum rather than two bools" above. debug_redactas the credential marker — inert in Go, so it would imply enforcement the runtime does not provide. Same section.- AIP-147 prescribes patterns, not an annotation:
INPUT_ONLYplusOUTPUT_ONLY bool <name>_set. We follow it inconsistently —ssl_ca_setdoes,directory_sync_token_configureddoes not, and that field is in no tagged release yet.
Implementation
- Annotation —
AuditBehaviorandaudit_behavior = 100010inproto/v1/v1/annotation.proto; 77 fields annotated across 17 service protos.OMITis defined as "must not be recorded, for any reason", which is the reading the rest of this design assumes: it carriesUser.phone,groupsandprofile,MaskingReason.semantic_type_iconandAIChatRequest.messagesalongside the bulk content fields, so deleting the ten rebuilds admits nothing new. - Redactor —
backend/api/v1/audit_redact.go. Plan per message type, cached on descriptor identity, withdescendarms holding the sub-descriptor rather than a nested plan so a cycle is followed lazily to any depth; copy-and-share byRange; maps and repeated messages rebuilt;SENSITIVEoneof arms blanked andOMITones dropped. All 35 hand-written redactors are gone. With nothing left to switch on,getRequestStringandgetResponseStringcollapse into onemarshalAuditPayload: the annotation says what to do with a field, and which side of the call it arrived on does not change that. The hook point is unchanged — it is still called fromcreateAuditLog, notWrapUnary, which is what keeps the streaming path covered. The row itself is not walked as a whole:service_dataandstatusare redacted where they are assigned, byredactAuditServiceDataandredactAuditStatus, andrequest_metadataandmcp_delegationare assigned untouched because nothing under them is annotated.TestAuditRowNeedsNoRedactionBeyondTheAnyPayloadsasserts that stays true. Anyregistry —backend/api/v1/audit_redact.go, keyed by destination field rather than by type, soStatus.detailsandservice_datapermit different things.TestLintAuditAnyFieldsAreRegisteredenforces the descriptor half: a newAnyfield reaching the row must name what may be packed in it. The other half — a new type packed into an existing field — changes no descriptor, so it is caught at runtime instead, by theslog.WarnredactPackedAnyemits naming the type it dropped.- Coverage — the redactor owes the row two things, and the tests are split
on that line rather than by message type, and both halves live in
audit_redact_test.go.TestAuditRedactionCoversEveryAnnotatedFieldsweeps 567 annotated field paths across 321 request/response types, one message per field so every oneof arm is exercised on its own rather than only the last-numbered one — that is the whole of "the secret leaves", so the hand-written per-credential tables that used to assert it by example are gone.TestAuditRowKeepsItsSubstancein the same file is the other half and has to stay hand-written: the sweep asserts absence, so a redactor that blanked the entire row would pass it. Mutation-checked in both directions — disabling redaction fails 91 subtests, replacing the output with an empty message fails all 13 survival rows and leaves the sweep green. - Inventory —
audit_redact_inventory_test.goholds the declaration lints, as againstaudit_redact_test.gowhich tests the code: 866 recorded(message, string-or-bytes field)pairs. Annotated fields are absent by design, so the list reads as "what the audit row writes down" and a new credential lands in it as one line of diff. - Read path —
assertNoInputOnlyValuesis generalized toassertNoSensitiveValues(INPUT_ONLY ∪ SENSITIVE, traversing map values) and runs unconditionally on the converters a test can construct. A static call-graph lint over everyv1pb-producing function inbackend/api/v1was prototyped and dropped: the read path is a different surface from the audit row, and a Go AST check that reports a credential leak when a converter is merely renamed is the wrong price to pay inside this change. If we want it, it belongs in its own PR. - Kept, not replaced —
mcpDenialRequestsUnderReviewstays alongside the inventory. It is per(method, field)and covers the message-typed request fields the inventory's scalar scope does not:DiffMetadata.target_metadatais a whole schema, and no string field names it. - Deliberately not fail-open —
AWSCredential.role_arnandexternal_idare annotatedSENSITIVE. They are not credentials, and the Redaction section above notes that deletingredactIAMExtensionwould start recording them; annotating them keeps today's behavior instead of taking that trade silently. - Widened on purpose, recorded here rather than discovered later —
redactDataSourceused to replace the wholeexternal_secretsubmessage with an empty one; only its credential leaves are annotated, soDataSourceExternalSecret'surl,engine_name,secret_name,password_key_name,mount_pathand the type enums now reach the row. Those say WHERE a secret lives, not what it is, and knowing which Vault path an instance was pointed at is the kind of thing the row exists for — but a customer's internal Vault endpoint and secret path are newly readable by anyone withbb.auditLogs.search, so this is a sign-off, not a side effect. QueryResult.messagesisOMIT— the deletedredactQueryResponsewas an allowlist that predated the field, so it dropped it by omission. Engine output (RAISE NOTICE,PRINT,DBMS_OUTPUT) is a result-data channel likerows, and column masking does not reach it.v1.AuditLog.service_dataisOMIT— otherwise SearchAuditLogs' own audit row unpacks, redacts and re-transcribes the before-image of every row the caller read, which is the one place cost scaled with payload size.- A SENSITIVE scalar with explicit presence is blanked, not dropped. The
Redaction section above argues for blanking a oneof arm so the row still
records which credential was supplied; a proto3
optionalcredential is the same question, andLoginRequest.otp_codeis the case that matters — a failed login has to stay distinguishable from one where no second factor was presented. Non-optional scalars are still dropped, which protojson renders identically. - Still uncovered —
Status.message, exactly as the Redaction section says.idp_service.go:283still interpolates an oauth2 error whoseError()can embed the IdP's raw response body. It has no descriptor, so nothing here reaches it; fixing it means deciding what that error should say instead.