Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Answer a member lookup that times out at submission with a retryable error

garrytane95bb4d1a596
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.md
index 0768c26da2..402ec25c06 100644
--- a/docs/guides/error-codes.md
+++ b/docs/guides/error-codes.md
@@ -78,7 +78,7 @@ More: [docs/guides/google-connect.md#troubleshooting](../../docs/guides/google-c
| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
|---|---|---|---|---|---|---|
-| The member a credential (access token, OAuth client or stdio binding) is bound to could not be looked up, so the request was refused and nothing was read or written. | Reading the binding failed with a database error. Reading the credential as unbound instead would open every unlabelled page to a labelled-only member, so the request stops, and the same request succeeds once the database answers. | Retry the same request in a few seconds (a write with the same request_id); if it keeps failing, ask the brain operator to run gbrain doctor --json on the brain host. Run: gbrain whoami --json | provider | `gbrain whoami --json` | 1 | yes |
+| The member a credential (access token, OAuth client or stdio binding) is bound to could not be looked up, so the request was refused and nothing was read or written. | Reading the binding failed with a database error, or with a lock or statement timeout or a dropped connection while the request was submitted. Reading the credential as unbound instead would open every unlabelled page to a labelled-only member, so the request stops, and the same request succeeds once the database answers. | Retry the same request in a few seconds (a write with the same request_id); if it keeps failing, ask the brain operator to run gbrain doctor --json on the brain host. Run: gbrain whoami --json | provider | `gbrain whoami --json` | 1 | yes |
More: [docs/guides/write-refusals.md#access_identity_unavailable](../../docs/guides/write-refusals.md#access_identity_unavailable)
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index 3cd631bfd3..717abe4285 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -1047,7 +1047,7 @@ same on a repo-less brain and on a git-backed one.
| <a id="access_member_invalid"></a>`access_member_invalid` | A member or group id given to `gbrain access`, to `--member` on `gbrain auth create`, `rescope` or `register-client`, or to a local `--as <member>` is ill-formed (`ill_formed`) or names no usable member (`unknown`, `disabled`, `removed`, `unknown_group`), or `member add` named an existing member (`exists`). A member's declared email (`--email` on `member add` or `member set`) is not an address (`invalid_email`), another member already declares it (`email_taken`), or `member link --email` found several members declaring it (`email_ambiguous`; the email is unique, so this means a damaged table). Nothing was read or changed: an unknown `--as` is refused before the command runs instead of reading as unbound. | Check the id with `gbrain access member list --all`; add the member with `gbrain access member add <id>` or enable it with `gbrain access member enable <id>`. Ids are 1-64 characters: lower-case letters, digits, `.`, `_` or `-`. An email has one `@`, no spaces and at most 254 characters; clear a taken one with `gbrain access member set <id> --email none`, and link by member id when an email is ambiguous. `member link --email` naming an email no member declares is `not_found`; it never creates a member. |
| <a id="access_member_id_retired"></a>`access_member_id_retired` | `gbrain access member add` or `enable` named an id that `gbrain access member rm --yes` removed. Member ids are never reused, because pages and audit history may still name the removed id. Nothing was created. | Choose a new member id. `gbrain access member disable` is the reversible way to pause a member. |
| <a id="access_member_disabled"></a>`access_member_disabled` | The access token or OAuth client is bound to a member who is disabled, removed or missing, so it was refused at verification (HTTP 401, `invalid_token` with this code in the description). A credential bound to a member reads as that member, so it is refused rather than silently treated as unbound. | Ask the brain operator to run `gbrain access member enable <id>`, or to rebind the credential with `gbrain auth rescope --token <name> --member <id>` (`--member none` unbinds it; an unbound credential sees no page that names readers). |
-| <a id="access_identity_unavailable"></a>`access_identity_unavailable` | The member a credential (access token, OAuth client or stdio binding) is bound to could not be looked up: the database failed the statement. The request was refused and nothing was read or written. Reading the credential as unbound instead would open every unlabelled page to a labelled-only member. | Retry the same request in a few seconds; a write keeps the same `request_id`. `whoami` shows the member the connection reads as once the lookup works again. If it keeps failing, the brain operator runs `gbrain doctor --json` on the brain host. |
+| <a id="access_identity_unavailable"></a>`access_identity_unavailable` | The member a credential (access token, OAuth client or stdio binding) is bound to could not be looked up: the database failed the statement, or timed out or dropped the connection while the request was submitted. The request was refused and nothing was read or written. Reading the credential as unbound instead would open every unlabelled page to a labelled-only member. | Retry the same request in a few seconds; a write keeps the same `request_id`. `whoami` shows the member the connection reads as once the lookup works again. If it keeps failing, the brain operator runs `gbrain doctor --json` on the brain host. |
| <a id="access_rule_invalid"></a>`access_rule_invalid` | `gbrain access rule add` was given a rule whose prefix is not a literal lower-case slug folder ending in `/` (or holds `..` or `*`), whose readers are not member ids or `group:<id>`, or whose prefix another rule already covers; nothing was written. Prefixes match slugs literally, so a pattern or typo would restrict the wrong pages. | Write the prefix as a folder such as `partners/alice-example/` and the readers as `alice-example,group:team-a`; to change who reads a folder that already has a rule, pass that rule's `--id`. |
| <a id="access_policy_invalid"></a>`access_policy_invalid` | A source's committed `.gbrain/access.yml` (or a file given to `gbrain access rule import`) does not validate. The last applied rules stay in force and every prefix the file names reads as nobody for remote callers (the whole source when the file does not parse), so a typo never opens a folder. In a partial clone (`git clone --filter`), a policy file or an earlier version of it that the clone never fetched makes the whole source read as nobody the same way, because sync fetches nothing. | `gbrain access rule list --source <id>` shows each problem with its line. Fix the file, commit it with `gbrain git commit-push`, and run `gbrain sync --source <id> --no-pull`. In a partial clone, run the `git cat-file -p` or `git log -p` the problem names in the checkout, which fetches what is missing, then sync the same way. |
| <a id="access_policy_pending"></a>`access_policy_pending` | A narrowing change to a source's folder rules is still being stamped onto its pages, so pages under the folders it touches read as nobody for remote callers until it finishes (other folders keep the last applied rules). Remote reads carry this as a notice: an empty answer there is not proof of no notes. | Wait and retry. If it stays pending the apply was interrupted: on the brain host run `gbrain access resume --source <id>` (database store) or `gbrain sync --source <id> --no-pull` (git store); `gbrain access rule status --source <id>` shows the pending folders. |
diff --git a/src/core/access/caller.ts b/src/core/access/caller.ts
index 6b2a92ec23..6a0c047a5f 100644
--- a/src/core/access/caller.ts
+++ b/src/core/access/caller.ts
@@ -32,7 +32,7 @@ import { remotePrivacyRelaxed, requestRemotePolicy, type CallerAccess, type Fold
import { ACCESS_ID_RE, identityFor, normalizePrincipal } from './principals.ts';
import { catalogueError } from '../error-catalogue.ts';
import type { Action } from '../agent-output.ts';
-import { OperationError, opError } from '../ops/contract.ts';
+import { markNotAdmitted, OperationError, opError } from '../ops/contract.ts';
import { hostOnlyError } from '../ops/op-fix.ts';
import { currentCallerAccess } from '../search/private-visibility.ts';
import { edgeTrimSql } from '../edge-whitespace.ts';
@@ -68,14 +68,37 @@ function identityUnavailable(): OperationError {
});
}
+/**
+ * Marks a transient failure liveIdentity rethrew unchanged (a non-enumerable property, so the error still reads the
+ * same to the persistence consumer), so the dispatch boundary can tell it from other statements' failures.
+ */
+const TRANSIENT_LOOKUP_FAILURE = Symbol('gbrain.transientIdentityLookupFailure');
+
+/**
+ * At the MCP dispatch boundary: access_identity_unavailable (retryable, not admitted) when `error`, or an error it was
+ * wrapped in, is a transient failure of a member-identity lookup; else null. A request that fails this way at
+ * submission has written nothing, so the same request_id can be sent again. The persistence consumer never sees the
+ * mapping: its lookups keep the original error, which it releases and retries.
+ */
+export function identityLookupUnavailable(error: unknown): OperationError | null {
+ for (let e = error, depth = 0; e && typeof e === 'object' && depth < 8; e = (e as { cause?: unknown }).cause, depth++) {
+ if ((e as { [TRANSIENT_LOOKUP_FAILURE]?: true })[TRANSIENT_LOOKUP_FAILURE] === true) return markNotAdmitted(identityUnavailable());
+ }
+ return null;
+}
+
async function liveIdentity(engine: Pick<BrainEngine, 'executeRaw'>, memberSql: string, param: string): Promise<string[]> {
let row: { id: string; groups: string[] } | undefined;
try {
[row] = await engine.executeRaw<{ id: string; groups: string[] }>(MEMBER_SQL(memberSql), [param]);
} catch (error) {
if (isMissingSchemaObject(error, 'table', MEMBER_TABLES) || isMissingSchemaObject(error, 'column', MEMBER_COLUMN)) return [];
+ if (error && typeof error === 'object' && (transientDatabaseFailure(error) || transientInterruption(error))) {
+ if (Object.isExtensible(error)) Object.defineProperty(error, TRANSIENT_LOOKUP_FAILURE, { value: true });
+ throw error;
+ }
const code = (error as { code?: unknown } | null)?.code;
- if (typeof code !== 'string' || !SQLSTATE_RE.test(code) || transientDatabaseFailure(error) || transientInterruption(error)) throw error;
+ if (typeof code !== 'string' || !SQLSTATE_RE.test(code)) throw error;
throw identityUnavailable();
}
if (!row || normalizePrincipal(`member:${row.id}`) !== `member:${row.id}`) return [];
diff --git a/src/core/error-registry.ts b/src/core/error-registry.ts
index 73d0d8b1bb..9c26f5ab82 100644
--- a/src/core/error-registry.ts
+++ b/src/core/error-registry.ts
@@ -64,7 +64,7 @@ export interface CodeEntry {
export const CODES = {
absent_from_worktree: { class: 'caller', retryable: false, reasons: ['skip_worktree', 'deleted_locally'], summary: "Sync held one file that is committed at HEAD but missing from the working tree (a sparse checkout or a skip-worktree bit, or a local deletion that is not committed); the rest of the source synced.", why: "Sync imports from the working tree, so it cannot import a file that is not on disk, and treating it as deleted would drop a committed file silently. Whether the file comes back or its deletion is committed is the user's choice; classic sync re-screens the hold on every run.", docs: 'docs/guides/write-refusals.md#absent_from_worktree', suggestion: 'Restore the file (git restore -- <path>, or widen the sparse checkout) and the next sync imports it; or commit the deletion with gbrain git commit-push -m <msg> -- <path> and the next sync clears the hold.' },
access_command_failed: { class: 'caller', summary: "Google connect credential error: access command failed.", docs: 'docs/guides/google-connect.md#troubleshooting' },
- access_identity_unavailable: { class: 'retryable', actor: 'provider', summary: "The member a credential (access token, OAuth client or stdio binding) is bound to could not be looked up, so the request was refused and nothing was read or written.", why: "Reading the binding failed with a database error. Reading the credential as unbound instead would open every unlabelled page to a labelled-only member, so the request stops, and the same request succeeds once the database answers.", docs: 'docs/guides/write-refusals.md#access_identity_unavailable', suggestion: 'Retry the same request in a few seconds (a write with the same request_id); if it keeps failing, ask the brain operator to run gbrain doctor --json on the brain host.', fix: { argv: ['gbrain', 'whoami', '--json'], consent: [], actor: 'provider', requires_exclusive: false, why: 'Read-only: after a few seconds whoami reads the binding again and shows the member this connection reads as; then repeat the original request.', verify: { argv: ['gbrain', 'whoami', '--json'] } } },
+ access_identity_unavailable: { class: 'retryable', actor: 'provider', summary: "The member a credential (access token, OAuth client or stdio binding) is bound to could not be looked up, so the request was refused and nothing was read or written.", why: "Reading the binding failed with a database error, or with a lock or statement timeout or a dropped connection while the request was submitted. Reading the credential as unbound instead would open every unlabelled page to a labelled-only member, so the request stops, and the same request succeeds once the database answers.", docs: 'docs/guides/write-refusals.md#access_identity_unavailable', suggestion: 'Retry the same request in a few seconds (a write with the same request_id); if it keeps failing, ask the brain operator to run gbrain doctor --json on the brain host.', fix: { argv: ['gbrain', 'whoami', '--json'], consent: [], actor: 'provider', requires_exclusive: false, why: 'Read-only: after a few seconds whoami reads the binding again and shows the member this connection reads as; then repeat the original request.', verify: { argv: ['gbrain', 'whoami', '--json'] } } },
access_member_disabled: { class: 'host_only', actor: 'user', reasons: ['disabled', 'removed', 'missing', 'malformed'], summary: "The credential (access token, OAuth client or stdio binding) is bound to a member who is disabled, removed or missing, so it was refused at verification and nothing was read.", why: 'A credential bound to a member reads as that member. While the member cannot read, the credential is refused rather than silently treated as unbound.', docs: 'docs/guides/write-refusals.md#access_member_disabled', suggestion: 'Ask the brain operator to enable the member (gbrain access member enable <id>) or rebind the credential (gbrain auth rescope --token <name>|--client <id> --member <id>|none).' },
on_behalf_of_invalid_audience: { class: 'caller', summary: "A shared agent's request named its audience (the X-Gbrain-On-Behalf-Of header or request-level _meta.on_behalf_of) in a form gbrain does not accept: empty, more than 50 entries, an entry that is not a member id, group:<id> or guest, or a header and _meta that name different audiences. Nothing was read.", why: 'The audience decides whose notes the agent may read, so an audience gbrain cannot read exactly is refused instead of guessed.', docs: 'docs/guides/write-refusals.md#on_behalf_of_invalid_audience', suggestion: 'Fix the harness code that sets the audience: 1-50 entries, each a member id, group:<id> or guest, the same in the header and _meta when both are sent.' },
on_behalf_of_unknown_member: { class: 'caller', summary: "A shared agent's audience names a member who is not enabled on this brain, or a group that does not exist, so nothing was read.", why: 'An unknown or disabled entry is refused rather than read as a guest, so a typo or a removed person never quietly changes what the agent reads.', docs: 'docs/guides/write-refusals.md#on_behalf_of_unknown_member', suggestion: 'Name enabled members and existing groups only (gbrain access member list --all, gbrain access group list on the brain host); a person who is not a member joins the audience as guest.' },
diff --git a/src/mcp/dispatch.ts b/src/mcp/dispatch.ts
index 09ba3d70b4..6a0c72a4b3 100644
--- a/src/mcp/dispatch.ts
+++ b/src/mcp/dispatch.ts
@@ -15,7 +15,7 @@ import type { Operation, OperationMeta } from '../core/ops/contract.ts';
import { loadConfig } from '../core/config.ts';
import { withRequestConfigSnapshot, withoutRequestConfigSnapshot } from '../core/config-snapshot.ts';
import { withCallerAccess } from '../core/search/private-visibility.ts';
-import { appliedAccessSummary, folderAccessFor, remoteCallerAccess } from '../core/access/caller.ts';
+import { appliedAccessSummary, folderAccessFor, identityLookupUnavailable, remoteCallerAccess } from '../core/access/caller.ts';
import { audienceInArguments, audienceInArgumentsError, logAudienceAssertion, parseAudience, type AudienceAuditEvent, type AudienceChannels, type AudiencePrincipal } from '../core/access/audience.ts';
import { ACTOR_ARGUMENT, actorNotAttested, EXTERNAL_ID_RE, resolveAttestedActor, type ActorChannels } from '../core/access/actor.ts';
import { attestedReadMode, SCOPE_READ_ACCESS, scopeReadLabelledOnlyRefused, withReadMode, type ReadMode } from '../core/access/scope-read.ts';
@@ -1033,7 +1033,8 @@ export async function dispatchToolCall(
// GBRA-284: an attested call reads its grant minus other members' personal sources, a scope read minus every personal source (access/member-sources.ts).
if (attested?.actor) scoped = await attestedReadScope(engine, resolved, attested.actor.member, readMode === 'scope');
if (opts.enforcedMember) access = { ...access, hidePrivate: true, owner: false, privateReadableSources: [] };
- } catch (e) {
+ } catch (thrown) {
+ const e = identityLookupUnavailable(thrown) ?? thrown;
// GST-359: a call that names a person gets every failure here (a lock timeout reading its link included) in the
// envelope QM reads, never a thrown call; a coded refusal (access_identity_unavailable) is an error result, before
// anything is admitted; anything else keeps the caller's own error path.
@@ -1396,7 +1397,9 @@ async function dispatchResolvedCall(
const over = op.mutating ? null : await frameOverCap(out, { remote: ctx.remote, transport: ctx.transport, engine, config: ctx.config });
if (over) return errorResult(resultTooLargeError(name, safeParams, over), opts, { op: name, mutating: false, idempotent: op.idempotent === true });
return out;
- } catch (e: unknown) {
+ } catch (thrown: unknown) {
+ // A transient failure of the member-identity lookup at submission is the retryable access_identity_unavailable.
+ const e = identityLookupUnavailable(thrown) ?? thrown;
logVerb(false);
if (op.mutating) invalidateHotMemoryForEngine(engine); // a failed write may have committed part of its work
// Agent contract v1 (A1): every failure — OperationError, classified DB
diff --git a/test/access-labelled-only-fail-closed.test.ts b/test/access-labelled-only-fail-closed.test.ts
index f55ccf2447..278d34e05d 100644
--- a/test/access-labelled-only-fail-closed.test.ts
+++ b/test/access-labelled-only-fail-closed.test.ts
@@ -59,6 +59,9 @@ const FAULT = 'XX000';
const DROPPED = '08006';
const LOCK_TIMEOUT = '55P03';
const ONE_FAILURE = 1;
+const SUBMIT_SLUG = 'notes/owner-submit-example';
+const SUBMIT_CANARY = 'canarylofcsubmitx';
+const SUBMIT_REQUEST = '00000000-0000-4000-8000-00000000a1f1';
/** Submission checks the writer's member twice (writerIdentities, writerLabelledOnly); the stored write's checks come after. */
const SUBMISSION_LOOKUPS = 2;
const COMBINED_PROBE = 'AS any_labelled';
@@ -308,6 +311,18 @@ for (const kind of testBackends()) {
expect((await pageRows(engine, OWNER_SLUG))[0]?.compiled_truth).toContain(OWNER_CANARY);
}, SLOW_MS);
+ test('fix 1 red-first (submission): an owner put_page whose submission lookup meets one lock timeout gets the retryable access_identity_unavailable, writes nothing, and commits when sent again with the same request_id', async () => {
+ const auth = await verify(engine, owner.token);
+ const params = { slug: SUBMIT_SLUG, content: note('Owner submit example', [], 'Owner note ' + SUBMIT_CANARY + ' here.'), request_id: SUBMIT_REQUEST };
+ const { value, failed } = await withFailures(engine, MEMBER_LOOKUP, ONE_FAILURE, LOCK_TIMEOUT, () => call(engine, auth, 'put_page', params));
+ expect({ failed, isError: value.isError, code: value.json.code, retryable: value.json.retryable, next: value.json.fix?.next })
+ .toEqual({ failed: ONE_FAILURE, isError: true, code: 'access_identity_unavailable', retryable: true, next: 'wait' });
+ expect(await pageRows(engine, SUBMIT_SLUG)).toEqual([]);
+ const again = await call(engine, auth, 'put_page', params);
+ expect({ isError: again.isError, code: again.json.code ?? null }).toEqual({ isError: false, code: null });
+ expect((await pageRows(engine, SUBMIT_SLUG))[0]?.compiled_truth).toContain(SUBMIT_CANARY);
+ }, SLOW_MS);
+
test('fix 1 red-first (stored audience and attested actor): a write for a labelled-only member stays labelled-only when re-resolving who it writes for fails', async () => {
const agent = await mint(engine, 'audience-agent');
await bindCredential(engine, { kind: 'token', id: agent.id }, { mayActFor: ['member:' + BOT] });