Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

13.6 KB
Raw Blame

Sync Service

Service responsible for syncing collaborative documents between multiple users.

Project Structure

.
├── bebop/ # bebop schema and ts generator for tests & qa
├── build.rs # build script for worker, to generate bebop rust bindings
├── justfile # justfile for running commands
├── src/ # rust worker-rs worker
├── tests/ # e2e tests using miniflare and vitest
└── wrangler.toml # wrangler configuration

Architecture

The sync service leverages Cloudflare's Durable Objects to create session servers or "rooms" for each document. A Durable Object functions as a worker with both in-memory state and persistent storage capabilities.

Using Cloudflare's Durable Objects API, we can dynamically create and request these objects on demand for individual documents. To connect and receive updates from a document, a client establishes a WebSocket connection to /document/:document_id.

When a connection is established, the worker checks if a Durable Object already exists for the document and creates a new one if necessary. Each Durable Object maintains two types of state:

  • In-memory state: Contains the loro_document itself, along with metadata about the document's current state
  • Persistable state: Stores data that survives even when the in-memory state is cleared

The in-memory state and loro_document are initialized using a loro-snapshot retrieved from the Durable Object's persistent storage. By default, Durable Objects evict in-memory state after 10 seconds of inactivity. This can occur even when connections to the Durable Object exist, as long as there are no WebSocket messages or alarm invocations within that timeframe.

Since generating and importing loro snapshots is computationally expensive, we want to maintain the in-memory state as long as active connections exist. To accomplish this, we implement a "heartbeat" system by scheduling an alarm for 5 seconds into the future, similar to a debounce mechanism. Even without incoming WebSocket messages, this alarm fires regularly, preventing the in-memory state from being evicted. When a client updates a document via a WebSocket message, we apply the update to the in-memory loro document. Whenever the alarm fires, we generate a new loro snapshot and store it in the Durable Object's persistent storage. When a new user connects to the document, we retrieve both the latest snapshot and any pending operations from persistent storage, then send a complete snapshot to the client.

Authentication

document_storage_service will generate a jwt with permissions for the document. This jwt will be passed to the connecting websocket using queryParams.
Unfortunately, query params are the best way to authenticate a websocket connection. Since we use tls, query params should be encrypted in transit. the sync-service will verify the jwt and ensure that the user has the correct permissions to access the document.

Document grants may also carry signed agent_correlation_id attribution. The worker preserves it through editor metadata and pending activity batches so separate code-mode invocations remain distinct, without changing permissions.

HTTP routes use Axum FromRequestParts extractors to authenticate before the handler runs. Document view/edit/admin routes declare their minimum grant; atomic state/update routes require an exact signed document grant, and internal management routes require the shared service credential. Surface reads and connects require an exact, kind-discriminated surface JWT. WebSocket query tokens are validated against the requested session before accepting the upgrade.

src/inbound/ owns routing, credential decoding and Cloudflare callbacks; src/domain/ owns permission/write policy and atomic document-update logic; src/outbound/ owns persistence, secrets and external clients. The Cloudflare callback adapter shares one isolate-local session through Rc, preserving the mutation barrier across HTTP requests, WebSocket messages, alarms and closes.

Socket write policy allows viewers to read, document commenters to change only comment marks, and editors to write full updates. Surface commenters cannot write. PeerCommentUpdate requires comment-only validation even on an editor's connection, so a staged annotation cannot alter document text. Deploy support for this message before deploying clients that send it; the dev push and production release workflows enforce this order while building the web app in parallel. The worker deployment job waits 300 seconds after deployment for active Durable Objects to adopt the new code. When the live version was built from the same inputs, recorded in its deployment message, the job skips migrations and deployment and waits only for what remains of those 300 seconds. Manual deployments must preserve the same order and wait at least five minutes before publishing clients. Keep the workflow wait aligned with any future code_update_strategy change. Older workers refuse the unknown variant without acknowledging or applying it. The HTTP API schema is generated with just openapi; WebSocket messages continue to use the Bebop schema. Regenerate the HTTP types from apps/web/src/lib/service-clients with bun run orval --config orval.config.ts --project syncService. The existing web transport handles binary payloads and permission tokens; Orval generates shared models only.

Comment validation configuration

Document commenters authenticate to lexical-service with the worker secret LEXICAL_SERVICE_AUTH_KEY. Provision it in each deployed environment with the matching lexical-service INTERNAL_AUTH_KEY through wrangler secret put. Never store its value in Wrangler vars or source control. Missing or invalid credentials fail closed. Current clients stage anchors outside the live editor, wait for acknowledgement, and retain the draft instead of posting a message when validation fails. Older clients can create the message independently while the anchor remains unacknowledged in their WAL and local snapshot; another browser then cannot display that inline comment.

CI and the just deploy-* recipes run check-secrets before migrations or worker deployment. This checks binding names only, not credential validity; validate comment-only writes after provisioning or rotating a key. Rejected validation spans include document.id and op.id for correlation with clients.

Bebop

The bebop schema is defined in ./bebop/schema.bop.

Generating typescript bindings locally

The typescript bindings are primarily used for testing and QA environemnt.

\cd bebop
../../../node_modules/.bin/bebopc build

Use the repository-pinned generator. Copy schema.bop to packages/collaboration/src/sync-service/generated/schema.bop and run ../../node_modules/.bin/bebopc build from packages/collaboration to update the web bindings from the same schema. The package's generate-sync-schema command fetches the deployed schema and is unsuitable for an unreleased change.

Generating rust bindings locally

Rust bindings get automatically generated when building the worker. The behavior for this is defined in ./build.rs.

worker-build --profile sync-service-release

Generating bindings elsewhere

The sync_service exposes an endpoint /schema which returns the bebop schema. A consuming client can fetch this schema during build time and generate the corresponding typescript / rust bindings.

Features

By default bebop-owned-all, do-sqlite-snapshot-storage and search-service are enabled. just worker-build adds create-default-state; alarm-keep-alive is opt-in.

  • bebop-owned-all: Enables all bebop owned types
  • create-default-state: As of writing this there is no mechanism to initialize a document's state from existing state on the client. If a client fetches a document for the first time the worker will create some default state for it. In the future, this feature should be disabled, and we should be inheriting default state from the client.
  • alarm-keep-alive: Retained for compatibility with existing build commands. Active sessions always schedule persistence/keep-alive alarms; this flag no longer changes runtime behavior. just test-no-alarm checks the legacy feature selection.

Development

We have 3 environments "test", "dev", and "prod". When developing we typically use "testing". This is deployed with npx wrangler deploy, dev with npx wrangler deploy --env dev, and prod with npx wrangler deploy --env prod.

Testing

The tests run in Node.js with Vitest and start the compiled Rust Worker through Miniflare directly (tests/utils.ts); no Vitest worker-pool plugin is needed. Use Node.js 22.12+ (22.x), 24.x, or 26+ for Vitest 5. Install dependencies with npm ci; package-lock.json is the lockfile used by setup and deployment.

Running Locally

Run:

just dev

just dev applies the local D1 migrations before starting Wrangler. If you run Wrangler directly, apply the migrations first:

CI=true npx wrangler d1 migrations apply USER_PEER_MAPPING --local
npx wrangler dev

If you run into build errors related to clang, follow this fix:

Ensure you have llvm installed and available on your PATH. For mac with zsh this looks like:

# install llvm
brew install llvm
# add it to your PATH for zsh
echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc

Running tests locally

# runs only unit tests
cargo test

# runs both unit tests and e2e tests
just test

# runs e2e tests with the alarm-keep-alive feature disabled
just test-no-alarm

when making a new deploy you need to make a cloudflare KV store and D1 database

Atomic document updates

GET /document/:id/state returns JSON {snapshot, revision}. Add ?shallow=true to return current state without prior edit history; its revision and subsequent edits are identical to a full snapshot. Both fields use standard base64: snapshot is a Loro snapshot and revision is an encoded Loro version vector from that same state. Requests require a signed Bearer document permission token. An internal API key does not replace this token.

POST /document/:id/update accepts JSON {expectedRevision, update}, where update is a base64 Loro update exported from the snapshot revision. Edit or owner access is required; viewers can read only. The service validates a disposable fork, compares the current revision, and imports synchronously before its first persistence await, sharing the websocket operation log and broadcast pipeline. It returns {revision, applied} only after persistence. A stale revision returns 409 without applying changes. Retrying a fully applied delta succeeds with applied: false, even after intervening edits.

Sync accepts arbitrary Loro document schemas. It validates the update encoding, operation/byte limits and causal dependencies, but does not interpret cells, styles, sheet names or document content types. Spreadsheet validation and operations live in packages/spreadsheet and the backend AI editing worker. Binary updates and read snapshots are limited to 4 MiB, revisions to 64 KiB, and HTTP bodies are bounded while streaming. Signed actor/user attribution is bounded and stored beside the existing operation log; request bodies cannot specify attribution.

The existing binary /snapshot endpoint is unchanged.

Content notifications go to DSS's internal POST /internal/documents/:id/sync-content-updated endpoint. The documents domain loads the stored file type and publishes document.sync_content_updated, which lets search choose its supported extractor. Sync no longer infers Markdown from CRDT roots or labels updates as Markdown. The search-service feature continues to control these notifications for compatibility with existing build commands.

Deploy DSS first, then roll out Sync and the AI editing worker together.

Content edits collect the editing socket's verified actor, or its user ID for human editors, in an in-memory list. Existing snapshot/search notifications flush distinct editors to DSS; collecting an editor adds no storage writes, network requests, background tasks, or timers to Sync. Anonymous edits, duplicate updates, and idle peers do not add editors.

DSS's independent Activity consumer records one event at the start of editing and refreshes a shared five-minute inactivity window on each subsequent batch. An edit after five quiet minutes starts another session. Timing follows the existing snapshot notification cadence. Redis or notification failures may drop Activity hints; they never block document saving or search extraction. If Redis applies a session refresh before timing out, that session can remain suppressed without an Activity row until five minutes of inactivity. This is an accepted best-effort delivery limit, rather than a reason to retry or emit on every batch.

Deploy DSS before Sync to enable Activity hints immediately. If an older DSS rejects the optional editors field, Sync retries once without it so search continues working. Old Sync requests and old Kafka events remain supported.

Each registered peer is recorded in D1 peer_user_map with the token's user and signed agent actor (NULL for humans); the metadata and blame endpoints return both, and the web labels agent edits with the agent's name and other AI-peer edits as <user> (AI).

Active peer queries (include_ai=false) exclude both signed agent actors and reserved AI peer IDs. Canvas workers use independent full-width random Loro peer identities across worker isolates; signed actor claims still supply durable provenance. Cursor names remain untrusted ephemeral presentation.