Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
Sync Service
Service responsible for syncing collaborative documents between multiple users.
Project Structure
.
├── bebop/ # bebop schema and ts generator for tests & qa
├── build.rs # build script for worker, to generate bebop rust bindings
├── justfile # justfile for running commands
├── src/ # rust worker-rs worker
├── tests/ # e2e tests using miniflare and vitest
└── wrangler.toml # wrangler configuration
Architecture
The sync service leverages Cloudflare's Durable Objects to create session servers or "rooms" for each document. A Durable Object functions as a worker with both in-memory state and persistent storage capabilities.
Using Cloudflare's Durable Objects API, we can dynamically create and request these objects on demand for individual documents. To connect and receive updates from a document, a client establishes a WebSocket connection to /document/:document_id.
When a connection is established, the worker checks if a Durable Object already exists for the document and creates a new one if necessary. Each Durable Object maintains two types of state:
- In-memory state: Contains the loro_document itself, along with metadata about the document's current state
- Persistable state: Stores data that survives even when the in-memory state is cleared
The in-memory state and loro_document are initialized using a loro-snapshot retrieved from the Durable Object's persistent storage. By default, Durable Objects evict in-memory state after 10 seconds of inactivity. This can occur even when connections to the Durable Object exist, as long as there are no WebSocket messages or alarm invocations within that timeframe.
Since generating and importing loro snapshots is computationally expensive, we want to maintain the in-memory state as long as active connections exist. To accomplish this, we implement a "heartbeat" system by scheduling an alarm for 5 seconds into the future, similar to a debounce mechanism. Even without incoming WebSocket messages, this alarm fires regularly, preventing the in-memory state from being evicted. When a client updates a document via a WebSocket message, we apply the update to the in-memory loro document. Whenever the alarm fires, we generate a new loro snapshot and store it in the Durable Object's persistent storage. When a new user connects to the document, we retrieve both the latest snapshot and any pending operations from persistent storage, then send a complete snapshot to the client.
Authentication
document_storage_service will generate a jwt with permissions for the document. This jwt will be passed to the connecting websocket using queryParams.
Unfortunately, query params are the best way to authenticate a websocket connection. Since we use tls, query params should be encrypted in transit.
the sync-service will verify the jwt and ensure that the user has the correct permissions to access the document.
Document grants may also carry signed agent_correlation_id attribution. The
worker preserves it through editor metadata and pending activity batches so
separate code-mode invocations remain distinct, without changing permissions.
HTTP routes use Axum FromRequestParts extractors to authenticate before the
handler runs. Document view/edit/admin routes declare their minimum grant;
atomic state/update routes require an exact signed document grant, and internal
management routes require the shared service credential. Surface reads and
connects require an exact, kind-discriminated surface JWT. WebSocket query tokens
are validated against the requested session before accepting the upgrade.
src/inbound/ owns routing, credential decoding and Cloudflare callbacks;
src/domain/ owns permission/write policy and atomic document-update logic;
src/outbound/ owns persistence, secrets and external clients. The Cloudflare
callback adapter shares one isolate-local session through Rc, preserving the
mutation barrier across HTTP requests, WebSocket messages, alarms and closes.
Socket write policy allows viewers to read, document commenters to change only
comment marks, and editors to write full updates. Surface commenters cannot
write. PeerCommentUpdate requires comment-only validation even on an editor's
connection, so a staged annotation cannot alter document text. Deploy support
for this message before deploying clients that send it; the dev push and
production release workflows enforce this order while building the web app in
parallel. The worker deployment job waits 300 seconds after deployment for
active Durable Objects to adopt the new code.
When the live version was built from the same inputs, recorded in its deployment
message, the job skips migrations and deployment and waits only for what remains
of those 300 seconds.
Manual deployments must preserve the same order and wait at least five minutes
before publishing clients. Keep the workflow wait aligned with any future
code_update_strategy change. Older workers refuse
the unknown variant without acknowledging or applying it. The HTTP API schema
is generated with just openapi; WebSocket messages
continue to use the Bebop schema. Regenerate the HTTP types from
apps/web/src/lib/service-clients with
bun run orval --config orval.config.ts --project syncService. The existing web
transport handles binary payloads and permission tokens; Orval generates shared
models only.
Comment validation configuration
Document commenters authenticate to lexical-service with the worker secret
LEXICAL_SERVICE_AUTH_KEY. Provision it in each deployed environment with the
matching lexical-service INTERNAL_AUTH_KEY through wrangler secret put.
Never store its value in Wrangler vars or source control. Missing or invalid
credentials fail closed. Current clients stage anchors outside the live editor,
wait for acknowledgement, and retain the draft instead of posting a message when
validation fails. Older clients can create the message independently while the
anchor remains unacknowledged in their WAL and local snapshot; another browser
then cannot display that inline comment.
CI and the just deploy-* recipes run check-secrets before migrations or
worker deployment. This checks binding names only, not credential validity;
validate comment-only writes after provisioning or rotating a key. Rejected
validation spans include document.id and op.id for correlation with clients.
Bebop
The bebop schema is defined in ./bebop/schema.bop.
Generating typescript bindings locally
The typescript bindings are primarily used for testing and QA environemnt.
\cd bebop
../../../node_modules/.bin/bebopc build
Use the repository-pinned generator. Copy schema.bop to
packages/collaboration/src/sync-service/generated/schema.bop and run
../../node_modules/.bin/bebopc build from packages/collaboration to update
the web bindings from the same schema. The package's generate-sync-schema
command fetches the deployed schema and is unsuitable for an unreleased change.
Generating rust bindings locally
Rust bindings get automatically generated when building the worker.
The behavior for this is defined in ./build.rs.
worker-build --profile sync-service-release
Generating bindings elsewhere
The sync_service exposes an endpoint /schema which returns the bebop schema.
A consuming client can fetch this schema during build time and generate the corresponding typescript / rust bindings.
Features
By default bebop-owned-all, do-sqlite-snapshot-storage and search-service are enabled. just worker-build
adds create-default-state; alarm-keep-alive is opt-in.
bebop-owned-all: Enables all bebop owned typescreate-default-state: As of writing this there is no mechanism to initialize a document's state from existing state on the client. If a client fetches a document for the first time the worker will create some default state for it. In the future, this feature should be disabled, and we should be inheriting default state from the client.alarm-keep-alive: Retained for compatibility with existing build commands. Active sessions always schedule persistence/keep-alive alarms; this flag no longer changes runtime behavior.just test-no-alarmchecks the legacy feature selection.
Development
We have 3 environments "test", "dev", and "prod". When developing we typically use "testing". This is deployed with npx wrangler deploy, dev with npx wrangler deploy --env dev, and prod with npx wrangler deploy --env prod.
Testing
The tests run in Node.js with Vitest and start the compiled Rust Worker through
Miniflare directly (tests/utils.ts); no Vitest worker-pool plugin is needed.
Use Node.js 22.12+ (22.x), 24.x, or 26+ for Vitest 5. Install dependencies with
npm ci; package-lock.json is the lockfile used by setup and deployment.
Running Locally
Run:
just dev
just dev applies the local D1 migrations before starting Wrangler. If you run Wrangler directly, apply the migrations first:
CI=true npx wrangler d1 migrations apply USER_PEER_MAPPING --local
npx wrangler dev
If you run into build errors related to clang, follow this fix:
Ensure you have llvm installed and available on your PATH. For mac with zsh this looks like:
# install llvm
brew install llvm
# add it to your PATH for zsh
echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc
Running tests locally
# runs only unit tests
cargo test
# runs both unit tests and e2e tests
just test
# runs e2e tests with the alarm-keep-alive feature disabled
just test-no-alarm
when making a new deploy you need to make a cloudflare KV store and D1 database
Atomic document updates
GET /document/:id/state returns JSON {snapshot, revision}.
Add ?shallow=true to return current state without prior edit history; its revision
and subsequent edits are identical to a full snapshot. Both fields use standard
base64: snapshot is a Loro snapshot and revision
is an encoded Loro version vector from that same state. Requests require a signed
Bearer document permission token. An internal API key does not replace this token.
POST /document/:id/update accepts JSON
{expectedRevision, update}, where update is a base64 Loro update exported from
the snapshot revision. Edit or owner access is required; viewers can read only.
The service validates a disposable fork, compares the current revision, and
imports synchronously before its first persistence await, sharing the websocket
operation log and broadcast pipeline. It returns {revision, applied} only after
persistence. A stale revision returns 409 without applying changes. Retrying a
fully applied delta succeeds with applied: false, even after intervening edits.
Sync accepts arbitrary Loro document schemas. It validates the update encoding,
operation/byte limits and causal dependencies, but does not interpret cells,
styles, sheet names or document content types. Spreadsheet validation and
operations live in packages/spreadsheet and the backend AI editing worker.
Binary updates and read snapshots are limited to 4 MiB, revisions to 64 KiB,
and HTTP bodies are bounded while streaming. Signed actor/user attribution is
bounded and stored beside the existing operation log; request bodies cannot
specify attribution.
The existing binary /snapshot endpoint is unchanged.
Content notifications go to DSS's internal
POST /internal/documents/:id/sync-content-updated endpoint. The documents domain
loads the stored file type and publishes document.sync_content_updated, which
lets search choose its supported extractor. Sync no longer infers Markdown from
CRDT roots or labels updates as Markdown. The search-service feature continues
to control these notifications for compatibility with existing build commands.
Deploy DSS first, then roll out Sync and the AI editing worker together.
Content edits collect the editing socket's verified actor, or its user ID for human editors, in an in-memory list. Existing snapshot/search notifications flush distinct editors to DSS; collecting an editor adds no storage writes, network requests, background tasks, or timers to Sync. Anonymous edits, duplicate updates, and idle peers do not add editors.
DSS's independent Activity consumer records one event at the start of editing and refreshes a shared five-minute inactivity window on each subsequent batch. An edit after five quiet minutes starts another session. Timing follows the existing snapshot notification cadence. Redis or notification failures may drop Activity hints; they never block document saving or search extraction. If Redis applies a session refresh before timing out, that session can remain suppressed without an Activity row until five minutes of inactivity. This is an accepted best-effort delivery limit, rather than a reason to retry or emit on every batch.
Deploy DSS before Sync to enable Activity hints immediately. If an older DSS
rejects the optional editors field, Sync retries once without it so search
continues working. Old Sync requests and old Kafka events remain supported.
Each registered peer is recorded in D1 peer_user_map with the token's user and
signed agent actor (NULL for humans); the metadata and blame endpoints return
both, and the web labels agent edits with the agent's name and other AI-peer
edits as <user> (AI).
Active peer queries (include_ai=false) exclude both signed agent actors and
reserved AI peer IDs. Canvas workers use independent full-width random
Loro peer identities across worker isolates; signed actor claims still supply
durable provenance. Cursor names remain untrusted ephemeral presentation.