Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

fix(bench): one connection owned nothing, and every wall time was that

esengine9fdad91286f5
diff --git a/bench/idle-maintenance/README.md b/bench/idle-maintenance/README.md
new file mode 100644
index 000000000..4481749a4
--- /dev/null
+++ b/bench/idle-maintenance/README.md
@@ -0,0 +1,66 @@
+# What a sample maintains when nothing has happened
+
+A stationary sample at 100,000 entities and 32 connections costs 3,573 µs, and
+`bench/interest-floor` accounts for 3,439 of it in the visibility pass. This asks
+what the OTHER work costs: reconciling the registry against its topology journal,
+refreshing an owner index nobody changed, gating a change journal, reading empty
+removal windows, and giving three kinds of reader a new floor.
+
+The hypothesis it was built to test: that a replication server pays rent per
+REPLICATED COMPONENT rather than per entity, so a wide schema costs a stationary
+server what a busy one costs.
+
+`node bench/idle-maintenance/probe.mjs`
+
+## It costs about a microsecond
+
+| | µs per sample |
+|---|---|
+| R registry topology read | 0.1 |
+| O owner write-journal read | 0.1 |
+| Dc Changed gate, one `anyChangedSince` per component | 0.2 |
+| Dr Removed-history read, one window per component | 0.2 |
+| Ac advance Removed readers, one per component | 0.3 |
+| Ar advance the topology reader | 0.2 |
+| Ao advance the write reader | 0.2 |
+| K ack and bookkeeping | 0.6 |
+
+Timed a second way, inside the real server rather than in a transcription:
+reconcile **0.4 µs**, owner refresh **0.1 µs**, dirty collection **0.8 µs**.
+Both say the same thing.
+
+## The schema-width rent is real and it is a rounding error
+
+Same world, same connections, nothing happening, only the number of replicated
+components moving:
+
+| replicated components | whole idle pass |
+|---|---|
+| 2 | 1.4 µs |
+| 6 | 2.8 µs |
+| 18 | 4.1 µs |
+| 66 | **4.7 µs** |
+
+It grows — and sub-linearly, because a component no entity carries answers its
+gate faster. Sixty-six replicated components cost **4.7 µs against a 3,573 µs
+stationary sample**. There is nothing here to collect.
+
+## The one thing that did cost 108 µs
+
+The first run of this probe reported `R` at 108 µs and `Ar` at 29 µs, reading and
+advancing an EMPTY topology window. Both vanished when the idle server whose
+table it borrowed was disposed first.
+
+A server that never samples never advances its floors, so its claim pins the
+journal at the tick it was created — and a second holder then reads every window
+as if it were the whole run. That is a property worth knowing about the
+reader-owned journals: **an idle claimant makes everyone else's reads expensive**,
+and it is invisible until someone measures with two readers in the room.
+
+## What this does not cover
+
+- The journals are exercised with nothing in them. What a window costs to read
+ when it HOLDS rows is the dirty path, measured in `bench/replication-routing`.
+- One server, one claim per replicated component. A host running several servers
+ against one world would have several, and the min-reader arithmetic behind
+ retention is where that would show.
diff --git a/bench/idle-maintenance/probe.mjs b/bench/idle-maintenance/probe.mjs
new file mode 100644
index 000000000..c7473da15
--- /dev/null
+++ b/bench/idle-maintenance/probe.mjs
@@ -0,0 +1,200 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file What a replication sample maintains when nothing has happened.
+ *
+ * A stationary sample costs 7,334 us at 100k entities and 32 connections. The
+ * visibility pass is 3,439 of it (bench/interest-floor); this is the other
+ * 3,895 — reconciling a registry against an empty topology window, refreshing an
+ * owner index nobody changed, gating a change journal, reading an empty removal
+ * window, and giving three kinds of reader a new floor.
+ *
+ * The suspicion it exists to kill or confirm: that this is rent paid per
+ * REPLICATED COMPONENT rather than per entity, so a wide schema costs a
+ * stationary server the same as a busy one.
+ *
+ * node bench/idle-maintenance/probe.mjs
+ */
+import path from 'node:path';
+import os from 'node:os';
+import { fileURLToPath, pathToFileURL } from 'node:url';
+
+const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..');
+const argv = process.argv.slice(2);
+const flag = (name, fallback) => {
+ const i = argv.indexOf(`--${name}`);
+ return i >= 0 && argv[i + 1] !== undefined ? argv[i + 1] : fallback;
+};
+const ENTITIES = Number(flag('entities', '100000'));
+const EXTRA = Number(flag('components', '0'));
+const SAMPLES = Number(flag('samples', '200'));
+const CONNECTIONS = Number(flag('connections', '32'));
+
+const { loadSdk, sdkIdentity } = await import(pathToFileURL(
+ path.join(ROOT, 'bench/replication-interest/workload.mjs')).href);
+const sdk = await loadSdk(ROOT);
+const wasmDir = process.env.ESENGINE_WASM_DIR ?? path.join(ROOT, 'build', 'wasm', 'web');
+const factory = (await import(pathToFileURL(path.join(wasmDir, 'esengine.js')).href)).default;
+const engine = await factory({ locateFile: (f) => path.join(wasmDir, f) });
+
+sdk.clearUserComponents();
+/** The knob: how wide the replication schema is, beside the builtins. */
+for (let i = 0; i < EXTRA; i++) {
+ sdk.defineComponent(`Wide${i}`, { a: 0, b: 0 }, { replicatedFields: ['a', 'b'] });
+}
+
+const app = sdk.App.new();
+app.connectCpp(new engine.Registry(), engine, { strict: false });
+app.addPlugin(sdk.replicationPlugin);
+const server = app.getResource(sdk.Net).startServer();
+const world = app.world;
+
+const side = Math.ceil(Math.sqrt(ENTITIES));
+const entities = new Array(ENTITIES);
+for (let i = 0; i < ENTITIES; i++) {
+ const e = world.spawn();
+ world.insert(e, sdk.Replicated, { owner: -1 });
+ world.insert(e, sdk.Transform, { position: { x: i % side, y: Math.floor(i / side), z: 0 } });
+ entities[i] = e;
+}
+// The table is built (and the observation claimed) on first access, from every
+// component registered by now — which is what the width knob moves.
+const table = server.table;
+// Its claims are released before anything is measured. A server that never
+// samples never advances its floors, and a second holder pinning history at tick
+// zero makes every window look like the whole run — 108 us to read an empty one.
+server.dispose();
+const known = new Set(entities);
+
+/**
+ * The reader set a server holds, mirrored: one removal claim per replicated
+ * component, one topology claim, one write claim. Claimed the same way, so what
+ * is timed below is the same work on the same journals.
+ */
+const observations = table.entries.map((te) => {
+ world.enableChangeTracking(te.def);
+ return { def: te.def, readerId: world.registerRemovedReaderFrom(te.def, world.getWorldTick() - 1) };
+});
+const registryReader = world.registerTopologyReaderFrom(sdk.Replicated, world.getWorldTick() - 1);
+const ownerReader = world.registerWriteReaderFrom(sdk.Replicated, world.getWorldTick() - 1);
+let floor = world.getWorldTick() - 1;
+
+const conns = Array.from({ length: CONNECTIONS }, (_, id) => ({ id, ready: true, applied: null, ackedSeq: 0 }));
+
+const ns = { R: 0n, O: 0n, Dc: 0n, Dr: 0n, Ac: 0n, Ar: 0n, Ao: 0n, K: 0n };
+const calls = { journalReads: 0, readerAdvances: 0 };
+
+/**
+ * One sample's maintenance, in the order the server does it. `arm` removes a
+ * layer: B stops reading the change and removal journals, C stops giving the
+ * readers a new floor, D does neither and keeps only the connection loop.
+ */
+function maintain(arm, counting) {
+ const t0 = process.hrtime.bigint();
+ let t = process.hrtime.bigint();
+ if (arm !== 'D') {
+ const candidates = world.getTopologyChangedEntitiesSince(sdk.Replicated, floor);
+ for (const e of candidates) void e;
+ if (counting) { ns.R += process.hrtime.bigint() - t; calls.journalReads++; }
+
+ t = process.hrtime.bigint();
+ const written = world.getWrittenEntitiesSince(sdk.Replicated, floor);
+ for (const e of written) void e;
+ if (counting) { ns.O += process.hrtime.bigint() - t; calls.journalReads++; }
+ }
+
+ if (arm === 'A') {
+ t = process.hrtime.bigint();
+ let anyChanged = 0;
+ for (const te of table.entries) {
+ if (world.anyChangedSince(te.def, floor)) {
+ anyChanged++;
+ for (const e of known) if (world.isChangedSince(e, te.def, floor)) anyChanged++;
+ }
+ if (counting) calls.journalReads++;
+ }
+ if (counting) ns.Dc += process.hrtime.bigint() - t;
+
+ t = process.hrtime.bigint();
+ for (const te of table.entries) {
+ const removed = world.getRemovedEntitiesSince(te.def, floor);
+ for (const e of removed) void e;
+ if (counting) calls.journalReads++;
+ }
+ if (counting) ns.Dr += process.hrtime.bigint() - t;
+ void anyChanged;
+ }
+
+ world.advanceTick();
+ const next = world.getWorldTick() - 1;
+ if (arm === 'A' || arm === 'B') {
+ t = process.hrtime.bigint();
+ for (const o of observations) {
+ world.advanceRemovedReader(o.def, o.readerId, next);
+ if (counting) calls.readerAdvances++;
+ }
+ if (counting) ns.Ac += process.hrtime.bigint() - t;
+
+ t = process.hrtime.bigint();
+ world.advanceTopologyReader(sdk.Replicated, registryReader, next);
+ if (counting) { ns.Ar += process.hrtime.bigint() - t; calls.readerAdvances++; }
+
+ t = process.hrtime.bigint();
+ world.advanceWriteReader(sdk.Replicated, ownerReader, next);
+ if (counting) { ns.Ao += process.hrtime.bigint() - t; calls.readerAdvances++; }
+ }
+ floor = next;
+
+ t = process.hrtime.bigint();
+ for (const conn of conns) {
+ if (!conn.ready || !conn.applied || conn.applied.seq <= conn.ackedSeq) continue;
+ conn.ackedSeq = conn.applied.seq;
+ }
+ if (counting) ns.K += process.hrtime.bigint() - t;
+ return Number(process.hrtime.bigint() - t0);
+}
+
+const arms = { A: Infinity, B: Infinity, C: Infinity, D: Infinity };
+for (let tick = 0; tick < SAMPLES + 20; tick++) {
+ const warm = tick < 20;
+ const order = ['A', 'B', 'C', 'D'];
+ for (let r = 0; r < tick % order.length; r++) order.push(order.shift());
+ for (const arm of order) {
+ const dt = maintain(arm, !warm && arm === 'A');
+ if (!warm) arms[arm] = Math.min(arms[arm], dt / 1000);
+ }
+}
+
+const n = SAMPLES;
+const us = (v) => Number(v) / 1000 / n;
+const say = (s = '') => console.log(s);
+const pad = (v, w) => String(v).padStart(w);
+say('');
+say(`=== a sample maintaining nothing (${ENTITIES} entities, ${table.entries.length} replicated`
+ + ` components, ${CONNECTIONS} connections) ===`);
+say('');
+say(' where it goes');
+say(` R registry topology read ${pad(us(ns.R).toFixed(1), 8)} us`);
+say(` O owner write-journal read ${pad(us(ns.O).toFixed(1), 8)} us`);
+say(` Dc Changed gate ${pad(us(ns.Dc).toFixed(1), 8)} us one anyChangedSince per component`);
+say(` Dr Removed-history read ${pad(us(ns.Dr).toFixed(1), 8)} us one window per component`);
+say(` Ac advance Removed readers ${pad(us(ns.Ac).toFixed(1), 8)} us one per component`);
+say(` Ar advance topology reader ${pad(us(ns.Ar).toFixed(1), 8)} us`);
+say(` Ao advance write reader ${pad(us(ns.Ao).toFixed(1), 8)} us`);
+say(` K ack / bookkeeping ${pad(us(ns.K).toFixed(1), 8)} us`);
+say('');
+say(` journal reads per sample ${pad(Math.round(calls.journalReads / n), 8)}`);
+say(` reader advances per sample ${pad(Math.round(calls.readerAdvances / n), 8)}`);
+say('');
+say(' and the counterfactuals (fastest sample)');
+say(' A everything, as the server does it');
+say(' B the change and removal journals not read');
+say(' C and no reader given a new floor');
+say(' D neither read nor advanced: the connection loop alone');
+say('');
+say(` A ${pad(arms.A.toFixed(1), 8)} us`);
+say(` B ${pad(arms.B.toFixed(1), 8)} us A - B = ${(arms.A - arms.B).toFixed(1)} us, the reading`);
+say(` C ${pad(arms.C.toFixed(1), 8)} us B - C = ${(arms.B - arms.C).toFixed(1)} us, the advancing`);
+say(` D ${pad(arms.D.toFixed(1), 8)} us C - D = ${(arms.C - arms.D).toFixed(1)} us, registry and owner`);
+say('');
+say(` ${JSON.stringify({ ...sdkIdentity(ROOT), components: table.entries.length, cpu: os.cpus()[0]?.model })}`);
diff --git a/bench/interest-floor/README.md b/bench/interest-floor/README.md
index a764e833a..d2db28758 100644
--- a/bench/interest-floor/README.md
+++ b/bench/interest-floor/README.md
@@ -1,9 +1,16 @@
# What a sample costs when nothing has happened
-With no dirty row and no removal, a stationary server spends **7,334 µs a sample
-— 44% of one core** at 100,000 entities and 32 connections. Nothing moved,
+With no dirty row and no removal, a stationary server spends **3,573 µs a sample
+— 21% of one core** at 100,000 entities and 32 connections. Nothing moved,
nothing entered anyone's view, nothing left it, and nothing was sent.
+> That figure was 7,334 when this was first written, because the bench it came
+> from had one connection owning nothing, failing open to `'all'`, and being
+> handed the whole population every sample. Nothing in THIS probe was affected —
+> it drives the provider directly with its own consistent keys — but the share it
+> was said to be of has changed: the visibility pass is not 47% of a stationary
+> sample, it is nearly all of one.
+
`node bench/interest-floor/probe.mjs`. Nothing here is installed on a server;
the counterfactual arms remove one layer at a time from a transcription of the
server's visibility pass.
@@ -11,7 +18,8 @@ server's visibility pass.
## Where it goes
The visibility pass — ask the provider, copy the answer, walk it for arrivals,
-walk the previous view for departures — is **3,439 µs**, 47% of the sample.
+walk the previous view for departures — is **3,439 µs**, which is essentially the
+whole stationary sample.
| | µs | |
|---|---|---|
@@ -71,13 +79,16 @@ would leave 82% of the floor on the table.
So the contract to design is one the provider consults BEFORE walking cells, not
one the server consumes after. Nothing is designed here — this measures.
-## And the other half of the sample
+## And the rest of the sample
+
+There is almost none. `bench/idle-maintenance` measures what a sample does
+BESIDES visibility when nothing has happened — reconciling the registry against
+its topology journal, gating the change journal, reading empty removal windows,
+giving three kinds of reader a new floor — and it is **about 1.3 µs**. Timed
+inside the real server: reconcile 0.4, owner refresh 0.1, dirty collection 0.8.
-3,439 µs of 7,334 is the visibility pass. The remaining **3,895 µs is everything
-else a sample does when nothing has happened**: reconciling the registry against
-its topology journal, diffing the shadow, refreshing the owner index, advancing
-three reader floors. None of it is measured here, and at 53% of a stationary
-sample it is the larger unknown of the two.
+The 3,895 µs of "idle maintenance" this section used to claim was the same
+`'all'` connection, counted a second time.
## What this does not cover
diff --git a/bench/replication-routing/README.md b/bench/replication-routing/README.md
index 46ec771ec..5fce70f27 100644
--- a/bench/replication-routing/README.md
+++ b/bench/replication-routing/README.md
@@ -33,24 +33,32 @@ only ever makes a sample slower, and the fastest reproduces to the microsecond
between runs where the mean moves by 5%. Two arms run back to back once differed
by 40% on the mean and not at all on the minimum.
+> **These numbers were re-measured.** The first run of this bench keyed each
+> connection's anchor by the loop counter while connection ids start at one, so
+> the last connection owned nothing, failed open to `'all'`, and was handed a
+> hundred-thousand-entry Set every sample. Every wall time was that Set. The
+> counts, which came from an oracle with its own consistent keys, were never
+> affected. `arm.mjs` now compares the server's own `viewerLinks` against the
+> oracle's total each census and refuses to report if they disagree.
+
## First: the two filters are not the wall
| workload | µs/sample | one core | over the floor |
|---|---|---|---|
-| **nothing moves at all** | 7,690 | 46% | — |
-| floor: 1% movement, nothing else | 14,477 | 87% | — |
-| dirty 0.1% | 17,524 | 105% | +3,047 |
-| **dirty 1%** | 19,343 | 116% | +4,866 |
-| dirty 10% | 33,756 | 203% | +19,279 |
-| dirty 100% | 90,828 | 545% | +76,351 |
-| removals 0.1% | 17,763 | 107% | +3,285 |
-| removals 1% | 18,007 | 108% | +3,530 |
-| removals 10% | 21,587 | 130% | +7,110 |
-| **mixed — 1% dirty, 0.1% removed** | 19,563 | 117% | +5,086 |
-
-At the rates a game actually runs at, **three quarters of the sample is there
-before a single extra dirty row exists**. Routing the mixed workload's rows adds
-26%. The filters only take over past 10% dirty, where they add 19 ms.
+| **nothing moves at all** | 3,573 | 21% | — |
+| floor: 1% movement, nothing else | 8,751 | 53% | — |
+| dirty 0.1% | 11,234 | 67% | +2,483 |
+| **dirty 1%** | 11,662 | 70% | +2,910 |
+| dirty 10% | 19,273 | 116% | +10,522 |
+| dirty 100% | 42,463 | 255% | +33,712 |
+| removals 0.1% | 11,225 | 67% | +2,473 |
+| removals 1% | 12,055 | 72% | +3,303 |
+| removals 10% | 15,697 | 94% | +6,945 |
+| **mixed — 1% dirty, 0.1% removed** | 12,234 | 73% | +3,483 |
+
+At the rates a game actually runs at, **more than two thirds of the sample is
+there before a single extra dirty row exists**. Routing the mixed workload's rows
+adds 28%. The filters only take over past 10% dirty.
## But what they visit is almost all waste
@@ -207,25 +215,25 @@ the mixed workload against the 66,848 visits the sort replaces.
for free. The reverse index is the server's — the projection of `conn.interest`,
maintained from the enters and leaves the visibility pass already produces.
+Measured against the old shape put back under the same (fixed) harness, at 100k
+entities and 32 connections:
+
| workload | before | after |
|---|---|---|
-| floor: 1% movement | 14,477 | 14,105 |
-| dirty 1% | 19,343 | 19,541 |
-| dirty 10% | 33,756 | **30,324** |
-| **dirty 100%** | 90,828 | **66,742** (545% → 400% of a core) |
-| removals 10% | 21,587 | **19,068** |
-| mixed | 19,563 | **17,921** |
-
-Unambiguous where routing was the cost and inside the noise where it was not,
-which is what N3a said would happen: the two filters were 26% of a realistic
-sample, so removing almost all of them moves that sample by under a tenth. The
-floor itself varies by 3% between runs, so nothing smaller than that in the top
-rows is a claim.
-
-The shape of the win is the point rather than its size today. What used to grow
-with `connections x rows` now grows with the smaller of `affected + fanout` and
-`total interest membership` — so a game that dirties a tenth of its world stopped
-paying 352,000 visits a sample to send 3,201 rows.
+| **dirty 100%** | 69,289 | **42,463** (a 39% cut) |
+| dirty 1% | 12,837 | 11,662 |
+| mixed | 12,426 | 12,234 |
+| removals 10% | 15,525 | 15,697 |
+
+Only the saturation case is outside what separate processes vary by. That is not
+a disappointment, it is N3a's decomposition holding: at realistic rates the two
+filters were a quarter of the sample, so removing almost all of them moves the
+whole sample by less than the harness can resolve. What the ROUTING itself costs
+is measured where it can be — same process, alternating arms — in the section
+above: 783 µs to 85 at the mixed workload, 37,434 to 3,091 with everything dirty.
+
+The shape is the result. What used to grow with `connections × rows` now grows
+with the smaller of `affected + fanout` and `total interest membership`.
## What this does not cover
@@ -233,10 +241,12 @@ paying 352,000 visits a sample to send 3,201 rows.
is actually SENT — 598 entries at the mixed point — so it is the floor a
perfect router leaves behind, not part of the wall. Separating it needs the
frame writer, which is not on the SDK's entry.
-- **The still floor is not explained here.** 7,690 µs with nothing moving is
- about 0.14 µs per (connection × visible entity) plus a constant that does not
- follow the population — a quarter of the entities at the same C × V costs
- 3,675. Whatever that constant is, it is not C × D and not C × R.
+- **The still floor is C × V and nothing else.** 3,573 µs with nothing moving,
+ and `µs per (connection × visible entity)` comes out 0.112 to 0.121 across
+ every point of the sweep — a quarter of the population at the same C × V costs
+ the same, and eight connections cost a quarter of thirty-two. There is no
+ population term and no constant. The constant the broken harness appeared to
+ have was the `'all'` connection.
- One anchor per connection, and a uniform grid world. `viewers per entity` is
the number the push/pull choice turns on, and it is a property of how players
are arranged: 1.00 spread across the map, 8.10 packed into a thousandth of it.
diff --git a/bench/replication-routing/arm.mjs b/bench/replication-routing/arm.mjs
index e639133ef..8b50f62c0 100644
--- a/bench/replication-routing/arm.mjs
+++ b/bench/replication-routing/arm.mjs
@@ -77,14 +77,20 @@ for (let i = 0; i < ENTITIES; i++) {
world.insert(e, Health, { hp: 100 });
entities[i] = e;
}
-/** One anchor per connection, spread so a view is a neighbourhood, not a corner. */
+/**
+ * One anchor per connection, spread so a view is a neighbourhood, not a corner.
+ *
+ * @note Keyed by the id the SERVER gave the connection, not the loop counter:
+ * ids start at one, so owning `c` left the last connection owning nothing
+ * and failing open to `'all'`.
+ */
const owned = new Map();
const span = Math.max(1, Math.floor(ENTITIES * CLUSTER));
const stride = Math.max(1, Math.floor(span / CONNECTIONS));
-for (let c = 0; c < CONNECTIONS; c++) {
+for (const [c, client] of clients.entries()) {
const e = entities[(c * stride) % ENTITIES];
- world.set(e, sdk.Replicated, { owner: c });
- owned.set(c, [e]);
+ world.set(e, sdk.Replicated, { owner: client.id });
+ owned.set(client.id, [e]);
}
const MOVE = Math.round(ENTITIES * MOVEMENT);
@@ -154,8 +160,8 @@ function census(made) {
world.ensureTransformsComposed();
const prepared = oracle.prepare({ world, entities, entityCount: entities.length });
const seenBy = new Map();
- for (let c = 0; c < CONNECTIONS; c++) {
- const answer = prepared.query({ connectionId: c, owned: owned.get(c) });
+ for (const [c, anchors] of owned) {
+ const answer = prepared.query({ connectionId: c, owned: anchors });
const visible = answer === 'all' ? new Set(entities) : answer;
totals.visible += visible.size;
for (const e of visible) seenBy.set(e, (seenBy.get(e) ?? 0) + 1);
@@ -177,6 +183,17 @@ function census(made) {
totals.dirtyVisits += CONNECTIONS * made.dirtied.length;
totals.removalVisits += CONNECTIONS * made.removed.length;
totals.samples++;
+
+ // The server's own view against the oracle's. They diverged silently for a
+ // whole campaign, and every wall time was the difference: one connection
+ // owning nothing fails open to 'all' and is handed the whole population.
+ const links = server.viewerLinks;
+ const counted = totals.visible / totals.samples;
+ if (Math.abs(links - counted) > CONNECTIONS) {
+ process.stderr.write(`the server holds ${links} viewer links where the oracle counts `
+ + `${Math.round(counted)} — they are not looking at the same world\n`);
+ process.exit(1);
+ }
}
let ns = 0n;
@@ -202,6 +219,7 @@ const per = (v) => v / totals.samples;
const usPerSample = us / ticks;
process.stdout.write(`${JSON.stringify({
entities: ENTITIES, connections: CONNECTIONS, visible: VISIBLE, movement: MOVEMENT,
+ viewerLinks: server.viewerLinks,
cluster: CLUSTER,
dirty: DIRTY, removals: REMOVALS, samples: totals.samples,
usPerSample, usPerSampleMin: fastest,