Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

fix(ingest): flush buffers at full speed and keep pre-worker events (#560)

Carl-Gerhard Lindesvärd7f6f7f9e2b97
diff --git a/apps/public/content/docs/self-hosting/environment-variables.mdx b/apps/public/content/docs/self-hosting/environment-variables.mdx
index 65c0f212c..0515ab759 100644
--- a/apps/public/content/docs/self-hosting/environment-variables.mdx
+++ b/apps/public/content/docs/self-hosting/environment-variables.mdx
@@ -46,19 +46,6 @@ Direct PostgreSQL connection string (bypasses connection pooling). Used for migr
DATABASE_URL_DIRECT=postgres://user:password@localhost:5432/openpanel?schema=public
```
-### DATABASE_URL_REPLICA
-
-**Type**: `string`
-**Required**: No
-**Default**: Same as `DATABASE_URL`
-
-Read replica connection string for read-heavy operations. If not set, uses the main database.
-
-**Example**:
-```bash
-DATABASE_URL_REPLICA=postgres://user:password@replica-host:5432/openpanel?schema=public
-```
-
### REDIS_URL
**Type**: `string`
@@ -725,19 +712,6 @@ SELF_HOSTED=true
## Worker & Queue
-### WORKER_PORT
-
-**Type**: `number`
-**Required**: No
-**Default**: `3000`
-
-Port for the worker service to listen on.
-
-**Example**:
-```bash
-WORKER_PORT=3000
-```
-
### DISABLE_BULLBOARD
**Type**: `boolean`
@@ -795,17 +769,17 @@ ENABLED_QUEUES=events,sessions,cron
`events` is the Kafka-based events consumer — there is only one events transport now, so there is no separate `events_kafka` queue name. If you're upgrading from a version that used `ENABLED_QUEUES=events_kafka`, rename it to `ENABLED_QUEUES=events`; the old name is now rejected at boot. See the [Event Streaming (Kafka)](#event-streaming-kafka) section for the full configuration. A common high-throughput layout is to run one worker pool with `ENABLED_QUEUES=events` and another pool with everything else.
</Callout>
-### EVENT_JOB_CONCURRENCY
+### CRON_CONCURRENCY
**Type**: `number`
**Required**: No
-**Default**: `10`
+**Default**: `4`
-Number of concurrent event processing jobs per worker.
+Number of scheduled jobs a worker runs at the same time: buffer flushes, the session reaper and the other cron jobs. Each buffer flush holds its own lock, so raising this lets different buffers flush in parallel without two workers flushing the same one. Raise it if one buffer's backlog keeps growing while the others drain.
**Example**:
```bash
-EVENT_JOB_CONCURRENCY=20
+CRON_CONCURRENCY=6
```
### GSC_CONCURRENCY
@@ -821,32 +795,6 @@ Number of [Google Search Console](/docs/self-hosting/google-search-console) sync
GSC_CONCURRENCY=2
```
-### EVENT_BLOCKING_TIMEOUT_SEC
-
-**Type**: `number`
-**Required**: No
-**Default**: `1`
-
-Blocking timeout in seconds for event queue workers.
-
-**Example**:
-```bash
-EVENT_BLOCKING_TIMEOUT_SEC=2
-```
-
-### EVENTS_GROUP_QUEUES_SHARDS
-
-**Type**: `number`
-**Required**: No
-**Default**: `1`
-
-Number of shards for the events group queue. Increase for better performance with high event volume.
-
-**Example**:
-```bash
-EVENTS_GROUP_QUEUES_SHARDS=4
-```
-
### QUEUE_CLUSTER
**Type**: `boolean`
@@ -855,62 +803,13 @@ EVENTS_GROUP_QUEUES_SHARDS=4
Enable Redis cluster mode for queues. When enabled, queue names are wrapped with `{}` for Redis cluster sharding.
-**Example**:
-```bash
-QUEUE_CLUSTER=true
-```
-
-### ORDERING_DELAY_MS
-
-**Type**: `number`
-**Required**: No
-**Default**: `100`
-
-Delay in milliseconds to hold events for correct ordering when events arrive out of order.
+Set it before the first start. Turning it on later renames every queue key, and jobs already queued under the old names are never processed.
**Example**:
```bash
-ORDERING_DELAY_MS=200
-```
-
-<Callout type="warn">
-Should not exceed 500ms. Higher values may cause delays in event processing.
-</Callout>
-
-### AUTO_BATCH_MAX_WAIT_MS
-
-**Type**: `number`
-**Required**: No
-**Default**: `0` (disabled)
-
-Maximum wait time in milliseconds for auto-batching events. Experimental feature.
-
-**Example**:
-```bash
-AUTO_BATCH_MAX_WAIT_MS=100
-```
-
-<Callout type="warn">
-⚠️ **Experimental**: This feature is experimental and not used in production. Do not use unless you have a good understanding of the implications and specific performance requirements.
-</Callout>
-
-### AUTO_BATCH_SIZE
-
-**Type**: `number`
-**Required**: No
-**Default**: `0` (disabled)
-
-Batch size for auto-batching events. Experimental feature.
-
-**Example**:
-```bash
-AUTO_BATCH_SIZE=100
+QUEUE_CLUSTER=true
```
-<Callout type="warn">
-⚠️ **Experimental**: This feature is experimental and not used in production. Do not use unless you have a good understanding of the implications and specific performance requirements.
-</Callout>
-
## Event Streaming (Kafka)
OpenPanel can optionally ingest events through a Kafka-compatible broker (Kafka itself, Redpanda, etc.) instead of the default Redis-backed events queue. This is useful at high throughput, where many consumer instances can process partitions in parallel.
@@ -1068,6 +967,8 @@ Set this once, before the first `openpanel up`. The Kafka/Redpanda partitioner i
Kafka consumer group ID used by the worker. All worker replicas with the same group ID share partition assignments cooperatively.
+Changing it on an existing install starts a new group, which reads the events topic from the oldest message it still retains (48 hours on the bundled Redpanda). Expect duplicate events for that window.
+
**Example**:
```bash
KAFKA_CONSUMER_GROUP=openpanel-events
@@ -1183,26 +1084,26 @@ KAFKA_REBALANCE_TIMEOUT_MS=60000
**Type**: `number`
**Required**: No
-**Default**: Buffer-specific default
+**Default**: `10000`
-Batch size for session buffer operations.
+Session rows read from Redis per batch. Rows for the same session are squashed within a batch, so larger batches insert fewer rows.
**Example**:
```bash
-SESSION_BUFFER_BATCH_SIZE=5000
+SESSION_BUFFER_BATCH_SIZE=20000
```
### SESSION_BUFFER_CHUNK_SIZE
**Type**: `number`
**Required**: No
-**Default**: Buffer-specific default
+**Default**: Same as `SESSION_BUFFER_BATCH_SIZE`
-Chunk size for session buffer operations.
+Session rows per ClickHouse insert. By default each batch is one insert. A smaller value splits the batch into chunks inserted in parallel, but if one chunk fails the whole batch is retried and the chunks that already landed are written twice, which corrupts those sessions. Leave it unset unless a single insert is too large for your ClickHouse.
**Example**:
```bash
-SESSION_BUFFER_CHUNK_SIZE=1000
+SESSION_BUFFER_CHUNK_SIZE=10000
```
### EVENT_BUFFER_BATCH_SIZE
@@ -1270,6 +1171,19 @@ Time-to-live in seconds for profile buffer entries.
PROFILE_BUFFER_TTL_IN_SECONDS=3600
```
+### BUFFER_CH_INSERT_CONCURRENCY
+
+**Type**: `number`
+**Required**: No
+**Default**: `5`
+
+How many ClickHouse inserts one buffer flush runs in parallel when it splits a batch into several chunks (`*_BUFFER_CHUNK_SIZE` rows each).
+
+**Example**:
+```bash
+BUFFER_CH_INSERT_CONCURRENCY=8
+```
+
### BOT_BUFFER_BATCH_SIZE
**Type**: `number`
@@ -1382,17 +1296,17 @@ IP_HEADER_ORDER=cf-connecting-ip,x-real-ip,x-forwarded-for
The default order includes: `openpanel-client-ip`, `cf-connecting-ip`, `true-client-ip`, `x-client-ip`, `x-forwarded-for`, `x-real-ip`, and others. See the [source code](https://github.com/Openpanel-dev/openpanel/blob/main/packages/common/server/get-client-ip.ts) for the complete default list.
</Callout>
-### SHUTDOWN_GRACE_PERIOD_MS
+### SHUTDOWN_FORCE_EXIT_MS
**Type**: `number`
**Required**: No
-**Default**: `5000`
+**Default**: `20000`
-Grace period in milliseconds for graceful shutdown of services.
+How long, in milliseconds, a graceful shutdown may take before the process exits anyway with code 1. A shutdown stops consuming, writes buffered events to Redis and lets running scheduled jobs finish. Keep it below your orchestrator's kill timeout so the process can exit on its own.
**Example**:
```bash
-SHUTDOWN_GRACE_PERIOD_MS=10000
+SHUTDOWN_FORCE_EXIT_MS=30000
```
### API_PORT
@@ -1401,7 +1315,7 @@ SHUTDOWN_GRACE_PERIOD_MS=10000
**Required**: No
**Default**: `3000`
-Port for the API service to listen on.
+Port for the API service to listen on. Workers listen on the same port for their health checks and `/metrics`.
**Example**:
```bash
diff --git a/apps/public/content/docs/self-hosting/high-volume.mdx b/apps/public/content/docs/self-hosting/high-volume.mdx
index e162cf31d..803c86c51 100644
--- a/apps/public/content/docs/self-hosting/high-volume.mdx
+++ b/apps/public/content/docs/self-hosting/high-volume.mdx
@@ -105,16 +105,16 @@ Events, sessions, and profiles flow through in-memory Redis buffers before being
### Event buffer
-The event buffer collects incoming events in Redis and flushes them to ClickHouse on a cron schedule.
+The event buffer collects incoming events in Redis and flushes them to ClickHouse on a cron schedule. Each flush keeps reading batches for up to 8 seconds, until the buffer is empty.
| Variable | Default | What it controls |
|---|---|---|
-| `EVENT_BUFFER_BATCH_SIZE` | `4000` | How many events are read from Redis and sent to ClickHouse per flush |
+| `EVENT_BUFFER_BATCH_SIZE` | `4000` | How many events are read from Redis and sent to ClickHouse per batch |
| `EVENT_BUFFER_CHUNK_SIZE` | `1000` | How many events are sent in a single ClickHouse insert call |
| `EVENT_BUFFER_MICRO_BATCH_MS` | `10` | How long (ms) to accumulate events in memory before writing to Redis |
| `EVENT_BUFFER_MICRO_BATCH_SIZE` | `100` | Max events to accumulate before forcing a Redis write |
-For high throughput, increase `EVENT_BUFFER_BATCH_SIZE` so each flush processes more events. Keep `EVENT_BUFFER_CHUNK_SIZE` at or below `EVENT_BUFFER_BATCH_SIZE`.
+For high throughput, increase `EVENT_BUFFER_BATCH_SIZE` so each batch carries more events. Keep `EVENT_BUFFER_CHUNK_SIZE` at or below `EVENT_BUFFER_BATCH_SIZE`.
```bash
EVENT_BUFFER_BATCH_SIZE=10000
@@ -127,12 +127,13 @@ Sessions are updated on each event and flushed to ClickHouse separately.
| Variable | Default | What it controls |
|---|---|---|
-| `SESSION_BUFFER_BATCH_SIZE` | `1000` | Events read per flush |
-| `SESSION_BUFFER_CHUNK_SIZE` | `1000` | Events per ClickHouse insert |
+| `SESSION_BUFFER_BATCH_SIZE` | `10000` | Session rows read per batch |
+| `SESSION_BUFFER_CHUNK_SIZE` | same as the batch size | Session rows per ClickHouse insert |
+
+Leave `SESSION_BUFFER_CHUNK_SIZE` unset so each batch is written in a single insert. With smaller chunks, a chunk that fails sends the whole batch back for a retry, and the chunks that had already landed are written twice, which corrupts those sessions.
```bash
-SESSION_BUFFER_BATCH_SIZE=5000
-SESSION_BUFFER_CHUNK_SIZE=2000
+SESSION_BUFFER_BATCH_SIZE=20000
```
### Profile buffer
@@ -156,23 +157,16 @@ PROFILE_BUFFER_CHUNK_SIZE=1000
## Scaling ingestion
-If the event queue is growing faster than workers can drain it, you have a few options.
-
-Start vertical before going horizontal. Each worker replica adds overhead: more Redis connections, more ClickHouse connections, more memory. Increasing concurrency on an existing replica is almost always cheaper and more effective than adding another one.
+Events take two hops before they reach ClickHouse. The API writes each event to Kafka (the bundled Redpanda service). Workers consume it, update the visitor's session and profile, and append rows to Redis buffers. Scheduled flush jobs then move those buffers into ClickHouse in batches.
-### Increase job concurrency (do this first)
+The two hops scale differently, so check which one is behind before changing anything:
-Each worker processes multiple jobs in parallel. The default is `10` per replica.
+- **Kafka consumer lag is growing:** workers are not consuming fast enough. Add worker replicas.
+- **Lag is flat but the Redis buffers keep growing:** consuming keeps up and flushing does not. Let more flushes run at once, raise the batch sizes above, or give ClickHouse more resources.
-```bash
-EVENT_JOB_CONCURRENCY=20
-```
+### Add worker replicas
-Raise this in steps and watch your queue depth. The limit is memory, not logic — values of `500`, `1000`, or even `2000+` are possible on hardware with enough RAM. Each concurrent job holds event data in memory, so monitor usage as you increase the value. Only add more replicas once concurrency alone stops helping.
-
-### Add more worker replicas
-
-If you have maxed out concurrency and the queue is still falling behind, add more replicas.
+Each partition of the events topic is consumed by one worker at a time. Adding replicas spreads the partitions across more workers, up to the partition count (24 by default). Replicas beyond that get no partitions to consume, though they still run flush and other scheduled jobs. Within one worker, `KAFKA_PARTITIONS_CONCURRENT` (default `8`) sets how many of its partitions it processes at the same time.
In `docker-compose.yml`:
@@ -188,34 +182,29 @@ Or at runtime:
docker compose up -d --scale op-worker=8
```
-### Shard the events queue
+The partition count is fixed when the topic is created. Raising it later means draining and recreating the topic, so pick it at install time. See [`KAFKA_EVENTS_TOPIC_PARTITIONS`](/docs/self-hosting/environment-variables#kafka_events_topic_partitions).
-<Callout type="warn">
-**Experimental.** Queue sharding requires either a Redis Cluster or Dragonfly. Dragonfly has seen minimal testing and Redis Cluster has not been tested at all. Do not use this in production without validating it in your environment first.
-</Callout>
+### Let buffers flush in parallel
-Redis is single-threaded, so a single queue instance can become the bottleneck at very high event rates. Queue sharding works around this by splitting the queue across multiple independent shards. Each shard can be backed by its own Redis instance, so the throughput scales with the number of instances rather than being capped by one core.
+Each buffer (events, sessions, profiles and a few smaller ones) is flushed by one worker at a time, so two workers never write the same rows. A flush keeps reading batches for up to 8 seconds, then hands over to the next scheduled run.
-Events are distributed across shards by project ID, so ordering within a project is preserved.
+Different buffers flush at the same time. `CRON_CONCURRENCY` (default `4`) sets how many scheduled jobs one worker runs at once, and every worker replica adds that many slots. If one buffer's backlog keeps growing while the others drain, raise it. Inside a single flush, `BUFFER_CH_INSERT_CONCURRENCY` (default `5`) sets how many ClickHouse inserts run in parallel.
```bash
-EVENTS_GROUP_QUEUES_SHARDS=4
-QUEUE_CLUSTER=true
+CRON_CONCURRENCY=6
```
-<Callout type="warn">
-Set `EVENTS_GROUP_QUEUES_SHARDS` before you have live traffic on the queue. Changing it while jobs are pending will cause those jobs to be looked up on the wrong shard and they will not be processed until the shard count is restored.
-</Callout>
+If you split workers into pools with [`ENABLED_QUEUES`](/docs/self-hosting/environment-variables#enabled_queues), the flushing happens in the pool that runs the `cron` queue.
-### Tune the ordering delay
+### Leave room for ClickHouse
-Events arriving out of order are held briefly before processing. The default is `100ms`.
+Every flush ends in a ClickHouse insert. When workers and ClickHouse share a machine, more worker replicas can make ingestion slower rather than faster, because they compete with ClickHouse for CPU. If insert times climb as you add workers, give ClickHouse its own resources before adding more.
-```bash
-ORDERING_DELAY_MS=100
-```
+### What to watch
-Lowering this reduces latency but increases the chance of out-of-order writes to ClickHouse. The value should not exceed `500ms`.
+- **Kafka consumer lag** for your `KAFKA_CONSUMER_GROUP`: `docker compose exec op-rp rpk group describe openpanel-events`, replacing `openpanel-events` (the default) if you set a different group.
+- **Buffer backlog:** `buffer_flush_llen_at_start` on a worker's `/metrics` endpoint, or the length of the `event_buffer:queue` and `session-buffer` lists in Redis.
+- **Insert latency:** `buffer_ch_insert_duration_ms` on the same endpoint. If it rises with load, ClickHouse is the limit.
---
@@ -233,16 +222,11 @@ DATABASE_URL_DIRECT=postgresql://postgres:postgres@op-db:5432/postgres?schema=pu
EVENT_BUFFER_BATCH_SIZE=10000
EVENT_BUFFER_CHUNK_SIZE=2000
-# Session buffer
-SESSION_BUFFER_BATCH_SIZE=5000
-SESSION_BUFFER_CHUNK_SIZE=2000
-
# Profile buffer
PROFILE_BUFFER_BATCH_SIZE=500
-# Queue
-EVENTS_GROUP_QUEUES_SHARDS=4
-EVENT_JOB_CONCURRENCY=20
+# Flushing
+CRON_CONCURRENCY=6
```
Then start with more workers:
@@ -251,4 +235,4 @@ Then start with more workers:
docker compose up -d --scale op-worker=8
```
-Monitor the Redis queue depth and ClickHouse insert latency as you tune. The right values depend on your hardware, event shape, and traffic pattern.
+Watch Kafka consumer lag, the Redis buffer backlog and ClickHouse insert latency as you tune. The right values depend on your hardware, event shape, and traffic pattern.
diff --git a/packages/core/src/buffers/base-buffer.ts b/packages/core/src/buffers/base-buffer.ts
index a5bb44ecb..84bde8ade 100644
--- a/packages/core/src/buffers/base-buffer.ts
+++ b/packages/core/src/buffers/base-buffer.ts
@@ -59,6 +59,8 @@ export type FlushPhaseTimings = {
* fetch-existing-profile-for-merge step). Only buffers that read CH
* during flush will populate this. */
chFetchMs?: number;
+ /** Parsing and reshaping rows between the read and the insert. */
+ transformMs?: number;
chInsertMs?: number;
trimMs?: number;
/** Total time spent inside onFlush (subclass-specific processing) */
@@ -68,6 +70,7 @@ export type FlushPhaseTimings = {
const FLUSH_PHASES = [
'lrangeMs',
'chFetchMs',
+ 'transformMs',
'chInsertMs',
'trimMs',
'onFlushMs',
@@ -272,9 +275,9 @@ export class BaseBuffer {
/**
* Run `fn` over `items` with a bounded concurrency. Preserves order of
- * `results`. If any worker throws, the helper propagates the first error
- * (consistent with `Promise.all`); other in-flight promises run to
- * completion but their resolved values/errors are discarded.
+ * `results`. After the first failure no new item starts, and the call
+ * rejects with that error only once the items already running have settled,
+ * so nothing is still writing when the caller releases its flush lock.
*/
protected async parallelLimit<T, R>(
items: T[],
@@ -293,13 +296,18 @@ export class BaseBuffer {
}
const results = new Array<R>(items.length);
let nextIndex = 0;
+ const failures: unknown[] = [];
const worker = async (): Promise<void> => {
- while (true) {
+ while (failures.length === 0) {
const idx = nextIndex++;
if (idx >= items.length) {
return;
}
- results[idx] = await fn(items[idx] as T, idx);
+ try {
+ results[idx] = await fn(items[idx] as T, idx);
+ } catch (error) {
+ failures.push(error);
+ }
}
};
const workers = Array.from(
@@ -307,6 +315,9 @@ export class BaseBuffer {
() => worker()
);
await Promise.all(workers);
+ if (failures.length > 0) {
+ throw failures[0];
+ }
return results;
}
@@ -423,6 +434,10 @@ export class BaseBuffer {
obs.phases?.chFetchMs != null
? Math.round(obs.phases.chFetchMs)
: undefined,
+ transformMs:
+ obs.phases?.transformMs != null
+ ? Math.round(obs.phases.transformMs)
+ : undefined,
chInsertMs:
obs.phases?.chInsertMs != null
? Math.round(obs.phases.chInsertMs)
diff --git a/packages/core/src/buffers/event-buffer.test.ts b/packages/core/src/buffers/event-buffer.test.ts
index c82761947..fac8a1940 100644
--- a/packages/core/src/buffers/event-buffer.test.ts
+++ b/packages/core/src/buffers/event-buffer.test.ts
@@ -651,6 +651,54 @@ describe('EventBuffer', () => {
);
expect(await eventBuffer.getBufferSize()).toBe(1);
});
+
+ it('a flush drains a backlog larger than one batch', async () => {
+ const config = testCoreConfig();
+ config.buffers.event.batchSize = 3;
+ const eb = new EventBuffer(bufferDepsWithCh({ insert: chInsert }, config));
+ for (let i = 0; i < 10; i++) {
+ eb.add({
+ project_id: 'p13',
+ name: `event${i}`,
+ created_at: new Date().toISOString(),
+ } as any);
+ }
+ await eb.flush();
+
+ await eb.tryFlush({ trigger: 'cron' });
+
+ expect(chInsert).toHaveBeenCalledTimes(4);
+ const names: string[] = [];
+ for (const [options] of chInsert.mock.calls) {
+ const lines = await streamToLines(
+ (options as { values: Readable }).values
+ );
+ names.push(...lines.map((line) => JSON.parse(line).name));
+ }
+ expect(names).toEqual(Array.from({ length: 10 }, (_, i) => `event${i}`));
+ expect(await eb.getBufferSize()).toBe(0);
+ });
+
+ it('a failed batch stops the drain and keeps the rest queued', async () => {
+ const config = testCoreConfig();
+ config.buffers.event.batchSize = 3;
+ const eb = new EventBuffer(bufferDepsWithCh({ insert: chInsert }, config));
+ for (let i = 0; i < 10; i++) {
+ eb.add({
+ project_id: 'p14',
+ name: `event${i}`,
+ created_at: new Date().toISOString(),
+ } as any);
+ }
+ await eb.flush();
+
+ chInsert.mockResolvedValueOnce(undefined);
+ chInsert.mockRejectedValueOnce(new Error('ClickHouse unavailable'));
+
+ await expect(eb.processBuffer()).rejects.toThrow('ClickHouse unavailable');
+ expect(chInsert).toHaveBeenCalledTimes(2);
+ expect(await eb.getBufferSize()).toBe(7);
+ });
});
describe('extractProjectId', () => {
diff --git a/packages/core/src/buffers/event-buffer.ts b/packages/core/src/buffers/event-buffer.ts
index 1553b3b2c..c22067188 100644
--- a/packages/core/src/buffers/event-buffer.ts
+++ b/packages/core/src/buffers/event-buffer.ts
@@ -385,6 +385,11 @@ export class EventBuffer extends BaseBuffer {
}
async processBuffer() {
+ await this.drainBatches(this.batchSize, () => this.processBatch());
+ }
+
+ /** Inserts and trims one batch from the head of the queue; returns its size. */
+ private async processBatch(): Promise<number> {
const redis = getRedisCache();
const lrangeStart = performance.now();
@@ -397,7 +402,7 @@ export class EventBuffer extends BaseBuffer {
if (queueEvents.length === 0) {
this.reportFlushStats({ rowsProcessed: 0, phases: { lrangeMs } });
- return;
+ return 0;
}
// The queued lines are already JSONEachRow, and the client's custom
@@ -444,6 +449,7 @@ export class EventBuffer extends BaseBuffer {
rowsProcessed: queueEvents.length,
phases: { lrangeMs, chInsertMs, trimMs },
});
+ return queueEvents.length;
}
public async getActiveVisitorCount(projectId: string): Promise<number> {
diff --git a/packages/core/src/buffers/session-buffer.test.ts b/packages/core/src/buffers/session-buffer.test.ts
index 38205c492..0d0891b49 100644
--- a/packages/core/src/buffers/session-buffer.test.ts
+++ b/packages/core/src/buffers/session-buffer.test.ts
@@ -4,6 +4,7 @@ import { bufferDepsWithCh } from '../../test/buffer-fixtures';
import { testCoreConfig } from '../../test/config-fixture';
import type { IServiceCreateEventPayload } from '../modules/event/event.service';
import type { IClickhouseSession } from '../modules/session/session.service';
+import type { FlushObservation } from './base-buffer';
// The client comes in as `BufferDeps.ch` — no module mock needed for the flush
// path at all. See event-buffer.test.ts's header.
@@ -474,8 +475,157 @@ describe('SessionBuffer', () => {
expect(chInsert).toHaveBeenCalledTimes(4);
expect(await buffer.getBufferSize()).toBe(0);
});
+
+ it('inserts the chunks of one batch concurrently', async () => {
+ const config = testCoreConfig();
+ config.buffers.session.chunkSize = 2;
+ const buffer = new SessionBuffer(
+ bufferDepsWithCh({ insert: chInsert }, config)
+ );
+ const t0 = Date.now();
+ for (let i = 0; i < 10; i++) {
+ await buffer.ingest(
+ makePayload({
+ deviceId: `device-${i}`,
+ sessionId: `session-${i}`,
+ createdAt: new Date(t0),
+ })
+ );
+ }
+
+ let inFlight = 0;
+ let maxInFlight = 0;
+ const insertedIds: string[] = [];
+ chInsert.mockImplementation(async ({ values }) => {
+ inFlight += 1;
+ maxInFlight = Math.max(maxInFlight, inFlight);
+ await Bun.sleep(INSERT_LATENCY_MS);
+ insertedIds.push(...values.map((row) => row.device_id));
+ inFlight -= 1;
+ });
+
+ await buffer.processBuffer();
+
+ expect(chInsert).toHaveBeenCalledTimes(5);
+ expect(maxInFlight).toBeGreaterThan(1);
+ expect(insertedIds.sort()).toEqual(
+ Array.from({ length: 10 }, (_, i) => `device-${i}`).sort()
+ );
+ expect(await buffer.getBufferSize()).toBe(0);
+ });
+
+ it('keeps the whole batch queued when one concurrent chunk fails', async () => {
+ const config = testCoreConfig();
+ config.buffers.session.chunkSize = 2;
+ const buffer = new SessionBuffer(
+ bufferDepsWithCh({ insert: chInsert }, config)
+ );
+ const t0 = Date.now();
+ for (let i = 0; i < 10; i++) {
+ await buffer.ingest(
+ makePayload({
+ deviceId: `device-${i}`,
+ sessionId: `session-${i}`,
+ createdAt: new Date(t0),
+ })
+ );
+ }
+
+ chInsert.mockImplementation(async ({ values }) => {
+ await Bun.sleep(INSERT_LATENCY_MS);
+ if (values.some((row) => row.device_id === 'device-5')) {
+ throw new Error('ClickHouse unavailable');
+ }
+ });
+
+ await expect(buffer.processBuffer()).rejects.toThrow(
+ 'ClickHouse unavailable'
+ );
+ expect(await buffer.getBufferSize()).toBe(10);
+ });
+
+ it('inserts a whole batch in one insert by default', async () => {
+ const buffer = new SessionBuffer(bufferDepsWithCh({ insert: chInsert }));
+ // More rows than any fixed chunk size short of the batch size would allow
+ // in one insert.
+ const rows = Array.from({ length: ROWS_OVER_OLD_CHUNK_SIZE }, (_, i) =>
+ JSON.stringify({ id: `session-${i}`, sign: 1, version: 1, duration: 0 })
+ );
+ await redis.rpush('session-buffer', ...rows);
+
+ await buffer.processBuffer();
+
+ expect(chInsert).toHaveBeenCalledTimes(1);
+ expect(chInsert.mock.calls[0]?.[0].values).toHaveLength(
+ ROWS_OVER_OLD_CHUNK_SIZE
+ );
+ });
+
+ it('after a failed chunk starts no new chunk and rejects once the running ones finish', async () => {
+ const config = testCoreConfig();
+ config.buffers.session.chunkSize = 2;
+ const buffer = new SessionBuffer(
+ bufferDepsWithCh({ insert: chInsert }, config)
+ );
+ const t0 = Date.now();
+ for (let i = 0; i < 20; i++) {
+ await buffer.ingest(
+ makePayload({
+ deviceId: `device-${i}`,
+ sessionId: `session-${i}`,
+ createdAt: new Date(t0),
+ })
+ );
+ }
+
+ let started = 0;
+ let finished = 0;
+ chInsert.mockImplementation(async ({ values }) => {
+ started += 1;
+ if (values.some((row) => row.device_id === 'device-0')) {
+ throw new Error('ClickHouse unavailable');
+ }
+ await Bun.sleep(INSERT_LATENCY_MS);
+ finished += 1;
+ });
+
+ await expect(buffer.processBuffer()).rejects.toThrow(
+ 'ClickHouse unavailable'
+ );
+ // Ten chunks, five at a time: the first wave runs, nothing after it.
+ expect(started).toBe(BUFFER_CH_INSERT_CONCURRENCY_DEFAULT);
+ expect(finished).toBe(BUFFER_CH_INSERT_CONCURRENCY_DEFAULT - 1);
+ expect(await buffer.getBufferSize()).toBe(20);
+ });
+
+ it('reports read, transform, insert and trim timings', async () => {
+ const buffer = bufferWithBatchSize(1000);
+ const observations: FlushObservation[] = [];
+ buffer.flushObserver = (observation) => observations.push(observation);
+ await buffer.ingest(makePayload());
+
+ await buffer.tryFlush({ trigger: 'cron' });
+
+ expect(observations).toHaveLength(1);
+ const [observation] = observations;
+ expect(observation?.result).toBe('success');
+ expect(observation?.rowsProcessed).toBe(1);
+ for (const phase of [
+ 'lrangeMs',
+ 'transformMs',
+ 'chInsertMs',
+ 'trimMs',
+ ] as const) {
+ expect(observation?.phases?.[phase]).toBeGreaterThanOrEqual(0);
+ }
+ });
});
+const INSERT_LATENCY_MS = 10;
+const ROWS_OVER_OLD_CHUNK_SIZE = 1500;
+/** `testCoreConfig()` leaves it unset, so the base buffer's default of 5 applies. */
+const BUFFER_CH_INSERT_CONCURRENCY_DEFAULT = 5;
+
const MONITOR_SENTINEL = 'session-buffer-test-done';
const MONITOR_POLL_MS = 5;
diff --git a/packages/core/src/buffers/session-buffer.ts b/packages/core/src/buffers/session-buffer.ts
index ffd84356e..eff8030e6 100644
--- a/packages/core/src/buffers/session-buffer.ts
+++ b/packages/core/src/buffers/session-buffer.ts
@@ -84,14 +84,17 @@ function pickUtm(
return v ? String(v) : '';
}
-const DEFAULT_BATCH_SIZE = 1000;
-const DEFAULT_CHUNK_SIZE = 1000;
+// Large enough that a busy session's versions share a batch and squash.
+const DEFAULT_BATCH_SIZE = 10_000;
export class SessionBuffer extends BaseBuffer {
private readonly batchSize =
this.deps.config.buffers.session.batchSize ?? DEFAULT_BATCH_SIZE;
+ // One insert per batch by default: a failed chunk keeps the whole batch
+ // queued, so chunks that already landed would be inserted again, and a
+ // duplicated sign row corrupts the collapsed session.
private readonly chunkSize =
- this.deps.config.buffers.session.chunkSize ?? DEFAULT_CHUNK_SIZE;
+ this.deps.config.buffers.session.chunkSize ?? this.batchSize;
private readonly squashEnabled = this.deps.config.buffers.session.squash;
private readonly idleTimeoutMs = resolveSessionTimeoutMs(this.deps.config);
@@ -529,15 +532,19 @@ export class SessionBuffer extends BaseBuffer {
/** Inserts and trims one batch from the head of the list; returns its size. */
private async processBatch(): Promise<number> {
+ const lrangeStart = performance.now();
const events = await this.redis.lrange(
this.redisKey,
0,
this.batchSize - 1
);
+ const lrangeMs = performance.now() - lrangeStart;
if (events.length === 0) {
+ this.reportFlushStats({ rowsProcessed: 0, phases: { lrangeMs } });
return 0;
}
+ const transformStart = performance.now();
const parsed: IClickhouseSession[] = [];
for (const e of events) {
const s = getSafeJson<IClickhouseSession>(e);
@@ -558,23 +565,32 @@ export class SessionBuffer extends BaseBuffer {
const sessions = this.squashEnabled
? this.squashSessionsByVersion(parsed)
: parsed;
+ const transformMs = performance.now() - transformStart;
+ // Chunks may land in any order: VersionedCollapsingMergeTree collapses by
+ // (sign, version), not by insert order.
const ch = this.resolveCh();
- for (const chunk of this.chunks(sessions, this.chunkSize)) {
- await ch.insert({
+ const chStart = performance.now();
+ await this.parallelLimit(this.chunks(sessions, this.chunkSize), (chunk) =>
+ ch.insert({
table: TABLE_NAMES.sessions,
values: chunk,
format: 'JSONEachRow',
- });
- }
+ })
+ );
+ const chInsertMs = performance.now() - chStart;
// Trim only after a successful insert — on failure the rows stay queued.
// Don't swallow: let it propagate so the base flush records result:'error'
// (and the next flush retries the still-queued rows).
+ const trimStart = performance.now();
await this.redis.ltrim(this.redisKey, events.length, -1);
+ const trimMs = performance.now() - trimStart;
- this.reportFlushStats({ rowsProcessed: events.length });
- this.logger.debug({ count: events.length }, 'Processed sessions');
+ this.reportFlushStats({
+ rowsProcessed: events.length,
+ phases: { lrangeMs, transformMs, chInsertMs, trimMs },
+ });
return events.length;
}
diff --git a/packages/core/src/jobs.registry.test.ts b/packages/core/src/jobs.registry.test.ts
index c1fbf33d0..83dd96133 100644
--- a/packages/core/src/jobs.registry.test.ts
+++ b/packages/core/src/jobs.registry.test.ts
@@ -55,7 +55,7 @@ const PINNED_DEFAULTS = {
const PINNED_CONCURRENCY = {
sessions: 1,
- cron: 1,
+ cron: 4,
notification: 1,
import: 1,
insights: 5,
diff --git a/packages/core/src/jobs.registry.ts b/packages/core/src/jobs.registry.ts
index 54183d87d..25cf75852 100644
--- a/packages/core/src/jobs.registry.ts
+++ b/packages/core/src/jobs.registry.ts
@@ -40,7 +40,7 @@ const COHORT_ATTEMPTS = 3;
// `<KEY>_CONCURRENCY` overrides one of these at boot.
const CONCURRENCY = {
sessions: 1,
- cron: 1,
+ cron: 4,
notification: 1,
import: 1,
insights: 5,
@@ -63,7 +63,8 @@ const sessions = defineQueue('sessions', {
},
});
-// One queue, one worker at concurrency 1, every scheduled job in the system.
+// One queue for every scheduled job in the system. Each buffer flush takes its
+// own lock, so concurrency lets different buffers flush side by side.
// Pausing it from bull-board halts all buffer flushing.
const cron = defineQueue('cron', {
defaults: {
diff --git a/packages/core/src/modules/ingest/src/consumer.test.ts b/packages/core/src/modules/ingest/src/consumer.test.ts
index aab8be1fa..298b534f7 100644
--- a/packages/core/src/modules/ingest/src/consumer.test.ts
+++ b/packages/core/src/modules/ingest/src/consumer.test.ts
@@ -15,11 +15,12 @@ import {
type DeadLetterRecord,
type DeadLetterRedisClient,
} from '@openpanel/redis';
-import type { EachBatchPayload, KafkaMessage } from 'kafkajs';
+import type { Consumer, EachBatchPayload, KafkaMessage } from 'kafkajs';
import {
createEventsBatchHandler,
type DeadLetterMessage,
type EventsBatchHandlerDeps,
+ startKafkaEventsConsumer,
} from './consumer';
const PARTITION = 7;
@@ -878,3 +879,44 @@ describe('dead letter to a capped redis list', () => {
expect(resolvedOffsets(resolveOffset)).toEqual(['0']);
});
});
+
+describe('startKafkaEventsConsumer', () => {
+ test('a partition with no committed offset starts at its oldest message', async () => {
+ const subscribe = mock(
+ async (_options: { topic: string; fromBeginning?: boolean }) => undefined
+ );
+ const consumer = {
+ connect: mock(async () => undefined),
+ subscribe,
+ on: mock(() => () => undefined),
+ run: mock(async () => undefined),
+ stop: mock(async () => undefined),
+ disconnect: mock(async () => undefined),
+ events: {
+ HEARTBEAT: 'consumer.heartbeat',
+ GROUP_JOIN: 'consumer.group_join',
+ REBALANCING: 'consumer.rebalancing',
+ CRASH: 'consumer.crash',
+ DISCONNECT: 'consumer.disconnect',
+ REQUEST_TIMEOUT: 'consumer.network.request_timeout',
+ },
+ };
+ const { deps } = makeDeps();
+ const { topic: _topic, ...batch } = deps;
+
+ await startKafkaEventsConsumer({
+ createConsumer: () => consumer as unknown as Consumer,
+ logger: deps.logger,
+ kafkaLogger: deps.logger,
+ topic: TOPIC,
+ partitionsConsumedConcurrently: 1,
+ batch,
+ });
+
+ expect(subscribe).toHaveBeenCalledTimes(1);
+ expect(subscribe).toHaveBeenCalledWith({
+ topic: TOPIC,
+ fromBeginning: true,
+ });
+ });
+});
diff --git a/packages/core/src/modules/ingest/src/consumer.ts b/packages/core/src/modules/ingest/src/consumer.ts
index 295dfe1cc..5087c1a06 100644
--- a/packages/core/src/modules/ingest/src/consumer.ts
+++ b/packages/core/src/modules/ingest/src/consumer.ts
@@ -461,9 +461,13 @@ export async function startKafkaEventsConsumer(
topic: deps.topic,
});
await consumer.connect();
+ // Only decides where a partition WITHOUT a committed offset starts: a new
+ // install, a new partition or a renamed group. From the newest message it
+ // would skip everything produced before this consumer first joined; from the
+ // oldest it can at worst re-read the topic's retention as duplicates.
await consumer.subscribe({
topic: deps.topic,
- fromBeginning: false,
+ fromBeginning: true,
});
consumer.on(consumer.events.HEARTBEAT, deps.batch.onActivity);
diff --git a/packages/core/src/modules/integration/integration.jobs.ts b/packages/core/src/modules/integration/integration.jobs.ts
index 423d805fe..158fbf71e 100644
--- a/packages/core/src/modules/integration/integration.jobs.ts
+++ b/packages/core/src/modules/integration/integration.jobs.ts
@@ -1,6 +1,7 @@
import { z } from 'zod';
import { defineJob } from '../../jobs/define';
import {
+ createRedisExportClaim,
type ExportClickhouse,
type ExportDb,
runFlushExportsCron,
@@ -30,6 +31,7 @@ export const integrationCronJobs = {
config,
ctx.config
),
+ claimExport: createRedisExportClaim(ctx.redis),
});
},
}),
diff --git a/packages/core/src/modules/integration/src/flush-exports.test.ts b/packages/core/src/modules/integration/src/flush-exports.test.ts
index 4f642ebbf..46f58eb1c 100644
--- a/packages/core/src/modules/integration/src/flush-exports.test.ts
+++ b/packages/core/src/modules/integration/src/flush-exports.test.ts
@@ -3,6 +3,7 @@
import { describe, expect, it, mock } from 'bun:test';
import { gunzipSync } from 'node:zlib';
+import { getRedisCache } from '@openpanel/redis';
import { testCoreConfig } from '../../../../test/config-fixture';
import type { Logger } from '../../../logger';
import {
@@ -16,6 +17,7 @@ import {
serializeManifest,
} from './export';
import {
+ createRedisExportClaim,
type ExportIntegrationRow,
type ExportWatermarkRow,
type FlushExportsDeps,
@@ -80,6 +82,7 @@ const watermark: ExportWatermarkRow = {
function fakeDeps(rows: unknown[][]) {
const uploads: { key: string; contentType: string }[] = [];
const watermarkUpdates: unknown[] = [];
+ const releaseClaim = mock(async () => undefined);
const windows = [...rows];
const query = mock(async () => ({
json: async () => windows.shift() ?? [],
@@ -92,9 +95,9 @@ function fakeDeps(rows: unknown[][]) {
exportWatermark: {
findUnique: async () => watermark,
create: async () => ({}),
- update: async (args) => {
+ updateMany: async (args) => {
watermarkUpdates.push(args);
- return {};
+ return { count: 1 };
},
},
},
@@ -114,9 +117,10 @@ function fakeDeps(rows: unknown[][]) {
return { key };
},
}) as never,
+ claimExport: async () => releaseClaim,
};
- return { deps, uploads, watermarkUpdates, query };
+ return { deps, uploads, watermarkUpdates, query, releaseClaim };
}
describe('runFlushExportsCron', () => {
@@ -142,6 +146,22 @@ describe('runFlushExportsCron', () => {
});
});
+ it('never moves the watermark backwards', async () => {
+ const { deps, watermarkUpdates } = fakeDeps([[1, 2, 3].map(chEvent)]);
+
+ await runFlushExportsCron(deps);
+
+ // Only applies while the stored watermark is not newer than this run's,
+ // so a run that outlived its claim cannot undo a later run's progress.
+ expect(watermarkUpdates[0]).toMatchObject({
+ where: {
+ projectId: 'proj_1',
+ integrationId: 'int_1',
+ lastInsertedAt: { lte: new Date('2026-08-26T11:03:00.000Z') },
+ },
+ });
+ });
+
it('does nothing when no integration declares an export capability', async () => {
const { deps, uploads } = fakeDeps([]);
deps.db.integration.findMany = async () => [
@@ -161,6 +181,84 @@ describe('runFlushExportsCron', () => {
// Two rows is far under BATCH_SIZE, so one window is the whole backlog.
expect(query).toHaveBeenCalledTimes(1);
});
+
+ it('skips an export another run has claimed', async () => {
+ const { deps, uploads, watermarkUpdates, query } = fakeDeps([
+ [1, 2, 3].map(chEvent),
+ ]);
+ deps.claimExport = async () => null;
+
+ await runFlushExportsCron(deps);
+
+ expect(query).not.toHaveBeenCalled();
+ expect(uploads).toHaveLength(0);
+ expect(watermarkUpdates).toHaveLength(0);
+ });
+
+ it('releases the claim after a successful export', async () => {
+ const { deps, releaseClaim } = fakeDeps([[1, 2, 3].map(chEvent)]);
+
+ await runFlushExportsCron(deps);
+
+ expect(releaseClaim).toHaveBeenCalledTimes(1);
+ });
+
+ it('releases the claim when the export fails', async () => {
+ const { deps, releaseClaim } = fakeDeps([[1, 2, 3].map(chEvent)]);
+ deps.createAdapter = () =>
+ ({
+ upload: async () => {
+ throw new Error('bucket unavailable');
+ },
+ }) as never;
+
+ await expect(runFlushExportsCron(deps)).rejects.toThrow(
+ 'exports failed with a retryable provider error'
+ );
+ expect(releaseClaim).toHaveBeenCalledTimes(1);
+ });
+});
+
+describe('createRedisExportClaim', () => {
+ const redis = getRedisCache();
+ const claim = createRedisExportClaim(redis);
+ const projectId = 'proj_claim_test';
+ const claimKey = (integrationId: string) =>
+ `lock:flush-exports:${projectId}:${integrationId}`;
+
+ it('grants one claim at a time per export', async () => {
+ await redis.del(claimKey('int_a'), claimKey('int_b'));
+
+ const first = await claim(projectId, 'int_a');
+ expect(first).not.toBeNull();
+ expect(await claim(projectId, 'int_a')).toBeNull();
+ // A different integration is a different export.
+ const other = await claim(projectId, 'int_b');
+ expect(other).not.toBeNull();
+
+ await first?.();
+ const again = await claim(projectId, 'int_a');
+ expect(again).not.toBeNull();
+
+ await again?.();
+ await other?.();
+ });
+
+ it('a release whose claim expired does not free the next holder', async () => {
+ await redis.del(claimKey('int_c'));
+
+ const stale = await claim(projectId, 'int_c');
+ // Simulate the TTL running out while the first run is still going.
+ await redis.del(claimKey('int_c'));
+ const current = await claim(projectId, 'int_c');
+ expect(current).not.toBeNull();
+
+ await stale?.();
+ expect(await claim(projectId, 'int_c')).toBeNull();
+
+ await current?.();
+ expect(await redis.exists(claimKey('int_c'))).toBe(0);
+ });
});
/**
diff --git a/packages/core/src/modules/integration/src/flush-exports.ts b/packages/core/src/modules/integration/src/flush-exports.ts
index 0a11c7bcf..9d79f2d62 100644
--- a/packages/core/src/modules/integration/src/flush-exports.ts
+++ b/packages/core/src/modules/integration/src/flush-exports.ts
@@ -1,7 +1,9 @@
// Prisma, ClickHouse and the object-store adapter factory are injected so the
// orchestration is testable without `mock.module`.
-import { DateTime } from '@openpanel/shared';
+import type { Redis } from '@openpanel/redis';
+import { DateTime, generateSecureId } from '@openpanel/shared';
+import { defineRedisScript } from '../../../buffers/redis-script';
import { isProviderError } from '../../../clients/provider-error';
import type { CoreConfig } from '../../../config';
import type { Logger } from '../../../logger';
@@ -32,6 +34,16 @@ const DEFAULT_LAG_SECONDS = 60;
const DEFAULT_BATCH_SIZE = 50_000;
const DEFAULT_MAX_BATCHES_PER_RUN = 20;
const DEFAULT_CONCURRENCY = 4;
+// Outlives a full run (DEFAULT_MAX_BATCHES_PER_RUN uploads); a run that takes
+// longer loses the claim and may overlap, which at-least-once already allows.
+const EXPORT_CLAIM_TTL_MS = 15 * 60 * 1000;
+
+const releaseClaimScript = defineRedisScript(`
+if redis.call("get", KEYS[1]) == ARGV[1] then
+ return redis.call("del", KEYS[1])
+end
+return 0
+`);
// Sentinel cursor id for the first window of a (project, integration). The
// events `id` column is a UUID, so the tie-breaker must compare as UUID — an
@@ -89,7 +101,7 @@ export interface ExportDb {
exportWatermark: {
findUnique(args: unknown): Promise<ExportWatermarkRow | null>;
create(args: unknown): Promise<unknown>;
- update(args: unknown): Promise<unknown>;
+ updateMany(args: unknown): Promise<{ count: number }>;
};
}
@@ -101,6 +113,41 @@ export interface FlushExportsDeps {
config: CoreConfig;
/** Resolves the object-store adapter for an export config, or undefined. */
createAdapter(config: ExportConfig): IObjectStoreAdapter | undefined;
+ /**
+ * Claims one (project, integration) for this run; null while another run
+ * holds it. Two overlapping runs would read the same watermark and upload the
+ * same window twice.
+ */
+ claimExport(
+ projectId: string,
+ integrationId: string
+ ): Promise<ReleaseClaim | null>;
+}
+
+export type ReleaseClaim = () => Promise<void>;
+
+/** `claimExport` backed by a Redis key per (project, integration). */
+export function createRedisExportClaim(
+ redis: Redis
+): FlushExportsDeps['claimExport'] {
+ return async (projectId, integrationId) => {
+ const key = `lock:flush-exports:${projectId}:${integrationId}`;
+ const token = generateSecureId('lock');
+ const claimed = await redis.set(
+ key,
+ token,
+ 'PX',
+ EXPORT_CLAIM_TTL_MS,
+ 'NX'
+ );
+ if (claimed !== 'OK') {
+ return null;
+ }
+ // Token-checked, so a run that outlived its claim never frees another's.
+ return async () => {
+ await releaseClaimScript(redis, [key], [token]);
+ };
+ };
}
function isExportConfig(config: IIntegrationConfig): config is ExportConfig {
@@ -204,6 +251,27 @@ async function processExport(
integrationId: string,
config: ExportConfig,
deps: FlushExportsDeps
+): Promise<void> {
+ const release = await deps.claimExport(projectId, integrationId);
+ if (!release) {
+ deps.logger.debug(
+ { projectId, integrationId },
+ 'Export already running in another cron run'
+ );
+ return;
+ }
+ try {
+ await exportWindows(projectId, integrationId, config, deps);
+ } finally {
+ await release();
+ }
+}
+
+async function exportWindows(
+ projectId: string,
+ integrationId: string,
+ config: ExportConfig,
+ deps: FlushExportsDeps
): Promise<void> {
let cursor = await loadCursor(projectId, integrationId, deps);
const adapter = deps.createAdapter(config);
@@ -351,12 +419,16 @@ async function saveCursor(
cursor: Cursor,
deps: FlushExportsDeps
): Promise<void> {
- await deps.db.exportWatermark.update({
- where: { projectId_integrationId: { projectId, integrationId } },
- data: {
- lastInsertedAt: convertClickhouseDateToJs(cursor.insertedAt),
- lastEventId: cursor.eventId,
+ const lastInsertedAt = convertClickhouseDateToJs(cursor.insertedAt);
+ // Forward only: a run that outlived its claim must not rewind a watermark
+ // that a later run has already moved past.
+ await deps.db.exportWatermark.updateMany({
+ where: {
+ projectId,
+ integrationId,
+ lastInsertedAt: { lte: lastInsertedAt },
},
+ data: { lastInsertedAt, lastEventId: cursor.eventId },
});
}
diff --git a/self-hosting/cli/src/doctor/checks.ts b/self-hosting/cli/src/doctor/checks.ts
index 89f63481c..bb860a283 100644
--- a/self-hosting/cli/src/doctor/checks.ts
+++ b/self-hosting/cli/src/doctor/checks.ts
@@ -32,6 +32,7 @@ const REMOVED_ENV_VARS = [
'AUTO_BATCH_SIZE',
'AUTO_BATCH_MAX_WAIT_MS',
'SHUTDOWN_GRACE_PERIOD_MS',
+ 'DATABASE_URL_REPLICA',
'FUNNEL_NON_STRICT_ORDERING',
'CHART_VALUES_LOOKBACK_DAYS',
] as const;
diff --git a/self-hosting/cli/test/doctor-edge.test.ts b/self-hosting/cli/test/doctor-edge.test.ts
index d3c8b76f4..866e1f560 100644
--- a/self-hosting/cli/test/doctor-edge.test.ts
+++ b/self-hosting/cli/test/doctor-edge.test.ts
@@ -230,3 +230,14 @@ describe('worker replicas', () => {
expect(replicas).toBe(`${DOLLAR}{OP_WORKER_REPLICAS:-1}`);
});
});
+
+test('flags DATABASE_URL_REPLICA, which v3 no longer reads', () => {
+ const install = loadInstall(V1_FIXTURE);
+ install.env.set('DATABASE_URL_REPLICA', 'postgresql://replica:5432/postgres');
+
+ const finding = runDoctor(install).find(
+ (candidate) => candidate.id === 'env/removed-vars'
+ );
+
+ expect(finding?.title).toContain('DATABASE_URL_REPLICA');
+});