Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

fix(dynamodb): route Kinesis CDC to the table owner's account and retry failed forwards (#3063)

Guilherme Scaransea9df87295605
diff --git a/docs/services/dynamodb.md b/docs/services/dynamodb.md
index 7f3ac0d8f6..8f6d3c0cd4 100644
--- a/docs/services/dynamodb.md
+++ b/docs/services/dynamodb.md
@@ -164,4 +164,38 @@ aws dynamodb list-exports \
--endpoint-url $AWS_ENDPOINT_URL
```
-The export writes to `s3://<bucket>/<prefix>/AWSDynamoDB/<exportId>/data/` as one or more `.json.gz` files, along with `manifest-summary.json` and `manifest-files.json`, the same layout as real AWS DynamoDB exports.
\ No newline at end of file
+The export writes to `s3://<bucket>/<prefix>/AWSDynamoDB/<exportId>/data/` as one or more `.json.gz` files, along with `manifest-summary.json` and `manifest-files.json`, the same layout as real AWS DynamoDB exports.
+
+## Kinesis change data capture (CDC)
+
+When a table has an **ACTIVE** Kinesis streaming destination (see
+`EnableKinesisStreamingDestination`), every item change, `INSERT`, `MODIFY`, and `REMOVE`,
+including TTL expirations, is forwarded to the destination stream as a Kinesis record in the
+AWS CDC envelope (`eventName`, `dynamodb.Keys`, `NewImage`/`OldImage`, `ApproximateCreationDateTime`).
+
+### Delivery contract
+
+Forwarding is **bounded best-effort with in-process retry**. A write is never blocked or failed by
+the destination stream: the change event is enqueued and delivered on a background drain, so a slow or
+unavailable Kinesis stream cannot stall a `PutItem`/`UpdateItem`/`DeleteItem` or the TTL sweep.
+
+The drain gives these guarantees per destination:
+
+- **Retries** transient/unknown send failures with capped exponential backoff (250 ms doubling to an
+ 8 s cap, up to 10 attempts) rather than dropping the record on the first exception.
+- **Preserves FIFO** across retries: the head record is not skipped past (stronger than AWS, which may
+ reorder or duplicate).
+- **Drops a poison record** immediately on a deterministic terminal failure
+ (`ValidationException`/`InvalidArgumentException`) so it cannot wedge the queue behind it.
+- **Gives up an episode** after the retry budget is exhausted, dropping the buffered records and marking
+ the destination `GAVE_UP`; a later change event starts a fresh episode and self-heals once the stream
+ recovers.
+- **Bounds memory** at 1000 buffered records per destination, evicting the oldest on overflow.
+
+Disabling a destination or deleting the table discards that destination's buffered records and stops all
+future sends for it; a send already in flight at that instant may still complete (teardown cannot recall
+an in-flight request, an inherent and harmless race). Buffered records are held **in memory only**: they
+are not durable and are lost on restart (this is an emulator, not an at-least-once pipeline). Records that are permanently dropped (terminal, give-up,
+or overflow) are counted and logged, and per-destination delivery health (forwarded/retried/dropped
+counts, queue depth, last error, and current health) is tracked for inspection.
+```
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java b/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java
index 2473a8fb86..61d90cb48d 100644
--- a/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java
@@ -66,13 +66,23 @@ public class AccountAwareStorageBackend<V> implements StorageBackend<String, V>
if (result.isPresent()) {
return result;
}
- // Backward-compat: try un-prefixed key (pre-multi-account data) and migrate on read.
- result = delegate.get(key);
- if (result.isPresent()) {
- delegate.put(prefixedKey, result.get());
- delegate.delete(key);
+ // Backward-compat: try un-prefixed key (pre-multi-account data) and migrate on read. The
+ // migration is a write (put then delete), so it runs under the same monitor the explicit-account
+ // migrators hold, or two readers resolving the same legacy key from different account contexts
+ // would each copy it into their own partition and the two copies would then diverge. The hit
+ // path above stays outside the monitor, so only a miss pays for it.
+ synchronized (this) {
+ Optional<V> migratedByAnotherReader = delegate.get(prefixedKey);
+ if (migratedByAnotherReader.isPresent()) {
+ return migratedByAnotherReader;
+ }
+ result = delegate.get(key);
+ if (result.isPresent()) {
+ delegate.put(prefixedKey, result.get());
+ delegate.delete(key);
+ }
+ return result;
}
- return result;
}
@Override
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DestinationForwardingStats.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DestinationForwardingStats.java
new file mode 100644
index 0000000000..150992f85a
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DestinationForwardingStats.java
@@ -0,0 +1,19 @@
+package io.github.hectorvent.floci.services.dynamodb;
+
+/**
+ * An immutable snapshot of a single Kinesis CDC destination's forwarding health, produced by
+ * {@link KinesisStreamingForwarder#forwardingStats()}. Copied under the destination's monitor so
+ * the counters and queue depth are internally consistent.
+ *
+ * @param health one of HEALTHY, RETRYING, GAVE_UP
+ */
+record DestinationForwardingStats(
+ String health,
+ long forwarded,
+ long retried,
+ long dropped,
+ long overflowed,
+ long discardedOnDisable,
+ int queueDepth,
+ String lastErrorType) {
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbJsonHandler.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbJsonHandler.java
index 4b6ad85e91..8b56a83c2c 100644
--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbJsonHandler.java
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbJsonHandler.java
@@ -1853,6 +1853,8 @@ public class DynamoDbJsonHandler {
existing.get().setDestinationStatus("DISABLED");
existing.get().setDestinationStatusDescription("Kinesis streaming is disabled for this table");
dynamoDbService.persistTable(resolvedTableName, table, region);
+ // Stop forwarding and discard buffered CDC records for this now-disabled destination.
+ dynamoDbService.onKinesisStreamingDestinationDisabled(resolvedTableName, streamArn, region);
ObjectNode response = objectMapper.createObjectNode();
response.put("TableName", resolvedTableName);
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java
index 7acb126e75..9b8039ee54 100644
--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java
@@ -480,9 +480,24 @@ public class DynamoDbService implements ResourceProvider {
if (streamService != null) {
streamService.deleteStream(canonicalTableName, region);
}
+ if (kinesisForwarder != null) {
+ // Discard any buffered CDC records and stop draining: the destination stream is gone.
+ kinesisForwarder.onTableDeleted(regionResolver.getAccountId(), region, canonicalTableName);
+ }
LOG.infov("Deleted table: {0}", canonicalTableName);
}
+ /**
+ * Notify the CDC forwarder that a Kinesis streaming destination was disabled so it discards any
+ * buffered records for it and stops draining. {@code tableName} must be the resolved (canonical)
+ * table name, the same value the forward path keys destination state on. No-op without a forwarder.
+ */
+ public void onKinesisStreamingDestinationDisabled(String tableName, String streamArn, String region) {
+ if (kinesisForwarder != null) {
+ kinesisForwarder.onDestinationDisabled(regionResolver.getAccountId(), region, tableName, streamArn);
+ }
+ }
+
public List<String> listTables(String region) {
String prefix = region + "::";
return tableStore.scan(k -> k.startsWith(prefix)).stream()
@@ -571,12 +586,16 @@ public class DynamoDbService implements ResourceProvider {
LOG.tracev("Put item in {0}: key={1} item={2}", canonicalTableName, itemKey, item);
String eventName = existing == null ? "INSERT" : "MODIFY";
+ // Captured in request scope on purpose: this event may be deferred to the batch drain,
+ // and resolving the account inside the lambda would fall back to the default account,
+ // which is the ambient-account bug this commit exists to remove.
+ String ownerAccountId = regionResolver.getAccountId();
Runnable streamEvent = () -> {
if (streamService != null) {
streamService.captureEvent(canonicalTableName, eventName, existing, item, table, region);
}
if (kinesisForwarder != null) {
- kinesisForwarder.forward(eventName, existing, item, table, region);
+ kinesisForwarder.forward(eventName, existing, item, table, region, ownerAccountId);
}
};
if (deferredStreamEvents != null) {
@@ -660,12 +679,16 @@ public class DynamoDbService implements ResourceProvider {
LOG.tracev("Deleted item from {0}: key={1} removed={2}", canonicalTableName, itemKey, removed);
if (removed != null) {
+ // Captured in request scope on purpose: this event may be deferred to the batch drain,
+ // and resolving the account inside the lambda would fall back to the default account,
+ // which is the ambient-account bug this commit exists to remove.
+ String ownerAccountId = regionResolver.getAccountId();
Runnable streamEvent = () -> {
if (streamService != null) {
streamService.captureEvent(canonicalTableName, "REMOVE", removed, null, table, region);
}
if (kinesisForwarder != null) {
- kinesisForwarder.forward("REMOVE", removed, null, table, region);
+ kinesisForwarder.forward("REMOVE", removed, null, table, region, ownerAccountId);
}
};
if (deferredStreamEvents != null) {
@@ -828,12 +851,16 @@ public class DynamoDbService implements ResourceProvider {
LOG.tracev("Updated item in {0}: key={1} updateExpression={2} item={3}",
canonicalTableName, itemKey, updateExpression, item);
+ // Captured in request scope on purpose: this event may be deferred to the batch drain,
+ // and resolving the account inside the lambda would fall back to the default account,
+ // which is the ambient-account bug this commit exists to remove.
+ String ownerAccountId = regionResolver.getAccountId();
Runnable streamEvent = () -> {
if (streamService != null) {
streamService.captureEvent(canonicalTableName, "MODIFY", existing, item, table, region);
}
if (kinesisForwarder != null) {
- kinesisForwarder.forward("MODIFY", existing, item, table, region);
+ kinesisForwarder.forward("MODIFY", existing, item, table, region, ownerAccountId);
}
};
if (deferredStreamEvents != null) {
@@ -1840,7 +1867,9 @@ public class DynamoDbService implements ResourceProvider {
streamService.captureEvent(table.getTableName(), "REMOVE", removed, null, table, region);
}
if (kinesisForwarder != null) {
- kinesisForwarder.forward("REMOVE", removed, null, table, region);
+ // Out of request scope here: pass the table owner's account explicitly so the CDC
+ // record lands in the owner's stream, not the default account's same-named stream.
+ kinesisForwarder.forward("REMOVE", removed, null, table, region, accountId);
}
}
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarder.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarder.java
index fbd11390d6..9b8a67f391 100644
--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarder.java
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarder.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.dynamodb;
+import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
import io.github.hectorvent.floci.services.dynamodb.model.KinesisStreamingDestination;
import io.github.hectorvent.floci.services.dynamodb.model.TableDefinition;
@@ -7,73 +8,416 @@ import io.github.hectorvent.floci.services.kinesis.KinesisService;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
+import jakarta.annotation.PreDestroy;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import org.jboss.logging.Logger;
+import java.time.Duration;
import java.time.Instant;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
-import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+/**
+ * Forwards DynamoDB change events to Kinesis with a <strong>bounded best-effort, in-process retry</strong>
+ * delivery contract (NOT durable at-least-once: the buffer is in memory and lost on restart).
+ *
+ * <p>The write path only ENQUEUES (it never performs Kinesis I/O or blocks: a slow/dead destination can
+ * neither stall a DynamoDB write nor the TTL sweep). A per-destination single-flight drain, scheduled on a
+ * small daemon pool, performs the actual {@code putRecord} outside the destination lock and:
+ * <ul>
+ * <li>retries transient/unknown failures with capped exponential backoff (≤{@value #MAX_PROBES} probes),</li>
+ * <li>drops a record immediately on a deterministic terminal failure (invalid argument / validation /
+ * serialization) so a poison record cannot wedge the queue,</li>
+ * <li>gives up the whole per-destination episode after the probe budget (dropping the buffered records
+ * and marking the destination GAVE_UP until the next successful send),</li>
+ * <li>preserves per-destination FIFO across retries (stronger than AWS, which may reorder/duplicate),</li>
+ * <li>evicts the oldest buffered record on overflow (max {@value #MAX_BUFFERED} per destination).</li>
+ * </ul>
+ *
+ * <p>The destination's {@code DestinationStatus} is intentionally NOT auto-demoted (AWS has no
+ * delivery-failure status); delivery health is surfaced separately via {@link #forwardingStats()}.
+ * Records dropped/retried are counted and their last error retained for observability.
+ */
@ApplicationScoped
public class KinesisStreamingForwarder {
private static final Logger LOG = Logger.getLogger(KinesisStreamingForwarder.class);
+ static final int MAX_BUFFERED = 1000;
+ static final int MAX_PROBES = 10;
+ static final long INITIAL_BACKOFF_MS = 250;
+ static final long MAX_BACKOFF_MS = 8000;
+
+ /** Schedules drain runs; abstracted so tests can drive drains deterministically. */
+ interface Scheduler {
+ void schedule(Runnable task, long delayMillis);
+
+ void shutdown();
+ }
+
+ enum Health { HEALTHY, RETRYING, GAVE_UP }
+
+ record PendingRecord(String accountId, String region, String streamName, String tableName,
+ byte[] data, String partitionKey, Instant enqueuedAt) {
+ }
+
+ /** Per-destination delivery state. Every field is accessed only while holding {@code this} monitor. */
+ static final class DestinationState {
+ final String key;
+ final String accountId;
+ final String region;
+ final String tableName;
+ final String streamArn;
+ final ArrayDeque<PendingRecord> queue = new ArrayDeque<>();
+ long epoch;
+ int consecutiveFailedProbes;
+ boolean drainScheduled;
+ long forwarded;
+ long retried;
+ long dropped;
+ long overflowed;
+ long discardedOnDisable;
+ String lastErrorType;
+ String lastErrorMessage;
+ Health health = Health.HEALTHY;
+
+ DestinationState(String key, String accountId, String region, String tableName, String streamArn) {
+ this.key = key;
+ this.accountId = accountId;
+ this.region = region;
+ this.tableName = tableName;
+ this.streamArn = streamArn;
+ }
+ }
+
private final KinesisService kinesisService;
private final ObjectMapper objectMapper;
- private final AtomicLong forwardFailures = new AtomicLong();
+ private final Scheduler scheduler;
+ private final ConcurrentHashMap<String, DestinationState> states = new ConcurrentHashMap<>();
@Inject
public KinesisStreamingForwarder(KinesisService kinesisService, ObjectMapper objectMapper) {
+ this(kinesisService, objectMapper, defaultScheduler());
+ }
+
+ /** Test seam: inject a deterministic Scheduler so drains run on demand rather than on real timers. */
+ KinesisStreamingForwarder(KinesisService kinesisService, ObjectMapper objectMapper, Scheduler scheduler) {
this.kinesisService = kinesisService;
this.objectMapper = objectMapper;
+ this.scheduler = scheduler;
+ }
+
+ private static Scheduler defaultScheduler() {
+ ScheduledExecutorService exec = Executors.newScheduledThreadPool(2, r -> {
+ Thread t = new Thread(r, "cdc-forward-drain");
+ t.setDaemon(true);
+ return t;
+ });
+ return new Scheduler() {
+ @Override
+ public void schedule(Runnable task, long delayMillis) {
+ exec.schedule(task, delayMillis, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void shutdown() {
+ exec.shutdownNow();
+ }
+ };
}
+ // ──────────────────────────── Enqueue (write path: never blocks, never throws) ────────────────────────────
+
public void forward(String eventName, JsonNode oldItem, JsonNode newItem,
- TableDefinition table, String region) {
+ TableDefinition table, String region, String ownerAccountId) {
List<KinesisStreamingDestination> destinations = table.getKinesisStreamingDestinations();
- if (destinations == null || destinations.isEmpty()) return;
-
+ if (destinations == null || destinations.isEmpty()) {
+ return;
+ }
Instant now = Instant.now();
JsonNode sourceItem = newItem != null ? newItem : oldItem;
ObjectNode keys = buildKeys(sourceItem, table);
+ byte[] data;
+ String partitionKey;
+ try {
+ ObjectNode payload = buildPayload(eventName, keys, newItem, oldItem, table.getTableName(), region, now);
+ data = objectMapper.writeValueAsBytes(payload);
+ partitionKey = extractPartitionKey(keys, table);
+ } catch (Exception e) {
+ // Serialization is a deterministic terminal failure: nothing can be forwarded for this event.
+ for (KinesisStreamingDestination dest : destinations) {
+ if (!"ACTIVE".equals(dest.getDestinationStatus())) {
+ continue;
+ }
+ DestinationState st = stateFor(ownerAccountId, region, table.getTableName(), dest.getStreamArn());
+ synchronized (st) {
+ st.dropped++;
+ recordError(st, e);
+ }
+ }
+ LOG.errorv(e, "Dropped DynamoDB CDC event for table {0}: failed to serialize payload", table.getTableName());
+ return;
+ }
+
for (KinesisStreamingDestination dest : destinations) {
- if (!"ACTIVE".equals(dest.getDestinationStatus())) continue;
+ if (!"ACTIVE".equals(dest.getDestinationStatus())) {
+ continue;
+ }
+ String streamName = extractStreamName(dest.getStreamArn());
+ DestinationState st = stateFor(ownerAccountId, region, table.getTableName(), dest.getStreamArn());
+ PendingRecord rec = new PendingRecord(ownerAccountId, region, streamName, table.getTableName(),
+ data, partitionKey, now);
+ synchronized (st) {
+ if (st.queue.size() >= MAX_BUFFERED) {
+ st.queue.pollFirst(); // drop OLDEST
+ st.overflowed++;
+ st.dropped++;
+ LOG.warnv("CDC buffer full for stream {0} (>{1} records); dropped the oldest",
+ st.streamArn, MAX_BUFFERED);
+ }
+ st.queue.addLast(rec);
+ if (!st.drainScheduled) {
+ scheduleDrain(st, 0);
+ }
+ }
+ }
+ }
- try {
- ObjectNode payload = buildPayload(eventName, keys, newItem, oldItem,
- table.getTableName(), region, now);
- byte[] data = objectMapper.writeValueAsBytes(payload);
+ private DestinationState stateFor(String accountId, String region, String tableName, String streamArn) {
+ String key = accountId + "|" + region + "|" + tableName + "|" + streamArn;
+ return states.computeIfAbsent(key,
+ k -> new DestinationState(k, accountId, region, tableName, streamArn));
+ }
- String partitionKey = extractPartitionKey(keys, table);
- String streamName = extractStreamName(dest.getStreamArn());
+ // ──────────────────────────── Drain (per-destination single-flight; I/O outside the lock) ────────────────────────────
- kinesisService.putRecord(streamName, data, partitionKey, region);
- LOG.debugv("Forwarded DynamoDB event to Kinesis stream {0}: {1} on {2}",
- streamName, eventName, table.getTableName());
+ private void drain(DestinationState st, long scheduledEpoch) {
+ while (true) {
+ PendingRecord head;
+ synchronized (st) {
+ if (scheduledEpoch != st.epoch) {
+ return; // this destination was disabled/deleted; a fresh episode (if any) owns the queue
+ }
+ if (st.queue.isEmpty()) {
+ st.drainScheduled = false;
+ if (st.health != Health.GAVE_UP) {
+ st.health = Health.HEALTHY;
+ }
+ return;
+ }
+ // Lease the head OUT of the queue for the duration of the send. A concurrent overflow
+ // eviction can then only drop a still-buffered record, never the one being delivered,
+ // and completion below reconciles exactly this leased record (no blind pollFirst).
+ head = st.queue.pollFirst();
+ }
+
+ boolean success = false;
+ boolean terminal = false;
+ Throwable failure = null;
+ try {
+ if (head.accountId() != null) {
+ kinesisService.putRecordForAccount(head.accountId(), head.streamName(), head.data(),
+ head.partitionKey(), head.region());
+ } else {
+ kinesisService.putRecord(head.streamName(), head.data(), head.partitionKey(), head.region());
+ }
+ success = true;
+ } catch (AwsException e) {
+ failure = e;
+ terminal = isTerminal(e);
} catch (Exception e) {
- // Do NOT rethrow: put/delete/update persist before forwarding, so a rethrow would
- // turn a committed write into a spurious 5xx; the TTL sweep forwards mid-loop and
- // persists only after the batch, so a rethrow would abort the sweep. Surface the
- // drop loudly (ERROR + stack trace) and count it for process-local diagnostics.
- forwardFailures.incrementAndGet();
- LOG.errorv(e, "Dropped DynamoDB CDC event: failed to forward {0} on table {1} to Kinesis destination {2}",
- eventName, table.getTableName(), dest.getStreamArn());
+ failure = e; // unknown/operational failure -> retryable
+ }
+
+ synchronized (st) {
+ if (scheduledEpoch != st.epoch) {
+ // Torn down mid-send: the leased record is discarded along with the rest of the buffer.
+ // The dispatched send may still complete, an inherent and harmless race for an in-memory
+ // emulator (teardown stops all FUTURE sends; it cannot recall one already in flight).
+ return;
+ }
+ if (success) {
+ st.forwarded++;
+ st.consecutiveFailedProbes = 0;
+ st.health = Health.HEALTHY;
+ continue;
+ }
+ recordError(st, failure);
+ if (terminal) {
+ // The leased poison record is already out of the queue: drop it (do NOT wedge) and let
+ // the records behind it flow. Its probe budget dies with it: the next record earns a fresh one.
+ st.dropped++;
+ st.consecutiveFailedProbes = 0;
+ LOG.errorv(failure, "Dropped DynamoDB CDC record (terminal {0}) for stream {1}",
+ st.lastErrorType, st.streamArn);
+ continue;
+ }
+ // retryable: return the leased record to the head so per-destination FIFO is preserved.
+ st.queue.addFirst(head);
+ st.retried++;
+ st.consecutiveFailedProbes++;
+ st.health = Health.RETRYING;
+ if (st.consecutiveFailedProbes >= MAX_PROBES) {
+ long gaveUp = st.queue.size();
+ st.dropped += gaveUp;
+ st.queue.clear();
+ st.health = Health.GAVE_UP;
+ st.drainScheduled = false;
+ // Reset the budget so a later event starts a fresh episode: a transient outage must
+ // not permanently wedge delivery: the next forward re-attempts and self-heals.
+ st.consecutiveFailedProbes = 0;
+ LOG.errorv("CDC delivery gave up for stream {0} after {1} attempts; dropped {2} buffered "
+ + "record(s). Last error: {3}",
+ st.streamArn, MAX_PROBES, gaveUp, st.lastErrorMessage);
+ return;
+ }
+ scheduleDrain(st, backoff(st.consecutiveFailedProbes));
+ return;
+ }
+ }
+ }
+
+ /**
+ * Schedule a drain run. The caller MUST hold {@code st}'s monitor. A scheduler fault (e.g.
+ * {@link java.util.concurrent.RejectedExecutionException} once the executor is shutting down) must
+ * never escape into the DynamoDB write path and must not wedge the destination: on failure the
+ * {@code drainScheduled} flag is cleared so a later event can schedule a fresh drain.
+ */
+ private void scheduleDrain(DestinationState st, long delayMillis) {
+ st.drainScheduled = true;
+ long ep = st.epoch;
+ try {
+ scheduler.schedule(() -> drain(st, ep), delayMillis);
+ } catch (RuntimeException rex) {
+ st.drainScheduled = false;
+ LOG.warnv(rex, "CDC drain scheduling failed for stream {0}; buffered records await a later event",
+ st.streamArn);
+ }
+ }
+
+ private static boolean isTerminal(AwsException e) {
+ String code = e.getErrorCode();
+ return "InvalidArgumentException".equals(code) || "ValidationException".equals(code);
+ }
+
+ private static long backoff(int probe) {
+ long shifted = INITIAL_BACKOFF_MS * (1L << Math.min(probe - 1, 20));
+ return Math.min(MAX_BACKOFF_MS, shifted);
+ }
+
+ private static void recordError(DestinationState st, Throwable failure) {
+ st.lastErrorType = failure instanceof AwsException a && a.getErrorCode() != null
+ ? a.getErrorCode() : failure.getClass().getSimpleName();
+ st.lastErrorMessage = failure.getMessage();
+ }
+
+ // ──────────────────────────── Lifecycle ────────────────────────────
+
+ /**
+ * Disable of a specific destination: discard its buffer and stop any in-flight drain (epoch bump).
+ * The state is RETAINED (not removed) so its cumulative delivery stats stay inspectable and a
+ * subsequent re-enable reuses it; only a table deletion removes it.
+ */
+ public void onDestinationDisabled(String accountId, String region, String tableName, String streamArn) {
+ DestinationState st = states.get(accountId + "|" + region + "|" + tableName + "|" + streamArn);
+ if (st != null) {
+ invalidate(st);
+ }
+ }
+
+ /** Table deletion: discard all of its destinations' buffers, stop their drains, and drop their state. */
+ public void onTableDeleted(String accountId, String region, String tableName) {
+ String prefix = accountId + "|" + region + "|" + tableName + "|";
+ states.entrySet().removeIf(entry -> {
+ if (entry.getKey().startsWith(prefix)) {
+ invalidate(entry.getValue());
+ return true;
}
+ return false;
+ });
+ }
+
+ private void invalidate(DestinationState st) {
+ synchronized (st) {
+ st.epoch++; // a scheduled drain from the old epoch will abort on its next check
+ st.discardedOnDisable += st.queue.size();
+ st.queue.clear();
+ st.drainScheduled = false;
+ st.consecutiveFailedProbes = 0; // a re-enable starts a fresh delivery episode
+ st.health = Health.HEALTHY;
}
}
- /** Process-local count of CDC events dropped because forwarding to Kinesis threw. */
+ @PreDestroy
+ public void shutdown() {
+ scheduler.shutdown();
+ }
+
+ /** Blocks until every destination's queue is empty and no drain is scheduled, or the timeout elapses. */
+ public boolean awaitIdle(Duration timeout) {
+ long deadlineNanos = System.nanoTime() + timeout.toNanos();
+ while (System.nanoTime() < deadlineNanos) {
+ boolean idle = true;
+ for (DestinationState st : states.values()) {
+ synchronized (st) {
+ if (!st.queue.isEmpty() || st.drainScheduled) {
+ idle = false;
+ break;
+ }
+ }
+ }
+ if (idle) {
+ return true;
+ }
+ try {
+ Thread.sleep(5);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return false;
+ }
+ }
+ return false;
+ }
+
+ // ──────────────────────────── Observability ────────────────────────────
+
+ /** Total CDC records permanently dropped across all destinations (terminal, give-up, or overflow). */
public long getForwardFailureCount() {
- return forwardFailures.get();
+ long sum = 0;
+ for (DestinationState st : states.values()) {
+ synchronized (st) {
+ sum += st.dropped;
+ }
+ }
+ return sum;
}
+ /** Immutable per-destination delivery-health snapshots (each copied under its destination lock). */
+ List<DestinationForwardingStats> forwardingStats() {
+ List<DestinationForwardingStats> out = new ArrayList<>();
+ for (DestinationState st : states.values()) {
+ synchronized (st) {
+ out.add(new DestinationForwardingStats(st.health.name(), st.forwarded,
+ st.retried, st.dropped, st.overflowed, st.discardedOnDisable,
+ st.queue.size(), st.lastErrorType));
+ }
+ }
+ return out;
+ }
+
+ // ──────────────────────────── Payload construction (unchanged) ────────────────────────────
+
private ObjectNode buildPayload(String eventName, JsonNode keys,
- JsonNode newImage, JsonNode oldImage,
- String tableName, String region, Instant timestamp) {
+ JsonNode newImage, JsonNode oldImage,
+ String tableName, String region, Instant timestamp) {
ObjectNode payload = objectMapper.createObjectNode();
payload.put("awsRegion", region);
payload.put("eventID", UUID.randomUUID().toString());
@@ -103,7 +447,9 @@ public class KinesisStreamingForwarder {
private ObjectNode buildKeys(JsonNode item, TableDefinition table) {
ObjectNode keys = objectMapper.createObjectNode();
- if (item == null) return keys;
+ if (item == null) {
+ return keys;
+ }
for (KeySchemaElement ks : table.getKeySchema()) {
String attrName = ks.getAttributeName();
if (item.has(attrName)) {
@@ -114,19 +460,31 @@ public class KinesisStreamingForwarder {
}
private String extractPartitionKey(JsonNode keys, TableDefinition table) {
- if (keys == null || keys.isEmpty()) return "default";
+ if (keys == null || keys.isEmpty()) {
+ return "default";
+ }
String pkName = table.getPartitionKeyName();
JsonNode pkValue = keys.get(pkName);
- if (pkValue == null) return "default";
- if (pkValue.has("S")) return pkValue.get("S").asText();
- if (pkValue.has("N")) return pkValue.get("N").asText();
- if (pkValue.has("B")) return pkValue.get("B").asText();
+ if (pkValue == null) {
+ return "default";
+ }
+ if (pkValue.has("S")) {
+ return pkValue.get("S").asText();
+ }
+ if (pkValue.has("N")) {
+ return pkValue.get("N").asText();
+ }
+ if (pkValue.has("B")) {
+ return pkValue.get("B").asText();
+ }
return pkValue.toString();
}
private String extractStreamName(String streamArn) {
int idx = streamArn.lastIndexOf('/');
- if (idx >= 0) return streamArn.substring(idx + 1);
+ if (idx >= 0) {
+ return streamArn.substring(idx + 1);
+ }
return streamArn;
}
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/kinesis/KinesisService.java b/src/main/java/io/github/hectorvent/floci/services/kinesis/KinesisService.java
index 2809c1b23f..75bde14c4e 100644
--- a/src/main/java/io/github/hectorvent/floci/services/kinesis/KinesisService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/kinesis/KinesisService.java
@@ -557,31 +557,65 @@ public class KinesisService implements ResourceProvider {
public PutRecordResult putRecordWithShardId(String streamName, byte[] data, String partitionKey,
String explicitHashKey, String region) {
+ return putRecordInternal(null, streamName, data, partitionKey, explicitHashKey, region);
+ }
+
+ /**
+ * Append a record on behalf of a specific account. Used by producers that run outside request scope
+ * (the DynamoDB TTL sweep and DynamoDB→Kinesis CDC forwarding) so the record lands in the table
+ * owner's stream rather than the ambient/default account's same-named stream. A {@code null}
+ * accountId behaves exactly like {@link #putRecord}.
+ */
+ public String putRecordForAccount(String accountId, String streamName, byte[] data, String partitionKey,
+ String region) {
+ return putRecordInternal(accountId, streamName, data, partitionKey, null, region).sequenceNumber();
+ }
+
+ private PutRecordResult putRecordInternal(String accountId, String streamName, byte[] data,
+ String partitionKey, String explicitHashKey, String region) {
String key = regionKey(region, streamName);
- KinesisStream stream = resolveStream(streamName, region);
- validateExplicitHashKey(explicitHashKey);
- validateRecordSize(stream, data, partitionKey);
putRecordBeforeLockHook.run();
// Serialize shard selection, sequence allocation, append and persistence per stream so
// concurrent producers can neither lose an append (plain ArrayList) nor store records
// out of sequence order (which would break AT_/AFTER_SEQUENCE_NUMBER scans and WAL replay).
+ // The lock is keyed on region+streamName (NOT the account): an account-qualified key would stop
+ // explicit-account appends from serializing against request-path producers on the same stream,
+ // reintroducing the append race. Cross-account same-named streams therefore share one monitor
+ // (over-coarse but safe).
synchronized (lockFor(key)) {
- // Re-resolve under the lock. deleteStream holds this same monitor, so a stream deleted
- // between the resolve above and here is now absent from the store. Appending to and
- // persisting the stale pre-delete instance would resurrect a deleted stream, so bind to
- // the instance currently in the store and fail like any missing stream if it is gone.
- KinesisStream current = resolveStream(streamName, region);
+ // Resolve under the lock, not before it. Resolving a legacy unprefixed stream MIGRATES it,
+ // which is a write: the explicit-account path moves it into the account partition behind an
+ // ownership predicate, while AccountAwareStorageBackend.get adopts it into the request
+ // account with no ownership check and no synchronization. Resolved outside this monitor,
+ // an ambient producer and the owner's producer could migrate the same stream concurrently
+ // and fork it across partitions. deleteStream holds this same monitor too, so a stream
+ // deleted before we get here is absent and this fails like any other missing stream rather
+ // than appending to a stale instance and resurrecting it.
+ KinesisStream current = resolveStreamForAccountMigrating(accountId, streamName, region);
+ // Ordered exactly as before this moved under the monitor: a missing stream still reports
+ // ResourceNotFoundException ahead of an invalid ExplicitHashKey, and the size check runs
+ // against the same lock-protected instance that shard selection and the append use.
+ validateExplicitHashKey(explicitHashKey);
+ validateRecordSize(current, data, partitionKey);
KinesisShard shard = selectShard(current, partitionKey, explicitHashKey);
String sequenceNumber = String.valueOf(sequenceGenerator.incrementAndGet());
putRecordAppendHook.run();
KinesisRecord record = new KinesisRecord(data, partitionKey, sequenceNumber, Instant.now());
shard.addRecord(record);
- store.put(key, current);
+ persistStream(accountId, key, current);
return new PutRecordResult(sequenceNumber, shard.getShardId());
}
}
+ private void persistStream(String accountId, String key, KinesisStream stream) {
+ if (accountId != null && store instanceof AccountAwareStorageBackend<KinesisStream> aware) {
+ aware.putForAccount(accountId, key, stream);
+ } else {
+ store.put(key, stream);
+ }
+ }
+
public String getShardIterator(String streamName, String shardId, String type, String sequenceNumber, String region) {
return getShardIterator(streamName, shardId, type, sequenceNumber, null, region);
}
@@ -806,6 +840,24 @@ public class KinesisService implements ResourceProvider {
return resolveStream(streamName, region);
}
+ /**
+ * Resolve for an out-of-request-scope PRODUCER (CDC forwarding, TTL sweep), migrating a legacy
+ * unprefixed stream into the account partition on write, mirroring the ambient {@code store.get()}
+ * fallback the producer used before account-scoping, so an installation with an existing CDC
+ * destination and an unmigrated stream keeps forwarding after upgrade. Ownership is validated by the
+ * stream's own accountId/ARN so one tenant cannot adopt another's unprefixed stream.
+ */
+ private KinesisStream resolveStreamForAccountMigrating(String accountId, String streamName, String region) {
+ if (accountId != null && store instanceof AccountAwareStorageBackend<KinesisStream> aware) {
+ return aware.getForAccountMigratingLegacy(accountId, regionKey(region, streamName),
+ s -> accountId.equals(s.getAccountId())
+ || (s.getStreamArn() != null && s.getStreamArn().contains(":" + accountId + ":")))
+ .orElseThrow(() -> new AwsException("ResourceNotFoundException",
+ "Stream " + streamName + " not found", 400));
+ }
+ return resolveStream(streamName, region);
+ }
+
public String getShardIteratorForAccount(String accountId, String streamName, String shardId,
String type, String sequenceNumber, String region) {
KinesisStream stream = resolveStreamForAccount(accountId, streamName, region);
diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackendTest.java b/src/test/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackendTest.java
index d789ea5527..aa06cd89b6 100644
--- a/src/test/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackendTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackendTest.java
@@ -6,8 +6,16 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Predicate;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
class AccountAwareStorageBackendTest {
@@ -210,4 +218,66 @@ class AccountAwareStorageBackendTest {
new AccountAwareStorageBackend.AccountEntry<>("000000000000", "us-east-1::legacy", "legacy")),
Set.copyOf(storage.scanAllAccountEntries(key -> key.startsWith("us-east-1::"))));
}
+
+ /**
+ * {@code get} and {@code getForAccountMigratingLegacy} both migrate a pre-multi-account entry, and
+ * both delete the legacy key once they have copied it. Unless they hold the same monitor, two readers
+ * resolving that key from different account contexts each copy it into their own partition and the two
+ * copies then diverge. Made deterministic by parking the first reader inside its legacy read, at the
+ * exact point where the unsynchronized version let the second reader in.
+ */
+ @Test
+ void concurrentLegacyMigrationsDoNotForkTheEntryAcrossAccounts() throws Exception {
+ InMemoryStorage<String, String> raw = new InMemoryStorage<>();
+ raw.put("resource", "v");
+
+ CountDownLatch parked = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+ StorageBackend<String, String> parking = new StorageBackend<>() {
+ @Override public void put(String k, String v) { raw.put(k, v); }
+ @Override public void delete(String k) { raw.delete(k); }
+ @Override public List<String> scan(Predicate<String> f) { return raw.scan(f); }
+ @Override public Set<String> keys() { return raw.keys(); }
+ @Override public void flush() { raw.flush(); }
+ @Override public void load() { raw.load(); }
+ @Override public void clear() { raw.clear(); }
+ @Override public Optional<String> get(String k) {
+ if ("resource".equals(k) && parked.getCount() > 0) {
+ parked.countDown();
+ try {
+ release.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ return raw.get(k);
+ }
+ };
+ AccountAwareStorageBackend<String> storage =
+ new AccountAwareStorageBackend<>(parking, null, "111111111111");
+
+ ExecutorService pool = Executors.newFixedThreadPool(2);
+ try {
+ Future<?> ambient = pool.submit(() -> storage.get("resource"));
+ assertTrue(parked.await(2, TimeUnit.SECONDS), "the ambient reader reached its legacy read");
+
+ Future<?> owner = pool.submit(
+ () -> storage.getForAccountMigratingLegacy("222222222222", "resource", v -> true));
+ // The owner must be unable to proceed: without a shared monitor it would migrate a second copy.
+ assertThrows(TimeoutException.class, () -> owner.get(300, TimeUnit.MILLISECONDS),
+ "the second migration ran concurrently with the first");
+
+ release.countDown();
+ ambient.get(2, TimeUnit.SECONDS);
+ owner.get(2, TimeUnit.SECONDS);
+ } finally {
+ pool.shutdownNow();
+ }
+
+ assertTrue(raw.get("resource").isEmpty(), "the legacy key is consumed exactly once");
+ long copies = List.of("111111111111/resource", "222222222222/resource").stream()
+ .filter(k -> raw.get(k).isPresent())
+ .count();
+ assertEquals(1, copies, "the entry landed in exactly one account partition, not forked into both");
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbConcurrencyIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbConcurrencyIntegrationTest.java
index 37d88e121c..046c16bbd3 100644
--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbConcurrencyIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbConcurrencyIntegrationTest.java
@@ -542,6 +542,11 @@ class DynamoDbConcurrencyIntegrationTest {
});
assertTrue(errors.isEmpty(), () -> "unexpected errors: " + errors);
+ // CDC delivery is asynchronous (bounded best-effort retry queue): wait for the background drain to
+ // flush every buffered record before reading the shard, otherwise this reads a partial stream.
+ assertTrue(forwarder.awaitIdle(java.time.Duration.ofSeconds(10)),
+ "the CDC drain must flush all buffered records");
+
// Paginated drain of the single shard.
String shardId = kinesis.describeStream("cdc-stream", region).getShards().getFirst().getShardId();
String iterator = kinesis.getShardIterator("cdc-stream", shardId, "TRIM_HORIZON", null, region);
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbKinesisStreamingIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbKinesisStreamingIntegrationTest.java
index d394f758a5..71070e32ba 100644
--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbKinesisStreamingIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbKinesisStreamingIntegrationTest.java
@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.dynamodb;
import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.response.Response;
+import jakarta.inject.Inject;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
@@ -12,6 +13,7 @@ import org.junit.jupiter.api.TestMethodOrder;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
+import java.time.Duration;
import java.util.Base64;
import static io.restassured.RestAssured.given;
@@ -26,11 +28,20 @@ class DynamoDbKinesisStreamingIntegrationTest {
private static final String KINESIS_CONTENT_TYPE = "application/x-amz-json-1.1";
private static String kinesisStreamArn;
+ // CDC forwarding is asynchronous (bounded best-effort retry queue): after a write enqueues, the send
+ // happens on a background drain. Await it before reading the stream so these assertions are deterministic.
+ @Inject
+ KinesisStreamingForwarder forwarder;
+
@BeforeAll
static void configureRestAssured() {
RestAssuredJsonUtils.configureAwsContentTypes();
}
+ private void awaitCdcDelivery() {
+ forwarder.awaitIdle(Duration.ofSeconds(5));
+ }
+
@Test
@Order(1)
void setupKinesisStream() {
@@ -178,6 +189,7 @@ class DynamoDbKinesisStreamingIntegrationTest {
.then()
.statusCode(200);
+ awaitCdcDelivery();
String shardIterator = given()
.header("X-Amz-Target", "Kinesis_20131202.GetShardIterator")
.contentType(KINESIS_CONTENT_TYPE)
@@ -248,6 +260,7 @@ class DynamoDbKinesisStreamingIntegrationTest {
.then()
.statusCode(200);
+ awaitCdcDelivery();
String shardIterator = given()
.header("X-Amz-Target", "Kinesis_20131202.GetShardIterator")
.contentType(KINESIS_CONTENT_TYPE)
@@ -292,6 +305,7 @@ class DynamoDbKinesisStreamingIntegrationTest {
.then()
.statusCode(200);
+ awaitCdcDelivery();
String shardIterator = given()
.header("X-Amz-Target", "Kinesis_20131202.GetShardIterator")
.contentType(KINESIS_CONTENT_TYPE)
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbTtlForwardingTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbTtlForwardingTest.java
index ea52c227b9..cc0cf2afa3 100644
--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbTtlForwardingTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbTtlForwardingTest.java
@@ -17,6 +17,7 @@ import io.github.hectorvent.floci.services.kinesis.model.KinesisRecord;
import io.github.hectorvent.floci.services.kinesis.model.KinesisStream;
import org.junit.jupiter.api.Test;
+import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashMap;
@@ -27,15 +28,30 @@ import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verifyNoInteractions;
import static org.mockito.Mockito.when;
/**
* TTL sweep must forward a REMOVE per expired item to Kinesis (happy path) and must isolate
- * forwarding failures: a throwing forwarder cannot abort the sweep or block persistence.
- * See issue #571.
+ * forwarding from Kinesis health: because forwarding is now enqueue-only (delivered/retried on a
+ * background drain, see {@link KinesisStreamingForwarderTest}), a broken Kinesis cannot abort the
+ * sweep, block persistence, or silently drop the change events. See issue #571.
*/
class DynamoDbTtlForwardingTest {
+ /** A Scheduler that never runs the drain, so buffered CDC records stay put for inspection. */
+ private static final KinesisStreamingForwarder.Scheduler NO_DRAIN =
+ new KinesisStreamingForwarder.Scheduler() {
+ @Override
+ public void schedule(Runnable task, long delayMillis) {
+ // intentionally never executes the drain
+ }
+
+ @Override
+ public void shutdown() {
+ }
+ };
+
private static final String REGION = "us-east-1";
private final ObjectMapper mapper = new ObjectMapper();
@@ -109,6 +125,8 @@ class DynamoDbTtlForwardingTest {
table.getKinesisStreamingDestinations().add(new KinesisStreamingDestination(stream.getStreamArn()));
svc.deleteExpiredItems();
+ // Delivery is asynchronous now; wait for the background drain to flush before reading the stream.
+ assertTrue(forwarder.awaitIdle(Duration.ofSeconds(5)), "CDC forwards should drain promptly");
List<KinesisRecord> drained = drain(kinesis, "ttl-stream");
assertEquals(expired, drained.size(), "one Kinesis record per expired item");
@@ -131,9 +149,9 @@ class DynamoDbTtlForwardingTest {
StorageBackend<String, Map<String, JsonNode>> itemStore = new InMemoryStorage<>();
KinesisService throwingKinesis = mock(KinesisService.class);
- when(throwingKinesis.putRecord(anyString(), any(byte[].class), anyString(), anyString()))
- .thenThrow(new RuntimeException("kinesis unavailable"));
- KinesisStreamingForwarder forwarder = new KinesisStreamingForwarder(throwingKinesis, mapper);
+ // No-drain scheduler: the sweep only ENQUEUES, so Kinesis is never touched during the sweep at
+ // all. (Retry/give-up drop accounting on the drain is covered by KinesisStreamingForwarderTest.)
+ KinesisStreamingForwarder forwarder = new KinesisStreamingForwarder(throwingKinesis, mapper, NO_DRAIN);
DynamoDbService svc = new DynamoDbService(
tableStore, itemStore, new RegionResolver("us-east-1", "000000000000"), null, forwarder);
@@ -143,11 +161,16 @@ class DynamoDbTtlForwardingTest {
table.getKinesisStreamingDestinations().add(new KinesisStreamingDestination(
"arn:aws:kinesis:us-east-1:000000000000:stream/ttl-stream"));
- // A forwarding failure must not propagate out of the sweep.
+ // A dead Kinesis must not propagate out of the sweep, and must not be called synchronously by it.
assertDoesNotThrow(svc::deleteExpiredItems);
-
- assertEquals(expired, forwarder.getForwardFailureCount(),
- "each failed forward must be counted, not swallowed silently");
+ verifyNoInteractions(throwingKinesis);
+
+ // Every expired change event is safely BUFFERED (not silently dropped): the fix for the
+ // drop-on-forward-exception gap. The drain would deliver these once Kinesis recovers.
+ List<DestinationForwardingStats> stats = forwarder.forwardingStats();
+ assertEquals(1, stats.size());
+ assertEquals(expired, stats.get(0).queueDepth(), "each expired item's REMOVE must be enqueued");
+ assertEquals(0L, forwarder.getForwardFailureCount(), "buffered records are not (yet) dropped");
// Removed in-memory AND persisted: a fresh service over the same stores must not see them.
for (int i = 0; i < expired; i++) {
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarderTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarderTest.java
index 04800d1533..79b7415f7a 100644
--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarderTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarderTest.java
@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.dynamodb;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
+import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.services.dynamodb.model.AttributeDefinition;
import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
import io.github.hectorvent.floci.services.dynamodb.model.KinesisStreamingDestination;
@@ -10,96 +11,496 @@ import io.github.hectorvent.floci.services.kinesis.KinesisService;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.mockito.ArgumentMatchers.*;
-import static org.mockito.Mockito.*;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoInteractions;
+import static org.mockito.Mockito.when;
+/**
+ * The CDC delivery contract (issue: DynamoDB→Kinesis records dropped on forward exception).
+ *
+ * <p>{@code forward} is proven ENQUEUE-ONLY (no synchronous Kinesis I/O, never throws); the actual send,
+ * retry, terminal-drop, give-up, overflow, FIFO, and lifecycle behaviour are exercised deterministically
+ * by driving a manual {@link KinesisStreamingForwarder.Scheduler} step by step: no real timers, no sleeps
+ * in the delivery assertions.
+ */
class KinesisStreamingForwarderTest {
+ private static final String ACCOUNT = "000000000000";
+ private static final String STREAM_ARN = "arn:aws:kinesis:us-east-1:000000000000:stream/test-stream";
+
private KinesisService kinesisService;
- private KinesisStreamingForwarder forwarder;
private ObjectMapper objectMapper;
+ private ManualScheduler scheduler;
+ private KinesisStreamingForwarder forwarder;
@BeforeEach
void setUp() {
kinesisService = mock(KinesisService.class);
objectMapper = new ObjectMapper();
- forwarder = new KinesisStreamingForwarder(kinesisService, objectMapper);
+ scheduler = new ManualScheduler();
+ forwarder = new KinesisStreamingForwarder(kinesisService, objectMapper, scheduler);
+ }
+
+ // ──────────────────────────── helpers ────────────────────────────
+
+ /** A Scheduler the test drives by hand: tasks queue up and run only when the test asks. */
+ static final class ManualScheduler implements KinesisStreamingForwarder.Scheduler {
+ final ArrayDeque<Runnable> tasks = new ArrayDeque<>();
+ final List<Long> delays = new ArrayList<>();
+ boolean shutdown;
+
+ @Override
+ public void schedule(Runnable task, long delayMillis) {
+ tasks.addLast(task);
+ delays.add(delayMillis);
+ }
+
+ @Override
+ public void shutdown() {
+ shutdown = true;
+ }
+
+ boolean runOne() {
+ Runnable r = tasks.pollFirst();
+ if (r == null) {
+ return false;
+ }
+ r.run();
+ return true;
+ }
+
+ int runUntilQuiet() {
+ int steps = 0;
+ while (steps < 20_000 && runOne()) {
+ steps++;
+ }
+ return steps;
+ }
+ }
+
+ /** A Scheduler that rejects while {@code reject} is set (executor-shutdown simulation), else queues. */
+ static final class RejectingScheduler implements KinesisStreamingForwarder.Scheduler {
+ final ArrayDeque<Runnable> tasks = new ArrayDeque<>();
+ boolean reject = true;
+
+ @Override
+ public void schedule(Runnable task, long delayMillis) {
+ if (reject) {
+ throw new RejectedExecutionException("executor is shutting down");
+ }
+ tasks.addLast(task);
+ }
+
+ @Override
+ public void shutdown() {
+ }
+
+ boolean runOne() {
+ Runnable r = tasks.pollFirst();
+ if (r == null) {
+ return false;
+ }
+ r.run();
+ return true;
+ }
}
- private TableDefinition createTable(String tableName) {
- return new TableDefinition(tableName,
+ private TableDefinition tableWithDestination(String... streamArns) {
+ TableDefinition table = new TableDefinition("test-table",
List.of(new KeySchemaElement("pk", "HASH")),
List.of(new AttributeDefinition("pk", "S")));
+ for (String arn : streamArns) {
+ table.getKinesisStreamingDestinations().add(new KinesisStreamingDestination(arn));
+ }
+ return table;
}
- private ObjectNode createItem(String pk) {
- ObjectNode item = objectMapper.createObjectNode();
+ private ObjectNode item(String pk) {
+ ObjectNode node = objectMapper.createObjectNode();
ObjectNode pkValue = objectMapper.createObjectNode();
pkValue.put("S", pk);
- item.set("pk", pkValue);
- return item;
+ node.set("pk", pkValue);
+ return node;
}
+ private DestinationForwardingStats onlyStat() {
+ List<DestinationForwardingStats> stats = forwarder.forwardingStats();
+ assertEquals(1, stats.size(), "expected exactly one destination state");
+ return stats.get(0);
+ }
+
+ // ──────────────────────────── enqueue path ────────────────────────────
+
@Test
- void forwardsToActiveDestination() {
- TableDefinition table = createTable("test-table");
- KinesisStreamingDestination dest = new KinesisStreamingDestination(
- "arn:aws:kinesis:us-east-1:000000000000:stream/test-stream");
- table.getKinesisStreamingDestinations().add(dest);
+ void forwardOnlyEnqueues_neverCallsKinesisSynchronouslyNorThrows() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenThrow(new RuntimeException("kinesis down"));
- when(kinesisService.putRecord(anyString(), any(byte[].class), anyString(), anyString()))
+ assertDoesNotThrow(() -> forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT));
+
+ // The write path did no Kinesis I/O: the record is buffered, not delivered and not dropped.
+ verifyNoInteractions(kinesisService);
+ DestinationForwardingStats s = onlyStat();
+ assertEquals(1, s.queueDepth());
+ assertEquals(0L, s.forwarded());
+ assertEquals(0L, s.dropped());
+ }
+
+ @Test
+ void skipsDisabledDestination() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ table.getKinesisStreamingDestinations().get(0).setDestinationStatus("DISABLED");
+
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
+
+ assertTrue(forwarder.forwardingStats().isEmpty(), "a disabled destination is never enqueued");
+ assertFalse(scheduler.runOne(), "nothing scheduled for a disabled destination");
+ }
+
+ @Test
+ void skipsWhenNoDestinations() {
+ forwarder.forward("INSERT", null, item("k1"), tableWithDestination(), "us-east-1", ACCOUNT);
+ verifyNoInteractions(kinesisService);
+ assertTrue(forwarder.forwardingStats().isEmpty());
+ }
+
+ // ──────────────────────────── happy-path drain ────────────────────────────
+
+ @Test
+ void deliversBufferedRecordOnDrain() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
.thenReturn("seq-1");
- forwarder.forward("INSERT", null, createItem("k1"), table, "us-east-1");
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
+ scheduler.runUntilQuiet();
- verify(kinesisService).putRecord(eq("test-stream"), any(byte[].class), eq("k1"), eq("us-east-1"));
+ verify(kinesisService).putRecordForAccount(eq(ACCOUNT), eq("test-stream"), any(byte[].class),
+ eq("k1"), eq("us-east-1"));
+ DestinationForwardingStats s = onlyStat();
+ assertEquals(1L, s.forwarded());
+ assertEquals(0, s.queueDepth());
+ assertEquals("HEALTHY", s.health());
}
@Test
- void skipsDisabledDestination() {
- TableDefinition table = createTable("test-table");
- KinesisStreamingDestination dest = new KinesisStreamingDestination(
- "arn:aws:kinesis:us-east-1:000000000000:stream/test-stream");
- dest.setDestinationStatus("DISABLED");
- table.getKinesisStreamingDestinations().add(dest);
+ void threadsTableOwnerAccountToThePut() {
+ // The owning account is threaded through so an out-of-request-scope forward lands in the owner's stream.
+ TableDefinition table = tableWithDestination("arn:aws:kinesis:us-east-1:111111111111:stream/test-stream");
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenReturn("seq-1");
+
+ forwarder.forward("REMOVE", item("k1"), null, table, "us-east-1", "111111111111");
+ scheduler.runUntilQuiet();
+
+ verify(kinesisService).putRecordForAccount(eq("111111111111"), eq("test-stream"), any(byte[].class),
+ eq("k1"), eq("us-east-1"));
+ }
+
+ @Test
+ void preservesFifoAcrossARetry() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ List<String> delivered = new ArrayList<>();
+ AtomicInteger calls = new AtomicInteger();
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenAnswer(inv -> {
+ if (calls.getAndIncrement() == 0) {
+ throw new RuntimeException("transient on the very first attempt");
+ }
+ delivered.add(inv.getArgument(3)); // partitionKey
+ return "seq";
+ });
- forwarder.forward("INSERT", null, createItem("k1"), table, "us-east-1");
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
+ forwarder.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT);
+ forwarder.forward("INSERT", null, item("k3"), table, "us-east-1", ACCOUNT);
+ scheduler.runUntilQuiet();
+
+ // k1 was retried, yet ordering is preserved: the head is not skipped past.
+ assertEquals(List.of("k1", "k2", "k3"), delivered);
+ DestinationForwardingStats s = onlyStat();
+ assertEquals(3L, s.forwarded());
+ assertEquals(1L, s.retried());
+ assertEquals(0L, s.dropped());
+ }
+
+ // ──────────────────────────── failure classification ────────────────────────────
+
+ @Test
+ void unknownFailureIsRetriedNotDropped() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenThrow(new RuntimeException("transient"));
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
+ scheduler.runOne(); // exactly one drain attempt
+
+ DestinationForwardingStats s = onlyStat();
+ assertEquals(1, s.queueDepth(), "an unknown failure keeps the record buffered for retry");
+ assertEquals(0L, s.dropped());
+ assertEquals(1L, s.retried());
+ assertEquals("RETRYING", s.health());
+ }
+
+ @Test
+ void terminalFailureDropsThePoisonRecordWithoutWedgingTheQueue() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ List<String> delivered = new ArrayList<>();
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenAnswer(inv -> {
+ String pk = inv.getArgument(3);
+ if ("k1".equals(pk)) {
+ throw new AwsException("ValidationException", "record too large", 400);
+ }
+ delivered.add(pk);
+ return "seq";
+ });
+
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT); // poison
+ forwarder.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT); // good
+ scheduler.runUntilQuiet();
+
+ // The poison record is dropped immediately; the good record behind it is still delivered.
+ assertEquals(List.of("k2"), delivered);
+ DestinationForwardingStats s = onlyStat();
+ assertEquals(1L, s.dropped());
+ assertEquals(1L, s.forwarded());
+ assertEquals("ValidationException", s.lastErrorType());
+ assertEquals(1L, forwarder.getForwardFailureCount());
+ }
+
+ @Test
+ void givesUpAfterProbeBudgetThenSelfHealsOnALaterEvent() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ AtomicBoolean healthy = new AtomicBoolean(false);
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenAnswer(inv -> {
+ if (!healthy.get()) {
+ throw new RuntimeException("stream unavailable");
+ }
+ return "seq";
+ });
+
+ // Episode 1: persistent failure exhausts the probe budget and drops the buffered record.
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
+ scheduler.runUntilQuiet();
+
+ DestinationForwardingStats gaveUp = onlyStat();
+ assertEquals("GAVE_UP", gaveUp.health());
+ assertEquals(1L, gaveUp.dropped());
+ assertEquals(0, gaveUp.queueDepth());
+ // Exactly MAX_PROBES (10) send attempts: the initial immediate run + 9 retry schedules, backoff
+ // 250ms doubling to the 8s cap. Give-up fires on the 10th failure (no 11th schedule).
+ assertEquals(List.of(0L, 250L, 500L, 1000L, 2000L, 4000L, 8000L, 8000L, 8000L, 8000L),
+ List.copyOf(scheduler.delays));
+
+ // Episode 2: a fresh event with a recovered stream re-attempts (budget was reset) and heals.
+ healthy.set(true);
+ forwarder.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT);
+ scheduler.runUntilQuiet();
+
+ DestinationForwardingStats healed = onlyStat();
+ assertEquals("HEALTHY", healed.health());
+ assertEquals(1L, healed.forwarded());
+ assertEquals(1L, healed.dropped(), "the episode-1 drop is retained in the cumulative count");
+ assertEquals(0, healed.queueDepth());
+ }
+
+ // ──────────────────────────── overflow ────────────────────────────
+
+ @Test
+ void overflowDropsOldestBeyondTheBufferCap() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ int over = 2;
+ for (int i = 0; i < KinesisStreamingForwarder.MAX_BUFFERED + over; i++) {
+ forwarder.forward("INSERT", null, item("k" + i), table, "us-east-1", ACCOUNT);
+ }
+ // Never drained: the buffer is capped and the oldest are evicted.
+ DestinationForwardingStats s = onlyStat();
+ assertEquals(KinesisStreamingForwarder.MAX_BUFFERED, s.queueDepth());
+ assertEquals(over, s.overflowed());
+ assertEquals(over, s.dropped());
+ assertEquals(over, forwarder.getForwardFailureCount());
+ }
+
+ // ──────────────────────────── lifecycle ────────────────────────────
+
+ @Test
+ void disableDiscardsBufferAndStalesTheInFlightDrain() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
+
+ forwarder.onDestinationDisabled(ACCOUNT, "us-east-1", "test-table", STREAM_ARN);
+
+ DestinationForwardingStats s = onlyStat(); // state retained on disable
+ assertEquals(0, s.queueDepth());
+ assertEquals(1L, s.discardedOnDisable());
+
+ // The drain scheduled before the disable is now stale: running it is a no-op (epoch bumped).
+ scheduler.runUntilQuiet();
verifyNoInteractions(kinesisService);
+ assertEquals(0L, onlyStat().forwarded());
}
@Test
- void skipsWhenNoDestinations() {
- TableDefinition table = createTable("test-table");
+ void tableDeleteRemovesStateAndStalesTheInFlightDrain() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
- forwarder.forward("INSERT", null, createItem("k1"), table, "us-east-1");
+ forwarder.onTableDeleted(ACCOUNT, "us-east-1", "test-table");
+ assertTrue(forwarder.forwardingStats().isEmpty(), "table delete drops the destination state");
+ scheduler.runUntilQuiet();
verifyNoInteractions(kinesisService);
}
@Test
- void continuesOnPutRecordFailure() {
- TableDefinition table = createTable("test-table");
- KinesisStreamingDestination dest1 = new KinesisStreamingDestination(
- "arn:aws:kinesis:us-east-1:000000000000:stream/stream-1");
- KinesisStreamingDestination dest2 = new KinesisStreamingDestination(
- "arn:aws:kinesis:us-east-1:000000000000:stream/stream-2");
- table.getKinesisStreamingDestinations().add(dest1);
- table.getKinesisStreamingDestinations().add(dest2);
+ void awaitIdleReflectsQueueState() {
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenReturn("seq");
- when(kinesisService.putRecord(eq("stream-1"), any(byte[].class), anyString(), anyString()))
- .thenThrow(new RuntimeException("stream-1 failed"));
- when(kinesisService.putRecord(eq("stream-2"), any(byte[].class), anyString(), anyString()))
- .thenReturn("seq-1");
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
+ assertFalse(forwarder.awaitIdle(Duration.ofMillis(50)), "buffered + drain pending is not idle");
+
+ scheduler.runUntilQuiet();
+ assertTrue(forwarder.awaitIdle(Duration.ofMillis(50)), "drained empty is idle");
+ }
+
+ // ──────────────────────────── race / robustness ────────────────────────────
+
+ @Test
+ void overflowWhileTheHeadIsInFlightNeverEvictsTheRecordBeingSent() {
+ // Deterministically reproduces the overflow-vs-in-flight race: while k0 is being sent, a burst of
+ // producers overfills the buffer. The in-flight record must not be evicted, and nothing may be lost
+ // without accounting (the peek-then-poll approach would evict k0 and then poll a different record).
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ int burst = KinesisStreamingForwarder.MAX_BUFFERED + 5;
+ List<String> delivered = new ArrayList<>();
+ AtomicBoolean filled = new AtomicBoolean(false);
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenAnswer(inv -> {
+ String pk = inv.getArgument(3);
+ if ("k0".equals(pk) && filled.compareAndSet(false, true)) {
+ for (int i = 1; i <= burst; i++) {
+ forwarder.forward("INSERT", null, item("k" + i), table, "us-east-1", ACCOUNT);
+ }
+ }
+ delivered.add(pk);
+ return "seq";
+ });
+
+ forwarder.forward("INSERT", null, item("k0"), table, "us-east-1", ACCOUNT);
+ scheduler.runUntilQuiet();
+
+ // k0 (in flight during the overflow) was delivered, not evicted; the 5 oldest buffered were dropped.
+ assertEquals("k0", delivered.get(0));
+ assertFalse(delivered.contains("k1"), "the oldest buffered record is the overflow victim, not k0");
+ DestinationForwardingStats s = onlyStat();
+ assertEquals(5L, s.overflowed());
+ assertEquals(5L, s.dropped());
+ assertEquals(burst + 1 - 5, s.forwarded());
+ // Every one of the burst+1 records is accounted for exactly once: no silent loss.
+ assertEquals(burst + 1, s.forwarded() + s.dropped());
+ assertEquals(0, s.queueDepth());
+ }
+
+ @Test
+ void schedulerRejectionNeitherEscapesForwardNorWedgesTheDestination() {
+ RejectingScheduler rejecting = new RejectingScheduler();
+ KinesisStreamingForwarder f = new KinesisStreamingForwarder(kinesisService, objectMapper, rejecting);
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenReturn("seq");
+
+ // Executor down: scheduling the initial drain is rejected, but forward must not throw, and the
+ // record stays buffered (not dropped).
+ assertDoesNotThrow(() -> f.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT));
+ assertEquals(1, f.forwardingStats().get(0).queueDepth());
+
+ // Executor recovers: a later event must be able to schedule a fresh drain (no wedge), and BOTH the
+ // previously-buffered and the new record drain in FIFO order.
+ rejecting.reject = false;
+ f.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT);
+ while (rejecting.runOne()) {
+ // drive the recovered drain to completion
+ }
+ DestinationForwardingStats s = f.forwardingStats().get(0);
+ assertEquals(2L, s.forwarded());
+ assertEquals(0, s.queueDepth());
+ }
+
+ @Test
+ void terminalHeadDropGivesTheNextRecordItsOwnFullProbeBudget() {
+ // k1 fails transiently then terminally; k2 must then get a FRESH MAX_PROBES budget, not the leftover.
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ Map<String, Integer> attempts = new HashMap<>();
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenAnswer(inv -> {
+ String pk = inv.getArgument(3);
+ int n = attempts.merge(pk, 1, Integer::sum);
+ if ("k1".equals(pk)) {
+ if (n <= 3) {
+ throw new RuntimeException("transient");
+ }
+ throw new AwsException("ValidationException", "poison", 400); // 4th attempt: terminal
+ }
+ throw new RuntimeException("k2 stream down"); // k2 always transient
+ });
+
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
+ forwarder.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT);
+ scheduler.runUntilQuiet();
+
+ assertEquals(4, attempts.get("k1"), "3 transient retries then the terminal attempt");
+ // If the budget had not been reset on the terminal drop, k2 would give up after only
+ // MAX_PROBES - 3 attempts. A fresh budget means exactly MAX_PROBES attempts for k2.
+ assertEquals(KinesisStreamingForwarder.MAX_PROBES, attempts.get("k2"));
+ DestinationForwardingStats s = onlyStat();
+ assertEquals(2L, s.dropped()); // k1 (terminal) + k2 (give-up)
+ assertEquals("GAVE_UP", s.health());
+ }
+
+ @Test
+ void disableDuringInFlightSendCompletesButDoesNotResurrectOrDoubleCount() {
+ // Characterizes the inherent teardown race (documented as best-effort): a send dispatched before a
+ // disable may complete, but the torn-down destination is neither re-counted nor resurrected.
+ TableDefinition table = tableWithDestination(STREAM_ARN);
+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))
+ .thenAnswer(inv -> {
+ forwarder.onDestinationDisabled(ACCOUNT, "us-east-1", "test-table", STREAM_ARN);
+ return "seq";
+ });
- forwarder.forward("INSERT", null, createItem("k1"), table, "us-east-1");
+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);
+ scheduler.runUntilQuiet();
- verify(kinesisService).putRecord(eq("stream-1"), any(byte[].class), anyString(), anyString());
- verify(kinesisService).putRecord(eq("stream-2"), any(byte[].class), anyString(), anyString());
- assertEquals(1L, forwarder.getForwardFailureCount(),
- "the single failed forward must be counted");
+ // The dispatched send ran exactly once; its result is discarded (epoch bumped), the buffer is empty,
+ // and no further drain runs.
+ verify(kinesisService, times(1)).putRecordForAccount(anyString(), anyString(), any(byte[].class),
+ anyString(), anyString());
+ DestinationForwardingStats s = onlyStat();
+ assertEquals(0, s.queueDepth());
+ assertEquals(0L, s.forwarded());
}
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceAccountTest.java b/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceAccountTest.java
new file mode 100644
index 0000000000..cd85504ec8
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceAccountTest.java
@@ -0,0 +1,102 @@
+package io.github.hectorvent.floci.services.kinesis;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.InMemoryStorage;
+import io.github.hectorvent.floci.services.kinesis.model.KinesisConsumer;
+import io.github.hectorvent.floci.services.kinesis.model.KinesisShard;
+import io.github.hectorvent.floci.services.kinesis.model.KinesisStream;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.jupiter.api.Test;
+
+import java.nio.charset.StandardCharsets;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+
+/**
+ * Verifies that {@code putRecordForAccount} resolves, persists, and legacy-migrates against the SPECIFIED
+ * account rather than the ambient/default one, so an out-of-request-scope producer (TTL sweep, CDC
+ * forwarding) never writes into another tenant's same-named stream and keeps working after an upgrade
+ * from unprefixed (pre-multi-account) storage. Real {@link KinesisService} over in-memory account-aware
+ * storage (no Vert.x → runs in the offline sandbox).
+ */
+class KinesisServiceAccountTest {
+
+ private static final String KEY_S = "us-east-1::s";
+ // jsr310 module registered so cloning a shard's Instant (creationTimestamp) round-trips.
+ private final ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();
+
+ private KinesisService serviceOver(AccountAwareStorageBackend<KinesisStream> streams) {
+ return new KinesisService(streams, AccountAwareStorageBackend.inMemory("000000000000"),
+ mock(RegionResolver.class));
+ }
+
+ private static int recordCount(KinesisStream stream) {
+ return stream.getShards().stream().mapToInt(KinesisShard::recordCount).sum();
+ }
+
+ private KinesisStream cloneStream(KinesisStream template) {
+ try {
+ return mapper.treeToValue(mapper.valueToTree(template), KinesisStream.class);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ void putRecordForAccountResolvesAgainstThatAccountNotDefault() {
+ KinesisService svc = serviceOver(AccountAwareStorageBackend.inMemory("000000000000"));
+ svc.createStream("s", 1, "us-east-1"); // created in the ambient/default account (000000000000)
+ byte[] data = "x".getBytes(StandardCharsets.UTF_8);
+
+ assertNotNull(svc.putRecord("s", data, "pk", "us-east-1"));
+ assertNotNull(svc.putRecordForAccount(null, "s", data, "pk", "us-east-1"));
+ assertNotNull(svc.putRecordForAccount("000000000000", "s", data, "pk", "us-east-1"));
+
+ AwsException ex = assertThrows(AwsException.class,
+ () -> svc.putRecordForAccount("111111111111", "s", data, "pk", "us-east-1"));
+ assertEquals("ResourceNotFoundException", ex.getErrorCode());
+ }
+
+ @Test
+ void putRecordForAccountPersistsToThatAccountPartition() {
+ AccountAwareStorageBackend<KinesisStream> streams = AccountAwareStorageBackend.inMemory("000000000000");
+ KinesisService svc = serviceOver(streams);
+ svc.createStream("s", 1, "us-east-1"); // in the default account
+ // Seed an INDEPENDENT same-named stream in account 111 (deep copy so the append can't leak back).
+ KinesisStream in111 = cloneStream(streams.getForAccount("000000000000", KEY_S).orElseThrow());
+ streams.putForAccount("111111111111", KEY_S, in111);
+
+ svc.putRecordForAccount("111111111111", "s", "x".getBytes(StandardCharsets.UTF_8), "pk", "us-east-1");
+
+ // Persisted into account 111's partition only; the default partition is untouched. (A regression
+ // that used the ambient put would land the record in the default account here.)
+ assertEquals(1, recordCount(streams.getForAccount("111111111111", KEY_S).orElseThrow()));
+ assertEquals(0, recordCount(streams.getForAccount("000000000000", KEY_S).orElseThrow()));
+ }
+
+ @Test
+ void putRecordForAccountMigratesLegacyUnprefixedStream() {
+ // Upgrade scenario: a pre-multi-account (unprefixed) stream owned by account 111 exists in the
+ // raw delegate. An out-of-request-scope forward for 111 must migrate + use it, not drop as RNF.
+ InMemoryStorage<String, KinesisStream> raw = new InMemoryStorage<>();
+ AccountAwareStorageBackend<KinesisStream> streams = new AccountAwareStorageBackend<>(raw, null, "000000000000");
+ KinesisService svc = serviceOver(streams);
+
+ // Build a shard-carrying stream (via createStream on a throwaway key) and place a copy under the
+ // UNPREFIXED legacy key, owned (by ARN) by account 111.
+ KinesisStream legacy = cloneStream(svc.createStream("tmpl", 1, "us-east-1"));
+ legacy.setStreamArn("arn:aws:kinesis:us-east-1:111111111111:stream/s2");
+ raw.put("us-east-1::s2", legacy); // unprefixed = legacy pre-multi-account data
+
+ // Migrates on write and appends; no ResourceNotFoundException.
+ assertNotNull(svc.putRecordForAccount("111111111111", "s2", "x".getBytes(StandardCharsets.UTF_8),
+ "pk", "us-east-1"));
+ // The record is now in account 111's partition (migrated).
+ assertEquals(1, recordCount(streams.getForAccount("111111111111", "us-east-1::s2").orElseThrow()));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceTest.java
index ec20fc862d..16c95adffd 100644
--- a/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceTest.java
@@ -835,7 +835,8 @@ class KinesisServiceTest {
* append lock, the producer must fail cleanly (ResourceNotFoundException) rather than append to
* and re-persist the stale instance. Deterministic via the pre-lock hook: the producer parks
* WITHOUT holding the monitor, so the deleter wins the lock first. Fails on code lacking the
- * in-lock re-resolve: the producer would resurrect the stream and throw nothing.
+ * in-lock resolve: a producer that resolved before the lock would append to and re-persist the
+ * stale pre-delete instance, resurrecting the stream and throwing nothing.
*/
@Test
void putRecord_failsCleanly_whenStreamDeletedBeforeLockAcquired() throws Exception {