Light, fluffy, and always free - The AWS Local Emulator alternative
fix(dynamodb): route Kinesis CDC to the table owner's account and retry failed forwards (#3063)
Guilherme Scaransea9df87295605
diff --git a/docs/services/dynamodb.md b/docs/services/dynamodb.mdindex 7f3ac0d8f6..8f6d3c0cd4 100644--- a/docs/services/dynamodb.md+++ b/docs/services/dynamodb.md@@ -164,4 +164,38 @@ aws dynamodb list-exports \--endpoint-url $AWS_ENDPOINT_URL```-The export writes to `s3://<bucket>/<prefix>/AWSDynamoDB/<exportId>/data/` as one or more `.json.gz` files, along with `manifest-summary.json` and `manifest-files.json`, the same layout as real AWS DynamoDB exports.\ No newline at end of file+The export writes to `s3://<bucket>/<prefix>/AWSDynamoDB/<exportId>/data/` as one or more `.json.gz` files, along with `manifest-summary.json` and `manifest-files.json`, the same layout as real AWS DynamoDB exports.++## Kinesis change data capture (CDC)++When a table has an **ACTIVE** Kinesis streaming destination (see+`EnableKinesisStreamingDestination`), every item change, `INSERT`, `MODIFY`, and `REMOVE`,+including TTL expirations, is forwarded to the destination stream as a Kinesis record in the+AWS CDC envelope (`eventName`, `dynamodb.Keys`, `NewImage`/`OldImage`, `ApproximateCreationDateTime`).++### Delivery contract++Forwarding is **bounded best-effort with in-process retry**. A write is never blocked or failed by+the destination stream: the change event is enqueued and delivered on a background drain, so a slow or+unavailable Kinesis stream cannot stall a `PutItem`/`UpdateItem`/`DeleteItem` or the TTL sweep.++The drain gives these guarantees per destination:++- **Retries** transient/unknown send failures with capped exponential backoff (250 ms doubling to an+ 8 s cap, up to 10 attempts) rather than dropping the record on the first exception.+- **Preserves FIFO** across retries: the head record is not skipped past (stronger than AWS, which may+ reorder or duplicate).+- **Drops a poison record** immediately on a deterministic terminal failure+ (`ValidationException`/`InvalidArgumentException`) so it cannot wedge the queue behind it.+- **Gives up an episode** after the retry budget is exhausted, dropping the buffered records and marking+ the destination `GAVE_UP`; a later change event starts a fresh episode and self-heals once the stream+ recovers.+- **Bounds memory** at 1000 buffered records per destination, evicting the oldest on overflow.++Disabling a destination or deleting the table discards that destination's buffered records and stops all+future sends for it; a send already in flight at that instant may still complete (teardown cannot recall+an in-flight request, an inherent and harmless race). Buffered records are held **in memory only**: they+are not durable and are lost on restart (this is an emulator, not an at-least-once pipeline). Records that are permanently dropped (terminal, give-up,+or overflow) are counted and logged, and per-destination delivery health (forwarded/retried/dropped+counts, queue depth, last error, and current health) is tracked for inspection.+```diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java b/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.javaindex 2473a8fb86..61d90cb48d 100644--- a/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java+++ b/src/main/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackend.java@@ -66,13 +66,23 @@ public class AccountAwareStorageBackend<V> implements StorageBackend<String, V>if (result.isPresent()) {return result;}- // Backward-compat: try un-prefixed key (pre-multi-account data) and migrate on read.- result = delegate.get(key);- if (result.isPresent()) {- delegate.put(prefixedKey, result.get());- delegate.delete(key);+ // Backward-compat: try un-prefixed key (pre-multi-account data) and migrate on read. The+ // migration is a write (put then delete), so it runs under the same monitor the explicit-account+ // migrators hold, or two readers resolving the same legacy key from different account contexts+ // would each copy it into their own partition and the two copies would then diverge. The hit+ // path above stays outside the monitor, so only a miss pays for it.+ synchronized (this) {+ Optional<V> migratedByAnotherReader = delegate.get(prefixedKey);+ if (migratedByAnotherReader.isPresent()) {+ return migratedByAnotherReader;+ }+ result = delegate.get(key);+ if (result.isPresent()) {+ delegate.put(prefixedKey, result.get());+ delegate.delete(key);+ }+ return result;}- return result;}@Overridediff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DestinationForwardingStats.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DestinationForwardingStats.javanew file mode 100644index 0000000000..150992f85a--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DestinationForwardingStats.java@@ -0,0 +1,19 @@+package io.github.hectorvent.floci.services.dynamodb;++/**+ * An immutable snapshot of a single Kinesis CDC destination's forwarding health, produced by+ * {@link KinesisStreamingForwarder#forwardingStats()}. Copied under the destination's monitor so+ * the counters and queue depth are internally consistent.+ *+ * @param health one of HEALTHY, RETRYING, GAVE_UP+ */+record DestinationForwardingStats(+ String health,+ long forwarded,+ long retried,+ long dropped,+ long overflowed,+ long discardedOnDisable,+ int queueDepth,+ String lastErrorType) {+}diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbJsonHandler.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbJsonHandler.javaindex 4b6ad85e91..8b56a83c2c 100644--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbJsonHandler.java+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbJsonHandler.java@@ -1853,6 +1853,8 @@ public class DynamoDbJsonHandler {existing.get().setDestinationStatus("DISABLED");existing.get().setDestinationStatusDescription("Kinesis streaming is disabled for this table");dynamoDbService.persistTable(resolvedTableName, table, region);+ // Stop forwarding and discard buffered CDC records for this now-disabled destination.+ dynamoDbService.onKinesisStreamingDestinationDisabled(resolvedTableName, streamArn, region);ObjectNode response = objectMapper.createObjectNode();response.put("TableName", resolvedTableName);diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.javaindex 7acb126e75..9b8039ee54 100644--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java@@ -480,9 +480,24 @@ public class DynamoDbService implements ResourceProvider {if (streamService != null) {streamService.deleteStream(canonicalTableName, region);}+ if (kinesisForwarder != null) {+ // Discard any buffered CDC records and stop draining: the destination stream is gone.+ kinesisForwarder.onTableDeleted(regionResolver.getAccountId(), region, canonicalTableName);+ }LOG.infov("Deleted table: {0}", canonicalTableName);}+ /**+ * Notify the CDC forwarder that a Kinesis streaming destination was disabled so it discards any+ * buffered records for it and stops draining. {@code tableName} must be the resolved (canonical)+ * table name, the same value the forward path keys destination state on. No-op without a forwarder.+ */+ public void onKinesisStreamingDestinationDisabled(String tableName, String streamArn, String region) {+ if (kinesisForwarder != null) {+ kinesisForwarder.onDestinationDisabled(regionResolver.getAccountId(), region, tableName, streamArn);+ }+ }+public List<String> listTables(String region) {String prefix = region + "::";return tableStore.scan(k -> k.startsWith(prefix)).stream()@@ -571,12 +586,16 @@ public class DynamoDbService implements ResourceProvider {LOG.tracev("Put item in {0}: key={1} item={2}", canonicalTableName, itemKey, item);String eventName = existing == null ? "INSERT" : "MODIFY";+ // Captured in request scope on purpose: this event may be deferred to the batch drain,+ // and resolving the account inside the lambda would fall back to the default account,+ // which is the ambient-account bug this commit exists to remove.+ String ownerAccountId = regionResolver.getAccountId();Runnable streamEvent = () -> {if (streamService != null) {streamService.captureEvent(canonicalTableName, eventName, existing, item, table, region);}if (kinesisForwarder != null) {- kinesisForwarder.forward(eventName, existing, item, table, region);+ kinesisForwarder.forward(eventName, existing, item, table, region, ownerAccountId);}};if (deferredStreamEvents != null) {@@ -660,12 +679,16 @@ public class DynamoDbService implements ResourceProvider {LOG.tracev("Deleted item from {0}: key={1} removed={2}", canonicalTableName, itemKey, removed);if (removed != null) {+ // Captured in request scope on purpose: this event may be deferred to the batch drain,+ // and resolving the account inside the lambda would fall back to the default account,+ // which is the ambient-account bug this commit exists to remove.+ String ownerAccountId = regionResolver.getAccountId();Runnable streamEvent = () -> {if (streamService != null) {streamService.captureEvent(canonicalTableName, "REMOVE", removed, null, table, region);}if (kinesisForwarder != null) {- kinesisForwarder.forward("REMOVE", removed, null, table, region);+ kinesisForwarder.forward("REMOVE", removed, null, table, region, ownerAccountId);}};if (deferredStreamEvents != null) {@@ -828,12 +851,16 @@ public class DynamoDbService implements ResourceProvider {LOG.tracev("Updated item in {0}: key={1} updateExpression={2} item={3}",canonicalTableName, itemKey, updateExpression, item);+ // Captured in request scope on purpose: this event may be deferred to the batch drain,+ // and resolving the account inside the lambda would fall back to the default account,+ // which is the ambient-account bug this commit exists to remove.+ String ownerAccountId = regionResolver.getAccountId();Runnable streamEvent = () -> {if (streamService != null) {streamService.captureEvent(canonicalTableName, "MODIFY", existing, item, table, region);}if (kinesisForwarder != null) {- kinesisForwarder.forward("MODIFY", existing, item, table, region);+ kinesisForwarder.forward("MODIFY", existing, item, table, region, ownerAccountId);}};if (deferredStreamEvents != null) {@@ -1840,7 +1867,9 @@ public class DynamoDbService implements ResourceProvider {streamService.captureEvent(table.getTableName(), "REMOVE", removed, null, table, region);}if (kinesisForwarder != null) {- kinesisForwarder.forward("REMOVE", removed, null, table, region);+ // Out of request scope here: pass the table owner's account explicitly so the CDC+ // record lands in the owner's stream, not the default account's same-named stream.+ kinesisForwarder.forward("REMOVE", removed, null, table, region, accountId);}}}diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarder.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarder.javaindex fbd11390d6..9b8a67f391 100644--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarder.java+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarder.java@@ -1,5 +1,6 @@package io.github.hectorvent.floci.services.dynamodb;+import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;import io.github.hectorvent.floci.services.dynamodb.model.KinesisStreamingDestination;import io.github.hectorvent.floci.services.dynamodb.model.TableDefinition;@@ -7,73 +8,416 @@ import io.github.hectorvent.floci.services.kinesis.KinesisService;import com.fasterxml.jackson.databind.JsonNode;import com.fasterxml.jackson.databind.ObjectMapper;import com.fasterxml.jackson.databind.node.ObjectNode;+import jakarta.annotation.PreDestroy;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;+import java.time.Duration;import java.time.Instant;+import java.util.ArrayDeque;+import java.util.ArrayList;import java.util.List;import java.util.UUID;-import java.util.concurrent.atomic.AtomicLong;+import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.Executors;+import java.util.concurrent.ScheduledExecutorService;+import java.util.concurrent.TimeUnit;+/**+ * Forwards DynamoDB change events to Kinesis with a <strong>bounded best-effort, in-process retry</strong>+ * delivery contract (NOT durable at-least-once: the buffer is in memory and lost on restart).+ *+ * <p>The write path only ENQUEUES (it never performs Kinesis I/O or blocks: a slow/dead destination can+ * neither stall a DynamoDB write nor the TTL sweep). A per-destination single-flight drain, scheduled on a+ * small daemon pool, performs the actual {@code putRecord} outside the destination lock and:+ * <ul>+ * <li>retries transient/unknown failures with capped exponential backoff (≤{@value #MAX_PROBES} probes),</li>+ * <li>drops a record immediately on a deterministic terminal failure (invalid argument / validation /+ * serialization) so a poison record cannot wedge the queue,</li>+ * <li>gives up the whole per-destination episode after the probe budget (dropping the buffered records+ * and marking the destination GAVE_UP until the next successful send),</li>+ * <li>preserves per-destination FIFO across retries (stronger than AWS, which may reorder/duplicate),</li>+ * <li>evicts the oldest buffered record on overflow (max {@value #MAX_BUFFERED} per destination).</li>+ * </ul>+ *+ * <p>The destination's {@code DestinationStatus} is intentionally NOT auto-demoted (AWS has no+ * delivery-failure status); delivery health is surfaced separately via {@link #forwardingStats()}.+ * Records dropped/retried are counted and their last error retained for observability.+ */@ApplicationScopedpublic class KinesisStreamingForwarder {private static final Logger LOG = Logger.getLogger(KinesisStreamingForwarder.class);+ static final int MAX_BUFFERED = 1000;+ static final int MAX_PROBES = 10;+ static final long INITIAL_BACKOFF_MS = 250;+ static final long MAX_BACKOFF_MS = 8000;++ /** Schedules drain runs; abstracted so tests can drive drains deterministically. */+ interface Scheduler {+ void schedule(Runnable task, long delayMillis);++ void shutdown();+ }++ enum Health { HEALTHY, RETRYING, GAVE_UP }++ record PendingRecord(String accountId, String region, String streamName, String tableName,+ byte[] data, String partitionKey, Instant enqueuedAt) {+ }++ /** Per-destination delivery state. Every field is accessed only while holding {@code this} monitor. */+ static final class DestinationState {+ final String key;+ final String accountId;+ final String region;+ final String tableName;+ final String streamArn;+ final ArrayDeque<PendingRecord> queue = new ArrayDeque<>();+ long epoch;+ int consecutiveFailedProbes;+ boolean drainScheduled;+ long forwarded;+ long retried;+ long dropped;+ long overflowed;+ long discardedOnDisable;+ String lastErrorType;+ String lastErrorMessage;+ Health health = Health.HEALTHY;++ DestinationState(String key, String accountId, String region, String tableName, String streamArn) {+ this.key = key;+ this.accountId = accountId;+ this.region = region;+ this.tableName = tableName;+ this.streamArn = streamArn;+ }+ }+private final KinesisService kinesisService;private final ObjectMapper objectMapper;- private final AtomicLong forwardFailures = new AtomicLong();+ private final Scheduler scheduler;+ private final ConcurrentHashMap<String, DestinationState> states = new ConcurrentHashMap<>();@Injectpublic KinesisStreamingForwarder(KinesisService kinesisService, ObjectMapper objectMapper) {+ this(kinesisService, objectMapper, defaultScheduler());+ }++ /** Test seam: inject a deterministic Scheduler so drains run on demand rather than on real timers. */+ KinesisStreamingForwarder(KinesisService kinesisService, ObjectMapper objectMapper, Scheduler scheduler) {this.kinesisService = kinesisService;this.objectMapper = objectMapper;+ this.scheduler = scheduler;+ }++ private static Scheduler defaultScheduler() {+ ScheduledExecutorService exec = Executors.newScheduledThreadPool(2, r -> {+ Thread t = new Thread(r, "cdc-forward-drain");+ t.setDaemon(true);+ return t;+ });+ return new Scheduler() {+ @Override+ public void schedule(Runnable task, long delayMillis) {+ exec.schedule(task, delayMillis, TimeUnit.MILLISECONDS);+ }++ @Override+ public void shutdown() {+ exec.shutdownNow();+ }+ };}+ // ──────────────────────────── Enqueue (write path: never blocks, never throws) ────────────────────────────+public void forward(String eventName, JsonNode oldItem, JsonNode newItem,- TableDefinition table, String region) {+ TableDefinition table, String region, String ownerAccountId) {List<KinesisStreamingDestination> destinations = table.getKinesisStreamingDestinations();- if (destinations == null || destinations.isEmpty()) return;-+ if (destinations == null || destinations.isEmpty()) {+ return;+ }Instant now = Instant.now();JsonNode sourceItem = newItem != null ? newItem : oldItem;ObjectNode keys = buildKeys(sourceItem, table);+ byte[] data;+ String partitionKey;+ try {+ ObjectNode payload = buildPayload(eventName, keys, newItem, oldItem, table.getTableName(), region, now);+ data = objectMapper.writeValueAsBytes(payload);+ partitionKey = extractPartitionKey(keys, table);+ } catch (Exception e) {+ // Serialization is a deterministic terminal failure: nothing can be forwarded for this event.+ for (KinesisStreamingDestination dest : destinations) {+ if (!"ACTIVE".equals(dest.getDestinationStatus())) {+ continue;+ }+ DestinationState st = stateFor(ownerAccountId, region, table.getTableName(), dest.getStreamArn());+ synchronized (st) {+ st.dropped++;+ recordError(st, e);+ }+ }+ LOG.errorv(e, "Dropped DynamoDB CDC event for table {0}: failed to serialize payload", table.getTableName());+ return;+ }+for (KinesisStreamingDestination dest : destinations) {- if (!"ACTIVE".equals(dest.getDestinationStatus())) continue;+ if (!"ACTIVE".equals(dest.getDestinationStatus())) {+ continue;+ }+ String streamName = extractStreamName(dest.getStreamArn());+ DestinationState st = stateFor(ownerAccountId, region, table.getTableName(), dest.getStreamArn());+ PendingRecord rec = new PendingRecord(ownerAccountId, region, streamName, table.getTableName(),+ data, partitionKey, now);+ synchronized (st) {+ if (st.queue.size() >= MAX_BUFFERED) {+ st.queue.pollFirst(); // drop OLDEST+ st.overflowed++;+ st.dropped++;+ LOG.warnv("CDC buffer full for stream {0} (>{1} records); dropped the oldest",+ st.streamArn, MAX_BUFFERED);+ }+ st.queue.addLast(rec);+ if (!st.drainScheduled) {+ scheduleDrain(st, 0);+ }+ }+ }+ }- try {- ObjectNode payload = buildPayload(eventName, keys, newItem, oldItem,- table.getTableName(), region, now);- byte[] data = objectMapper.writeValueAsBytes(payload);+ private DestinationState stateFor(String accountId, String region, String tableName, String streamArn) {+ String key = accountId + "|" + region + "|" + tableName + "|" + streamArn;+ return states.computeIfAbsent(key,+ k -> new DestinationState(k, accountId, region, tableName, streamArn));+ }- String partitionKey = extractPartitionKey(keys, table);- String streamName = extractStreamName(dest.getStreamArn());+ // ──────────────────────────── Drain (per-destination single-flight; I/O outside the lock) ────────────────────────────- kinesisService.putRecord(streamName, data, partitionKey, region);- LOG.debugv("Forwarded DynamoDB event to Kinesis stream {0}: {1} on {2}",- streamName, eventName, table.getTableName());+ private void drain(DestinationState st, long scheduledEpoch) {+ while (true) {+ PendingRecord head;+ synchronized (st) {+ if (scheduledEpoch != st.epoch) {+ return; // this destination was disabled/deleted; a fresh episode (if any) owns the queue+ }+ if (st.queue.isEmpty()) {+ st.drainScheduled = false;+ if (st.health != Health.GAVE_UP) {+ st.health = Health.HEALTHY;+ }+ return;+ }+ // Lease the head OUT of the queue for the duration of the send. A concurrent overflow+ // eviction can then only drop a still-buffered record, never the one being delivered,+ // and completion below reconciles exactly this leased record (no blind pollFirst).+ head = st.queue.pollFirst();+ }++ boolean success = false;+ boolean terminal = false;+ Throwable failure = null;+ try {+ if (head.accountId() != null) {+ kinesisService.putRecordForAccount(head.accountId(), head.streamName(), head.data(),+ head.partitionKey(), head.region());+ } else {+ kinesisService.putRecord(head.streamName(), head.data(), head.partitionKey(), head.region());+ }+ success = true;+ } catch (AwsException e) {+ failure = e;+ terminal = isTerminal(e);} catch (Exception e) {- // Do NOT rethrow: put/delete/update persist before forwarding, so a rethrow would- // turn a committed write into a spurious 5xx; the TTL sweep forwards mid-loop and- // persists only after the batch, so a rethrow would abort the sweep. Surface the- // drop loudly (ERROR + stack trace) and count it for process-local diagnostics.- forwardFailures.incrementAndGet();- LOG.errorv(e, "Dropped DynamoDB CDC event: failed to forward {0} on table {1} to Kinesis destination {2}",- eventName, table.getTableName(), dest.getStreamArn());+ failure = e; // unknown/operational failure -> retryable+ }++ synchronized (st) {+ if (scheduledEpoch != st.epoch) {+ // Torn down mid-send: the leased record is discarded along with the rest of the buffer.+ // The dispatched send may still complete, an inherent and harmless race for an in-memory+ // emulator (teardown stops all FUTURE sends; it cannot recall one already in flight).+ return;+ }+ if (success) {+ st.forwarded++;+ st.consecutiveFailedProbes = 0;+ st.health = Health.HEALTHY;+ continue;+ }+ recordError(st, failure);+ if (terminal) {+ // The leased poison record is already out of the queue: drop it (do NOT wedge) and let+ // the records behind it flow. Its probe budget dies with it: the next record earns a fresh one.+ st.dropped++;+ st.consecutiveFailedProbes = 0;+ LOG.errorv(failure, "Dropped DynamoDB CDC record (terminal {0}) for stream {1}",+ st.lastErrorType, st.streamArn);+ continue;+ }+ // retryable: return the leased record to the head so per-destination FIFO is preserved.+ st.queue.addFirst(head);+ st.retried++;+ st.consecutiveFailedProbes++;+ st.health = Health.RETRYING;+ if (st.consecutiveFailedProbes >= MAX_PROBES) {+ long gaveUp = st.queue.size();+ st.dropped += gaveUp;+ st.queue.clear();+ st.health = Health.GAVE_UP;+ st.drainScheduled = false;+ // Reset the budget so a later event starts a fresh episode: a transient outage must+ // not permanently wedge delivery: the next forward re-attempts and self-heals.+ st.consecutiveFailedProbes = 0;+ LOG.errorv("CDC delivery gave up for stream {0} after {1} attempts; dropped {2} buffered "+ + "record(s). Last error: {3}",+ st.streamArn, MAX_PROBES, gaveUp, st.lastErrorMessage);+ return;+ }+ scheduleDrain(st, backoff(st.consecutiveFailedProbes));+ return;+ }+ }+ }++ /**+ * Schedule a drain run. The caller MUST hold {@code st}'s monitor. A scheduler fault (e.g.+ * {@link java.util.concurrent.RejectedExecutionException} once the executor is shutting down) must+ * never escape into the DynamoDB write path and must not wedge the destination: on failure the+ * {@code drainScheduled} flag is cleared so a later event can schedule a fresh drain.+ */+ private void scheduleDrain(DestinationState st, long delayMillis) {+ st.drainScheduled = true;+ long ep = st.epoch;+ try {+ scheduler.schedule(() -> drain(st, ep), delayMillis);+ } catch (RuntimeException rex) {+ st.drainScheduled = false;+ LOG.warnv(rex, "CDC drain scheduling failed for stream {0}; buffered records await a later event",+ st.streamArn);+ }+ }++ private static boolean isTerminal(AwsException e) {+ String code = e.getErrorCode();+ return "InvalidArgumentException".equals(code) || "ValidationException".equals(code);+ }++ private static long backoff(int probe) {+ long shifted = INITIAL_BACKOFF_MS * (1L << Math.min(probe - 1, 20));+ return Math.min(MAX_BACKOFF_MS, shifted);+ }++ private static void recordError(DestinationState st, Throwable failure) {+ st.lastErrorType = failure instanceof AwsException a && a.getErrorCode() != null+ ? a.getErrorCode() : failure.getClass().getSimpleName();+ st.lastErrorMessage = failure.getMessage();+ }++ // ──────────────────────────── Lifecycle ────────────────────────────++ /**+ * Disable of a specific destination: discard its buffer and stop any in-flight drain (epoch bump).+ * The state is RETAINED (not removed) so its cumulative delivery stats stay inspectable and a+ * subsequent re-enable reuses it; only a table deletion removes it.+ */+ public void onDestinationDisabled(String accountId, String region, String tableName, String streamArn) {+ DestinationState st = states.get(accountId + "|" + region + "|" + tableName + "|" + streamArn);+ if (st != null) {+ invalidate(st);+ }+ }++ /** Table deletion: discard all of its destinations' buffers, stop their drains, and drop their state. */+ public void onTableDeleted(String accountId, String region, String tableName) {+ String prefix = accountId + "|" + region + "|" + tableName + "|";+ states.entrySet().removeIf(entry -> {+ if (entry.getKey().startsWith(prefix)) {+ invalidate(entry.getValue());+ return true;}+ return false;+ });+ }++ private void invalidate(DestinationState st) {+ synchronized (st) {+ st.epoch++; // a scheduled drain from the old epoch will abort on its next check+ st.discardedOnDisable += st.queue.size();+ st.queue.clear();+ st.drainScheduled = false;+ st.consecutiveFailedProbes = 0; // a re-enable starts a fresh delivery episode+ st.health = Health.HEALTHY;}}- /** Process-local count of CDC events dropped because forwarding to Kinesis threw. */+ @PreDestroy+ public void shutdown() {+ scheduler.shutdown();+ }++ /** Blocks until every destination's queue is empty and no drain is scheduled, or the timeout elapses. */+ public boolean awaitIdle(Duration timeout) {+ long deadlineNanos = System.nanoTime() + timeout.toNanos();+ while (System.nanoTime() < deadlineNanos) {+ boolean idle = true;+ for (DestinationState st : states.values()) {+ synchronized (st) {+ if (!st.queue.isEmpty() || st.drainScheduled) {+ idle = false;+ break;+ }+ }+ }+ if (idle) {+ return true;+ }+ try {+ Thread.sleep(5);+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ return false;+ }+ }+ return false;+ }++ // ──────────────────────────── Observability ────────────────────────────++ /** Total CDC records permanently dropped across all destinations (terminal, give-up, or overflow). */public long getForwardFailureCount() {- return forwardFailures.get();+ long sum = 0;+ for (DestinationState st : states.values()) {+ synchronized (st) {+ sum += st.dropped;+ }+ }+ return sum;}+ /** Immutable per-destination delivery-health snapshots (each copied under its destination lock). */+ List<DestinationForwardingStats> forwardingStats() {+ List<DestinationForwardingStats> out = new ArrayList<>();+ for (DestinationState st : states.values()) {+ synchronized (st) {+ out.add(new DestinationForwardingStats(st.health.name(), st.forwarded,+ st.retried, st.dropped, st.overflowed, st.discardedOnDisable,+ st.queue.size(), st.lastErrorType));+ }+ }+ return out;+ }++ // ──────────────────────────── Payload construction (unchanged) ────────────────────────────+private ObjectNode buildPayload(String eventName, JsonNode keys,- JsonNode newImage, JsonNode oldImage,- String tableName, String region, Instant timestamp) {+ JsonNode newImage, JsonNode oldImage,+ String tableName, String region, Instant timestamp) {ObjectNode payload = objectMapper.createObjectNode();payload.put("awsRegion", region);payload.put("eventID", UUID.randomUUID().toString());@@ -103,7 +447,9 @@ public class KinesisStreamingForwarder {private ObjectNode buildKeys(JsonNode item, TableDefinition table) {ObjectNode keys = objectMapper.createObjectNode();- if (item == null) return keys;+ if (item == null) {+ return keys;+ }for (KeySchemaElement ks : table.getKeySchema()) {String attrName = ks.getAttributeName();if (item.has(attrName)) {@@ -114,19 +460,31 @@ public class KinesisStreamingForwarder {}private String extractPartitionKey(JsonNode keys, TableDefinition table) {- if (keys == null || keys.isEmpty()) return "default";+ if (keys == null || keys.isEmpty()) {+ return "default";+ }String pkName = table.getPartitionKeyName();JsonNode pkValue = keys.get(pkName);- if (pkValue == null) return "default";- if (pkValue.has("S")) return pkValue.get("S").asText();- if (pkValue.has("N")) return pkValue.get("N").asText();- if (pkValue.has("B")) return pkValue.get("B").asText();+ if (pkValue == null) {+ return "default";+ }+ if (pkValue.has("S")) {+ return pkValue.get("S").asText();+ }+ if (pkValue.has("N")) {+ return pkValue.get("N").asText();+ }+ if (pkValue.has("B")) {+ return pkValue.get("B").asText();+ }return pkValue.toString();}private String extractStreamName(String streamArn) {int idx = streamArn.lastIndexOf('/');- if (idx >= 0) return streamArn.substring(idx + 1);+ if (idx >= 0) {+ return streamArn.substring(idx + 1);+ }return streamArn;}}diff --git a/src/main/java/io/github/hectorvent/floci/services/kinesis/KinesisService.java b/src/main/java/io/github/hectorvent/floci/services/kinesis/KinesisService.javaindex 2809c1b23f..75bde14c4e 100644--- a/src/main/java/io/github/hectorvent/floci/services/kinesis/KinesisService.java+++ b/src/main/java/io/github/hectorvent/floci/services/kinesis/KinesisService.java@@ -557,31 +557,65 @@ public class KinesisService implements ResourceProvider {public PutRecordResult putRecordWithShardId(String streamName, byte[] data, String partitionKey,String explicitHashKey, String region) {+ return putRecordInternal(null, streamName, data, partitionKey, explicitHashKey, region);+ }++ /**+ * Append a record on behalf of a specific account. Used by producers that run outside request scope+ * (the DynamoDB TTL sweep and DynamoDB→Kinesis CDC forwarding) so the record lands in the table+ * owner's stream rather than the ambient/default account's same-named stream. A {@code null}+ * accountId behaves exactly like {@link #putRecord}.+ */+ public String putRecordForAccount(String accountId, String streamName, byte[] data, String partitionKey,+ String region) {+ return putRecordInternal(accountId, streamName, data, partitionKey, null, region).sequenceNumber();+ }++ private PutRecordResult putRecordInternal(String accountId, String streamName, byte[] data,+ String partitionKey, String explicitHashKey, String region) {String key = regionKey(region, streamName);- KinesisStream stream = resolveStream(streamName, region);- validateExplicitHashKey(explicitHashKey);- validateRecordSize(stream, data, partitionKey);putRecordBeforeLockHook.run();// Serialize shard selection, sequence allocation, append and persistence per stream so// concurrent producers can neither lose an append (plain ArrayList) nor store records// out of sequence order (which would break AT_/AFTER_SEQUENCE_NUMBER scans and WAL replay).+ // The lock is keyed on region+streamName (NOT the account): an account-qualified key would stop+ // explicit-account appends from serializing against request-path producers on the same stream,+ // reintroducing the append race. Cross-account same-named streams therefore share one monitor+ // (over-coarse but safe).synchronized (lockFor(key)) {- // Re-resolve under the lock. deleteStream holds this same monitor, so a stream deleted- // between the resolve above and here is now absent from the store. Appending to and- // persisting the stale pre-delete instance would resurrect a deleted stream, so bind to- // the instance currently in the store and fail like any missing stream if it is gone.- KinesisStream current = resolveStream(streamName, region);+ // Resolve under the lock, not before it. Resolving a legacy unprefixed stream MIGRATES it,+ // which is a write: the explicit-account path moves it into the account partition behind an+ // ownership predicate, while AccountAwareStorageBackend.get adopts it into the request+ // account with no ownership check and no synchronization. Resolved outside this monitor,+ // an ambient producer and the owner's producer could migrate the same stream concurrently+ // and fork it across partitions. deleteStream holds this same monitor too, so a stream+ // deleted before we get here is absent and this fails like any other missing stream rather+ // than appending to a stale instance and resurrecting it.+ KinesisStream current = resolveStreamForAccountMigrating(accountId, streamName, region);+ // Ordered exactly as before this moved under the monitor: a missing stream still reports+ // ResourceNotFoundException ahead of an invalid ExplicitHashKey, and the size check runs+ // against the same lock-protected instance that shard selection and the append use.+ validateExplicitHashKey(explicitHashKey);+ validateRecordSize(current, data, partitionKey);KinesisShard shard = selectShard(current, partitionKey, explicitHashKey);String sequenceNumber = String.valueOf(sequenceGenerator.incrementAndGet());putRecordAppendHook.run();KinesisRecord record = new KinesisRecord(data, partitionKey, sequenceNumber, Instant.now());shard.addRecord(record);- store.put(key, current);+ persistStream(accountId, key, current);return new PutRecordResult(sequenceNumber, shard.getShardId());}}+ private void persistStream(String accountId, String key, KinesisStream stream) {+ if (accountId != null && store instanceof AccountAwareStorageBackend<KinesisStream> aware) {+ aware.putForAccount(accountId, key, stream);+ } else {+ store.put(key, stream);+ }+ }+public String getShardIterator(String streamName, String shardId, String type, String sequenceNumber, String region) {return getShardIterator(streamName, shardId, type, sequenceNumber, null, region);}@@ -806,6 +840,24 @@ public class KinesisService implements ResourceProvider {return resolveStream(streamName, region);}+ /**+ * Resolve for an out-of-request-scope PRODUCER (CDC forwarding, TTL sweep), migrating a legacy+ * unprefixed stream into the account partition on write, mirroring the ambient {@code store.get()}+ * fallback the producer used before account-scoping, so an installation with an existing CDC+ * destination and an unmigrated stream keeps forwarding after upgrade. Ownership is validated by the+ * stream's own accountId/ARN so one tenant cannot adopt another's unprefixed stream.+ */+ private KinesisStream resolveStreamForAccountMigrating(String accountId, String streamName, String region) {+ if (accountId != null && store instanceof AccountAwareStorageBackend<KinesisStream> aware) {+ return aware.getForAccountMigratingLegacy(accountId, regionKey(region, streamName),+ s -> accountId.equals(s.getAccountId())+ || (s.getStreamArn() != null && s.getStreamArn().contains(":" + accountId + ":")))+ .orElseThrow(() -> new AwsException("ResourceNotFoundException",+ "Stream " + streamName + " not found", 400));+ }+ return resolveStream(streamName, region);+ }+public String getShardIteratorForAccount(String accountId, String streamName, String shardId,String type, String sequenceNumber, String region) {KinesisStream stream = resolveStreamForAccount(accountId, streamName, region);diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackendTest.java b/src/test/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackendTest.javaindex d789ea5527..aa06cd89b6 100644--- a/src/test/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackendTest.java+++ b/src/test/java/io/github/hectorvent/floci/core/storage/AccountAwareStorageBackendTest.java@@ -6,8 +6,16 @@ import java.util.List;import java.util.Map;import java.util.Optional;import java.util.Set;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.Future;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.TimeoutException;+import java.util.function.Predicate;import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertThrows;import static org.junit.jupiter.api.Assertions.assertTrue;class AccountAwareStorageBackendTest {@@ -210,4 +218,66 @@ class AccountAwareStorageBackendTest {new AccountAwareStorageBackend.AccountEntry<>("000000000000", "us-east-1::legacy", "legacy")),Set.copyOf(storage.scanAllAccountEntries(key -> key.startsWith("us-east-1::"))));}++ /**+ * {@code get} and {@code getForAccountMigratingLegacy} both migrate a pre-multi-account entry, and+ * both delete the legacy key once they have copied it. Unless they hold the same monitor, two readers+ * resolving that key from different account contexts each copy it into their own partition and the two+ * copies then diverge. Made deterministic by parking the first reader inside its legacy read, at the+ * exact point where the unsynchronized version let the second reader in.+ */+ @Test+ void concurrentLegacyMigrationsDoNotForkTheEntryAcrossAccounts() throws Exception {+ InMemoryStorage<String, String> raw = new InMemoryStorage<>();+ raw.put("resource", "v");++ CountDownLatch parked = new CountDownLatch(1);+ CountDownLatch release = new CountDownLatch(1);+ StorageBackend<String, String> parking = new StorageBackend<>() {+ @Override public void put(String k, String v) { raw.put(k, v); }+ @Override public void delete(String k) { raw.delete(k); }+ @Override public List<String> scan(Predicate<String> f) { return raw.scan(f); }+ @Override public Set<String> keys() { return raw.keys(); }+ @Override public void flush() { raw.flush(); }+ @Override public void load() { raw.load(); }+ @Override public void clear() { raw.clear(); }+ @Override public Optional<String> get(String k) {+ if ("resource".equals(k) && parked.getCount() > 0) {+ parked.countDown();+ try {+ release.await();+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ }+ }+ return raw.get(k);+ }+ };+ AccountAwareStorageBackend<String> storage =+ new AccountAwareStorageBackend<>(parking, null, "111111111111");++ ExecutorService pool = Executors.newFixedThreadPool(2);+ try {+ Future<?> ambient = pool.submit(() -> storage.get("resource"));+ assertTrue(parked.await(2, TimeUnit.SECONDS), "the ambient reader reached its legacy read");++ Future<?> owner = pool.submit(+ () -> storage.getForAccountMigratingLegacy("222222222222", "resource", v -> true));+ // The owner must be unable to proceed: without a shared monitor it would migrate a second copy.+ assertThrows(TimeoutException.class, () -> owner.get(300, TimeUnit.MILLISECONDS),+ "the second migration ran concurrently with the first");++ release.countDown();+ ambient.get(2, TimeUnit.SECONDS);+ owner.get(2, TimeUnit.SECONDS);+ } finally {+ pool.shutdownNow();+ }++ assertTrue(raw.get("resource").isEmpty(), "the legacy key is consumed exactly once");+ long copies = List.of("111111111111/resource", "222222222222/resource").stream()+ .filter(k -> raw.get(k).isPresent())+ .count();+ assertEquals(1, copies, "the entry landed in exactly one account partition, not forked into both");+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbConcurrencyIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbConcurrencyIntegrationTest.javaindex 37d88e121c..046c16bbd3 100644--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbConcurrencyIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbConcurrencyIntegrationTest.java@@ -542,6 +542,11 @@ class DynamoDbConcurrencyIntegrationTest {});assertTrue(errors.isEmpty(), () -> "unexpected errors: " + errors);+ // CDC delivery is asynchronous (bounded best-effort retry queue): wait for the background drain to+ // flush every buffered record before reading the shard, otherwise this reads a partial stream.+ assertTrue(forwarder.awaitIdle(java.time.Duration.ofSeconds(10)),+ "the CDC drain must flush all buffered records");+// Paginated drain of the single shard.String shardId = kinesis.describeStream("cdc-stream", region).getShards().getFirst().getShardId();String iterator = kinesis.getShardIterator("cdc-stream", shardId, "TRIM_HORIZON", null, region);diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbKinesisStreamingIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbKinesisStreamingIntegrationTest.javaindex d394f758a5..71070e32ba 100644--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbKinesisStreamingIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbKinesisStreamingIntegrationTest.java@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.dynamodb;import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;import io.quarkus.test.junit.QuarkusTest;import io.restassured.response.Response;+import jakarta.inject.Inject;import org.junit.jupiter.api.BeforeAll;import org.junit.jupiter.api.MethodOrderer;import org.junit.jupiter.api.Order;@@ -12,6 +13,7 @@ import org.junit.jupiter.api.TestMethodOrder;import com.fasterxml.jackson.databind.JsonNode;import com.fasterxml.jackson.databind.ObjectMapper;+import java.time.Duration;import java.util.Base64;import static io.restassured.RestAssured.given;@@ -26,11 +28,20 @@ class DynamoDbKinesisStreamingIntegrationTest {private static final String KINESIS_CONTENT_TYPE = "application/x-amz-json-1.1";private static String kinesisStreamArn;+ // CDC forwarding is asynchronous (bounded best-effort retry queue): after a write enqueues, the send+ // happens on a background drain. Await it before reading the stream so these assertions are deterministic.+ @Inject+ KinesisStreamingForwarder forwarder;+@BeforeAllstatic void configureRestAssured() {RestAssuredJsonUtils.configureAwsContentTypes();}+ private void awaitCdcDelivery() {+ forwarder.awaitIdle(Duration.ofSeconds(5));+ }+@Test@Order(1)void setupKinesisStream() {@@ -178,6 +189,7 @@ class DynamoDbKinesisStreamingIntegrationTest {.then().statusCode(200);+ awaitCdcDelivery();String shardIterator = given().header("X-Amz-Target", "Kinesis_20131202.GetShardIterator").contentType(KINESIS_CONTENT_TYPE)@@ -248,6 +260,7 @@ class DynamoDbKinesisStreamingIntegrationTest {.then().statusCode(200);+ awaitCdcDelivery();String shardIterator = given().header("X-Amz-Target", "Kinesis_20131202.GetShardIterator").contentType(KINESIS_CONTENT_TYPE)@@ -292,6 +305,7 @@ class DynamoDbKinesisStreamingIntegrationTest {.then().statusCode(200);+ awaitCdcDelivery();String shardIterator = given().header("X-Amz-Target", "Kinesis_20131202.GetShardIterator").contentType(KINESIS_CONTENT_TYPE)diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbTtlForwardingTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbTtlForwardingTest.javaindex ea52c227b9..cc0cf2afa3 100644--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbTtlForwardingTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbTtlForwardingTest.java@@ -17,6 +17,7 @@ import io.github.hectorvent.floci.services.kinesis.model.KinesisRecord;import io.github.hectorvent.floci.services.kinesis.model.KinesisStream;import org.junit.jupiter.api.Test;+import java.time.Duration;import java.time.Instant;import java.util.ArrayList;import java.util.HashMap;@@ -27,15 +28,30 @@ import static org.junit.jupiter.api.Assertions.*;import static org.mockito.ArgumentMatchers.any;import static org.mockito.ArgumentMatchers.anyString;import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.verifyNoInteractions;import static org.mockito.Mockito.when;/*** TTL sweep must forward a REMOVE per expired item to Kinesis (happy path) and must isolate- * forwarding failures: a throwing forwarder cannot abort the sweep or block persistence.- * See issue #571.+ * forwarding from Kinesis health: because forwarding is now enqueue-only (delivered/retried on a+ * background drain, see {@link KinesisStreamingForwarderTest}), a broken Kinesis cannot abort the+ * sweep, block persistence, or silently drop the change events. See issue #571.*/class DynamoDbTtlForwardingTest {+ /** A Scheduler that never runs the drain, so buffered CDC records stay put for inspection. */+ private static final KinesisStreamingForwarder.Scheduler NO_DRAIN =+ new KinesisStreamingForwarder.Scheduler() {+ @Override+ public void schedule(Runnable task, long delayMillis) {+ // intentionally never executes the drain+ }++ @Override+ public void shutdown() {+ }+ };+private static final String REGION = "us-east-1";private final ObjectMapper mapper = new ObjectMapper();@@ -109,6 +125,8 @@ class DynamoDbTtlForwardingTest {table.getKinesisStreamingDestinations().add(new KinesisStreamingDestination(stream.getStreamArn()));svc.deleteExpiredItems();+ // Delivery is asynchronous now; wait for the background drain to flush before reading the stream.+ assertTrue(forwarder.awaitIdle(Duration.ofSeconds(5)), "CDC forwards should drain promptly");List<KinesisRecord> drained = drain(kinesis, "ttl-stream");assertEquals(expired, drained.size(), "one Kinesis record per expired item");@@ -131,9 +149,9 @@ class DynamoDbTtlForwardingTest {StorageBackend<String, Map<String, JsonNode>> itemStore = new InMemoryStorage<>();KinesisService throwingKinesis = mock(KinesisService.class);- when(throwingKinesis.putRecord(anyString(), any(byte[].class), anyString(), anyString()))- .thenThrow(new RuntimeException("kinesis unavailable"));- KinesisStreamingForwarder forwarder = new KinesisStreamingForwarder(throwingKinesis, mapper);+ // No-drain scheduler: the sweep only ENQUEUES, so Kinesis is never touched during the sweep at+ // all. (Retry/give-up drop accounting on the drain is covered by KinesisStreamingForwarderTest.)+ KinesisStreamingForwarder forwarder = new KinesisStreamingForwarder(throwingKinesis, mapper, NO_DRAIN);DynamoDbService svc = new DynamoDbService(tableStore, itemStore, new RegionResolver("us-east-1", "000000000000"), null, forwarder);@@ -143,11 +161,16 @@ class DynamoDbTtlForwardingTest {table.getKinesisStreamingDestinations().add(new KinesisStreamingDestination("arn:aws:kinesis:us-east-1:000000000000:stream/ttl-stream"));- // A forwarding failure must not propagate out of the sweep.+ // A dead Kinesis must not propagate out of the sweep, and must not be called synchronously by it.assertDoesNotThrow(svc::deleteExpiredItems);-- assertEquals(expired, forwarder.getForwardFailureCount(),- "each failed forward must be counted, not swallowed silently");+ verifyNoInteractions(throwingKinesis);++ // Every expired change event is safely BUFFERED (not silently dropped): the fix for the+ // drop-on-forward-exception gap. The drain would deliver these once Kinesis recovers.+ List<DestinationForwardingStats> stats = forwarder.forwardingStats();+ assertEquals(1, stats.size());+ assertEquals(expired, stats.get(0).queueDepth(), "each expired item's REMOVE must be enqueued");+ assertEquals(0L, forwarder.getForwardFailureCount(), "buffered records are not (yet) dropped");// Removed in-memory AND persisted: a fresh service over the same stores must not see them.for (int i = 0; i < expired; i++) {diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarderTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarderTest.javaindex 04800d1533..79b7415f7a 100644--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarderTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/KinesisStreamingForwarderTest.java@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.dynamodb;import com.fasterxml.jackson.databind.ObjectMapper;import com.fasterxml.jackson.databind.node.ObjectNode;+import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.services.dynamodb.model.AttributeDefinition;import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;import io.github.hectorvent.floci.services.dynamodb.model.KinesisStreamingDestination;@@ -10,96 +11,496 @@ import io.github.hectorvent.floci.services.kinesis.KinesisService;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;+import java.time.Duration;+import java.util.ArrayDeque;+import java.util.ArrayList;+import java.util.HashMap;import java.util.List;+import java.util.Map;+import java.util.concurrent.RejectedExecutionException;+import java.util.concurrent.atomic.AtomicBoolean;+import java.util.concurrent.atomic.AtomicInteger;+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;import static org.junit.jupiter.api.Assertions.assertEquals;-import static org.mockito.ArgumentMatchers.*;-import static org.mockito.Mockito.*;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.times;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.verifyNoInteractions;+import static org.mockito.Mockito.when;+/**+ * The CDC delivery contract (issue: DynamoDB→Kinesis records dropped on forward exception).+ *+ * <p>{@code forward} is proven ENQUEUE-ONLY (no synchronous Kinesis I/O, never throws); the actual send,+ * retry, terminal-drop, give-up, overflow, FIFO, and lifecycle behaviour are exercised deterministically+ * by driving a manual {@link KinesisStreamingForwarder.Scheduler} step by step: no real timers, no sleeps+ * in the delivery assertions.+ */class KinesisStreamingForwarderTest {+ private static final String ACCOUNT = "000000000000";+ private static final String STREAM_ARN = "arn:aws:kinesis:us-east-1:000000000000:stream/test-stream";+private KinesisService kinesisService;- private KinesisStreamingForwarder forwarder;private ObjectMapper objectMapper;+ private ManualScheduler scheduler;+ private KinesisStreamingForwarder forwarder;@BeforeEachvoid setUp() {kinesisService = mock(KinesisService.class);objectMapper = new ObjectMapper();- forwarder = new KinesisStreamingForwarder(kinesisService, objectMapper);+ scheduler = new ManualScheduler();+ forwarder = new KinesisStreamingForwarder(kinesisService, objectMapper, scheduler);+ }++ // ──────────────────────────── helpers ────────────────────────────++ /** A Scheduler the test drives by hand: tasks queue up and run only when the test asks. */+ static final class ManualScheduler implements KinesisStreamingForwarder.Scheduler {+ final ArrayDeque<Runnable> tasks = new ArrayDeque<>();+ final List<Long> delays = new ArrayList<>();+ boolean shutdown;++ @Override+ public void schedule(Runnable task, long delayMillis) {+ tasks.addLast(task);+ delays.add(delayMillis);+ }++ @Override+ public void shutdown() {+ shutdown = true;+ }++ boolean runOne() {+ Runnable r = tasks.pollFirst();+ if (r == null) {+ return false;+ }+ r.run();+ return true;+ }++ int runUntilQuiet() {+ int steps = 0;+ while (steps < 20_000 && runOne()) {+ steps++;+ }+ return steps;+ }+ }++ /** A Scheduler that rejects while {@code reject} is set (executor-shutdown simulation), else queues. */+ static final class RejectingScheduler implements KinesisStreamingForwarder.Scheduler {+ final ArrayDeque<Runnable> tasks = new ArrayDeque<>();+ boolean reject = true;++ @Override+ public void schedule(Runnable task, long delayMillis) {+ if (reject) {+ throw new RejectedExecutionException("executor is shutting down");+ }+ tasks.addLast(task);+ }++ @Override+ public void shutdown() {+ }++ boolean runOne() {+ Runnable r = tasks.pollFirst();+ if (r == null) {+ return false;+ }+ r.run();+ return true;+ }}- private TableDefinition createTable(String tableName) {- return new TableDefinition(tableName,+ private TableDefinition tableWithDestination(String... streamArns) {+ TableDefinition table = new TableDefinition("test-table",List.of(new KeySchemaElement("pk", "HASH")),List.of(new AttributeDefinition("pk", "S")));+ for (String arn : streamArns) {+ table.getKinesisStreamingDestinations().add(new KinesisStreamingDestination(arn));+ }+ return table;}- private ObjectNode createItem(String pk) {- ObjectNode item = objectMapper.createObjectNode();+ private ObjectNode item(String pk) {+ ObjectNode node = objectMapper.createObjectNode();ObjectNode pkValue = objectMapper.createObjectNode();pkValue.put("S", pk);- item.set("pk", pkValue);- return item;+ node.set("pk", pkValue);+ return node;}+ private DestinationForwardingStats onlyStat() {+ List<DestinationForwardingStats> stats = forwarder.forwardingStats();+ assertEquals(1, stats.size(), "expected exactly one destination state");+ return stats.get(0);+ }++ // ──────────────────────────── enqueue path ────────────────────────────+@Test- void forwardsToActiveDestination() {- TableDefinition table = createTable("test-table");- KinesisStreamingDestination dest = new KinesisStreamingDestination(- "arn:aws:kinesis:us-east-1:000000000000:stream/test-stream");- table.getKinesisStreamingDestinations().add(dest);+ void forwardOnlyEnqueues_neverCallsKinesisSynchronouslyNorThrows() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenThrow(new RuntimeException("kinesis down"));- when(kinesisService.putRecord(anyString(), any(byte[].class), anyString(), anyString()))+ assertDoesNotThrow(() -> forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT));++ // The write path did no Kinesis I/O: the record is buffered, not delivered and not dropped.+ verifyNoInteractions(kinesisService);+ DestinationForwardingStats s = onlyStat();+ assertEquals(1, s.queueDepth());+ assertEquals(0L, s.forwarded());+ assertEquals(0L, s.dropped());+ }++ @Test+ void skipsDisabledDestination() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ table.getKinesisStreamingDestinations().get(0).setDestinationStatus("DISABLED");++ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);++ assertTrue(forwarder.forwardingStats().isEmpty(), "a disabled destination is never enqueued");+ assertFalse(scheduler.runOne(), "nothing scheduled for a disabled destination");+ }++ @Test+ void skipsWhenNoDestinations() {+ forwarder.forward("INSERT", null, item("k1"), tableWithDestination(), "us-east-1", ACCOUNT);+ verifyNoInteractions(kinesisService);+ assertTrue(forwarder.forwardingStats().isEmpty());+ }++ // ──────────────────────────── happy-path drain ────────────────────────────++ @Test+ void deliversBufferedRecordOnDrain() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString())).thenReturn("seq-1");- forwarder.forward("INSERT", null, createItem("k1"), table, "us-east-1");+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);+ scheduler.runUntilQuiet();- verify(kinesisService).putRecord(eq("test-stream"), any(byte[].class), eq("k1"), eq("us-east-1"));+ verify(kinesisService).putRecordForAccount(eq(ACCOUNT), eq("test-stream"), any(byte[].class),+ eq("k1"), eq("us-east-1"));+ DestinationForwardingStats s = onlyStat();+ assertEquals(1L, s.forwarded());+ assertEquals(0, s.queueDepth());+ assertEquals("HEALTHY", s.health());}@Test- void skipsDisabledDestination() {- TableDefinition table = createTable("test-table");- KinesisStreamingDestination dest = new KinesisStreamingDestination(- "arn:aws:kinesis:us-east-1:000000000000:stream/test-stream");- dest.setDestinationStatus("DISABLED");- table.getKinesisStreamingDestinations().add(dest);+ void threadsTableOwnerAccountToThePut() {+ // The owning account is threaded through so an out-of-request-scope forward lands in the owner's stream.+ TableDefinition table = tableWithDestination("arn:aws:kinesis:us-east-1:111111111111:stream/test-stream");+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenReturn("seq-1");++ forwarder.forward("REMOVE", item("k1"), null, table, "us-east-1", "111111111111");+ scheduler.runUntilQuiet();++ verify(kinesisService).putRecordForAccount(eq("111111111111"), eq("test-stream"), any(byte[].class),+ eq("k1"), eq("us-east-1"));+ }++ @Test+ void preservesFifoAcrossARetry() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ List<String> delivered = new ArrayList<>();+ AtomicInteger calls = new AtomicInteger();+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenAnswer(inv -> {+ if (calls.getAndIncrement() == 0) {+ throw new RuntimeException("transient on the very first attempt");+ }+ delivered.add(inv.getArgument(3)); // partitionKey+ return "seq";+ });- forwarder.forward("INSERT", null, createItem("k1"), table, "us-east-1");+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);+ forwarder.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT);+ forwarder.forward("INSERT", null, item("k3"), table, "us-east-1", ACCOUNT);+ scheduler.runUntilQuiet();++ // k1 was retried, yet ordering is preserved: the head is not skipped past.+ assertEquals(List.of("k1", "k2", "k3"), delivered);+ DestinationForwardingStats s = onlyStat();+ assertEquals(3L, s.forwarded());+ assertEquals(1L, s.retried());+ assertEquals(0L, s.dropped());+ }++ // ──────────────────────────── failure classification ────────────────────────────++ @Test+ void unknownFailureIsRetriedNotDropped() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenThrow(new RuntimeException("transient"));+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);+ scheduler.runOne(); // exactly one drain attempt++ DestinationForwardingStats s = onlyStat();+ assertEquals(1, s.queueDepth(), "an unknown failure keeps the record buffered for retry");+ assertEquals(0L, s.dropped());+ assertEquals(1L, s.retried());+ assertEquals("RETRYING", s.health());+ }++ @Test+ void terminalFailureDropsThePoisonRecordWithoutWedgingTheQueue() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ List<String> delivered = new ArrayList<>();+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenAnswer(inv -> {+ String pk = inv.getArgument(3);+ if ("k1".equals(pk)) {+ throw new AwsException("ValidationException", "record too large", 400);+ }+ delivered.add(pk);+ return "seq";+ });++ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT); // poison+ forwarder.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT); // good+ scheduler.runUntilQuiet();++ // The poison record is dropped immediately; the good record behind it is still delivered.+ assertEquals(List.of("k2"), delivered);+ DestinationForwardingStats s = onlyStat();+ assertEquals(1L, s.dropped());+ assertEquals(1L, s.forwarded());+ assertEquals("ValidationException", s.lastErrorType());+ assertEquals(1L, forwarder.getForwardFailureCount());+ }++ @Test+ void givesUpAfterProbeBudgetThenSelfHealsOnALaterEvent() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ AtomicBoolean healthy = new AtomicBoolean(false);+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenAnswer(inv -> {+ if (!healthy.get()) {+ throw new RuntimeException("stream unavailable");+ }+ return "seq";+ });++ // Episode 1: persistent failure exhausts the probe budget and drops the buffered record.+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);+ scheduler.runUntilQuiet();++ DestinationForwardingStats gaveUp = onlyStat();+ assertEquals("GAVE_UP", gaveUp.health());+ assertEquals(1L, gaveUp.dropped());+ assertEquals(0, gaveUp.queueDepth());+ // Exactly MAX_PROBES (10) send attempts: the initial immediate run + 9 retry schedules, backoff+ // 250ms doubling to the 8s cap. Give-up fires on the 10th failure (no 11th schedule).+ assertEquals(List.of(0L, 250L, 500L, 1000L, 2000L, 4000L, 8000L, 8000L, 8000L, 8000L),+ List.copyOf(scheduler.delays));++ // Episode 2: a fresh event with a recovered stream re-attempts (budget was reset) and heals.+ healthy.set(true);+ forwarder.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT);+ scheduler.runUntilQuiet();++ DestinationForwardingStats healed = onlyStat();+ assertEquals("HEALTHY", healed.health());+ assertEquals(1L, healed.forwarded());+ assertEquals(1L, healed.dropped(), "the episode-1 drop is retained in the cumulative count");+ assertEquals(0, healed.queueDepth());+ }++ // ──────────────────────────── overflow ────────────────────────────++ @Test+ void overflowDropsOldestBeyondTheBufferCap() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ int over = 2;+ for (int i = 0; i < KinesisStreamingForwarder.MAX_BUFFERED + over; i++) {+ forwarder.forward("INSERT", null, item("k" + i), table, "us-east-1", ACCOUNT);+ }+ // Never drained: the buffer is capped and the oldest are evicted.+ DestinationForwardingStats s = onlyStat();+ assertEquals(KinesisStreamingForwarder.MAX_BUFFERED, s.queueDepth());+ assertEquals(over, s.overflowed());+ assertEquals(over, s.dropped());+ assertEquals(over, forwarder.getForwardFailureCount());+ }++ // ──────────────────────────── lifecycle ────────────────────────────++ @Test+ void disableDiscardsBufferAndStalesTheInFlightDrain() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);++ forwarder.onDestinationDisabled(ACCOUNT, "us-east-1", "test-table", STREAM_ARN);++ DestinationForwardingStats s = onlyStat(); // state retained on disable+ assertEquals(0, s.queueDepth());+ assertEquals(1L, s.discardedOnDisable());++ // The drain scheduled before the disable is now stale: running it is a no-op (epoch bumped).+ scheduler.runUntilQuiet();verifyNoInteractions(kinesisService);+ assertEquals(0L, onlyStat().forwarded());}@Test- void skipsWhenNoDestinations() {- TableDefinition table = createTable("test-table");+ void tableDeleteRemovesStateAndStalesTheInFlightDrain() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);- forwarder.forward("INSERT", null, createItem("k1"), table, "us-east-1");+ forwarder.onTableDeleted(ACCOUNT, "us-east-1", "test-table");+ assertTrue(forwarder.forwardingStats().isEmpty(), "table delete drops the destination state");+ scheduler.runUntilQuiet();verifyNoInteractions(kinesisService);}@Test- void continuesOnPutRecordFailure() {- TableDefinition table = createTable("test-table");- KinesisStreamingDestination dest1 = new KinesisStreamingDestination(- "arn:aws:kinesis:us-east-1:000000000000:stream/stream-1");- KinesisStreamingDestination dest2 = new KinesisStreamingDestination(- "arn:aws:kinesis:us-east-1:000000000000:stream/stream-2");- table.getKinesisStreamingDestinations().add(dest1);- table.getKinesisStreamingDestinations().add(dest2);+ void awaitIdleReflectsQueueState() {+ TableDefinition table = tableWithDestination(STREAM_ARN);+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenReturn("seq");- when(kinesisService.putRecord(eq("stream-1"), any(byte[].class), anyString(), anyString()))- .thenThrow(new RuntimeException("stream-1 failed"));- when(kinesisService.putRecord(eq("stream-2"), any(byte[].class), anyString(), anyString()))- .thenReturn("seq-1");+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);+ assertFalse(forwarder.awaitIdle(Duration.ofMillis(50)), "buffered + drain pending is not idle");++ scheduler.runUntilQuiet();+ assertTrue(forwarder.awaitIdle(Duration.ofMillis(50)), "drained empty is idle");+ }++ // ──────────────────────────── race / robustness ────────────────────────────++ @Test+ void overflowWhileTheHeadIsInFlightNeverEvictsTheRecordBeingSent() {+ // Deterministically reproduces the overflow-vs-in-flight race: while k0 is being sent, a burst of+ // producers overfills the buffer. The in-flight record must not be evicted, and nothing may be lost+ // without accounting (the peek-then-poll approach would evict k0 and then poll a different record).+ TableDefinition table = tableWithDestination(STREAM_ARN);+ int burst = KinesisStreamingForwarder.MAX_BUFFERED + 5;+ List<String> delivered = new ArrayList<>();+ AtomicBoolean filled = new AtomicBoolean(false);+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenAnswer(inv -> {+ String pk = inv.getArgument(3);+ if ("k0".equals(pk) && filled.compareAndSet(false, true)) {+ for (int i = 1; i <= burst; i++) {+ forwarder.forward("INSERT", null, item("k" + i), table, "us-east-1", ACCOUNT);+ }+ }+ delivered.add(pk);+ return "seq";+ });++ forwarder.forward("INSERT", null, item("k0"), table, "us-east-1", ACCOUNT);+ scheduler.runUntilQuiet();++ // k0 (in flight during the overflow) was delivered, not evicted; the 5 oldest buffered were dropped.+ assertEquals("k0", delivered.get(0));+ assertFalse(delivered.contains("k1"), "the oldest buffered record is the overflow victim, not k0");+ DestinationForwardingStats s = onlyStat();+ assertEquals(5L, s.overflowed());+ assertEquals(5L, s.dropped());+ assertEquals(burst + 1 - 5, s.forwarded());+ // Every one of the burst+1 records is accounted for exactly once: no silent loss.+ assertEquals(burst + 1, s.forwarded() + s.dropped());+ assertEquals(0, s.queueDepth());+ }++ @Test+ void schedulerRejectionNeitherEscapesForwardNorWedgesTheDestination() {+ RejectingScheduler rejecting = new RejectingScheduler();+ KinesisStreamingForwarder f = new KinesisStreamingForwarder(kinesisService, objectMapper, rejecting);+ TableDefinition table = tableWithDestination(STREAM_ARN);+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenReturn("seq");++ // Executor down: scheduling the initial drain is rejected, but forward must not throw, and the+ // record stays buffered (not dropped).+ assertDoesNotThrow(() -> f.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT));+ assertEquals(1, f.forwardingStats().get(0).queueDepth());++ // Executor recovers: a later event must be able to schedule a fresh drain (no wedge), and BOTH the+ // previously-buffered and the new record drain in FIFO order.+ rejecting.reject = false;+ f.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT);+ while (rejecting.runOne()) {+ // drive the recovered drain to completion+ }+ DestinationForwardingStats s = f.forwardingStats().get(0);+ assertEquals(2L, s.forwarded());+ assertEquals(0, s.queueDepth());+ }++ @Test+ void terminalHeadDropGivesTheNextRecordItsOwnFullProbeBudget() {+ // k1 fails transiently then terminally; k2 must then get a FRESH MAX_PROBES budget, not the leftover.+ TableDefinition table = tableWithDestination(STREAM_ARN);+ Map<String, Integer> attempts = new HashMap<>();+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenAnswer(inv -> {+ String pk = inv.getArgument(3);+ int n = attempts.merge(pk, 1, Integer::sum);+ if ("k1".equals(pk)) {+ if (n <= 3) {+ throw new RuntimeException("transient");+ }+ throw new AwsException("ValidationException", "poison", 400); // 4th attempt: terminal+ }+ throw new RuntimeException("k2 stream down"); // k2 always transient+ });++ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);+ forwarder.forward("INSERT", null, item("k2"), table, "us-east-1", ACCOUNT);+ scheduler.runUntilQuiet();++ assertEquals(4, attempts.get("k1"), "3 transient retries then the terminal attempt");+ // If the budget had not been reset on the terminal drop, k2 would give up after only+ // MAX_PROBES - 3 attempts. A fresh budget means exactly MAX_PROBES attempts for k2.+ assertEquals(KinesisStreamingForwarder.MAX_PROBES, attempts.get("k2"));+ DestinationForwardingStats s = onlyStat();+ assertEquals(2L, s.dropped()); // k1 (terminal) + k2 (give-up)+ assertEquals("GAVE_UP", s.health());+ }++ @Test+ void disableDuringInFlightSendCompletesButDoesNotResurrectOrDoubleCount() {+ // Characterizes the inherent teardown race (documented as best-effort): a send dispatched before a+ // disable may complete, but the torn-down destination is neither re-counted nor resurrected.+ TableDefinition table = tableWithDestination(STREAM_ARN);+ when(kinesisService.putRecordForAccount(anyString(), anyString(), any(byte[].class), anyString(), anyString()))+ .thenAnswer(inv -> {+ forwarder.onDestinationDisabled(ACCOUNT, "us-east-1", "test-table", STREAM_ARN);+ return "seq";+ });- forwarder.forward("INSERT", null, createItem("k1"), table, "us-east-1");+ forwarder.forward("INSERT", null, item("k1"), table, "us-east-1", ACCOUNT);+ scheduler.runUntilQuiet();- verify(kinesisService).putRecord(eq("stream-1"), any(byte[].class), anyString(), anyString());- verify(kinesisService).putRecord(eq("stream-2"), any(byte[].class), anyString(), anyString());- assertEquals(1L, forwarder.getForwardFailureCount(),- "the single failed forward must be counted");+ // The dispatched send ran exactly once; its result is discarded (epoch bumped), the buffer is empty,+ // and no further drain runs.+ verify(kinesisService, times(1)).putRecordForAccount(anyString(), anyString(), any(byte[].class),+ anyString(), anyString());+ DestinationForwardingStats s = onlyStat();+ assertEquals(0, s.queueDepth());+ assertEquals(0L, s.forwarded());}}diff --git a/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceAccountTest.java b/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceAccountTest.javanew file mode 100644index 0000000000..cd85504ec8--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceAccountTest.java@@ -0,0 +1,102 @@+package io.github.hectorvent.floci.services.kinesis;++import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.InMemoryStorage;+import io.github.hectorvent.floci.services.kinesis.model.KinesisConsumer;+import io.github.hectorvent.floci.services.kinesis.model.KinesisShard;+import io.github.hectorvent.floci.services.kinesis.model.KinesisStream;+import com.fasterxml.jackson.databind.ObjectMapper;+import org.junit.jupiter.api.Test;++import java.nio.charset.StandardCharsets;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertNotNull;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.mockito.Mockito.mock;++/**+ * Verifies that {@code putRecordForAccount} resolves, persists, and legacy-migrates against the SPECIFIED+ * account rather than the ambient/default one, so an out-of-request-scope producer (TTL sweep, CDC+ * forwarding) never writes into another tenant's same-named stream and keeps working after an upgrade+ * from unprefixed (pre-multi-account) storage. Real {@link KinesisService} over in-memory account-aware+ * storage (no Vert.x → runs in the offline sandbox).+ */+class KinesisServiceAccountTest {++ private static final String KEY_S = "us-east-1::s";+ // jsr310 module registered so cloning a shard's Instant (creationTimestamp) round-trips.+ private final ObjectMapper mapper = new ObjectMapper().findAndRegisterModules();++ private KinesisService serviceOver(AccountAwareStorageBackend<KinesisStream> streams) {+ return new KinesisService(streams, AccountAwareStorageBackend.inMemory("000000000000"),+ mock(RegionResolver.class));+ }++ private static int recordCount(KinesisStream stream) {+ return stream.getShards().stream().mapToInt(KinesisShard::recordCount).sum();+ }++ private KinesisStream cloneStream(KinesisStream template) {+ try {+ return mapper.treeToValue(mapper.valueToTree(template), KinesisStream.class);+ } catch (Exception e) {+ throw new RuntimeException(e);+ }+ }++ @Test+ void putRecordForAccountResolvesAgainstThatAccountNotDefault() {+ KinesisService svc = serviceOver(AccountAwareStorageBackend.inMemory("000000000000"));+ svc.createStream("s", 1, "us-east-1"); // created in the ambient/default account (000000000000)+ byte[] data = "x".getBytes(StandardCharsets.UTF_8);++ assertNotNull(svc.putRecord("s", data, "pk", "us-east-1"));+ assertNotNull(svc.putRecordForAccount(null, "s", data, "pk", "us-east-1"));+ assertNotNull(svc.putRecordForAccount("000000000000", "s", data, "pk", "us-east-1"));++ AwsException ex = assertThrows(AwsException.class,+ () -> svc.putRecordForAccount("111111111111", "s", data, "pk", "us-east-1"));+ assertEquals("ResourceNotFoundException", ex.getErrorCode());+ }++ @Test+ void putRecordForAccountPersistsToThatAccountPartition() {+ AccountAwareStorageBackend<KinesisStream> streams = AccountAwareStorageBackend.inMemory("000000000000");+ KinesisService svc = serviceOver(streams);+ svc.createStream("s", 1, "us-east-1"); // in the default account+ // Seed an INDEPENDENT same-named stream in account 111 (deep copy so the append can't leak back).+ KinesisStream in111 = cloneStream(streams.getForAccount("000000000000", KEY_S).orElseThrow());+ streams.putForAccount("111111111111", KEY_S, in111);++ svc.putRecordForAccount("111111111111", "s", "x".getBytes(StandardCharsets.UTF_8), "pk", "us-east-1");++ // Persisted into account 111's partition only; the default partition is untouched. (A regression+ // that used the ambient put would land the record in the default account here.)+ assertEquals(1, recordCount(streams.getForAccount("111111111111", KEY_S).orElseThrow()));+ assertEquals(0, recordCount(streams.getForAccount("000000000000", KEY_S).orElseThrow()));+ }++ @Test+ void putRecordForAccountMigratesLegacyUnprefixedStream() {+ // Upgrade scenario: a pre-multi-account (unprefixed) stream owned by account 111 exists in the+ // raw delegate. An out-of-request-scope forward for 111 must migrate + use it, not drop as RNF.+ InMemoryStorage<String, KinesisStream> raw = new InMemoryStorage<>();+ AccountAwareStorageBackend<KinesisStream> streams = new AccountAwareStorageBackend<>(raw, null, "000000000000");+ KinesisService svc = serviceOver(streams);++ // Build a shard-carrying stream (via createStream on a throwaway key) and place a copy under the+ // UNPREFIXED legacy key, owned (by ARN) by account 111.+ KinesisStream legacy = cloneStream(svc.createStream("tmpl", 1, "us-east-1"));+ legacy.setStreamArn("arn:aws:kinesis:us-east-1:111111111111:stream/s2");+ raw.put("us-east-1::s2", legacy); // unprefixed = legacy pre-multi-account data++ // Migrates on write and appends; no ResourceNotFoundException.+ assertNotNull(svc.putRecordForAccount("111111111111", "s2", "x".getBytes(StandardCharsets.UTF_8),+ "pk", "us-east-1"));+ // The record is now in account 111's partition (migrated).+ assertEquals(1, recordCount(streams.getForAccount("111111111111", "us-east-1::s2").orElseThrow()));+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceTest.javaindex ec20fc862d..16c95adffd 100644--- a/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/kinesis/KinesisServiceTest.java@@ -835,7 +835,8 @@ class KinesisServiceTest {* append lock, the producer must fail cleanly (ResourceNotFoundException) rather than append to* and re-persist the stale instance. Deterministic via the pre-lock hook: the producer parks* WITHOUT holding the monitor, so the deleter wins the lock first. Fails on code lacking the- * in-lock re-resolve: the producer would resurrect the stream and throw nothing.+ * in-lock resolve: a producer that resolved before the lock would append to and re-persist the+ * stale pre-delete instance, resurrecting the stream and throwing nothing.*/@Testvoid putRecord_failsCleanly_whenStreamDeletedBeforeLockAcquired() throws Exception {