Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

fix(dynamodb): route stream consumers through one account-scoped reader (#4394)

Mario Toffia2df052181d84
diff --git a/docs/services/lambda.md b/docs/services/lambda.md
index d8afca65a4..fd1e04aad1 100644
--- a/docs/services/lambda.md
+++ b/docs/services/lambda.md
@@ -849,7 +849,7 @@ aws lambda update-function-code \
Connect Lambda to SQS, Kinesis, or DynamoDB Streams. Self-managed Apache Kafka event source mappings are accepted, validated, persisted, and returned on the wire, but Floci does not run an active Kafka consumer poller:
-For DynamoDB Streams mappings, Floci retries failed batches with exponential backoff, honors `MaximumRetryAttempts` and `MaximumRecordAgeInSeconds`, and sends discarded batches to configured SQS, SNS, or S3 `DestinationConfig.OnFailure` destinations. With `BisectBatchOnFunctionError`, a batch that fails with a function error is split in half and retried, narrowing it down to the failing record, which then follows the normal retry and record age rules. Splits do not count as retry attempts, and only function errors split: throttles do not, and a partial batch response retries from the reported record. If the OnFailure destination refuses a discarded batch, Floci keeps the checkpoint and retries the send with backoff instead of advancing past the batch.
+For DynamoDB Streams mappings, Floci retries failed batches with exponential backoff, honors `MaximumRetryAttempts` and `MaximumRecordAgeInSeconds`, and sends discarded batches to configured SQS, SNS, or S3 `DestinationConfig.OnFailure` destinations. With `BisectBatchOnFunctionError`, a batch that fails with a function error is split in half and retried, narrowing it down to the failing record, which then follows the normal retry and record age rules. Splits do not count as retry attempts, and only function errors split: throttles do not, and a partial batch response retries from the reported record. If the OnFailure destination refuses a discarded batch, Floci logs the failure, drops the batch and advances past it.
A DynamoDB Streams mapping created with `StartingPosition: LATEST` delivers only records written after it is created, so a stream that is empty at creation delivers everything written later. After a restart it still resumes from the trim horizon, because native stream records are volatile. Deleting or disabling a mapping stops any poll that has not yet invoked the function. An invocation already running completes, but its checkpoint is not saved: a deleted mapping is never recreated and its result is dropped, and a disabled mapping re-reads that batch when it is enabled again.
diff --git a/docs/services/pipes.md b/docs/services/pipes.md
index fd87ff2f5b..d5cec1e65b 100644
--- a/docs/services/pipes.md
+++ b/docs/services/pipes.md
@@ -119,9 +119,23 @@ the field on a source its parameter block does not describe, for example
!!! note "Enforcement status"
The configured `ParallelizationFactor` is persisted and returned on the wire, but the poller
does not yet process concurrent batches per shard. Floci opens an iterator on a single shard
- (`shardId-000000000000`) per Kinesis or DynamoDB Streams pipe and delivers one batch at a time
- regardless of the configured value. Multi-shard polling and real per-shard concurrency are
- tracked as follow-ups.
+ (`shardId-000000000000`) per Kinesis pipe, and reads the shards of a DynamoDB stream one batch
+ at a time, regardless of the configured value. Multi-shard Kinesis polling and real per-shard
+ concurrency are tracked as follow-ups.
+
+## DynamoDB Streams Source
+
+- `StartingPosition` `TRIM_HORIZON` and `LATEST` are honored. `LATEST` is pinned on the pipe's
+ first poll, so only records written after that poll are delivered.
+- Progress survives `StopPipe` and `StartPipe`: a restarted pipe resumes after the last record it
+ delivered or sent to the DLQ. The progress is kept in memory, so it is lost when Floci restarts.
+- Records are delivered in the AWS DynamoDB Streams record shape (`eventName`, `dynamodb.Keys`,
+ `dynamodb.NewImage` and so on) with `eventSourceARN` set to the stream ARN, so filter patterns on
+ `dynamodb.*` fields match as on AWS.
+- A failed batch is retried until it is delivered or sent to the `DeadLetterConfig` queue. When
+ some records of a batch reach a non-Lambda target and others fail, the whole batch is retried.
+- `MaximumRetryAttempts`, `MaximumRecordAgeInSeconds` and `OnPartialBatchItemFailure` are accepted
+ and returned but not enforced, so they do not bound or split those retries.
## Enrichment
diff --git a/docs/services/redshift.md b/docs/services/redshift.md
index a9f8837cb9..0092c1b78b 100644
--- a/docs/services/redshift.md
+++ b/docs/services/redshift.md
@@ -79,8 +79,11 @@ Each stream record is written to a stable landing table named `floci_zetl_<integ
The landing table stores the event id, event name, source, region, sequence number, creation time,
and DynamoDB keys and images as JSON text. Record event ids are unique, so retries are idempotent.
-The integration consumer has its own stream checkpoint and does not share Lambda event source
-mapping state. Deleting an integration stops its consumer while retaining the landing table.
+The integration consumer keeps its own checkpoint for each stream shard and does not share Lambda
+event source mapping state. A batch is checkpointed only once it is written, so a failed write is
+retried with the same records. After a Floci restart the consumer resumes from the trim horizon,
+because native stream records are volatile. Deleting an integration stops its consumer while
+retaining the landing table.
Items already present in the source table when `CreateIntegration` runs are backfilled into the
landing table with a paginated `Scan`, one page per poll tick, and the scan resumes after a Floci
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java
index a9d3bb078e..14ed108f3f 100644
--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbService.java
@@ -795,7 +795,7 @@ public class DynamoDbService {
if (streamService == null) {
return table;
}
- streamService.disableStream(table.getTableName(), region);
+ streamService.disableStream(table.getTableArn());
table.setStreamEnabled(false);
persistTable(tableName, table, region);
return table;
@@ -820,7 +820,7 @@ public class DynamoDbService {
itemStore.delete(storageKey);
}
if (streamService != null) {
- streamService.deleteStream(canonicalTableName, region);
+ streamService.deleteStream(table.getTableArn());
}
if (kinesisForwarder != null) {
// Discard any buffered CDC records and stop draining: the destination stream is gone.
@@ -948,7 +948,7 @@ public class DynamoDbService {
String ownerAccountId = regionResolver.getAccountId();
Runnable streamEvent = () -> {
if (streamService != null) {
- streamService.captureEvent(canonicalTableName, eventName, existing, item, table, region);
+ streamService.captureEvent(eventName, existing, item, table, region);
}
if (kinesisForwarder != null) {
kinesisForwarder.forward(eventName, existing, item, table, region, ownerAccountId);
@@ -1050,7 +1050,7 @@ public class DynamoDbService {
String ownerAccountId = regionResolver.getAccountId();
Runnable streamEvent = () -> {
if (streamService != null) {
- streamService.captureEvent(canonicalTableName, "REMOVE", removed, null, table, region);
+ streamService.captureEvent("REMOVE", removed, null, table, region);
}
if (kinesisForwarder != null) {
kinesisForwarder.forward("REMOVE", removed, null, table, region, ownerAccountId);
@@ -1265,7 +1265,7 @@ public class DynamoDbService {
String ownerAccountId = regionResolver.getAccountId();
Runnable streamEvent = () -> {
if (streamService != null) {
- streamService.captureEvent(canonicalTableName, "MODIFY", existing, item, table, region);
+ streamService.captureEvent("MODIFY", existing, item, table, region);
}
if (kinesisForwarder != null) {
kinesisForwarder.forward("MODIFY", existing, item, table, region, ownerAccountId);
@@ -2753,8 +2753,7 @@ public class DynamoDbService {
}
deletedForTable++;
if (streamService != null) {
- streamService.captureEvent(scan.table().getTableName(), "REMOVE", removed, null,
- scan.table(), scan.region());
+ streamService.captureEvent("REMOVE", removed, null, scan.table(), scan.region());
}
if (kinesisForwarder != null) {
// Out of request scope here: pass the table owner's account explicitly so the CDC
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamService.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamService.java
index d31c9482b0..f5327c2311 100644
--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamService.java
@@ -1,6 +1,8 @@
package io.github.hectorvent.floci.services.dynamodb;
+import io.github.hectorvent.floci.core.common.AwsArnUtils;
import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.StorageBackend;
import io.github.hectorvent.floci.core.storage.StorageFactory;
import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
@@ -32,13 +34,14 @@ public class DynamoDbStreamService {
private static final Logger LOG = Logger.getLogger(DynamoDbStreamService.class);
- public static final String SHARD_ID = "shardId-0000000001-00000000001";
+ static final String SHARD_ID = "shardId-0000000001-00000000001";
static final int MAX_RECORDS = 1000;
private static final String ZERO_SEQUENCE_NUMBER = "000000000000000000000";
private static final DateTimeFormatter STREAM_LABEL_FORMAT =
DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS").withZone(ZoneOffset.UTC);
+ /** Keyed by table ARN, which is unique per partition, region, account and table. */
private final ConcurrentHashMap<String, StreamDescription> streams = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, ConcurrentLinkedDeque<DynamoDbStreamRecord>> records =
new ConcurrentHashMap<>();
@@ -61,13 +64,23 @@ public class DynamoDbStreamService {
private void loadPersistedStreams(StorageBackend<String, TableDefinition> tableStore) {
- if (tableStore == null) return;
- for (String tableKey : tableStore.keys()){
- String region = tableKey.split("::", 2)[0];
- tableStore.get(tableKey).ifPresent(table -> {
- if (!table.isStreamEnabled()) return;
- this.enableStream(table.getTableName(), table.getTableArn(), table.getStreamViewType(), region, table.getStreamArn());
- });
+ if (tableStore == null) {
+ return;
+ }
+ // No request scope at startup, so keys() would only see the default account's tables.
+ if (tableStore instanceof AccountAwareStorageBackend<TableDefinition> aware) {
+ aware.scanAllAccountsRaw().values().forEach(this::loadPersistedStream);
+ return;
+ }
+ for (String tableKey : tableStore.keys()) {
+ tableStore.get(tableKey).ifPresent(this::loadPersistedStream);
+ }
+ }
+
+ private void loadPersistedStream(TableDefinition table) {
+ if (table.isStreamEnabled()) {
+ enableStream(table.getTableName(), table.getTableArn(), table.getStreamViewType(),
+ AwsArnUtils.parse(table.getTableArn()).region(), table.getStreamArn());
}
}
@@ -76,8 +89,7 @@ public class DynamoDbStreamService {
}
public StreamDescription enableStream(String tableName, String tableArn, String viewType, String region, String streamArnInput) {
- String key = streamKey(region, tableName);
- StreamDescription existing = streams.get(key);
+ StreamDescription existing = streams.get(tableArn);
if (existing != null && "ENABLED".equals(existing.getStreamStatus())) {
// Re-enabling a live stream with a different view type retargets it. The records this
// stream emits are built from the description's view type, so leaving it untouched
@@ -110,37 +122,33 @@ public class DynamoDbStreamService {
sd.setCreationDateTime(now);
sd.setStartingSequenceNumber(String.format("%021d", 1));
- streams.put(key, sd);
+ streams.put(tableArn, sd);
records.put(streamArn, new ConcurrentLinkedDeque<>());
streamRecordCounts.put(streamArn, new AtomicLong());
LOG.infov("Enabled stream for table {0} in region {1}: {2}", tableName, region, streamArn);
return sd;
}
- public void disableStream(String tableName, String region) {
- String key = streamKey(region, tableName);
- StreamDescription sd = streams.get(key);
+ public void disableStream(String tableArn) {
+ StreamDescription sd = streams.get(tableArn);
if (sd != null) {
sd.setStreamStatus("DISABLED");
- LOG.infov("Disabled stream for table {0} in region {1}", tableName, region);
+ LOG.infov("Disabled stream for table {0}", tableArn);
}
}
- public void deleteStream(String tableName, String region) {
- String key = streamKey(region, tableName);
- StreamDescription sd = streams.remove(key);
+ public void deleteStream(String tableArn) {
+ StreamDescription sd = streams.remove(tableArn);
if (sd != null) {
records.remove(sd.getStreamArn());
streamRecordCounts.remove(sd.getStreamArn());
- LOG.infov("Deleted stream for table {0} in region {1}", tableName, region);
+ LOG.infov("Deleted stream for table {0}", tableArn);
}
}
- public void captureEvent(String tableName, String eventName,
- JsonNode oldItem, JsonNode newItem,
+ public void captureEvent(String eventName, JsonNode oldItem, JsonNode newItem,
TableDefinition table, String region) {
- String key = streamKey(region, tableName);
- StreamDescription sd = streams.get(key);
+ StreamDescription sd = streams.get(table.getTableArn());
if (sd == null || !"ENABLED".equals(sd.getStreamStatus())) {
return;
}
@@ -208,13 +216,14 @@ public class DynamoDbStreamService {
};
}
- public List<StreamDescription> listStreams(String tableNameFilter, String region) {
+ public List<StreamDescription> listStreams(String tableNameFilter, String accountId, String region) {
List<StreamDescription> result = new ArrayList<>();
for (StreamDescription sd : streams.values()) {
if (tableNameFilter != null && !tableNameFilter.equals(sd.getTableName())) {
continue;
}
- if (region != null && !sd.getStreamArn().contains(":" + region + ":")) {
+ AwsArnUtils.Arn streamArn = AwsArnUtils.parse(sd.getStreamArn());
+ if (!accountId.equals(streamArn.accountId()) || !region.equals(streamArn.region())) {
continue;
}
result.add(sd);
@@ -222,7 +231,7 @@ public class DynamoDbStreamService {
return result;
}
- public StreamDescription describeStream(String streamArn) {
+ StreamDescription describeStream(String streamArn) {
for (StreamDescription sd : streams.values()) {
if (streamArn.equals(sd.getStreamArn())) {
return sd;
@@ -232,8 +241,8 @@ public class DynamoDbStreamService {
"Stream not found: " + streamArn, 400);
}
- public String getShardIterator(String streamArn, String shardId,
- String iteratorType, String sequenceNumber) {
+ String getShardIterator(String streamArn, String shardId,
+ String iteratorType, String sequenceNumber) {
StreamDescription sd = describeStream(streamArn);
if (!"ENABLED".equals(sd.getStreamStatus()) && !"DISABLED".equals(sd.getStreamStatus())) {
throw new AwsException("ResourceNotFoundException",
@@ -283,21 +292,9 @@ public class DynamoDbStreamService {
return records.size();
}
- public record GetRecordsResult(List<DynamoDbStreamRecord> records, String nextShardIterator) {}
-
- /** Sequence number of the newest record still retained, or null when the stream holds none. */
- public String latestSequenceNumber(String streamArn) {
- ConcurrentLinkedDeque<DynamoDbStreamRecord> deque = records.get(streamArn);
- if (deque == null) {
- return null;
- }
- synchronized (deque) {
- DynamoDbStreamRecord last = deque.peekLast();
- return last == null ? null : last.getSequenceNumber();
- }
- }
+ record GetRecordsResult(List<DynamoDbStreamRecord> records, String nextShardIterator) {}
- public GetRecordsResult getRecords(String shardIterator, Integer limit) {
+ GetRecordsResult getRecords(String shardIterator, Integer limit) {
String[] parts = decodeIterator(shardIterator);
String streamArn = parts[0];
String cursorSequence = parts[1];
@@ -345,6 +342,11 @@ public class DynamoDbStreamService {
return Base64.getEncoder().encodeToString(raw.getBytes());
}
+ /** The stream an iterator reads, rejecting a malformed iterator as GetRecords does. */
+ String streamArnOf(String shardIterator) {
+ return decodeIterator(shardIterator)[0];
+ }
+
private String[] decodeIterator(String iterator) {
try {
String raw = new String(Base64.getDecoder().decode(iterator));
@@ -382,8 +384,4 @@ public class DynamoDbStreamService {
private String zeroSequence() {
return ZERO_SEQUENCE_NUMBER;
}
-
- private String streamKey(String region, String tableName) {
- return region + "::" + tableName;
- }
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbStreamsJsonHandler.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbStreamsJsonHandler.java
index a94dcf9734..bc7c4288f0 100644
--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbStreamsJsonHandler.java
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbStreamsJsonHandler.java
@@ -1,7 +1,9 @@
package io.github.hectorvent.floci.services.dynamodb;
+import io.github.hectorvent.floci.core.common.AwsArnUtils;
import io.github.hectorvent.floci.core.common.AwsErrorResponse;
import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.common.RegionResolver;
import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
import io.github.hectorvent.floci.services.dynamodb.model.StreamDescription;
@@ -28,13 +30,15 @@ public class NativeDynamoDbStreamsJsonHandler {
private final DynamoDbStreamService streamService;
private final DynamoDbService dynamoDbService;
+ private final RegionResolver regionResolver;
private final ObjectMapper objectMapper;
@Inject
public NativeDynamoDbStreamsJsonHandler(DynamoDbStreamService streamService, DynamoDbService dynamoDbService,
- ObjectMapper objectMapper) {
+ RegionResolver regionResolver, ObjectMapper objectMapper) {
this.streamService = streamService;
this.dynamoDbService = dynamoDbService;
+ this.regionResolver = regionResolver;
this.objectMapper = objectMapper;
}
@@ -55,7 +59,8 @@ public class NativeDynamoDbStreamsJsonHandler {
private Response handleListStreams(JsonNode request, String region) {
String tableNameFilter = request.has("TableName") ? request.get("TableName").asText() : null;
- List<StreamDescription> streams = streamService.listStreams(tableNameFilter, region);
+ List<StreamDescription> streams =
+ streamService.listStreams(tableNameFilter, regionResolver.getAccountId(), region);
ObjectNode response = objectMapper.createObjectNode();
ArrayNode streamList = objectMapper.createArrayNode();
@@ -72,7 +77,7 @@ public class NativeDynamoDbStreamsJsonHandler {
private Response handleDescribeStream(JsonNode request, String region) {
String streamArn = request.path("StreamArn").asText();
- StreamDescription sd = streamService.describeStream(streamArn);
+ StreamDescription sd = streamService.describeStream(requireInScope(streamArn, region));
// Fetch key schema from the table
List<KeySchemaElement> keySchema = List.of();
@@ -130,7 +135,8 @@ public class NativeDynamoDbStreamsJsonHandler {
String sequenceNumber = request.has("SequenceNumber")
? request.get("SequenceNumber").asText() : null;
- String iterator = streamService.getShardIterator(streamArn, shardId, iteratorType, sequenceNumber);
+ String iterator = streamService.getShardIterator(requireInScope(streamArn, region), shardId, iteratorType,
+ sequenceNumber);
ObjectNode response = objectMapper.createObjectNode();
response.put("ShardIterator", iterator);
@@ -141,6 +147,7 @@ public class NativeDynamoDbStreamsJsonHandler {
String shardIterator = request.path("ShardIterator").asText();
Integer limit = request.has("Limit") ? request.get("Limit").asInt() : null;
+ requireInScope(streamService.streamArnOf(shardIterator), region);
DynamoDbStreamService.GetRecordsResult result = streamService.getRecords(shardIterator, limit);
ObjectNode response = objectMapper.createObjectNode();
@@ -154,6 +161,15 @@ public class NativeDynamoDbStreamsJsonHandler {
return Response.ok(response).build();
}
+ /** A stream of another account or region is not found, as ListStreams leaves it out. */
+ private String requireInScope(String streamArn, String region) {
+ if (!streamArn.startsWith(
+ AwsArnUtils.Arn.of("dynamodb", region, regionResolver.getAccountId(), "table/").toString())) {
+ throw new AwsException("ResourceNotFoundException", "Stream not found: " + streamArn, 400);
+ }
+ return streamArn;
+ }
+
private ObjectNode recordToNode(DynamoDbStreamRecord record) {
ObjectNode node = objectMapper.createObjectNode();
node.put("eventID", record.getEventId());
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbTableService.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbTableService.java
index 08f5b93340..93ed32aaba 100644
--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbTableService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbTableService.java
@@ -239,7 +239,7 @@ public class NativeDynamoDbTableService {
if (settings.streamEnabled()) {
enableStream(table, settings.streamViewType(), region);
} else {
- streamService.disableStream(table.getTableName(), region);
+ streamService.disableStream(table.getTableArn());
table.setStreamEnabled(false);
}
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbApiStreamReader.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbApiStreamReader.java
new file mode 100644
index 0000000000..f3f7f7aa53
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbApiStreamReader.java
@@ -0,0 +1,103 @@
+package io.github.hectorvent.floci.services.dynamodb.backend;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.JsonNodeFactory;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Api;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Call;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Reply;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * A {@link DynamoDbStreamReader} that speaks the DynamoDB Streams wire API through
+ * {@link DynamoDbOperations}, so it reads whichever engine is selected. It parses AWS JSON only.
+ */
+public final class DynamoDbApiStreamReader implements DynamoDbStreamReader {
+
+ private static final JsonNodeFactory JSON = JsonNodeFactory.instance;
+
+ private final DynamoDbOperations operations;
+ private final CheckpointLifetime checkpointLifetime;
+
+ public DynamoDbApiStreamReader(DynamoDbOperations operations, CheckpointLifetime checkpointLifetime) {
+ this.operations = operations;
+ this.checkpointLifetime = checkpointLifetime;
+ }
+
+ @Override
+ public ShardsPage describeStream(Stream stream, String exclusiveStartShardId, Integer limit) {
+ ObjectNode body = JSON.objectNode().put("StreamArn", stream.arn());
+ if (exclusiveStartShardId != null) {
+ body.put("ExclusiveStartShardId", exclusiveStartShardId);
+ }
+ if (limit != null) {
+ body.put("Limit", limit);
+ }
+ JsonNode description = call(stream, "DescribeStream", body).path("StreamDescription");
+ List<Shard> shards = new ArrayList<>();
+ for (JsonNode shard : description.path("Shards")) {
+ JsonNode range = shard.path("SequenceNumberRange");
+ shards.add(new Shard(text(shard, "ShardId"), text(shard, "ParentShardId"),
+ text(range, "StartingSequenceNumber"), text(range, "EndingSequenceNumber")));
+ }
+ return new ShardsPage(stream, List.copyOf(shards), text(description, "LastEvaluatedShardId"));
+ }
+
+ @Override
+ public Cursor getShardIterator(Stream stream, String shardId, Position position, String sequenceNumber) {
+ ObjectNode body = JSON.objectNode()
+ .put("StreamArn", stream.arn())
+ .put("ShardId", shardId)
+ .put("ShardIteratorType", position.name());
+ if (sequenceNumber != null) {
+ body.put("SequenceNumber", sequenceNumber);
+ }
+ return new Cursor(stream, shardId, text(call(stream, "GetShardIterator", body), "ShardIterator"));
+ }
+
+ @Override
+ public RecordsPage getRecords(Cursor cursor, int limit) {
+ ObjectNode body = JSON.objectNode().put("ShardIterator", cursor.token())
+ .put("Limit", Math.min(limit, MAX_RECORDS_PER_READ));
+ JsonNode reply = call(cursor.stream(), "GetRecords", body);
+ List<Record> records = new ArrayList<>();
+ for (JsonNode awsRecord : reply.path("Records")) {
+ records.add(new Record(text(awsRecord.path("dynamodb"), "SequenceNumber"), awsRecord));
+ }
+ String next = text(reply, "NextShardIterator");
+ return new RecordsPage(List.copyOf(records),
+ next == null ? null : new Cursor(cursor.stream(), cursor.shardId(), next));
+ }
+
+ @Override
+ public CheckpointLifetime checkpointLifetime() {
+ return checkpointLifetime;
+ }
+
+ private JsonNode call(Stream stream, String action, ObjectNode body) {
+ Reply reply;
+ try {
+ reply = operations.execute(new Call(stream.scope(), Api.DYNAMODB_STREAMS, action, body));
+ } catch (RuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new IllegalStateException(e);
+ }
+ JsonNode replyBody = reply.body() == null ? JSON.missingNode() : reply.body();
+ if (reply.status() >= 300) {
+ String type = text(replyBody, "__type");
+ String code = type == null ? "InternalServerError" : type.substring(type.lastIndexOf('#') + 1);
+ String message = text(replyBody, "message");
+ throw new AwsException(code, message != null ? message : text(replyBody, "Message"), reply.status());
+ }
+ return replyBody;
+ }
+
+ private static String text(JsonNode node, String field) {
+ JsonNode value = node.get(field);
+ return value == null || value.isNull() ? null : value.asText();
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbBackendSelector.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbBackendSelector.java
index 48d345f2fd..4c18b42c72 100644
--- a/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbBackendSelector.java
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbBackendSelector.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.dynamodb.backend;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.CheckpointLifetime;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.inject.Produces;
import jakarta.inject.Inject;
@@ -7,7 +8,8 @@ import jakarta.inject.Inject;
/**
* The one immutable DynamoDB backend selection. Native is the only engine today; a configurable
* choice arrives with the alternative engine. {@link NativeDynamoDbBackend} is typed to its concrete
- * class, so these producers are the only unqualified beans for the three seam interfaces.
+ * class and {@link DynamoDbApiStreamReader} is not a bean, so these producers are the only
+ * unqualified beans for the four seam interfaces.
*/
@ApplicationScoped
public class DynamoDbBackendSelector {
@@ -33,4 +35,10 @@ public class DynamoDbBackendSelector {
DynamoDbTableAccess tableAccess() {
return nativeBackend;
}
+
+ /** Native stream history lives only in memory, so committed progress ends with the process. */
+ @Produces
+ DynamoDbStreamReader streamReader() {
+ return new DynamoDbApiStreamReader(nativeBackend, CheckpointLifetime.PROCESS);
+ }
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbStreamReader.java b/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbStreamReader.java
new file mode 100644
index 0000000000..a6396c284d
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbStreamReader.java
@@ -0,0 +1,162 @@
+package io.github.hectorvent.floci.services.dynamodb.backend;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import io.github.hectorvent.floci.core.common.AwsArnUtils;
+import io.github.hectorvent.floci.core.common.AwsException;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * How Java consumers read DynamoDB Streams, whatever engine serves them. {@link Record#awsRecord()}
+ * is the AWS {@code Records[]} element exactly as returned and is never mutated, so a consumer
+ * copies it before changing it. A shard is read completely when {@link #getRecords} returns no
+ * records and no next cursor. An engine behind this reader must return records whenever the shard
+ * holds any after the cursor, so an empty page with a next cursor means the cursor is at the tip of
+ * an open shard. The native engine and DynamoDB Local behave this way; real AWS does not guarantee it.
+ */
+public interface DynamoDbStreamReader {
+
+ /** The most records one GetRecords call returns, as AWS allows. */
+ int MAX_RECORDS_PER_READ = 1000;
+
+ enum Position { TRIM_HORIZON, LATEST, AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER }
+
+ /** How long committed progress stays valid: for this process only, or as long as the stream. */
+ enum CheckpointLifetime { PROCESS, STREAM }
+
+ record Stream(DynamoDbOperations.Scope scope, String arn) {
+
+ /** The stream under its owner's account and region, read off the ARN. */
+ public static Stream of(String streamArn) {
+ AwsArnUtils.Arn parsed = AwsArnUtils.parse(streamArn);
+ return new Stream(new DynamoDbOperations.Scope(parsed.accountId(), parsed.region()), streamArn);
+ }
+ }
+
+ record Shard(String shardId, String parentShardId, String startingSequenceNumber, String endingSequenceNumber) {}
+
+ record ShardsPage(Stream stream, List<Shard> shards, String lastEvaluatedShardId) {}
+
+ record Cursor(Stream stream, String shardId, String token) {}
+
+ record Record(String sequenceNumber, JsonNode awsRecord) {}
+
+ record RecordsPage(List<Record> records, Cursor nextCursor) {
+
+ /** The shard was read to its end. */
+ public boolean closed() {
+ return records.isEmpty() && nextCursor == null;
+ }
+ }
+
+ ShardsPage describeStream(Stream stream, String exclusiveStartShardId, Integer limit);
+
+ Cursor getShardIterator(Stream stream, String shardId, Position position, String sequenceNumber);
+
+ RecordsPage getRecords(Cursor cursor, int limit);
+
+ CheckpointLifetime checkpointLifetime();
+
+ /**
+ * Up to {@code limit} records after {@code committedSequence}, or from the trim horizon when
+ * nothing is committed, filled from as many reads as it needs. It stops early at the tip of an
+ * open shard or the end of a closed one. A trim on a later read returns what was already read.
+ */
+ default RecordsPage readAfter(Stream stream, String shardId, String committedSequence, int limit) {
+ Cursor cursor = committedSequence == null
+ ? getShardIterator(stream, shardId, Position.TRIM_HORIZON, null)
+ : getShardIterator(stream, shardId, Position.AFTER_SEQUENCE_NUMBER, committedSequence);
+ List<Record> records = new ArrayList<>();
+ RecordsPage page;
+ do {
+ try {
+ page = getRecords(cursor, Math.min(limit - records.size(), MAX_RECORDS_PER_READ));
+ } catch (AwsException e) {
+ if (!"TrimmedDataAccessException".equals(e.getErrorCode()) || records.isEmpty()) {
+ throw e;
+ }
+ // The records already read are gone from the stream, so a later read being trimmed
+ // must not drop them; the next read after them surfaces the trim to the caller.
+ return new RecordsPage(List.copyOf(records), cursor);
+ }
+ records.addAll(page.records());
+ cursor = page.nextCursor();
+ } while (records.size() < limit && !page.records().isEmpty() && cursor != null);
+ return new RecordsPage(List.copyOf(records), cursor);
+ }
+
+ /** Every shard of the stream, following {@code LastEvaluatedShardId} to the last page. */
+ default List<Shard> shards(Stream stream) {
+ List<Shard> shards = new ArrayList<>();
+ String exclusiveStartShardId = null;
+ do {
+ ShardsPage page = describeStream(stream, exclusiveStartShardId, null);
+ shards.addAll(page.shards());
+ exclusiveStartShardId = page.lastEvaluatedShardId();
+ } while (exclusiveStartShardId != null);
+ return shards;
+ }
+
+ /**
+ * The shards a consumer may read now, in input order: not finished, and whose parent is absent,
+ * no longer listed because it was trimmed away, or finished.
+ */
+ static List<Shard> readable(List<Shard> shards, Set<String> finishedShardIds) {
+ Set<String> listed = new HashSet<>();
+ for (Shard shard : shards) {
+ listed.add(shard.shardId());
+ }
+ List<Shard> readable = new ArrayList<>();
+ for (Shard shard : shards) {
+ if (finishedShardIds.contains(shard.shardId())) {
+ continue;
+ }
+ String parent = shard.parentShardId();
+ if (parent == null || !listed.contains(parent) || finishedShardIds.contains(parent)) {
+ readable.add(shard);
+ }
+ }
+ return readable;
+ }
+
+ /**
+ * The newest retained sequence number of each shard that holds records, so a LATEST start can
+ * be pinned as committed progress.
+ */
+ default Map<String, String> newestSequenceNumbers(Stream stream) {
+ // ponytail: a tail scan instead of holding a LATEST iterator keeps consumer progress
+ // sequence-only; it costs one pass over the retained records once per consumer start, and
+ // relies on the engine requirement above that an empty page means the tip of the shard.
+ Map<String, String> newest = new LinkedHashMap<>();
+ for (Shard shard : shards(stream)) {
+ Cursor cursor = getShardIterator(stream, shard.shardId(), Position.TRIM_HORIZON, null);
+ int reopenings = 0;
+ while (cursor != null) {
+ RecordsPage page;
+ try {
+ page = getRecords(cursor, MAX_RECORDS_PER_READ);
+ } catch (AwsException e) {
+ if (!"TrimmedDataAccessException".equals(e.getErrorCode()) || reopenings == 3) {
+ throw e;
+ }
+ // The oldest records were trimmed between opening the iterator and reading it,
+ // so the scan restarts at the new trim horizon.
+ reopenings++;
+ cursor = getShardIterator(stream, shard.shardId(), Position.TRIM_HORIZON, null);
+ continue;
+ }
+ if (page.records().isEmpty()) {
+ break;
+ }
+ newest.put(shard.shardId(), page.records().get(page.records().size() - 1).sequenceNumber());
+ cursor = page.nextCursor();
+ }
+ }
+ return newest;
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePoller.java b/src/main/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePoller.java
index 265f8095c5..a9b311df9d 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePoller.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePoller.java
@@ -9,8 +9,7 @@ import io.github.hectorvent.floci.core.common.AwsArnUtils;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.core.common.Resettable;
import io.github.hectorvent.floci.core.common.RequestScopes;
-import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
-import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader;
import io.github.hectorvent.floci.services.lambda.model.EventSourceMapping;
import io.github.hectorvent.floci.services.lambda.model.InvocationType;
import io.github.hectorvent.floci.services.lambda.model.InvokeResult;
@@ -40,6 +39,7 @@ import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.function.LongSupplier;
+import java.util.stream.Collectors;
@ApplicationScoped
public class DynamoDbStreamsEventSourcePoller implements Resettable {
@@ -55,7 +55,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
static final long MAX_RETRY_BACKOFF_MS = 60_000;
private final Vertx vertx;
- private final DynamoDbStreamService streamService;
+ private final DynamoDbStreamReader streamReader;
private final LambdaExecutorService executorService;
private final LambdaTargetResolver targetResolver;
private final EsmStore esmStore;
@@ -73,6 +73,8 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
private final ConcurrentHashMap<String, Long> retryNotBefore = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, Integer> bisectLimits = new ConcurrentHashMap<>();
final Set<String> stopped = ConcurrentHashMap.newKeySet();
+ /** {@code uuid:shardId} of shards a mapping read to their end; re-derived from committed progress. */
+ private final Set<String> finishedShards = ConcurrentHashMap.newKeySet();
private volatile boolean resetting;
private final ExecutorService pollExecutor = Executors.newCachedThreadPool(r -> {
Thread t = new Thread(r, "dynamodb-streams-esm-poller");
@@ -81,7 +83,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
});
@Inject
- public DynamoDbStreamsEventSourcePoller(Vertx vertx, DynamoDbStreamService streamService,
+ public DynamoDbStreamsEventSourcePoller(Vertx vertx, DynamoDbStreamReader streamReader,
LambdaExecutorService executorService,
LambdaTargetResolver targetResolver,
EsmStore esmStore,
@@ -91,11 +93,11 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
SqsService sqsService,
SnsService snsService,
S3Service s3Service) {
- this(vertx, streamService, executorService, targetResolver, esmStore, objectMapper, config,
+ this(vertx, streamReader, executorService, targetResolver, esmStore, objectMapper, config,
filterMatcher, sqsService, snsService, s3Service, System::currentTimeMillis);
}
- DynamoDbStreamsEventSourcePoller(Vertx vertx, DynamoDbStreamService streamService,
+ DynamoDbStreamsEventSourcePoller(Vertx vertx, DynamoDbStreamReader streamReader,
LambdaExecutorService executorService,
LambdaTargetResolver targetResolver,
EsmStore esmStore,
@@ -107,7 +109,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
S3Service s3Service,
LongSupplier clock) {
this.vertx = vertx;
- this.streamService = streamService;
+ this.streamReader = streamReader;
this.executorService = executorService;
this.targetResolver = targetResolver;
this.esmStore = esmStore;
@@ -124,7 +126,9 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
public void startPersistedPollers() {
for (EventSourceMapping esm : esmStore.listAll()) {
if (esm.getEventSourceArn() != null && esm.getEventSourceArn().contains(":dynamodb:")) {
- discardStaleShardCheckpoints(esm);
+ if (streamReader.checkpointLifetime() == DynamoDbStreamReader.CheckpointLifetime.PROCESS) {
+ discardStaleShardCheckpoints(esm);
+ }
if (esm.isEnabled()) {
startPolling(esm);
}
@@ -134,18 +138,21 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
}
/**
- * Discards any shard checkpoints a DynamoDB Streams ESM persisted during a previous run,
- * before its poller is (re)started at startup.
+ * Discards any shard checkpoints a DynamoDB Streams ESM persisted during a previous run, before
+ * its poller is (re)started at startup, when the engine's checkpoints last only for the process
+ * ({@link DynamoDbStreamReader.CheckpointLifetime#PROCESS}).
*
- * <p>A DynamoDB stream is volatile: its record buffer and its {@code AtomicLong} sequence
- * counter live only in memory (see {@link DynamoDbStreamService}) and are never persisted, so a
- * restart recreates the stream empty and its sequence numbers start over from
- * {@code 000000000000000000001}. A {@code shardSequenceNumbers} checkpoint saved during the
- * previous run therefore points <em>past</em> every record in the new stream epoch: resuming
- * from it with an {@code AFTER_SEQUENCE_NUMBER} iterator silently skips every freshly written
- * record — no invoke, no error, no log — until the new sequence numbers climb back above the
- * stale value. Clearing the checkpoint lets the poller resume from {@code TRIM_HORIZON}, which
- * matches the volatility of the stream itself. See issue #2076.
+ * <p>Such a stream's history is volatile: the native engine keeps its records and sequence
+ * counter only in memory, so a restart recreates the stream empty and its sequence numbers start
+ * over from {@code 000000000000000000001}. A sequence number from a previous process is therefore
+ * meaningless, and a {@code shardSequenceNumbers} checkpoint saved then usually points
+ * <em>past</em> every record in the new stream epoch: resuming from it with an
+ * {@code AFTER_SEQUENCE_NUMBER} iterator silently skips every freshly written record (no invoke,
+ * no error, no log) until the new sequence numbers climb back above the stale value. Clearing the
+ * checkpoint lets the poller resume from {@code TRIM_HORIZON}, which matches the volatility of the
+ * stream itself. An engine whose checkpoints last as long as the stream
+ * ({@link DynamoDbStreamReader.CheckpointLifetime#STREAM}) keeps its history across restarts, so
+ * its progress is kept. See issue #2076.
*/
private void discardStaleShardCheckpoints(EventSourceMapping esm) {
if (esm.getShardSequenceNumbers().isEmpty()) {
@@ -158,14 +165,23 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
esmStore.saveForAccount(esm.getAccountId(), esm);
}
- /** Pins a new LATEST mapping after the stream's newest record, so only later writes are delivered. */
+ /**
+ * Pins a new LATEST mapping after each shard's newest record, so only later writes are delivered.
+ * A stream that does not exist yet has nothing to skip.
+ */
public void initializeStartingPosition(EventSourceMapping esm) {
if (!"LATEST".equals(esm.getStartingPosition()) || !esm.getShardSequenceNumbers().isEmpty()) {
return;
}
- String latest = streamService.latestSequenceNumber(esm.getEventSourceArn());
- if (latest != null) {
- esm.getShardSequenceNumbers().put(DynamoDbStreamService.SHARD_ID, latest);
+ try {
+ esm.getShardSequenceNumbers().putAll(
+ streamReader.newestSequenceNumbers(DynamoDbStreamReader.Stream.of(esm.getEventSourceArn())));
+ } catch (AwsException e) {
+ if (!"ResourceNotFoundException".equals(e.getErrorCode())) {
+ throw e;
+ }
+ LOG.debugv("DynamoDB Streams ESM {0}: stream {1} does not exist yet, so LATEST is not pinned",
+ esm.getUuid(), esm.getEventSourceArn());
}
}
@@ -195,6 +211,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
retryCounts.clear();
retryNotBefore.clear();
bisectLimits.clear();
+ finishedShards.clear();
}
public synchronized void startPolling(EventSourceMapping esm) {
@@ -232,6 +249,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
retryCounts.keySet().removeIf(k -> k.startsWith(uuid + ":"));
retryNotBefore.keySet().removeIf(k -> k.startsWith(uuid + ":"));
bisectLimits.keySet().removeIf(k -> k.startsWith(uuid + ":"));
+ finishedShards.removeIf(k -> k.startsWith(uuid + ":"));
}
@@ -241,200 +259,211 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
}
pollExecutor.submit(() -> {
try {
- String streamArn = esm.getEventSourceArn();
- String shardId = DynamoDbStreamService.SHARD_ID;
- String lastSeq = esm.getShardSequenceNumbers().get(shardId);
- String persistedKey = batchStateKey(esm.getUuid(), shardId, lastSeq);
-
LambdaFunction fn = targetResolver.resolveMappingTarget(esm).orElse(null);
if (fn == null) {
LOG.warnv("DynamoDB Streams ESM {0}: function {1} not found, skipping",
esm.getUuid(), esm.getFunctionName());
return;
}
-
- boolean checkpointTrimmed = false;
- DynamoDbStreamService.GetRecordsResult result;
- try {
- String iterator = lastSeq == null
- ? streamService.getShardIterator(streamArn, shardId, "TRIM_HORIZON", null)
- : streamService.getShardIterator(streamArn, shardId, "AFTER_SEQUENCE_NUMBER", lastSeq);
- result = streamService.getRecords(iterator,
- Math.min(bisectLimit(esm, persistedKey), esm.getBatchSize()));
- } catch (AwsException e) {
- if (!TRIMMED_DATA_ACCESS_EXCEPTION.equals(e.getErrorCode())) {
- throw e;
- }
- checkpointTrimmed = true;
- clearBatchState(persistedKey);
- // The checkpoint fell outside the retained window, so the cursor it names can
- // never succeed again. Retrying it wedges the ESM permanently: every later
- // write reaches the stream and none is ever delivered. Resume from the oldest
- // record still held instead, which is what AWS does when a consumer is
- // overtaken by the trim horizon. Records written between the lost checkpoint
- // and that record are gone from the stream and are not delivered.
- LOG.warnv("DynamoDB Streams ESM {0}: checkpoint {1} was trimmed, resuming from the "
- + "trim horizon; records between were dropped from the stream",
- esm.getUuid(), lastSeq);
- String horizon = streamService.getShardIterator(streamArn, shardId, "TRIM_HORIZON", null);
- result = streamService.getRecords(horizon, Math.min(esm.getBatchSize(),
- bisectLimit(esm, batchStateKey(esm.getUuid(), shardId, null))));
+ DynamoDbStreamReader.Stream stream = DynamoDbStreamReader.Stream.of(esm.getEventSourceArn());
+ List<DynamoDbStreamReader.Shard> shards = streamReader.shards(stream);
+ Set<String> finished = shards.stream()
+ .map(DynamoDbStreamReader.Shard::shardId)
+ .filter(shardId -> finishedShards.contains(finishedShardKey(esm.getUuid(), shardId)))
+ .collect(Collectors.toSet());
+ // ponytail: a shard whose read or invoke throws ends the tick for the shards after it; the next
+ // tick starts over from committed progress, so it only delays them.
+ for (DynamoDbStreamReader.Shard shard : DynamoDbStreamReader.readable(shards, finished)) {
+ pollShard(esm, fn, stream, shard.shardId());
}
- List<DynamoDbStreamRecord> records = result.records();
+ } catch (Exception e) {
+ LOG.warnv("DynamoDB Streams ESM {0} poll error: {1}", esm.getUuid(), e.getMessage());
+ } finally {
+ activePolls.remove(esm.getUuid());
+ }
+ });
+ }
- if (records.isEmpty()) {
- return;
- }
- // ponytail: a stop or reset racing this check does not cancel an in-flight invoke; the stop checks in
- // advanceCheckpoint and dispose drop its checkpoint and OnFailure send. Only a stop landing
- // between dispose's check and the send itself still delivers; a lock around the send would close it.
- boolean enabled = esmStore.getForAccount(esm.getAccountId(), esm.getUuid())
- .map(EventSourceMapping::isEnabled)
- .orElse(false);
- if (resetting || !enabled) {
- return;
- }
+ private void pollShard(EventSourceMapping esm, LambdaFunction fn, DynamoDbStreamReader.Stream stream,
+ String shardId) {
+ String lastSeq = esm.getShardSequenceNumbers().get(shardId);
+ String persistedKey = batchStateKey(esm.getUuid(), shardId, lastSeq);
- // Advance to the newest FETCHED record whenever the batch is disposed of, invoked or
- // fully filtered out, so filtered-out records are consumed, not re-read forever. Leave
- // the checkpoint unmoved only when an attempted invoke fails (the window retries).
- String newestFetchedSeq = records.get(records.size() - 1).getSequenceNumber();
-
- List<DynamoDbStreamRecord> matched = records;
- JsonNode filterParams = EsmFilterCriteriaUtils.matcherSourceParameters(objectMapper, esm.getFilterCriteria());
- if (filterParams != null) {
- List<JsonNode> filterNodes = new ArrayList<>(records.size());
- for (DynamoDbStreamRecord rec : records) {
- filterNodes.add(buildDynamoDbRecordNode(rec, esm));
- }
- matched = EsmFilterCriteriaUtils.selectMatched(
- records, filterNodes, filterMatcher.applyFilterCriteria(filterNodes, filterParams));
- }
+ boolean checkpointTrimmed = false;
+ DynamoDbStreamReader.RecordsPage page;
+ try {
+ page = streamReader.readAfter(stream, shardId, lastSeq,
+ Math.min(bisectLimit(esm, persistedKey), esm.getBatchSize()));
+ } catch (AwsException e) {
+ if (!TRIMMED_DATA_ACCESS_EXCEPTION.equals(e.getErrorCode())) {
+ throw e;
+ }
+ checkpointTrimmed = true;
+ clearBatchState(persistedKey);
+ // The checkpoint fell outside the retained window, so the cursor it names can
+ // never succeed again. Retrying it wedges the ESM permanently: every later
+ // write reaches the stream and none is ever delivered. Resume from the oldest
+ // record still held instead, which is what AWS does when a consumer is
+ // overtaken by the trim horizon. Records written between the lost checkpoint
+ // and that record are gone from the stream and are not delivered.
+ LOG.warnv("DynamoDB Streams ESM {0}: checkpoint {1} was trimmed, resuming from the "
+ + "trim horizon; records between were dropped from the stream",
+ esm.getUuid(), lastSeq);
+ page = streamReader.readAfter(stream, shardId, null, Math.min(esm.getBatchSize(),
+ bisectLimit(esm, batchStateKey(esm.getUuid(), shardId, null))));
+ }
+ List<DynamoDbStreamReader.Record> records = page.records();
- String batchKey = batchStateKey(esm.getUuid(), shardId, checkpointTrimmed ? null : lastSeq);
- if (matched.isEmpty()) {
- clearBatchState(batchKey);
- advanceCheckpoint(esm, shardId, newestFetchedSeq);
- return;
- }
+ if (records.isEmpty()) {
+ if (page.closed()) {
+ finishedShards.add(finishedShardKey(esm.getUuid(), shardId));
+ }
+ return;
+ }
+ // ponytail: a stop or reset racing this check does not cancel an in-flight invoke; the stop checks in
+ // advanceCheckpoint and dispose drop its checkpoint and OnFailure send. Only a stop landing
+ // between dispose's check and the send itself still delivers; a lock around the send would close it.
+ boolean enabled = esmStore.getForAccount(esm.getAccountId(), esm.getUuid())
+ .map(EventSourceMapping::isEnabled)
+ .orElse(false);
+ if (resetting || !enabled) {
+ return;
+ }
- long now = clock.getAsLong();
- if (now < retryNotBefore.getOrDefault(batchKey, 0L)) {
- return;
- }
+ // Advance to the newest FETCHED record whenever the batch is disposed of, invoked or
+ // fully filtered out, so filtered-out records are consumed, not re-read forever. Leave
+ // the checkpoint unmoved only when an attempted invoke fails (the window retries).
+ String newestFetchedSeq = records.get(records.size() - 1).sequenceNumber();
+
+ List<DynamoDbStreamReader.Record> matched = records;
+ JsonNode filterParams = EsmFilterCriteriaUtils.matcherSourceParameters(objectMapper, esm.getFilterCriteria());
+ if (filterParams != null) {
+ List<JsonNode> filterNodes = new ArrayList<>(records.size());
+ for (DynamoDbStreamReader.Record rec : records) {
+ filterNodes.add(buildDynamoDbRecordNode(rec, esm));
+ }
+ matched = EsmFilterCriteriaUtils.selectMatched(
+ records, filterNodes, filterMatcher.applyFilterCriteria(filterNodes, filterParams));
+ }
- LOG.infov("DynamoDB Streams ESM {0}: delivering {1} of {2} record(s) to {3}",
- esm.getUuid(), matched.size(), records.size(), esm.getFunctionName());
-
- String eventJson = buildDynamoDbEvent(matched, esm);
- InvokeResult invokeResult;
- try {
- invokeResult = executorService.invoke(fn, eventJson.getBytes(), InvocationType.RequestResponse);
- } catch (AwsException e) {
- if ("TooManyRequestsException".equals(e.getErrorCode())) {
- LOG.infov("DynamoDB Streams ESM {0}: function {1} throttled, shard iterator not advanced",
- esm.getUuid(), fn.getFunctionName());
- return;
- }
- throw e;
- }
- if (!exists(esm)) {
- LOG.debugv("DynamoDB Streams ESM {0} was deleted during the invocation, dropping its result",
- esm.getUuid());
- return;
- }
+ String batchKey = batchStateKey(esm.getUuid(), shardId, checkpointTrimmed ? null : lastSeq);
+ if (matched.isEmpty()) {
+ clearBatchState(batchKey);
+ advanceCheckpoint(esm, shardId, newestFetchedSeq);
+ return;
+ }
- CheckpointOutcome outcome = invokeResult.getFunctionError() == null
- ? successfulInvocationCheckpoint(esm, invokeResult, lastSeq, records, matched)
- : new CheckpointOutcome(null, 0);
-
- if (outcome.lowestFailedIndex() == records.size()) {
- clearBatchState(batchKey);
- advanceCheckpoint(esm, shardId, outcome.checkpoint());
- } else if (outcome.lowestFailedIndex() > 0) {
- clearBatchState(batchKey);
-
- String nextCheckpoint = outcome.checkpoint();
- String nextBatchKey = batchStateKey(esm.getUuid(), shardId, nextCheckpoint);
- Integer maxRetries = esm.getMaximumRetryAttempts();
- int currentRetries = retryCounts.merge(nextBatchKey, 1, Integer::sum);
- Set<String> deliveredSeqs = new HashSet<>();
- for (DynamoDbStreamRecord rec : matched) {
- deliveredSeqs.add(rec.getSequenceNumber());
- }
- List<DynamoDbStreamRecord> failedRecords = new ArrayList<>();
- for (int i = outcome.lowestFailedIndex(); i < records.size(); i++) {
- DynamoDbStreamRecord rec = records.get(i);
- if (deliveredSeqs.contains(rec.getSequenceNumber())) {
- failedRecords.add(rec);
- }
- }
- if (failedRecords.isEmpty()) {
- failedRecords = records.subList(outcome.lowestFailedIndex(), records.size());
- }
+ long now = clock.getAsLong();
+ if (now < retryNotBefore.getOrDefault(batchKey, 0L)) {
+ return;
+ }
- if (hasExceededMaximumRecordAge(esm, failedRecords, now)) {
- LOG.warnv("DynamoDB Streams ESM {0}: maximum record age exceeded for batch ending at {1}",
- esm.getUuid(), newestFetchedSeq);
- dispose(esm, shardId, nextBatchKey, new PendingFailure(failedRecords, invokeResult,
- currentRetries, "MaximumRecordAgeExceeded", newestFetchedSeq));
- } else if (maxRetries != null && maxRetries >= 0 && currentRetries > maxRetries) {
- LOG.warnv("DynamoDB Streams ESM {0}: maximum retry attempts ({1}) exhausted for batch ending at {2}",
- esm.getUuid(), maxRetries, newestFetchedSeq);
- dispose(esm, shardId, nextBatchKey, new PendingFailure(failedRecords, invokeResult,
- currentRetries, "RetryAttemptsExhausted", newestFetchedSeq));
- } else {
- retryNotBefore.put(nextBatchKey, now + retryBackoffMs(currentRetries));
- LOG.warnv("DynamoDB Streams ESM {0}: Lambda returned error [batchItemFailures], retry {1}, records will be retried",
- esm.getUuid(), currentRetries);
- advanceCheckpoint(esm, shardId, nextCheckpoint);
- }
- } else {
- Integer maxRetries = esm.getMaximumRetryAttempts();
- int currentRetries = retryCounts.getOrDefault(batchKey, 0) + 1;
- if (hasExceededMaximumRecordAge(esm, matched, now)) {
- LOG.warnv("DynamoDB Streams ESM {0}: maximum record age exceeded for batch ending at {1}",
- esm.getUuid(), newestFetchedSeq);
- clearBatchState(batchKey);
- dispose(esm, shardId, persistedKey, new PendingFailure(matched, invokeResult,
- currentRetries, "MaximumRecordAgeExceeded", newestFetchedSeq));
- } else if (invokeResult.getFunctionError() != null
- && Boolean.TRUE.equals(esm.getBisectBatchOnFunctionError()) && matched.size() > 1) {
- // ponytail: the halved limit is keyed by checkpoint, so once a good half advances the
- // rest is refetched at full batch size and split again. One window converges on the
- // poison record in O(log n) invocations, a busy stream in O(log^2 n) worst case;
- // carrying the right half's end sequence forward would match AWS exactly.
- int limit = (records.size() + 1) / 2;
- bisectLimits.put(batchKey, limit);
- LOG.infov("DynamoDB Streams ESM {0}: function error on {1} record(s), bisecting to {2}",
- esm.getUuid(), records.size(), limit);
- } else if (maxRetries != null && maxRetries >= 0 && currentRetries > maxRetries) {
- LOG.warnv("DynamoDB Streams ESM {0}: maximum retry attempts ({1}) exhausted for batch ending at {2}",
- esm.getUuid(), maxRetries, newestFetchedSeq);
- clearBatchState(batchKey);
- dispose(esm, shardId, persistedKey, new PendingFailure(matched, invokeResult,
- currentRetries, "RetryAttemptsExhausted", newestFetchedSeq));
- } else {
- retryCounts.put(batchKey, currentRetries);
- retryNotBefore.put(batchKey, now + retryBackoffMs(currentRetries));
- String error = invokeResult.getFunctionError() != null
- ? invokeResult.getFunctionError()
- : "batchItemFailures";
- LOG.warnv("DynamoDB Streams ESM {0}: Lambda returned error [{1}], retry {2}, records will be retried",
- esm.getUuid(), error, currentRetries);
- }
+ LOG.infov("DynamoDB Streams ESM {0}: delivering {1} of {2} record(s) to {3}",
+ esm.getUuid(), matched.size(), records.size(), esm.getFunctionName());
+
+ String eventJson = buildDynamoDbEvent(matched, esm);
+ InvokeResult invokeResult;
+ try {
+ invokeResult = executorService.invoke(fn, eventJson.getBytes(), InvocationType.RequestResponse);
+ } catch (AwsException e) {
+ if ("TooManyRequestsException".equals(e.getErrorCode())) {
+ LOG.infov("DynamoDB Streams ESM {0}: function {1} throttled, shard iterator not advanced",
+ esm.getUuid(), fn.getFunctionName());
+ return;
+ }
+ throw e;
+ }
+ if (!exists(esm)) {
+ LOG.debugv("DynamoDB Streams ESM {0} was deleted during the invocation, dropping its result",
+ esm.getUuid());
+ return;
+ }
+
+ CheckpointOutcome outcome = invokeResult.getFunctionError() == null
+ ? successfulInvocationCheckpoint(esm, invokeResult, lastSeq, records, matched)
+ : new CheckpointOutcome(null, 0);
+
+ if (outcome.lowestFailedIndex() == records.size()) {
+ clearBatchState(batchKey);
+ advanceCheckpoint(esm, shardId, outcome.checkpoint());
+ } else if (outcome.lowestFailedIndex() > 0) {
+ clearBatchState(batchKey);
+
+ String nextCheckpoint = outcome.checkpoint();
+ String nextBatchKey = batchStateKey(esm.getUuid(), shardId, nextCheckpoint);
+ Integer maxRetries = esm.getMaximumRetryAttempts();
+ int currentRetries = retryCounts.merge(nextBatchKey, 1, Integer::sum);
+ Set<String> deliveredSeqs = new HashSet<>();
+ for (DynamoDbStreamReader.Record rec : matched) {
+ deliveredSeqs.add(rec.sequenceNumber());
+ }
+ List<DynamoDbStreamReader.Record> failedRecords = new ArrayList<>();
+ for (int i = outcome.lowestFailedIndex(); i < records.size(); i++) {
+ DynamoDbStreamReader.Record rec = records.get(i);
+ if (deliveredSeqs.contains(rec.sequenceNumber())) {
+ failedRecords.add(rec);
}
- } catch (Exception e) {
- LOG.warnv("DynamoDB Streams ESM {0} poll error: {1}", esm.getUuid(), e.getMessage());
- } finally {
- activePolls.remove(esm.getUuid());
}
- });
+ if (failedRecords.isEmpty()) {
+ failedRecords = records.subList(outcome.lowestFailedIndex(), records.size());
+ }
+
+ if (hasExceededMaximumRecordAge(esm, failedRecords, now)) {
+ LOG.warnv("DynamoDB Streams ESM {0}: maximum record age exceeded for batch ending at {1}",
+ esm.getUuid(), newestFetchedSeq);
+ dispose(esm, shardId, nextBatchKey, new PendingFailure(failedRecords, invokeResult,
+ currentRetries, "MaximumRecordAgeExceeded", newestFetchedSeq));
+ } else if (maxRetries != null && maxRetries >= 0 && currentRetries > maxRetries) {
+ LOG.warnv("DynamoDB Streams ESM {0}: maximum retry attempts ({1}) exhausted for batch ending at {2}",
+ esm.getUuid(), maxRetries, newestFetchedSeq);
+ dispose(esm, shardId, nextBatchKey, new PendingFailure(failedRecords, invokeResult,
+ currentRetries, "RetryAttemptsExhausted", newestFetchedSeq));
+ } else {
+ retryNotBefore.put(nextBatchKey, now + retryBackoffMs(currentRetries));
+ LOG.warnv("DynamoDB Streams ESM {0}: Lambda returned error [batchItemFailures], retry {1}, records will be retried",
+ esm.getUuid(), currentRetries);
+ advanceCheckpoint(esm, shardId, nextCheckpoint);
+ }
+ } else {
+ Integer maxRetries = esm.getMaximumRetryAttempts();
+ int currentRetries = retryCounts.getOrDefault(batchKey, 0) + 1;
+ if (hasExceededMaximumRecordAge(esm, matched, now)) {
+ LOG.warnv("DynamoDB Streams ESM {0}: maximum record age exceeded for batch ending at {1}",
+ esm.getUuid(), newestFetchedSeq);
+ clearBatchState(batchKey);
+ dispose(esm, shardId, persistedKey, new PendingFailure(matched, invokeResult,
+ currentRetries, "MaximumRecordAgeExceeded", newestFetchedSeq));
+ } else if (invokeResult.getFunctionError() != null
+ && Boolean.TRUE.equals(esm.getBisectBatchOnFunctionError()) && matched.size() > 1) {
+ // ponytail: the halved limit is keyed by checkpoint, so once a good half advances the
+ // rest is refetched at full batch size and split again. One window converges on the
+ // poison record in O(log n) invocations, a busy stream in O(log^2 n) worst case;
+ // carrying the right half's end sequence forward would match AWS exactly.
+ int limit = (records.size() + 1) / 2;
+ bisectLimits.put(batchKey, limit);
+ LOG.infov("DynamoDB Streams ESM {0}: function error on {1} record(s), bisecting to {2}",
+ esm.getUuid(), records.size(), limit);
+ } else if (maxRetries != null && maxRetries >= 0 && currentRetries > maxRetries) {
+ LOG.warnv("DynamoDB Streams ESM {0}: maximum retry attempts ({1}) exhausted for batch ending at {2}",
+ esm.getUuid(), maxRetries, newestFetchedSeq);
+ clearBatchState(batchKey);
+ dispose(esm, shardId, persistedKey, new PendingFailure(matched, invokeResult,
+ currentRetries, "RetryAttemptsExhausted", newestFetchedSeq));
+ } else {
+ retryCounts.put(batchKey, currentRetries);
+ retryNotBefore.put(batchKey, now + retryBackoffMs(currentRetries));
+ String error = invokeResult.getFunctionError() != null
+ ? invokeResult.getFunctionError()
+ : "batchItemFailures";
+ LOG.warnv("DynamoDB Streams ESM {0}: Lambda returned error [{1}], retry {2}, records will be retried",
+ esm.getUuid(), error, currentRetries);
+ }
+ }
}
private record CheckpointOutcome(String checkpoint, int lowestFailedIndex) {}
- private record PendingFailure(List<DynamoDbStreamRecord> records, InvokeResult invokeResult, int invokeCount,
+ private record PendingFailure(List<DynamoDbStreamReader.Record> records, InvokeResult invokeResult, int invokeCount,
String condition, String advanceTo) {}
/** Sends a discarded batch to its OnFailure destination and checkpoints past it. */
@@ -461,6 +490,10 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
: Integer.MAX_VALUE;
}
+ private static String finishedShardKey(String uuid, String shardId) {
+ return uuid + ":" + shardId;
+ }
+
private static String batchStateKey(String uuid, String shardId, String sequence) {
return uuid + ":" + shardId + ":" + (sequence == null ? "TRIM_HORIZON" : sequence);
}
@@ -472,9 +505,9 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
*/
private CheckpointOutcome successfulInvocationCheckpoint(EventSourceMapping esm, InvokeResult invokeResult,
String previousCheckpoint,
- List<DynamoDbStreamRecord> fetched,
- List<DynamoDbStreamRecord> delivered) {
- String newestFetchedSeq = fetched.get(fetched.size() - 1).getSequenceNumber();
+ List<DynamoDbStreamReader.Record> fetched,
+ List<DynamoDbStreamReader.Record> delivered) {
+ String newestFetchedSeq = fetched.get(fetched.size() - 1).sequenceNumber();
byte[] payload = invokeResult.getPayload();
if (!esm.isReportBatchItemFailures() || payload == null || payload.length == 0) {
return new CheckpointOutcome(newestFetchedSeq, fetched.size());
@@ -493,11 +526,11 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
Map<String, Integer> fetchedIndexes = new HashMap<>();
for (int i = 0; i < fetched.size(); i++) {
- fetchedIndexes.put(fetched.get(i).getSequenceNumber(), i);
+ fetchedIndexes.put(fetched.get(i).sequenceNumber(), i);
}
Set<String> deliveredSequences = new HashSet<>();
- for (DynamoDbStreamRecord record : delivered) {
- deliveredSequences.add(record.getSequenceNumber());
+ for (DynamoDbStreamReader.Record record : delivered) {
+ deliveredSequences.add(record.sequenceNumber());
}
int lowestFailedIndex = fetched.size();
@@ -523,7 +556,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
}
String checkpoint = lowestFailedIndex == 0
? previousCheckpoint
- : fetched.get(lowestFailedIndex - 1).getSequenceNumber();
+ : fetched.get(lowestFailedIndex - 1).sequenceNumber();
return new CheckpointOutcome(checkpoint, lowestFailedIndex);
} catch (Exception e) {
return new CheckpointOutcome(
@@ -538,7 +571,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
return previousCheckpoint;
}
- private boolean hasExceededMaximumRecordAge(EventSourceMapping esm, List<DynamoDbStreamRecord> records,
+ private boolean hasExceededMaximumRecordAge(EventSourceMapping esm, List<DynamoDbStreamReader.Record> records,
long now) {
Integer configuredAge = esm.getMaximumRecordAgeInSeconds();
if (configuredAge == null || configuredAge <= 0) {
@@ -546,7 +579,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
}
long oldestRecordSeconds = records.stream()
- .mapToLong(DynamoDbStreamRecord::getApproximateCreationDateTime)
+ .mapToLong(DynamoDbStreamsEventSourcePoller::approximateCreationSeconds)
.filter(seconds -> seconds > 0)
.min()
.orElse(0);
@@ -556,13 +589,17 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
return now - oldestRecordSeconds * 1_000 >= configuredAge * 1_000L;
}
+ private static long approximateCreationSeconds(DynamoDbStreamReader.Record rec) {
+ return rec.awsRecord().path("dynamodb").path("ApproximateCreationDateTime").asLong();
+ }
+
long retryBackoffMs(int retries) {
int doublings = Math.min(retries, 16);
return Math.min(pollIntervalMs * (1L << doublings), MAX_RETRY_BACKOFF_MS);
}
private void sendToOnFailureDestination(EventSourceMapping esm, String shardId,
- List<DynamoDbStreamRecord> records,
+ List<DynamoDbStreamReader.Record> records,
InvokeResult invokeResult,
int invokeCount,
String condition) {
@@ -610,7 +647,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
}
private String buildS3OnFailurePayload(EventSourceMapping esm, String shardId,
- List<DynamoDbStreamRecord> records,
+ List<DynamoDbStreamReader.Record> records,
InvokeResult invokeResult,
int invokeCount,
String condition) {
@@ -631,7 +668,7 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
}
private String buildOnFailurePayload(EventSourceMapping esm, String shardId,
- List<DynamoDbStreamRecord> records,
+ List<DynamoDbStreamReader.Record> records,
InvokeResult invokeResult,
int invokeCount,
String condition) {
@@ -655,14 +692,14 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
ObjectNode batchInfo = root.putObject("DDBStreamBatchInfo");
batchInfo.put("shardId", shardId);
- String startSeq = records.isEmpty() ? "" : records.get(0).getSequenceNumber();
- String endSeq = records.isEmpty() ? "" : records.get(records.size() - 1).getSequenceNumber();
+ String startSeq = records.isEmpty() ? "" : records.get(0).sequenceNumber();
+ String endSeq = records.isEmpty() ? "" : records.get(records.size() - 1).sequenceNumber();
batchInfo.put("startSequenceNumber", startSeq);
batchInfo.put("endSequenceNumber", endSeq);
if (!records.isEmpty()) {
- long firstArrival = records.get(0).getApproximateCreationDateTime();
- long lastArrival = records.get(records.size() - 1).getApproximateCreationDateTime();
+ long firstArrival = approximateCreationSeconds(records.get(0));
+ long lastArrival = approximateCreationSeconds(records.get(records.size() - 1));
batchInfo.put("approximateArrivalOfFirstRecord",
DateTimeFormatter.ISO_INSTANT.format(Instant.ofEpochSecond(firstArrival)));
batchInfo.put("approximateArrivalOfLastRecord",
@@ -678,11 +715,11 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
}
}
- private String buildDynamoDbEvent(List<DynamoDbStreamRecord> records, EventSourceMapping esm) {
+ private String buildDynamoDbEvent(List<DynamoDbStreamReader.Record> records, EventSourceMapping esm) {
try {
ObjectNode root = objectMapper.createObjectNode();
ArrayNode array = root.putArray("Records");
- for (DynamoDbStreamRecord rec : records) {
+ for (DynamoDbStreamReader.Record rec : records) {
array.add(buildDynamoDbRecordNode(rec, esm));
}
return objectMapper.writeValueAsString(root);
@@ -692,33 +729,19 @@ public class DynamoDbStreamsEventSourcePoller implements Resettable {
}
/**
- * Builds the single-record node: top-level {@code eventName}/metadata plus the {@code dynamodb} map
- * with AttributeValue-wrapped images. This is both the delivery record shape and the exact structure a
+ * Builds the single-record node: a copy of the AWS stream record, whose top-level {@code eventName}
+ * and metadata and {@code dynamodb} map with AttributeValue-wrapped images Lambda delivers as they
+ * are, plus {@code eventSourceARN}. {@code ApproximateCreationDateTime} is written as a double, as
+ * Floci has always delivered it. This is both the delivery record shape and the exact structure a
* DynamoDB filter pattern matches against, so it serves the matcher unchanged. (Numeric operators
* naturally never match here because AttributeValue numbers are JSON strings, AWS parity for free.)
*/
- private ObjectNode buildDynamoDbRecordNode(DynamoDbStreamRecord rec, EventSourceMapping esm) {
- ObjectNode item = objectMapper.createObjectNode();
- item.put("eventID", rec.getEventId());
- item.put("eventVersion", rec.getEventVersion());
- item.put("awsRegion", rec.getAwsRegion());
- item.put("eventName", rec.getEventName());
+ private ObjectNode buildDynamoDbRecordNode(DynamoDbStreamReader.Record rec, EventSourceMapping esm) {
+ ObjectNode item = rec.awsRecord().deepCopy();
item.put("eventSourceARN", esm.getEventSourceArn());
- item.put("eventSource", rec.getEventSource());
-
- ObjectNode dynamodb = item.putObject("dynamodb");
- dynamodb.put("StreamViewType", rec.getStreamViewType());
- dynamodb.put("SequenceNumber", rec.getSequenceNumber());
- dynamodb.put("SizeBytes", 100);
- dynamodb.put("ApproximateCreationDateTime", (double) rec.getApproximateCreationDateTime());
- if (rec.getKeys() != null) {
- dynamodb.set("Keys", rec.getKeys());
- }
- if (rec.getNewImage() != null) {
- dynamodb.set("NewImage", rec.getNewImage());
- }
- if (rec.getOldImage() != null) {
- dynamodb.set("OldImage", rec.getOldImage());
+ JsonNode created = item.path("dynamodb").path("ApproximateCreationDateTime");
+ if (created.isNumber()) {
+ ((ObjectNode) item.get("dynamodb")).put("ApproximateCreationDateTime", created.asDouble());
}
return item;
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/pipes/PipesPoller.java b/src/main/java/io/github/hectorvent/floci/services/pipes/PipesPoller.java
index 42a8021e2d..8d7b479735 100644
--- a/src/main/java/io/github/hectorvent/floci/services/pipes/PipesPoller.java
+++ b/src/main/java/io/github/hectorvent/floci/services/pipes/PipesPoller.java
@@ -8,7 +8,7 @@ import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.AwsArnUtils;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.core.common.Resettable;
-import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader;
import io.github.hectorvent.floci.services.kinesis.KinesisService;
import io.github.hectorvent.floci.services.pipes.model.DesiredState;
import io.github.hectorvent.floci.services.pipes.model.Pipe;
@@ -43,7 +43,7 @@ public class PipesPoller implements Resettable {
private final Vertx vertx;
private final SqsService sqsService;
private final KinesisService kinesisService;
- private final DynamoDbStreamService dynamoDbStreamService;
+ private final DynamoDbStreamReader streamReader;
private final PipesKafkaConsumerManager kafkaConsumerManager;
private final PipesTargetInvoker targetInvoker;
private final PipesFilterMatcher filterMatcher;
@@ -52,7 +52,8 @@ public class PipesPoller implements Resettable {
private final ConcurrentHashMap<String, Long> timerIds = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, Boolean> activePolls = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, String> kinesisIterators = new ConcurrentHashMap<>();
- private final ConcurrentHashMap<String, String> dynamoDbIterators = new ConcurrentHashMap<>();
+ /** Per pipe, its DynamoDB Stream progress, which a poll only starts while the pipe polls. */
+ private final ConcurrentHashMap<String, DynamoDbProgress> dynamoDbProgress = new ConcurrentHashMap<>();
private final ExecutorService pollExecutor = Executors.newCachedThreadPool(r -> {
Thread t = new Thread(r, "pipes-poller");
t.setDaemon(true);
@@ -63,7 +64,7 @@ public class PipesPoller implements Resettable {
public PipesPoller(Vertx vertx,
SqsService sqsService,
KinesisService kinesisService,
- DynamoDbStreamService dynamoDbStreamService,
+ DynamoDbStreamReader streamReader,
PipesKafkaConsumerManager kafkaConsumerManager,
PipesTargetInvoker targetInvoker,
PipesFilterMatcher filterMatcher,
@@ -72,7 +73,7 @@ public class PipesPoller implements Resettable {
this.vertx = vertx;
this.sqsService = sqsService;
this.kinesisService = kinesisService;
- this.dynamoDbStreamService = dynamoDbStreamService;
+ this.streamReader = streamReader;
this.kafkaConsumerManager = kafkaConsumerManager;
this.targetInvoker = targetInvoker;
this.filterMatcher = filterMatcher;
@@ -93,7 +94,7 @@ public class PipesPoller implements Resettable {
timerIds.clear();
activePolls.clear();
kinesisIterators.clear();
- dynamoDbIterators.clear();
+ dynamoDbProgress.clear();
}
public void startPolling(Pipe pipe) {
@@ -112,7 +113,6 @@ public class PipesPoller implements Resettable {
if (timerId != null) {
vertx.cancelTimer(timerId);
kinesisIterators.remove(pipeKey);
- dynamoDbIterators.remove(pipeKey);
kafkaConsumerManager.close(pipe);
LOG.infov("Pipe {0}: stopped polling", pipe.getName());
}
@@ -122,6 +122,11 @@ public class PipesPoller implements Resettable {
return timerIds.containsKey(pipeKey(pipe));
}
+ /** Drops the pipe's DynamoDB Stream progress, which a stop keeps so a restart resumes from it. */
+ public void forget(Pipe pipe) {
+ dynamoDbProgress.remove(pipeKey(pipe));
+ }
+
private void pollAndInvoke(Pipe pipe) {
String pipeKey = pipeKey(pipe);
if (activePolls.putIfAbsent(pipeKey, Boolean.TRUE) != null) {
@@ -298,46 +303,77 @@ public class PipesPoller implements Resettable {
}
}
- private void pollDynamoDbStreams(Pipe pipe, String region) {
- String pipeKey = pipeKey(pipe);
- String streamArn = pipe.getSource();
- int batchSize = getBatchSize(pipe, "DynamoDBStreamParameters");
- String iterator = dynamoDbIterators.get(pipeKey);
- if (iterator == null) {
- iterator = initDynamoDbIterator(streamArn);
- if (iterator == null) {
- return;
- }
+ /**
+ * Reads every readable shard once. A LATEST pipe is pinned after each shard's newest record on
+ * its first poll, so only later writes are delivered.
+ */
+ void pollDynamoDbStreams(Pipe pipe, String region) {
+ DynamoDbStreamReader.Stream stream = DynamoDbStreamReader.Stream.of(pipe.getSource());
+ JsonNode sp = pipe.getSourceParameters();
+ boolean latest = sp != null
+ && "LATEST".equals(sp.path("DynamoDBStreamParameters").path("StartingPosition").asText());
+ // A delete stops polling before it forgets, so a poll it raced finds no timer for its pipe's lifetime and
+ // cannot start progress again, nor touch a pipe recreated under the same ARN.
+ DynamoDbProgress progress = dynamoDbProgress.computeIfAbsent(pipeKey(pipe), key -> timerIds.containsKey(key)
+ ? new DynamoDbProgress(latest
+ ? new ConcurrentHashMap<>(streamReader.newestSequenceNumbers(stream))
+ : new ConcurrentHashMap<>(), ConcurrentHashMap.newKeySet())
+ : null);
+ if (progress == null) {
+ return;
+ }
+ // ponytail: a shard whose read throws ends the tick for the shards after it; the next tick
+ // starts over from committed progress, so it only delays them.
+ for (DynamoDbStreamReader.Shard shard
+ : DynamoDbStreamReader.readable(streamReader.shards(stream), progress.finished())) {
+ pollDynamoDbShard(pipe, region, stream, shard.shardId(), progress.committed(), progress.finished());
}
+ }
+
+ /** The newest sequence delivered or sent to the DLQ by shard, and the shards read to their end. */
+ private record DynamoDbProgress(Map<String, String> committed, Set<String> finished) {}
+
+ /**
+ * Reads one batch after the shard's committed sequence and commits it only once the batch is
+ * filtered out, delivered, or sent to the DLQ, so an undisposed batch is read again next poll.
+ */
+ private void pollDynamoDbShard(Pipe pipe, String region, DynamoDbStreamReader.Stream stream, String shardId,
+ Map<String, String> committed, Set<String> finished) {
+ String sequence = committed.get(shardId);
+ DynamoDbStreamReader.RecordsPage page;
try {
- var result = dynamoDbStreamService.getRecords(iterator, batchSize);
- String nextIterator = result.nextShardIterator();
- if (nextIterator != null) {
- dynamoDbIterators.put(pipeKey, nextIterator);
- }
- var records = result.records();
- if (records == null || records.isEmpty()) {
- return;
- }
- LOG.infov("Pipe {0}: received {1} DynamoDB Stream record(s)", pipe.getName(), records.size());
- List<ObjectNode> recordNodes = buildDynamoDbRecordNodes(records, pipe, region);
- List<JsonNode> filtered = filterMatcher.applyFilterCriteria(
- new ArrayList<>(recordNodes), pipe.getSourceParameters());
- if (filtered.isEmpty()) {
- return;
- }
- int failed = deliverRecords(pipe, filtered, region);
- if (failed > 0) {
- LOG.warnv("Pipe {0}: {1} DynamoDB Stream record(s) dropped — delivery and DLQ both failed",
- pipe.getName(), failed);
- }
+ page = streamReader.readAfter(stream, shardId, sequence, getBatchSize(pipe, "DynamoDBStreamParameters"));
} catch (AwsException e) {
- if ("ExpiredIteratorException".equals(e.getErrorCode()) ||
- "TrimmedDataAccessException".equals(e.getErrorCode())) {
- dynamoDbIterators.remove(pipeKey);
+ if (!"TrimmedDataAccessException".equals(e.getErrorCode())) {
+ throw e;
}
- throw e;
+ LOG.warnv("Pipe {0}: DynamoDB Stream checkpoint {1} on {2} was trimmed, restarting at the trim "
+ + "horizon; records between were dropped from the stream", pipe.getName(), sequence, shardId);
+ committed.remove(shardId);
+ return;
+ }
+ List<DynamoDbStreamReader.Record> records = page.records();
+ if (records.isEmpty()) {
+ if (page.closed()) {
+ finished.add(shardId);
+ }
+ return;
}
+ LOG.infov("Pipe {0}: received {1} DynamoDB Stream record(s)", pipe.getName(), records.size());
+ List<JsonNode> recordNodes = new ArrayList<>(records.size());
+ for (DynamoDbStreamReader.Record record : records) {
+ ObjectNode node = record.awsRecord().deepCopy();
+ node.put("eventSourceARN", pipe.getSource());
+ recordNodes.add(node);
+ }
+ List<JsonNode> filtered = filterMatcher.applyFilterCriteria(recordNodes, pipe.getSourceParameters());
+ // ponytail: a partially delivered non-Lambda batch is redelivered whole (at-least-once).
+ if (!filtered.isEmpty() && deliverRecords(pipe, filtered, region) > 0) {
+ LOG.warnv("Pipe {0}: DynamoDB Stream batch on {1} was neither delivered nor sent to a DLQ, "
+ + "retrying it next poll", pipe.getName(), shardId);
+ return;
+ }
+ committed.put(shardId, records.get(records.size() - 1).sequenceNumber());
}
void pollKafka(Pipe pipe, String region) {
@@ -511,20 +547,6 @@ public class PipesPoller implements Resettable {
return failed;
}
- /**
- * Opens an iterator on the stream's first shard, under the same single-shard limit as
- * {@link #initKinesisIterator}.
- */
- private String initDynamoDbIterator(String streamArn) {
- try {
- return dynamoDbStreamService.getShardIterator(
- streamArn, DynamoDbStreamService.SHARD_ID, "TRIM_HORIZON", null);
- } catch (Exception e) {
- LOG.warnv("Failed to get DynamoDB stream iterator for {0}: {1}", streamArn, e.getMessage());
- return null;
- }
- }
-
// ──────────────────────────── Invocation & DLQ ────────────────────────────
private int deliverRecords(Pipe pipe, List<JsonNode> records, String region) {
@@ -736,18 +758,6 @@ public class PipesPoller implements Resettable {
return nodes;
}
- private List<ObjectNode> buildDynamoDbRecordNodes(List<?> records, Pipe pipe, String region) {
- List<ObjectNode> nodes = new ArrayList<>();
- for (Object record : records) {
- ObjectNode node = objectMapper.valueToTree(record);
- node.put("eventSource", "aws:dynamodb");
- node.put("eventSourceARN", pipe.getSource());
- node.put("awsRegion", region);
- nodes.add(node);
- }
- return nodes;
- }
-
private List<ObjectNode> buildKafkaRecordNodes(List<KafkaRecordDto> records, Pipe pipe) {
List<ObjectNode> nodes = new ArrayList<>();
String eventSource = pipe.getSource().contains(":kafka:") ? "aws:kafka" : "SelfManagedKafka";
@@ -890,8 +900,9 @@ public class PipesPoller implements Resettable {
return partitions;
}
+ /** Keys polling state by pipe lifetime, so a pipe recreated under the same ARN starts afresh. */
private static String pipeKey(Pipe pipe) {
- return pipe.getArn();
+ return pipe.getArn() + "@" + pipe.getCreationTime();
}
private static String extractRegionFromArn(String arn) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/pipes/PipesService.java b/src/main/java/io/github/hectorvent/floci/services/pipes/PipesService.java
index b4ba5106ec..b2f2bb83da 100644
--- a/src/main/java/io/github/hectorvent/floci/services/pipes/PipesService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/pipes/PipesService.java
@@ -219,6 +219,7 @@ public class PipesService implements TagHandler, ResourceProvider {
.orElseThrow(() -> new AwsException("NotFoundException",
"Pipe " + name + " does not exist.", 404));
poller.stopPolling(pipe);
+ poller.forget(pipe);
storage.delete(key);
LOG.infov("Deleted pipe: {0}", name);
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumer.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumer.java
index 187f363c23..dd214e8f97 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumer.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumer.java
@@ -9,10 +9,9 @@ import io.github.hectorvent.floci.core.common.AwsArnUtils;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.core.common.RequestScopes;
import io.github.hectorvent.floci.services.dynamodb.DynamoDbFacade;
-import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbItemAccess.ScanPage;
import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Scope;
-import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader;
import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
import io.github.hectorvent.floci.services.dynamodb.model.TableDefinition;
import io.github.hectorvent.floci.services.redshift.model.Integration;
@@ -29,6 +28,8 @@ import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.List;
+import java.util.Map;
+import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
@@ -40,7 +41,7 @@ public class RedshiftDynamoDbZeroEtlConsumer {
private static final int BATCH_SIZE = 100;
private final Vertx vertx;
- private final DynamoDbStreamService streamService;
+ private final DynamoDbStreamReader streamReader;
private final DynamoDbFacade dynamoDb;
private final RedshiftService redshiftService;
private final RedshiftZeroEtlWriter writer;
@@ -48,6 +49,8 @@ public class RedshiftDynamoDbZeroEtlConsumer {
private final long pollIntervalMs;
private final ConcurrentHashMap<String, Long> timerIds = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, Boolean> activePolls = new ConcurrentHashMap<>();
+ /** Per integration, the DynamoDB Stream shards read to their end. */
+ private final ConcurrentHashMap<String, Set<String>> finishedShards = new ConcurrentHashMap<>();
private final ExecutorService pollExecutor = Executors.newCachedThreadPool(runnable -> {
Thread thread = new Thread(runnable, "redshift-zero-etl");
thread.setDaemon(true);
@@ -56,32 +59,32 @@ public class RedshiftDynamoDbZeroEtlConsumer {
@Inject
public RedshiftDynamoDbZeroEtlConsumer(Vertx vertx,
- DynamoDbStreamService streamService,
+ DynamoDbStreamReader streamReader,
DynamoDbFacade dynamoDb,
RedshiftService redshiftService,
RedshiftZeroEtlWriter writer,
ObjectMapper objectMapper,
EmulatorConfig config) {
- this(vertx, streamService, dynamoDb, redshiftService, writer, objectMapper,
+ this(vertx, streamReader, dynamoDb, redshiftService, writer, objectMapper,
config.services().redshift().pollIntervalMs());
}
- RedshiftDynamoDbZeroEtlConsumer(DynamoDbStreamService streamService,
+ RedshiftDynamoDbZeroEtlConsumer(DynamoDbStreamReader streamReader,
DynamoDbFacade dynamoDb,
RedshiftService redshiftService,
RedshiftZeroEtlWriter writer) {
- this(null, streamService, dynamoDb, redshiftService, writer, new ObjectMapper(), 1000);
+ this(null, streamReader, dynamoDb, redshiftService, writer, new ObjectMapper(), 1000);
}
private RedshiftDynamoDbZeroEtlConsumer(Vertx vertx,
- DynamoDbStreamService streamService,
+ DynamoDbStreamReader streamReader,
DynamoDbFacade dynamoDb,
RedshiftService redshiftService,
RedshiftZeroEtlWriter writer,
ObjectMapper objectMapper,
long pollIntervalMs) {
this.vertx = vertx;
- this.streamService = streamService;
+ this.streamReader = streamReader;
this.dynamoDb = dynamoDb;
this.redshiftService = redshiftService;
this.writer = writer;
@@ -93,8 +96,16 @@ public class RedshiftDynamoDbZeroEtlConsumer {
startPersistedIntegrations();
}
+ /**
+ * Starts the persisted integrations, first discarding their stream progress when it lasts only
+ * for the process: native stream history is volatile, so a sequence number saved by a previous
+ * run would skip the records of the new stream epoch.
+ */
public void startPersistedIntegrations() {
for (Integration integration : redshiftService.listDynamoDbZeroEtlIntegrations()) {
+ if (streamReader.checkpointLifetime() == DynamoDbStreamReader.CheckpointLifetime.PROCESS) {
+ integration.getShardSequenceNumbers().clear();
+ }
if (integration.isPollingEnabled()) {
startPolling(integration);
}
@@ -115,6 +126,7 @@ public class RedshiftDynamoDbZeroEtlConsumer {
vertx.cancelTimer(timerId);
}
activePolls.remove(integrationArn);
+ finishedShards.remove(integrationArn);
}
void pollOnce(Integration integration) {
@@ -124,19 +136,45 @@ public class RedshiftDynamoDbZeroEtlConsumer {
pollBackfillPage(integration);
return;
}
- String iteratorType = integration.getCheckpointSequenceNumber() == null
- ? "TRIM_HORIZON" : "AFTER_SEQUENCE_NUMBER";
- String iterator = streamService.getShardIterator(integration.getSourceStreamArn(),
- DynamoDbStreamService.SHARD_ID, iteratorType, integration.getCheckpointSequenceNumber());
- DynamoDbStreamService.GetRecordsResult result = streamService.getRecords(iterator, BATCH_SIZE);
- if (result.records().isEmpty()) {
+ DynamoDbStreamReader.Stream stream = DynamoDbStreamReader.Stream.of(integration.getSourceStreamArn());
+ Set<String> finished = finishedShards.computeIfAbsent(integration.getIntegrationArn(),
+ ignored -> ConcurrentHashMap.newKeySet());
+ // ponytail: a shard whose read or write throws ends the tick for the shards after it; the next
+ // tick starts over from committed progress, so it only delays them.
+ for (DynamoDbStreamReader.Shard shard : DynamoDbStreamReader.readable(streamReader.shards(stream), finished)) {
+ pollShard(integration, stream, shard.shardId(), finished);
+ }
+ }
+
+ /**
+ * Writes one batch after the shard's committed sequence and commits it only once written, so a
+ * failed write is read and written again next poll.
+ */
+ private void pollShard(Integration integration, DynamoDbStreamReader.Stream stream, String shardId,
+ Set<String> finished) {
+ Map<String, String> committed = integration.getShardSequenceNumbers();
+ DynamoDbStreamReader.RecordsPage page;
+ try {
+ page = streamReader.readAfter(stream, shardId, committed.get(shardId), BATCH_SIZE);
+ } catch (AwsException e) {
+ if ("TrimmedDataAccessException".equals(e.getErrorCode())) {
+ committed.remove(shardId);
+ }
+ throw e;
+ }
+ List<DynamoDbStreamReader.Record> records = page.records();
+ if (records.isEmpty()) {
+ if (page.closed()) {
+ finished.add(shardId);
+ }
return;
}
- String sequence = writer.writeBatch(integration.getAccountId(), integration.getTargetClusterIdentifier(),
- integration.getLandingTableName(), result.records());
+ writer.writeBatch(integration.getAccountId(), integration.getTargetClusterIdentifier(),
+ integration.getLandingTableName(),
+ records.stream().map(DynamoDbStreamReader.Record::awsRecord).toList());
+ committed.put(shardId, records.get(records.size() - 1).sequenceNumber());
redshiftService.updateIntegrationRuntime(integration.getAccountId(), integration.getIntegrationArn(),
- sequence, true, null);
- integration.setCheckpointSequenceNumber(sequence);
+ committed, true, null);
}
private void pollBackfillPage(Integration integration) {
@@ -150,7 +188,7 @@ public class RedshiftDynamoDbZeroEtlConsumer {
ScanPage result = dynamoDb.items().scan(scope, tableName, null, null, null, null,
BATCH_SIZE, exclusiveStartKey);
if (!result.items().isEmpty()) {
- List<DynamoDbStreamRecord> records = result.items().stream()
+ List<JsonNode> records = result.items().stream()
.map(item -> toBackfillRecord(integration, item, table))
.toList();
writer.writeBatch(integration.getAccountId(), integration.getTargetClusterIdentifier(),
@@ -166,7 +204,8 @@ public class RedshiftDynamoDbZeroEtlConsumer {
});
}
- private DynamoDbStreamRecord toBackfillRecord(Integration integration, JsonNode item, TableDefinition table) {
+ /** The item as an AWS stream {@code INSERT} record, with an event id stable across scans. */
+ private JsonNode toBackfillRecord(Integration integration, JsonNode item, TableDefinition table) {
ObjectNode keys = objectMapper.createObjectNode();
for (KeySchemaElement keySchemaElement : table.getKeySchema()) {
String attributeName = keySchemaElement.getAttributeName();
@@ -174,12 +213,12 @@ public class RedshiftDynamoDbZeroEtlConsumer {
keys.set(attributeName, item.get(attributeName));
}
}
- DynamoDbStreamRecord record = new DynamoDbStreamRecord();
- record.setEventId("backfill#" + integration.getIntegrationArn() + "#" + sha256Hex(keys.toString()));
- record.setEventName("INSERT");
- record.setSequenceNumber("backfill");
- record.setKeys(keys);
- record.setNewImage(item);
+ ObjectNode record = objectMapper.createObjectNode()
+ .put("eventID", "backfill#" + integration.getIntegrationArn() + "#" + sha256Hex(keys.toString()))
+ .put("eventName", "INSERT");
+ ObjectNode dynamodb = record.putObject("dynamodb").put("SequenceNumber", "backfill");
+ dynamodb.set("Keys", keys);
+ dynamodb.set("NewImage", item);
return record;
}
@@ -227,6 +266,7 @@ public class RedshiftDynamoDbZeroEtlConsumer {
stopPolling(integrationArn);
}
activePolls.clear();
+ finishedShards.clear();
}
@PreDestroy
@@ -254,13 +294,9 @@ public class RedshiftDynamoDbZeroEtlConsumer {
pollOnce(integration);
} catch (Exception e) {
LOG.warnv(e, "Zero-ETL polling failed for integration {0}", integration.getIntegrationArn());
- if (e instanceof AwsException awsException
- && "TrimmedDataAccessException".equals(awsException.getErrorCode())) {
- integration.setCheckpointSequenceNumber(null);
- }
try {
redshiftService.updateIntegrationRuntime(integration.getAccountId(), integration.getIntegrationArn(),
- integration.getCheckpointSequenceNumber(), false, e.getMessage());
+ integration.getShardSequenceNumbers(), false, e.getMessage());
} catch (Exception updateError) {
LOG.warnv(updateError, "Could not persist zero-ETL failure for integration {0}",
integration.getIntegrationArn());
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
index 8a6a9f49ec..fca7dcf4bb 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/RedshiftService.java
@@ -11,7 +11,7 @@ import io.github.hectorvent.floci.core.common.RegionResolver;
import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.StorageFactory;
-import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader;
import io.github.hectorvent.floci.services.rds.proxy.PasswordValidator;
import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerHandle;
import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerManager;
@@ -83,7 +83,7 @@ public class RedshiftService {
private final RedshiftCredentialBroker credentialBroker;
private final SecretsManagerService secretsManagerService;
private final ObjectMapper objectMapper;
- private final DynamoDbStreamService streamService;
+ private final DynamoDbStreamReader streamReader;
// Proxy ports currently handed out, so allocateProxyPort never double-assigns within this JVM.
private final Set<Integer> usedPorts = ConcurrentHashMap.newKeySet();
@@ -93,7 +93,7 @@ public class RedshiftService {
RedshiftProxyManager proxyManager, DockerHostResolver dockerHostResolver,
RedshiftCredentialBroker credentialBroker,
SecretsManagerService secretsManagerService, ObjectMapper objectMapper,
- DynamoDbStreamService streamService) {
+ DynamoDbStreamReader streamReader) {
this.clusters = storageFactory.create("redshift", "redshift-clusters.json", new TypeReference<Map<String, Cluster>>() {});
this.snapshots = storageFactory.create("redshift", "redshift-snapshots.json", new TypeReference<Map<String, Snapshot>>() {});
this.parameterGroups = storageFactory.create("redshift", "redshift-parameter-groups.json", new TypeReference<Map<String, ClusterParameterGroup>>() {});
@@ -108,7 +108,7 @@ public class RedshiftService {
this.credentialBroker = credentialBroker;
this.secretsManagerService = secretsManagerService;
this.objectMapper = objectMapper;
- this.streamService = streamService;
+ this.streamReader = streamReader;
}
RedshiftService(StorageFactory storageFactory, RedshiftContainerManager containerManager,
@@ -335,10 +335,10 @@ public class RedshiftService {
throw new AwsException("InvalidParameterValue",
"SourceArn must identify a DynamoDB stream.", 400);
}
- if (streamService == null) {
+ if (streamReader == null) {
throw new AwsException("InternalFailure", "DynamoDB stream service is unavailable.", 500);
}
- streamService.describeStream(sourceArn);
+ streamReader.describeStream(DynamoDbStreamReader.Stream.of(sourceArn), null, null);
AwsArnUtils.Arn target = parseZeroEtlArn(targetArn, "Redshift cluster");
if (!"redshift".equals(target.service()) || !target.resource().startsWith("cluster:")) {
throw new AwsException("InvalidParameterValue",
@@ -378,7 +378,6 @@ public class RedshiftService {
integration.setSourceStreamArn(sourceArn);
integration.setTargetClusterIdentifier(clusterIdentifier);
integration.setLandingTableName("floci_zetl_" + integrationId.replace('-', '_'));
- integration.setCheckpointSequenceNumber(null);
integration.setRetryCount(0);
integration.setLastError(null);
integration.setPollingEnabled(true);
@@ -467,7 +466,7 @@ public class RedshiftService {
}
public synchronized void updateIntegrationRuntime(String accountId, String integrationArn,
- String checkpointSequenceNumber,
+ Map<String, String> shardSequenceNumbers,
boolean successful, String error) {
for (String key : integrations.keysForAccount(accountId)) {
Optional<Integration> stored = integrations.getForAccount(accountId, key);
@@ -476,7 +475,7 @@ public class RedshiftService {
}
Integration integration = stored.get();
if (successful) {
- integration.setCheckpointSequenceNumber(checkpointSequenceNumber);
+ integration.setShardSequenceNumbers(shardSequenceNumbers);
integration.setRetryCount(0);
integration.setLastError(null);
integration.setStatus("active");
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Integration.java b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Integration.java
index 4b26c99003..152eed6459 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshift/model/Integration.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshift/model/Integration.java
@@ -1,11 +1,13 @@
package io.github.hectorvent.floci.services.redshift.model;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import io.quarkus.runtime.annotations.RegisterForReflection;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
/**
* A zero-ETL integration between a source (for example a DynamoDB table) and a Redshift target.
@@ -14,6 +16,7 @@ import java.util.Map;
* ({@code active}), and {@code errors} is present but empty on a healthy integration.
*/
@RegisterForReflection
+@JsonIgnoreProperties(ignoreUnknown = true)
public class Integration {
private String integrationArn;
private String accountId;
@@ -23,7 +26,11 @@ public class Integration {
private String sourceStreamArn;
private String targetClusterIdentifier;
private String landingTableName;
- private String checkpointSequenceNumber;
+ /**
+ * The newest DynamoDB Stream sequence written to the landing table, by shard. Concurrent because
+ * a store flush can serialize it while the consumer commits.
+ */
+ private Map<String, String> shardSequenceNumbers = new ConcurrentHashMap<>();
private String backfillLastEvaluatedKey;
private boolean backfillCompleted;
private int retryCount;
@@ -55,8 +62,11 @@ public class Integration {
public void setTargetClusterIdentifier(String targetClusterIdentifier) { this.targetClusterIdentifier = targetClusterIdentifier; }
public String getLandingTableName() { return landingTableName; }
public void setLandingTableName(String landingTableName) { this.landingTableName = landingTableName; }
- public String getCheckpointSequenceNumber() { return checkpointSequenceNumber; }
- public void setCheckpointSequenceNumber(String checkpointSequenceNumber) { this.checkpointSequenceNumber = checkpointSequenceNumber; }
+ public Map<String, String> getShardSequenceNumbers() { return shardSequenceNumbers; }
+ public void setShardSequenceNumbers(Map<String, String> shardSequenceNumbers) {
+ this.shardSequenceNumbers = shardSequenceNumbers == null
+ ? new ConcurrentHashMap<>() : new ConcurrentHashMap<>(shardSequenceNumbers);
+ }
public String getBackfillLastEvaluatedKey() { return backfillLastEvaluatedKey; }
public void setBackfillLastEvaluatedKey(String backfillLastEvaluatedKey) { this.backfillLastEvaluatedKey = backfillLastEvaluatedKey; }
public boolean isBackfillCompleted() { return backfillCompleted; }
diff --git a/src/main/java/io/github/hectorvent/floci/services/redshiftdata/RedshiftZeroEtlWriter.java b/src/main/java/io/github/hectorvent/floci/services/redshiftdata/RedshiftZeroEtlWriter.java
index 8d7cb06912..c05b1c3b78 100644
--- a/src/main/java/io/github/hectorvent/floci/services/redshiftdata/RedshiftZeroEtlWriter.java
+++ b/src/main/java/io/github/hectorvent/floci/services/redshiftdata/RedshiftZeroEtlWriter.java
@@ -4,7 +4,6 @@ import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.github.hectorvent.floci.core.common.AwsException;
-import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
import io.github.hectorvent.floci.services.redshift.RedshiftService;
import io.github.hectorvent.floci.services.redshift.model.Cluster;
import jakarta.enterprise.context.ApplicationScoped;
@@ -61,40 +60,44 @@ public class RedshiftZeroEtlWriter {
}
}
- public String writeBatch(String accountId, String clusterIdentifier, String tableName,
- List<DynamoDbStreamRecord> records) {
+ /** Writes AWS DynamoDB Streams {@code Records[]} elements, skipping event ids already landed. */
+ public void writeBatch(String accountId, String clusterIdentifier, String tableName, List<JsonNode> records) {
if (records == null || records.isEmpty()) {
- return null;
+ return;
}
String safeTableName = validateTableName(tableName);
- String newestSequence = null;
try (Connection connection = connectionFactory.open(target(accountId, clusterIdentifier));
PreparedStatement statement = connection.prepareStatement(INSERT_RECORD.formatted(safeTableName))) {
- for (DynamoDbStreamRecord record : records) {
+ for (JsonNode record : records) {
bind(statement, record);
statement.executeUpdate();
- newestSequence = record.getSequenceNumber();
}
} catch (SQLException e) {
throw new AwsException("InternalFailure", "Could not write Redshift zero-ETL records.", 500);
}
- return newestSequence;
}
- private void bind(PreparedStatement statement, DynamoDbStreamRecord record) throws SQLException {
- statement.setString(1, record.getEventId());
- statement.setString(2, record.getEventName());
- setNullableString(statement, 3, record.getEventSource());
- setNullableString(statement, 4, record.getAwsRegion());
- statement.setString(5, record.getSequenceNumber());
- if (record.getApproximateCreationDateTime() > 0) {
- statement.setObject(6, Instant.ofEpochSecond(record.getApproximateCreationDateTime()));
+ private void bind(PreparedStatement statement, JsonNode record) throws SQLException {
+ JsonNode dynamodb = record.path("dynamodb");
+ statement.setString(1, text(record, "eventID"));
+ statement.setString(2, text(record, "eventName"));
+ setNullableString(statement, 3, text(record, "eventSource"));
+ setNullableString(statement, 4, text(record, "awsRegion"));
+ statement.setString(5, text(dynamodb, "SequenceNumber"));
+ long approximateCreationDateTime = dynamodb.path("ApproximateCreationDateTime").asLong();
+ if (approximateCreationDateTime > 0) {
+ statement.setObject(6, Instant.ofEpochSecond(approximateCreationDateTime));
} else {
statement.setNull(6, Types.TIMESTAMP_WITH_TIMEZONE);
}
- setJson(statement, 7, record.getKeys());
- setJson(statement, 8, record.getOldImage());
- setJson(statement, 9, record.getNewImage());
+ setJson(statement, 7, dynamodb.get("Keys"));
+ setJson(statement, 8, dynamodb.get("OldImage"));
+ setJson(statement, 9, dynamodb.get("NewImage"));
+ }
+
+ private static String text(JsonNode node, String field) {
+ JsonNode value = node.get(field);
+ return value == null || value.isNull() ? null : value.asText();
}
private void setNullableString(PreparedStatement statement, int index, String value) throws SQLException {
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamServiceTest.java
index 9cdd712398..fc9792bd86 100644
--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamServiceTest.java
@@ -1,6 +1,7 @@
package io.github.hectorvent.floci.services.dynamodb;
import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.InMemoryStorage;
import io.github.hectorvent.floci.core.storage.StorageBackend;
import io.github.hectorvent.floci.services.dynamodb.model.AttributeDefinition;
@@ -20,6 +21,9 @@ class DynamoDbStreamServiceTest {
private static final String TABLE_ARN =
"arn:aws:dynamodb:us-east-1:000000000000:table/ViewTypeTable";
+ private static final String SHARED_TABLE = "Shared";
+ private static final String ACCOUNT_A = "111111111111";
+ private static final String ACCOUNT_B = "222222222222";
private DynamoDbStreamService service;
private ObjectMapper mapper;
@@ -74,7 +78,7 @@ class DynamoDbStreamServiceTest {
service.enableStream("ViewTypeTable", TABLE_ARN, "KEYS_ONLY", "us-east-1");
JsonNode item = mapper.readTree("{\"userId\":{\"S\":\"u1\"},\"name\":{\"S\":\"a\"}}");
- service.captureEvent("ViewTypeTable", "INSERT", null, item, table, "us-east-1");
+ service.captureEvent("INSERT", null, item, table, "us-east-1");
String iterator = service.getShardIterator(
sd.getStreamArn(), "shardId-000000000000", "TRIM_HORIZON", null);
@@ -88,25 +92,43 @@ class DynamoDbStreamServiceTest {
@Test
void loadsStreamOnStartup() {
- var streams = service.listStreams(null, null);
+ List<StreamDescription> streams = service.listStreams(null, "000000000000", "us-west-2");
assertEquals(1, streams.size());
StreamDescription stream = streams.get(0);
assertEquals("TestTable", stream.getTableName());
assertEquals("ENABLED", stream.getStreamStatus());
}
+ @Test
+ void loadsPersistedStreamsOfEveryAccountOnStartup() {
+ AccountAwareStorageBackend<TableDefinition> tables = AccountAwareStorageBackend.inMemory(ACCOUNT_A);
+ TableDefinition tableA = sharedNameTable(ACCOUNT_A);
+ TableDefinition tableB = sharedNameTable(ACCOUNT_B);
+ tableA.setStreamArn(tableA.getTableArn() + "/stream/2026-01-01T00:00:00.000");
+ tableB.setStreamArn(tableB.getTableArn() + "/stream/2026-01-02T00:00:00.000");
+ tables.putForAccount(ACCOUNT_A, "us-east-1::" + SHARED_TABLE, tableA);
+ tables.putForAccount(ACCOUNT_B, "us-east-1::" + SHARED_TABLE, tableB);
+
+ DynamoDbStreamService restarted = new DynamoDbStreamService(mapper, tables);
+
+ assertEquals(List.of(tableA.getStreamArn()),
+ streamArns(restarted.listStreams(null, ACCOUNT_A, "us-east-1")));
+ assertEquals(List.of(tableB.getStreamArn()),
+ streamArns(restarted.listStreams(null, ACCOUNT_B, "us-east-1")));
+ }
+
@Test
void rejectsAnIteratorWhenItsSequenceNumberHasBeenTrimmed() throws Exception {
var table = createTestTableWithStream();
- service.enableStream(table.getTableName(), TABLE_ARN, "NEW_IMAGE", "us-east-1");
+ service.enableStream(table.getTableName(), table.getTableArn(), "NEW_IMAGE", "us-east-1");
JsonNode item = mapper.readTree("{\"userId\":{\"S\":\"u1\"}}");
- service.captureEvent(table.getTableName(), "INSERT", null, item, table, "us-east-1");
+ service.captureEvent("INSERT", null, item, table, "us-east-1");
String iterator = service.getShardIterator(
table.getStreamArn(), DynamoDbStreamService.SHARD_ID, "AT_SEQUENCE_NUMBER", "000000000000000000001");
for (int i = 0; i < DynamoDbStreamService.MAX_RECORDS; i++) {
- service.captureEvent(table.getTableName(), "INSERT", null, item, table, "us-east-1");
+ service.captureEvent("INSERT", null, item, table, "us-east-1");
}
AwsException exception = assertThrows(AwsException.class,
@@ -123,13 +145,13 @@ class DynamoDbStreamServiceTest {
"us-east-1", "000000000000");
table.setStreamEnabled(true);
StreamDescription stream = service.enableStream(
- table.getTableName(), TABLE_ARN, "NEW_IMAGE", "us-east-1");
+ table.getTableName(), table.getTableArn(), "NEW_IMAGE", "us-east-1");
JsonNode item = mapper.readTree("{\"userId\":{\"S\":\"u1\"}}");
String iterator = service.getShardIterator(
stream.getStreamArn(), DynamoDbStreamService.SHARD_ID, "TRIM_HORIZON", null);
for (int i = 0; i <= DynamoDbStreamService.MAX_RECORDS; i++) {
- service.captureEvent(table.getTableName(), "INSERT", null, item, table, "us-east-1");
+ service.captureEvent("INSERT", null, item, table, "us-east-1");
}
AwsException exception = assertThrows(AwsException.class,
@@ -146,13 +168,13 @@ class DynamoDbStreamServiceTest {
"us-east-1", "000000000000");
table.setStreamEnabled(true);
StreamDescription stream = service.enableStream(
- table.getTableName(), TABLE_ARN, "NEW_IMAGE", "us-east-1");
+ table.getTableName(), table.getTableArn(), "NEW_IMAGE", "us-east-1");
JsonNode item = mapper.readTree("{\"userId\":{\"S\":\"u1\"}}");
String iterator = service.getShardIterator(
stream.getStreamArn(), DynamoDbStreamService.SHARD_ID, "LATEST", null);
for (int i = 0; i <= DynamoDbStreamService.MAX_RECORDS; i++) {
- service.captureEvent(table.getTableName(), "INSERT", null, item, table, "us-east-1");
+ service.captureEvent("INSERT", null, item, table, "us-east-1");
}
AwsException exception = assertThrows(AwsException.class,
@@ -169,13 +191,13 @@ class DynamoDbStreamServiceTest {
"us-east-1", "000000000000");
table.setStreamEnabled(true);
StreamDescription stream = service.enableStream(
- table.getTableName(), TABLE_ARN, "NEW_IMAGE", "us-east-1");
+ table.getTableName(), table.getTableArn(), "NEW_IMAGE", "us-east-1");
JsonNode item = mapper.readTree("{\"userId\":{\"S\":\"u1\"}}");
String iterator = service.getShardIterator(
stream.getStreamArn(), DynamoDbStreamService.SHARD_ID, "TRIM_HORIZON", null);
for (int i = 0; i < DynamoDbStreamService.MAX_RECORDS; i++) {
- service.captureEvent(table.getTableName(), "INSERT", null, item, table, "us-east-1");
+ service.captureEvent("INSERT", null, item, table, "us-east-1");
}
var result = service.getRecords(iterator, 1);
@@ -187,11 +209,11 @@ class DynamoDbStreamServiceTest {
@Test
void appliesInclusiveAndExclusiveSequencePositions() throws Exception {
var table = createTestTableWithStream();
- service.enableStream(table.getTableName(), TABLE_ARN, "NEW_IMAGE", "us-east-1");
+ service.enableStream(table.getTableName(), table.getTableArn(), "NEW_IMAGE", "us-east-1");
JsonNode item = mapper.readTree("{\"userId\":{\"S\":\"u1\"}}");
- service.captureEvent(table.getTableName(), "INSERT", null, item, table, "us-east-1");
- service.captureEvent(table.getTableName(), "MODIFY", item, item, table, "us-east-1");
+ service.captureEvent("INSERT", null, item, table, "us-east-1");
+ service.captureEvent("MODIFY", item, item, table, "us-east-1");
String at = service.getShardIterator(
table.getStreamArn(), DynamoDbStreamService.SHARD_ID,
@@ -211,17 +233,17 @@ class DynamoDbStreamServiceTest {
@Test
void resumesAtTheRetainedSequenceAfterTheHeadIsTrimmed() throws Exception {
var table = createTestTableWithStream();
- service.enableStream(table.getTableName(), TABLE_ARN, "NEW_IMAGE", "us-east-1");
+ service.enableStream(table.getTableName(), table.getTableArn(), "NEW_IMAGE", "us-east-1");
JsonNode item = mapper.readTree("{\"userId\":{\"S\":\"u1\"}}");
- service.captureEvent(table.getTableName(), "EVENT_1", null, item, table, "us-east-1");
- service.captureEvent(table.getTableName(), "EVENT_2", null, item, table, "us-east-1");
+ service.captureEvent("EVENT_1", null, item, table, "us-east-1");
+ service.captureEvent("EVENT_2", null, item, table, "us-east-1");
String iterator = service.getShardIterator(
table.getStreamArn(), DynamoDbStreamService.SHARD_ID,
"AT_SEQUENCE_NUMBER", "000000000000000000002");
for (int i = 3; i <= DynamoDbStreamService.MAX_RECORDS + 1; i++) {
- service.captureEvent(table.getTableName(), "EVENT_" + i, null, item, table, "us-east-1");
+ service.captureEvent("EVENT_" + i, null, item, table, "us-east-1");
}
var records = service.getRecords(iterator, 1).records();
@@ -233,11 +255,11 @@ class DynamoDbStreamServiceTest {
@Test
void advancesToTheNextSequenceWithoutDuplicatingRecords() throws Exception {
var table = createTestTableWithStream();
- service.enableStream(table.getTableName(), TABLE_ARN, "NEW_IMAGE", "us-east-1");
+ service.enableStream(table.getTableName(), table.getTableArn(), "NEW_IMAGE", "us-east-1");
JsonNode item = mapper.readTree("{\"userId\":{\"S\":\"u1\"}}");
- service.captureEvent(table.getTableName(), "EVENT_1", null, item, table, "us-east-1");
- service.captureEvent(table.getTableName(), "EVENT_2", null, item, table, "us-east-1");
+ service.captureEvent("EVENT_1", null, item, table, "us-east-1");
+ service.captureEvent("EVENT_2", null, item, table, "us-east-1");
String iterator = service.getShardIterator(
table.getStreamArn(), DynamoDbStreamService.SHARD_ID,
"TRIM_HORIZON", null);
@@ -249,24 +271,63 @@ class DynamoDbStreamServiceTest {
assertEquals("000000000000000000002", secondPage.records().get(0).getSequenceNumber());
}
- @Test
- void latestSequenceNumberIsNullForAnUnknownOrEmptyStream() {
- TableDefinition table = createTestTableWithStream();
+ private TableDefinition sharedNameTable(String accountId) {
+ TableDefinition table = new TableDefinition(SHARED_TABLE,
+ List.of(new KeySchemaElement("userId", "HASH")),
+ List.of(new AttributeDefinition("userId", "S")),
+ "us-east-1", accountId);
+ table.setStreamEnabled(true);
+ return table;
+ }
- assertNull(service.latestSequenceNumber(TABLE_ARN + "/stream/unknown"));
- assertNull(service.latestSequenceNumber(table.getStreamArn()));
+ private List<String> userIdsIn(String streamArn) {
+ String iterator = service.getShardIterator(streamArn, DynamoDbStreamService.SHARD_ID, "TRIM_HORIZON", null);
+ return service.getRecords(iterator, 100).records().stream()
+ .map(record -> record.getKeys().path("userId").path("S").asText())
+ .toList();
+ }
+
+ private static List<String> streamArns(List<StreamDescription> streams) {
+ return streams.stream().map(StreamDescription::getStreamArn).toList();
}
@Test
- void latestSequenceNumberIsTheNewestRetainedRecord() throws Exception {
- TableDefinition table = createTestTableWithStream();
- service.enableStream(table.getTableName(), TABLE_ARN, "NEW_IMAGE", "us-east-1");
- JsonNode item = mapper.readTree("{\"userId\":{\"S\":\"u1\"}}");
+ void sameNamedTablesInTwoAccountsKeepSeparateStreams() throws Exception {
+ TableDefinition tableA = sharedNameTable(ACCOUNT_A);
+ TableDefinition tableB = sharedNameTable(ACCOUNT_B);
+ StreamDescription streamA = service.enableStream(SHARED_TABLE, tableA.getTableArn(), "NEW_IMAGE", "us-east-1");
+ StreamDescription streamB = service.enableStream(SHARED_TABLE, tableB.getTableArn(), "NEW_IMAGE", "us-east-1");
+
+ service.captureEvent("INSERT", null, mapper.readTree("{\"userId\":{\"S\":\"a\"}}"),
+ tableA, "us-east-1");
+ service.captureEvent("INSERT", null, mapper.readTree("{\"userId\":{\"S\":\"b\"}}"),
+ tableB, "us-east-1");
+
+ assertNotEquals(streamA.getStreamArn(), streamB.getStreamArn());
+ assertEquals(List.of(streamA.getStreamArn()),
+ streamArns(service.listStreams(SHARED_TABLE, ACCOUNT_A, "us-east-1")));
+ assertEquals(List.of(streamB.getStreamArn()),
+ streamArns(service.listStreams(null, ACCOUNT_B, "us-east-1")));
+ assertEquals(List.of("a"), userIdsIn(streamA.getStreamArn()));
+ assertEquals(List.of("b"), userIdsIn(streamB.getStreamArn()));
+ }
+
+ @Test
+ void disablingOrDeletingOneAccountsStreamLeavesTheOtherEnabled() {
+ TableDefinition tableA = sharedNameTable(ACCOUNT_A);
+ TableDefinition tableB = sharedNameTable(ACCOUNT_B);
+ StreamDescription streamA = service.enableStream(SHARED_TABLE, tableA.getTableArn(), "NEW_IMAGE", "us-east-1");
+ StreamDescription streamB = service.enableStream(SHARED_TABLE, tableB.getTableArn(), "NEW_IMAGE", "us-east-1");
+
+ service.disableStream(tableA.getTableArn());
+
+ assertEquals("DISABLED", service.describeStream(streamA.getStreamArn()).getStreamStatus());
+ assertEquals("ENABLED", service.describeStream(streamB.getStreamArn()).getStreamStatus());
- service.captureEvent(table.getTableName(), "INSERT", null, item, table, "us-east-1");
- service.captureEvent(table.getTableName(), "MODIFY", item, item, table, "us-east-1");
+ service.deleteStream(tableA.getTableArn());
- assertEquals("000000000000000000002", service.latestSequenceNumber(table.getStreamArn()));
+ assertThrows(AwsException.class, () -> service.describeStream(streamA.getStreamArn()));
+ assertEquals("ENABLED", service.describeStream(streamB.getStreamArn()).getStreamStatus());
}
@Test
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamViewTypeDurabilityTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamViewTypeDurabilityTest.java
index d93e4ed5c4..fdd75a81fa 100644
--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamViewTypeDurabilityTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/DynamoDbStreamViewTypeDurabilityTest.java
@@ -103,7 +103,7 @@ class DynamoDbStreamViewTypeDurabilityTest {
// The restart itself: a fresh stream service rebuilds streams from the persisted table.
StorageBackend<String, TableDefinition> reopened = diskStore(file);
DynamoDbStreamService afterRestart = new DynamoDbStreamService(mapper, reopened);
- StreamDescription sd = afterRestart.listStreams(TABLE, REGION).get(0);
+ StreamDescription sd = afterRestart.listStreams(TABLE, "000000000000", REGION).get(0);
assertEquals("KEYS_ONLY", sd.getStreamViewType(),
"a restarted stream must not resume the old image shape");
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbTableServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbTableServiceTest.java
index 0870b5fd50..d3b193fc86 100644
--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbTableServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/NativeDynamoDbTableServiceTest.java
@@ -133,7 +133,7 @@ class NativeDynamoDbTableServiceTest {
private void assertNoTableAndNoStream() {
assertTrue(dynamoDbService.findTable(TABLE, REGION).isEmpty(), "no table may be left behind");
- assertTrue(streams.listStreams(TABLE, REGION).isEmpty(), "no stream may be left behind");
+ assertTrue(streams.listStreams(TABLE, ACCOUNT, REGION).isEmpty(), "no stream may be left behind");
}
@Test
@@ -206,7 +206,7 @@ class NativeDynamoDbTableServiceTest {
persisted.getKmsMasterKeyArn());
StreamDescription stream = new DynamoDbStreamService(mapper, diskStore(file))
- .listStreams(TABLE, REGION).get(0);
+ .listStreams(TABLE, ACCOUNT, REGION).get(0);
assertEquals("NEW_IMAGE", stream.getStreamViewType(),
"a restarted stream must come back with the persisted view type");
}
@@ -270,7 +270,7 @@ class NativeDynamoDbTableServiceTest {
assertNull(persisted.getKmsMasterKeyArn());
StreamDescription stream = new DynamoDbStreamService(mapper, diskStore(file))
- .listStreams(TABLE, REGION).get(0);
+ .listStreams(TABLE, ACCOUNT, REGION).get(0);
assertEquals("NEW_IMAGE", stream.getStreamViewType(),
"a restarted stream must not resume the old image shape");
}
@@ -283,7 +283,7 @@ class NativeDynamoDbTableServiceTest {
tableService.updateTable(updateRequest(List.of(), List.of(), settings(null, false, null, null, null)), REGION);
assertFalse(reloadFromDisk().isStreamEnabled());
- assertEquals("DISABLED", streams.listStreams(TABLE, REGION).get(0).getStreamStatus());
+ assertEquals("DISABLED", streams.listStreams(TABLE, ACCOUNT, REGION).get(0).getStreamStatus());
}
@Test
@@ -330,7 +330,7 @@ class NativeDynamoDbTableServiceTest {
assertFalse(persisted.isDeletionProtectionEnabled());
assertNull(persisted.getTableClass());
assertFalse(persisted.isStreamEnabled());
- assertTrue(streams.listStreams(TABLE, REGION).isEmpty(), "the rejected update must not enable a stream");
+ assertTrue(streams.listStreams(TABLE, ACCOUNT, REGION).isEmpty(), "the rejected update must not enable a stream");
}
@Test
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbApiStreamReaderTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbApiStreamReaderTest.java
new file mode 100644
index 0000000000..0bdae70eed
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbApiStreamReaderTest.java
@@ -0,0 +1,458 @@
+package io.github.hectorvent.floci.services.dynamodb.backend;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Api;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Call;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Reply;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Scope;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.CheckpointLifetime;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.Cursor;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.Position;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.RecordsPage;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.Shard;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.Stream;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** The DynamoDB Streams reader over the raw seam, driven by a scripted engine. */
+class DynamoDbApiStreamReaderTest {
+
+ private static final String STREAM_ARN =
+ "arn:aws:dynamodb:eu-west-1:333333333333:table/orders/stream/2026-09-25T00:00:00.000";
+ private static final Stream STREAM = Stream.of(STREAM_ARN);
+ private static final String ERROR_PREFIX = "com.amazonaws.dynamodb.v20120810#";
+
+ private final ObjectMapper mapper = new ObjectMapper();
+ private final List<Call> calls = new ArrayList<>();
+
+ private interface Script {
+ Reply answer(Call call) throws Exception;
+ }
+
+ private DynamoDbStreamReader reader(Script script) {
+ return new DynamoDbApiStreamReader(call -> {
+ calls.add(call);
+ return script.answer(call);
+ }, CheckpointLifetime.PROCESS);
+ }
+
+ private Reply ok(String json) throws IOException {
+ return new Reply(200, mapper.readTree(json), Map.of());
+ }
+
+ private JsonNode json(String text) throws IOException {
+ return mapper.readTree(text);
+ }
+
+ private static String recordJson(String sequenceNumber) {
+ return "{\"eventName\":\"INSERT\",\"dynamodb\":{\"SequenceNumber\":\"" + sequenceNumber + "\"}}";
+ }
+
+ private static Shard shard(String shardId, String parentShardId) {
+ return new Shard(shardId, parentShardId, null, null);
+ }
+
+ private Reply trimmed() throws IOException {
+ return new Reply(400, json("{\"__type\":\"" + ERROR_PREFIX
+ + "TrimmedDataAccessException\",\"message\":\"Trimmed\"}"), Map.of());
+ }
+
+ private long trimHorizonIteratorCalls() {
+ return calls.stream()
+ .filter(call -> "GetShardIterator".equals(call.action()))
+ .filter(call -> "TRIM_HORIZON".equals(call.body().path("ShardIteratorType").asText()))
+ .count();
+ }
+
+ @Test
+ void streamOfScopesToTheStreamOwner() {
+ assertEquals(new Scope("333333333333", "eu-west-1"), STREAM.scope());
+ assertEquals(STREAM_ARN, STREAM.arn());
+ }
+
+ @Test
+ void shardsFollowsLastEvaluatedShardIdAcrossPages() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> call.body().has("ExclusiveStartShardId")
+ ? ok("""
+ {"StreamDescription":{"Shards":[{"ShardId":"child","ParentShardId":"parent",
+ "SequenceNumberRange":{"StartingSequenceNumber":"5","EndingSequenceNumber":"9"}}],
+ "LastEvaluatedShardId":null}}""")
+ : ok("""
+ {"StreamDescription":{"Shards":[{"ShardId":"parent",
+ "SequenceNumberRange":{"StartingSequenceNumber":"1"}}],
+ "LastEvaluatedShardId":"parent"}}"""));
+
+ List<Shard> shards = reader.shards(STREAM);
+
+ assertEquals(List.of(new Shard("parent", null, "1", null), new Shard("child", "parent", "5", "9")), shards);
+ assertEquals(json("{\"StreamArn\":\"" + STREAM_ARN + "\"}"), calls.get(0).body());
+ assertEquals(json("{\"StreamArn\":\"" + STREAM_ARN + "\",\"ExclusiveStartShardId\":\"parent\"}"),
+ calls.get(1).body());
+ assertEquals(2, calls.size());
+ }
+
+ @Test
+ void describeStreamSendsTheLimitAndReadsAMissingLastEvaluatedShardIdAsNull() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> ok("{\"StreamDescription\":{\"Shards\":[]}}"));
+
+ DynamoDbStreamReader.ShardsPage page = reader.describeStream(STREAM, "s0", 2);
+
+ assertEquals(json("{\"StreamArn\":\"" + STREAM_ARN + "\",\"ExclusiveStartShardId\":\"s0\",\"Limit\":2}"),
+ calls.get(0).body());
+ assertSame(STREAM, page.stream());
+ assertTrue(page.shards().isEmpty());
+ assertNull(page.lastEvaluatedShardId());
+ }
+
+ @Test
+ void readableHoldsAChildUntilItsParentIsFinished() {
+ List<Shard> shards = List.of(shard("parent", null), shard("child", "parent"),
+ shard("orphan", "trimmed-away"), shard("grandchild", "child"));
+
+ assertEquals(List.of(shard("parent", null), shard("orphan", "trimmed-away")),
+ DynamoDbStreamReader.readable(shards, Set.of()));
+ assertEquals(List.of(shard("child", "parent"), shard("orphan", "trimmed-away")),
+ DynamoDbStreamReader.readable(shards, Set.of("parent")));
+ assertEquals(List.of(shard("orphan", "trimmed-away"), shard("grandchild", "child")),
+ DynamoDbStreamReader.readable(shards, Set.of("parent", "child")));
+ }
+
+ @Test
+ void anOpenEmptyPageKeepsItsCursorWhileAClosedShardEndsWithNone() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> switch (call.body().path("ShardIterator").asText()) {
+ case "open" -> ok("{\"Records\":[],\"NextShardIterator\":\"open-2\"}");
+ case "closed" -> ok("{\"Records\":[],\"NextShardIterator\":null}");
+ default -> ok("{\"Records\":[]}");
+ });
+
+ RecordsPage open = reader.getRecords(new Cursor(STREAM, "s1", "open"), 10);
+ RecordsPage closed = reader.getRecords(new Cursor(STREAM, "s1", "closed"), 10);
+ RecordsPage missing = reader.getRecords(new Cursor(STREAM, "s1", "missing"), 10);
+
+ assertTrue(open.records().isEmpty());
+ assertEquals(new Cursor(STREAM, "s1", "open-2"), open.nextCursor());
+ assertTrue(closed.records().isEmpty());
+ assertNull(closed.nextCursor());
+ assertNull(missing.nextCursor());
+ }
+
+ @Test
+ void errorRepliesBecomeAwsExceptionsWithTheShortCode() {
+ DynamoDbStreamReader reader = reader(call -> switch (call.body().path("ShardIterator").asText()) {
+ case "expired" -> new Reply(400, json("{\"__type\":\"" + ERROR_PREFIX
+ + "ExpiredIteratorException\",\"message\":\"Iterator expired\"}"), Map.of());
+ default -> new Reply(400, json("{\"__type\":\"" + ERROR_PREFIX
+ + "TrimmedDataAccessException\",\"Message\":\"Trimmed\"}"), Map.of());
+ });
+
+ AwsException expired = assertThrows(AwsException.class,
+ () -> reader.getRecords(new Cursor(STREAM, "s1", "expired"), 10));
+ AwsException trimmed = assertThrows(AwsException.class,
+ () -> reader.getRecords(new Cursor(STREAM, "s1", "trimmed"), 10));
+
+ assertEquals("ExpiredIteratorException", expired.getErrorCode());
+ assertEquals("Iterator expired", expired.getMessage());
+ assertEquals(400, expired.getHttpStatus());
+ assertEquals("TrimmedDataAccessException", trimmed.getErrorCode());
+ assertEquals("Trimmed", trimmed.getMessage());
+ assertEquals(400, trimmed.getHttpStatus());
+ }
+
+ @Test
+ void anErrorReplyWithoutATypeIsAnInternalServerError() {
+ DynamoDbStreamReader reader = reader(call -> new Reply(500, null, Map.of()));
+
+ AwsException error = assertThrows(AwsException.class, () -> reader.describeStream(STREAM, null, null));
+
+ assertEquals("InternalServerError", error.getErrorCode());
+ assertEquals(500, error.getHttpStatus());
+ }
+
+ @Test
+ void aThrownAwsExceptionPropagatesUnchanged() {
+ AwsException thrown = new AwsException("ResourceNotFoundException", "Stream not found", 400);
+ DynamoDbStreamReader reader = reader(call -> {
+ throw thrown;
+ });
+
+ assertSame(thrown, assertThrows(AwsException.class, () -> reader.describeStream(STREAM, null, null)));
+ }
+
+ @Test
+ void aCheckedExceptionBecomesAnIllegalStateException() {
+ IOException thrown = new IOException("boom");
+ DynamoDbStreamReader reader = reader(call -> {
+ throw thrown;
+ });
+
+ IllegalStateException error = assertThrows(IllegalStateException.class,
+ () -> reader.describeStream(STREAM, null, null));
+ assertSame(thrown, error.getCause());
+ }
+
+ @Test
+ void everyCallRunsUnderTheStreamScopeOnTheStreamsApi() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> switch (call.action()) {
+ case "DescribeStream" -> ok("{\"StreamDescription\":{\"Shards\":[]}}");
+ case "GetShardIterator" -> ok("{\"ShardIterator\":\"it\"}");
+ default -> ok("{\"Records\":[],\"NextShardIterator\":\"it\"}");
+ });
+
+ reader.describeStream(STREAM, null, null);
+ Cursor cursor = reader.getShardIterator(STREAM, "s1", Position.LATEST, null);
+ reader.getRecords(cursor, 10);
+
+ assertEquals(List.of("DescribeStream", "GetShardIterator", "GetRecords"),
+ calls.stream().map(Call::action).toList());
+ for (Call call : calls) {
+ assertEquals(new Scope("333333333333", "eu-west-1"), call.scope());
+ assertEquals(Api.DYNAMODB_STREAMS, call.api());
+ }
+ }
+
+ @Test
+ void getShardIteratorSendsTheSequenceNumberOnlyWhenGiven() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> ok("{\"ShardIterator\":\"it-" + calls.size() + "\"}"));
+
+ Cursor fromStart = reader.getShardIterator(STREAM, "s1", Position.TRIM_HORIZON, null);
+ Cursor afterFive = reader.getShardIterator(STREAM, "s1", Position.AFTER_SEQUENCE_NUMBER, "5");
+
+ assertEquals(json("{\"StreamArn\":\"" + STREAM_ARN
+ + "\",\"ShardId\":\"s1\",\"ShardIteratorType\":\"TRIM_HORIZON\"}"), calls.get(0).body());
+ assertEquals(json("{\"StreamArn\":\"" + STREAM_ARN
+ + "\",\"ShardId\":\"s1\",\"ShardIteratorType\":\"AFTER_SEQUENCE_NUMBER\",\"SequenceNumber\":\"5\"}"),
+ calls.get(1).body());
+ assertEquals(new Cursor(STREAM, "s1", "it-1"), fromStart);
+ assertEquals(new Cursor(STREAM, "s1", "it-2"), afterFive);
+ }
+
+ @Test
+ void repeatReadsOfOneCursorSendIdenticalRequestsAndLeaveTheRecordUntouched() throws Exception {
+ Reply reply = ok("{\"Records\":[" + recordJson("7") + "],\"NextShardIterator\":\"next\"}");
+ DynamoDbStreamReader reader = reader(call -> reply);
+ Cursor cursor = new Cursor(STREAM, "s1", "it");
+
+ RecordsPage first = reader.getRecords(cursor, 10);
+ RecordsPage second = reader.getRecords(cursor, 10);
+
+ assertEquals(json("{\"ShardIterator\":\"it\",\"Limit\":10}"), calls.get(0).body());
+ assertEquals(calls.get(0).body(), calls.get(1).body());
+ assertEquals("7", first.records().get(0).sequenceNumber());
+ assertEquals(json(recordJson("7")), first.records().get(0).awsRecord());
+ assertEquals(json(recordJson("7")), second.records().get(0).awsRecord());
+ assertEquals(new Cursor(STREAM, "s1", "next"), first.nextCursor());
+ }
+
+ @Test
+ void newestSequenceNumbersRestartsAShardWhoseOldestRecordsWereTrimmedMeanwhile() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> switch (call.action()) {
+ case "DescribeStream" -> ok("{\"StreamDescription\":{\"Shards\":[{\"ShardId\":\"s1\"}]}}");
+ case "GetShardIterator" -> ok("{\"ShardIterator\":\"s1-" + trimHorizonIteratorCalls() + "\"}");
+ default -> switch (call.body().path("ShardIterator").asText()) {
+ case "s1-1" -> trimmed();
+ case "s1-2" -> ok("{\"Records\":[" + recordJson("5") + "," + recordJson("6")
+ + "],\"NextShardIterator\":\"s1-3\"}");
+ default -> ok("{\"Records\":[],\"NextShardIterator\":\"idle\"}");
+ };
+ });
+
+ Map<String, String> newest = reader.newestSequenceNumbers(STREAM);
+
+ assertEquals(Map.of("s1", "6"), newest);
+ assertEquals(2, trimHorizonIteratorCalls());
+ }
+
+ @Test
+ void newestSequenceNumbersGivesUpAfterRepeatedTrims() {
+ DynamoDbStreamReader reader = reader(call -> switch (call.action()) {
+ case "DescribeStream" -> ok("{\"StreamDescription\":{\"Shards\":[{\"ShardId\":\"s1\"}]}}");
+ case "GetShardIterator" -> ok("{\"ShardIterator\":\"it\"}");
+ default -> trimmed();
+ });
+
+ AwsException error = assertThrows(AwsException.class, () -> reader.newestSequenceNumbers(STREAM));
+
+ assertEquals("TrimmedDataAccessException", error.getErrorCode());
+ assertEquals(4, trimHorizonIteratorCalls());
+ }
+
+ @Test
+ void newestSequenceNumbersScansEveryShardFromTrimHorizon() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> switch (call.action()) {
+ case "DescribeStream" -> ok("""
+ {"StreamDescription":{"Shards":[{"ShardId":"s1"},{"ShardId":"s2","ParentShardId":"s1"},
+ {"ShardId":"s3"}]}}""");
+ case "GetShardIterator" -> ok("{\"ShardIterator\":\"" + call.body().path("ShardId").asText() + "-0\"}");
+ default -> switch (call.body().path("ShardIterator").asText()) {
+ case "s1-0" -> ok("{\"Records\":[" + recordJson("1") + "," + recordJson("2")
+ + "],\"NextShardIterator\":\"s1-1\"}");
+ case "s1-1" -> ok("{\"Records\":[" + recordJson("3") + "],\"NextShardIterator\":\"s1-2\"}");
+ case "s2-0" -> ok("{\"Records\":[" + recordJson("10") + "],\"NextShardIterator\":null}");
+ default -> ok("{\"Records\":[],\"NextShardIterator\":\"idle\"}");
+ };
+ });
+
+ Map<String, String> newest = reader.newestSequenceNumbers(STREAM);
+
+ assertEquals(Map.of("s1", "3", "s2", "10"), newest);
+ assertFalse(calls.stream().anyMatch(call -> "idle".equals(call.body().path("ShardIterator").asText())));
+ for (Call call : calls) {
+ if ("GetShardIterator".equals(call.action())) {
+ assertEquals("TRIM_HORIZON", call.body().path("ShardIteratorType").asText());
+ } else if ("GetRecords".equals(call.action())) {
+ assertEquals(1000, call.body().path("Limit").asInt());
+ }
+ }
+ }
+
+ @Test
+ void getRecordsCapsTheLimitAtTheAwsMaximum() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> ok("{\"Records\":[],\"NextShardIterator\":\"it\"}"));
+
+ reader.getRecords(new Cursor(STREAM, "s1", "it"), 5000);
+
+ assertEquals(json("{\"ShardIterator\":\"it\",\"Limit\":1000}"), calls.get(0).body());
+ }
+
+ @Test
+ void readAfterStartsAtTheTrimHorizonWithoutACommittedSequenceAndAfterItOtherwise() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> {
+ if ("GetShardIterator".equals(call.action())) {
+ return ok("{\"ShardIterator\":\"it\"}");
+ }
+ return "it".equals(call.body().path("ShardIterator").asText())
+ ? ok("{\"Records\":[" + recordJson("7") + "],\"NextShardIterator\":\"next\"}")
+ : ok("{\"Records\":[],\"NextShardIterator\":\"next\"}");
+ });
+
+ reader.readAfter(STREAM, "s1", null, 10);
+ assertEquals(json("{\"StreamArn\":\"" + STREAM_ARN
+ + "\",\"ShardId\":\"s1\",\"ShardIteratorType\":\"TRIM_HORIZON\"}"), calls.get(0).body());
+ assertEquals(json("{\"ShardIterator\":\"it\",\"Limit\":10}"), calls.get(1).body());
+ calls.clear();
+ RecordsPage page = reader.readAfter(STREAM, "s1", "5", 10);
+
+ assertEquals(json("{\"StreamArn\":\"" + STREAM_ARN
+ + "\",\"ShardId\":\"s1\",\"ShardIteratorType\":\"AFTER_SEQUENCE_NUMBER\",\"SequenceNumber\":\"5\"}"),
+ calls.get(0).body());
+ assertEquals(json("{\"ShardIterator\":\"it\",\"Limit\":10}"), calls.get(1).body());
+ assertEquals("7", page.records().get(0).sequenceNumber());
+ }
+
+ /** A shard of 3000 records whose iterator {@code at-N} reads after the Nth one. */
+ private DynamoDbStreamReader shardOf3000Records() {
+ return reader(call -> {
+ if ("GetShardIterator".equals(call.action())) {
+ return ok("{\"ShardIterator\":\"at-0\"}");
+ }
+ int after = Integer.parseInt(call.body().path("ShardIterator").asText().substring(3));
+ int end = Math.min(after + call.body().path("Limit").asInt(), 3000);
+ StringBuilder records = new StringBuilder();
+ for (int i = after + 1; i <= end; i++) {
+ records.append(i > after + 1 ? "," : "").append(recordJson(String.valueOf(i)));
+ }
+ return ok("{\"Records\":[" + records + "],\"NextShardIterator\":\"at-" + end + "\"}");
+ });
+ }
+
+ @Test
+ void readAfterFillsALimitAboveTheAwsMaximumFromSeveralReads() throws Exception {
+ RecordsPage page = shardOf3000Records().readAfter(STREAM, "s1", null, 2500);
+
+ assertEquals(List.of(1000, 1000, 500), calls.stream()
+ .filter(call -> "GetRecords".equals(call.action()))
+ .map(call -> call.body().path("Limit").asInt())
+ .toList());
+ assertEquals(2500, page.records().size());
+ assertEquals("1", page.records().get(0).sequenceNumber());
+ assertEquals("2500", page.records().get(2499).sequenceNumber());
+ assertEquals(new Cursor(STREAM, "s1", "at-2500"), page.nextCursor());
+ }
+
+ @Test
+ void readAfterKeepsTheRecordsAlreadyReadWhenALaterReadIsTrimmed() throws Exception {
+ List<String> firstPage = new ArrayList<>();
+ StringBuilder records = new StringBuilder();
+ for (int i = 1; i <= 1000; i++) {
+ firstPage.add(String.valueOf(i));
+ records.append(i > 1 ? "," : "").append(recordJson(String.valueOf(i)));
+ }
+ DynamoDbStreamReader reader = reader(call -> {
+ if ("GetShardIterator".equals(call.action())) {
+ return ok("{\"ShardIterator\":\"first\"}");
+ }
+ return "first".equals(call.body().path("ShardIterator").asText())
+ ? ok("{\"Records\":[" + records + "],\"NextShardIterator\":\"second\"}")
+ : trimmed();
+ });
+
+ RecordsPage page = reader.readAfter(STREAM, "s1", null, 1500);
+
+ assertEquals(firstPage, page.records().stream().map(DynamoDbStreamReader.Record::sequenceNumber).toList());
+ assertFalse(page.closed());
+ }
+
+ @Test
+ void readAfterReturnsWhatItReadAtTheTipOfAnOpenShardAndTheEndOfAClosedOne() throws Exception {
+ DynamoDbStreamReader reader = reader(call -> {
+ if ("GetShardIterator".equals(call.action())) {
+ return ok("{\"ShardIterator\":\"" + call.body().path("ShardId").asText() + "\"}");
+ }
+ return switch (call.body().path("ShardIterator").asText()) {
+ case "open" -> ok("{\"Records\":[" + recordJson("1") + "],\"NextShardIterator\":\"tip\"}");
+ case "tip" -> ok("{\"Records\":[],\"NextShardIterator\":\"tip-2\"}");
+ case "closing" -> ok("{\"Records\":[" + recordJson("2") + "],\"NextShardIterator\":null}");
+ default -> ok("{\"Records\":[],\"NextShardIterator\":null}");
+ };
+ });
+
+ RecordsPage open = reader.readAfter(STREAM, "open", null, 10);
+ RecordsPage closing = reader.readAfter(STREAM, "closing", null, 10);
+ RecordsPage closed = reader.readAfter(STREAM, "closed", null, 10);
+
+ assertEquals(List.of("1"), open.records().stream().map(DynamoDbStreamReader.Record::sequenceNumber).toList());
+ assertEquals(new Cursor(STREAM, "open", "tip-2"), open.nextCursor());
+ assertEquals(List.of("2"), closing.records().stream().map(DynamoDbStreamReader.Record::sequenceNumber).toList());
+ assertTrue(closed.closed());
+ assertEquals(List.of(10, 9, 10, 10), calls.stream()
+ .filter(call -> "GetRecords".equals(call.action()))
+ .map(call -> call.body().path("Limit").asInt())
+ .toList());
+ }
+
+ @Test
+ void aPageIsClosedOnlyWhenItHasNoRecordsAndNoNextCursor() throws Exception {
+ List<DynamoDbStreamReader.Record> one = List.of(new DynamoDbStreamReader.Record("7", json(recordJson("7"))));
+ Cursor next = new Cursor(STREAM, "s1", "next");
+
+ assertTrue(new RecordsPage(List.of(), null).closed());
+ assertFalse(new RecordsPage(List.of(), next).closed());
+ assertFalse(new RecordsPage(one, null).closed());
+ }
+
+ @Test
+ void checkpointLifetimeEchoesTheConstructorValue() {
+ DynamoDbOperations unused = call -> {
+ throw new AssertionError("no call expected");
+ };
+
+ assertEquals(CheckpointLifetime.PROCESS,
+ new DynamoDbApiStreamReader(unused, CheckpointLifetime.PROCESS).checkpointLifetime());
+ assertEquals(CheckpointLifetime.STREAM,
+ new DynamoDbApiStreamReader(unused, CheckpointLifetime.STREAM).checkpointLifetime());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbBackendSelectionTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbBackendSelectionTest.java
index 8f24febc52..712518ff15 100644
--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbBackendSelectionTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbBackendSelectionTest.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.dynamodb.backend;
+import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import io.github.hectorvent.floci.core.common.AwsException;
@@ -8,8 +9,15 @@ import io.github.hectorvent.floci.services.dynamodb.DynamoDbFacade;
import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Api;
import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Call;
import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations.Scope;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.CheckpointLifetime;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.Cursor;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.Position;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.RecordsPage;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.Shard;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.Stream;
import io.github.hectorvent.floci.services.dynamodb.model.AttributeDefinition;
import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
+import io.github.hectorvent.floci.services.dynamodb.model.TableDefinition;
import io.quarkus.arc.ClientProxy;
import io.quarkus.test.junit.QuarkusTest;
import jakarta.enterprise.inject.Instance;
@@ -21,6 +29,7 @@ import java.util.UUID;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -42,6 +51,9 @@ class DynamoDbBackendSelectionTest {
@Inject
DynamoDbTableAccess tables;
+ @Inject
+ DynamoDbStreamReader streamReader;
+
// Collected exactly as ResourceExplorer2Service collects its providers.
@Inject
Instance<ResourceProvider> providers;
@@ -87,4 +99,95 @@ class DynamoDbBackendSelectionTest {
tables.deleteTable(ACCOUNT_A, tableName);
}
}
+
+ private void createTable(Scope scope, String tableName) {
+ tables.createTable(scope, tableName, List.of(new KeySchemaElement("id", "HASH")),
+ List.of(new AttributeDefinition("id", "S")), 5L, 5L, List.of(), List.of());
+ }
+
+ @Test
+ void theStreamReaderReadsTheSelectedBackendAsTheStreamOwner() {
+ String tableName = "backend-stream-" + UUID.randomUUID();
+ createTable(ACCOUNT_A, tableName);
+ try {
+ TableDefinition table = tables.enableStream(ACCOUNT_A, tableName, "NEW_AND_OLD_IMAGES");
+ ObjectNode item = mapper.createObjectNode();
+ item.putObject("id").put("S", "streamed");
+ items.putItem(ACCOUNT_A, tableName, item, null, null, null);
+
+ Stream stream = Stream.of(table.getStreamArn());
+ List<Shard> shards = streamReader.shards(stream);
+ Cursor cursor = streamReader.getShardIterator(stream, shards.get(0).shardId(), Position.TRIM_HORIZON, null);
+ RecordsPage page = streamReader.getRecords(cursor, 10);
+
+ assertEquals(ACCOUNT_A, stream.scope());
+ assertEquals(1, shards.size());
+ assertEquals(1, page.records().size());
+ assertEquals(item, page.records().get(0).awsRecord().path("dynamodb").path("Keys"));
+ assertEquals(CheckpointLifetime.PROCESS, streamReader.checkpointLifetime());
+ } finally {
+ tables.deleteTable(ACCOUNT_A, tableName);
+ }
+ }
+
+ @Test
+ void listStreamsAnswersOnlyTheScopeAccount() throws Exception {
+ String tableName = "backend-list-streams-" + UUID.randomUUID();
+ createTable(ACCOUNT_A, tableName);
+ createTable(ACCOUNT_B, tableName);
+ try {
+ String streamA = tables.enableStream(ACCOUNT_A, tableName, "KEYS_ONLY").getStreamArn();
+ String streamB = tables.enableStream(ACCOUNT_B, tableName, "KEYS_ONLY").getStreamArn();
+ assertNotEquals(streamA, streamB);
+
+ assertEquals(List.of(streamA), listedStreamArns(ACCOUNT_A, tableName));
+ assertEquals(List.of(streamB), listedStreamArns(ACCOUNT_B, tableName));
+ } finally {
+ tables.deleteTable(ACCOUNT_A, tableName);
+ tables.deleteTable(ACCOUNT_B, tableName);
+ }
+ }
+
+ private List<String> listedStreamArns(Scope scope, String tableName) throws Exception {
+ ObjectNode body = mapper.createObjectNode().put("TableName", tableName);
+ JsonNode reply = operations.execute(new Call(scope, Api.DYNAMODB_STREAMS, "ListStreams", body)).body();
+ return reply.path("Streams").findValuesAsText("StreamArn");
+ }
+
+ @Test
+ void aStreamOutsideTheScopeIsNotFound() throws Exception {
+ String tableName = "backend-foreign-stream-" + UUID.randomUUID();
+ createTable(ACCOUNT_A, tableName);
+ try {
+ String streamArn = tables.enableStream(ACCOUNT_A, tableName, "KEYS_ONLY").getStreamArn();
+ ObjectNode item = mapper.createObjectNode();
+ item.putObject("id").put("S", "private");
+ items.putItem(ACCOUNT_A, tableName, item, null, null, null);
+ ObjectNode stream = mapper.createObjectNode().put("StreamArn", streamArn);
+ String shardId = streams(ACCOUNT_A, "DescribeStream", stream)
+ .path("StreamDescription").path("Shards").path(0).path("ShardId").asText();
+ ObjectNode fromStart = stream.deepCopy().put("ShardId", shardId).put("ShardIteratorType", "TRIM_HORIZON");
+ String iterator = streams(ACCOUNT_A, "GetShardIterator", fromStart).path("ShardIterator").asText();
+ ObjectNode read = mapper.createObjectNode().put("ShardIterator", iterator);
+
+ for (Scope outside : List.of(ACCOUNT_B, new Scope(ACCOUNT_A.accountId(), "eu-west-1"))) {
+ assertStreamNotFound(outside, "DescribeStream", stream, streamArn);
+ assertStreamNotFound(outside, "GetShardIterator", fromStart, streamArn);
+ assertStreamNotFound(outside, "GetRecords", read, streamArn);
+ }
+ assertEquals(1, streams(ACCOUNT_A, "GetRecords", read).path("Records").size());
+ } finally {
+ tables.deleteTable(ACCOUNT_A, tableName);
+ }
+ }
+
+ private JsonNode streams(Scope scope, String action, ObjectNode body) throws Exception {
+ return operations.execute(new Call(scope, Api.DYNAMODB_STREAMS, action, body)).body();
+ }
+
+ private void assertStreamNotFound(Scope scope, String action, ObjectNode body, String streamArn) {
+ AwsException notFound = assertThrows(AwsException.class, () -> streams(scope, action, body));
+ assertEquals("ResourceNotFoundException", notFound.getErrorCode());
+ assertEquals("Stream not found: " + streamArn, notFound.getMessage());
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbNativeBackendTest.java b/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbNativeBackendTest.java
index fa1fbe7cb7..956a9a7f2b 100644
--- a/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbNativeBackendTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/dynamodb/backend/DynamoDbNativeBackendTest.java
@@ -63,7 +63,8 @@ class DynamoDbNativeBackendTest {
mock(KinesisStreamingForwarder.class), null, mapper, mock(EmulatorConfig.class, RETURNS_DEEP_STUBS));
NativeDynamoDbTableService tables = new NativeDynamoDbTableService(service, streams, mock(KinesisService.class));
backend = new NativeDynamoDbBackend(new NativeDynamoDbJsonHandler(service, tables, mapper),
- new NativeDynamoDbStreamsJsonHandler(streams, service, mapper), service, mapper);
+ new NativeDynamoDbStreamsJsonHandler(streams, service, new RegionResolver(REGION, ACCOUNT), mapper),
+ service, mapper);
}
private Reply call(Api api, String action, JsonNode body) throws Exception {
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePollerTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePollerTest.java
index 82a0ce0055..9100047e08 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePollerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/DynamoDbStreamsEventSourcePollerTest.java
@@ -11,11 +11,9 @@ import io.github.hectorvent.floci.core.common.RequestScopes;
import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.InMemoryStorage;
import io.github.hectorvent.floci.core.storage.StorageFactory;
-import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
-import io.github.hectorvent.floci.services.dynamodb.model.AttributeDefinition;
-import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
-import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
-import io.github.hectorvent.floci.services.dynamodb.model.TableDefinition;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.CheckpointLifetime;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.Position;
import io.github.hectorvent.floci.services.eventbridge.EventBridgeService;
import io.github.hectorvent.floci.services.iam.IamService;
import io.github.hectorvent.floci.services.lambda.model.EventSourceMapping;
@@ -48,7 +46,9 @@ import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
@@ -66,8 +66,11 @@ import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.atLeast;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.timeout;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
@@ -90,9 +93,11 @@ class DynamoDbStreamsEventSourcePollerTest {
private static final String STREAM_ARN =
"arn:aws:dynamodb:us-east-1:000000000000:table/t/stream/2026-08-01T00:00:00.000";
private static final String STALE_CHECKPOINT = "000000000000000000634";
+ private static final String SHARD_ID = "shardId-0000000001-00000000001";
private DynamoDbStreamsEventSourcePoller poller;
- private DynamoDbStreamService streamService;
+ private FakeStreamReader reader;
+ private List<DynamoDbStreamReader.Record> shard;
private LambdaExecutorService executorService;
private LambdaFunctionStore functionStore;
private LambdaAliasStore aliasStore;
@@ -103,6 +108,7 @@ class DynamoDbStreamsEventSourcePollerTest {
private io.github.hectorvent.floci.services.sns.SnsService snsService;
private io.github.hectorvent.floci.services.s3.S3Service s3Service;
private final AtomicLong clock = new AtomicLong();
+ private final AtomicLong sequence = new AtomicLong();
@Inject
Instance<RequestContext> requestContextInstance;
@@ -118,7 +124,9 @@ class DynamoDbStreamsEventSourcePollerTest {
when(lambdaConfig.pollIntervalMs()).thenReturn(1000L);
when(config.effectiveBaseUrl()).thenReturn("http://localhost:4566");
- streamService = mock(DynamoDbStreamService.class);
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
+ shard = streams.shard(SHARD_ID, null);
+ reader = spy(streams);
executorService = mock(LambdaExecutorService.class);
functionStore = mock(LambdaFunctionStore.class);
aliasStore = mock(LambdaAliasStore.class);
@@ -128,11 +136,75 @@ class DynamoDbStreamsEventSourcePollerTest {
snsService = mock(io.github.hectorvent.floci.services.sns.SnsService.class);
s3Service = mock(io.github.hectorvent.floci.services.s3.S3Service.class);
- // A mocked Vertx makes setPeriodic a no-op, so startPolling registers no live timer and
- // the tests drive pollAndInvoke deterministically.
- poller = new DynamoDbStreamsEventSourcePoller(
- mock(Vertx.class), streamService, executorService, new LambdaTargetResolver(functionStore, aliasStore),
- esmStore, OBJECT_MAPPER, config, filterMatcher, sqsService, snsService, s3Service, clock::get);
+ poller = pollerOver(reader, esmStore);
+ }
+
+ /** A mocked Vertx makes setPeriodic a no-op, so startPolling registers no live timer and tests drive the polls. */
+ private DynamoDbStreamsEventSourcePoller pollerOver(DynamoDbStreamReader streamReader, EsmStore store) {
+ return new DynamoDbStreamsEventSourcePoller(
+ mock(Vertx.class), streamReader, executorService, new LambdaTargetResolver(functionStore, aliasStore),
+ store, OBJECT_MAPPER, config, filterMatcher, sqsService, snsService, s3Service, clock::get);
+ }
+
+ /**
+ * Serves shards from memory the way the native engine does. A cursor is the sequence to read
+ * after, or empty for the trim horizon, and a sequence older than the oldest retained record is
+ * trimmed. A closed shard read to its end answers without a next cursor, and a read returns at
+ * most {@link DynamoDbStreamReader#MAX_RECORDS_PER_READ} records, as GetRecords does.
+ */
+ static class FakeStreamReader implements DynamoDbStreamReader {
+
+ private final List<Shard> shards = new CopyOnWriteArrayList<>();
+ private final Map<String, List<Record>> records = new ConcurrentHashMap<>();
+ private final Set<String> closed = ConcurrentHashMap.newKeySet();
+ private final CheckpointLifetime lifetime;
+
+ FakeStreamReader(CheckpointLifetime lifetime) {
+ this.lifetime = lifetime;
+ }
+
+ List<Record> shard(String shardId, String parentShardId) {
+ shards.add(new Shard(shardId, parentShardId, null, null));
+ return records.computeIfAbsent(shardId, ignored -> new CopyOnWriteArrayList<>());
+ }
+
+ void close(String shardId) {
+ closed.add(shardId);
+ }
+
+ @Override
+ public ShardsPage describeStream(Stream stream, String exclusiveStartShardId, Integer limit) {
+ return new ShardsPage(stream, List.copyOf(shards), null);
+ }
+
+ @Override
+ public Cursor getShardIterator(Stream stream, String shardId, Position position, String sequenceNumber) {
+ return new Cursor(stream, shardId, position == Position.TRIM_HORIZON ? "" : sequenceNumber);
+ }
+
+ @Override
+ public RecordsPage getRecords(Cursor cursor, int limit) {
+ List<Record> held = new ArrayList<>(records.get(cursor.shardId()));
+ String after = cursor.token();
+ if (!after.isEmpty() && !held.isEmpty() && after.compareTo(held.get(0).sequenceNumber()) < 0) {
+ throw new AwsException("TrimmedDataAccessException",
+ "The requested sequence number has been trimmed", 400);
+ }
+ List<Record> page = held.stream()
+ .filter(record -> record.sequenceNumber().compareTo(after) > 0)
+ .limit(Math.min(limit, MAX_RECORDS_PER_READ))
+ .toList();
+ if (page.isEmpty()) {
+ return new RecordsPage(page, closed.contains(cursor.shardId()) ? null : cursor);
+ }
+ return new RecordsPage(page,
+ new Cursor(cursor.stream(), cursor.shardId(), page.get(page.size() - 1).sequenceNumber()));
+ }
+
+ @Override
+ public CheckpointLifetime checkpointLifetime() {
+ return lifetime;
+ }
}
private EventSourceMapping persistedStreamsEsmWithStaleCheckpoint() {
@@ -144,7 +216,7 @@ class DynamoDbStreamsEventSourcePollerTest {
esm.setEventSourceArn(STREAM_ARN);
esm.setBatchSize(10);
esm.setEnabled(true);
- esm.getShardSequenceNumbers().put(DynamoDbStreamService.SHARD_ID, STALE_CHECKPOINT);
+ esm.getShardSequenceNumbers().put(SHARD_ID, STALE_CHECKPOINT);
esmStore.saveForAccount(ACCOUNT_ID, esm);
return esm;
}
@@ -158,7 +230,7 @@ class DynamoDbStreamsEventSourcePollerTest {
EventSourceMapping reloaded = esmStore.getForAccount(ACCOUNT_ID, "esm-1").orElseThrow();
assertTrue(reloaded.getShardSequenceNumbers().isEmpty(),
"a checkpoint persisted before restart must be discarded at startup, since the "
- + "stream's sequence numbers reset to 1 — otherwise every new record is skipped");
+ + "stream's sequence numbers reset to 1, otherwise every new record is skipped");
}
@Test
@@ -168,7 +240,7 @@ class DynamoDbStreamsEventSourcePollerTest {
esmStore.saveForAccount(ACCOUNT_ID, esm);
Vertx vertx = mock(Vertx.class);
DynamoDbStreamsEventSourcePoller p = new DynamoDbStreamsEventSourcePoller(
- vertx, streamService, executorService, new LambdaTargetResolver(functionStore, aliasStore),
+ vertx, reader, executorService, new LambdaTargetResolver(functionStore, aliasStore),
esmStore, OBJECT_MAPPER, config, filterMatcher, sqsService, snsService, s3Service, clock::get);
p.startPersistedPollers();
@@ -180,6 +252,17 @@ class DynamoDbStreamsEventSourcePollerTest {
verifyNoInteractions(vertx);
}
+ @Test
+ void startPersistedPollersKeepsTheCheckpointsOfAStreamLifetimeEngine() {
+ persistedStreamsEsmWithStaleCheckpoint();
+
+ pollerOver(new FakeStreamReader(CheckpointLifetime.STREAM), esmStore).startPersistedPollers();
+
+ assertEquals(Map.of(SHARD_ID, STALE_CHECKPOINT),
+ esmStore.getForAccount(ACCOUNT_ID, "esm-1").orElseThrow().getShardSequenceNumbers(),
+ "an engine that keeps stream history across restarts keeps the progress committed against it");
+ }
+
@Test
void startPersistedPollersSkipsADisabledMappingWithoutAnEventSourceArn() {
EventSourceMapping kafka = new EventSourceMapping();
@@ -197,36 +280,19 @@ class DynamoDbStreamsEventSourcePollerTest {
void pollerDeliversPostRestartRecordAfterStartupCheckpointReset() {
persistedStreamsEsmWithStaleCheckpoint();
- // The new stream epoch has a single record at sequence 1.
- DynamoDbStreamRecord record = new DynamoDbStreamRecord();
- record.setEventName("INSERT");
- record.setEventSource("aws:dynamodb");
- record.setAwsRegion("us-east-1");
- record.setSequenceNumber("000000000000000000001");
-
- // Resuming from the stale checkpoint (AFTER_SEQUENCE_NUMBER 634) sees nothing — the bug: the
- // record's sequence (1) is far below the stale checkpoint, so it is silently skipped.
- when(streamService.getShardIterator(STREAM_ARN, DynamoDbStreamService.SHARD_ID,
- "AFTER_SEQUENCE_NUMBER", STALE_CHECKPOINT)).thenReturn("iterator-stale");
- when(streamService.getRecords("iterator-stale", 10))
- .thenReturn(new DynamoDbStreamService.GetRecordsResult(List.of(), "iterator-stale"));
- // Resuming from TRIM_HORIZON — where the fix makes the poller restart — delivers the record.
- when(streamService.getShardIterator(STREAM_ARN, DynamoDbStreamService.SHARD_ID,
- "TRIM_HORIZON", null)).thenReturn("iterator-trim");
- when(streamService.getRecords("iterator-trim", 10))
- .thenReturn(new DynamoDbStreamService.GetRecordsResult(List.of(record), "iterator-trim"));
-
- LambdaFunction fn = new LambdaFunction();
- fn.setFunctionName("fn");
- when(functionStore.getForAccount(ACCOUNT_ID, "us-east-1", "fn")).thenReturn(Optional.of(fn));
+ // The new stream epoch has a single record at sequence 1. Resuming from the stale checkpoint
+ // (AFTER_SEQUENCE_NUMBER 634) sees nothing, the bug: the record's sequence (1) is far below the
+ // stale checkpoint, so it is silently skipped. Resuming from TRIM_HORIZON, where the fix makes
+ // the poller restart, delivers the record.
+ LambdaFunction fn = stubTrimHorizon(List.of(ddbRecord("000000000000000000001", "INSERT", "{}")));
when(executorService.invoke(eq(fn), any(byte[].class), eq(InvocationType.RequestResponse)))
- .thenReturn(new InvokeResult()); // success — no functionError
+ .thenReturn(new InvokeResult()); // success, no functionError
// Startup invalidates the stale checkpoint...
poller.startPersistedPollers();
// ...so the poll resumes from TRIM_HORIZON and delivers the record instead of dropping it.
// (Were the checkpoint not cleared, the poll would take the AFTER_SEQUENCE_NUMBER branch,
- // find nothing, and invoke would never be called — this verify would then fail.)
+ // find nothing, and invoke would never be called, so this verify would then fail.)
poller.pollAndInvoke(esmStore.getForAccount(ACCOUNT_ID, "esm-1").orElseThrow());
verify(executorService, timeout(2000))
@@ -239,9 +305,7 @@ class DynamoDbStreamsEventSourcePollerTest {
// The mock records writes for verification; reads see what filterEsm stored, as a real store would.
when(store.getForAccount(anyString(), anyString()))
.thenAnswer(inv -> esmStore.getForAccount(inv.getArgument(0), inv.getArgument(1)));
- return new DynamoDbStreamsEventSourcePoller(
- mock(Vertx.class), streamService, executorService, new LambdaTargetResolver(functionStore, aliasStore),
- store, OBJECT_MAPPER, config, filterMatcher, sqsService, snsService, s3Service, clock::get);
+ return pollerOver(reader, store);
}
private void advancePastRetry(DynamoDbStreamsEventSourcePoller poller) {
@@ -257,7 +321,7 @@ class DynamoDbStreamsEventSourcePollerTest {
while (true) {
p.pollAndInvoke(esm);
try {
- verify(streamService, atLeast(2)).getRecords("it", 10);
+ verify(reader, atLeast(2)).getRecords(any(), eq(10));
return;
} catch (AssertionError retry) {
if (System.currentTimeMillis() > deadline) {
@@ -297,31 +361,40 @@ class DynamoDbStreamsEventSourcePollerTest {
return esm;
}
- private DynamoDbStreamRecord ddbRecord(String seq, String eventName, String newImageJson) {
- DynamoDbStreamRecord rec = new DynamoDbStreamRecord();
- rec.setSequenceNumber(seq);
- rec.setEventName(eventName);
- rec.setEventSource("aws:dynamodb");
- rec.setAwsRegion("us-east-1");
- rec.setStreamViewType("NEW_AND_OLD_IMAGES");
+ /** A GetRecords {@code Records[]} element as the native engine returns it. */
+ private DynamoDbStreamReader.Record ddbRecord(String seq, String eventName, String newImageJson) {
+ ObjectNode awsRecord = OBJECT_MAPPER.createObjectNode()
+ .put("eventID", "event-" + seq)
+ .put("eventName", eventName)
+ .put("eventVersion", "1.1")
+ .put("eventSource", "aws:dynamodb")
+ .put("awsRegion", "us-east-1");
+ ObjectNode dynamodb = awsRecord.putObject("dynamodb").put("ApproximateCreationDateTime", 0L);
try {
- rec.setNewImage(OBJECT_MAPPER.readTree(newImageJson));
+ dynamodb.set("NewImage", OBJECT_MAPPER.readTree(newImageJson));
} catch (Exception e) {
throw new RuntimeException(e);
}
- return rec;
+ dynamodb.put("SequenceNumber", seq).put("SizeBytes", 100).put("StreamViewType", "NEW_AND_OLD_IMAGES");
+ return new DynamoDbStreamReader.Record(seq, awsRecord);
+ }
+
+ private static DynamoDbStreamReader.Record createdAt(DynamoDbStreamReader.Record record, long epochSeconds) {
+ ((ObjectNode) record.awsRecord().path("dynamodb")).put("ApproximateCreationDateTime", epochSeconds);
+ return record;
}
- private void stubTrimHorizon(List<DynamoDbStreamRecord> records) {
- // Match any iterator type/seq so a re-kicked poll after a checkpoint advance (AFTER_SEQUENCE_NUMBER)
- // still resolves an iterator: the second-fetch barrier depends on poll N+1 fetching.
- when(streamService.getShardIterator(eq(STREAM_ARN), eq(DynamoDbStreamService.SHARD_ID), anyString(), any()))
- .thenReturn("it");
- when(streamService.getRecords("it", 10))
- .thenReturn(new DynamoDbStreamService.GetRecordsResult(records, "it"));
+ private LambdaFunction stubFunction() {
LambdaFunction fn = new LambdaFunction();
fn.setFunctionName("fn");
when(functionStore.getForAccount(ACCOUNT_ID, "us-east-1", "fn")).thenReturn(Optional.of(fn));
+ return fn;
+ }
+
+ /** The mapping's shard holds {@code records} from its trim horizon, and its function resolves. */
+ private LambdaFunction stubTrimHorizon(List<DynamoDbStreamReader.Record> records) {
+ shard.addAll(records);
+ return stubFunction();
}
private static final String PATTERN =
@@ -386,7 +459,7 @@ class DynamoDbStreamsEventSourcePollerTest {
assertEquals(1, records.size());
assertEquals("s1", records.get(0).path("dynamodb").path("SequenceNumber").asText());
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s2", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s2", esm.getShardSequenceNumbers().get(SHARD_ID));
}
@Test
@@ -402,7 +475,7 @@ class DynamoDbStreamsEventSourcePollerTest {
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
awaitPollCompletedViaSecondFetch(p, esm);
- assertEquals("s2", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s2", esm.getShardSequenceNumbers().get(SHARD_ID));
verify(executorService, never()).invoke(any(), any(byte[].class), any());
}
@@ -423,7 +496,7 @@ class DynamoDbStreamsEventSourcePollerTest {
awaitPollCompletedViaSecondFetch(p, esm);
verify(store, never()).saveForAccount(anyString(), any());
- assertNull(esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertNull(esm.getShardSequenceNumbers().get(SHARD_ID));
// retry-from-old-checkpoint: both polls re-derive TRIM_HORIZON and deliver the same window.
ArgumentCaptor<byte[]> cap = ArgumentCaptor.forClass(byte[].class);
verify(executorService, timeout(2000).atLeast(2)).invoke(any(), cap.capture(), eq(InvocationType.RequestResponse));
@@ -446,7 +519,7 @@ class DynamoDbStreamsEventSourcePollerTest {
verify(executorService, timeout(2000)).invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse));
awaitPollCompletedViaSecondFetch(p, esm);
verify(store, never()).saveForAccount(anyString(), any());
- assertNull(esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertNull(esm.getShardSequenceNumbers().get(SHARD_ID));
}
@Test
@@ -465,7 +538,7 @@ class DynamoDbStreamsEventSourcePollerTest {
verify(executorService, timeout(2000)).invoke(any(), payload.capture(), eq(InvocationType.RequestResponse));
assertEquals(2, readRecords(payload.getValue()).size());
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s2", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s2", esm.getShardSequenceNumbers().get(SHARD_ID));
}
@Test
@@ -485,12 +558,13 @@ class DynamoDbStreamsEventSourcePollerTest {
pollerWith(store).pollAndInvoke(esm);
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s1", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s1", esm.getShardSequenceNumbers().get(SHARD_ID));
}
@Test
void partialBatchFailureAtFirstRecordKeepsPriorCheckpoint() {
stubTrimHorizon(List.of(
+ ddbRecord("s0", "INSERT", "{}"),
ddbRecord("s1", "INSERT", "{}"),
ddbRecord("s2", "INSERT", "{}")));
InvokeResult result = new InvokeResult();
@@ -500,7 +574,7 @@ class DynamoDbStreamsEventSourcePollerTest {
EsmStore store = mock(EsmStore.class);
EventSourceMapping esm = filterEsm();
esm.setFunctionResponseTypes(List.of("ReportBatchItemFailures"));
- esm.getShardSequenceNumbers().put(DynamoDbStreamService.SHARD_ID, "s0");
+ esm.getShardSequenceNumbers().put(SHARD_ID, "s0");
DynamoDbStreamsEventSourcePoller p = pollerWith(store);
p.pollAndInvoke(esm);
@@ -509,7 +583,7 @@ class DynamoDbStreamsEventSourcePollerTest {
awaitPollCompletedViaSecondFetch(p, esm);
verify(store, never()).saveForAccount(anyString(), any());
- assertEquals("s0", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s0", esm.getShardSequenceNumbers().get(SHARD_ID));
}
@Test
@@ -532,7 +606,7 @@ class DynamoDbStreamsEventSourcePollerTest {
awaitPollCompletedViaSecondFetch(p, esm);
verify(store, never()).saveForAccount(anyString(), any());
- assertNull(esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertNull(esm.getShardSequenceNumbers().get(SHARD_ID));
}
@Test
@@ -550,7 +624,7 @@ class DynamoDbStreamsEventSourcePollerTest {
pollerWith(store).pollAndInvoke(esm);
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s2", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s2", esm.getShardSequenceNumbers().get(SHARD_ID));
}
// ──────────────────── FilterCriteria + partial batch failure ────────────────────
@@ -588,9 +662,9 @@ class DynamoDbStreamsEventSourcePollerTest {
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
// s2 was never sent to the function, yet it is the record immediately before the lowest
- // failure (s3) in the fetched batch — so it is the checkpoint: s1 and s2 are consumed and
+ // failure (s3) in the fetched batch, so it is the checkpoint: s1 and s2 are consumed and
// the next poll resumes AFTER s2, re-delivering s3 and s4.
- assertEquals("s2", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s2", esm.getShardSequenceNumbers().get(SHARD_ID));
}
/**
@@ -601,6 +675,7 @@ class DynamoDbStreamsEventSourcePollerTest {
@Test
void partialFailureReportingFilteredOutRecordRetriesWholeBatch() {
stubTrimHorizon(List.of(
+ ddbRecord("s0", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
ddbRecord("s2", "INSERT", "{\"status\":{\"S\":\"inactive\"}}"), // filtered out
ddbRecord("s3", "INSERT", "{\"status\":{\"S\":\"active\"}}")));
@@ -612,7 +687,7 @@ class DynamoDbStreamsEventSourcePollerTest {
EsmStore store = mock(EsmStore.class);
EventSourceMapping esm = filterEsm(PATTERN);
esm.setFunctionResponseTypes(List.of("ReportBatchItemFailures"));
- esm.getShardSequenceNumbers().put(DynamoDbStreamService.SHARD_ID, "s0");
+ esm.getShardSequenceNumbers().put(SHARD_ID, "s0");
DynamoDbStreamsEventSourcePoller p = pollerWith(store);
p.pollAndInvoke(esm);
@@ -627,7 +702,7 @@ class DynamoDbStreamsEventSourcePollerTest {
// Had the identifier been accepted, the checkpoint would have landed on s1 (the record
// before s2). Instead the prior checkpoint is kept and the entire window is retried.
verify(store, never()).saveForAccount(anyString(), any());
- assertEquals("s0", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s0", esm.getShardSequenceNumbers().get(SHARD_ID));
}
/**
@@ -637,26 +712,14 @@ class DynamoDbStreamsEventSourcePollerTest {
*/
@Test
void trimmedCheckpointResumesFromTrimHorizonInsteadOfWedging() {
- LambdaFunction fn = new LambdaFunction();
- fn.setFunctionName("fn");
- when(functionStore.getForAccount(ACCOUNT_ID, "us-east-1", "fn")).thenReturn(Optional.of(fn));
-
- when(streamService.getShardIterator(eq(STREAM_ARN), eq(DynamoDbStreamService.SHARD_ID),
- eq("AFTER_SEQUENCE_NUMBER"), any())).thenReturn("it-trimmed");
- when(streamService.getShardIterator(eq(STREAM_ARN), eq(DynamoDbStreamService.SHARD_ID),
- eq("TRIM_HORIZON"), any())).thenReturn("it-horizon");
- when(streamService.getRecords("it-trimmed", 10)).thenThrow(
- new AwsException("TrimmedDataAccessException",
- "The requested sequence number has been trimmed", 400));
- when(streamService.getRecords("it-horizon", 10)).thenReturn(
- new DynamoDbStreamService.GetRecordsResult(
- List.of(ddbRecord("s9", "INSERT", "{\"status\":{\"S\":\"active\"}}")), "it-horizon"));
+ // The stale checkpoint is older than s9, the oldest record retained, so reading after it is trimmed.
+ stubTrimHorizon(List.of(ddbRecord("s9", "INSERT", "{\"status\":{\"S\":\"active\"}}")));
when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse)))
.thenReturn(new InvokeResult());
EsmStore store = mock(EsmStore.class);
EventSourceMapping esm = filterEsm();
- esm.getShardSequenceNumbers().put(DynamoDbStreamService.SHARD_ID, STALE_CHECKPOINT);
+ esm.getShardSequenceNumbers().put(SHARD_ID, STALE_CHECKPOINT);
pollerWith(store).pollAndInvoke(esm);
@@ -664,31 +727,26 @@ class DynamoDbStreamsEventSourcePollerTest {
verify(executorService, timeout(2000)).invoke(any(), payload.capture(), eq(InvocationType.RequestResponse));
assertEquals(1, readRecords(payload.getValue()).size());
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s9", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s9", esm.getShardSequenceNumbers().get(SHARD_ID));
+ verify(reader).getShardIterator(any(), eq(SHARD_ID), eq(Position.TRIM_HORIZON), any());
}
/** Only a trimmed checkpoint resets the cursor; any other stream failure retries the same window. */
@Test
void nonTrimmedStreamErrorLeavesTheCheckpointAlone() {
- LambdaFunction fn = new LambdaFunction();
- fn.setFunctionName("fn");
- when(functionStore.getForAccount(ACCOUNT_ID, "us-east-1", "fn")).thenReturn(Optional.of(fn));
-
- when(streamService.getShardIterator(eq(STREAM_ARN), eq(DynamoDbStreamService.SHARD_ID),
- anyString(), any())).thenReturn("it");
- when(streamService.getRecords("it", 10)).thenThrow(
- new AwsException("InternalServerError", "boom", 500));
+ stubFunction();
+ doThrow(new AwsException("InternalServerError", "boom", 500)).when(reader).getRecords(any(), anyInt());
EsmStore store = mock(EsmStore.class);
EventSourceMapping esm = filterEsm();
- esm.getShardSequenceNumbers().put(DynamoDbStreamService.SHARD_ID, STALE_CHECKPOINT);
+ esm.getShardSequenceNumbers().put(SHARD_ID, STALE_CHECKPOINT);
pollerWith(store).pollAndInvoke(esm);
awaitPollCompletedViaSecondFetch(pollerWith(store), esm);
verify(executorService, never()).invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse));
verify(store, never()).saveForAccount(anyString(), any());
- assertEquals(STALE_CHECKPOINT, esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals(STALE_CHECKPOINT, esm.getShardSequenceNumbers().get(SHARD_ID));
}
@Test
@@ -737,7 +795,7 @@ class DynamoDbStreamsEventSourcePollerTest {
// Checkpoint advanced to "s1"
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s1", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s1", esm.getShardSequenceNumbers().get(SHARD_ID));
// SQS delivery invoked
ArgumentCaptor<String> bodyCaptor = ArgumentCaptor.forClass(String.class);
@@ -750,7 +808,7 @@ class DynamoDbStreamsEventSourcePollerTest {
assertEquals(2, dlqPayload.path("requestContext").path("approximateInvokeCount").asInt());
assertEquals("req-123", dlqPayload.path("requestContext").path("requestId").asText());
assertEquals("Unhandled", dlqPayload.path("responseContext").path("functionError").asText());
- assertEquals(DynamoDbStreamService.SHARD_ID, dlqPayload.path("DDBStreamBatchInfo").path("shardId").asText());
+ assertEquals(SHARD_ID, dlqPayload.path("DDBStreamBatchInfo").path("shardId").asText());
assertEquals("s1", dlqPayload.path("DDBStreamBatchInfo").path("startSequenceNumber").asText());
assertEquals("s1", dlqPayload.path("DDBStreamBatchInfo").path("endSequenceNumber").asText());
assertEquals(1, dlqPayload.path("DDBStreamBatchInfo").path("batchSize").asInt());
@@ -847,7 +905,7 @@ class DynamoDbStreamsEventSourcePollerTest {
pollerWith(store).pollAndInvoke(esm);
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s1", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s1", esm.getShardSequenceNumbers().get(SHARD_ID));
return new S3OnFailureDelivery(s3Service, objects);
}
@@ -900,9 +958,8 @@ class DynamoDbStreamsEventSourcePollerTest {
// Chosen so the seconds/millis interpretations land 46+ years apart: a millis bug cannot
// accidentally parse back to this epoch-seconds value.
long knownEpochSeconds = 1_700_000_000L;
- DynamoDbStreamRecord rec = ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}");
- rec.setApproximateCreationDateTime(knownEpochSeconds);
- stubTrimHorizon(List.of(rec));
+ stubTrimHorizon(List.of(createdAt(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
+ knownEpochSeconds)));
InvokeResult err = new InvokeResult();
err.setFunctionError("Unhandled");
@@ -959,30 +1016,19 @@ class DynamoDbStreamsEventSourcePollerTest {
p.pollAndInvoke(esm);
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s1", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s1", esm.getShardSequenceNumbers().get(SHARD_ID));
verify(snsService, timeout(2000)).publish(eq(snsArn), any(), anyString(), eq("ESM OnFailure"), eq("us-east-1"));
}
@Test
void maxRetryAttemptsExhaustedTracksRetriesAcrossExpandingBatchWindow() throws Exception {
- when(streamService.getShardIterator(eq(STREAM_ARN), eq(DynamoDbStreamService.SHARD_ID),
- eq("TRIM_HORIZON"), any())).thenReturn("it-1", "it-2");
- when(streamService.getRecords(eq("it-1"), anyInt())).thenReturn(
- new DynamoDbStreamService.GetRecordsResult(
- List.of(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}")), "it-1"));
- when(streamService.getRecords(eq("it-2"), anyInt())).thenReturn(
- new DynamoDbStreamService.GetRecordsResult(
- List.of(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
- ddbRecord("s2", "INSERT", "{\"status\":{\"S\":\"active\"}}")), "it-2"));
+ stubTrimHorizon(List.of(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}")));
InvokeResult err = new InvokeResult();
err.setFunctionError("Unhandled");
when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse)))
.thenReturn(err);
- LambdaFunction fn = new LambdaFunction();
- fn.setFunctionName("fn");
- when(functionStore.getForAccount(ACCOUNT_ID, "us-east-1", "fn")).thenReturn(Optional.of(fn));
EsmStore store = mock(EsmStore.class);
EventSourceMapping esm = filterEsm();
@@ -1008,11 +1054,12 @@ class DynamoDbStreamsEventSourcePollerTest {
Thread.sleep(25);
}
+ shard.add(ddbRecord("s2", "INSERT", "{\"status\":{\"S\":\"active\"}}"));
advancePastRetry(p);
p.pollAndInvoke(esm);
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s2", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s2", esm.getShardSequenceNumbers().get(SHARD_ID));
ArgumentCaptor<String> bodyCaptor = ArgumentCaptor.forClass(String.class);
String expectedQueueUrl = "http://localhost:4566/000000000000/my-dlq";
@@ -1073,7 +1120,7 @@ class DynamoDbStreamsEventSourcePollerTest {
// Checkpoint advanced to "s1"
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s1", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s1", esm.getShardSequenceNumbers().get(SHARD_ID));
// SQS delivery invoked
ArgumentCaptor<String> bodyCaptor = ArgumentCaptor.forClass(String.class);
@@ -1087,7 +1134,7 @@ class DynamoDbStreamsEventSourcePollerTest {
assertEquals("req-batch-fail", dlqPayload.path("requestContext").path("requestId").asText());
assertEquals(200, dlqPayload.path("responseContext").path("statusCode").asInt());
assertFalse(dlqPayload.path("responseContext").has("functionError"));
- assertEquals(DynamoDbStreamService.SHARD_ID, dlqPayload.path("DDBStreamBatchInfo").path("shardId").asText());
+ assertEquals(SHARD_ID, dlqPayload.path("DDBStreamBatchInfo").path("shardId").asText());
assertEquals("s1", dlqPayload.path("DDBStreamBatchInfo").path("startSequenceNumber").asText());
assertEquals("s1", dlqPayload.path("DDBStreamBatchInfo").path("endSequenceNumber").asText());
assertEquals(1, dlqPayload.path("DDBStreamBatchInfo").path("batchSize").asInt());
@@ -1097,12 +1144,8 @@ class DynamoDbStreamsEventSourcePollerTest {
@Test
void reportBatchItemFailuresPartialSuccessWithZeroRetriesExhaustsImmediately() throws Exception {
- when(streamService.getShardIterator(eq(STREAM_ARN), eq(DynamoDbStreamService.SHARD_ID),
- eq("TRIM_HORIZON"), any())).thenReturn("it-1");
- when(streamService.getRecords(eq("it-1"), anyInt())).thenReturn(
- new DynamoDbStreamService.GetRecordsResult(
- List.of(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
- ddbRecord("s2", "INSERT", "{\"status\":{\"S\":\"active\"}}")), "it-1"));
+ stubTrimHorizon(List.of(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
+ ddbRecord("s2", "INSERT", "{\"status\":{\"S\":\"active\"}}")));
InvokeResult partialFailure = new InvokeResult();
partialFailure.setStatusCode(200);
@@ -1110,10 +1153,6 @@ class DynamoDbStreamsEventSourcePollerTest {
when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse)))
.thenReturn(partialFailure);
- LambdaFunction fn = new LambdaFunction();
- fn.setFunctionName("fn");
- when(functionStore.getForAccount(ACCOUNT_ID, "us-east-1", "fn")).thenReturn(Optional.of(fn));
-
EsmStore store = mock(EsmStore.class);
EventSourceMapping esm = filterEsm();
esm.setFunctionResponseTypes(List.of("ReportBatchItemFailures"));
@@ -1133,7 +1172,7 @@ class DynamoDbStreamsEventSourcePollerTest {
// Checkpoint advanced to "s2"
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s2", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s2", esm.getShardSequenceNumbers().get(SHARD_ID));
ArgumentCaptor<String> bodyCaptor = ArgumentCaptor.forClass(String.class);
String expectedQueueUrl = "http://localhost:4566/000000000000/my-dlq";
@@ -1153,17 +1192,8 @@ class DynamoDbStreamsEventSourcePollerTest {
@Test
void reportBatchItemFailuresPartialSuccessAllowsOneRetryBeforeExhaustion() throws Exception {
- when(streamService.getShardIterator(eq(STREAM_ARN), eq(DynamoDbStreamService.SHARD_ID),
- eq("TRIM_HORIZON"), any())).thenReturn("it-1");
- when(streamService.getShardIterator(eq(STREAM_ARN), eq(DynamoDbStreamService.SHARD_ID),
- eq("AFTER_SEQUENCE_NUMBER"), eq("s1"))).thenReturn("it-2");
- when(streamService.getRecords(eq("it-1"), anyInt())).thenReturn(
- new DynamoDbStreamService.GetRecordsResult(
- List.of(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
- ddbRecord("s2", "INSERT", "{\"status\":{\"S\":\"active\"}}")), "it-1"));
- when(streamService.getRecords(eq("it-2"), anyInt())).thenReturn(
- new DynamoDbStreamService.GetRecordsResult(
- List.of(ddbRecord("s2", "INSERT", "{\"status\":{\"S\":\"active\"}}")), "it-2"));
+ stubTrimHorizon(List.of(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
+ ddbRecord("s2", "INSERT", "{\"status\":{\"S\":\"active\"}}")));
InvokeResult partialFailure = new InvokeResult();
partialFailure.setStatusCode(200);
@@ -1171,10 +1201,6 @@ class DynamoDbStreamsEventSourcePollerTest {
when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse)))
.thenReturn(partialFailure);
- LambdaFunction fn = new LambdaFunction();
- fn.setFunctionName("fn");
- when(functionStore.getForAccount(ACCOUNT_ID, "us-east-1", "fn")).thenReturn(Optional.of(fn));
-
EsmStore store = mock(EsmStore.class);
EventSourceMapping esm = filterEsm();
esm.setFunctionResponseTypes(List.of("ReportBatchItemFailures"));
@@ -1192,7 +1218,7 @@ class DynamoDbStreamsEventSourcePollerTest {
// First poll: s1 succeeds, s2 fails -> checkpoint advances to s1, retry count carried over, no DLQ delivery
p.pollAndInvoke(esm);
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s1", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s1", esm.getShardSequenceNumbers().get(SHARD_ID));
verify(sqsService, never()).sendMessage(anyString(), anyString(), anyInt(), anyString());
long deadline = System.currentTimeMillis() + 3000;
@@ -1208,7 +1234,7 @@ class DynamoDbStreamsEventSourcePollerTest {
p.pollAndInvoke(esm);
verify(store, timeout(2000).times(2)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s2", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s2", esm.getShardSequenceNumbers().get(SHARD_ID));
ArgumentCaptor<String> bodyCaptor = ArgumentCaptor.forClass(String.class);
String expectedQueueUrl = "http://localhost:4566/000000000000/my-dlq";
@@ -1270,9 +1296,8 @@ class DynamoDbStreamsEventSourcePollerTest {
}
private void assertOldBatchIsRetriedWithoutAnAgeCutoff(EventSourceMapping esm) throws Exception {
- DynamoDbStreamRecord record = ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}");
- record.setApproximateCreationDateTime(clock.get() / 1_000 - 86_401);
- stubTrimHorizon(List.of(record));
+ stubTrimHorizon(List.of(createdAt(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
+ clock.get() / 1_000 - 86_401)));
InvokeResult error = new InvokeResult();
error.setFunctionError("Unhandled");
when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse)))
@@ -1301,9 +1326,8 @@ class DynamoDbStreamsEventSourcePollerTest {
@Test
void batchOlderThanMaximumRecordAgeIsDiscardedAndDeliveredToOnFailure() throws Exception {
- DynamoDbStreamRecord record = ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}");
- record.setApproximateCreationDateTime(clock.get() / 1_000 - 61);
- stubTrimHorizon(List.of(record));
+ stubTrimHorizon(List.of(createdAt(ddbRecord("s1", "INSERT", "{\"status\":{\"S\":\"active\"}}"),
+ clock.get() / 1_000 - 61)));
InvokeResult error = new InvokeResult();
error.setFunctionError("Unhandled");
when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse)))
@@ -1321,7 +1345,7 @@ class DynamoDbStreamsEventSourcePollerTest {
pollerWith(store).pollAndInvoke(esm);
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s1", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s1", esm.getShardSequenceNumbers().get(SHARD_ID));
ArgumentCaptor<String> bodyCaptor = ArgumentCaptor.forClass(String.class);
verify(sqsService, timeout(2000)).sendMessage(anyString(), bodyCaptor.capture(), anyInt(), anyString());
JsonNode payload = OBJECT_MAPPER.readTree(bodyCaptor.getValue());
@@ -1330,34 +1354,13 @@ class DynamoDbStreamsEventSourcePollerTest {
// ──────────────────── BisectBatchOnFunctionError and refused destinations ────────────────────
- /**
- * Serves the listed sequences as the stream and returns it for trimming or appending. The iterator
- * is the sequence to read after, or empty for the trim horizon; a checkpoint missing from the stream
- * behaves as trimmed.
- */
- private List<DynamoDbStreamRecord> stubStream(String... sequences) {
- List<DynamoDbStreamRecord> stream = new CopyOnWriteArrayList<>();
+ /** Serves the listed sequences as the mapping's shard and returns it for trimming or appending. */
+ private List<DynamoDbStreamReader.Record> stubStream(String... sequences) {
for (String seq : sequences) {
- stream.add(ddbRecord(seq, "INSERT", "{}"));
+ shard.add(ddbRecord(seq, "INSERT", "{}"));
}
- when(streamService.getShardIterator(eq(STREAM_ARN), eq(DynamoDbStreamService.SHARD_ID), anyString(), any()))
- .thenAnswer(inv -> "TRIM_HORIZON".equals(inv.getArgument(2)) ? "" : inv.getArgument(3));
- when(streamService.getRecords(anyString(), anyInt())).thenAnswer(inv -> {
- String after = inv.getArgument(0);
- List<DynamoDbStreamRecord> snapshot = new ArrayList<>(stream);
- int from = after.isEmpty() ? 0
- : snapshot.stream().map(DynamoDbStreamRecord::getSequenceNumber).toList().indexOf(after) + 1;
- if (!after.isEmpty() && from == 0) {
- throw new AwsException("TrimmedDataAccessException", "trimmed", 400);
- }
- int limit = inv.getArgument(1);
- return new DynamoDbStreamService.GetRecordsResult(
- new ArrayList<>(snapshot.subList(from, Math.min(from + limit, snapshot.size()))), "unused");
- });
- LambdaFunction fn = new LambdaFunction();
- fn.setFunctionName("fn");
- when(functionStore.getForAccount(ACCOUNT_ID, "us-east-1", "fn")).thenReturn(Optional.of(fn));
- return stream;
+ stubFunction();
+ return shard;
}
/** Records the sequences of every delivered batch and answers each with {@code respond}. */
@@ -1419,7 +1422,7 @@ class DynamoDbStreamsEventSourcePollerTest {
}
private String checkpoint(EventSourceMapping esm) {
- return esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID);
+ return esm.getShardSequenceNumbers().get(SHARD_ID);
}
private JsonNode batchInfo(String body) throws Exception {
@@ -1445,7 +1448,7 @@ class DynamoDbStreamsEventSourcePollerTest {
EventSourceMapping esm = esmWithDlq(0);
esm.setBisectBatchOnFunctionError(true);
if (startingCheckpoint != null) {
- esm.getShardSequenceNumbers().put(DynamoDbStreamService.SHARD_ID, startingCheckpoint);
+ esm.getShardSequenceNumbers().put(SHARD_ID, startingCheckpoint);
}
DynamoDbStreamsEventSourcePoller p = pollerWith(mock(EsmStore.class));
@@ -1491,8 +1494,8 @@ class DynamoDbStreamsEventSourcePollerTest {
@Test
void halvedWindowFullyFilteredOutDoesNotLeakItsLimitIntoALaterHorizonRefetch() throws Exception {
- List<DynamoDbStreamRecord> stream = stubStream("s1", "s2", "s3", "s4", "s5", "s6", "s7", "s8");
- stream.subList(0, 4).forEach(r -> r.setEventName("MODIFY"));
+ List<DynamoDbStreamReader.Record> stream = stubStream("s1", "s2", "s3", "s4", "s5", "s6", "s7", "s8");
+ stream.subList(0, 4).forEach(r -> ((ObjectNode) r.awsRecord()).put("eventName", "MODIFY"));
List<List<String>> invocations = failInvocationsContaining("s5");
EventSourceMapping esm = filterEsm("{\"eventName\":[\"INSERT\"]}");
esm.setBisectBatchOnFunctionError(true);
@@ -1501,7 +1504,7 @@ class DynamoDbStreamsEventSourcePollerTest {
pollOnce(p, esm);
pollOnce(p, esm);
assertEquals("s4", checkpoint(esm), "the halved window [s1..s4] is fully filtered out");
- stream.removeIf(r -> "MODIFY".equals(r.getEventName()));
+ stream.removeIf(r -> "MODIFY".equals(r.awsRecord().path("eventName").asText()));
stream.add(ddbRecord("s9", "INSERT", "{}"));
pollOnce(p, esm);
@@ -1615,7 +1618,7 @@ class DynamoDbStreamsEventSourcePollerTest {
@Test
void failedOnFailureDeliveryAllowsNextRecordsToBeProcessed() throws Exception {
- List<DynamoDbStreamRecord> stream = stubStream("s1");
+ List<DynamoDbStreamReader.Record> stream = stubStream("s1");
List<List<String>> invocations = failInvocationsContaining("s1");
List<String> delivered = refuseSqsSends(1);
EventSourceMapping esm = esmWithDlq(0);
@@ -1688,26 +1691,12 @@ class DynamoDbStreamsEventSourcePollerTest {
// ──────────────────── StartingPosition and mapping lifecycle (#4311) ────────────────────
- private static final String TABLE_ARN = "arn:aws:dynamodb:us-east-1:000000000000:table/t";
-
- private DynamoDbStreamService realStream() {
- DynamoDbStreamService streams = new DynamoDbStreamService(OBJECT_MAPPER, mock(StorageFactory.class));
- streams.enableStream("t", TABLE_ARN, "NEW_AND_OLD_IMAGES", "us-east-1", STREAM_ARN);
- return streams;
- }
-
- private void write(DynamoDbStreamService streams, String pk) {
- TableDefinition table = new TableDefinition("t", List.of(new KeySchemaElement("pk", "HASH")),
- List.of(new AttributeDefinition("pk", "S")), "us-east-1", ACCOUNT_ID);
- ObjectNode item = OBJECT_MAPPER.createObjectNode();
- item.putObject("pk").put("S", pk);
- streams.captureEvent("t", "INSERT", null, item, table, "us-east-1");
- }
-
- private DynamoDbStreamsEventSourcePoller pollerOver(DynamoDbStreamService streams) {
- return new DynamoDbStreamsEventSourcePoller(
- mock(Vertx.class), streams, executorService, new LambdaTargetResolver(functionStore, aliasStore),
- esmStore, OBJECT_MAPPER, config, filterMatcher, sqsService, snsService, s3Service, clock::get);
+ /** Appends an INSERT of item {@code pk} to the mapping's shard, numbered like native sequence numbers. */
+ private void write(String pk) {
+ DynamoDbStreamReader.Record record =
+ ddbRecord(String.format("%021d", sequence.incrementAndGet()), "INSERT", "{}");
+ ((ObjectNode) record.awsRecord().path("dynamodb")).putObject("Keys").putObject("pk").put("S", pk);
+ shard.add(record);
}
private EventSourceMapping streamsEsm(String uuid, String startingPosition) {
@@ -1724,9 +1713,7 @@ class DynamoDbStreamsEventSourcePollerTest {
}
private void stubFunctionSucceeds() {
- LambdaFunction fn = new LambdaFunction();
- fn.setFunctionName("fn");
- when(functionStore.getForAccount(ACCOUNT_ID, "us-east-1", "fn")).thenReturn(Optional.of(fn));
+ stubFunction();
when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse)))
.thenReturn(new InvokeResult());
}
@@ -1741,12 +1728,12 @@ class DynamoDbStreamsEventSourcePollerTest {
/** One-record read that signals {@code reading} and then holds until {@code release} opens. */
private void stubBlockingRead(CountDownLatch reading, CountDownLatch release) {
- stubTrimHorizon(List.of());
- when(streamService.getRecords("it", 10)).thenAnswer(inv -> {
+ stubStream("s1");
+ doAnswer(inv -> {
reading.countDown();
release.await(5, TimeUnit.SECONDS);
- return new DynamoDbStreamService.GetRecordsResult(List.of(ddbRecord("s1", "INSERT", "{}")), "it");
- });
+ return inv.callRealMethod();
+ }).when(reader).getRecords(any(), anyInt());
}
/** Invocation that signals {@code invoking} and then holds until {@code release} opens. */
@@ -1761,17 +1748,15 @@ class DynamoDbStreamsEventSourcePollerTest {
@Test
void latestMappingDeliversOnlyRecordsWrittenAfterItWasCreated() {
- DynamoDbStreamService streams = realStream();
- write(streams, "old-1");
- write(streams, "old-2");
+ write("old-1");
+ write("old-2");
stubFunctionSucceeds();
- DynamoDbStreamsEventSourcePoller p = pollerOver(streams);
EventSourceMapping esm = streamsEsm("esm-latest", "LATEST");
- p.initializeStartingPosition(esm);
+ poller.initializeStartingPosition(esm);
esmStore.saveForAccount(ACCOUNT_ID, esm);
- write(streams, "new");
- p.pollAndInvoke(esm);
+ write("new");
+ poller.pollAndInvoke(esm);
ArgumentCaptor<byte[]> payload = ArgumentCaptor.forClass(byte[].class);
verify(executorService, timeout(2000)).invoke(any(), payload.capture(), eq(InvocationType.RequestResponse));
@@ -1780,21 +1765,19 @@ class DynamoDbStreamsEventSourcePollerTest {
@Test
void latestMappingOnAnEmptyStreamDeliversEveryLaterRecord() throws Exception {
- DynamoDbStreamService streams = realStream();
stubFunctionSucceeds();
- DynamoDbStreamsEventSourcePoller p = pollerOver(streams);
EventSourceMapping esm = streamsEsm("esm-latest", "LATEST");
- p.initializeStartingPosition(esm);
+ poller.initializeStartingPosition(esm);
esmStore.saveForAccount(ACCOUNT_ID, esm);
assertTrue(esm.getShardSequenceNumbers().isEmpty(), "an empty stream has no record to start after");
- p.pollAndInvoke(esm);
- awaitPollCompleted(p);
+ poller.pollAndInvoke(esm);
+ awaitPollCompleted(poller);
assertTrue(esm.getShardSequenceNumbers().isEmpty(), "an empty poll must not pin the cursor");
verify(executorService, never()).invoke(any(), any(byte[].class), any());
- write(streams, "first");
- p.pollAndInvoke(esm);
+ write("first");
+ poller.pollAndInvoke(esm);
ArgumentCaptor<byte[]> payload = ArgumentCaptor.forClass(byte[].class);
verify(executorService, timeout(2000)).invoke(any(), payload.capture(), eq(InvocationType.RequestResponse));
@@ -1803,37 +1786,34 @@ class DynamoDbStreamsEventSourcePollerTest {
@Test
void trimHorizonMappingIsNotPinnedAtCreation() {
- DynamoDbStreamService streams = realStream();
- write(streams, "old");
+ write("old");
EventSourceMapping esm = streamsEsm("esm-trim", "TRIM_HORIZON");
- pollerOver(streams).initializeStartingPosition(esm);
+ poller.initializeStartingPosition(esm);
assertTrue(esm.getShardSequenceNumbers().isEmpty());
}
@Test
void reEnabledMappingResumesAfterItsLastProcessedRecord() throws Exception {
- DynamoDbStreamService streams = realStream();
- write(streams, "old");
+ write("old");
stubFunctionSucceeds();
- DynamoDbStreamsEventSourcePoller p = pollerOver(streams);
EventSourceMapping esm = streamsEsm("esm-latest", "LATEST");
- p.initializeStartingPosition(esm);
+ poller.initializeStartingPosition(esm);
esmStore.saveForAccount(ACCOUNT_ID, esm);
- write(streams, "a");
- p.pollAndInvoke(esm);
+ write("a");
+ poller.pollAndInvoke(esm);
verify(executorService, timeout(2000)).invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse));
- awaitPollCompleted(p);
+ awaitPollCompleted(poller);
esm.setEnabled(false);
esmStore.saveForAccount(ACCOUNT_ID, esm);
- p.stopPolling(esm.getUuid());
- write(streams, "b");
+ poller.stopPolling(esm.getUuid());
+ write("b");
esm.setEnabled(true);
esmStore.saveForAccount(ACCOUNT_ID, esm);
- p.startPolling(esm);
- p.pollAndInvoke(esm);
+ poller.startPolling(esm);
+ poller.pollAndInvoke(esm);
ArgumentCaptor<byte[]> payload = ArgumentCaptor.forClass(byte[].class);
verify(executorService, timeout(2000).times(2))
@@ -1944,7 +1924,7 @@ class DynamoDbStreamsEventSourcePollerTest {
assertEquals(List.of(List.of("s1", "s2", "s3", "s4")), invocations);
// A recorded bisect would halve the next fetch; both fetches must still use the full batch size.
- verify(streamService, times(2)).getRecords(anyString(), eq(10));
+ verify(reader, times(2)).getRecords(any(), eq(10));
assertNull(checkpoint(esm));
verify(sqsService, never()).sendMessage(anyString(), anyString(), anyInt(), anyString());
}
@@ -2027,7 +2007,7 @@ class DynamoDbStreamsEventSourcePollerTest {
p.pollAndInvoke(esm);
verify(store, timeout(2000)).saveForAccount(eq(ACCOUNT_ID), any());
- assertEquals("s1", esm.getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ assertEquals("s1", esm.getShardSequenceNumbers().get(SHARD_ID));
}
@Test
@@ -2063,6 +2043,132 @@ class DynamoDbStreamsEventSourcePollerTest {
"a mapping wiped by a reset must not be written back by its in-flight poll");
}
+ // ──────────────────── Shards and the Lambda event ────────────────────
+
+ /** A closed parent shard holding p1 and its child holding c1. */
+ private FakeStreamReader parentAndChild() {
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
+ streams.shard("parent", null).add(ddbRecord("p1", "INSERT", "{}"));
+ streams.close("parent");
+ streams.shard("child", "parent").add(ddbRecord("c1", "INSERT", "{}"));
+ return spy(streams);
+ }
+
+ @Test
+ void childShardIsReadOnlyAfterItsParentWasReadToItsEnd() throws Exception {
+ stubFunction();
+ List<List<String>> invocations = recordInvocations(seqs -> new InvokeResult());
+ EventSourceMapping esm = filterEsm();
+ DynamoDbStreamsEventSourcePoller p = pollerOver(parentAndChild(), esmStore);
+
+ pollOnce(p, esm);
+ pollOnce(p, esm);
+ assertEquals(List.of(List.of("p1")), invocations,
+ "the parent stays open to the mapping until a read finds no records and no next cursor");
+ pollOnce(p, esm);
+
+ assertEquals(List.of(List.of("p1"), List.of("c1")), invocations);
+ assertEquals(Map.of("parent", "p1", "child", "c1"), esm.getShardSequenceNumbers());
+ }
+
+ @Test
+ void stopPollingForgetsWhichShardsTheMappingFinished() throws Exception {
+ stubFunction();
+ recordInvocations(seqs -> new InvokeResult());
+ EventSourceMapping esm = filterEsm();
+ FakeStreamReader streams = parentAndChild();
+ DynamoDbStreamsEventSourcePoller p = pollerOver(streams, esmStore);
+ pollOnce(p, esm);
+ pollOnce(p, esm);
+
+ p.stopPolling(esm.getUuid());
+ p.startPolling(esm);
+ pollOnce(p, esm);
+
+ verify(streams, times(3)).getShardIterator(any(), eq("parent"), any(), any());
+ verify(streams, never()).getShardIterator(any(), eq("child"), any(), any());
+ }
+
+ @Test
+ void latestMappingPinsEveryShardAfterItsNewestRecord() {
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
+ streams.shard("a", null).addAll(List.of(ddbRecord("a1", "INSERT", "{}"), ddbRecord("a2", "INSERT", "{}")));
+ streams.shard("b", null).add(ddbRecord("b1", "INSERT", "{}"));
+ streams.shard("empty", null);
+ EventSourceMapping esm = streamsEsm("esm-latest", "LATEST");
+
+ pollerOver(streams, esmStore).initializeStartingPosition(esm);
+
+ assertEquals(Map.of("a", "a2", "b", "b1"), esm.getShardSequenceNumbers());
+ }
+
+ @Test
+ void latestMappingOnAnUnknownStreamIsNotPinned() {
+ doThrow(new AwsException("ResourceNotFoundException", "Stream not found", 400))
+ .when(reader).describeStream(any(), any(), any());
+ EventSourceMapping esm = streamsEsm("esm-latest", "LATEST");
+
+ poller.initializeStartingPosition(esm);
+
+ assertTrue(esm.getShardSequenceNumbers().isEmpty(),
+ "a stream that does not exist yet delivers everything written to it once it does");
+ }
+
+ @Test
+ void lambdaEventIsACopyOfTheAwsRecordWithTheEventSourceArn() throws Exception {
+ JsonNode awsRecord = OBJECT_MAPPER.readTree("""
+ {"eventID": "e-1", "eventName": "MODIFY", "eventVersion": "1.1", "eventSource": "aws:dynamodb",
+ "awsRegion": "us-east-1", "dynamodb": {"ApproximateCreationDateTime": 1700000000,
+ "Keys": {"pk": {"S": "k"}}, "NewImage": {"pk": {"S": "k"}, "n": {"N": "2"}},
+ "OldImage": {"pk": {"S": "k"}, "n": {"N": "1"}}, "SequenceNumber": "s1", "SizeBytes": 42,
+ "StreamViewType": "NEW_AND_OLD_IMAGES"}}""");
+ JsonNode original = awsRecord.deepCopy();
+ stubTrimHorizon(List.of(new DynamoDbStreamReader.Record("s1", awsRecord)));
+ when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse)))
+ .thenReturn(new InvokeResult());
+
+ pollOnce(poller, filterEsm());
+
+ ArgumentCaptor<byte[]> payload = ArgumentCaptor.forClass(byte[].class);
+ verify(executorService).invoke(any(), payload.capture(), eq(InvocationType.RequestResponse));
+ ObjectNode expected = original.deepCopy();
+ expected.put("eventSourceARN", STREAM_ARN);
+ ((ObjectNode) expected.get("dynamodb")).put("ApproximateCreationDateTime", 1_700_000_000d);
+ JsonNode delivered = readRecords(payload.getValue()).get(0);
+ assertEquals(expected, delivered);
+ assertTrue(delivered.path("dynamodb").path("ApproximateCreationDateTime").isDouble());
+ assertEquals(original, awsRecord, "the reader's record is never mutated");
+ }
+
+ @Test
+ void missingFunctionSkipsTheTickBeforeReadingTheStream() throws Exception {
+ shard.add(ddbRecord("s1", "INSERT", "{}"));
+ EventSourceMapping esm = filterEsm();
+
+ pollOnce(poller, esm);
+
+ verify(reader, never()).describeStream(any(), any(), any());
+ verifyNoInteractions(executorService);
+ assertNull(checkpoint(esm));
+ }
+
+ @Test
+ void batchSizeAboveTheReadMaximumIsDeliveredInOneInvocation() throws Exception {
+ List<String> sequences = new ArrayList<>();
+ for (int i = 1; i <= 1500; i++) {
+ sequences.add(String.format("s%04d", i));
+ }
+ stubStream(sequences.toArray(String[]::new));
+ List<List<String>> invocations = recordInvocations(seqs -> new InvokeResult());
+ EventSourceMapping esm = filterEsm();
+ esm.setBatchSize(1500);
+
+ pollOnce(pollerWith(mock(EsmStore.class)), esm);
+
+ assertEquals(List.of(sequences), invocations);
+ assertEquals("s1500", checkpoint(esm));
+ }
+
private void awaitPollCompleted(DynamoDbStreamsEventSourcePoller poller) throws InterruptedException {
long deadline = System.currentTimeMillis() + 3_000;
while (!poller.activePolls.isEmpty() && System.currentTimeMillis() < deadline) {
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/EsmIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/EsmIntegrationTest.java
index 50de9f1e13..207f5b6744 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambda/EsmIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/EsmIntegrationTest.java
@@ -1,6 +1,5 @@
package io.github.hectorvent.floci.services.lambda;
-import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
import io.quarkus.test.junit.QuarkusTest;
import io.restassured.path.json.JsonPath;
import io.restassured.path.json.config.JsonPathConfig;
@@ -283,9 +282,10 @@ class EsmIntegrationTest {
{"TableName": "%s", "Item": {"pk": {"S": "%s"}}}
""".formatted(tableName, pk));
}
+ String shardId = "shardId-0000000001-00000000001";
String iterator = dynamoDb("DynamoDBStreams_20120810.GetShardIterator", """
{"StreamArn": "%s", "ShardId": "%s", "ShardIteratorType": "TRIM_HORIZON"}
- """.formatted(streamArn, DynamoDbStreamService.SHARD_ID)).getString("ShardIterator");
+ """.formatted(streamArn, shardId)).getString("ShardIterator");
String newestSequence = dynamoDb("DynamoDBStreams_20120810.GetRecords", """
{"ShardIterator": "%s"}
""".formatted(iterator)).getString("Records[-1].dynamodb.SequenceNumber");
@@ -309,7 +309,7 @@ class EsmIntegrationTest {
.path("UUID");
assertEquals(newestSequence, esmStore.getForAccount(ACCOUNT_ID, uuid).orElseThrow()
- .getShardSequenceNumbers().get(DynamoDbStreamService.SHARD_ID));
+ .getShardSequenceNumbers().get(shardId));
given().delete(LAMBDA_BASE + "/event-source-mappings/" + uuid).then().statusCode(202);
dynamoDb("DynamoDB_20120810.DeleteTable", "{\"TableName\": \"%s\"}".formatted(tableName));
diff --git a/src/test/java/io/github/hectorvent/floci/services/pipes/PipesPollerTest.java b/src/test/java/io/github/hectorvent/floci/services/pipes/PipesPollerTest.java
index 700b3b6999..2416074f20 100644
--- a/src/test/java/io/github/hectorvent/floci/services/pipes/PipesPollerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/pipes/PipesPollerTest.java
@@ -2,12 +2,19 @@ package io.github.hectorvent.floci.services.pipes;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
import io.github.hectorvent.floci.config.EmulatorConfig;
-import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader;
import io.github.hectorvent.floci.services.kinesis.KinesisService;
+import io.github.hectorvent.floci.services.pipes.model.DesiredState;
import io.github.hectorvent.floci.services.pipes.model.Pipe;
import io.github.hectorvent.floci.services.sqs.SqsService;
import io.github.hectorvent.floci.services.sqs.model.Message;
+import io.vertx.core.Handler;
import io.vertx.core.Vertx;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -17,20 +24,36 @@ import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.util.ArrayList;
import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyList;
import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.contains;
import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.lenient;
+import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@@ -38,25 +61,74 @@ import static org.mockito.Mockito.when;
class PipesPollerTest {
private static final ObjectMapper MAPPER = new ObjectMapper();
+ private static final String STREAM_ARN =
+ "arn:aws:dynamodb:us-east-1:000000000000:table/orders/stream/2026-09-01T00:00:00.000";
+ private static final String SHARD_ID = "shardId-0000000001-00000000001";
+ private static final String DLQ_ARN = "arn:aws:sqs:us-east-1:000000000000:pipe-dlq";
@Mock private Vertx vertx;
@Mock private SqsService sqsService;
@Mock private KinesisService kinesisService;
- @Mock private DynamoDbStreamService dynamoDbStreamService;
@Mock private PipesKafkaConsumerManager kafkaConsumerManager;
@Mock private PipesTargetInvoker targetInvoker;
@Mock private EmulatorConfig config;
+ private FakeStreamReader streams;
private PipesPoller poller;
@BeforeEach
void setUp() {
when(config.effectiveBaseUrl()).thenReturn("http://localhost:4566");
lenient().when(kafkaConsumerManager.resolveBatchSize(any(), anyInt())).thenReturn(10);
- poller = new PipesPoller(vertx, sqsService, kinesisService, dynamoDbStreamService,
+ streams = new FakeStreamReader();
+ poller = pollerOver(streams);
+ }
+
+ private PipesPoller pollerOver(DynamoDbStreamReader streamReader) {
+ return new PipesPoller(vertx, sqsService, kinesisService, streamReader,
kafkaConsumerManager, targetInvoker, new PipesFilterMatcher(MAPPER), MAPPER, config);
}
+ /**
+ * Serves one open shard from memory the way the native engine does. A cursor is the sequence to
+ * read after, or empty for the trim horizon, and a sequence older than the oldest retained record
+ * is trimmed.
+ */
+ static class FakeStreamReader implements DynamoDbStreamReader {
+
+ final List<Record> shard = new CopyOnWriteArrayList<>();
+
+ @Override
+ public ShardsPage describeStream(Stream stream, String exclusiveStartShardId, Integer limit) {
+ return new ShardsPage(stream, List.of(new Shard(SHARD_ID, null, null, null)), null);
+ }
+
+ @Override
+ public Cursor getShardIterator(Stream stream, String shardId, Position position, String sequenceNumber) {
+ return new Cursor(stream, shardId, position == Position.TRIM_HORIZON ? "" : sequenceNumber);
+ }
+
+ @Override
+ public RecordsPage getRecords(Cursor cursor, int limit) {
+ String after = cursor.token();
+ if (!after.isEmpty() && !shard.isEmpty() && after.compareTo(shard.get(0).sequenceNumber()) < 0) {
+ throw new AwsException("TrimmedDataAccessException",
+ "The requested sequence number has been trimmed", 400);
+ }
+ List<Record> page = shard.stream()
+ .filter(record -> record.sequenceNumber().compareTo(after) > 0)
+ .limit(limit)
+ .toList();
+ return new RecordsPage(page, page.isEmpty() ? cursor
+ : new Cursor(cursor.stream(), cursor.shardId(), page.get(page.size() - 1).sequenceNumber()));
+ }
+
+ @Override
+ public CheckpointLifetime checkpointLifetime() {
+ return CheckpointLifetime.PROCESS;
+ }
+ }
+
@Test
void asEventArrayWrapsSingleObjectInBatchArray() throws Exception {
// Pipes delivers events to a target as a batch array; a single-object enrichment response
@@ -275,6 +347,306 @@ class PipesPollerTest {
assertEquals(2L, singleOffset(offsetsCaptor.getValue(), "orders", 0));
}
+ // ──────────────────────────── DynamoDB Streams ────────────────────────────
+
+ @Test
+ void targetFailureWithoutDlqRereadsTheSameDynamoDbRecordsNextPoll() throws Exception {
+ Pipe pipe = dynamoDbPipe("{\"StartingPosition\":\"TRIM_HORIZON\"}");
+ streams.shard.add(ddbRecord("001", "active"));
+ doThrow(new RuntimeException("target down")).doNothing()
+ .when(targetInvoker).invoke(eq(pipe), anyString(), eq("us-east-1"));
+
+ poller.startPolling(pipe);
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ assertEquals(List.of(List.of("001"), List.of("001")), deliveredSequences(pipe));
+ }
+
+ @Test
+ void dynamoDbBatchSentToTheDeadLetterQueueIsCommitted() throws Exception {
+ Pipe pipe = dynamoDbPipe("{\"DeadLetterConfig\":{\"Arn\":\"" + DLQ_ARN + "\"}}");
+ streams.shard.add(ddbRecord("001", "active"));
+ doThrow(new RuntimeException("target down"))
+ .when(targetInvoker).invoke(eq(pipe), anyString(), eq("us-east-1"));
+
+ poller.startPolling(pipe);
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ streams.shard.add(ddbRecord("002", "active"));
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ assertEquals(List.of(List.of("001"), List.of("002")), deliveredSequences(pipe));
+ verify(sqsService, times(2)).sendMessage(contains("pipe-dlq"), anyString(), eq(0), eq("us-east-1"));
+ }
+
+ @Test
+ void dynamoDbBatchTheDeadLetterQueueRejectsIsRereadNextPoll() throws Exception {
+ Pipe pipe = dynamoDbPipe("{\"DeadLetterConfig\":{\"Arn\":\"" + DLQ_ARN + "\"}}");
+ streams.shard.add(ddbRecord("001", "active"));
+ doThrow(new RuntimeException("target down"))
+ .when(targetInvoker).invoke(eq(pipe), anyString(), eq("us-east-1"));
+ doThrow(new RuntimeException("dlq down"))
+ .when(sqsService).sendMessage(anyString(), anyString(), anyInt(), anyString());
+
+ poller.startPolling(pipe);
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ assertEquals(List.of(List.of("001"), List.of("001")), deliveredSequences(pipe));
+ }
+
+ @Test
+ void laterDynamoDbRecordsWaitBehindAnUndeliveredBatch() throws Exception {
+ Pipe pipe = dynamoDbPipe("{\"BatchSize\":1}");
+ streams.shard.add(ddbRecord("001", "active"));
+ streams.shard.add(ddbRecord("002", "active"));
+ doThrow(new RuntimeException("target down")).doNothing()
+ .when(targetInvoker).invoke(eq(pipe), anyString(), eq("us-east-1"));
+
+ poller.startPolling(pipe);
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ assertEquals(List.of(List.of("001"), List.of("001"), List.of("002")), deliveredSequences(pipe));
+ }
+
+ @Test
+ void latestStartingPositionDeliversOnlyDynamoDbRecordsWrittenAfterTheFirstPoll() throws Exception {
+ Pipe pipe = dynamoDbPipe("{\"StartingPosition\":\"LATEST\"}");
+ streams.shard.add(ddbRecord("001", "active"));
+
+ poller.startPolling(pipe);
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ streams.shard.add(ddbRecord("002", "active"));
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ assertEquals(List.of(List.of("002")), deliveredSequences(pipe));
+ }
+
+ @Test
+ void stoppedAndRestartedLatestPipeResumesAfterItsLastDeliveredRecord() throws Exception {
+ Pipe pipe = dynamoDbPipe("{\"StartingPosition\":\"LATEST\"}");
+ streams.shard.add(ddbRecord("000", "active"));
+ poller.startPolling(pipe);
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ streams.shard.add(ddbRecord("001", "active"));
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ poller.stopPolling(pipe);
+ streams.shard.add(ddbRecord("002", "active"));
+ poller.startPolling(pipe);
+ streams.shard.add(ddbRecord("003", "active"));
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ assertEquals(List.of(List.of("001"), List.of("002", "003")), deliveredSequences(pipe));
+ }
+
+ @Test
+ void deletedPipeForgetsItsProgress() throws Exception {
+ PipesService pipes = pipesOverPoller();
+ Pipe pipe = createDynamoDbPipe(pipes);
+ streams.shard.add(ddbRecord("001", "active"));
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ pipes.deletePipe(pipe.getName(), "us-east-1");
+ Pipe recreated = createDynamoDbPipe(pipes);
+ poller.pollDynamoDbStreams(recreated, "us-east-1");
+
+ assertEquals(List.of(List.of("001")), deliveredSequences(recreated));
+ }
+
+ @Test
+ void aPollThatOutlivesItsDeletedPipeLeavesTheRecreatedPipeAlone() throws Exception {
+ Pipe pipe = dynamoDbPipe("{\"StartingPosition\":\"TRIM_HORIZON\"}");
+ pipe.setCreationTime(Instant.parse("2026-09-01T00:00:00Z"));
+ streams.shard.add(ddbRecord("001", "active"));
+ CountDownLatch delivering = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+ doAnswer(invocation -> {
+ delivering.countDown();
+ release.await(5, TimeUnit.SECONDS);
+ return null;
+ }).when(targetInvoker).invoke(any(), anyString(), eq("us-east-1"));
+ poller.startPolling(pipe);
+ CompletableFuture<Void> inFlight =
+ CompletableFuture.runAsync(() -> poller.pollDynamoDbStreams(pipe, "us-east-1"));
+ assertTrue(delivering.await(5, TimeUnit.SECONDS));
+
+ poller.stopPolling(pipe);
+ poller.forget(pipe);
+ release.countDown();
+ inFlight.get(5, TimeUnit.SECONDS);
+ Pipe recreated = dynamoDbPipe("{\"StartingPosition\":\"TRIM_HORIZON\"}");
+ recreated.setCreationTime(Instant.parse("2026-09-02T00:00:00Z"));
+ poller.startPolling(recreated);
+ // A tick the delete raced: it starts once the pipe is recreated under the same ARN.
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ poller.pollDynamoDbStreams(recreated, "us-east-1");
+
+ assertEquals(List.of(List.of("001")), deliveredSequences(recreated));
+ assertEquals(List.of(List.of("001")), deliveredSequences(pipe));
+ }
+
+ @Test
+ void latestPinFailureDoesNotFailStartPolling() throws Exception {
+ AtomicBoolean readerDown = new AtomicBoolean(true);
+ FakeStreamReader flaky = new FakeStreamReader() {
+ @Override
+ public Map<String, String> newestSequenceNumbers(Stream stream) {
+ if (readerDown.get()) {
+ throw new IllegalStateException("engine unreachable");
+ }
+ return super.newestSequenceNumbers(stream);
+ }
+ };
+ PipesPoller flakyPoller = pollerOver(flaky);
+ Pipe pipe = dynamoDbPipe("{\"StartingPosition\":\"LATEST\"}");
+ flaky.shard.add(ddbRecord("001", "active"));
+
+ flakyPoller.startPolling(pipe);
+ assertThrows(IllegalStateException.class, () -> flakyPoller.pollDynamoDbStreams(pipe, "us-east-1"));
+ readerDown.set(false);
+ flakyPoller.pollDynamoDbStreams(pipe, "us-east-1");
+ flaky.shard.add(ddbRecord("002", "active"));
+ flakyPoller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ assertEquals(List.of(List.of("002")), deliveredSequences(pipe));
+ }
+
+ @Test
+ void dynamoDbRecordsAreFilteredAndDeliveredInTheAwsStreamRecordShape() throws Exception {
+ Pipe pipe = dynamoDbPipe("{}");
+ pipe.setSourceParameters(MAPPER.readTree("""
+ {"DynamoDBStreamParameters":{"StartingPosition":"TRIM_HORIZON"},
+ "FilterCriteria":{"Filters":[
+ {"Pattern":"{\\"dynamodb\\":{\\"NewImage\\":{\\"status\\":{\\"S\\":[\\"active\\"]}}}}"}]}}"""));
+ DynamoDbStreamReader.Record active = ddbRecord("001", "active");
+ JsonNode original = active.awsRecord().deepCopy();
+ streams.shard.add(active);
+ streams.shard.add(ddbRecord("002", "inactive"));
+
+ poller.startPolling(pipe);
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ ArgumentCaptor<String> payload = ArgumentCaptor.forClass(String.class);
+ verify(targetInvoker).invoke(eq(pipe), payload.capture(), eq("us-east-1"));
+ ObjectNode expected = original.deepCopy();
+ expected.put("eventSourceARN", STREAM_ARN);
+ ObjectNode event = MAPPER.createObjectNode();
+ event.putArray("Records").add(expected);
+ assertEquals(event, MAPPER.readTree(payload.getValue()));
+ assertEquals(original, active.awsRecord(), "the reader's record is never mutated");
+ }
+
+ @Test
+ void trimmedDynamoDbCheckpointRestartsAtTheTrimHorizon() throws Exception {
+ Pipe pipe = dynamoDbPipe("{}");
+ streams.shard.add(ddbRecord("001", "active"));
+ poller.startPolling(pipe);
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ streams.shard.clear();
+ streams.shard.add(ddbRecord("003", "active"));
+
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+ poller.pollDynamoDbStreams(pipe, "us-east-1");
+
+ assertEquals(List.of(List.of("001"), List.of("003")), deliveredSequences(pipe));
+ }
+
+ @Test
+ void pollTickReturnsAtOnceAndReadsTheStreamOnAPipesPollerThread() throws Exception {
+ CountDownLatch reading = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+ AtomicReference<String> readThread = new AtomicReference<>();
+ AtomicBoolean readReturned = new AtomicBoolean();
+ PipesPoller blockedPoller = pollerOver(new FakeStreamReader() {
+ @Override
+ public ShardsPage describeStream(Stream stream, String exclusiveStartShardId, Integer limit) {
+ readThread.set(Thread.currentThread().getName());
+ reading.countDown();
+ try {
+ release.await(5, TimeUnit.SECONDS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ readReturned.set(true);
+ return super.describeStream(stream, exclusiveStartShardId, limit);
+ }
+ });
+ Pipe pipe = dynamoDbPipe("{}");
+ blockedPoller.startPolling(pipe);
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<Handler<Long>> tick = ArgumentCaptor.forClass(Handler.class);
+ verify(vertx).setPeriodic(eq(1000L), tick.capture());
+
+ try {
+ tick.getValue().handle(1L);
+
+ assertFalse(readReturned.get(), "the tick returned while the read is still blocked");
+ assertTrue(reading.await(5, TimeUnit.SECONDS), "the tick hands the read to the poll executor");
+ assertEquals("pipes-poller", readThread.get());
+ } finally {
+ release.countDown();
+ }
+ }
+
+ private Pipe dynamoDbPipe(String dynamoDbStreamParameters) throws Exception {
+ Pipe pipe = new Pipe();
+ pipe.setName("orders-stream-pipe");
+ pipe.setArn("arn:aws:pipes:us-east-1:000000000000:pipe/orders-stream-pipe");
+ pipe.setSource(STREAM_ARN);
+ pipe.setTarget("arn:aws:lambda:us-east-1:000000000000:function:orders-target");
+ pipe.setDesiredState(DesiredState.RUNNING);
+ pipe.setSourceParameters(MAPPER.readTree("{\"DynamoDBStreamParameters\":" + dynamoDbStreamParameters + "}"));
+ return pipe;
+ }
+
+ private PipesService pipesOverPoller() {
+ StorageFactory storageFactory = mock(StorageFactory.class);
+ doReturn(AccountAwareStorageBackend.inMemory("000000000000"))
+ .when(storageFactory).create(anyString(), anyString(), any());
+ return new PipesService(storageFactory, new RegionResolver("us-east-1", "000000000000"), poller);
+ }
+
+ private Pipe createDynamoDbPipe(PipesService pipes) throws Exception {
+ return pipes.createPipe("orders-stream-pipe", STREAM_ARN,
+ "arn:aws:lambda:us-east-1:000000000000:function:orders-target",
+ "arn:aws:iam::000000000000:role/pipe-role", null, null, null,
+ MAPPER.readTree("{\"DynamoDBStreamParameters\":{\"StartingPosition\":\"TRIM_HORIZON\"}}"),
+ null, null, null, "us-east-1");
+ }
+
+ /** A GetRecords {@code Records[]} element as the native engine returns it. */
+ private static DynamoDbStreamReader.Record ddbRecord(String sequenceNumber, String status) {
+ ObjectNode awsRecord = MAPPER.createObjectNode()
+ .put("eventID", "event-" + sequenceNumber)
+ .put("eventName", "INSERT")
+ .put("eventVersion", "1.1")
+ .put("eventSource", "aws:dynamodb")
+ .put("awsRegion", "us-east-1");
+ ObjectNode dynamodb = awsRecord.putObject("dynamodb");
+ dynamodb.putObject("NewImage").putObject("status").put("S", status);
+ dynamodb.put("SequenceNumber", sequenceNumber).put("SizeBytes", 50).put("StreamViewType", "NEW_IMAGE");
+ return new DynamoDbStreamReader.Record(sequenceNumber, awsRecord);
+ }
+
+ /** The sequence numbers of each batch the Lambda target was invoked with, in invocation order. */
+ private List<List<String>> deliveredSequences(Pipe pipe) throws Exception {
+ ArgumentCaptor<String> payloads = ArgumentCaptor.forClass(String.class);
+ verify(targetInvoker, atLeastOnce()).invoke(eq(pipe), payloads.capture(), eq("us-east-1"));
+ List<List<String>> batches = new ArrayList<>();
+ for (String payload : payloads.getAllValues()) {
+ List<String> batch = new ArrayList<>();
+ for (JsonNode record : MAPPER.readTree(payload).path("Records")) {
+ batch.add(record.path("dynamodb").path("SequenceNumber").asText());
+ }
+ batches.add(batch);
+ }
+ return batches;
+ }
+
private Pipe selfManagedKafkaPipe() throws Exception {
Pipe pipe = new Pipe();
pipe.setName("orders-pipe");
diff --git a/src/test/java/io/github/hectorvent/floci/services/pipes/PipesServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/pipes/PipesServiceTest.java
index 24a2e1a467..49c04adfab 100644
--- a/src/test/java/io/github/hectorvent/floci/services/pipes/PipesServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/pipes/PipesServiceTest.java
@@ -541,6 +541,32 @@ class PipesServiceTest {
assertTrue(ex.getMessage().contains("must be between 1 and 10"), ex.getMessage());
}
+ @Test
+ void dynamoDbStreamRetryOptionsAreAcceptedAndReturned() {
+ String created = """
+ {"StartingPosition":"TRIM_HORIZON","MaximumRetryAttempts":3,"MaximumRecordAgeInSeconds":60,
+ "OnPartialBatchItemFailure":"AUTOMATIC_BISECT"}""";
+ String updated = """
+ {"MaximumRetryAttempts":10000,"MaximumRecordAgeInSeconds":604800,
+ "OnPartialBatchItemFailure":"AUTOMATIC_BISECT"}""";
+
+ pipesService.createPipe("ddb-retry-pipe",
+ "arn:aws:dynamodb:us-east-1:000000000000:table/orders/stream/2024-01-01T00:00:00.000",
+ "arn:aws:sqs:us-east-1:000000000000:target", "arn:aws:iam::000000000000:role/role",
+ null, null, null, sourceParameters("{\"DynamoDBStreamParameters\":" + created + "}"),
+ null, null, null, "us-east-1");
+ assertEquals(sourceParameters(created), pipesService.describePipe("ddb-retry-pipe", "us-east-1")
+ .getSourceParameters().path("DynamoDBStreamParameters"));
+
+ pipesService.updatePipe("ddb-retry-pipe", null, null, null, null, null,
+ sourceParameters("{\"DynamoDBStreamParameters\":" + updated + "}"), null, null, "us-east-1");
+ JsonNode stored = pipesService.describePipe("ddb-retry-pipe", "us-east-1")
+ .getSourceParameters().path("DynamoDBStreamParameters");
+ assertEquals(10000, stored.path("MaximumRetryAttempts").asInt());
+ assertEquals(604800, stored.path("MaximumRecordAgeInSeconds").asInt());
+ assertEquals("AUTOMATIC_BISECT", stored.path("OnPartialBatchItemFailure").asText());
+ }
+
private static JsonNode sourceParameters(String json) {
try {
return MAPPER.readTree(json);
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumerTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumerTest.java
index 0420cdcb3d..b4bd501c06 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftDynamoDbZeroEtlConsumerTest.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.redshift;
+import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import io.github.hectorvent.floci.core.common.AwsException;
@@ -7,11 +8,11 @@ import io.github.hectorvent.floci.core.common.RegionResolver;
import io.github.hectorvent.floci.core.common.RequestScopes;
import io.github.hectorvent.floci.services.dynamodb.DynamoDbFacade;
import io.github.hectorvent.floci.services.dynamodb.DynamoDbService;
-import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.CheckpointLifetime;
import io.github.hectorvent.floci.services.dynamodb.backend.RecordingDynamoDbBackend;
import io.github.hectorvent.floci.services.dynamodb.backend.RecordingDynamoDbBackend.Invocation;
import io.github.hectorvent.floci.services.dynamodb.model.AttributeDefinition;
-import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
import io.github.hectorvent.floci.services.dynamodb.model.KeySchemaElement;
import io.github.hectorvent.floci.services.redshift.model.Integration;
import io.github.hectorvent.floci.services.redshiftdata.RedshiftZeroEtlWriter;
@@ -20,7 +21,12 @@ import jakarta.inject.Inject;
import org.junit.jupiter.api.Test;
import org.mockito.ArgumentCaptor;
+import java.util.ArrayList;
import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
@@ -29,6 +35,7 @@ import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
@@ -46,37 +53,222 @@ class RedshiftDynamoDbZeroEtlConsumerTest {
@Inject
DynamoDbFacade dynamoDb;
+ private static final String SHARD_ID = "shardId-0000000001-00000000001";
+
+ /**
+ * Serves shards from memory the way the native engine does. A cursor is the sequence to read
+ * after, or empty for the trim horizon, a sequence older than the oldest retained record is
+ * trimmed, and a closed shard read to its end has no next cursor.
+ */
+ static class FakeStreamReader implements DynamoDbStreamReader {
+
+ final List<String> iteratorRequests = new CopyOnWriteArrayList<>();
+ private final List<Shard> shards = new CopyOnWriteArrayList<>();
+ private final Map<String, List<Record>> records = new ConcurrentHashMap<>();
+ private final Set<String> closed = ConcurrentHashMap.newKeySet();
+ private final CheckpointLifetime lifetime;
+
+ FakeStreamReader(CheckpointLifetime lifetime) {
+ this.lifetime = lifetime;
+ }
+
+ List<Record> shard(String shardId, String parentShardId) {
+ shards.add(new Shard(shardId, parentShardId, null, null));
+ return records.computeIfAbsent(shardId, ignored -> new CopyOnWriteArrayList<>());
+ }
+
+ void close(String shardId) {
+ closed.add(shardId);
+ }
+
+ @Override
+ public ShardsPage describeStream(Stream stream, String exclusiveStartShardId, Integer limit) {
+ return new ShardsPage(stream, List.copyOf(shards), null);
+ }
+
+ @Override
+ public Cursor getShardIterator(Stream stream, String shardId, Position position, String sequenceNumber) {
+ iteratorRequests.add(shardId + ":" + position + ":" + sequenceNumber);
+ return new Cursor(stream, shardId, position == Position.TRIM_HORIZON ? "" : sequenceNumber);
+ }
+
+ @Override
+ public RecordsPage getRecords(Cursor cursor, int limit) {
+ List<Record> held = new ArrayList<>(records.get(cursor.shardId()));
+ String after = cursor.token();
+ if (!after.isEmpty() && !held.isEmpty() && after.compareTo(held.get(0).sequenceNumber()) < 0) {
+ throw new AwsException("TrimmedDataAccessException",
+ "The requested sequence number has been trimmed", 400);
+ }
+ List<Record> page = held.stream()
+ .filter(record -> record.sequenceNumber().compareTo(after) > 0)
+ .limit(limit)
+ .toList();
+ if (page.isEmpty()) {
+ return new RecordsPage(page, closed.contains(cursor.shardId()) ? null : cursor);
+ }
+ return new RecordsPage(page,
+ new Cursor(cursor.stream(), cursor.shardId(), page.get(page.size() - 1).sequenceNumber()));
+ }
+
+ @Override
+ public CheckpointLifetime checkpointLifetime() {
+ return lifetime;
+ }
+ }
+
@Test
- void writesRecordsAndAdvancesCheckpointOnlyAfterSuccessfulBatch() {
- DynamoDbStreamService streamService = mock(DynamoDbStreamService.class);
+ void writesTheAwsStreamRecordsAndCommitsShardProgressAfterTheWrite() {
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
RedshiftService redshiftService = mock(RedshiftService.class);
RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
- Integration integration = integration("orders");
- integration.setBackfillCompleted(true);
- DynamoDbStreamRecord record = new DynamoDbStreamRecord();
- record.setEventId("event-1");
- record.setSequenceNumber("000000000000000000001");
- String iterator = "iterator";
- when(streamService.getShardIterator(eq(integration.getSourceStreamArn()), eq(DynamoDbStreamService.SHARD_ID),
- eq("TRIM_HORIZON"), eq(null))).thenReturn(iterator);
- when(streamService.getRecords(iterator, 100)).thenReturn(
- new DynamoDbStreamService.GetRecordsResult(List.of(record), "next"));
- when(writer.writeBatch(integration.getAccountId(), "warehouse", "floci_zetl_orders", List.of(record)))
- .thenReturn(record.getSequenceNumber());
+ Integration integration = streamingIntegration();
+ DynamoDbStreamReader.Record first = streamRecord("s1");
+ DynamoDbStreamReader.Record second = streamRecord("s2");
+ streams.shard(SHARD_ID, null).addAll(List.of(first, second));
- RedshiftDynamoDbZeroEtlConsumer consumer =
- new RedshiftDynamoDbZeroEtlConsumer(streamService, dynamoDb, redshiftService, writer);
- consumer.pollOnce(integration);
+ new RedshiftDynamoDbZeroEtlConsumer(streams, dynamoDb, redshiftService, writer).pollOnce(integration);
verify(writer).createLandingTable(integration.getAccountId(), "warehouse", "floci_zetl_orders");
- verify(writer).writeBatch(integration.getAccountId(), "warehouse", "floci_zetl_orders", List.of(record));
+ verify(writer).writeBatch(integration.getAccountId(), "warehouse", "floci_zetl_orders",
+ List.of(first.awsRecord(), second.awsRecord()));
verify(redshiftService).updateIntegrationRuntime(integration.getAccountId(), integration.getIntegrationArn(),
- record.getSequenceNumber(), true, null);
+ Map.of(SHARD_ID, "s2"), true, null);
+ assertEquals(Map.of(SHARD_ID, "s2"), integration.getShardSequenceNumbers());
+ assertEquals(List.of(SHARD_ID + ":TRIM_HORIZON:null"), streams.iteratorRequests);
+ }
+
+ @Test
+ void commitsProgressForEachShardAndResumesAfterIt() {
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
+ RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
+ Integration integration = streamingIntegration();
+ streams.shard("shard-a", null).add(streamRecord("a1"));
+ streams.shard("shard-b", null).add(streamRecord("b1"));
+ RedshiftDynamoDbZeroEtlConsumer consumer =
+ new RedshiftDynamoDbZeroEtlConsumer(streams, dynamoDb, mock(RedshiftService.class), writer);
+
+ consumer.pollOnce(integration);
+ consumer.pollOnce(integration);
+
+ assertEquals(Map.of("shard-a", "a1", "shard-b", "b1"), integration.getShardSequenceNumbers());
+ assertEquals(List.of("shard-a:TRIM_HORIZON:null", "shard-b:TRIM_HORIZON:null",
+ "shard-a:AFTER_SEQUENCE_NUMBER:a1", "shard-b:AFTER_SEQUENCE_NUMBER:b1"), streams.iteratorRequests);
+ verify(writer, times(2)).writeBatch(any(), any(), any(), any());
+ }
+
+ @Test
+ void failedWriteKeepsProgressSoTheSameRecordsAreWrittenNextTick() {
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
+ RedshiftService redshiftService = mock(RedshiftService.class);
+ RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
+ doThrow(new AwsException("InternalFailure", "Could not write Redshift zero-ETL records.", 500))
+ .doNothing()
+ .when(writer).writeBatch(any(), any(), any(), any());
+ Integration integration = streamingIntegration();
+ integration.setShardSequenceNumbers(Map.of(SHARD_ID, "s1"));
+ DynamoDbStreamReader.Record second = streamRecord("s2");
+ streams.shard(SHARD_ID, null).addAll(List.of(streamRecord("s1"), second));
+ RedshiftDynamoDbZeroEtlConsumer consumer =
+ new RedshiftDynamoDbZeroEtlConsumer(streams, dynamoDb, redshiftService, writer);
+
+ assertThrows(AwsException.class, () -> consumer.pollOnce(integration));
+
+ assertEquals(Map.of(SHARD_ID, "s1"), integration.getShardSequenceNumbers());
+ verify(redshiftService, never()).updateIntegrationRuntime(any(), any(), any(), eq(true), any());
+
+ consumer.pollOnce(integration);
+
+ verify(writer, times(2)).writeBatch(integration.getAccountId(), "warehouse", "floci_zetl_orders",
+ List.of(second.awsRecord()));
+ assertEquals(Map.of(SHARD_ID, "s2"), integration.getShardSequenceNumbers());
+ }
+
+ @Test
+ void trimmedCheckpointRestartsTheShardAtTheTrimHorizon() {
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
+ RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
+ Integration integration = streamingIntegration();
+ integration.setShardSequenceNumbers(Map.of(SHARD_ID, "s0"));
+ streams.shard(SHARD_ID, null).addAll(List.of(streamRecord("s1"), streamRecord("s2")));
+ RedshiftDynamoDbZeroEtlConsumer consumer =
+ new RedshiftDynamoDbZeroEtlConsumer(streams, dynamoDb, mock(RedshiftService.class), writer);
+
+ AwsException trimmed = assertThrows(AwsException.class, () -> consumer.pollOnce(integration));
+ assertEquals("TrimmedDataAccessException", trimmed.getErrorCode());
+ assertTrue(integration.getShardSequenceNumbers().isEmpty());
+
+ consumer.pollOnce(integration);
+
+ assertEquals(Map.of(SHARD_ID, "s2"), integration.getShardSequenceNumbers());
+ }
+
+ @Test
+ void readsAChildShardOnlyAfterItsParentIsReadToItsEnd() {
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
+ Integration integration = streamingIntegration();
+ streams.shard("parent", null).add(streamRecord("p1"));
+ streams.close("parent");
+ streams.shard("child", "parent").add(streamRecord("c1"));
+ RedshiftDynamoDbZeroEtlConsumer consumer = new RedshiftDynamoDbZeroEtlConsumer(streams, dynamoDb,
+ mock(RedshiftService.class), mock(RedshiftZeroEtlWriter.class));
+
+ consumer.pollOnce(integration);
+ consumer.pollOnce(integration);
+ assertEquals(Map.of("parent", "p1"), integration.getShardSequenceNumbers());
+
+ consumer.pollOnce(integration);
+ consumer.pollOnce(integration);
+
+ assertEquals(Map.of("parent", "p1", "child", "c1"), integration.getShardSequenceNumbers());
+ assertEquals(List.of("parent:TRIM_HORIZON:null", "parent:AFTER_SEQUENCE_NUMBER:p1",
+ "child:TRIM_HORIZON:null", "child:AFTER_SEQUENCE_NUMBER:c1"), streams.iteratorRequests);
+ }
+
+ @Test
+ void processLifetimeDiscardsPersistedShardProgressAtStartup() {
+ Integration integration = streamingIntegration();
+ integration.setShardSequenceNumbers(Map.of(SHARD_ID, "s9"));
+
+ startPersisted(CheckpointLifetime.PROCESS, integration);
+
+ assertTrue(integration.getShardSequenceNumbers().isEmpty());
+ }
+
+ @Test
+ void streamLifetimeKeepsPersistedShardProgressAtStartup() {
+ Integration integration = streamingIntegration();
+ integration.setShardSequenceNumbers(Map.of(SHARD_ID, "s9"));
+
+ startPersisted(CheckpointLifetime.STREAM, integration);
+
+ assertEquals(Map.of(SHARD_ID, "s9"), integration.getShardSequenceNumbers());
+ }
+
+ private void startPersisted(CheckpointLifetime lifetime, Integration integration) {
+ RedshiftService redshiftService = mock(RedshiftService.class);
+ when(redshiftService.listDynamoDbZeroEtlIntegrations()).thenReturn(List.of(integration));
+ new RedshiftDynamoDbZeroEtlConsumer(new FakeStreamReader(lifetime), dynamoDb, redshiftService,
+ mock(RedshiftZeroEtlWriter.class)).startPersistedIntegrations();
+ }
+
+ private static DynamoDbStreamReader.Record streamRecord(String sequenceNumber) {
+ ObjectNode awsRecord = MAPPER.createObjectNode()
+ .put("eventID", "event-" + sequenceNumber)
+ .put("eventName", "MODIFY");
+ awsRecord.putObject("dynamodb").put("SequenceNumber", sequenceNumber);
+ return new DynamoDbStreamReader.Record(sequenceNumber, awsRecord);
+ }
+
+ private static Integration streamingIntegration() {
+ Integration integration = integration("orders");
+ integration.setBackfillCompleted(true);
+ return integration;
}
@Test
void backfillsOnePageAtATimeBeforeSwitchingToStreamPolling() {
- DynamoDbStreamService streamService = mock(DynamoDbStreamService.class);
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
RedshiftService redshiftService = mock(RedshiftService.class);
RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
String tableName = "orders-page-" + System.nanoTime();
@@ -99,28 +291,31 @@ class RedshiftDynamoDbZeroEtlConsumerTest {
assertThrows(AwsException.class, () -> dynamoDbService.describeTable(tableName, "us-east-1"));
RedshiftDynamoDbZeroEtlConsumer consumer =
- new RedshiftDynamoDbZeroEtlConsumer(streamService, dynamoDb, redshiftService, writer);
+ new RedshiftDynamoDbZeroEtlConsumer(streams, dynamoDb, redshiftService, writer);
consumer.pollOnce(integration);
assertFalse(integration.isBackfillCompleted());
assertNotNull(integration.getBackfillLastEvaluatedKey());
verify(redshiftService).updateIntegrationBackfillProgress(eq(integration.getAccountId()),
eq(integration.getIntegrationArn()), eq(integration.getBackfillLastEvaluatedKey()), eq(false));
- verify(streamService, never()).getShardIterator(any(), any(), any(), any());
+ assertTrue(streams.iteratorRequests.isEmpty());
@SuppressWarnings("unchecked")
- ArgumentCaptor<List<DynamoDbStreamRecord>> captor = ArgumentCaptor.forClass(List.class);
+ ArgumentCaptor<List<JsonNode>> captor = ArgumentCaptor.forClass(List.class);
verify(writer).writeBatch(eq(integration.getAccountId()), eq("warehouse"), eq("floci_zetl_orders"),
captor.capture());
assertEquals(100, captor.getValue().size());
- DynamoDbStreamRecord written = captor.getValue().get(0);
- assertEquals("INSERT", written.getEventName());
- assertTrue(written.getEventId().startsWith("backfill#" + integration.getIntegrationArn() + "#"));
+ JsonNode written = captor.getValue().get(0);
+ assertEquals("INSERT", written.path("eventName").asText());
+ assertTrue(written.path("eventID").asText().startsWith("backfill#" + integration.getIntegrationArn() + "#"));
+ assertEquals("backfill", written.path("dynamodb").path("SequenceNumber").asText());
+ assertEquals(written.path("dynamodb").path("Keys").path("id"),
+ written.path("dynamodb").path("NewImage").path("id"));
}
@Test
void backfillCompletesWhenScanReturnsNoLastEvaluatedKey() {
- DynamoDbStreamService streamService = mock(DynamoDbStreamService.class);
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
RedshiftService redshiftService = mock(RedshiftService.class);
RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
String tableName = "orders-empty-" + System.nanoTime();
@@ -136,7 +331,7 @@ class RedshiftDynamoDbZeroEtlConsumerTest {
assertThrows(AwsException.class, () -> dynamoDbService.describeTable(tableName, "us-east-1"));
RedshiftDynamoDbZeroEtlConsumer consumer =
- new RedshiftDynamoDbZeroEtlConsumer(streamService, dynamoDb, redshiftService, writer);
+ new RedshiftDynamoDbZeroEtlConsumer(streams, dynamoDb, redshiftService, writer);
consumer.pollOnce(integration);
assertTrue(integration.isBackfillCompleted());
@@ -147,7 +342,7 @@ class RedshiftDynamoDbZeroEtlConsumerTest {
@Test
void sameItemProducesTheSameEventIdAcrossScans() {
- DynamoDbStreamService streamService = mock(DynamoDbStreamService.class);
+ FakeStreamReader streams = new FakeStreamReader(CheckpointLifetime.PROCESS);
RedshiftService redshiftService = mock(RedshiftService.class);
RedshiftZeroEtlWriter writer = mock(RedshiftZeroEtlWriter.class);
String tableName = "orders-same-" + System.nanoTime();
@@ -163,16 +358,16 @@ class RedshiftDynamoDbZeroEtlConsumerTest {
});
RedshiftDynamoDbZeroEtlConsumer consumer =
- new RedshiftDynamoDbZeroEtlConsumer(streamService, dynamoDb, redshiftService, writer);
+ new RedshiftDynamoDbZeroEtlConsumer(streams, dynamoDb, redshiftService, writer);
consumer.pollOnce(integration);
integration.setBackfillCompleted(false);
consumer.pollOnce(integration);
@SuppressWarnings("unchecked")
- ArgumentCaptor<List<DynamoDbStreamRecord>> captor = ArgumentCaptor.forClass(List.class);
+ ArgumentCaptor<List<JsonNode>> captor = ArgumentCaptor.forClass(List.class);
verify(writer, times(2)).writeBatch(any(), any(), any(), captor.capture());
- assertEquals(captor.getAllValues().get(0).get(0).getEventId(),
- captor.getAllValues().get(1).get(0).getEventId());
+ assertEquals(captor.getAllValues().get(0).get(0).path("eventID"),
+ captor.getAllValues().get(1).get(0).path("eventID"));
}
@Test
@@ -182,7 +377,7 @@ class RedshiftDynamoDbZeroEtlConsumerTest {
Integration integration = integration("orders");
integration.setSourceStreamArn("arn:aws:dynamodb:eu-west-1:111111111111:table/orders/stream/one");
- new RedshiftDynamoDbZeroEtlConsumer(mock(DynamoDbStreamService.class), recording,
+ new RedshiftDynamoDbZeroEtlConsumer(new FakeStreamReader(CheckpointLifetime.PROCESS), recording,
mock(RedshiftService.class), mock(RedshiftZeroEtlWriter.class)).pollOnce(integration);
assertEquals(List.of(
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
index a863204a90..0fc0079510 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/RedshiftServiceTest.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.redshift;
+import com.fasterxml.jackson.databind.ObjectMapper;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.core.common.PaginatedResult;
@@ -8,8 +9,9 @@ import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.StorageFactory;
import io.github.hectorvent.floci.services.rds.proxy.PasswordValidator;
-import io.github.hectorvent.floci.services.dynamodb.DynamoDbStreamService;
-import io.github.hectorvent.floci.services.dynamodb.model.StreamDescription;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbApiStreamReader;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbOperations;
+import io.github.hectorvent.floci.services.dynamodb.backend.DynamoDbStreamReader.CheckpointLifetime;
import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerHandle;
import io.github.hectorvent.floci.services.redshift.container.RedshiftContainerManager;
import io.github.hectorvent.floci.services.redshift.model.Cluster;
@@ -33,6 +35,8 @@ import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -63,7 +67,7 @@ class RedshiftServiceTest {
private DockerHostResolver dockerHostResolver;
private RedshiftCredentialBroker credentialBroker;
private SecretsManagerService secretsManagerService;
- private DynamoDbStreamService streamService;
+ private final Set<String> existingStreams = new HashSet<>();
private RedshiftService service;
@BeforeEach
@@ -109,22 +113,27 @@ class RedshiftServiceTest {
credentialBroker = new RedshiftCredentialBroker();
secretsManagerService = mock(SecretsManagerService.class);
- streamService = mock(DynamoDbStreamService.class);
+ ObjectMapper mapper = new ObjectMapper();
+ // Answers DescribeStream the way the native handler does: it throws for an unknown stream.
+ DynamoDbOperations streams = call -> {
+ String streamArn = call.body().path("StreamArn").asText();
+ if (!existingStreams.contains(streamArn)) {
+ throw new AwsException("ResourceNotFoundException", "Stream not found: " + streamArn, 400);
+ }
+ return new DynamoDbOperations.Reply(200, mapper.readTree("{\"StreamDescription\":{\"Shards\":[]}}"),
+ Map.of());
+ };
service = new RedshiftService(sf, cm, config, regionResolver, proxyManager, dockerHostResolver,
- credentialBroker, secretsManagerService, new com.fasterxml.jackson.databind.ObjectMapper(),
- streamService);
+ credentialBroker, secretsManagerService, mapper,
+ new DynamoDbApiStreamReader(streams, CheckpointLifetime.PROCESS));
}
@Test
void createDynamoDbZeroEtlIntegrationRequiresExistingProvisionedResources() {
String streamArn = "arn:aws:dynamodb:us-east-1:111111111111:table/orders/stream/2026-09-18T00:00:00.000";
String targetArn = "arn:aws:redshift:us-east-1:111111111111:cluster:warehouse";
- StreamDescription stream = new StreamDescription();
- stream.setStreamArn(streamArn);
- stream.setTableName("orders");
- stream.setStreamStatus("ENABLED");
- when(streamService.describeStream(streamArn)).thenReturn(stream);
+ existingStreams.add(streamArn);
Cluster cluster = new Cluster();
cluster.setClusterIdentifier("warehouse");
when(clusterBackend.get("warehouse")).thenReturn(Optional.of(cluster));
@@ -140,15 +149,48 @@ class RedshiftServiceTest {
assertFalse(integration.isBackfillCompleted());
}
+ @Test
+ void createDynamoDbZeroEtlIntegrationRejectsAMissingStreamWithResourceNotFound() {
+ String streamArn = "arn:aws:dynamodb:us-east-1:111111111111:table/orders/stream/2026-09-18T00:00:00.000";
+
+ AwsException error = assertThrows(AwsException.class, () -> service.createIntegration(
+ "orders-to-warehouse", streamArn, "arn:aws:redshift:us-east-1:111111111111:cluster:warehouse",
+ null, null, Map.of(), Map.of(), "us-east-1"));
+
+ assertEquals("ResourceNotFoundException", error.getErrorCode());
+ assertEquals("Stream not found: " + streamArn, error.getMessage());
+ assertEquals(400, error.getHttpStatus());
+ verify(integrationBackend, never()).put(any(), any());
+ }
+
+ @Test
+ void updateIntegrationRuntimeStoresACopyOfShardProgressOnlyOnSuccess() {
+ String integrationArn = "arn:aws:redshift:us-east-1:111111111111:integration:one";
+ Integration integration = new Integration();
+ integration.setIntegrationArn(integrationArn);
+ when(integrationBackend.keysForAccount("111111111111")).thenReturn(Set.of("one"));
+ when(integrationBackend.getForAccount("111111111111", "one")).thenReturn(Optional.of(integration));
+ Map<String, String> progress = new HashMap<>(Map.of("shard-1", "s1"));
+
+ service.updateIntegrationRuntime("111111111111", integrationArn, progress, true, null);
+ progress.put("shard-1", "s2");
+
+ assertEquals(Map.of("shard-1", "s1"), integration.getShardSequenceNumbers());
+ assertEquals("active", integration.getStatus());
+
+ service.updateIntegrationRuntime("111111111111", integrationArn, progress, false, "boom");
+
+ assertEquals(Map.of("shard-1", "s1"), integration.getShardSequenceNumbers());
+ assertEquals("failed", integration.getStatus());
+ assertEquals("boom", integration.getLastError());
+ verify(integrationBackend, times(2)).putForAccount("111111111111", "one", integration);
+ }
+
@Test
void updateIntegrationBackfillProgressPersistsCheckpointAndFlipsStatusOnCompletion() {
String streamArn = "arn:aws:dynamodb:us-east-1:111111111111:table/orders/stream/2026-09-18T00:00:00.000";
String targetArn = "arn:aws:redshift:us-east-1:111111111111:cluster:warehouse";
- StreamDescription stream = new StreamDescription();
- stream.setStreamArn(streamArn);
- stream.setTableName("orders");
- stream.setStreamStatus("ENABLED");
- when(streamService.describeStream(streamArn)).thenReturn(stream);
+ existingStreams.add(streamArn);
Cluster cluster = new Cluster();
cluster.setClusterIdentifier("warehouse");
when(clusterBackend.get("warehouse")).thenReturn(Optional.of(cluster));
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshift/model/IntegrationModelTest.java b/src/test/java/io/github/hectorvent/floci/services/redshift/model/IntegrationModelTest.java
new file mode 100644
index 0000000000..8b60be13ef
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/redshift/model/IntegrationModelTest.java
@@ -0,0 +1,43 @@
+package io.github.hectorvent.floci.services.redshift.model;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class IntegrationModelTest {
+
+ @Test
+ void legacyJsonWithCheckpointSequenceNumberLoadsWithEmptyShardSequenceNumbers() throws Exception {
+ // A default ObjectMapper, as the storage backends use, so an unknown property fails the load.
+ Integration restored = new ObjectMapper().readValue("""
+ {"integrationArn":"arn:aws:redshift:us-east-1:111111111111:integration:one",
+ "checkpointSequenceNumber":"000000000000000000042","backfillCompleted":true}
+ """, Integration.class);
+
+ assertEquals("arn:aws:redshift:us-east-1:111111111111:integration:one", restored.getIntegrationArn());
+ assertTrue(restored.isBackfillCompleted());
+ assertTrue(restored.getShardSequenceNumbers().isEmpty());
+ }
+
+ @Test
+ void shardSequenceNumbersStayNonNullAndAreCopiedOnSet() throws Exception {
+ Integration integration = new Integration();
+ assertTrue(integration.getShardSequenceNumbers().isEmpty());
+
+ integration.setShardSequenceNumbers(null);
+ assertTrue(integration.getShardSequenceNumbers().isEmpty());
+
+ Map<String, String> progress = new HashMap<>(Map.of("shard-1", "s1"));
+ integration.setShardSequenceNumbers(progress);
+ progress.put("shard-2", "s2");
+ assertEquals(Map.of("shard-1", "s1"), integration.getShardSequenceNumbers());
+
+ Integration nulled = new ObjectMapper().readValue("{\"shardSequenceNumbers\":null}", Integration.class);
+ assertTrue(nulled.getShardSequenceNumbers().isEmpty());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/redshiftdata/RedshiftZeroEtlWriterTest.java b/src/test/java/io/github/hectorvent/floci/services/redshiftdata/RedshiftZeroEtlWriterTest.java
index 33181964b5..ab105a3d5b 100644
--- a/src/test/java/io/github/hectorvent/floci/services/redshiftdata/RedshiftZeroEtlWriterTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/redshiftdata/RedshiftZeroEtlWriterTest.java
@@ -1,8 +1,8 @@
package io.github.hectorvent.floci.services.redshiftdata;
+import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.node.JsonNodeFactory;
-import io.github.hectorvent.floci.services.dynamodb.model.DynamoDbStreamRecord;
+import com.fasterxml.jackson.databind.node.ObjectNode;
import io.github.hectorvent.floci.services.redshift.RedshiftService;
import io.github.hectorvent.floci.services.redshift.model.Cluster;
import org.junit.jupiter.api.Test;
@@ -10,9 +10,10 @@ import org.junit.jupiter.api.Test;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.Statement;
+import java.sql.Types;
+import java.time.Instant;
import java.util.List;
-import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.ArgumentMatchers.contains;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.*;
@@ -40,7 +41,7 @@ class RedshiftZeroEtlWriterTest {
}
@Test
- void writesStreamRecordsIdempotentlyAndReturnsNewestSequence() throws Exception {
+ void writesAwsStreamRecordsIdempotently() throws Exception {
RedshiftService redshiftService = mock(RedshiftService.class);
RedshiftDataConnectionFactory connectionFactory = mock(RedshiftDataConnectionFactory.class);
ObjectMapper objectMapper = new ObjectMapper();
@@ -50,16 +51,23 @@ class RedshiftZeroEtlWriterTest {
when(connectionFactory.open(any())).thenReturn(connection);
when(connection.prepareStatement(contains("ON CONFLICT (event_id) DO NOTHING"))).thenReturn(statement);
- DynamoDbStreamRecord first = record("event-1", "000000000000000000001");
- DynamoDbStreamRecord second = record("event-2", "000000000000000000002");
+ JsonNode first = record("event-1", "000000000000000000001");
+ JsonNode second = record("event-2", "000000000000000000002");
RedshiftZeroEtlWriter writer = new RedshiftZeroEtlWriter(redshiftService, connectionFactory, objectMapper);
- String sequence = writer.writeBatch("111111111111", "warehouse", "floci_zetl_orders", List.of(first, second));
+ writer.writeBatch("111111111111", "warehouse", "floci_zetl_orders", List.of(first, second));
- assertEquals("000000000000000000002", sequence);
verify(statement, times(2)).executeUpdate();
verify(statement).setString(1, "event-1");
verify(statement).setString(1, "event-2");
+ verify(statement, times(2)).setString(2, "INSERT");
+ verify(statement, times(2)).setString(3, "aws:dynamodb");
+ verify(statement, times(2)).setString(4, "us-east-1");
+ verify(statement).setString(5, "000000000000000000002");
+ verify(statement, times(2)).setObject(6, Instant.ofEpochSecond(1_758_153_600L));
+ verify(statement, times(2)).setString(7, "{\"id\":\"1\"}");
+ verify(statement, times(2)).setNull(8, Types.VARCHAR);
+ verify(statement, times(2)).setString(9, "{\"value\":\"new\"}");
verify(connection).close();
}
@@ -73,16 +81,17 @@ class RedshiftZeroEtlWriterTest {
return cluster;
}
- private static DynamoDbStreamRecord record(String eventId, String sequenceNumber) {
- DynamoDbStreamRecord record = new DynamoDbStreamRecord();
- record.setEventId(eventId);
- record.setEventName("INSERT");
- record.setEventSource("aws:dynamodb");
- record.setAwsRegion("us-east-1");
- record.setSequenceNumber(sequenceNumber);
- record.setApproximateCreationDateTime(1_758_153_600L);
- record.setKeys(JsonNodeFactory.instance.objectNode().put("id", "1"));
- record.setNewImage(JsonNodeFactory.instance.objectNode().put("value", "new"));
+ private static JsonNode record(String eventId, String sequenceNumber) {
+ ObjectNode record = new ObjectMapper().createObjectNode()
+ .put("eventID", eventId)
+ .put("eventName", "INSERT")
+ .put("eventSource", "aws:dynamodb")
+ .put("awsRegion", "us-east-1");
+ ObjectNode dynamodb = record.putObject("dynamodb")
+ .put("SequenceNumber", sequenceNumber)
+ .put("ApproximateCreationDateTime", 1_758_153_600L);
+ dynamodb.putObject("Keys").put("id", "1");
+ dynamodb.putObject("NewImage").put("value", "new");
return record;
}
}