Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

fix(lambda): honor MaximumRetryAttempts and DestinationConfig.OnFailure for Kinesis ESM (#6133)

Attila Kisbb32ba06eb81
diff --git a/docs/services/lambda.md b/docs/services/lambda.md
index 52a1826686..b459a20bfa 100644
--- a/docs/services/lambda.md
+++ b/docs/services/lambda.md
@@ -1108,7 +1108,9 @@ Connect Lambda to SQS, Kinesis, or DynamoDB Streams. Self-managed Apache Kafka e
For DynamoDB Streams mappings, Floci retries failed batches with exponential backoff, honors `MaximumRetryAttempts` and `MaximumRecordAgeInSeconds`, and sends discarded batches to configured SQS, SNS, or S3 `DestinationConfig.OnFailure` destinations. With `BisectBatchOnFunctionError`, a batch that fails with a function error is split in half and retried, narrowing it down to the failing record, which then follows the normal retry and record age rules. Splits do not count as retry attempts, and only function errors split: throttles do not, and a partial batch response retries from the reported record. If the OnFailure destination refuses a discarded batch, Floci logs the failure, drops the batch and advances past it.
-For Kinesis mappings, a function error or throttle leaves the shard checkpoint in place and the same batch is read again on the next poll; there is no retry limit, backoff, or OnFailure destination. With `FunctionResponseTypes: ["ReportBatchItemFailures"]`, a partial batch response moves the checkpoint up to the lowest reported record, and a malformed response retries the whole batch. `FunctionResponseTypes` can be set or cleared on an existing mapping with `UpdateEventSourceMapping`.
+For Kinesis mappings, a function error leaves the shard checkpoint in place and the same batch is retried with exponential backoff. With `FunctionResponseTypes: ["ReportBatchItemFailures"]`, a partial batch response moves the checkpoint up to the lowest reported record and retries from there, and a malformed response retries the whole batch. Each of these retries counts toward `MaximumRetryAttempts`; when a partial batch response moves the checkpoint forward, the attempts count again from the new checkpoint, as they do for DynamoDB Streams. When the attempts run out, or a failed batch is older than `MaximumRecordAgeInSeconds`, Floci discards the batch: it sends one invocation record with a `KinesisBatchInfo` block to the SQS, SNS, or S3 `DestinationConfig.OnFailure` destination, if one is set, moves the checkpoint past the batch and goes on with the next records of the shard. If the destination refuses the record, Floci logs the failure and discards the batch all the same. With the defaults (`-1` or unset) a failing batch is retried until it succeeds. A throttle leaves the checkpoint in place, without a backoff and without using an attempt, until the batch is older than `MaximumRecordAgeInSeconds`: then it is discarded the same way, with an `approximateInvokeCount` of the invocations that ran. The record age is checked only when an invoke fails or is throttled, so records are not dropped before their first invoke. `BisectBatchOnFunctionError` is stored and returned but not applied to Kinesis mappings: a failing batch is retried whole. `FunctionResponseTypes` can be set or cleared on an existing mapping with `UpdateEventSourceMapping`.
+
+For both stream sources, the first retry waits twice `FLOCI_SERVICES_LAMBDA_POLL_INTERVAL_MS` (2 seconds by default), and each later retry waits twice as long as the one before, up to 60 seconds.
For SQS mappings, a function error returns every delivered message to the queue with the queue's `VisibilityTimeout`, so it is received again after that timeout and the queue's `RedrivePolicy` applies as usual. With `FunctionResponseTypes: ["ReportBatchItemFailures"]`, only the messages the function reports are returned and the rest are deleted; a malformed response returns the whole batch.
diff --git a/src/main/java/io/github/hectorvent/floci/lifecycle/EmulatorLifecycle.java b/src/main/java/io/github/hectorvent/floci/lifecycle/EmulatorLifecycle.java
index bcb9306aba..e3908ec840 100644
--- a/src/main/java/io/github/hectorvent/floci/lifecycle/EmulatorLifecycle.java
+++ b/src/main/java/io/github/hectorvent/floci/lifecycle/EmulatorLifecycle.java
@@ -398,8 +398,9 @@ public class EmulatorLifecycle {
// SIGTERM grace window and trigger SIGKILL; if the flush ran last it would be skipped and
// in-memory (hybrid) data would be lost on an otherwise-graceful shutdown. shutdownAll()
// still runs at the end to stop the flush schedulers and capture any shutdown-time writes.
- // DynamoDB stream consumers stop first, so the flush holds their final checkpoints.
+ // Stream consumers stop first, so the flush holds their final checkpoints.
runCleanup("DynamoDB Streams poller", dynamodbStreamsPoller::shutdown);
+ runCleanup("Kinesis poller", kinesisPoller::shutdown);
runCleanup("Redshift zero-ETL consumer", redshiftZeroEtlConsumer::shutdown);
runCleanup("storage flush", storageFactory::flushAll);
runCleanup("EC2 metadata server", () -> {
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/KinesisEventSourcePoller.java b/src/main/java/io/github/hectorvent/floci/services/lambda/KinesisEventSourcePoller.java
index 02b415aaf3..8e444d702d 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambda/KinesisEventSourcePoller.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/KinesisEventSourcePoller.java
@@ -32,15 +32,18 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
+import java.util.function.LongSupplier;
@ApplicationScoped
public class KinesisEventSourcePoller implements Resettable {
private static final Logger LOG = Logger.getLogger(KinesisEventSourcePoller.class);
+ private static final String BATCH_INFO_NAME = "KinesisBatchInfo";
private final Vertx vertx;
private final KinesisService kinesisService;
@@ -50,8 +53,17 @@ public class KinesisEventSourcePoller implements Resettable {
private final long pollIntervalMs;
private final ObjectMapper objectMapper;
private final PipesFilterMatcher filterMatcher;
+ private final StreamBatchFailures batchFailures;
+ private final LongSupplier clock;
private final ConcurrentHashMap<String, Long> timerIds = new ConcurrentHashMap<>();
- private final ConcurrentHashMap<String, Boolean> activePolls = new ConcurrentHashMap<>();
+ final ConcurrentHashMap<String, Boolean> activePolls = new ConcurrentHashMap<>();
+ /** Keyed by {@link StreamBatchFailures#batchStateKey}. */
+ private final ConcurrentHashMap<String, Integer> retryCounts = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, Long> retryNotBefore = new ConcurrentHashMap<>();
+ /** Set during a reset and after shutdown: a failed invoke then counts for nothing. */
+ private volatile boolean quiesced;
+ /** Mappings stopped since their last start; a delete stops the mapping before the store forgets it. */
+ final Set<String> stopped = ConcurrentHashMap.newKeySet();
private final ExecutorService pollExecutor = Executors.newCachedThreadPool(r -> {
Thread t = new Thread(r, "kinesis-esm-poller");
t.setDaemon(true);
@@ -64,7 +76,20 @@ public class KinesisEventSourcePoller implements Resettable {
LambdaTargetResolver targetResolver,
EsmStore esmStore, EmulatorConfig config,
ObjectMapper objectMapper,
- PipesFilterMatcher filterMatcher) {
+ PipesFilterMatcher filterMatcher,
+ StreamBatchFailures batchFailures) {
+ this(vertx, kinesisService, executorService, targetResolver, esmStore, config, objectMapper,
+ filterMatcher, batchFailures, System::currentTimeMillis);
+ }
+
+ KinesisEventSourcePoller(Vertx vertx, KinesisService kinesisService,
+ LambdaExecutorService executorService,
+ LambdaTargetResolver targetResolver,
+ EsmStore esmStore, EmulatorConfig config,
+ ObjectMapper objectMapper,
+ PipesFilterMatcher filterMatcher,
+ StreamBatchFailures batchFailures,
+ LongSupplier clock) {
this.vertx = vertx;
this.kinesisService = kinesisService;
this.executorService = executorService;
@@ -73,6 +98,8 @@ public class KinesisEventSourcePoller implements Resettable {
this.pollIntervalMs = config.services().lambda().pollIntervalMs();
this.objectMapper = objectMapper;
this.filterMatcher = filterMatcher;
+ this.batchFailures = batchFailures;
+ this.clock = clock;
}
public void startPersistedPollers() {
@@ -84,20 +111,36 @@ public class KinesisEventSourcePoller implements Resettable {
}
}
+ /** Runs before the Lambda containers are stopped, so an invoke cut short by them is not taken for a failure. */
@PreDestroy
- void shutdown() {
+ public synchronized void shutdown() {
+ quiesced = true;
pollExecutor.shutdownNow();
timerIds.values().forEach(vertx::cancelTimer);
timerIds.clear();
}
+ @Override
+ public synchronized void beforeReset() {
+ quiesced = true;
+ }
+
+ @Override
+ public synchronized void afterReset() {
+ quiesced = pollExecutor.isShutdown();
+ }
+
public void clear() {
timerIds.values().forEach(vertx::cancelTimer);
timerIds.clear();
activePolls.clear();
+ retryCounts.clear();
+ retryNotBefore.clear();
+ stopped.clear();
}
- public void startPolling(EventSourceMapping esm) {
+ public synchronized void startPolling(EventSourceMapping esm) {
+ stopped.remove(esm.getUuid());
if (timerIds.containsKey(esm.getUuid())) return;
String uuid = esm.getUuid();
String accountId = esm.getAccountId();
@@ -110,9 +153,19 @@ public class KinesisEventSourcePoller implements Resettable {
LOG.infov("Started Kinesis polling for ESM {0} → {1}", uuid, esm.getEventSourceArn());
}
- public void stopPolling(String uuid) {
+ public synchronized void stopPolling(String uuid) {
+ stopped.add(uuid);
Long timerId = timerIds.remove(uuid);
- if (timerId != null) vertx.cancelTimer(timerId);
+ if (timerId != null) {
+ vertx.cancelTimer(timerId);
+ }
+ retryCounts.keySet().removeIf(k -> k.startsWith(uuid + ":"));
+ retryNotBefore.keySet().removeIf(k -> k.startsWith(uuid + ":"));
+ }
+
+ /** Drops the stop tombstone once the store has forgotten the mapping, which stillActive then sees. */
+ void mappingDeleted(String uuid) {
+ stopped.remove(uuid);
}
/** Package-private (not private) only so unit tests can drive a single poll directly. */
@@ -128,71 +181,7 @@ public class KinesisEventSourcePoller implements Resettable {
esm.getAccountId(), streamName, esm.getRegion());
for (KinesisShard shard : stream.getShards()) {
- String lastSeq = esm.getShardSequenceNumbers().get(shard.getShardId());
- String iterator;
- if (lastSeq == null) {
- iterator = kinesisService.getShardIteratorForAccount(esm.getAccountId(),
- streamName, shard.getShardId(), "TRIM_HORIZON", null, esm.getRegion());
- } else {
- iterator = kinesisService.getShardIteratorForAccount(esm.getAccountId(),
- streamName, shard.getShardId(), "AFTER_SEQUENCE_NUMBER", lastSeq, esm.getRegion());
- }
-
- Map<String, Object> result = kinesisService.getRecordsForAccount(
- esm.getAccountId(), iterator, esm.getBatchSize(), esm.getRegion());
- List<KinesisRecord> records = (List<KinesisRecord>) result.get("Records");
-
- if (records.isEmpty()) {
- continue;
- }
-
- // The checkpoint must advance to the newest FETCHED record whenever the batch is
- // disposed of, whether by a successful invoke or because a filter matched nothing,
- // so filtered-out records are consumed, not re-read forever. Only an invoke that was
- // attempted and failed leaves the checkpoint unmoved (the whole window retries), and
- // reported batch item failures move it only up to the lowest failed record.
- String newestFetchedSeq = records.get(records.size() - 1).getSequenceNumber();
-
- List<KinesisRecord> matched = records;
- JsonNode filterParams = EsmFilterCriteriaUtils.matcherSourceParameters(objectMapper, esm.getFilterCriteria());
- if (filterParams != null) {
- List<JsonNode> filterNodes = new ArrayList<>(records.size());
- for (KinesisRecord rec : records) {
- filterNodes.add(buildKinesisFilterNode(rec));
- }
- matched = EsmFilterCriteriaUtils.selectMatched(
- records, filterNodes, filterMatcher.applyFilterCriteria(filterNodes, filterParams));
- }
-
- if (matched.isEmpty()) {
- // Whole batch filtered out: consume it (advance past), do not invoke, do not retry.
- advanceCheckpoint(esm, shard.getShardId(), newestFetchedSeq);
- continue;
- }
-
- String eventJson = buildKinesisEvent(matched, esm, shard.getShardId());
- InvokeResult invokeResult;
- try {
- String qualifier = LambdaTargetResolver.mappingQualifier(esm);
- invokeResult = qualifier == null
- ? executorService.invoke(fn, eventJson.getBytes(), InvocationType.RequestResponse)
- : executorService.invoke(fn, eventJson.getBytes(), InvocationType.RequestResponse, 0, qualifier);
- } catch (AwsException e) {
- if ("TooManyRequestsException".equals(e.getErrorCode())) {
- LOG.infov("Kinesis ESM {0}: function {1} throttled, shard iterator not advanced",
- esm.getUuid(), fn.getFunctionName());
- continue;
- }
- throw e;
- }
-
- if (invokeResult.getFunctionError() == null) {
- String checkpoint = successfulInvocationCheckpoint(
- esm, invokeResult.getPayload(), lastSeq, records, matched);
- if (checkpoint != null && !checkpoint.equals(lastSeq)) {
- advanceCheckpoint(esm, shard.getShardId(), checkpoint);
- }
- }
+ pollShard(esm, fn, streamName, shard.getShardId());
}
} catch (Exception e) {
LOG.warnv("Kinesis ESM {0} error: {1}", esm.getUuid(), e.getMessage());
@@ -202,6 +191,201 @@ public class KinesisEventSourcePoller implements Resettable {
});
}
+ private void pollShard(EventSourceMapping esm, LambdaFunction fn, String streamName, String shardId) {
+ String lastSeq = esm.getShardSequenceNumbers().get(shardId);
+ String batchKey = StreamBatchFailures.batchStateKey(esm.getUuid(), shardId, lastSeq);
+ if (clock.getAsLong() < retryNotBefore.getOrDefault(batchKey, 0L)) {
+ return;
+ }
+
+ String iterator;
+ if (lastSeq == null) {
+ iterator = kinesisService.getShardIteratorForAccount(esm.getAccountId(),
+ streamName, shardId, "TRIM_HORIZON", null, esm.getRegion());
+ } else {
+ iterator = kinesisService.getShardIteratorForAccount(esm.getAccountId(),
+ streamName, shardId, "AFTER_SEQUENCE_NUMBER", lastSeq, esm.getRegion());
+ }
+
+ Map<String, Object> result = kinesisService.getRecordsForAccount(
+ esm.getAccountId(), iterator, esm.getBatchSize(), esm.getRegion());
+ List<KinesisRecord> records = (List<KinesisRecord>) result.get("Records");
+
+ if (records.isEmpty()) {
+ return;
+ }
+
+ // The checkpoint must advance to the newest FETCHED record whenever the batch is
+ // disposed of: by a successful invoke, because a filter matched nothing, or because the
+ // batch was discarded after its retries or its record age ran out. Filtered-out records are
+ // consumed, not re-read forever. A failed invoke that may still be retried leaves the
+ // checkpoint unmoved, and reported batch item failures move it only up to the lowest failed
+ // record.
+ String newestFetchedSeq = records.get(records.size() - 1).getSequenceNumber();
+
+ List<KinesisRecord> matched = records;
+ JsonNode filterParams = EsmFilterCriteriaUtils.matcherSourceParameters(objectMapper, esm.getFilterCriteria());
+ if (filterParams != null) {
+ List<JsonNode> filterNodes = new ArrayList<>(records.size());
+ for (KinesisRecord rec : records) {
+ filterNodes.add(buildKinesisFilterNode(rec));
+ }
+ matched = EsmFilterCriteriaUtils.selectMatched(
+ records, filterNodes, filterMatcher.applyFilterCriteria(filterNodes, filterParams));
+ }
+
+ if (matched.isEmpty()) {
+ // Whole batch filtered out: consume it (advance past), do not invoke, do not retry.
+ clearBatchState(batchKey);
+ advanceCheckpoint(esm, shardId, newestFetchedSeq);
+ return;
+ }
+
+ String eventJson = buildKinesisEvent(matched, esm, shardId);
+ InvokeResult invokeResult;
+ try {
+ String qualifier = LambdaTargetResolver.mappingQualifier(esm);
+ invokeResult = qualifier == null
+ ? executorService.invoke(fn, eventJson.getBytes(), InvocationType.RequestResponse)
+ : executorService.invoke(fn, eventJson.getBytes(), InvocationType.RequestResponse, 0, qualifier);
+ } catch (AwsException e) {
+ if (!"TooManyRequestsException".equals(e.getErrorCode())) {
+ throw e;
+ }
+ // A throttled batch is retried until it is older than MaximumRecordAgeInSeconds, as on AWS.
+ if (stillActive(esm) && StreamBatchFailures.maximumRecordAgeExceeded(
+ esm, matched.get(0).getApproximateArrivalTimestamp(), clock.getAsLong())) {
+ discard(esm, shardId, batchKey, matched, new InvokeResult(429, null, null, null, null),
+ retryCounts.getOrDefault(batchKey, 0), StreamBatchFailures.MAXIMUM_RECORD_AGE_EXCEEDED,
+ newestFetchedSeq);
+ return;
+ }
+ LOG.infov("Kinesis ESM {0}: function {1} throttled, shard iterator not advanced",
+ esm.getUuid(), fn.getFunctionName());
+ return;
+ }
+
+ CheckpointOutcome outcome = invokeResult.getFunctionError() == null
+ ? successfulInvocationCheckpoint(esm, invokeResult.getPayload(), lastSeq, records, matched)
+ : new CheckpointOutcome(lastSeq, 0);
+ if (outcome.lowestFailedIndex() == records.size()) {
+ clearBatchState(batchKey);
+ advanceCheckpoint(esm, shardId, outcome.checkpoint());
+ return;
+ }
+ // Records before the lowest reported failure are done: the checkpoint moves up to them, as it
+ // always did, and the attempts of the retried batch are counted under the new checkpoint.
+ String retryKey = StreamBatchFailures.batchStateKey(esm.getUuid(), shardId, outcome.checkpoint());
+ if (!retryKey.equals(batchKey)) {
+ clearBatchState(batchKey);
+ advanceCheckpoint(esm, shardId, outcome.checkpoint());
+ }
+ // A failure is counted, and may discard the batch, only for a mapping that is still polled.
+ if (!stillActive(esm)) {
+ LOG.debugv("Kinesis ESM {0} was stopped during a failed invocation, dropping its result", esm.getUuid());
+ return;
+ }
+
+ // The backoff and the record age count from the end of the failed invoke, so a slow
+ // function is not retried at once.
+ long now = clock.getAsLong();
+ List<KinesisRecord> failed = failedRecords(records, matched, outcome.lowestFailedIndex());
+ int invokeCount = retryCounts.getOrDefault(retryKey, 0) + 1;
+ Optional<String> discardCondition = discardCondition(esm, failed, invokeCount, now);
+ if (discardCondition.isPresent()) {
+ discard(esm, shardId, retryKey, failed, invokeResult, invokeCount, discardCondition.get(),
+ newestFetchedSeq);
+ } else {
+ scheduleRetry(esm, shardId, retryKey, failed.size(), invokeResult, invokeCount, now);
+ }
+ }
+
+ /**
+ * Sends the discarded records to the OnFailure destination, then checkpoints past the batch. The
+ * send can be slow, so the mapping is checked again under the monitor that stopPolling, a reset
+ * and shutdown take: a mapping stopped meanwhile keeps no state and is not written back.
+ */
+ private void discard(EventSourceMapping esm, String shardId, String stateKey, List<KinesisRecord> discarded,
+ InvokeResult invokeResult, int invokeCount, String condition, String newestFetchedSeq) {
+ LOG.warnv("Kinesis ESM {0}: {1} after {2} invocation(s), discarding {3} record(s) of shard {4}",
+ esm.getUuid(), condition, invokeCount, discarded.size(), shardId);
+ batchFailures.sendToOnFailureDestination(esm, discardedBatch(esm, shardId, discarded), invokeResult,
+ invokeCount, condition);
+ synchronized (this) {
+ if (stillActive(esm)) {
+ clearBatchState(stateKey);
+ advanceCheckpoint(esm, shardId, newestFetchedSeq);
+ }
+ }
+ }
+
+ /** Records the attempt and its backoff, unless stopPolling, a reset or shutdown came first. */
+ private synchronized void scheduleRetry(EventSourceMapping esm, String shardId, String retryKey, int records,
+ InvokeResult invokeResult, int invokeCount, long now) {
+ if (!stillActive(esm)) {
+ return;
+ }
+ long backoffMs = retryBackoffMs(invokeCount);
+ retryCounts.put(retryKey, invokeCount);
+ retryNotBefore.put(retryKey, now + backoffMs);
+ LOG.warnv("Kinesis ESM {0}: invocation {1} failed [{2}], retrying {3} record(s) of shard {4} in {5,number,#} ms",
+ esm.getUuid(), invokeCount,
+ invokeResult.getFunctionError() != null ? invokeResult.getFunctionError() : "batchItemFailures",
+ records, shardId, backoffMs);
+ }
+
+ private record CheckpointOutcome(String checkpoint, int lowestFailedIndex) {}
+
+ long retryBackoffMs(int retries) {
+ return StreamBatchFailures.retryBackoffMs(pollIntervalMs, retries);
+ }
+
+ private static Optional<String> discardCondition(EventSourceMapping esm, List<KinesisRecord> failed,
+ int invokeCount, long now) {
+ if (StreamBatchFailures.maximumRecordAgeExceeded(esm, failed.get(0).getApproximateArrivalTimestamp(), now)) {
+ return Optional.of(StreamBatchFailures.MAXIMUM_RECORD_AGE_EXCEEDED);
+ }
+ if (StreamBatchFailures.retryAttemptsExhausted(esm, invokeCount)) {
+ return Optional.of(StreamBatchFailures.RETRY_ATTEMPTS_EXHAUSTED);
+ }
+ return Optional.empty();
+ }
+
+ /** The delivered records from the lowest failed one on. A filtered-out record was never sent, so it did not fail. */
+ private static List<KinesisRecord> failedRecords(List<KinesisRecord> fetched, List<KinesisRecord> delivered,
+ int lowestFailedIndex) {
+ Set<String> retried = new HashSet<>();
+ for (KinesisRecord rec : fetched.subList(lowestFailedIndex, fetched.size())) {
+ retried.add(rec.getSequenceNumber());
+ }
+ return delivered.stream().filter(rec -> retried.contains(rec.getSequenceNumber())).toList();
+ }
+
+ private StreamBatchFailures.DiscardedBatch discardedBatch(EventSourceMapping esm, String shardId,
+ List<KinesisRecord> records) {
+ KinesisRecord first = records.get(0);
+ KinesisRecord last = records.get(records.size() - 1);
+ return new StreamBatchFailures.DiscardedBatch(BATCH_INFO_NAME, shardId,
+ first.getSequenceNumber(), last.getSequenceNumber(),
+ AsyncInvokeDestinationRouter.formatRecordTimestamp(first.getApproximateArrivalTimestamp()),
+ AsyncInvokeDestinationRouter.formatRecordTimestamp(last.getApproximateArrivalTimestamp()),
+ records.size(), streamArnFromArn(esm.getEventSourceArn()),
+ () -> buildKinesisEvent(records, esm, shardId));
+ }
+
+ /** False once the mapping is stopped, deleted or disabled, or the poller is reset or shut down. */
+ private boolean stillActive(EventSourceMapping esm) {
+ return !quiesced && !stopped.contains(esm.getUuid())
+ && esmStore.getForAccount(esm.getAccountId(), esm.getUuid())
+ .map(EventSourceMapping::isEnabled)
+ .orElse(false);
+ }
+
+ private void clearBatchState(String key) {
+ retryCounts.remove(key);
+ retryNotBefore.remove(key);
+ }
+
private String buildKinesisEvent(List<KinesisRecord> records, EventSourceMapping esm, String shardId) {
try {
ArrayNode recordsArray = objectMapper.createArrayNode();
@@ -258,22 +442,25 @@ public class KinesisEventSourcePoller implements Resettable {
}
/**
- * The sequence number to checkpoint after a successful invocation. With
+ * The sequence number to checkpoint after a successful invocation, and the fetched index of the
+ * lowest failed record ({@code fetched.size()} when none failed). With
* {@code ReportBatchItemFailures} the shard resumes at the lowest reported failure; Floci
* resumes {@code AFTER_SEQUENCE_NUMBER}, so that is the fetched record just before it. A
* malformed {@code batchItemFailures} response retries the whole batch, as on AWS.
*/
- private String successfulInvocationCheckpoint(EventSourceMapping esm, byte[] payload, String previousCheckpoint,
- List<KinesisRecord> fetched, List<KinesisRecord> delivered) {
+ private CheckpointOutcome successfulInvocationCheckpoint(EventSourceMapping esm, byte[] payload,
+ String previousCheckpoint,
+ List<KinesisRecord> fetched,
+ List<KinesisRecord> delivered) {
String newestFetchedSeq = fetched.get(fetched.size() - 1).getSequenceNumber();
if (!esm.isReportBatchItemFailures() || payload == null || payload.length == 0) {
- return newestFetchedSeq;
+ return new CheckpointOutcome(newestFetchedSeq, fetched.size());
}
try {
JsonNode failures = objectMapper.readTree(payload).get("batchItemFailures");
if (failures == null || failures.isNull()) {
- return newestFetchedSeq;
+ return new CheckpointOutcome(newestFetchedSeq, fetched.size());
}
if (!failures.isArray()) {
return retryWholeBatch(esm, previousCheckpoint, "batchItemFailures is not an array");
@@ -305,22 +492,23 @@ public class KinesisEventSourcePoller implements Resettable {
}
if (lowestFailedIndex == fetched.size()) {
- return newestFetchedSeq;
+ return new CheckpointOutcome(newestFetchedSeq, fetched.size());
}
LOG.warnv("Kinesis ESM {0}: function reported batch item failures, resuming at {1}",
esm.getUuid(), fetched.get(lowestFailedIndex).getSequenceNumber());
- return lowestFailedIndex == 0
+ String checkpoint = lowestFailedIndex == 0
? previousCheckpoint
: fetched.get(lowestFailedIndex - 1).getSequenceNumber();
+ return new CheckpointOutcome(checkpoint, lowestFailedIndex);
} catch (Exception e) {
return retryWholeBatch(esm, previousCheckpoint, "response is not valid JSON: " + e.getMessage());
}
}
- private String retryWholeBatch(EventSourceMapping esm, String previousCheckpoint, String reason) {
+ private CheckpointOutcome retryWholeBatch(EventSourceMapping esm, String previousCheckpoint, String reason) {
LOG.warnv("Kinesis ESM {0}: malformed batchItemFailures response ({1}), retrying the whole batch",
esm.getUuid(), reason);
- return previousCheckpoint;
+ return new CheckpointOutcome(previousCheckpoint, 0);
}
private void advanceCheckpoint(EventSourceMapping esm, String shardId, String newestSeq) {
@@ -328,6 +516,12 @@ public class KinesisEventSourcePoller implements Resettable {
esmStore.saveForAccount(esm.getAccountId(), esm);
}
+ /** The stream ARN of a stream or stream-consumer ARN: an enhanced fan-out mapping reads its stream. */
+ static String streamArnFromArn(String arn) {
+ int consumer = arn.indexOf("/consumer/");
+ return consumer >= 0 ? arn.substring(0, consumer) : arn;
+ }
+
/**
* The stream name from a stream ARN ({@code .../stream/S}) or a stream-consumer ARN
* ({@code .../stream/S/consumer/C:123}): the segment right after {@code stream/}. Only a
diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java
index ee27bfc767..e6e388e953 100644
--- a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java
@@ -2231,6 +2231,8 @@ public class LambdaService implements ResourceProvider {
esmStore.delete(uuid);
if (esm.getEventSourceArn() != null && esm.getEventSourceArn().contains(":dynamodb:")) {
dynamodbStreamsPoller.mappingDeleted(uuid);
+ } else if (esm.getEventSourceArn() != null && esm.getEventSourceArn().contains(":kinesis:")) {
+ kinesisPoller.mappingDeleted(uuid);
}
LOG.infov("Deleted ESM {0}", uuid);
}
diff --git a/src/test/java/io/github/hectorvent/floci/lifecycle/EmulatorLifecycleTest.java b/src/test/java/io/github/hectorvent/floci/lifecycle/EmulatorLifecycleTest.java
index e160ca834c..3e5f1a2610 100644
--- a/src/test/java/io/github/hectorvent/floci/lifecycle/EmulatorLifecycleTest.java
+++ b/src/test/java/io/github/hectorvent/floci/lifecycle/EmulatorLifecycleTest.java
@@ -216,12 +216,13 @@ class EmulatorLifecycleTest {
}
@Test
- void stopsDynamoDbStreamConsumersBeforeTheFlushAndTheBackendBeforeStorageShutdown() {
+ void stopsStreamConsumersBeforeTheFlushAndTheBackendBeforeStorageShutdown() {
emulatorLifecycle.onStop(Mockito.mock(ShutdownEvent.class));
- InOrder inOrder = Mockito.inOrder(dynamodbStreamsPoller, redshiftZeroEtlConsumer, storageFactory,
- flociUiManager, dynamoDbRuntime);
+ InOrder inOrder = Mockito.inOrder(dynamodbStreamsPoller, kinesisPoller, redshiftZeroEtlConsumer,
+ storageFactory, flociUiManager, dynamoDbRuntime);
inOrder.verify(dynamodbStreamsPoller).shutdown();
+ inOrder.verify(kinesisPoller).shutdown();
inOrder.verify(redshiftZeroEtlConsumer).shutdown();
inOrder.verify(storageFactory).flushAll();
inOrder.verify(flociUiManager).shutdown();
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/KinesisEsmOnFailureDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/KinesisEsmOnFailureDockerIntegrationTest.java
new file mode 100644
index 0000000000..f2dab9dda0
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/KinesisEsmOnFailureDockerIntegrationTest.java
@@ -0,0 +1,169 @@
+package io.github.hectorvent.floci.services.lambda;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.github.dockerjava.api.DockerClient;
+import io.github.hectorvent.floci.core.common.RequestScopes;
+import io.github.hectorvent.floci.services.kinesis.KinesisService;
+import io.github.hectorvent.floci.services.sqs.SqsService;
+import io.github.hectorvent.floci.testing.DockerAvailability;
+import io.github.hectorvent.floci.testing.SqsTestQueue;
+import io.quarkus.test.junit.QuarkusTest;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Base64;
+import java.util.Map;
+import java.util.UUID;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+
+import static io.restassured.RestAssured.given;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assertions.fail;
+
+/**
+ * Runs a real function that throws on a poison record behind a Kinesis mapping with
+ * {@code MaximumRetryAttempts = 1}: the batch is invoked twice, its invocation record reaches the
+ * SQS OnFailure destination, and the shard goes on with the next record.
+ */
+@QuarkusTest
+class KinesisEsmOnFailureDockerIntegrationTest {
+
+ private static final String BASE_PATH = "/2015-03-31";
+ private static final String REGION = "us-east-1";
+ private static final String ACCOUNT = "000000000000";
+ private static final String ROLE = "arn:aws:iam::" + ACCOUNT + ":role/lambda-role";
+ private static final Duration DEADLINE = Duration.ofSeconds(90);
+
+ @Inject
+ DockerClient dockerClient;
+
+ @Inject
+ SqsService sqsService;
+
+ @Inject
+ KinesisService kinesisService;
+
+ @Inject
+ EsmStore esmStore;
+
+ @BeforeEach
+ void requireDocker() {
+ Assumptions.assumeTrue(DockerAvailability.answers(dockerClient),
+ "Docker daemon must be available for the Kinesis OnFailure integration test");
+ }
+
+ @Test
+ void exhaustedBatchReachesTheSqsDestinationAndTheShardGoesOn() throws Exception {
+ String suffix = UUID.randomUUID().toString().substring(0, 8);
+ String streamName = "esm-onfailure-" + suffix;
+ String functionName = "esm-onfailure-" + suffix;
+ String streamArn = RequestScopes.callAs(ACCOUNT, () ->
+ kinesisService.createStream(streamName, 1, REGION).getStreamArn());
+ SqsTestQueue queue = SqsTestQueue.create(sqsService, ACCOUNT, REGION, "esm-onfailure-");
+ createFunction(functionName);
+ String uuid = null;
+ try {
+ String poisonSequence = putRecord(streamName, "poison");
+ uuid = given()
+ .contentType("application/json")
+ .body("""
+ {
+ "FunctionName": "%s",
+ "EventSourceArn": "%s",
+ "StartingPosition": "TRIM_HORIZON",
+ "MaximumRetryAttempts": 1,
+ "DestinationConfig": {"OnFailure": {"Destination": "%s"}}
+ }
+ """.formatted(functionName, streamArn, queue.arn()))
+ .when()
+ .post(BASE_PATH + "/event-source-mappings/")
+ .then()
+ .statusCode(202)
+ .extract().path("UUID");
+
+ JsonNode record = queue.awaitJson(DEADLINE);
+
+ assertEquals("RetryAttemptsExhausted", record.at("/requestContext/condition").asText(), record.toString());
+ assertEquals(2, record.at("/requestContext/approximateInvokeCount").asInt());
+ assertEquals("Unhandled", record.at("/responseContext/functionError").asText());
+ assertEquals(poisonSequence, record.at("/KinesisBatchInfo/startSequenceNumber").asText());
+ assertEquals(poisonSequence, record.at("/KinesisBatchInfo/endSequenceNumber").asText());
+ assertEquals(1, record.at("/KinesisBatchInfo/batchSize").asInt());
+ assertEquals(streamArn, record.at("/KinesisBatchInfo/streamArn").asText());
+
+ String goodSequence = putRecord(streamName, "good");
+ awaitCheckpoint(uuid, goodSequence);
+ assertTrue(queue.receive().isEmpty(), "the good record must not reach the OnFailure destination");
+ } finally {
+ if (uuid != null) {
+ given().when().delete(BASE_PATH + "/event-source-mappings/" + uuid);
+ }
+ given().when().delete(BASE_PATH + "/functions/" + functionName);
+ RequestScopes.runAs(ACCOUNT, () -> kinesisService.deleteStream(streamName, REGION));
+ queue.delete();
+ }
+ }
+
+ private String putRecord(String streamName, String data) {
+ return RequestScopes.callAs(ACCOUNT, () ->
+ kinesisService.putRecord(streamName, data.getBytes(StandardCharsets.UTF_8), "pk", REGION));
+ }
+
+ private void createFunction(String functionName) throws Exception {
+ given()
+ .contentType("application/json")
+ .body("""
+ {
+ "FunctionName": "%s",
+ "Runtime": "nodejs20.x",
+ "Role": "%s",
+ "Handler": "index.handler",
+ "Timeout": 10,
+ "Code": {"ZipFile": "%s"}
+ }
+ """.formatted(functionName, ROLE, poisonRaisingZipBase64()))
+ .when()
+ .post(BASE_PATH + "/functions")
+ .then()
+ .statusCode(201);
+ }
+
+ private void awaitCheckpoint(String uuid, String sequenceNumber) throws InterruptedException {
+ long deadline = System.nanoTime() + DEADLINE.toNanos();
+ Map<String, String> checkpoints = Map.of();
+ while (System.nanoTime() < deadline) {
+ checkpoints = esmStore.get(uuid).orElseThrow().getShardSequenceNumbers();
+ if (checkpoints.containsValue(sequenceNumber)) {
+ return;
+ }
+ Thread.sleep(250);
+ }
+ fail("the shard was not checkpointed at " + sequenceNumber + " within " + DEADLINE.toSeconds()
+ + " s, checkpoints: " + checkpoints);
+ }
+
+ private static String poisonRaisingZipBase64() throws Exception {
+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+ try (ZipOutputStream zip = new ZipOutputStream(bytes)) {
+ zip.putNextEntry(new ZipEntry("index.js"));
+ zip.write("""
+ exports.handler = async (event) => {
+ for (const record of event.Records) {
+ if (Buffer.from(record.kinesis.data, 'base64').toString() === 'poison') {
+ throw new Error('poison record');
+ }
+ }
+ };
+ """.getBytes(StandardCharsets.UTF_8));
+ zip.closeEntry();
+ }
+ return Base64.getEncoder().encodeToString(bytes.toByteArray());
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/KinesisEventSourcePollerTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/KinesisEventSourcePollerTest.java
index 1f6ba8152f..606b726762 100644
--- a/src/test/java/io/github/hectorvent/floci/services/lambda/KinesisEventSourcePollerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/KinesisEventSourcePollerTest.java
@@ -14,30 +14,47 @@ import io.github.hectorvent.floci.services.lambda.model.InvokeResult;
import io.github.hectorvent.floci.services.lambda.model.LambdaAlias;
import io.github.hectorvent.floci.services.lambda.model.LambdaFunction;
import io.github.hectorvent.floci.services.pipes.PipesFilterMatcher;
+import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.services.sns.SnsService;
+import io.github.hectorvent.floci.services.sqs.SqsService;
import io.vertx.core.Vertx;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.NullSource;
+import org.junit.jupiter.params.provider.ValueSource;
import org.mockito.ArgumentCaptor;
import java.nio.charset.StandardCharsets;
+import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Base64;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Function;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.ArgumentMatchers.isNull;
import static org.mockito.Mockito.atLeast;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.timeout;
+import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoInteractions;
import static org.mockito.Mockito.when;
/**
@@ -49,6 +66,8 @@ import static org.mockito.Mockito.when;
* signal via Mockito {@code timeout(...)}: {@code esmStore.saveForAccount} for a checkpoint advance, or the
* invoke itself. {@code never()} assertions target actions the exercised code path never performs on any thread
* (invoke in a fully-filtered path; save in an invoke-error path), so they are timing-independent.
+ * Tests that drive several polls in a row wait for each one with {@link #pollOnce}, and move the injected
+ * clock past the retry backoff themselves.
*/
class KinesisEventSourcePollerTest {
@@ -57,6 +76,13 @@ class KinesisEventSourcePollerTest {
private static final String REGION = "us-east-1";
private static final String STREAM_ARN = "arn:aws:kinesis:us-east-1:000000000000:stream/s";
private static final String SHARD = "shardId-000000000000";
+ private static final String FUNCTION_ARN = "arn:aws:lambda:us-east-1:000000000000:function:fn";
+ private static final String QUEUE_ARN = "arn:aws:sqs:us-east-1:000000000000:k-dlq";
+ private static final String QUEUE_URL = "http://localhost:4566/000000000000/k-dlq";
+ private static final String TOPIC_ARN = "arn:aws:sns:us-east-1:000000000000:k-failures";
+ private static final String BUCKET_ARN = "arn:aws:s3:::k-failures";
+ /** The first record of {@link #stubShard}; each later record arrives one second after the one before. */
+ private static final Instant FIRST_ARRIVAL = Instant.parse("2026-10-20T12:07:26.012Z");
private KinesisEventSourcePoller poller;
private KinesisService kinesisService;
@@ -64,24 +90,39 @@ class KinesisEventSourcePollerTest {
private LambdaFunctionStore functionStore;
private LambdaAliasStore aliasStore;
private EsmStore esmStore;
+ private SqsService sqsService;
+ private SnsService snsService;
+ private S3Service s3Service;
+ private final AtomicLong clock = new AtomicLong();
+ /** What the store holds for the mapping while a poll runs; a test changes it to delete or disable it. */
+ private final AtomicReference<Optional<EventSourceMapping>> storedMapping = new AtomicReference<>();
@BeforeEach
void setUp() {
+ clock.set(FIRST_ARRIVAL.plusSeconds(30).toEpochMilli());
EmulatorConfig config = mock(EmulatorConfig.class);
EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);
EmulatorConfig.LambdaServiceConfig lambdaConfig = mock(EmulatorConfig.LambdaServiceConfig.class);
when(config.services()).thenReturn(services);
when(services.lambda()).thenReturn(lambdaConfig);
when(lambdaConfig.pollIntervalMs()).thenReturn(1000L);
+ when(config.effectiveBaseUrl()).thenReturn("http://localhost:4566");
kinesisService = mock(KinesisService.class);
executorService = mock(LambdaExecutorService.class);
functionStore = mock(LambdaFunctionStore.class);
aliasStore = mock(LambdaAliasStore.class);
esmStore = mock(EsmStore.class);
+ storedMapping.set(Optional.of(storedAs(true)));
+ when(esmStore.getForAccount(anyString(), anyString())).thenAnswer(inv -> storedMapping.get());
+ sqsService = mock(SqsService.class);
+ snsService = mock(SnsService.class);
+ s3Service = mock(S3Service.class);
poller = new KinesisEventSourcePoller(mock(Vertx.class), kinesisService, executorService,
- new LambdaTargetResolver(functionStore, aliasStore), esmStore, config, MAPPER, new PipesFilterMatcher(MAPPER));
+ new LambdaTargetResolver(functionStore, aliasStore), esmStore, config, MAPPER,
+ new PipesFilterMatcher(MAPPER), new StreamBatchFailures(sqsService, snsService, s3Service, MAPPER, config),
+ clock::get);
}
private EventSourceMapping esm(String... patterns) {
@@ -308,7 +349,7 @@ class KinesisEventSourcePollerTest {
}
@Test
- void invokeErrorDoesNotAdvanceCheckpointAndRetriesSameWindow() {
+ void invokeErrorDoesNotAdvanceCheckpointAndRetriesSameWindow() throws Exception {
stubFunction();
stubStreamWith(List.of(record("s1", "p1", "{\"type\":\"order\"}")));
InvokeResult err = new InvokeResult();
@@ -316,20 +357,18 @@ class KinesisEventSourcePollerTest {
when(executorService.invoke(any(), any(), eq(InvocationType.RequestResponse))).thenReturn(err);
EventSourceMapping esm = esm("{\"data\":{\"type\":[\"order\"]}}");
- poller.pollAndInvoke(esm);
- verify(executorService, timeout(2000)).invoke(any(), any(), eq(InvocationType.RequestResponse));
- // Barrier: a 2nd poll ran (so poll 1 finished). Since the checkpoint never advanced, both polls
- // re-derive TRIM_HORIZON and fetch the same window.
- awaitPollCompletedViaSecondFetch(esm);
+ pollOnce(esm);
+ clock.addAndGet(poller.retryBackoffMs(1));
+ pollOnce(esm);
verify(esmStore, never()).saveForAccount(eq(ACCOUNT), any());
assertNull(esm.getShardSequenceNumbers().get(SHARD));
// retry-from-old-checkpoint: both invokes carried the same fetched window.
ArgumentCaptor<byte[]> cap = ArgumentCaptor.forClass(byte[].class);
- verify(executorService, timeout(2000).atLeast(2)).invoke(any(), cap.capture(), eq(InvocationType.RequestResponse));
+ verify(executorService, times(2)).invoke(any(), cap.capture(), eq(InvocationType.RequestResponse));
List<byte[]> payloads = cap.getAllValues();
assertEquals(new String(payloads.get(0), StandardCharsets.UTF_8),
- new String(payloads.get(payloads.size() - 1), StandardCharsets.UTF_8));
+ new String(payloads.get(1), StandardCharsets.UTF_8));
}
@Test
@@ -377,33 +416,33 @@ class KinesisEventSourcePollerTest {
}
@Test
- void reportedFailureOnTheFirstRecordRetriesTheWholeBatch() {
- stubFunction();
- stubStreamWith(List.of(record("s1", "p1", "{}"), record("s2", "p2", "{}")));
- when(executorService.invoke(any(), any(), eq(InvocationType.RequestResponse))).thenReturn(
- batchItemFailures("{\"batchItemFailures\":[{\"itemIdentifier\":\"s1\"}]}"));
+ void reportedFailureOnTheFirstRecordRetriesTheWholeBatch() throws Exception {
+ stubShard("s1", "s2");
+ List<byte[]> events = recordInvocations(seqs -> batchItemFailures(
+ "{\"batchItemFailures\":[{\"itemIdentifier\":\"s1\"}]}"));
EventSourceMapping esm = reportingEsm();
- poller.pollAndInvoke(esm);
- verify(executorService, timeout(2000)).invoke(any(), any(), eq(InvocationType.RequestResponse));
- awaitPollCompletedViaSecondFetch(esm);
+ pollOnce(esm);
+ clock.addAndGet(poller.retryBackoffMs(1));
+ pollOnce(esm);
+ assertEquals(List.of(List.of("s1", "s2"), List.of("s1", "s2")), invokedSequences(events));
verify(esmStore, never()).saveForAccount(eq(ACCOUNT), any());
assertNull(esm.getShardSequenceNumbers().get(SHARD));
}
@Test
- void unknownItemIdentifierRetriesTheWholeBatch() {
- stubFunction();
- stubStreamWith(List.of(record("s1", "p1", "{}"), record("s2", "p2", "{}")));
- when(executorService.invoke(any(), any(), eq(InvocationType.RequestResponse))).thenReturn(
- batchItemFailures("{\"batchItemFailures\":[{\"itemIdentifier\":\"nope\"}]}"));
+ void unknownItemIdentifierRetriesTheWholeBatch() throws Exception {
+ stubShard("s1", "s2");
+ List<byte[]> events = recordInvocations(seqs -> batchItemFailures(
+ "{\"batchItemFailures\":[{\"itemIdentifier\":\"nope\"}]}"));
EventSourceMapping esm = reportingEsm();
- poller.pollAndInvoke(esm);
- verify(executorService, timeout(2000)).invoke(any(), any(), eq(InvocationType.RequestResponse));
- awaitPollCompletedViaSecondFetch(esm);
+ pollOnce(esm);
+ clock.addAndGet(poller.retryBackoffMs(1));
+ pollOnce(esm);
+ assertEquals(List.of(List.of("s1", "s2"), List.of("s1", "s2")), invokedSequences(events));
verify(esmStore, never()).saveForAccount(eq(ACCOUNT), any());
assertNull(esm.getShardSequenceNumbers().get(SHARD));
}
@@ -479,4 +518,585 @@ class KinesisEventSourcePollerTest {
assertEquals("orders/not-a-consumer", KinesisEventSourcePoller.streamNameFromArn(
"arn:aws:kinesis:us-east-1:000000000000:stream/orders/not-a-consumer"));
}
+
+ // ─────────── MaximumRetryAttempts, MaximumRecordAgeInSeconds and DestinationConfig.OnFailure ───────────
+
+ @Test
+ void maximumRetryAttemptsAllowsExactlyNPlusOneInvocationsThenDiscardsOnceAndMovesOn() throws Exception {
+ List<KinesisRecord> shard = stubShard("s1");
+ List<byte[]> events = failInvocationsContaining("s1");
+ EventSourceMapping esm = esmWithOnFailure(10, QUEUE_ARN);
+
+ for (int invocation = 1; invocation <= 10; invocation++) {
+ pollOnce(esm);
+ assertEquals(invocation, events.size(), "invocations after poll " + invocation);
+ verify(sqsService, never()).sendMessage(anyString(), anyString(), anyInt(), anyString());
+ assertNull(checkpoint(esm), "a batch with retries left must not be checkpointed");
+ clock.addAndGet(StreamBatchFailures.MAX_RETRY_BACKOFF_MS);
+ }
+ pollOnce(esm);
+
+ assertEquals(11, events.size());
+ verify(sqsService, times(1)).sendMessage(eq(QUEUE_URL), anyString(), eq(0), eq(REGION));
+ assertEquals("s1", checkpoint(esm), "the discarded batch must be checkpointed");
+
+ shard.add(record("s2", FIRST_ARRIVAL.plusSeconds(1)));
+ // No clock move: the discarded batch's backoff must not hold back the next records.
+ pollOnce(esm);
+
+ assertEquals(12, events.size());
+ assertEquals(List.of("s2"), sequences(events.get(11)));
+ assertEquals("s2", checkpoint(esm));
+ verify(sqsService, times(1)).sendMessage(anyString(), anyString(), anyInt(), anyString());
+ }
+
+ @Test
+ void zeroMaximumRetryAttemptsDiscardsAfterOneInvocationAndSendsTheAwsRecordToSqs() throws Exception {
+ stubShard("s1");
+ List<byte[]> events = failInvocationsContaining("s1");
+
+ EventSourceMapping esm = esmWithOnFailure(0, QUEUE_ARN);
+ pollOnce(esm);
+
+ assertEquals(1, events.size());
+ assertEquals("s1", checkpoint(esm));
+ ArgumentCaptor<String> body = ArgumentCaptor.forClass(String.class);
+ verify(sqsService, times(1)).sendMessage(eq(QUEUE_URL), body.capture(), eq(0), eq(REGION));
+ JsonNode record = MAPPER.readTree(body.getValue());
+ assertEquals(List.of("requestContext", "responseContext", "version", "timestamp", "KinesisBatchInfo"),
+ fieldNames(record), "SQS gets the metadata only, no payload: " + record);
+ assertEquals("req-1", record.at("/requestContext/requestId").asText());
+ assertEquals(FUNCTION_ARN, record.at("/requestContext/functionArn").asText());
+ assertEquals("RetryAttemptsExhausted", record.at("/requestContext/condition").asText());
+ assertEquals(1, record.at("/requestContext/approximateInvokeCount").asInt());
+ assertEquals(200, record.at("/responseContext/statusCode").asInt());
+ assertEquals("$LATEST", record.at("/responseContext/executedVersion").asText());
+ assertEquals("Unhandled", record.at("/responseContext/functionError").asText());
+ assertEquals("1.0", record.path("version").asText());
+ assertTrue(record.path("timestamp").asText().matches("\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z"),
+ "timestamp must be ISO-8601 with milliseconds: " + record.path("timestamp"));
+ assertBatchInfo(record, "s1", "s1", "2026-10-20T12:07:26.012Z", "2026-10-20T12:07:26.012Z", 1);
+ }
+
+ @Test
+ void snsOnFailureDestinationGetsTheRecordWithoutPayload() throws Exception {
+ stubShard("s1");
+ failInvocationsContaining("s1");
+
+ pollOnce(esmWithOnFailure(0, TOPIC_ARN));
+
+ ArgumentCaptor<String> message = ArgumentCaptor.forClass(String.class);
+ verify(snsService, times(1)).publish(eq(TOPIC_ARN), isNull(), message.capture(), eq("ESM OnFailure"), eq(REGION));
+ JsonNode record = MAPPER.readTree(message.getValue());
+ assertFalse(record.has("payload"), "SNS gets the metadata only: " + record);
+ assertEquals("RetryAttemptsExhausted", record.at("/requestContext/condition").asText());
+ assertBatchInfo(record, "s1", "s1", "2026-10-20T12:07:26.012Z", "2026-10-20T12:07:26.012Z", 1);
+ verifyNoInteractions(sqsService, s3Service);
+ }
+
+ @Test
+ void enhancedFanOutMappingReportsTheStreamArnNotTheConsumerArn() throws Exception {
+ stubShard("s1");
+ failInvocationsContaining("s1");
+ EventSourceMapping esm = esmWithOnFailure(0, QUEUE_ARN);
+ esm.setEventSourceArn(STREAM_ARN + "/consumer/reader:1700000000");
+
+ pollOnce(esm);
+
+ ArgumentCaptor<String> body = ArgumentCaptor.forClass(String.class);
+ verify(sqsService, times(1)).sendMessage(eq(QUEUE_URL), body.capture(), eq(0), eq(REGION));
+ assertEquals(STREAM_ARN, MAPPER.readTree(body.getValue()).at("/KinesisBatchInfo/streamArn").asText());
+ }
+
+ @Test
+ void s3OnFailureDestinationGetsTheExactEventAsAPayloadString() throws Exception {
+ stubShard("s1", "s2");
+ List<byte[]> events = failInvocationsContaining("s1");
+ EventSourceMapping esm = esmWithOnFailure(1, BUCKET_ARN);
+
+ pollOnce(esm);
+ clock.addAndGet(StreamBatchFailures.MAX_RETRY_BACKOFF_MS);
+ pollOnce(esm);
+
+ ArgumentCaptor<String> key = ArgumentCaptor.forClass(String.class);
+ ArgumentCaptor<byte[]> body = ArgumentCaptor.forClass(byte[].class);
+ verify(s3Service, times(1)).putObject(eq("k-failures"), key.capture(), body.capture(),
+ eq("application/json"), eq(Map.of()));
+ assertTrue(key.getValue().matches("aws/lambda/k-esm/shardId-000000000000/\\d{4}/\\d{2}/\\d{2}/"
+ + "\\d{4}-\\d{2}-\\d{2}T\\d{2}\\.\\d{2}\\.\\d{2}-[0-9a-f]{8}(-[0-9a-f]{4}){3}-[0-9a-f]{12}"),
+ "unexpected S3 key " + key.getValue());
+ JsonNode record = MAPPER.readTree(body.getValue());
+ assertEquals(List.of("requestContext", "responseContext", "version", "payload", "timestamp", "KinesisBatchInfo"),
+ fieldNames(record));
+ assertEquals(2, record.at("/requestContext/approximateInvokeCount").asInt());
+ assertTrue(record.path("payload").isTextual(), "payload must be a JSON string, not an object: " + record);
+ assertEquals(MAPPER.readTree(events.get(1)), MAPPER.readTree(record.path("payload").asText()));
+ assertBatchInfo(record, "s1", "s2", "2026-10-20T12:07:26.012Z", "2026-10-20T12:07:27.012Z", 2);
+ }
+
+ @Test
+ void reportedBatchItemFailuresUseUpTheRetryAttempts() throws Exception {
+ stubShard("s1", "s2", "s3");
+ List<byte[]> events = recordInvocations(seqs -> batchItemFailures(
+ "{\"batchItemFailures\":[{\"itemIdentifier\":\"s2\"}]}"));
+ EventSourceMapping esm = esmWithOnFailure(1, QUEUE_ARN);
+ esm.setFunctionResponseTypes(List.of("ReportBatchItemFailures"));
+
+ pollOnce(esm);
+ assertEquals("s1", checkpoint(esm), "the records before the lowest failure are done");
+ verify(sqsService, never()).sendMessage(anyString(), anyString(), anyInt(), anyString());
+
+ clock.addAndGet(StreamBatchFailures.MAX_RETRY_BACKOFF_MS);
+ pollOnce(esm);
+
+ assertEquals(List.of(List.of("s1", "s2", "s3"), List.of("s2", "s3")), invokedSequences(events));
+ assertEquals("s3", checkpoint(esm));
+ ArgumentCaptor<String> body = ArgumentCaptor.forClass(String.class);
+ verify(sqsService, times(1)).sendMessage(eq(QUEUE_URL), body.capture(), eq(0), eq(REGION));
+ JsonNode record = MAPPER.readTree(body.getValue());
+ assertEquals("RetryAttemptsExhausted", record.at("/requestContext/condition").asText());
+ assertEquals(2, record.at("/requestContext/approximateInvokeCount").asInt());
+ assertEquals(200, record.at("/responseContext/statusCode").asInt());
+ assertFalse(record.path("responseContext").has("functionError"),
+ "a partial batch response is not a function error: " + record);
+ assertBatchInfo(record, "s2", "s3", "2026-10-20T12:07:27.012Z", "2026-10-20T12:07:28.012Z", 2);
+ }
+
+ @Test
+ void reportedBatchItemFailureWithZeroRetryAttemptsDiscardsTheRestOfTheBatchAtOnce() throws Exception {
+ stubShard("s1", "s2", "s3");
+ List<byte[]> events = recordInvocations(seqs -> batchItemFailures(
+ "{\"batchItemFailures\":[{\"itemIdentifier\":\"s2\"}]}"));
+ EventSourceMapping esm = esmWithOnFailure(0, QUEUE_ARN);
+ esm.setFunctionResponseTypes(List.of("ReportBatchItemFailures"));
+
+ pollOnce(esm);
+
+ assertEquals(1, events.size());
+ assertEquals("s3", checkpoint(esm));
+ ArgumentCaptor<String> body = ArgumentCaptor.forClass(String.class);
+ verify(sqsService, times(1)).sendMessage(eq(QUEUE_URL), body.capture(), eq(0), eq(REGION));
+ JsonNode record = MAPPER.readTree(body.getValue());
+ assertEquals(1, record.at("/requestContext/approximateInvokeCount").asInt());
+ assertBatchInfo(record, "s2", "s3", "2026-10-20T12:07:27.012Z", "2026-10-20T12:07:28.012Z", 2);
+ }
+
+ @Test
+ void batchOlderThanMaximumRecordAgeIsDiscardedWithTheAgeCondition() throws Exception {
+ stubShard("s1");
+ List<byte[]> events = failInvocationsContaining("s1");
+ EventSourceMapping esm = esmWithOnFailure(null, QUEUE_ARN);
+ esm.setMaximumRecordAgeInSeconds(60);
+
+ clock.set(FIRST_ARRIVAL.plusSeconds(58).toEpochMilli());
+ pollOnce(esm);
+ assertNull(checkpoint(esm), "a 58 s old batch is within the 60 s maximum age");
+ verify(sqsService, never()).sendMessage(anyString(), anyString(), anyInt(), anyString());
+
+ clock.set(FIRST_ARRIVAL.plusSeconds(61).toEpochMilli());
+ pollOnce(esm);
+
+ assertEquals(2, events.size());
+ assertEquals("s1", checkpoint(esm));
+ ArgumentCaptor<String> body = ArgumentCaptor.forClass(String.class);
+ verify(sqsService, times(1)).sendMessage(eq(QUEUE_URL), body.capture(), eq(0), eq(REGION));
+ JsonNode record = MAPPER.readTree(body.getValue());
+ assertEquals("MaximumRecordAgeExceeded", record.at("/requestContext/condition").asText());
+ assertEquals(2, record.at("/requestContext/approximateInvokeCount").asInt());
+ }
+
+ @Test
+ void failedBatchIsNotRetriedBeforeItsDoublingBackoffElapses() throws Exception {
+ stubShard("s1");
+ List<byte[]> events = failInvocationsContaining("s1");
+ EventSourceMapping esm = esmWithOnFailure(null, null);
+
+ pollOnce(esm);
+ clock.addAndGet(1_999);
+ pollOnce(esm);
+ assertEquals(1, events.size(), "retried before the first backoff (2 x the 1 s poll interval) elapsed");
+ verify(kinesisService, times(1)).getRecordsForAccount(eq(ACCOUNT), anyString(), anyInt(), eq(REGION));
+ clock.addAndGet(1);
+ pollOnce(esm);
+ assertEquals(2, events.size(), "not retried once the first backoff elapsed");
+
+ clock.addAndGet(3_999);
+ pollOnce(esm);
+ assertEquals(2, events.size(), "retried before the second backoff (4 s) elapsed");
+ clock.addAndGet(1);
+ pollOnce(esm);
+ assertEquals(3, events.size(), "not retried once the second backoff elapsed");
+ }
+
+ @Test
+ void backoffStartsWhenTheFailedInvocationEnds() throws Exception {
+ stubShard("s1");
+ List<byte[]> events = recordInvocations(seqs -> {
+ clock.addAndGet(10_000);
+ return new InvokeResult(200, "Unhandled", new byte[0], null, "req-1");
+ });
+ EventSourceMapping esm = esmWithOnFailure(null, null);
+
+ pollOnce(esm);
+ clock.addAndGet(1_999);
+ pollOnce(esm);
+ assertEquals(1, events.size(), "a function that took 10 s to fail was retried before the 2 s backoff");
+ clock.addAndGet(1);
+ pollOnce(esm);
+ assertEquals(2, events.size());
+ }
+
+ @ParameterizedTest
+ @NullSource
+ @ValueSource(ints = -1)
+ void unlimitedRetryAttemptsAndRecordAgeNeverDiscardAFailingBatch(Integer unlimited) throws Exception {
+ stubShard("s1");
+ List<byte[]> events = failInvocationsContaining("s1");
+ EventSourceMapping esm = esmWithOnFailure(unlimited, QUEUE_ARN);
+ esm.setMaximumRecordAgeInSeconds(unlimited);
+ clock.set(FIRST_ARRIVAL.plus(Duration.ofDays(8)).toEpochMilli());
+
+ for (int poll = 0; poll < 20; poll++) {
+ pollOnce(esm);
+ clock.addAndGet(StreamBatchFailures.MAX_RETRY_BACKOFF_MS);
+ }
+
+ assertEquals(20, events.size());
+ assertNull(checkpoint(esm));
+ verify(esmStore, never()).saveForAccount(anyString(), any());
+ verifyNoInteractions(sqsService);
+ }
+
+ @Test
+ void exhaustedBatchWithoutOnFailureDestinationIsStillDiscarded() throws Exception {
+ List<KinesisRecord> shard = stubShard("s1");
+ List<byte[]> events = failInvocationsContaining("s1");
+ EventSourceMapping esm = esmWithOnFailure(0, null);
+
+ pollOnce(esm);
+ assertEquals("s1", checkpoint(esm));
+ shard.add(record("s2", FIRST_ARRIVAL.plusSeconds(1)));
+ pollOnce(esm);
+
+ assertEquals(List.of(List.of("s1"), List.of("s2")), invokedSequences(events));
+ assertEquals("s2", checkpoint(esm));
+ verifyNoInteractions(sqsService, snsService, s3Service);
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void mappingDeletedOrDisabledDuringAFailedInvokeSendsNothingAndSavesNothing(boolean deleted) throws Exception {
+ stubShard("s1");
+ List<byte[]> events = recordInvocations(seqs -> {
+ storedMapping.set(deleted ? Optional.empty() : Optional.of(storedAs(false)));
+ return new InvokeResult(200, "Unhandled", new byte[0], null, "req-1");
+ });
+ EventSourceMapping esm = esmWithOnFailure(0, QUEUE_ARN);
+
+ pollOnce(esm);
+
+ assertEquals(1, events.size());
+ verifyNoInteractions(sqsService);
+ verify(esmStore, never()).saveForAccount(anyString(), any());
+ assertNull(checkpoint(esm));
+ }
+
+ @Test
+ void mappingDisabledDuringAFailedInvokeKeepsNoAttemptOrBackoffForItsReEnable() throws Exception {
+ stubShard("s1");
+ EventSourceMapping esm = esmWithOnFailure(1, QUEUE_ARN);
+ AtomicBoolean disableDuringInvoke = new AtomicBoolean(true);
+ List<byte[]> events = recordInvocations(seqs -> {
+ if (disableDuringInvoke.getAndSet(false)) {
+ storedMapping.set(Optional.of(storedAs(false)));
+ poller.stopPolling(esm.getUuid());
+ }
+ return new InvokeResult(200, "Unhandled", new byte[0], null, "req-1");
+ });
+
+ pollOnce(esm);
+ storedMapping.set(Optional.of(storedAs(true)));
+ pollOnce(esm);
+
+ assertEquals(2, events.size(), "the re-enabled mapping must not wait out a backoff of the disabled one");
+ verifyNoInteractions(sqsService);
+ assertNull(checkpoint(esm), "the re-enabled mapping must not carry the disabled one's attempt");
+ }
+
+ @Test
+ void shutdownDuringAFailedInvokeKeepsTheBatchForTheNextStart() throws Exception {
+ stubShard("s1");
+ recordInvocations(seqs -> {
+ poller.shutdown();
+ return new InvokeResult(200, "Unhandled", new byte[0], null, "req-1");
+ });
+
+ pollOnce(esmWithOnFailure(0, QUEUE_ARN));
+
+ verifyNoInteractions(sqsService);
+ verify(esmStore, never()).saveForAccount(anyString(), any());
+ }
+
+ @Test
+ void resetDuringAFailedInvokeDropsItsResultAndPollingResumesAfterIt() throws Exception {
+ stubShard("s1");
+ AtomicBoolean resetDuringInvoke = new AtomicBoolean(true);
+ List<byte[]> events = recordInvocations(seqs -> {
+ if (resetDuringInvoke.getAndSet(false)) {
+ poller.beforeReset();
+ }
+ return new InvokeResult(200, "Unhandled", new byte[0], null, "req-1");
+ });
+ EventSourceMapping esm = esmWithOnFailure(0, QUEUE_ARN);
+
+ pollOnce(esm);
+ verifyNoInteractions(sqsService);
+ verify(esmStore, never()).saveForAccount(anyString(), any());
+
+ poller.afterReset();
+ pollOnce(esm);
+
+ assertEquals(2, events.size());
+ verify(sqsService, times(1)).sendMessage(eq(QUEUE_URL), anyString(), eq(0), eq(REGION));
+ assertEquals("s1", checkpoint(esm));
+ }
+
+ @Test
+ void throttledBatchOlderThanMaximumRecordAgeIsDiscardedWithTheAgeCondition() throws Exception {
+ stubShard("s1");
+ when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse)))
+ .thenThrow(new AwsException("TooManyRequestsException", "Rate exceeded", 429));
+ EventSourceMapping esm = esmWithOnFailure(null, QUEUE_ARN);
+ esm.setMaximumRecordAgeInSeconds(60);
+
+ clock.set(FIRST_ARRIVAL.plusSeconds(58).toEpochMilli());
+ pollOnce(esm);
+ assertNull(checkpoint(esm), "a throttled batch within the 60 s maximum age is kept");
+ verify(sqsService, never()).sendMessage(anyString(), anyString(), anyInt(), anyString());
+
+ clock.set(FIRST_ARRIVAL.plusSeconds(61).toEpochMilli());
+ pollOnce(esm);
+
+ assertEquals("s1", checkpoint(esm), "a throttled batch past the maximum age must not block the shard");
+ ArgumentCaptor<String> body = ArgumentCaptor.forClass(String.class);
+ verify(sqsService, times(1)).sendMessage(eq(QUEUE_URL), body.capture(), eq(0), eq(REGION));
+ JsonNode record = MAPPER.readTree(body.getValue());
+ assertEquals("MaximumRecordAgeExceeded", record.at("/requestContext/condition").asText(), record.toString());
+ assertEquals(0, record.at("/requestContext/approximateInvokeCount").asInt(), "the function never ran");
+ assertEquals(429, record.at("/responseContext/statusCode").asInt());
+ assertFalse(record.path("responseContext").has("functionError"), record.toString());
+ assertBatchInfo(record, "s1", "s1", "2026-10-20T12:07:26.012Z", "2026-10-20T12:07:26.012Z", 1);
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"disabled", "shutdown"})
+ void recordsTheFunctionHandledStayCheckpointedWhenTheMappingStopsDuringAPartialResponse(String stop)
+ throws Exception {
+ stubShard("s1", "s2", "s3");
+ recordInvocations(seqs -> {
+ if ("disabled".equals(stop)) {
+ storedMapping.set(Optional.of(storedAs(false)));
+ } else {
+ poller.shutdown();
+ }
+ return batchItemFailures("{\"batchItemFailures\":[{\"itemIdentifier\":\"s2\"}]}");
+ });
+ EventSourceMapping esm = esmWithOnFailure(0, QUEUE_ARN);
+ esm.setFunctionResponseTypes(List.of("ReportBatchItemFailures"));
+
+ pollOnce(esm);
+
+ assertEquals("s1", checkpoint(esm), "s1 was handled by the function and must not be sent to it again");
+ verifyNoInteractions(sqsService);
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"deleted", "reset"})
+ void mappingStoppedDuringTheOnFailureSendIsNotWrittenBack(String stop) throws Exception {
+ stubShard("s1");
+ failInvocationsContaining("s1");
+ when(sqsService.sendMessage(anyString(), anyString(), anyInt(), anyString())).thenAnswer(inv -> {
+ if ("deleted".equals(stop)) {
+ storedMapping.set(Optional.empty());
+ } else {
+ poller.beforeReset();
+ }
+ return null;
+ });
+
+ pollOnce(esmWithOnFailure(0, QUEUE_ARN));
+
+ verify(sqsService, times(1)).sendMessage(eq(QUEUE_URL), anyString(), eq(0), eq(REGION));
+ verify(esmStore, never()).saveForAccount(anyString(), any());
+ }
+
+ @Test
+ void stopPollingDuringAFailedInvokeKeepsNoAttemptBeforeTheStoreDeleteRuns() throws Exception {
+ stubShard("s1");
+ EventSourceMapping esm = esmWithOnFailure(1, QUEUE_ARN);
+ AtomicBoolean stopDuringInvoke = new AtomicBoolean(true);
+ List<byte[]> events = recordInvocations(seqs -> {
+ if (stopDuringInvoke.getAndSet(false)) {
+ // DeleteEventSourceMapping stops polling first; the store still holds the mapping.
+ poller.stopPolling(esm.getUuid());
+ }
+ return new InvokeResult(200, "Unhandled", new byte[0], null, "req-1");
+ });
+
+ pollOnce(esm);
+ poller.startPolling(esm);
+ pollOnce(esm);
+
+ assertEquals(2, events.size(), "a mapping stopped during its invoke must not keep a backoff");
+ verifyNoInteractions(sqsService);
+ }
+
+ @Test
+ void mappingDeletedReleasesItsStopTombstone() {
+ poller.stopPolling("k-esm-gone");
+
+ poller.mappingDeleted("k-esm-gone");
+
+ assertTrue(poller.stopped.isEmpty());
+ }
+
+ @Test
+ void stopPollingForgetsTheAttemptsAndBackoffOfItsMapping() throws Exception {
+ stubShard("s1");
+ List<byte[]> events = failInvocationsContaining("s1");
+ EventSourceMapping esm = esmWithOnFailure(1, QUEUE_ARN);
+
+ pollOnce(esm);
+ poller.stopPolling(esm.getUuid());
+ pollOnce(esm);
+
+ assertEquals(2, events.size(), "a stopped mapping must not keep its backoff");
+ assertNull(checkpoint(esm), "a stopped mapping must not keep its attempt count");
+ verifyNoInteractions(sqsService);
+ }
+
+ private void pollOnce(EventSourceMapping esm) throws InterruptedException {
+ poller.pollAndInvoke(esm);
+ long deadline = System.currentTimeMillis() + 3_000;
+ while (!poller.activePolls.isEmpty() && System.currentTimeMillis() < deadline) {
+ Thread.sleep(5);
+ }
+ assertTrue(poller.activePolls.isEmpty(), "poll did not complete before the test timeout");
+ }
+
+ private static KinesisRecord record(String seq, Instant arrival) {
+ return new KinesisRecord("{}".getBytes(StandardCharsets.UTF_8), "pk-" + seq, seq, arrival);
+ }
+
+ /**
+ * Serves one shard as Kinesis does: TRIM_HORIZON reads from the first record, AFTER_SEQUENCE_NUMBER
+ * after the checkpoint, at most the batch size. Adding to the returned list writes to the shard.
+ */
+ private List<KinesisRecord> stubShard(String... sequences) {
+ List<KinesisRecord> shard = new CopyOnWriteArrayList<>();
+ for (int i = 0; i < sequences.length; i++) {
+ shard.add(record(sequences[i], FIRST_ARRIVAL.plusSeconds(i)));
+ }
+ KinesisStream stream = new KinesisStream("s", STREAM_ARN);
+ stream.setShards(List.of(new KinesisShard(SHARD, "0", "1", "0")));
+ when(kinesisService.describeStreamForAccount(eq(ACCOUNT), anyString(), eq(REGION))).thenReturn(stream);
+ when(kinesisService.getShardIteratorForAccount(eq(ACCOUNT), anyString(), eq(SHARD), anyString(), any(), eq(REGION)))
+ .thenAnswer(inv -> "after:" + (inv.getArgument(4) == null ? "" : inv.getArgument(4)));
+ when(kinesisService.getRecordsForAccount(eq(ACCOUNT), anyString(), anyInt(), eq(REGION))).thenAnswer(inv -> {
+ int from = indexAfter(shard, inv.<String>getArgument(1).substring("after:".length()));
+ int to = Math.min(shard.size(), from + inv.<Integer>getArgument(2));
+ return Map.of("Records", new ArrayList<>(shard.subList(from, to)));
+ });
+ stubFunction();
+ return shard;
+ }
+
+ /** The index of the record after {@code checkpoint}, or 0 (the trim horizon) when no record has it. */
+ private static int indexAfter(List<KinesisRecord> shard, String checkpoint) {
+ for (int i = 0; i < shard.size(); i++) {
+ if (shard.get(i).getSequenceNumber().equals(checkpoint)) {
+ return i + 1;
+ }
+ }
+ return 0;
+ }
+
+ /** Answers every invoke with {@code respond}, given the delivered sequences, and keeps each event. */
+ private List<byte[]> recordInvocations(Function<List<String>, InvokeResult> respond) {
+ List<byte[]> events = new CopyOnWriteArrayList<>();
+ when(executorService.invoke(any(), any(byte[].class), eq(InvocationType.RequestResponse))).thenAnswer(inv -> {
+ byte[] event = inv.getArgument(1);
+ events.add(event);
+ return respond.apply(sequences(event));
+ });
+ return events;
+ }
+
+ /** The function raises, as a handler that throws does: status 200 with an Unhandled function error. */
+ private List<byte[]> failInvocationsContaining(String poison) {
+ return recordInvocations(seqs -> {
+ if (!seqs.contains(poison)) {
+ return new InvokeResult();
+ }
+ InvokeResult error = new InvokeResult(200, "Unhandled",
+ "{\"errorMessage\":\"boom\"}".getBytes(StandardCharsets.UTF_8), null, "req-1");
+ error.setExecutedVersion("$LATEST");
+ return error;
+ });
+ }
+
+ private List<String> sequences(byte[] event) {
+ List<String> sequences = new ArrayList<>();
+ deliveredRecords(event).forEach(r -> sequences.add(r.path("kinesis").path("sequenceNumber").asText()));
+ return sequences;
+ }
+
+ private List<List<String>> invokedSequences(List<byte[]> events) {
+ return events.stream().map(this::sequences).toList();
+ }
+
+ private EventSourceMapping esmWithOnFailure(Integer maximumRetryAttempts, String destinationArn) {
+ EventSourceMapping esm = esm();
+ esm.setFunctionArn(FUNCTION_ARN);
+ esm.setMaximumRetryAttempts(maximumRetryAttempts);
+ if (destinationArn != null) {
+ EventSourceMapping.OnFailure onFailure = new EventSourceMapping.OnFailure();
+ onFailure.setDestination(destinationArn);
+ EventSourceMapping.DestinationConfig destinationConfig = new EventSourceMapping.DestinationConfig();
+ destinationConfig.setOnFailure(onFailure);
+ esm.setDestinationConfig(destinationConfig);
+ }
+ return esm;
+ }
+
+ private static EventSourceMapping storedAs(boolean enabled) {
+ EventSourceMapping stored = new EventSourceMapping();
+ stored.setEnabled(enabled);
+ return stored;
+ }
+
+ private static String checkpoint(EventSourceMapping esm) {
+ return esm.getShardSequenceNumbers().get(SHARD);
+ }
+
+ private static List<String> fieldNames(JsonNode node) {
+ List<String> names = new ArrayList<>();
+ node.fieldNames().forEachRemaining(names::add);
+ return names;
+ }
+
+ private static void assertBatchInfo(JsonNode record, String startSequence, String endSequence,
+ String firstArrival, String lastArrival, int batchSize) {
+ JsonNode batchInfo = record.path("KinesisBatchInfo");
+ assertEquals(List.of("shardId", "startSequenceNumber", "endSequenceNumber", "approximateArrivalOfFirstRecord",
+ "approximateArrivalOfLastRecord", "batchSize", "streamArn"), fieldNames(batchInfo), record.toString());
+ assertEquals(SHARD, batchInfo.path("shardId").asText());
+ assertEquals(startSequence, batchInfo.path("startSequenceNumber").asText());
+ assertEquals(endSequence, batchInfo.path("endSequenceNumber").asText());
+ assertEquals(firstArrival, batchInfo.path("approximateArrivalOfFirstRecord").asText());
+ assertEquals(lastArrival, batchInfo.path("approximateArrivalOfLastRecord").asText());
+ assertTrue(batchInfo.path("batchSize").isInt(), "batchSize must be a number: " + batchInfo);
+ assertEquals(batchSize, batchInfo.path("batchSize").asInt());
+ assertEquals(STREAM_ARN, batchInfo.path("streamArn").asText());
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/testing/DockerAvailability.java b/src/test/java/io/github/hectorvent/floci/testing/DockerAvailability.java
index 0f378bc34c..f2dd772717 100644
--- a/src/test/java/io/github/hectorvent/floci/testing/DockerAvailability.java
+++ b/src/test/java/io/github/hectorvent/floci/testing/DockerAvailability.java
@@ -1,5 +1,7 @@
package io.github.hectorvent.floci.testing;
+import com.github.dockerjava.api.DockerClient;
+
import java.io.IOException;
import java.util.concurrent.TimeUnit;
@@ -29,4 +31,15 @@ public final class DockerAvailability {
}
return process.exitValue() == 0;
}
+
+ /** True when the daemon Floci itself talks to, through {@code dockerClient}, answers a ping. */
+ public static boolean answers(DockerClient dockerClient) {
+ try {
+ dockerClient.pingCmd().exec();
+ return true;
+ } catch (Exception expected) {
+ // An unreachable daemon only means the caller skips its test rather than failing it.
+ return false;
+ }
+ }
}