Light, fluffy, and always free - The AWS Local Emulator alternative
feat(stepfunctions): add the elasticmapreduce optimized integrations (#6261)
Priamo Germosen94c4572fed77
diff --git a/docs/services/emr.md b/docs/services/emr.mdindex 37cfab145f..2b697d7ee0 100644--- a/docs/services/emr.md+++ b/docs/services/emr.md@@ -62,12 +62,27 @@ steps added later complete, reporting `ALL_STEPS_COMPLETED`. A cluster with no s`WAITING` after its steps complete. Automatic step-completion termination also applies whentermination protection is enabled.`AddJobFlowSteps` rejects a terminated cluster without adding steps or changing its-termination status.+termination status, and answers an unknown cluster with `ValidationException` "Specified job flow ID+does not exist.", as AWS does.After automatic or manual termination, instance groups and fleets report `TERMINATED` with zerorunning or provisioned capacity; their requested and target capacities remain recorded.`ListInstances` retains each group's instances with their original synthetic IDs and `TERMINATED`status for 30 days after termination. Capacity modifications and new groups or fleets are rejected-on terminated clusters. `InstanceStates` filters the modeled `RUNNING` and `TERMINATED` instances.+on terminated clusters. As on AWS, `ModifyInstanceGroups` answers `ValidationException` "An instance+group may only be modified when the cluster is running or waiting.", and `ModifyInstanceFleet` answers+`InvalidRequestException` "A job flow that is shutting down, terminated, or finished may not be+modified." with `ErrorCode` `INVALID_JOB_SHUT_DOWN`. `InstanceStates` filters the modeled `RUNNING` and+`TERMINATED` instances.++`TerminateJobFlows`, the four `Set*` cluster setters and `CancelSteps` refuse a request without a+member their model requires (`JobFlowIds` and the setting, or `StepIds`) with `ValidationException`+"1 validation error detected: Value null at 'jobFlowIds' failed to satisfy constraint: Member must+not be null", AWS's framework wording, as `RunJobFlow` does for `Name`. They used to read a missing+list as empty and a missing setting as its default, and answer success. An empty `JobFlowIds` is+still accepted.++A state machine drives clusters and steps through the `elasticmapreduce:` integrations, `.sync`+included; see [Step Functions: Amazon EMR tasks](step-functions.md#amazon-emr-tasks).## Release cataloguediff --git a/docs/services/step-functions.md b/docs/services/step-functions.mdindex 51859550cb..490adae37c 100644--- a/docs/services/step-functions.md+++ b/docs/services/step-functions.md@@ -80,12 +80,13 @@ that does not match the selected task account fails with`HeartbeatSeconds` as a literal number.`resourceType` is the integrated service (`states`, `ecs`, `aws-sdk:sqs`) and `resource` theAPI with its suffix (`startExecution.sync:2`, `runTask.sync`), as AWS names them. A `.sync`- Task (`states:startExecution.sync` and `.sync:2`, `ecs:runTask.sync`) also records+ Task (`states:startExecution.sync` and `.sync:2`, `ecs:runTask.sync`, the `elasticmapreduce:`+ ones) also records`TaskSubmitted` after `TaskStarted`, once the call that started its job has returned and beforeit waits on the job. `taskSubmittedEventDetails` carries `resourceType`, `resource`,`outputDetails`, and that call's response as `output`, in PascalCase: `{"ExecutionArn","StartDate"}` with `StartDate` in epoch milliseconds for a nested execution, `{"Failures",- "Tasks"}` for an ECS task. AWS folds the SDK's `SdkHttpMetadata` and `SdkResponseMetadata`+ "Tasks"}` for an ECS task, `{"ClusterId"}`, `{"StepId"}` or `{}` for an EMR one. AWS folds the SDK's `SdkHttpMetadata` and `SdkResponseMetadata`into that response as well; Floci records the response fields alone.- A direct Lambda function ARN emits `LambdaFunctionScheduled`, `LambdaFunctionStarted`,and then `LambdaFunctionSucceeded` or `LambdaFunctionFailed`. `LambdaFunctionScheduled`@@ -269,7 +270,7 @@ One deviation. Every pause is capped at `floci.services.stepfunctions.max-wait-sASL carries two `TimeoutSeconds` fields and Floci enforces both, in the two terminal shapesAWS uses. The state machine's own field bounds every state; a `Task`'s own field bounds one thatwaits: for a task token (an activity, or a `.waitForTaskToken` integration) or for a job to end-(`ecs:runTask.sync`, `states:startExecution.sync` and `.sync:2`). A Lambda or other SDK task that+(`ecs:runTask.sync`, the `elasticmapreduce:` `.sync` ones, `states:startExecution.sync` and `.sync:2`). A Lambda or other SDK task thatreturns directly is not bound by it.The state machine's own `TimeoutSeconds` is the whole execution's budget. It is checked before@@ -302,6 +303,8 @@ both carry the cause `The Task state in AWS Step Functions execution [<arn>] whithis resource was aborted`. A `StopExecution` that lands while the state waits ends the wait andstops the job with the same cause, and so does the execution's budget, and so does a failure inanother branch of the `Parallel`, or in another iteration of the `Map`, that the `Task` is in.+An `addStep.sync` step is cancelled with `CancelSteps` instead, and the cluster of a+`createCluster.sync` is left running (see [Amazon EMR tasks](#amazon-emr-tasks)).One deviation. AWS starts the `TimeoutSeconds` clock when a worker picks the task up, the instantit emits `ActivityStarted`. Floci emits `ActivityStarted` at schedule time, so both clocks start@@ -840,6 +843,59 @@ Successful responses preserve their headers and status. JSON response text is paits content type, matching the live AWS integration; other text remains a string. Non-2xx responsesraise catchable `ApiGateway.<status>` failures.+## Amazon EMR tasks++`arn:...:states:::elasticmapreduce:` serves the optimized EMR integrations the Step Functions guide+lists, request-response for all seven and `.sync` for `createCluster`, `terminateCluster` and+`addStep`. Each one calls Floci's EMR in the state machine's Region, under the guide's field names:++| Resource | EMR calls | Result |+| --- | --- | --- |+| `createCluster` | `RunJobFlow` | `{ClusterId}` |+| `createCluster.sync` | `RunJobFlow`, then `DescribeCluster` until `WAITING` | `Cluster`, then `ClusterId` |+| `setClusterTerminationProtection` | `SetTerminationProtection`, `ClusterId` as `JobFlowIds` | `{}` |+| `terminateCluster` | `TerminateJobFlows`, `ClusterId` as `JobFlowIds` | `{}` |+| `terminateCluster.sync` | the same, then `DescribeCluster` until `TERMINATED` or `TERMINATED_WITH_ERRORS` | `Cluster` |+| `addStep` | `AddJobFlowSteps` with the one `Step`, and `ExecutionRoleArn` when given | `{StepId}` |+| `addStep.sync` | the same, then `DescribeStep` until `COMPLETED` | `Step`, then `StepId` |+| `cancelStep` | `CancelSteps` with the one `StepId` | `{CancelStepsInfo}` |+| `modifyInstanceFleetByName` | `ListInstanceFleets` for the fleet named `InstanceFleetName`, then `ModifyInstanceFleet` | `{}` |+| `modifyInstanceGroupByName` | `ListInstanceGroups` for the group named `InstanceGroupName`, then `ModifyInstanceGroups` | `{}` |++A `.sync` result is the describe call's response with the starting call's result after it. That is+the `createCluster.sync` output recorded on AWS: `Cluster`, the SDK's `SdkHttpMetadata` and+`SdkResponseMetadata`, then `ClusterId`. Floci leaves the SDK metadata out, as it does everywhere.+State machines read an `addStep.sync` result's `Step` (`$.result.Step.Status.State`); its `StepId`,+and `terminateCluster.sync`'s whole result, follow the same rule without a recording of their own.+The timestamps inside are EMR's epoch seconds, which no recording confirms either.++A `.sync` Task records `TaskSubmitted` with the starting call's result, then polls every 100 ms under+the same two clocks as the other `.sync` integrations (see [Timeouts](#timeouts)). A step that ends+`FAILED`, `CANCELLED` or `INTERRUPTED`, and a cluster that terminates before it reaches `WAITING`,+fail the Task with `States.TaskFailed`, the describe response as the cause: a JSON string, which a+`Catch` reads with `States.StringToJson` (for example its `Step.Status.State`). When+the wait ends early, `addStep.sync` cancels its step with `CancelSteps`, as Step Functions "will make+a best-effort attempt to cancel the task". A cluster is never terminated: "Step Functions does not+terminate an Amazon EMR cluster automatically if execution is stopped." EMR in Floci reaches each+state at once, so every wait ends at its first poll.++An error from EMR fails the Task with the SDK exception class under the `EMR.` prefix:+`EMR.InvalidRequestException` and `EMR.InternalServerException` for the two errors EMR models,+`EMR.AmazonElasticMapReduceException` for any other, such as the `ValidationException`+"Specified job flow ID does not exist." that `AddJobFlowSteps` answers for an unknown cluster. The cause is EMR's message; AWS+also appends `(Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ...; Request ID: ...)`.++The integrations refuse what the guide rules out with `States.TaskFailed`, since no source gives+AWS's error; the `createCluster` messages are the guide's own rules, the others Floci's words: a+`createCluster` whose `Instances.KeepJobFlowAliveWhenNoSteps` is not `true` or that carries `Steps`,+an `addStep` without a `Step`, an `InstanceGroupId` or `InstanceFleetId` in a ByName request, and a+name that matches no group or fleet, or more than one. They also refuse a request without the+integration's own fields, before EMR sees it: a `ClusterId` (every API but `createCluster`),+`cancelStep`'s `StepId`, and a ByName request's name. A member EMR itself requires, such as+`TerminationProtected`, is left to EMR, which answers its validation error.+`.waitForTaskToken`, which AWS does not offer for EMR, and `.sync` on any other API fail as an+unsupported resource.+## Configuration| Variable | Default | Description |diff --git a/src/main/java/io/github/hectorvent/floci/core/common/SmithyValidationErrors.java b/src/main/java/io/github/hectorvent/floci/core/common/SmithyValidationErrors.javaindex d665f81ae1..7e1a2913ff 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/SmithyValidationErrors.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/SmithyValidationErrors.java@@ -26,6 +26,11 @@ public final class SmithyValidationErrors {return count + String.join("; ", violations);}+ /** The violation for a required member the request leaves out. */+ public static String notNull(String path) {+ return "Value null at '" + path + "' failed to satisfy constraint: Member must not be null";+ }+/** The 400 error carrying {@link #message} under the protocol's validation error code. */public static AwsException exception(String errorCode, List<String> violations) {return new AwsException(errorCode, message(violations), 400);diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrHandler.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrHandler.javaindex 55d90f28b2..3d847aefec 100644--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrHandler.java+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrHandler.java@@ -10,7 +10,9 @@ import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.core.common.AwsJson11Controller;import io.github.hectorvent.floci.core.common.AwsJsonTimestamp;import io.github.hectorvent.floci.core.common.AwsRegions;+import io.github.hectorvent.floci.core.common.JsonErrorResponseUtils;import io.github.hectorvent.floci.core.common.PaginatedResult;+import io.github.hectorvent.floci.core.common.SmithyValidationErrors;import io.github.hectorvent.floci.services.emr.model.EmrBlockPublicAccess;import io.github.hectorvent.floci.services.emr.model.EmrCluster;import io.github.hectorvent.floci.services.emr.model.EmrInstanceFleet;@@ -98,6 +100,10 @@ public class EmrHandler {.build();};} catch (AwsException e) {+ if (e.getExtendedData() != null) {+ // Members an error adds to its message, such as InvalidRequestException's ErrorCode.+ return JsonErrorResponseUtils.createErrorResponse(e);+ }return Response.status(e.getHttpStatus()).entity(new AwsErrorResponse(e.jsonType(), e.getMessage())).build();@@ -180,29 +186,34 @@ public class EmrHandler {}private Response handleTerminateJobFlows(JsonNode request) {+ requireMembers(request, "JobFlowIds");service.terminateJobFlows(stringList(request.path("JobFlowIds")));return Response.ok(objectMapper.createObjectNode()).build();}private Response handleSetTerminationProtection(JsonNode request) {+ requireMembers(request, "JobFlowIds", "TerminationProtected");service.setTerminationProtection(stringList(request.path("JobFlowIds")),request.path("TerminationProtected").asBoolean(false));return Response.ok(objectMapper.createObjectNode()).build();}private Response handleSetVisibleToAllUsers(JsonNode request) {+ requireMembers(request, "JobFlowIds", "VisibleToAllUsers");service.setVisibleToAllUsers(stringList(request.path("JobFlowIds")),request.path("VisibleToAllUsers").asBoolean(true));return Response.ok(objectMapper.createObjectNode()).build();}private Response handleSetKeepAlive(JsonNode request) {+ requireMembers(request, "JobFlowIds", "KeepJobFlowAliveWhenNoSteps");service.setKeepJobFlowAliveWhenNoSteps(stringList(request.path("JobFlowIds")),request.path("KeepJobFlowAliveWhenNoSteps").asBoolean(false));return Response.ok(objectMapper.createObjectNode()).build();}private Response handleSetUnhealthyNodeReplacement(JsonNode request) {+ requireMembers(request, "JobFlowIds", "UnhealthyNodeReplacement");service.setUnhealthyNodeReplacement(stringList(request.path("JobFlowIds")),request.path("UnhealthyNodeReplacement").asBoolean(false));return Response.ok(objectMapper.createObjectNode()).build();@@ -244,6 +255,7 @@ public class EmrHandler {}private Response handleCancelSteps(JsonNode request) {+ requireMembers(request, "StepIds");List<EmrService.CancelStepInfo> infos = service.cancelSteps(text(request, "ClusterId"), stringList(request.path("StepIds")));ObjectNode response = objectMapper.createObjectNode();@@ -860,6 +872,25 @@ public class EmrHandler {return values;}+ /**+ * Refuses a request that leaves out members its model requires, which would otherwise read as an+ * empty list or a default flag and succeed on nothing. The wording is AWS's framework validation, as+ * for RunJobFlow's Name; an explicitly empty JobFlowIds is still a valid list.+ */+ private static void requireMembers(JsonNode request, String... members) {+ List<String> violations = new ArrayList<>();+ for (String member : members) {+ JsonNode value = request.path(member);+ if (value.isMissingNode() || value.isNull()) {+ violations.add(SmithyValidationErrors.notNull(+ Character.toLowerCase(member.charAt(0)) + member.substring(1)));+ }+ }+ if (!violations.isEmpty()) {+ throw SmithyValidationErrors.validationException(violations);+ }+ }+private String text(JsonNode request, String field) {JsonNode node = request.path(field);return node.isMissingNode() || node.isNull() ? null : node.asText(null);diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.javaindex 228654a558..7c0b2eb1f8 100644--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java@@ -45,6 +45,9 @@ public class EmrService {private static final String UPPER_ALNUM = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";private static final SecureRandom RANDOM = new SecureRandom();+ // AWS's answer to a change on a terminated cluster, for AddJobFlowSteps and ModifyInstanceFleet.+ private static final String JOB_FLOW_SHUT_DOWN =+ "A job flow that is shutting down, terminated, or finished may not be modified.";private final StorageBackend<String, EmrCluster> clusterStore;private final StorageBackend<String, SecurityConfiguration> secConfigStore;@@ -81,9 +84,7 @@ public class EmrService {// Only a missing Name fails AWS's framework validation: the member is REQUIRED but its// constraint is len=[0..256], so an explicit empty string is accepted by real EMR.if (cluster.getName() == null) {- throw SmithyValidationErrors.validationException(- "Value null at 'name' failed to satisfy constraint: "- + "Member must not be null");+ throw SmithyValidationErrors.validationException(SmithyValidationErrors.notNull("name"));}String id = "j-" + randomId(13);cluster.setId(id);@@ -212,10 +213,12 @@ public class EmrService {// ──────────────────────────── Steps ────────────────────────────public synchronized List<String> addJobFlowSteps(String clusterId, List<EmrStep> steps) {- EmrCluster cluster = requireCluster(clusterId);+ // AWS's own wording for an unknown job flow here, not DescribeCluster's: a botocore ClientError+ // (raster-foundry/raster-foundry#352) and a CloudTrail record on AWS re:Post both show it.+ EmrCluster cluster = clusterId == null ? requireCluster(null) : findCluster(clusterId).orElseThrow(+ () -> new AwsException("ValidationException", "Specified job flow ID does not exist.", 400));if ("TERMINATED".equals(cluster.getState())) {- throw new AwsException("ValidationException",- "A job flow that is shutting down, terminated, or finished may not be modified.", 400);+ throw new AwsException("ValidationException", JOB_FLOW_SHUT_DOWN, 400);}List<String> ids = new ArrayList<>();for (EmrStep step : steps) {@@ -315,7 +318,12 @@ public class EmrService {List<EmrInstanceGroup> groups = new ArrayList<>();for (InstanceGroupModification mod : modifications) {EmrCluster found = clusterId != null ? requireCluster(clusterId) : clusterOwning(region, mod.instanceGroupId());- rejectTerminatedCluster(found);+ // AWS's answer here, recorded by Terraform users+ // (hashicorp/terraform-provider-aws#1355 and #9400).+ if ("TERMINATED".equals(found.getState())) {+ throw new AwsException("ValidationException",+ "An instance group may only be modified when the cluster is running or waiting.", 400);+ }// One copy per cluster, so two groups of the same cluster are changed and stored together.EmrCluster cluster = touched.computeIfAbsent(found.getId(), id -> found);EmrInstanceGroup group = requireInstanceGroup(cluster, mod.instanceGroupId());@@ -368,7 +376,12 @@ public class EmrService {if (instanceFleetId == null) {throw invalid("InstanceFleetId is required.");}- rejectTerminatedCluster(cluster);+ // AWS's answer for a cluster terminated a while before, ErrorCode member included+ // (hashicorp/terraform-provider-aws#31375, 2023).+ if ("TERMINATED".equals(cluster.getState())) {+ throw new AwsException("InvalidRequestException", JOB_FLOW_SHUT_DOWN, 400,+ Map.of("ErrorCode", "INVALID_JOB_SHUT_DOWN"));+ }EmrInstanceFleet fleet = cluster.getInstanceFleets().stream().filter(f -> instanceFleetId.equals(f.getId())).findFirst()diff --git a/src/main/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutor.java b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutor.javaindex c4d262b511..5012487395 100644--- a/src/main/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutor.java+++ b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutor.java@@ -183,8 +183,8 @@ public class AslExecutor {private static final int INLINE_MAP_MAX_CONCURRENCY = 40;private static final int DISTRIBUTED_MAP_MAX_CONCURRENCY = 10_000;- // How often a .sync integration (ecs:runTask, states:startExecution) re-reads the job it waits- // on. The wait itself is bounded by the Task's TimeoutSeconds and the execution's budget.+ // How often a .sync integration (ecs:runTask, elasticmapreduce, states:startExecution) re-reads the job+ // it waits on. The wait itself is bounded by the Task's TimeoutSeconds and the execution's budget.private static final long SYNC_POLL_INTERVAL_MS = 100;// AWS caps the string input of States.Base64Encode/Base64Decode/Hash at 10,000 characters@@ -267,6 +267,7 @@ public class AslExecutor {private final Instance<StepFunctionsService> sfnService;private final WebClient webClient;private final ApiGatewayTaskInvoker apiGatewayTaskInvoker;+ private final EmrTaskInvoker emrTaskInvoker;private final EmulatorConfig config;private final NestedTaskCredentials nestedTaskCredentials;private final CustomResourceLiveness customResourceLiveness;@@ -297,14 +298,15 @@ public class AslExecutor {Instance<StepFunctionsService> sfnService, EmulatorConfig config, Vertx vertx,CustomResourceLiveness customResourceLiveness, RedshiftDataJsonHandler redshiftDataHandler,ServiceRegistry serviceRegistry, NestedTaskCredentials nestedTaskCredentials,- LambdaTaskAuthorizer lambdaTaskAuthorizer, ApiGatewayTaskInvoker apiGatewayTaskInvoker) {+ LambdaTaskAuthorizer lambdaTaskAuthorizer, ApiGatewayTaskInvoker apiGatewayTaskInvoker,+ EmrTaskInvoker emrTaskInvoker) {this(lambdaExecutor, targetResolver, dynamoDb, dynamoDbJsonHandler,sqsJsonHandler, snsJsonHandler, cloudFormationHandler,ec2Service, s3Service, ecsService, ecsJsonHandler,eventBridgeHandler, schedulerService, schedulerController, rdsDataService,objectMapper, jsonataEvaluator, sfnService, config, vertx, customResourceLiveness,Clock.systemUTC(), TimeUnit.NANOSECONDS::sleep, null, redshiftDataHandler, serviceRegistry,- nestedTaskCredentials, lambdaTaskAuthorizer, null, null, apiGatewayTaskInvoker);+ nestedTaskCredentials, lambdaTaskAuthorizer, null, null, apiGatewayTaskInvoker, emrTaskInvoker);}AslExecutor(LambdaExecutorService lambdaExecutor, LambdaTargetResolver targetResolver,@@ -322,8 +324,9 @@ public class AslExecutor {RedshiftDataJsonHandler redshiftDataHandler, ServiceRegistry serviceRegistry,NestedTaskCredentials nestedTaskCredentials, LambdaTaskAuthorizer lambdaTaskAuthorizer,Duration expressMaxExecutionTimeOverride, LongSupplier nanoTimeOverride,- ApiGatewayTaskInvoker apiGatewayTaskInvoker) {+ ApiGatewayTaskInvoker apiGatewayTaskInvoker, EmrTaskInvoker emrTaskInvoker) {this.apiGatewayTaskInvoker = apiGatewayTaskInvoker;+ this.emrTaskInvoker = emrTaskInvoker;this.nestedTaskCredentials = nestedTaskCredentials;this.serviceRegistry = serviceRegistry;this.lambdaTaskAuthorizer = lambdaTaskAuthorizer;@@ -386,7 +389,7 @@ public class AslExecutor {ec2Service, s3Service, ecsService, ecsJsonHandler,eventBridgeHandler, schedulerService, schedulerController, rdsDataService,objectMapper, jsonataEvaluator, sfnService, config, vertx, customResourceLiveness,- Clock.systemUTC(), TimeUnit.NANOSECONDS::sleep, null, null, null, null, null, null, null, null);+ Clock.systemUTC(), TimeUnit.NANOSECONDS::sleep, null, null, null, null, null, null, null, null, null);}// Kept for tests that do not exercise the Redshift Data API integration.@@ -407,7 +410,7 @@ public class AslExecutor {ec2Service, s3Service, ecsService, ecsJsonHandler,eventBridgeHandler, schedulerService, schedulerController, rdsDataService,objectMapper, jsonataEvaluator, sfnService, config, vertx, customResourceLiveness,- clock, sleeper, maxWaitSecondsOverride, null, null, null, null, null, null, null);+ clock, sleeper, maxWaitSecondsOverride, null, null, null, null, null, null, null, null);}AslExecutor(LambdaExecutorService lambdaExecutor, LambdaFunctionStore functionStore,@@ -1444,6 +1447,14 @@ public class AslExecutor {jobSubmitted);}+ // EMR optimized integrations: request-response and .sync. AWS supports no .waitForTaskToken+ // variant, so one carrying a token falls through to Unsupported resource.+ if (integration.isOptimizedService("elasticmapreduce") && taskToken == null) {+ String region = extractRegionFromArn(sm.getStateMachineArn());+ return invokeEmr(integration, input, region, executionArn, executionDeadlineNanos, taskDeadlineNanos,+ jobSubmitted);+ }+// AWS SDK service integrations: Step Functionsif (integration.isSdkService("sfn")) {String region = extractRegionFromArn(sm.getStateMachineArn());@@ -2193,6 +2204,43 @@ public class AslExecutor {}}+ /**+ * The {@code elasticmapreduce:} integrations. A {@code .sync} Task records its call's response as+ * submitted and polls the cluster or step as {@code ecs:runTask.sync} polls its tasks, bounded by+ * the same two clocks; whatever ends the wait first abandons the job.+ */+ private JsonNode invokeEmr(StatesIntegration integration, JsonNode input, String region, String executionArn,+ long executionDeadlineNanos, long taskDeadlineNanos,+ Consumer<JsonNode> jobSubmitted) throws Exception {+ EmrTaskInvoker.Job job = emrTaskInvoker.start(integration, input, region);+ if (integration.suffix().isEmpty()) {+ return job.response();+ }+ try {+ jobSubmitted.accept(job.response());+ } catch (RuntimeException e) {+ emrTaskInvoker.abandon(job, region);+ throw e;+ }+ while (true) {+ try {+ sleepOrTimeOutTask(TimeUnit.MILLISECONDS.toNanos(SYNC_POLL_INTERVAL_MS),+ executionDeadlineNanos, taskDeadlineNanos);+ } catch (TaskTimedOutException | ExecutionTimedOutException | InterruptedException e) {+ emrTaskInvoker.abandon(job, region);+ throw e;+ }+ if (abortedByCaller(executionArn)) {+ emrTaskInvoker.abandon(job, region);+ throw new ExecutionAbortedException();+ }+ Optional<JsonNode> result = emrTaskInvoker.poll(job, region);+ if (result.isPresent()) {+ return result.get();+ }+ }+ }+/*** The reason AWS writes on a job it stops because the {@code .sync} Task waiting on it ended* first, whichever way it ended: verbatim, as the ECS task's {@code stoppedReason} and the childdiff --git a/src/main/java/io/github/hectorvent/floci/services/stepfunctions/EmrTaskInvoker.java b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/EmrTaskInvoker.javanew file mode 100644index 0000000000..0449bf624d--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/EmrTaskInvoker.java@@ -0,0 +1,258 @@+package io.github.hectorvent.floci.services.stepfunctions;++import com.fasterxml.jackson.databind.JsonNode;+import com.fasterxml.jackson.databind.ObjectMapper;+import com.fasterxml.jackson.databind.node.ObjectNode;+import io.github.hectorvent.floci.core.common.RequestScopes;+import io.github.hectorvent.floci.services.emr.EmrHandler;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.inject.Inject;+import jakarta.ws.rs.core.Response;+import org.jboss.logging.Logger;++import java.util.ArrayList;+import java.util.List;+import java.util.Optional;+import java.util.Set;++/**+ * The optimized {@code arn:<partition>:states:::elasticmapreduce:*} integrations. Each one is an EMR+ * API under renamed fields, as the Step Functions guide's "Optimized Amazon EMR APIs" table lists+ * them, run through {@link EmrHandler} in the state machine's Region. A {@code .sync} Task calls+ * {@link #start}, then {@link #poll} until its cluster or step is done; {@link AslExecutor} owns the+ * wait between polls.+ */+@ApplicationScoped+public class EmrTaskInvoker {++ private static final Logger LOG = Logger.getLogger(EmrTaskInvoker.class);++ // A failure is named after the SDK exception class: the errors EMR models keep their own, and any+ // other code is the base class. AWS re:Post, "AWS Step Function does not recognize job flow ID":+ // a ValidationException fails addStep.sync as EMR.AmazonElasticMapReduceException.+ private static final Set<String> MODELED_ERRORS = Set.of("InvalidRequestException", "InternalServerException");+ private static final String BASE_ERROR = "AmazonElasticMapReduceException";+ private static final Set<String> APIS = Set.of("createCluster", "setClusterTerminationProtection",+ "terminateCluster", "addStep", "cancelStep", "modifyInstanceFleetByName", "modifyInstanceGroupByName");+ private static final Set<String> SYNC_APIS = Set.of("createCluster", "terminateCluster", "addStep");+ private static final Set<String> TERMINATED_STATES = Set.of("TERMINATED", "TERMINATED_WITH_ERRORS");+ private static final Set<String> FAILED_STEP_STATES = Set.of("FAILED", "CANCELLED", "INTERRUPTED");++ private final EmrHandler emr;+ private final ObjectMapper objectMapper;++ @Inject+ public EmrTaskInvoker(EmrHandler emr, ObjectMapper objectMapper) {+ this.emr = emr;+ this.objectMapper = objectMapper;+ }++ /**+ * A started call: the integration's response, and the cluster, or step, a {@code .sync} Task+ * waits on.+ */+ record Job(String api, String clusterId, String stepId, ObjectNode response) {}++ Job start(StatesIntegration integration, JsonNode input, String region) {+ String api = integration.api();+ if (!APIS.contains(api)+ || !integration.suffix().isEmpty() && !(".sync".equals(integration.suffix()) && SYNC_APIS.contains(api))) {+ throw unsupported(integration);+ }+ JsonNode request = input != null && input.isObject() ? input : objectMapper.createObjectNode();+ // The integration's own fields, named as the Task wrote them: renamed into JobFlowIds or StepIds,+ // a missing one would reach EMR as a "null" id.+ String clusterId = "createCluster".equals(api) ? null : required(request, "ClusterId");+ return switch (api) {+ case "createCluster" -> createCluster(request, region);+ case "setClusterTerminationProtection" -> new Job(api, clusterId, null,+ call("SetTerminationProtection", withJobFlowIds(request), region));+ case "terminateCluster" -> new Job(api, clusterId, null,+ call("TerminateJobFlows", withJobFlowIds(request), region));+ case "addStep" -> addStep(request, clusterId, region);+ case "cancelStep" -> cancelStep(request, clusterId, region);+ case "modifyInstanceFleetByName" -> modifyInstanceFleetByName(request, clusterId, region);+ case "modifyInstanceGroupByName" -> modifyInstanceGroupByName(request, clusterId, region);+ default -> throw unsupported(integration);+ };+ }++ /**+ * The Task's result once the job is done, empty while it runs. The result is the describe call's+ * response with the starting call's merged in: a {@code createCluster.sync} output recorded on+ * AWS is {@code Cluster}, the SDK metadata, then {@code ClusterId}.+ */+ Optional<JsonNode> poll(Job job, String region) {+ if ("addStep".equals(job.api())) {+ ObjectNode request = clusterRequest(job.clusterId());+ request.put("StepId", job.stepId());+ JsonNode described = call("DescribeStep", request, region);+ String state = described.path("Step").path("Status").path("State").asText();+ if (FAILED_STEP_STATES.contains(state)) {+ throw failed(described);+ }+ return "COMPLETED".equals(state) ? Optional.of(merged(described, job.response())) : Optional.empty();+ }+ JsonNode described = call("DescribeCluster", clusterRequest(job.clusterId()), region);+ String state = described.path("Cluster").path("Status").path("State").asText();+ if ("terminateCluster".equals(job.api())) {+ return TERMINATED_STATES.contains(state)+ ? Optional.of(merged(described, job.response())) : Optional.empty();+ }+ if (TERMINATED_STATES.contains(state)) {+ throw failed(described);+ }+ return "WAITING".equals(state) ? Optional.of(merged(described, job.response())) : Optional.empty();+ }++ /**+ * Best effort, for a {@code .sync} Task that ends before its job does: Step Functions "will make a+ * best-effort attempt to cancel the task", and addStep's IAM policy grants it CancelSteps. A cluster+ * is left running, since "Step Functions does not terminate an Amazon EMR cluster automatically if+ * execution is stopped."+ */+ void abandon(Job job, String region) {+ if (!"addStep".equals(job.api())) {+ return;+ }+ ObjectNode request = clusterRequest(job.clusterId());+ request.putArray("StepIds").add(job.stepId());+ try {+ call("CancelSteps", request, region);+ } catch (RuntimeException e) {+ LOG.warnv("elasticmapreduce:addStep.sync ended before its step {0} did, and the step could not be "+ + "cancelled: {1}", job.stepId(), e.getMessage());+ }+ }++ private Job createCluster(JsonNode request, String region) {+ // The two differences the guide lists from RunJobFlow. No source gives the error AWS answers either with.+ if (!request.path("Instances").path("KeepJobFlowAliveWhenNoSteps").booleanValue()) {+ throw refused("The field Instances.KeepJobFlowAliveWhenNoSteps is mandatory, "+ + "and must have the Boolean value TRUE.");+ }+ if (request.has("Steps")) {+ throw refused("The field Steps is not allowed.");+ }+ String clusterId = call("RunJobFlow", request, region).path("JobFlowId").asText();+ return new Job("createCluster", clusterId, null, objectMapper.createObjectNode().put("ClusterId", clusterId));+ }++ private Job addStep(JsonNode request, String clusterId, String region) {+ if (!request.path("Step").isObject()) {+ throw refused("The field Step is required.");+ }+ ObjectNode add = request.deepCopy();+ add.remove(List.of("ClusterId", "Step"));+ add.put("JobFlowId", clusterId);+ add.putArray("Steps").add(request.get("Step"));+ String stepId = call("AddJobFlowSteps", add, region).path("StepIds").path(0).asText();+ return new Job("addStep", clusterId, stepId, objectMapper.createObjectNode().put("StepId", stepId));+ }++ private Job cancelStep(JsonNode request, String clusterId, String region) {+ String stepId = required(request, "StepId");+ ObjectNode cancel = request.deepCopy();+ cancel.remove("StepId");+ cancel.putArray("StepIds").add(stepId);+ JsonNode info = call("CancelSteps", cancel, region).path("CancelStepsInfoList").path(0);+ ObjectNode response = objectMapper.createObjectNode();+ response.set("CancelStepsInfo", info.isMissingNode() ? objectMapper.createObjectNode() : info);+ return new Job("cancelStep", clusterId, null, response);+ }++ private Job modifyInstanceFleetByName(JsonNode request, String clusterId, String region) {+ JsonNode fleet = request.path("InstanceFleet");+ if (fleet.has("InstanceFleetId")) {+ throw refused("The field InstanceFleet.InstanceFleetId is not allowed.");+ }+ String name = required(request, "InstanceFleetName");+ JsonNode fleets = call("ListInstanceFleets", clusterRequest(clusterId), region).path("InstanceFleets");+ ObjectNode modification = fleet.isObject() ? fleet.deepCopy() : objectMapper.createObjectNode();+ modification.put("InstanceFleetId", idOfNamed(fleets, name, "instance fleet"));+ ObjectNode modify = clusterRequest(clusterId);+ modify.set("InstanceFleet", modification);+ return new Job("modifyInstanceFleetByName", clusterId, null, call("ModifyInstanceFleet", modify, region));+ }++ private Job modifyInstanceGroupByName(JsonNode request, String clusterId, String region) {+ JsonNode group = request.path("InstanceGroup");+ if (group.has("InstanceGroupId")) {+ throw refused("The field InstanceGroup.InstanceGroupId is not allowed.");+ }+ String name = required(request, "InstanceGroupName");+ JsonNode groups = call("ListInstanceGroups", clusterRequest(clusterId), region).path("InstanceGroups");+ ObjectNode modification = group.isObject() ? group.deepCopy() : objectMapper.createObjectNode();+ modification.put("InstanceGroupId", idOfNamed(groups, name, "instance group"));+ ObjectNode modify = clusterRequest(clusterId);+ modify.putArray("InstanceGroups").add(modification);+ return new Job("modifyInstanceGroupByName", clusterId, null, call("ModifyInstanceGroups", modify, region));+ }++ /** The guide asks for unique names whenever a ByName API is used; the lookup reads ListInstance*. */+ private static String idOfNamed(JsonNode members, String name, String kind) {+ List<String> ids = new ArrayList<>();+ for (JsonNode member : members) {+ if (name.equals(member.path("Name").asText(null))) {+ ids.add(member.path("Id").asText());+ }+ }+ if (ids.size() == 1) {+ return ids.get(0);+ }+ throw refused(ids.isEmpty()+ ? "The cluster has no " + kind + " named '" + name + "'."+ : "The cluster has " + ids.size() + " " + kind + "s named '" + name + "'.");+ }++ /** The request with {@code ClusterId} renamed to EMR's {@code JobFlowIds}. */+ private ObjectNode withJobFlowIds(JsonNode request) {+ ObjectNode renamed = request.deepCopy();+ renamed.putArray("JobFlowIds").add(renamed.remove("ClusterId"));+ return renamed;+ }++ private ObjectNode clusterRequest(String clusterId) {+ return objectMapper.createObjectNode().put("ClusterId", clusterId);+ }++ private ObjectNode merged(JsonNode described, ObjectNode response) {+ ObjectNode result = described.deepCopy();+ result.setAll(response);+ return result;+ }++ /** Runs in the state machine's Region, which EMR reads to find its clusters. */+ private ObjectNode call(String action, JsonNode request, String region) {+ Response response = RequestScopes.callAs(null, region, () -> emr.handle(action, request, region));+ if (response.getStatus() >= 400) {+ JsonNode error = objectMapper.valueToTree(response.getEntity());+ String type = error.path("__type").asText();+ String name = MODELED_ERRORS.contains(type) ? type : BASE_ERROR;+ throw new AslExecutor.FailStateException("EMR." + name, error.path("message").asText());+ }+ return response.getEntity() instanceof ObjectNode body ? body : objectMapper.createObjectNode();+ }++ /** A job that ended without succeeding: the cause is its describe response, as a JSON string. */+ private static AslExecutor.FailStateException failed(JsonNode described) {+ return new AslExecutor.FailStateException("States.TaskFailed", described.toString());+ }++ /** A field the integration's request names a cluster, step or member by, as a non-blank string. */+ private static String required(JsonNode request, String field) {+ JsonNode value = request.path(field);+ if (!value.isTextual() || value.asText().isBlank()) {+ throw refused("The field " + field + " is required.");+ }+ return value.asText();+ }++ private static AslExecutor.FailStateException refused(String cause) {+ return new AslExecutor.FailStateException("States.TaskFailed", cause);+ }++ private static AslExecutor.FailStateException unsupported(StatesIntegration integration) {+ return new AslExecutor.FailStateException("States.TaskFailed", "Unsupported resource: " + integration);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrClusterRegionIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrClusterRegionIntegrationTest.javaindex 6fbf8895d8..4ebeff695a 100644--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrClusterRegionIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrClusterRegionIntegrationTest.java@@ -42,7 +42,8 @@ class EmrClusterRegionIntegrationTest {emr(OTHER, "AddJobFlowSteps", "{\"JobFlowId\":\"" + clusterId + "\",\"Steps\":[{\"Name\":\"s\","+ "\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\",\"Args\":[\"echo\"]}}]}").then().statusCode(400)- .body("__type", equalTo("InvalidRequestException"));+ .body("__type", equalTo("ValidationException"))+ .body("message", equalTo("Specified job flow ID does not exist."));emr(OTHER, "ListClusters", "{}").then().statusCode(200).body("Clusters.Id", not(hasItem(clusterId)));diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.javaindex 7a0a017529..f8380c9cd5 100644--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java@@ -366,13 +366,13 @@ class EmrIntegrationTest {@Test@Order(17)- void addStepsToUnknownClusterIsInvalidRequest() {+ void addStepsToUnknownClusterIsAValidationException() {emr("AddJobFlowSteps","{\"JobFlowId\":\"j-DOESNOTEXIST0\",\"Steps\":[{\"Name\":\"s\","+ "\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}").then().statusCode(400)- .body("__type", equalTo("InvalidRequestException"))- .body("message", equalTo("Cluster id 'j-DOESNOTEXIST0' is not valid."));+ .body("__type", equalTo("ValidationException"))+ .body("message", equalTo("Specified job flow ID does not exist."));}@Test@@ -404,6 +404,40 @@ class EmrIntegrationTest {.body("Cluster.Status.State", equalTo("WAITING"));}+ @Test+ void settersAndCancelStepsRefuseARequestWithoutTheMembersTheyRequire() {+ String id = emr("RunJobFlow", "{\"Name\":\"required\",\"Instances\":{\"KeepJobFlowAliveWhenNoSteps\":true}}")+ .then().statusCode(200).extract().path("JobFlowId");+ String ids = "\"JobFlowIds\":[\"" + id + "\"]";++ assertMissing("TerminateJobFlows", "{}", "jobFlowIds");+ assertMissing("SetTerminationProtection", "{\"TerminationProtected\":true}", "jobFlowIds");+ assertMissing("SetTerminationProtection", "{" + ids + "}", "terminationProtected");+ assertMissing("SetVisibleToAllUsers", "{" + ids + "}", "visibleToAllUsers");+ assertMissing("SetKeepJobFlowAliveWhenNoSteps", "{" + ids + "}", "keepJobFlowAliveWhenNoSteps");+ assertMissing("SetUnhealthyNodeReplacement", "{" + ids + "}", "unhealthyNodeReplacement");+ assertMissing("CancelSteps", "{\"ClusterId\":\"" + id + "\"}", "stepIds");+ emr("SetVisibleToAllUsers", "{}").then().statusCode(400)+ .body("message", equalTo("2 validation errors detected: "+ + "Value null at 'jobFlowIds' failed to satisfy constraint: Member must not be null; "+ + "Value null at 'visibleToAllUsers' failed to satisfy constraint: Member must not be null"));+ emr("TerminateJobFlows", "{\"JobFlowIds\":[]}").then().statusCode(200);++ emr("DescribeCluster", "{\"ClusterId\":\"" + id + "\"}").then().statusCode(200)+ .body("Cluster.Status.State", equalTo("WAITING"))+ .body("Cluster.TerminationProtected", equalTo(false))+ .body("Cluster.VisibleToAllUsers", equalTo(true))+ .body("Cluster.AutoTerminate", equalTo(false));+ emr("TerminateJobFlows", "{" + ids + "}").then().statusCode(200);+ }++ private static void assertMissing(String action, String body, String member) {+ emr(action, body).then().statusCode(400)+ .body("__type", equalTo("ValidationException"))+ .body("message", equalTo("1 validation error detected: Value null at '" + member+ + "' failed to satisfy constraint: Member must not be null"));+ }+// Catches: a cluster stored before the DNS fix keeps reporting its old us-east-1 master name.@Testvoid storedClusterReportsTheRegionalMasterDnsName() {diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrTerminatedCapacityIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrTerminatedCapacityIntegrationTest.javaindex 7a40269026..a1083c228a 100644--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrTerminatedCapacityIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrTerminatedCapacityIntegrationTest.java@@ -31,6 +31,10 @@ class EmrTerminatedCapacityIntegrationTest {+ "\"TargetSpotCapacity\":3,\"InstanceTypeConfigs\":[{\"InstanceType\":\"m5.xlarge\"}]}]";private static final String STEP = "{\"Name\":\"complete\","+ "\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}";+ private static final String GROUP_REFUSAL =+ "An instance group may only be modified when the cluster is running or waiting.";+ private static final String FLEET_REFUSAL =+ "A job flow that is shutting down, terminated, or finished may not be modified.";@BeforeAllstatic void configure() {@@ -133,7 +137,9 @@ class EmrTerminatedCapacityIntegrationTest {emr("ModifyInstanceGroups", "{\"InstanceGroups\":["+ "{\"InstanceGroupId\":\"" + activeGroup + "\",\"InstanceCount\":4},"+ "{\"InstanceGroupId\":\"" + terminatedGroup + "\",\"InstanceCount\":4}]}")- .then().statusCode(400).body("__type", equalTo("InvalidRequestException"));+ .then().statusCode(400)+ .body("__type", equalTo("ValidationException"))+ .body("message", equalTo(GROUP_REFUSAL));emr("ListInstanceGroups", clusterRequest(activeId)).then().statusCode(200).body("InstanceGroups.find { it.Id == '" + activeGroup + "' }.RunningInstanceCount", equalTo(2));assertTerminatedGroup(terminatedId);@@ -172,7 +178,9 @@ class EmrTerminatedCapacityIntegrationTest {String clusterId = includeClusterId ? "\"ClusterId\":\"" + id + "\"," : "";emr("ModifyInstanceGroups", "{" + clusterId + "\"InstanceGroups\":[{\"InstanceGroupId\":\""+ coreGroupId(id) + "\",\"InstanceCount\":4}]}")- .then().statusCode(400).body("__type", equalTo("InvalidRequestException"));+ .then().statusCode(400)+ .body("__type", equalTo("ValidationException"))+ .body("message", equalTo(GROUP_REFUSAL));assertTerminatedGroup(id);}@@ -181,7 +189,10 @@ class EmrTerminatedCapacityIntegrationTest {.extract().jsonPath().getString("InstanceFleets.find { it.InstanceFleetType == 'CORE' }.Id");emr("ModifyInstanceFleet", "{\"ClusterId\":\"" + id + "\",\"InstanceFleet\":{"+ "\"InstanceFleetId\":\"" + fleetId + "\",\"TargetOnDemandCapacity\":4}}")- .then().statusCode(400).body("__type", equalTo("InvalidRequestException"));+ .then().statusCode(400)+ .body("__type", equalTo("InvalidRequestException"))+ .body("message", equalTo(FLEET_REFUSAL))+ .body("ErrorCode", equalTo("INVALID_JOB_SHUT_DOWN"));assertTerminatedFleet(id);}diff --git a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutorEmrSyncTest.java b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutorEmrSyncTest.javanew file mode 100644index 0000000000..b4846bdfeb--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutorEmrSyncTest.java@@ -0,0 +1,208 @@+package io.github.hectorvent.floci.services.stepfunctions;++import com.fasterxml.jackson.databind.JsonNode;+import com.fasterxml.jackson.databind.ObjectMapper;+import io.github.hectorvent.floci.services.emr.EmrHandler;+import io.github.hectorvent.floci.services.emr.EmrService;+import io.github.hectorvent.floci.services.emr.model.EmrCluster;+import io.github.hectorvent.floci.services.emr.model.EmrStep;+import io.github.hectorvent.floci.services.stepfunctions.model.Execution;+import io.github.hectorvent.floci.services.stepfunctions.model.HistoryEvent;+import io.github.hectorvent.floci.services.stepfunctions.model.StateMachine;+import jakarta.enterprise.inject.Instance;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.util.ArrayList;+import java.util.List;+import java.util.concurrent.atomic.AtomicInteger;+import java.util.concurrent.atomic.AtomicLong;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertNull;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++/**+ * The {@code .sync} waits of the {@code elasticmapreduce:} integrations, on a cluster or step held in+ * a state Floci's instant lifecycle never shows. Time is a test clock the waits move forward.+ */+class AslExecutorEmrSyncTest {++ private static final String REGION = "eu-west-3";+ private static final String ACCOUNT = "000000000000";+ private static final String CLUSTER_ID = "j-SYNCTEST00001";+ private static final String STEP_ID = "s-SYNCTEST00001";+ private static final String ADD_STEP = """+ {"ClusterId": "j-SYNCTEST00001",+ "Step": {"Name": "job", "HadoopJarStep": {"Jar": "command-runner.jar", "Args": ["true"]}}}+ """;+ private static final String CREATE_CLUSTER = """+ {"Name": "sync", "Instances": {"KeepJobFlowAliveWhenNoSteps": true}}+ """;++ private final ObjectMapper objectMapper = new ObjectMapper();+ private final AtomicLong testNanos = new AtomicLong();+ private EmrService emrService;+ private StepFunctionsService sfnService;+ private AslExecutor executor;+ private List<HistoryEvent> history;++ @BeforeEach+ @SuppressWarnings("unchecked")+ void setUp() {+ emrService = mock(EmrService.class);+ when(emrService.addJobFlowSteps(eq(CLUSTER_ID), any())).thenReturn(List.of(STEP_ID));+ when(emrService.runJobFlow(any(), anyString())).thenAnswer(invocation -> {+ EmrCluster cluster = invocation.getArgument(0);+ cluster.setId(CLUSTER_ID);+ return cluster;+ });+ sfnService = mock(StepFunctionsService.class);+ when(sfnService.describeExecution(any())).thenAnswer(invocation -> parent(invocation.getArgument(0), "RUNNING"));+ Instance<StepFunctionsService> instance = mock(Instance.class);+ when(instance.get()).thenReturn(sfnService);+ executor = AslExecutorFixture.builder()+ .objectMapper(objectMapper)+ .sfnService(instance)+ .customResourceLiveness(null)+ .emrTaskInvoker(new EmrTaskInvoker(new EmrHandler(emrService, objectMapper), objectMapper))+ .nanoTime(testNanos::get)+ .sleeper(testNanos::addAndGet)+ .build();+ }++ @Test+ void addStepSyncCancelsItsStepWhenTheTaskTimesOut() {+ when(emrService.describeStep(CLUSTER_ID, STEP_ID)).thenReturn(step("RUNNING"));++ Execution execution = run("""+ {"StartAt": "Add", "States": {"Add": {"Type": "Task",+ "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync", "TimeoutSeconds": 1, "End": true}}}+ """, ADD_STEP);++ assertEquals("FAILED", execution.getStatus());+ assertEquals("States.Timeout", execution.getError());+ verify(emrService).cancelSteps(CLUSTER_ID, List.of(STEP_ID));+ }++ @Test+ void addStepSyncFailsWithTheStepWhenTheStepFails() throws Exception {+ EmrStep failed = step("FAILED");+ failed.setStateChangeReason("Step failed");+ when(emrService.describeStep(CLUSTER_ID, STEP_ID)).thenReturn(failed);++ Execution execution = run(task("addStep.sync"), ADD_STEP);++ assertEquals("FAILED", execution.getStatus());+ assertEquals("States.TaskFailed", execution.getError());+ JsonNode cause = objectMapper.readTree(execution.getCause());+ assertEquals(STEP_ID, cause.path("Step").path("Id").asText());+ assertEquals("FAILED", cause.path("Step").path("Status").path("State").asText());+ verify(emrService, never()).cancelSteps(any(), any());+ }++ @Test+ void addStepSyncWaitsUntilTheStepCompletes() throws Exception {+ AtomicInteger polls = new AtomicInteger();+ when(emrService.describeStep(CLUSTER_ID, STEP_ID))+ .thenAnswer(invocation -> step(polls.incrementAndGet() < 3 ? "RUNNING" : "COMPLETED"));++ Execution execution = run(task("addStep.sync"), ADD_STEP);++ assertEquals("SUCCEEDED", execution.getStatus(), execution.getCause());+ assertEquals(3, polls.get());+ JsonNode output = objectMapper.readTree(execution.getOutput());+ assertEquals(List.of("Step", "StepId"), fieldNames(output));+ assertEquals("COMPLETED", output.path("Step").path("Status").path("State").asText());+ assertEquals(STEP_ID, output.path("StepId").asText());+ }++ @Test+ void createClusterSyncFailsWhenTheClusterTerminatesInsteadOfWaiting() throws Exception {+ when(emrService.describeCluster(CLUSTER_ID)).thenReturn(cluster("TERMINATED_WITH_ERRORS"));++ Execution execution = run(task("createCluster.sync"), CREATE_CLUSTER);++ assertEquals("FAILED", execution.getStatus());+ assertEquals("States.TaskFailed", execution.getError());+ JsonNode cause = objectMapper.readTree(execution.getCause());+ assertEquals("TERMINATED_WITH_ERRORS", cause.path("Cluster").path("Status").path("State").asText());+ }++ @Test+ void stopExecutionLeavesTheClusterOfACreateClusterSyncRunning() {+ when(emrService.describeCluster(CLUSTER_ID)).thenReturn(cluster("STARTING"));+ AtomicInteger parentReads = new AtomicInteger();+ when(sfnService.describeExecution(any())).thenAnswer(invocation ->+ parent(invocation.getArgument(0), parentReads.incrementAndGet() < 3 ? "RUNNING" : "ABORTED"));++ Execution execution = run(task("createCluster.sync"), CREATE_CLUSTER);++ assertNull(execution.getError());+ verify(emrService, never()).terminateJobFlows(any());+ assertEquals(List.of(), history.stream().map(HistoryEvent::getType)+ .filter(type -> type.equals("TaskFailed") || type.equals("TaskSucceeded")).toList());+ }++ private static String task(String api) {+ return """+ {"StartAt": "Emr", "States": {"Emr": {"Type": "Task",+ "Resource": "arn:aws:states:::elasticmapreduce:%s", "End": true}}}+ """.formatted(api);+ }++ private static EmrStep step(String state) {+ EmrStep step = new EmrStep();+ step.setId(STEP_ID);+ step.setName("job");+ step.setActionOnFailure("CONTINUE");+ step.setState(state);+ return step;+ }++ private static EmrCluster cluster(String state) {+ EmrCluster cluster = new EmrCluster();+ cluster.setId(CLUSTER_ID);+ cluster.setName("sync");+ cluster.setState(state);+ return cluster;+ }++ private static Execution parent(String executionArn, String status) {+ Execution parent = new Execution();+ parent.setExecutionArn(executionArn);+ parent.setStatus(status);+ return parent;+ }++ private static List<String> fieldNames(JsonNode node) {+ List<String> names = new ArrayList<>();+ node.fieldNames().forEachRemaining(names::add);+ return names;+ }++ private Execution run(String definition, String input) {+ StateMachine stateMachine = new StateMachine();+ stateMachine.setName("emr-sync-test");+ stateMachine.setStateMachineArn("arn:aws:states:%s:%s:stateMachine:emr-sync-test".formatted(REGION, ACCOUNT));+ stateMachine.setRoleArn("arn:aws:iam::%s:role/test-role".formatted(ACCOUNT));+ stateMachine.setDefinition(definition);++ Execution execution = new Execution();+ execution.setName("emr-sync-execution");+ execution.setExecutionArn("arn:aws:states:%s:%s:execution:emr-sync-test:emr-sync-execution"+ .formatted(REGION, ACCOUNT));+ execution.setStateMachineArn(stateMachine.getStateMachineArn());+ execution.setInput(input);++ history = new ArrayList<>();+ executor.executeSync(stateMachine, execution, history, (updated, events) -> { });+ return execution;+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutorFixture.java b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutorFixture.javaindex 2fbf220295..87e7673f7f 100644--- a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutorFixture.java+++ b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/AslExecutorFixture.java@@ -37,7 +37,7 @@ import static org.mockito.Mockito.mock;* <p>Every other collaborator defaults to what the convenience constructors pass: a Mockito mock* for each integration, the system clock, a real sleeper, {@code System.nanoTime()} for deadlines,* no Wait cap override, AWS's 5-minute Express limit, and no RDS Data, Redshift Data, service- * registry or Lambda task authorizer. A Lambda target resolver is built over the function store+ * registry, Lambda task authorizer or EMR task invoker. A Lambda target resolver is built over the function store* unless a test supplies its own.*/final class AslExecutorFixture {@@ -82,6 +82,7 @@ final class AslExecutorFixture {private LambdaTaskAuthorizer lambdaTaskAuthorizer;private Duration expressMaxExecutionTime;private LongSupplier nanoTime;+ private EmrTaskInvoker emrTaskInvoker;private Builder() {}@@ -236,6 +237,11 @@ final class AslExecutorFixture {return this;}+ Builder emrTaskInvoker(EmrTaskInvoker emrTaskInvoker) {+ this.emrTaskInvoker = emrTaskInvoker;+ return this;+ }+AslExecutor build() {return new AslExecutor(lambdaExecutor,targetResolver != null ? targetResolver : new LambdaTargetResolver(functionStore, null),@@ -245,7 +251,7 @@ final class AslExecutorFixture {jsonataEvaluator != null ? jsonataEvaluator : new JsonataEvaluator(objectMapper),sfnService, config, vertx, customResourceLiveness, clock, sleeper, maxWaitSeconds,redshiftDataHandler, serviceRegistry, null, lambdaTaskAuthorizer, expressMaxExecutionTime,- nanoTime, null);+ nanoTime, null, emrTaskInvoker);}}}diff --git a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsEmrIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsEmrIntegrationTest.javanew file mode 100644index 0000000000..8a89762dbb--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsEmrIntegrationTest.java@@ -0,0 +1,272 @@+package io.github.hectorvent.floci.services.stepfunctions;++import com.fasterxml.jackson.databind.JsonNode;+import com.fasterxml.jackson.databind.ObjectMapper;+import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;+import io.github.hectorvent.floci.testing.StepFunctionsExecutions;+import io.quarkus.test.junit.QuarkusTest;+import io.restassured.response.Response;+import org.junit.jupiter.api.BeforeAll;+import org.junit.jupiter.api.Test;++import java.util.ArrayList;+import java.util.List;+import java.util.UUID;++import static io.github.hectorvent.floci.testing.EmrRequests.emr;+import static io.restassured.RestAssured.given;+import static org.hamcrest.Matchers.equalTo;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertTrue;++/**+ * The {@code elasticmapreduce:} optimized integrations against Floci's EMR, from a state machine in a+ * Region other than the default, so every call has to reach the clusters of the state machine's own.+ */+@QuarkusTest+class StepFunctionsEmrIntegrationTest {++ private static final String REGION = "eu-west-3";+ private static final String ROLE_ARN = "arn:aws:iam::000000000000:role/test-role";+ private static final String RESOURCE = "arn:aws:states:::elasticmapreduce:";+ private static final String GROUPS_CLUSTER = """+ {"Name": "sfn-emr", "ReleaseLabel": "emr-7.5.0",+ "Instances": {"KeepJobFlowAliveWhenNoSteps": true, "InstanceGroups": [+ {"Name": "Primary", "InstanceRole": "MASTER", "InstanceType": "m5.xlarge", "InstanceCount": 1},+ {"Name": "Core", "InstanceRole": "CORE", "InstanceType": "m5.xlarge", "InstanceCount": 2}]}}+ """;+ private static final String FLEETS_CLUSTER = """+ {"Name": "sfn-emr-fleets", "ReleaseLabel": "emr-7.5.0",+ "Instances": {"KeepJobFlowAliveWhenNoSteps": true, "InstanceFleets": [+ {"Name": "Primary", "InstanceFleetType": "MASTER", "TargetOnDemandCapacity": 1,+ "InstanceTypeConfigs": [{"InstanceType": "m5.xlarge"}]},+ {"Name": "Core", "InstanceFleetType": "CORE", "TargetOnDemandCapacity": 2,+ "InstanceTypeConfigs": [{"InstanceType": "m5.xlarge"}]}]}}+ """;+ private static final String STEP = """+ {"Name": "job", "ActionOnFailure": "CONTINUE",+ "HadoopJarStep": {"Jar": "command-runner.jar", "Args": ["spark-submit", "s3://bucket/job.py"]}}+ """;+ private static final ObjectMapper MAPPER = new ObjectMapper();++ @BeforeAll+ static void configureRestAssured() {+ RestAssuredJsonUtils.configureAwsContentTypes();+ }++ @Test+ void aClusterIsCreatedGivenAStepResizedAndTerminated() throws Exception {+ String definition = """+ {"StartAt": "Create", "States": {+ "Create": {"Type": "Task", "Resource": "%1$screateCluster.sync", "Parameters": %2$s,+ "ResultPath": "$.cluster", "Next": "Run"},+ "Run": {"Type": "Task", "Resource": "%1$saddStep.sync",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId", "Step": %3$s},+ "ResultPath": "$.step", "Next": "Queue"},+ "Queue": {"Type": "Task", "Resource": "%1$saddStep",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId", "Step": %3$s},+ "ResultPath": "$.queued", "Next": "Cancel"},+ "Cancel": {"Type": "Task", "Resource": "%1$scancelStep",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId", "StepId.$": "$.queued.StepId"},+ "ResultPath": "$.cancel", "Next": "Resize"},+ "Resize": {"Type": "Task", "Resource": "%1$smodifyInstanceGroupByName",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId", "InstanceGroupName": "Core",+ "InstanceGroup": {"InstanceCount": 4}},+ "ResultPath": "$.resize", "Next": "Terminate"},+ "Terminate": {"Type": "Task", "Resource": "%1$sterminateCluster.sync",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId"},+ "ResultPath": "$.terminated", "End": true}}}+ """.formatted(RESOURCE, GROUPS_CLUSTER, STEP);++ Response execution = run(definition);++ assertEquals("SUCCEEDED", execution.jsonPath().getString("status"), execution.asString());+ JsonNode output = MAPPER.readTree(execution.jsonPath().getString("output"));+ JsonNode cluster = output.path("cluster");+ String clusterId = cluster.path("ClusterId").asText();+ assertEquals(List.of("Cluster", "ClusterId"), fieldNames(cluster));+ assertEquals(clusterId, cluster.path("Cluster").path("Id").asText());+ assertEquals("WAITING", cluster.path("Cluster").path("Status").path("State").asText());+ JsonNode step = output.path("step");+ assertEquals(List.of("Step", "StepId"), fieldNames(step));+ assertEquals(step.path("StepId").asText(), step.path("Step").path("Id").asText());+ assertEquals("COMPLETED", step.path("Step").path("Status").path("State").asText());+ String queuedId = output.path("queued").path("StepId").asText();+ assertEquals(List.of("StepId"), fieldNames(output.path("queued")));+ assertTrue(queuedId.startsWith("s-"), queuedId);+ JsonNode cancelled = output.path("cancel").path("CancelStepsInfo");+ assertEquals(queuedId, cancelled.path("StepId").asText());+ assertEquals("FAILED", cancelled.path("Status").asText());+ assertEquals(MAPPER.createObjectNode(), output.path("resize"));+ assertEquals("TERMINATED", output.path("terminated").path("Cluster").path("Status").path("State").asText());++ emr(REGION, "ListInstanceGroups", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(200)+ .body("InstanceGroups.find { it.Name == 'Core' }.RequestedInstanceCount", equalTo(4));+ emr("DescribeCluster", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(400);++ JsonNode submitted = history(execution).stream()+ .filter(event -> "TaskSubmitted".equals(event.path("type").asText()))+ .findFirst().orElseThrow().path("taskSubmittedEventDetails");+ assertEquals("elasticmapreduce", submitted.path("resourceType").asText());+ assertEquals("createCluster.sync", submitted.path("resource").asText());+ assertEquals(clusterId, MAPPER.readTree(submitted.path("output").asText()).path("ClusterId").asText());+ }++ @Test+ void anInstanceFleetIsResizedByName() throws Exception {+ String definition = """+ {"StartAt": "Create", "States": {+ "Create": {"Type": "Task", "Resource": "%1$screateCluster", "Parameters": %2$s,+ "ResultPath": "$.cluster", "Next": "Resize"},+ "Resize": {"Type": "Task", "Resource": "%1$smodifyInstanceFleetByName",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId", "InstanceFleetName": "Core",+ "InstanceFleet": {"TargetOnDemandCapacity": 5}},+ "ResultPath": "$.resize", "End": true}}}+ """.formatted(RESOURCE, FLEETS_CLUSTER);++ Response execution = run(definition);++ assertEquals("SUCCEEDED", execution.jsonPath().getString("status"), execution.asString());+ JsonNode output = MAPPER.readTree(execution.jsonPath().getString("output"));+ assertEquals(List.of("ClusterId"), fieldNames(output.path("cluster")));+ emr(REGION, "ListInstanceFleets", "{\"ClusterId\":\"" + output.path("cluster").path("ClusterId").asText() + "\"}")+ .then().statusCode(200)+ .body("InstanceFleets.find { it.Name == 'Core' }.TargetOnDemandCapacity", equalTo(5));+ }++ @Test+ void createClusterRefusesWhatTheGuideRulesOut() {+ assertFails(task("createCluster", "{\"Name\": \"transient\", \"Instances\": {}}"), "States.TaskFailed",+ "The field Instances.KeepJobFlowAliveWhenNoSteps is mandatory, and must have the Boolean value TRUE.");+ assertFails(task("createCluster", """+ {"Name": "with-steps", "Instances": {"KeepJobFlowAliveWhenNoSteps": true}, "Steps": [%s]}+ """.formatted(STEP)), "States.TaskFailed", "The field Steps is not allowed.");+ }++ @Test+ void aTaskWithoutTheFieldsItsApiActsOnFails() {+ assertFails(task("terminateCluster", "{}"), "States.TaskFailed", "The field ClusterId is required.");+ assertFails(task("addStep.sync", "{\"ClusterId\": \" \", \"Step\": " + STEP + "}"), "States.TaskFailed",+ "The field ClusterId is required.");+ assertFails(task("cancelStep", "{\"ClusterId\": \"j-ANY\"}"), "States.TaskFailed",+ "The field StepId is required.");+ assertFails(task("setClusterTerminationProtection", "{\"ClusterId\": \"j-ANY\"}"),+ "EMR.AmazonElasticMapReduceException", "1 validation error detected: Value null at "+ + "'terminationProtected' failed to satisfy constraint: Member must not be null");+ assertFails(task("modifyInstanceGroupByName", "{\"ClusterId\": \"j-ANY\", \"InstanceGroup\": {}}"),+ "States.TaskFailed", "The field InstanceGroupName is required.");+ assertFails(task("modifyInstanceFleetByName", "{\"ClusterId\": \"j-ANY\", \"InstanceFleet\": {}}"),+ "States.TaskFailed", "The field InstanceFleetName is required.");+ }++ @Test+ void emrErrorsAreNamedAfterTheSdkExceptionClass() {+ assertFails(task("addStep.sync", "{\"ClusterId\": \"j-NOSUCHCLUSTER\", \"Step\": " + STEP + "}"),+ "EMR.AmazonElasticMapReduceException", "Specified job flow ID does not exist.");+ assertFails(task("cancelStep", "{\"ClusterId\": \"j-NOSUCHCLUSTER\", \"StepId\": \"s-NOSUCHSTEP\"}"),+ "EMR.InvalidRequestException", "Cluster id 'j-NOSUCHCLUSTER' is not valid.");+ String protectedCluster = """+ {"StartAt": "Create", "States": {+ "Create": {"Type": "Task", "Resource": "%1$screateCluster", "Parameters": %2$s,+ "ResultPath": "$.cluster", "Next": "Protect"},+ "Protect": {"Type": "Task", "Resource": "%1$ssetClusterTerminationProtection",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId", "TerminationProtected": true},+ "ResultPath": null, "Next": "Terminate"},+ "Terminate": {"Type": "Task", "Resource": "%1$sterminateCluster",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId"}, "End": true}}}+ """.formatted(RESOURCE, GROUPS_CLUSTER);+ assertFails(protectedCluster, "EMR.AmazonElasticMapReduceException",+ "Could not shut down one or more job flows since they are termination protected.");+ }++ @Test+ void aResizeAfterTheClusterTerminatedFailsAsOnAws() {+ assertFails(resizeAfterTermination(GROUPS_CLUSTER, "modifyInstanceGroupByName",+ "\"InstanceGroupName\": \"Core\", \"InstanceGroup\": {\"InstanceCount\": 4}"),+ "EMR.AmazonElasticMapReduceException",+ "An instance group may only be modified when the cluster is running or waiting.");+ assertFails(resizeAfterTermination(FLEETS_CLUSTER, "modifyInstanceFleetByName",+ "\"InstanceFleetName\": \"Core\", \"InstanceFleet\": {\"TargetOnDemandCapacity\": 5}"),+ "EMR.InvalidRequestException",+ "A job flow that is shutting down, terminated, or finished may not be modified.");+ }++ @Test+ void aByNameLookupNeedsOneMatchAndAnUnsupportedPatternFails() {+ String unknownGroup = """+ {"StartAt": "Create", "States": {+ "Create": {"Type": "Task", "Resource": "%1$screateCluster", "Parameters": %2$s,+ "ResultPath": "$.cluster", "Next": "Resize"},+ "Resize": {"Type": "Task", "Resource": "%1$smodifyInstanceGroupByName",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId", "InstanceGroupName": "Task",+ "InstanceGroup": {"InstanceCount": 1}}, "End": true}}}+ """.formatted(RESOURCE, GROUPS_CLUSTER);+ assertFails(unknownGroup, "States.TaskFailed", "The cluster has no instance group named 'Task'.");+ assertFails(task("cancelStep.sync", "{\"ClusterId\": \"j-ANY\", \"StepId\": \"s-ANY\"}"),+ "States.TaskFailed", "Unsupported resource: " + RESOURCE + "cancelStep.sync");+ }++ private static String resizeAfterTermination(String cluster, String api, String resize) {+ return """+ {"StartAt": "Create", "States": {+ "Create": {"Type": "Task", "Resource": "%1$screateCluster", "Parameters": %2$s,+ "ResultPath": "$.cluster", "Next": "Terminate"},+ "Terminate": {"Type": "Task", "Resource": "%1$sterminateCluster.sync",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId"},+ "ResultPath": null, "Next": "Resize"},+ "Resize": {"Type": "Task", "Resource": "%1$s%3$s",+ "Parameters": {"ClusterId.$": "$.cluster.ClusterId", %4$s}, "End": true}}}+ """.formatted(RESOURCE, cluster, api, resize);+ }++ private static String task(String api, String parameters) {+ return """+ {"StartAt": "Emr", "States": {"Emr": {"Type": "Task", "Resource": "%s%s",+ "Parameters": %s, "End": true}}}+ """.formatted(RESOURCE, api, parameters);+ }++ private void assertFails(String definition, String error, String cause) {+ Response execution = run(definition);+ assertEquals("FAILED", execution.jsonPath().getString("status"), execution.asString());+ assertEquals(error, execution.jsonPath().getString("error"));+ assertEquals(cause, execution.jsonPath().getString("cause"));+ }++ private Response run(String definition) {+ String stateMachineArn = sfn("CreateStateMachine", MAPPER.createObjectNode()+ .put("name", "sfn-emr-" + UUID.randomUUID())+ .put("definition", definition)+ .put("roleArn", ROLE_ARN).toString())+ .then().statusCode(200).extract().path("stateMachineArn");+ String executionArn = sfn("StartExecution", MAPPER.createObjectNode()+ .put("stateMachineArn", stateMachineArn).put("input", "{}").toString())+ .then().statusCode(200).extract().path("executionArn");+ return StepFunctionsExecutions.waitForTerminalState(executionArn);+ }++ private List<JsonNode> history(Response execution) throws Exception {+ String body = sfn("GetExecutionHistory", MAPPER.createObjectNode()+ .put("executionArn", execution.jsonPath().getString("executionArn")).toString())+ .then().statusCode(200).extract().asString();+ List<JsonNode> events = new ArrayList<>();+ MAPPER.readTree(body).path("events").forEach(events::add);+ return events;+ }++ private static Response sfn(String action, String body) {+ return given()+ .header("Authorization", "AWS4-HMAC-SHA256 Credential=AKID/20260215/" + REGION+ + "/states/aws4_request, SignedHeaders=host, Signature=abc")+ .header("X-Amz-Target", "AWSStepFunctions." + action)+ .contentType("application/x-amz-json-1.0")+ .body(body)+ .when().post("/");+ }++ private static List<String> fieldNames(JsonNode node) {+ List<String> names = new ArrayList<>();+ node.fieldNames().forEachRemaining(names::add);+ return names;+ }+}