Light, fluffy, and always free - The AWS Local Emulator alternative
feat(emr): add an optional timed lifecycle for clusters and steps (#6441)
Priamo Germosen55b060b891eb
diff --git a/docs/services/emr.md b/docs/services/emr.mdindex 8790784736..fb4f93a16d 100644--- a/docs/services/emr.md+++ b/docs/services/emr.md@@ -55,25 +55,45 @@ Floci emulates the Amazon EMR (Elastic MapReduce) management API. Clusters (job## Cluster lifecycle-Steps complete immediately in this metadata-only emulator, in the order the Management Guide gives:-a request's steps are all created `PENDING`, then each completes before the next starts. A cluster-created with `Instances.KeepJobFlowAliveWhenNoSteps` set to `false` terminates after its initial steps or-steps added later complete, reporting `ALL_STEPS_COMPLETED`. A cluster with no steps remains-`WAITING` until steps are added. Set `KeepJobFlowAliveWhenNoSteps` to `true` to keep the cluster-`WAITING` after its steps complete. Automatic step-completion termination also applies when-termination protection is enabled.-`AddJobFlowSteps` rejects a terminated cluster without adding steps or changing its+Nothing runs in this metadata-only emulator, but a cluster and its steps go through the states of the+Management Guide's cluster lifecycle, each held for as long as the [configuration](#configuration)+says. A cluster is `STARTING` for the first half of `cluster-startup-delay-seconds` and `BOOTSTRAPPING`+for the second, then `RUNNING` while it runs steps and `WAITING` once it has none left. A request's+steps are all created `PENDING`, then start in the order they were added, at most `StepConcurrencyLevel`+at a time, each `RUNNING` for `step-duration-seconds`. Steps added while the cluster starts wait for it+to be ready. Raising the level with `ModifyCluster` starts the steps that now fit at once, and lowering+it lets the running ones complete. `TerminateJobFlows` makes a cluster `TERMINATING` for+`cluster-termination-delay-seconds` and then `TERMINATED`, and cancels any step not yet completed, as its+API reference says. Each call finds the clusters it reads as they stand at that moment. With every+duration at its default of 0, a cluster is ready, and its steps complete, before the call that started+them returns.++`CancelSteps` cancels a `PENDING` step, and a `RUNNING` one from release `emr-5.28.0`, as its API+reference says. A cancelled `RUNNING` step makes room for the next `PENDING` one. On an earlier release+a `RUNNING` step answers `FAILED` with "Step is not in a cancellable state: RUNNING".++A cluster created with `Instances.KeepJobFlowAliveWhenNoSteps` set to `false` terminates after its+initial steps or steps added later complete, reporting `ALL_STEPS_COMPLETED`, without waiting first. A+cluster with no steps remains `WAITING` until steps are added. Set `KeepJobFlowAliveWhenNoSteps` to+`true` to keep the cluster `WAITING` after its steps complete. Automatic step-completion termination+also applies when termination protection is enabled.+`AddJobFlowSteps` rejects a `TERMINATING` or `TERMINATED` cluster without adding steps or changing itstermination status, and answers an unknown cluster with `ValidationException` "Specified job flow IDdoes not exist.", as AWS does.-After automatic or manual termination, instance groups and fleets report `TERMINATED` with zero-running or provisioned capacity; their requested and target capacities remain recorded.-`ListInstances` retains each group's instances with their original synthetic IDs and `TERMINATED`-status for 30 days after termination. Capacity modifications and new groups or fleets are rejected+Instance groups and fleets report `RUNNING` from the cluster's creation until it is `TERMINATED`. Then+they report `TERMINATED` with zero running or provisioned capacity; their requested and target+capacities remain recorded. `ListInstances` retains each group's instances with their original+synthetic IDs and `TERMINATED` status for 30 days after termination. New groups or fleets are rejectedon terminated clusters. As on AWS, `ModifyInstanceGroups` answers `ValidationException` "An instance-group may only be modified when the cluster is running or waiting.", and `ModifyInstanceFleet` answers-`InvalidRequestException` "A job flow that is shutting down, terminated, or finished may not be-modified." with `ErrorCode` `INVALID_JOB_SHUT_DOWN`. `InstanceStates` filters the modeled `RUNNING` and-`TERMINATED` instances.+group may only be modified when the cluster is running or waiting." for a cluster in any other state,+and `ModifyInstanceFleet` answers `InvalidRequestException` "A job flow that is shutting down,+terminated, or finished may not be modified." with `ErrorCode` `INVALID_JOB_SHUT_DOWN` for a+`TERMINATING` or `TERMINATED` one, and "An instance fleet may only be modified when the cluster is+running or waiting." with `ErrorCode` `MODIFY_INSTANCE_FLEET_OF_RUNNING_JOBFLOW_ONLY` for one in any+other state, such as `STARTING` or `BOOTSTRAPPING`. `AddInstanceFleet` gives a `TERMINATING` cluster+the "shutting down" answer, and `AddInstanceGroups` gives it as `ValidationException`, as+`AddJobFlowSteps` does. `InstanceStates` filters the modeled `RUNNING` and `TERMINATED`+instances.`TerminateJobFlows`, the four `Set*` cluster setters and `CancelSteps` refuse a request without amember their model requires (`JobFlowIds` and the setting, or `StepIds`) with `ValidationException`@@ -95,29 +115,43 @@ examples. A cluster's detail holds `severity`, `stateChangeReason`, `name`, `clu`message`. The severity and message are the ones the guide's "Monitoring Amazon EMR events" tablesgive each state, with times to the minute (`2026-10-24 09:41 UTC`).-Since steps complete at once, each call sends:+Each change sends its events:-| Call | Events, in order |+| Change | Events, in order ||---|---|-| `RunJobFlow` | the cluster `STARTING`; each step `PENDING`, then each `COMPLETED`; then the cluster `WAITING`, or `TERMINATED` with `ALL_STEPS_COMPLETED` when it ends after its steps |-| `AddJobFlowSteps` | each step `PENDING`, then each `COMPLETED`; then the cluster `TERMINATED` with `ALL_STEPS_COMPLETED` when it ends after them |-| `TerminateJobFlows` | `TERMINATED` with `USER_REQUEST` for each cluster it ends, also when a protected cluster fails the call |-| `CancelSteps` | `CANCELLED` for each step it cancels, none while steps complete at once |+| `RunJobFlow` | the cluster `STARTING`, then each step `PENDING` |+| `AddJobFlowSteps` | each step `PENDING` |+| The cluster is ready, or a step completes | the step `COMPLETED`; each step it makes room for `RUNNING`; once none is left, the cluster `WAITING`, or `TERMINATED` with `ALL_STEPS_COMPLETED` when it ends after its steps |+| `TerminateJobFlows` | `CANCELLED` for each step not yet completed; the cluster `TERMINATED` with `USER_REQUEST` once it has been `TERMINATING` for the delay, also when a protected cluster fails the call |+| `CancelSteps` | `CANCELLED` for each step it cancels, and `RUNNING` for the step a cancelled `RUNNING` one makes room for |++A change that falls due between calls, such as a step completing after `step-duration-seconds`, is+made at that time and its events sent then, whether or not anything reads the cluster.A `STARTING` or `WAITING` cluster's `stateChangeReason` is `{"code":""}`, as in the guide's example,and a `TERMINATED` one's carries the reason's code and message, such as`{"code":"USER_REQUEST","message":"Terminated by user request"}`. `TERMINATED` is `INFO` for`USER_REQUEST` and `ALL_STEPS_COMPLETED` and `CRITICAL` for any other reason; a cancelled step is`WARN`. The tables have no row for a cluster's `BOOTSTRAPPING`, `RUNNING` or `TERMINATING`, which-send nothing. A step is never seen `RUNNING`, and a `WAITING` cluster that runs added steps stays-`WAITING`, so neither sends an event for it. The guide gives two `WAITING` messages without saying-when each applies: a cluster that ran steps reports it "finished running all pending steps", one that-did not that it "was created ... and is ready for use".+send nothing, so a `WAITING` cluster that runs added steps sends `WAITING` again when they are done.+The guide gives two `WAITING` messages without saying when each applies: a cluster that ran steps+reports it "finished running all pending steps", one that did not that it "was created ... and is+ready for use".Events go out in the order their changes were made, once the call that made them no longer holds thecluster. A rule's target that calls EMR while an event is delivered, such as one ending the cluster onits `STARTING` event, has its events sent after the rest of that call's.+## Configuration++| Variable | Default | Description |+|---|---|---|+| `FLOCI_SERVICES_EMR_ENABLED` | `true` | Enable or disable the service |+| `FLOCI_SERVICES_EMR_DEFAULT_RELEASE_LABEL` | `emr-7.5.0` | The release label of a cluster created without one; see [Release catalogue](#release-catalogue) |+| `FLOCI_SERVICES_EMR_CLUSTER_STARTUP_DELAY_SECONDS` | `0` | Seconds a new cluster takes to be ready to run steps, the first half `STARTING` and the second `BOOTSTRAPPING`; `0` makes it ready as soon as it is created |+| `FLOCI_SERVICES_EMR_STEP_DURATION_SECONDS` | `0` | Seconds a step stays `RUNNING` before it completes; `0` completes it as soon as it starts |+| `FLOCI_SERVICES_EMR_CLUSTER_TERMINATION_DELAY_SECONDS` | `0` | Seconds a cluster stays `TERMINATING` before it is `TERMINATED`; `0` ends it at once |+## Release catalogue`ListReleaseLabels`, `DescribeReleaseLabel` and `ListSupportedInstanceTypes` read a curated catalogue,diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex 34ec80ebf6..2ab9f25014 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -1937,9 +1937,20 @@ public interface EmulatorConfig {@WithDefault("emr-7.5.0")String defaultReleaseLabel();- /** Delay before a cluster reaches WAITING; 0 = advance synchronously. */+ /**+ * How long a new cluster takes to be ready to run steps, the first half STARTING and the second+ * BOOTSTRAPPING; 0 makes it ready as soon as it is created.+ */@WithDefault("0")int clusterStartupDelaySeconds();++ /** How long a step stays RUNNING before it completes; 0 completes it as soon as it starts. */+ @WithDefault("0")+ int stepDurationSeconds();++ /** How long a cluster stays TERMINATING before it is TERMINATED; 0 ends it at once. */+ @WithDefault("0")+ int clusterTerminationDelaySeconds();}interface WafV2ServiceConfig {diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.javanew file mode 100644index 0000000000..8beb619342--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.java@@ -0,0 +1,258 @@+package io.github.hectorvent.floci.services.emr;++import io.github.hectorvent.floci.services.emr.model.EmrCluster;+import io.github.hectorvent.floci.services.emr.model.EmrInstanceFleet;+import io.github.hectorvent.floci.services.emr.model.EmrInstanceGroup;+import io.github.hectorvent.floci.services.emr.model.EmrStep;++import java.time.Duration;+import java.time.Instant;+import java.util.Comparator;+import java.util.List;+import java.util.regex.Matcher;+import java.util.regex.Pattern;++/**+ * A cluster's and its steps' lifecycle, as the Management Guide gives it in "Understanding the cluster+ * lifecycle" and "Running steps to process data". A cluster is STARTING, then BOOTSTRAPPING, then+ * RUNNING while it runs its steps, in the order they were added and at most its StepConcurrencyLevel+ * at a time, then WAITING, or TERMINATING and TERMINATED when it ends after them. Nothing runs: each+ * change falls due at a time worked out from three durations and the times already recorded, and+ * {@link #advance} makes every change due by a given time, in the order they fall due, queuing the+ * event of each state the guide's event tables report. With the durations at 0, every change falls+ * due at the time of the call that leads to it.+ */+final class EmrClusterLifecycle {++ static final String STARTING = "STARTING";+ static final String BOOTSTRAPPING = "BOOTSTRAPPING";+ static final String RUNNING = "RUNNING";+ static final String WAITING = "WAITING";+ static final String TERMINATING = "TERMINATING";+ static final String TERMINATED = "TERMINATED";+ static final String PENDING = "PENDING";+ static final String COMPLETED = "COMPLETED";+ static final String CANCELLED = "CANCELLED";+ private static final Pattern RELEASE_LABEL = Pattern.compile("emr-(\\d{1,6})\\.(\\d{1,6})(?:\\..*)?");++ private final Duration startup;+ private final Duration stepRun;+ private final Duration termination;+ private final EmrEventPublisher events;++ EmrClusterLifecycle(int startupDelaySeconds, int stepDurationSeconds, int terminationDelaySeconds,+ EmrEventPublisher events) {+ this.startup = seconds("cluster-startup-delay-seconds", startupDelaySeconds);+ this.stepRun = seconds("step-duration-seconds", stepDurationSeconds);+ this.termination = seconds("cluster-termination-delay-seconds", terminationDelaySeconds);+ this.events = events;+ }++ private static Duration seconds(String setting, int value) {+ if (value < 0) {+ throw new IllegalArgumentException("floci.services.emr." + setting + " must not be negative: " + value);+ }+ return Duration.ofSeconds(value);+ }++ /** A new cluster, STARTING at {@code now}, with its steps PENDING. */+ void start(EmrCluster cluster, Instant now, String region) {+ cluster.setState(STARTING);+ cluster.setCreationDateTime(now);+ events.queue(events.clusterStateChange(cluster, STARTING, now), region);+ submit(cluster, cluster.getSteps(), now, region);+ }++ /**+ * {@code steps}, already on the cluster's list, added at {@code now}: "The state of all steps is set+ * to PENDING", and a cluster that is ready starts what it has room for. Adding none changes nothing.+ */+ void submit(EmrCluster cluster, List<EmrStep> steps, Instant now, String region) {+ if (steps.isEmpty()) {+ return;+ }+ for (EmrStep step : steps) {+ step.setState(PENDING);+ step.setCreationDateTime(now);+ if (step.getActionOnFailure() == null) {+ step.setActionOnFailure("TERMINATE_CLUSTER");+ }+ events.queue(events.stepStatusChange(cluster, step, PENDING), region);+ }+ if (RUNNING.equals(cluster.getState()) || WAITING.equals(cluster.getState())) {+ runSteps(cluster, now, region);+ }+ }++ /**+ * Ends the cluster for {@code code}: TERMINATING from {@code now}, then TERMINATED. TerminateJobFlows:+ * "When a job flow is shut down, any step not yet completed is canceled".+ */+ void terminate(EmrCluster cluster, String code, String message, Instant now, String region) {+ cluster.setState(TERMINATING);+ cluster.setStateChangeReasonCode(code);+ cluster.setStateChangeMessage(message);+ cluster.setTerminatingDateTime(now);+ for (EmrStep step : cluster.getSteps()) {+ if (PENDING.equals(step.getState()) || RUNNING.equals(step.getState())) {+ cancelStep(cluster, step, now, region);+ }+ }+ }++ /**+ * CancelSteps for one step: null once it is cancelled, or why it cannot be. CancelSteps: "When you+ * use Amazon EMR releases 5.28.0 and later, you can cancel steps that are in a PENDING or RUNNING+ * state." A cancelled RUNNING step makes room for the next PENDING one.+ */+ String cancel(EmrCluster cluster, EmrStep step, Instant now, String region) {+ boolean running = RUNNING.equals(step.getState()) && cancelsRunningSteps(cluster.getReleaseLabel());+ if (!running && !PENDING.equals(step.getState())) {+ return "Step is not in a cancellable state: " + step.getState();+ }+ cancelStep(cluster, step, now, region);+ if (running) {+ runSteps(cluster, now, region);+ }+ return null;+ }++ /**+ * The cluster's StepConcurrencyLevel changed: a RUNNING cluster starts what it now has room for. A+ * lower level lets the steps already running complete, as the guide says EMR does.+ */+ void concurrencyChanged(EmrCluster cluster, Instant now, String region) {+ if (RUNNING.equals(cluster.getState())) {+ runSteps(cluster, now, region);+ }+ }++ /** Makes every change due by {@code now}, in the order they fall due; true when there was one. */+ boolean advance(EmrCluster cluster, Instant now, String region) {+ boolean changed = false;+ Instant due;+ while ((due = nextChange(cluster)) != null && !due.isAfter(now)) {+ change(cluster, due, region);+ changed = true;+ }+ return changed;+ }++ /** When the cluster's next change falls due, or null while it waits for a call. */+ Instant nextChange(EmrCluster cluster) {+ return switch (cluster.getState()) {+ case STARTING -> cluster.getCreationDateTime().plus(startup.dividedBy(2));+ case BOOTSTRAPPING -> cluster.getCreationDateTime().plus(startup);+ case RUNNING -> cluster.getSteps().stream()+ .filter(step -> RUNNING.equals(step.getState()))+ .map(step -> step.getStartDateTime().plus(stepRun))+ .min(Comparator.naturalOrder())+ .orElse(null);+ case TERMINATING -> cluster.getTerminatingDateTime().plus(termination);+ default -> null;+ };+ }++ private void change(EmrCluster cluster, Instant at, String region) {+ switch (cluster.getState()) {+ case STARTING -> cluster.setState(BOOTSTRAPPING);+ case BOOTSTRAPPING -> {+ cluster.setReadyDateTime(at);+ cluster.setState(RUNNING);+ runSteps(cluster, at, region);+ }+ case RUNNING -> {+ for (EmrStep step : cluster.getSteps()) {+ if (RUNNING.equals(step.getState()) && !step.getStartDateTime().plus(stepRun).isAfter(at)) {+ step.setState(COMPLETED);+ step.setEndDateTime(at);+ events.queue(events.stepStatusChange(cluster, step, COMPLETED), region);+ }+ }+ runSteps(cluster, at, region);+ }+ case TERMINATING -> {+ cluster.setState(TERMINATED);+ cluster.setEndDateTime(at);+ releaseInstanceCapacity(cluster);+ events.queue(events.clusterStateChange(cluster, TERMINATED, at), region);+ }+ default -> throw new IllegalStateException("Nothing falls due for a " + cluster.getState() + " cluster");+ }+ }++ /**+ * Starts PENDING steps in the order they were added while the cluster has room: "pending steps in+ * queue transition to running state in the order they were submitted". With none left, the+ * cluster is done.+ */+ private void runSteps(EmrCluster cluster, Instant at, String region) {+ long running = cluster.getSteps().stream().filter(step -> RUNNING.equals(step.getState())).count();+ // The model allows 1 to 256; Floci stores any level it is given, and a lower one would never start a step.+ int level = Math.max(1, cluster.getStepConcurrencyLevel());+ for (EmrStep step : cluster.getSteps()) {+ if (running >= level) {+ break;+ }+ if (PENDING.equals(step.getState())) {+ step.setState(RUNNING);+ step.setStartDateTime(at);+ events.queue(events.stepStatusChange(cluster, step, RUNNING), region);+ running++;+ }+ }+ if (running > 0) {+ // A WAITING cluster's "ready to run steps" no longer holds; no source gives a RUNNING one's reason.+ cluster.setState(RUNNING);+ cluster.setStateChangeReasonCode(null);+ cluster.setStateChangeMessage(null);+ } else {+ finish(cluster, at, region);+ }+ }++ /**+ * The cluster has no step left to run: it waits for more, or, set to end after its steps, ends+ * without waiting. One created without steps waits either way.+ */+ private void finish(EmrCluster cluster, Instant at, String region) {+ if (cluster.isAutoTerminate() && !cluster.getSteps().isEmpty()) {+ terminate(cluster, "ALL_STEPS_COMPLETED", "All steps completed.", at, region);+ return;+ }+ cluster.setState(WAITING);+ cluster.setStateChangeReasonCode(cluster.getSteps().isEmpty() ? null : "ALL_STEPS_COMPLETED");+ cluster.setStateChangeMessage("Cluster ready to run steps.");+ events.queue(events.clusterStateChange(cluster, WAITING, at), region);+ }++ private void cancelStep(EmrCluster cluster, EmrStep step, Instant at, String region) {+ step.setState(CANCELLED);+ step.setEndDateTime(at);+ events.queue(events.stepStatusChange(cluster, step, CANCELLED), region);+ }++ private static void releaseInstanceCapacity(EmrCluster cluster) {+ for (EmrInstanceGroup group : cluster.getInstanceGroups()) {+ group.setState(TERMINATED);+ group.setTerminatedInstanceCount(group.getRunningInstanceCount());+ group.setRunningInstanceCount(0);+ }+ for (EmrInstanceFleet fleet : cluster.getInstanceFleets()) {+ fleet.setState(TERMINATED);+ fleet.setProvisionedOnDemandCapacity(0);+ fleet.setProvisionedSpotCapacity(0);+ }+ }++ /** Releases from 5.28.0 cancel RUNNING steps; a label not of the emr-x.y form is taken as current. */+ static boolean cancelsRunningSteps(String releaseLabel) {+ Matcher matcher = RELEASE_LABEL.matcher(releaseLabel == null ? "" : releaseLabel);+ if (!matcher.matches()) {+ return true;+ }+ int major = Integer.parseInt(matcher.group(1));+ int minor = Integer.parseInt(matcher.group(2));+ return major > 5 || (major == 5 && minor >= 28);+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrEventPublisher.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrEventPublisher.javaindex 7902a1642e..38d2db4b34 100644--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrEventPublisher.java+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrEventPublisher.java@@ -59,8 +59,8 @@ public class EmrEventPublisher {this.regionResolver = regionResolver;}- /** {@code cluster} entering {@code state}: STARTING, WAITING or TERMINATED. */- Event clusterStateChange(EmrCluster cluster, String state) {+ /** {@code cluster} entering {@code state} at {@code at}: STARTING, WAITING or TERMINATED. */+ Event clusterStateChange(EmrCluster cluster, String state, Instant at) {boolean terminated = "TERMINATED".equals(state);String code = cluster.getStateChangeReasonCode();// The examples' reason is a JSON string: {"code":""} for a WAITING cluster, and the reason's code@@ -76,11 +76,11 @@ public class EmrEventPublisher {detail.put("name", cluster.getName());detail.put("clusterId", cluster.getId());detail.put("state", state);- detail.put("message", clusterMessage(cluster, state));+ detail.put("message", clusterMessage(cluster, state, at));return new Event(CLUSTER_STATE_CHANGE, detail.toString());}- /** {@code step} of {@code cluster} entering {@code state}: PENDING, COMPLETED or CANCELLED. */+ /** {@code step} of {@code cluster} entering {@code state}: PENDING, RUNNING, COMPLETED or CANCELLED. */Event stepStatusChange(EmrCluster cluster, EmrStep step, String state) {ObjectNode detail = objectMapper.createObjectNode();detail.put("severity", "CANCELLED".equals(state) ? "WARN" : "INFO");@@ -126,17 +126,15 @@ public class EmrEventPublisher {SOURCE, queued.event().detailType(), queued.event().detail(), null, queued.region()));}- private static String clusterMessage(EmrCluster cluster, String state) {+ private static String clusterMessage(EmrCluster cluster, String state, Instant at) {String named = cluster.getId() + " (" + cluster.getName() + ")";return switch (state) {- case "STARTING" -> "Amazon EMR cluster " + named + " was requested at "- + time(cluster.getCreationDateTime()) + " and is being created.";+ case "STARTING" -> "Amazon EMR cluster " + named + " was requested at " + time(at)+ + " and is being created.";case "WAITING" -> cluster.getSteps().isEmpty()- ? "Amazon EMR cluster " + named + " was created at " + time(cluster.getReadyDateTime())- + " and is ready for use."- : "Amazon EMR cluster " + named + " finished running all pending steps at "- + time(cluster.getReadyDateTime()) + ".";- case "TERMINATED" -> "Amazon EMR Cluster " + named + " has terminated at " + time(cluster.getEndDateTime())+ ? "Amazon EMR cluster " + named + " was created at " + time(at) + " and is ready for use."+ : "Amazon EMR cluster " + named + " finished running all pending steps at " + time(at) + ".";+ case "TERMINATED" -> "Amazon EMR Cluster " + named + " has terminated at " + time(at)+ " with a reason of " + cluster.getStateChangeReasonCode() + ".";default -> throw new IllegalArgumentException("No cluster event for state " + state);};@@ -148,6 +146,8 @@ public class EmrEventPublisher {return switch (state) {case "PENDING" -> "Step " + stepNamed + " was added to Amazon EMR cluster " + clusterNamed + " at "+ time(step.getCreationDateTime()) + " and is pending execution.";+ case "RUNNING" -> "Step " + stepNamed + " in Amazon EMR cluster " + clusterNamed + " started running at "+ + time(step.getStartDateTime()) + ".";case "COMPLETED" -> "Step " + stepNamed + " in Amazon EMR cluster " + clusterNamed+ " completed execution at " + time(step.getEndDateTime()) + ". The step started running at "+ time(step.getStartDateTime())diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.javaindex 5b4caf1844..c96d850ea9 100644--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java@@ -2,13 +2,17 @@ package io.github.hectorvent.floci.services.emr;import com.fasterxml.jackson.core.type.TypeReference;import com.fasterxml.jackson.databind.JsonNode;+import com.fasterxml.jackson.databind.ObjectMapper;+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.core.common.AwsRegions;import io.github.hectorvent.floci.core.common.PaginatedResult;import io.github.hectorvent.floci.core.common.Pagination;import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.common.RequestScopes;import io.github.hectorvent.floci.core.common.SmithyValidationErrors;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;import io.github.hectorvent.floci.core.storage.StorageBackend;import io.github.hectorvent.floci.core.storage.StorageFactory;import io.github.hectorvent.floci.services.emr.model.EmrBlockPublicAccess;@@ -17,10 +21,16 @@ import io.github.hectorvent.floci.services.emr.model.EmrInstanceFleet;import io.github.hectorvent.floci.services.emr.model.EmrInstanceGroup;import io.github.hectorvent.floci.services.emr.model.EmrStep;import io.github.hectorvent.floci.services.emr.model.SecurityConfiguration;+import io.quarkus.runtime.StartupEvent;+import jakarta.annotation.PreDestroy;import jakarta.enterprise.context.ApplicationScoped;+import jakarta.enterprise.event.Observes;import jakarta.inject.Inject;+import org.jboss.logging.Logger;import java.security.SecureRandom;+import java.time.Clock;+import java.time.Duration;import java.time.Instant;import java.time.temporal.ChronoUnit;import java.util.ArrayList;@@ -31,32 +41,46 @@ import java.util.Map;import java.util.Objects;import java.util.Optional;import java.util.Set;+import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.ScheduledExecutorService;+import java.util.concurrent.TimeUnit;import java.util.function.Function;+import java.util.function.Supplier;/**- * EMR management-plane business logic. Cluster and step lifecycles are simulated- * through immediate state transitions (no Hadoop runs): steps complete immediately,- * leaving persistent clusters WAITING and transient clusters with steps TERMINATED.- * The {@code clusterStartupDelaySeconds} config exists for- * future deferred transitions; Phase 1 advances synchronously for deterministic tests.- * Every state a cluster or step enters is published as an {@code aws.emr} event, once the call that- * made the change no longer holds the service.+ * EMR management-plane business logic. Nothing runs: clusters and steps go through the states of+ * {@link EmrClusterLifecycle}, timed by {@code cluster-startup-delay-seconds},+ * {@code step-duration-seconds} and {@code cluster-termination-delay-seconds}, all 0 by default, which+ * makes a cluster ready, and a step complete, before the call that started it returns. A call brings+ * each cluster it reads up to the time it is made, and the service wakes at each cluster's next+ * change, so the change's event goes out without anyone asking. Events are published once the call+ * that made the change no longer holds the service.*/@ApplicationScopedpublic class EmrService {+ private static final Logger LOG = Logger.getLogger(EmrService.class);private static final String UPPER_ALNUM = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";private static final SecureRandom RANDOM = new SecureRandom();- // AWS's answer to a change on a terminated cluster, for AddJobFlowSteps and ModifyInstanceFleet.+ // AWS's answer to a change on a terminated cluster, for AddJobFlowSteps and ModifyInstanceFleet; Floci+ // gives it to AddInstanceGroups and AddInstanceFleet too, for one shutting down.private static final String JOB_FLOW_SHUT_DOWN ="A job flow that is shutting down, terminated, or finished may not be modified.";+ private static final ObjectMapper SNAPSHOT_MAPPER = new ObjectMapper().registerModule(new JavaTimeModule());- private final StorageBackend<String, EmrCluster> clusterStore;+ private final AccountAwareStorageBackend<EmrCluster> clusterStore;private final StorageBackend<String, SecurityConfiguration> secConfigStore;// Block public access is an account setting per Region, so this store is keyed by Region.private final StorageBackend<String, EmrBlockPublicAccess> blockPublicAccessStore;private final RegionResolver regionResolver;private final EmrEventPublisher eventPublisher;+ private final EmrClusterLifecycle lifecycle;+ private final Clock clock;+ private final Wakeups wakeups;+ // Each cluster's next wakeup, by account and id, so a call that reads a cluster does not add another.+ private final Map<String, Instant> wakeupsDue = new ConcurrentHashMap<>();private static final Set<String> COMPUTE_LIMITS_UNIT_TYPES = Set.of("InstanceFleetUnits", "Instances", "VCPU");private static final Set<String> SCALING_STRATEGIES = Set.of("DEFAULT", "ADVANCED");@@ -69,9 +93,62 @@ public class EmrService {+ "\"PermittedPublicSecurityGroupRuleRanges\":[{\"MinRange\":22,\"MaxRange\":22}]}";private final String defaultReleaseLabel;+ /**+ * When the service next brings a cluster up to date by itself, and where it then sends the events+ * that queued; a test runs both when it chooses.+ */+ interface Wakeups {++ void at(Instant due, Runnable task);++ /** Sends a wakeup's events, by default on the wakeup's own thread. */+ default void deliver(Runnable delivery) {+ delivery.run();+ }++ default void close() {+ }++ /**+ * Wakeups on a daemon thread, each at its time by {@code clock}, and their events sent on another,+ * in the order they queued: a rule's target can take seconds to answer, and no cluster's change+ * waits for it.+ */+ static Wakeups timer(Clock clock) {+ ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(+ Thread.ofPlatform().daemon().name("emr-lifecycle").factory());+ ExecutorService deliveries = Executors.newSingleThreadExecutor(+ Thread.ofPlatform().daemon().name("emr-events").factory());+ return new Wakeups() {+ @Override+ public void at(Instant due, Runnable task) {+ long delay = Math.max(0, Duration.between(clock.instant(), due).toMillis());+ executor.schedule(task, delay, TimeUnit.MILLISECONDS);+ }++ @Override+ public void deliver(Runnable delivery) {+ deliveries.execute(delivery);+ }++ @Override+ public void close() {+ executor.shutdownNow();+ deliveries.shutdownNow();+ }+ };+ }+ }+@Injectpublic EmrService(StorageFactory storageFactory, EmulatorConfig config, RegionResolver regionResolver,EmrEventPublisher eventPublisher) {+ this(storageFactory, config, regionResolver, eventPublisher, Clock.systemUTC(),+ Wakeups.timer(Clock.systemUTC()));+ }++ EmrService(StorageFactory storageFactory, EmulatorConfig config, RegionResolver regionResolver,+ EmrEventPublisher eventPublisher, Clock clock, Wakeups wakeups) {this.clusterStore = storageFactory.create("emr", "emr-clusters.json",new TypeReference<Map<String, EmrCluster>>() {});this.secConfigStore = storageFactory.create("emr", "emr-security-configs.json",@@ -81,55 +158,68 @@ public class EmrService {this.regionResolver = regionResolver;this.eventPublisher = eventPublisher;this.defaultReleaseLabel = config.services().emr().defaultReleaseLabel();+ this.lifecycle = new EmrClusterLifecycle(config.services().emr().clusterStartupDelaySeconds(),+ config.services().emr().stepDurationSeconds(),+ config.services().emr().clusterTerminationDelaySeconds(), eventPublisher);+ this.clock = clock;+ this.wakeups = wakeups;+ // After a restart, a stored cluster with a change still to come wakes the service as before.+ for (AccountAwareStorageBackend.AccountEntry<EmrCluster> entry+ : clusterStore.scanAllAccountEntries(k -> true)) {+ Instant due = lifecycle.nextChange(entry.value());+ if (due != null) {+ wakeAt(entry.accountId(), entry.value(), due);+ }+ }}- // ──────────────────────────── Cluster lifecycle ────────────────────────────+ // A normal-scoped bean is built on first use, so without this a cluster left mid-change by a restart+ // would not move on, nor send its events, until the next EMR call.+ void onStart(@Observes StartupEvent ignored) {+ LOG.debugv("EMR service ready");+ }- public EmrCluster runJobFlow(EmrCluster cluster, String region) {- try {- return runJobFlowLocked(cluster, region);- } finally {- eventPublisher.publishQueued();- }+ @PreDestroy+ void shutdown() {+ wakeups.close();}- private synchronized EmrCluster runJobFlowLocked(EmrCluster cluster, String region) {- // Only a missing Name fails AWS's framework validation: the member is REQUIRED but its- // constraint is len=[0..256], so an explicit empty string is accepted by real EMR.- if (cluster.getName() == null) {- throw SmithyValidationErrors.validationException(SmithyValidationErrors.notNull("name"));- }- String id = "j-" + randomId(13);- cluster.setId(id);- cluster.setRegion(region);- cluster.setClusterArn(regionResolver.buildArn("elasticmapreduce", region, "cluster/" + id));- if (cluster.getReleaseLabel() == null) {- cluster.setReleaseLabel(defaultReleaseLabel);- }- cluster.setInstanceCollectionType(cluster.getInstanceFleets().isEmpty()- ? "INSTANCE_GROUP" : "INSTANCE_FLEET");- cluster.setAutoTerminate(!cluster.isKeepJobFlowAliveWhenNoSteps());- cluster.setMasterPublicDnsName(masterDnsName(region));- cluster.setCreationDateTime(Instant.now().truncatedTo(ChronoUnit.MILLIS));- for (EmrInstanceGroup g : cluster.getInstanceGroups()) {- g.setId("ig-" + randomId(13));- g.setRunningInstanceCount(g.getRequestedInstanceCount());- g.setState("RUNNING");- }- for (EmrInstanceFleet f : cluster.getInstanceFleets()) {- f.setId("if-" + randomId(13));- f.setProvisionedOnDemandCapacity(f.getTargetOnDemandCapacity());- f.setProvisionedSpotCapacity(f.getTargetSpotCapacity());- f.setState("RUNNING");- }- eventPublisher.queue(eventPublisher.clusterStateChange(cluster, "STARTING"), region);- runSteps(cluster, cluster.getSteps(), region);- advanceToWaiting(cluster);- terminateAfterCompletedSteps(cluster);- clusterStore.put(id, cluster);- // The WAITING a transient cluster passes through here is never seen, so it has no event.- eventPublisher.queue(eventPublisher.clusterStateChange(cluster, cluster.getState()), region);- return cluster;+ // ──────────────────────────── Cluster lifecycle ────────────────────────────++ public EmrCluster runJobFlow(EmrCluster cluster, String region) {+ return call(() -> {+ // Only a missing Name fails AWS's framework validation: the member is REQUIRED but its+ // constraint is len=[0..256], so an explicit empty string is accepted by real EMR.+ if (cluster.getName() == null) {+ throw SmithyValidationErrors.validationException(SmithyValidationErrors.notNull("name"));+ }+ String id = "j-" + randomId(13);+ cluster.setId(id);+ cluster.setRegion(region);+ cluster.setClusterArn(regionResolver.buildArn("elasticmapreduce", region, "cluster/" + id));+ if (cluster.getReleaseLabel() == null) {+ cluster.setReleaseLabel(defaultReleaseLabel);+ }+ cluster.setInstanceCollectionType(cluster.getInstanceFleets().isEmpty()+ ? "INSTANCE_GROUP" : "INSTANCE_FLEET");+ cluster.setAutoTerminate(!cluster.isKeepJobFlowAliveWhenNoSteps());+ cluster.setMasterPublicDnsName(masterDnsName(region));+ for (EmrInstanceGroup g : cluster.getInstanceGroups()) {+ g.setId("ig-" + randomId(13));+ g.setRunningInstanceCount(g.getRequestedInstanceCount());+ g.setState("RUNNING");+ }+ for (EmrInstanceFleet f : cluster.getInstanceFleets()) {+ f.setId("if-" + randomId(13));+ f.setProvisionedOnDemandCapacity(f.getTargetOnDemandCapacity());+ f.setProvisionedSpotCapacity(f.getTargetSpotCapacity());+ f.setState("RUNNING");+ }+ cluster.getSteps().forEach(step -> step.setId("s-" + randomId(13)));+ lifecycle.start(cluster, now(), region);+ store(cluster);+ return cluster;+ });}/** The master node's private DNS name, derived from the region so a stored name never goes stale. */@@ -138,209 +228,180 @@ public class EmrService {}public EmrCluster describeCluster(String id) {- return requireCluster(id);+ return call(() -> snapshotOf(requireCluster(id)));}public List<EmrCluster> listClusters(String region, List<String> states,Instant createdAfter, Instant createdBefore) {- List<EmrCluster> result = new ArrayList<>();- for (EmrCluster c : clusterStore.scan(k -> true)) {- if (!region.equals(c.getRegion())) {- continue;- }- if (states != null && !states.isEmpty() && !states.contains(c.getState())) {- continue;- }- Instant createdAt = c.getCreationDateTime();- if (createdAt != null) {- createdAt = createdAt.truncatedTo(ChronoUnit.MILLIS);- }- if (createdAfter != null && (createdAt == null || !createdAt.isAfter(createdAfter))) {- continue;- }- if (createdBefore != null && (createdAt == null || !createdAt.isBefore(createdBefore))) {- continue;+ return call(() -> {+ List<EmrCluster> result = new ArrayList<>();+ for (EmrCluster c : clusterStore.scan(k -> true)) {+ if (!region.equals(c.getRegion())) {+ continue;+ }+ settle(c);+ if (states != null && !states.isEmpty() && !states.contains(c.getState())) {+ continue;+ }+ Instant createdAt = c.getCreationDateTime();+ if (createdAt != null) {+ createdAt = createdAt.truncatedTo(ChronoUnit.MILLIS);+ }+ if (createdAfter != null && (createdAt == null || !createdAt.isAfter(createdAfter))) {+ continue;+ }+ if (createdBefore != null && (createdAt == null || !createdAt.isBefore(createdBefore))) {+ continue;+ }+ result.add(snapshotOf(c));}- result.add(c);- }- return result;+ return result;+ });}public void terminateJobFlows(List<String> ids) {- try {- terminateJobFlowsLocked(ids);- } finally {- // A request naming a protected cluster still terminates the others before it fails.- eventPublisher.publishQueued();- }- }-- private synchronized void terminateJobFlowsLocked(List<String> ids) {- // AWS terminates the unprotected clusters in the request and then fails it with a- // ValidationException if any were termination protected.- boolean anyProtected = false;- for (String id : ids) {- EmrCluster cluster = findCluster(id).orElse(null);- if (cluster == null) {- continue;- }- if ("TERMINATED".equals(cluster.getState())) {- continue;+ // A request naming a protected cluster still terminates the others before it fails.+ run(() -> {+ // AWS terminates the unprotected clusters in the request and then fails it with a+ // ValidationException if any were termination protected.+ boolean anyProtected = false;+ for (String id : ids) {+ EmrCluster cluster = findCluster(id).orElse(null);+ if (cluster == null || isShutDown(cluster)) {+ continue;+ }+ if (cluster.isTerminationProtected()) {+ anyProtected = true;+ continue;+ }+ lifecycle.terminate(cluster, "USER_REQUEST", "Terminated by user request", now(), regionOf(cluster));+ store(cluster);}- if (cluster.isTerminationProtected()) {- anyProtected = true;- continue;+ if (anyProtected) {+ throw new AwsException("ValidationException",+ "Could not shut down one or more job flows since they are termination protected.", 400);}- cluster.setState("TERMINATED");- releaseInstanceCapacity(cluster);- cluster.setStateChangeReasonCode("USER_REQUEST");- cluster.setStateChangeMessage("Terminated by user request");- cluster.setEndDateTime(Instant.now());- clusterStore.put(id, cluster);- eventPublisher.queue(eventPublisher.clusterStateChange(cluster, "TERMINATED"),- regionResolver.getRegion());- }- if (anyProtected) {- throw new AwsException("ValidationException",- "Could not shut down one or more job flows since they are termination protected.", 400);- }+ });}- public synchronized void setTerminationProtection(List<String> ids, boolean value) {- mutateClusters(ids, c -> c.setTerminationProtected(value));+ public void setTerminationProtection(List<String> ids, boolean value) {+ run(() -> mutateClusters(ids, c -> c.setTerminationProtected(value)));}- public synchronized void setVisibleToAllUsers(List<String> ids, boolean value) {- mutateClusters(ids, c -> c.setVisibleToAllUsers(value));+ public void setVisibleToAllUsers(List<String> ids, boolean value) {+ run(() -> mutateClusters(ids, c -> c.setVisibleToAllUsers(value)));}- public synchronized void setKeepJobFlowAliveWhenNoSteps(List<String> ids, boolean value) {- mutateClusters(ids, c -> {+ public void setKeepJobFlowAliveWhenNoSteps(List<String> ids, boolean value) {+ run(() -> mutateClusters(ids, c -> {c.setKeepJobFlowAliveWhenNoSteps(value);c.setAutoTerminate(!value);- });+ }));}- public synchronized void setUnhealthyNodeReplacement(List<String> ids, boolean value) {- mutateClusters(ids, c -> c.setUnhealthyNodeReplacement(value));+ public void setUnhealthyNodeReplacement(List<String> ids, boolean value) {+ run(() -> mutateClusters(ids, c -> c.setUnhealthyNodeReplacement(value)));}- public synchronized int modifyCluster(String id, Integer stepConcurrencyLevel) {- EmrCluster cluster = requireCluster(id);- if (stepConcurrencyLevel != null) {- cluster.setStepConcurrencyLevel(stepConcurrencyLevel);- clusterStore.put(id, cluster);- }- return cluster.getStepConcurrencyLevel();+ public int modifyCluster(String id, Integer stepConcurrencyLevel) {+ return call(() -> {+ EmrCluster cluster = requireCluster(id);+ if (stepConcurrencyLevel != null) {+ cluster.setStepConcurrencyLevel(stepConcurrencyLevel);+ lifecycle.concurrencyChanged(cluster, now(), regionOf(cluster));+ store(cluster);+ }+ return cluster.getStepConcurrencyLevel();+ });}// ──────────────────────────── Steps ────────────────────────────public List<String> addJobFlowSteps(String clusterId, List<EmrStep> steps) {- try {- return addJobFlowStepsLocked(clusterId, steps);- } finally {- eventPublisher.publishQueued();- }- }-- private synchronized List<String> addJobFlowStepsLocked(String clusterId, List<EmrStep> steps) {- // AWS's own wording for an unknown job flow here, not DescribeCluster's: a botocore ClientError- // (raster-foundry/raster-foundry#352) and a CloudTrail record on AWS re:Post both show it.- EmrCluster cluster = clusterId == null ? requireCluster(null) : findCluster(clusterId).orElseThrow(- () -> new AwsException("ValidationException", "Specified job flow ID does not exist.", 400));- if ("TERMINATED".equals(cluster.getState())) {- throw new AwsException("ValidationException", JOB_FLOW_SHUT_DOWN, 400);- }- String region = regionResolver.getRegion();- runSteps(cluster, steps, region);- cluster.getSteps().addAll(steps);- List<String> ids = steps.stream().map(EmrStep::getId).toList();- if (!steps.isEmpty()) {- terminateAfterCompletedSteps(cluster);- }- clusterStore.put(clusterId, cluster);- if ("TERMINATED".equals(cluster.getState())) {- eventPublisher.queue(eventPublisher.clusterStateChange(cluster, "TERMINATED"), region);- }- return ids;+ return call(() -> {+ // AWS's own wording for an unknown job flow here, not DescribeCluster's: a botocore ClientError+ // (raster-foundry/raster-foundry#352) and a CloudTrail record on AWS re:Post both show it.+ EmrCluster cluster = clusterId == null ? requireCluster(null) : findCluster(clusterId).orElseThrow(+ () -> new AwsException("ValidationException", "Specified job flow ID does not exist.", 400));+ if (isShutDown(cluster)) {+ throw jobFlowShutDownValidation();+ }+ steps.forEach(step -> step.setId("s-" + randomId(13)));+ cluster.getSteps().addAll(steps);+ lifecycle.submit(cluster, steps, now(), regionOf(cluster));+ store(cluster);+ return steps.stream().map(EmrStep::getId).toList();+ });}public EmrStep describeStep(String clusterId, String stepId) {- EmrCluster cluster = requireCluster(clusterId);- return cluster.getSteps().stream()- .filter(s -> s.getId().equals(stepId))- .findFirst()- .orElseThrow(() -> new AwsException("InvalidRequestException",- "Step " + stepId + " does not exist.", 400));+ return call(() -> {+ EmrCluster cluster = snapshotOf(requireCluster(clusterId));+ return cluster.getSteps().stream()+ .filter(s -> s.getId().equals(stepId))+ .findFirst()+ .orElseThrow(() -> new AwsException("InvalidRequestException",+ "Step " + stepId + " does not exist.", 400));+ });}public List<EmrStep> listSteps(String clusterId, List<String> states, List<String> stepIds) {- EmrCluster cluster = requireCluster(clusterId);- List<EmrStep> result = new ArrayList<>();- // AWS returns steps in reverse submission order (newest first).- List<EmrStep> steps = cluster.getSteps();- for (int i = steps.size() - 1; i >= 0; i--) {- EmrStep step = steps.get(i);- if (states != null && !states.isEmpty() && !states.contains(step.getState())) {- continue;- }- if (stepIds != null && !stepIds.isEmpty() && !stepIds.contains(step.getId())) {- continue;+ return call(() -> {+ EmrCluster cluster = snapshotOf(requireCluster(clusterId));+ List<EmrStep> result = new ArrayList<>();+ // AWS returns steps in reverse submission order (newest first).+ List<EmrStep> steps = cluster.getSteps();+ for (int i = steps.size() - 1; i >= 0; i--) {+ EmrStep step = steps.get(i);+ if (states != null && !states.isEmpty() && !states.contains(step.getState())) {+ continue;+ }+ if (stepIds != null && !stepIds.isEmpty() && !stepIds.contains(step.getId())) {+ continue;+ }+ result.add(step);}- result.add(step);- }- return result;+ return result;+ });}public record CancelStepInfo(String stepId, String status, String reason) {}public List<CancelStepInfo> cancelSteps(String clusterId, List<String> stepIds) {- try {- return cancelStepsLocked(clusterId, stepIds);- } finally {- eventPublisher.publishQueued();- }- }-- private synchronized List<CancelStepInfo> cancelStepsLocked(String clusterId, List<String> stepIds) {- EmrCluster cluster = requireCluster(clusterId);- List<CancelStepInfo> result = new ArrayList<>();- for (String stepId : stepIds) {- EmrStep step = cluster.getSteps().stream()- .filter(s -> s.getId().equals(stepId)).findFirst().orElse(null);- if (step == null) {- result.add(new CancelStepInfo(stepId, "FAILED", "Step does not exist."));- } else if ("PENDING".equals(step.getState())) {- step.setState("CANCELLED");- step.setEndDateTime(Instant.now());- result.add(new CancelStepInfo(stepId, "SUBMITTED", null));- eventPublisher.queue(eventPublisher.stepStatusChange(cluster, step, "CANCELLED"),- regionResolver.getRegion());- } else {- result.add(new CancelStepInfo(stepId, "FAILED",- "Step is not in a cancellable state: " + step.getState()));+ return call(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ List<CancelStepInfo> result = new ArrayList<>();+ for (String stepId : stepIds) {+ EmrStep step = cluster.getSteps().stream()+ .filter(s -> s.getId().equals(stepId)).findFirst().orElse(null);+ String refusal = step == null ? "Step does not exist."+ : lifecycle.cancel(cluster, step, now(), regionOf(cluster));+ result.add(refusal == null+ ? new CancelStepInfo(stepId, "SUBMITTED", null)+ : new CancelStepInfo(stepId, "FAILED", refusal));}- }- clusterStore.put(clusterId, cluster);- return result;+ store(cluster);+ return result;+ });}// ──────────────────────────── Instance groups / fleets ────────────────────────────- public synchronized List<String> addInstanceGroups(String clusterId, List<EmrInstanceGroup> groups) {- EmrCluster cluster = requireCluster(clusterId);- rejectTerminatedCluster(cluster);- List<String> ids = new ArrayList<>();- for (EmrInstanceGroup group : groups) {- group.setId("ig-" + randomId(13));- group.setRunningInstanceCount(group.getRequestedInstanceCount());- group.setState("RUNNING");- cluster.getInstanceGroups().add(group);- ids.add(group.getId());- }- clusterStore.put(clusterId, cluster);- return ids;+ public List<String> addInstanceGroups(String clusterId, List<EmrInstanceGroup> groups) {+ return call(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ rejectShutDownCluster(cluster, EmrService::jobFlowShutDownValidation);+ List<String> ids = new ArrayList<>();+ for (EmrInstanceGroup group : groups) {+ group.setId("ig-" + randomId(13));+ group.setRunningInstanceCount(group.getRequestedInstanceCount());+ group.setState("RUNNING");+ cluster.getInstanceGroups().add(group);+ ids.add(group.getId());+ }+ clusterStore.put(clusterId, cluster);+ return ids;+ });}/** One InstanceGroupModifyConfig: a null count or configurations leaves that setting as it is. */@@ -351,110 +412,123 @@ public class EmrService {* without it each group is looked up across the account's clusters in the Region. Every modification is* checked before any is applied.*/- public synchronized void modifyInstanceGroups(String clusterId, String region,- List<InstanceGroupModification> modifications) {- if (modifications.isEmpty()) {- throw invalid("InstanceGroups must contain at least one instance group.");- }- Map<String, EmrCluster> touched = new LinkedHashMap<>();- List<EmrInstanceGroup> groups = new ArrayList<>();- for (InstanceGroupModification mod : modifications) {- EmrCluster found = clusterId != null ? requireCluster(clusterId) : clusterOwning(region, mod.instanceGroupId());- // AWS's answer here, recorded by Terraform users- // (hashicorp/terraform-provider-aws#1355 and #9400).- if ("TERMINATED".equals(found.getState())) {- throw new AwsException("ValidationException",- "An instance group may only be modified when the cluster is running or waiting.", 400);+ public void modifyInstanceGroups(String clusterId, String region, List<InstanceGroupModification> modifications) {+ run(() -> {+ if (modifications.isEmpty()) {+ throw invalid("InstanceGroups must contain at least one instance group.");}- // One copy per cluster, so two groups of the same cluster are changed and stored together.- EmrCluster cluster = touched.computeIfAbsent(found.getId(), id -> found);- EmrInstanceGroup group = requireInstanceGroup(cluster, mod.instanceGroupId());- if (mod.instanceCount() != null) {- if (mod.instanceCount() < 0) {- throw invalid("InstanceCount must not be negative.");+ Map<String, EmrCluster> touched = new LinkedHashMap<>();+ List<EmrInstanceGroup> groups = new ArrayList<>();+ for (InstanceGroupModification mod : modifications) {+ EmrCluster found = clusterId != null+ ? requireCluster(clusterId) : clusterOwning(region, mod.instanceGroupId());+ // AWS's answer here, recorded by Terraform users+ // (hashicorp/terraform-provider-aws#1355 and #9400).+ if (!runsOrWaits(found)) {+ throw new AwsException("ValidationException",+ "An instance group may only be modified when the cluster is running or waiting.", 400);}- if ("MASTER".equals(group.getInstanceGroupType())- && mod.instanceCount() != group.getRequestedInstanceCount()) {- throw invalid("The instance count of a master instance group cannot be modified.");+ // One copy per cluster, so two groups of the same cluster are changed and stored together.+ EmrCluster cluster = touched.computeIfAbsent(found.getId(), id -> found);+ EmrInstanceGroup group = requireInstanceGroup(cluster, mod.instanceGroupId());+ if (mod.instanceCount() != null) {+ if (mod.instanceCount() < 0) {+ throw invalid("InstanceCount must not be negative.");+ }+ if ("MASTER".equals(group.getInstanceGroupType())+ && mod.instanceCount() != group.getRequestedInstanceCount()) {+ throw invalid("The instance count of a master instance group cannot be modified.");+ }}+ groups.add(group);}- groups.add(group);- }- for (int i = 0; i < modifications.size(); i++) {- InstanceGroupModification mod = modifications.get(i);- EmrInstanceGroup group = groups.get(i);- if (mod.instanceCount() != null) {- group.setRequestedInstanceCount(mod.instanceCount());- group.setRunningInstanceCount(mod.instanceCount());- }- if (mod.configurations() != null) {- group.setConfigurations(mod.configurations());- group.setConfigurationsVersion(group.getConfigurationsVersion() + 1);+ for (int i = 0; i < modifications.size(); i++) {+ InstanceGroupModification mod = modifications.get(i);+ EmrInstanceGroup group = groups.get(i);+ if (mod.instanceCount() != null) {+ group.setRequestedInstanceCount(mod.instanceCount());+ group.setRunningInstanceCount(mod.instanceCount());+ }+ if (mod.configurations() != null) {+ group.setConfigurations(mod.configurations());+ group.setConfigurationsVersion(group.getConfigurationsVersion() + 1);+ }}- }- touched.forEach(clusterStore::put);+ touched.forEach(clusterStore::put);+ });}public List<EmrInstanceGroup> listInstanceGroups(String clusterId) {- return requireCluster(clusterId).getInstanceGroups();- }-- public synchronized String addInstanceFleet(String clusterId, EmrInstanceFleet fleet) {- EmrCluster cluster = requireCluster(clusterId);- rejectTerminatedCluster(cluster);- fleet.setId("if-" + randomId(13));- fleet.setProvisionedOnDemandCapacity(fleet.getTargetOnDemandCapacity());- fleet.setProvisionedSpotCapacity(fleet.getTargetSpotCapacity());- fleet.setState("RUNNING");- cluster.getInstanceFleets().add(fleet);- clusterStore.put(clusterId, cluster);- return fleet.getId();+ return call(() -> snapshotOf(requireCluster(clusterId)).getInstanceGroups());+ }++ public String addInstanceFleet(String clusterId, EmrInstanceFleet fleet) {+ return call(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ rejectShutDownCluster(cluster, EmrService::jobFlowShutDown);+ fleet.setId("if-" + randomId(13));+ fleet.setProvisionedOnDemandCapacity(fleet.getTargetOnDemandCapacity());+ fleet.setProvisionedSpotCapacity(fleet.getTargetSpotCapacity());+ fleet.setState("RUNNING");+ cluster.getInstanceFleets().add(fleet);+ clusterStore.put(clusterId, cluster);+ return fleet.getId();+ });}/** Resizes an instance fleet; a null target leaves that capacity as it is. */- public synchronized void modifyInstanceFleet(String clusterId, String instanceFleetId,- Integer targetOnDemandCapacity, Integer targetSpotCapacity) {- EmrCluster cluster = requireCluster(clusterId);- if (instanceFleetId == null) {- throw invalid("InstanceFleetId is required.");- }- // AWS's answer for a cluster terminated a while before, ErrorCode member included- // (hashicorp/terraform-provider-aws#31375, 2023).- if ("TERMINATED".equals(cluster.getState())) {- throw new AwsException("InvalidRequestException", JOB_FLOW_SHUT_DOWN, 400,- Map.of("ErrorCode", "INVALID_JOB_SHUT_DOWN"));- }- EmrInstanceFleet fleet = cluster.getInstanceFleets().stream()- .filter(f -> instanceFleetId.equals(f.getId()))- .findFirst()- .orElseThrow(() -> invalid("Instance fleet " + instanceFleetId + " is not in cluster " + clusterId + "."));- int onDemand = targetOnDemandCapacity != null ? targetOnDemandCapacity : fleet.getTargetOnDemandCapacity();- int spot = targetSpotCapacity != null ? targetSpotCapacity : fleet.getTargetSpotCapacity();- if (onDemand < 0 || spot < 0) {- throw invalid("Target capacities must not be negative.");- }- if ("MASTER".equals(fleet.getInstanceFleetType())- && (onDemand != fleet.getTargetOnDemandCapacity() || spot != fleet.getTargetSpotCapacity())) {- throw invalid("The target capacity of a master instance fleet cannot be modified.");- }- fleet.setTargetOnDemandCapacity(onDemand);- fleet.setTargetSpotCapacity(spot);- fleet.setProvisionedOnDemandCapacity(onDemand);- fleet.setProvisionedSpotCapacity(spot);- clusterStore.put(clusterId, cluster);+ public void modifyInstanceFleet(String clusterId, String instanceFleetId,+ Integer targetOnDemandCapacity, Integer targetSpotCapacity) {+ run(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ if (instanceFleetId == null) {+ throw invalid("InstanceFleetId is required.");+ }+ // AWS's answer for a cluster terminated a while before, ErrorCode member included+ // (hashicorp/terraform-provider-aws#31375, 2023); its wording covers one shutting down.+ if (isShutDown(cluster)) {+ throw jobFlowShutDown();+ }+ // And for one in any other state but running or waiting, a STARTING or BOOTSTRAPPING one+ // included, ErrorCode member too (hashicorp/terraform-provider-aws#15548, 2020).+ if (!runsOrWaits(cluster)) {+ throw new AwsException("InvalidRequestException",+ "An instance fleet may only be modified when the cluster is running or waiting.", 400,+ Map.of("ErrorCode", "MODIFY_INSTANCE_FLEET_OF_RUNNING_JOBFLOW_ONLY"));+ }+ EmrInstanceFleet fleet = cluster.getInstanceFleets().stream()+ .filter(f -> instanceFleetId.equals(f.getId()))+ .findFirst()+ .orElseThrow(() -> invalid("Instance fleet " + instanceFleetId + " is not in cluster "+ + clusterId + "."));+ int onDemand = targetOnDemandCapacity != null ? targetOnDemandCapacity : fleet.getTargetOnDemandCapacity();+ int spot = targetSpotCapacity != null ? targetSpotCapacity : fleet.getTargetSpotCapacity();+ if (onDemand < 0 || spot < 0) {+ throw invalid("Target capacities must not be negative.");+ }+ if ("MASTER".equals(fleet.getInstanceFleetType())+ && (onDemand != fleet.getTargetOnDemandCapacity() || spot != fleet.getTargetSpotCapacity())) {+ throw invalid("The target capacity of a master instance fleet cannot be modified.");+ }+ fleet.setTargetOnDemandCapacity(onDemand);+ fleet.setTargetSpotCapacity(spot);+ fleet.setProvisionedOnDemandCapacity(onDemand);+ fleet.setProvisionedSpotCapacity(spot);+ clusterStore.put(clusterId, cluster);+ });}/** The cluster's bootstrap actions as given to RunJobFlow, raw JSON; null when it had none. */public String listBootstrapActions(String clusterId) {- return requireCluster(clusterId).getBootstrapActions();+ return call(() -> requireCluster(clusterId).getBootstrapActions());}public List<EmrInstanceFleet> listInstanceFleets(String clusterId) {- return requireCluster(clusterId).getInstanceFleets();+ return call(() -> snapshotOf(requireCluster(clusterId)).getInstanceFleets());}public EmrCluster getClusterForInstances(String clusterId) {- return requireCluster(clusterId);+ return call(() -> snapshotOf(requireCluster(clusterId)));}// ──────────────────────────── Security configurations ────────────────────────────@@ -585,73 +659,90 @@ public class EmrService {// ──────────────────────────── Tags ────────────────────────────- public synchronized void addTags(String resourceId, Map<String, String> tags) {- EmrCluster cluster = requireCluster(resourceId);- cluster.getTags().putAll(tags);- clusterStore.put(resourceId, cluster);+ public void addTags(String resourceId, Map<String, String> tags) {+ run(() -> {+ EmrCluster cluster = requireCluster(resourceId);+ cluster.getTags().putAll(tags);+ clusterStore.put(resourceId, cluster);+ });}- public synchronized void removeTags(String resourceId, List<String> keys) {- EmrCluster cluster = requireCluster(resourceId);- keys.forEach(cluster.getTags()::remove);- clusterStore.put(resourceId, cluster);+ public void removeTags(String resourceId, List<String> keys) {+ run(() -> {+ EmrCluster cluster = requireCluster(resourceId);+ keys.forEach(cluster.getTags()::remove);+ clusterStore.put(resourceId, cluster);+ });}// ──────────────────────────── Helpers ────────────────────────────// ──────────────────────────── Scaling and termination policies ────────────────────────────- public synchronized void putManagedScalingPolicy(String clusterId, JsonNode policy) {- EmrCluster cluster = requireCluster(clusterId);- validateManagedScalingPolicy(policy);- cluster.setManagedScalingPolicy(policy.toString());- clusterStore.put(clusterId, cluster);+ public void putManagedScalingPolicy(String clusterId, JsonNode policy) {+ run(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ validateManagedScalingPolicy(policy);+ cluster.setManagedScalingPolicy(policy.toString());+ clusterStore.put(clusterId, cluster);+ });}/** The cluster's managed scaling policy as raw JSON, or null when it has none. */public String getManagedScalingPolicy(String clusterId) {- return requireCluster(clusterId).getManagedScalingPolicy();+ return call(() -> requireCluster(clusterId).getManagedScalingPolicy());}- public synchronized void removeManagedScalingPolicy(String clusterId) {- EmrCluster cluster = requireCluster(clusterId);- cluster.setManagedScalingPolicy(null);- clusterStore.put(clusterId, cluster);+ public void removeManagedScalingPolicy(String clusterId) {+ run(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ cluster.setManagedScalingPolicy(null);+ clusterStore.put(clusterId, cluster);+ });}/** AutoTerminationPolicy is optional in the request; an absent one is stored as an empty policy. */- public synchronized void putAutoTerminationPolicy(String clusterId, JsonNode policy) {- EmrCluster cluster = requireCluster(clusterId);- validateAutoTerminationPolicy(policy);- cluster.setAutoTerminationPolicy(policy == null || policy.isMissingNode() || policy.isNull() ? "{}" : policy.toString());- clusterStore.put(clusterId, cluster);+ public void putAutoTerminationPolicy(String clusterId, JsonNode policy) {+ run(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ validateAutoTerminationPolicy(policy);+ cluster.setAutoTerminationPolicy(+ policy == null || policy.isMissingNode() || policy.isNull() ? "{}" : policy.toString());+ clusterStore.put(clusterId, cluster);+ });}/** The cluster's auto termination policy as raw JSON, or null when it has none. */public String getAutoTerminationPolicy(String clusterId) {- return requireCluster(clusterId).getAutoTerminationPolicy();+ return call(() -> requireCluster(clusterId).getAutoTerminationPolicy());}- public synchronized void removeAutoTerminationPolicy(String clusterId) {- EmrCluster cluster = requireCluster(clusterId);- cluster.setAutoTerminationPolicy(null);- clusterStore.put(clusterId, cluster);+ public void removeAutoTerminationPolicy(String clusterId) {+ run(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ cluster.setAutoTerminationPolicy(null);+ clusterStore.put(clusterId, cluster);+ });}/** Attaches an automatic scaling policy to a core or task instance group and returns the group. */- public synchronized EmrInstanceGroup putAutoScalingPolicy(String clusterId, String instanceGroupId, JsonNode policy) {- EmrCluster cluster = requireCluster(clusterId);- EmrInstanceGroup group = requireInstanceGroup(cluster, instanceGroupId);- validateAutoScalingPolicy(policy, group.getInstanceGroupType());- group.setAutoScalingPolicy(policy.toString());- clusterStore.put(clusterId, cluster);- return group;+ public EmrInstanceGroup putAutoScalingPolicy(String clusterId, String instanceGroupId, JsonNode policy) {+ return call(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ EmrInstanceGroup group = requireInstanceGroup(cluster, instanceGroupId);+ validateAutoScalingPolicy(policy, group.getInstanceGroupType());+ group.setAutoScalingPolicy(policy.toString());+ clusterStore.put(clusterId, cluster);+ return group;+ });}- public synchronized void removeAutoScalingPolicy(String clusterId, String instanceGroupId) {- EmrCluster cluster = requireCluster(clusterId);- requireInstanceGroup(cluster, instanceGroupId).setAutoScalingPolicy(null);- clusterStore.put(clusterId, cluster);+ public void removeAutoScalingPolicy(String clusterId, String instanceGroupId) {+ run(() -> {+ EmrCluster cluster = requireCluster(clusterId);+ requireInstanceGroup(cluster, instanceGroupId).setAutoScalingPolicy(null);+ clusterStore.put(clusterId, cluster);+ });}// ──────────────────────────── Block public access ────────────────────────────@@ -811,6 +902,7 @@ public class EmrService {.filter(c -> region.equals(c.getRegion())).filter(c -> c.getInstanceGroups().stream().anyMatch(g -> instanceGroupId.equals(g.getId()))).findFirst()+ .map(this::settle).orElseThrow(() -> invalid("Instance group " + instanceGroupId + " is not valid."));}@@ -818,8 +910,30 @@ public class EmrService {return new AwsException("InvalidRequestException", message, 400);}- private static void rejectTerminatedCluster(EmrCluster cluster) {- if ("TERMINATED".equals(cluster.getState())) {+ /** ModifyInstanceFleet's answer for a job flow that is shutting down or ended, ErrorCode member included. */+ private static AwsException jobFlowShutDown() {+ return new AwsException("InvalidRequestException", JOB_FLOW_SHUT_DOWN, 400,+ Map.of("ErrorCode", "INVALID_JOB_SHUT_DOWN"));+ }++ /**+ * AddJobFlowSteps' answer for a job flow that is shutting down or ended. AddInstanceGroups gives it too:+ * like AddJobFlowSteps, its model declares no InvalidRequestException.+ */+ private static AwsException jobFlowShutDownValidation() {+ return new AwsException("ValidationException", JOB_FLOW_SHUT_DOWN, 400);+ }++ /**+ * AddInstanceGroups and AddInstanceFleet on a cluster that is shutting down or ended. A TERMINATING one+ * gets {@code shuttingDown}, the operation's answer with AWS's wording, which covers it; a TERMINATED+ * one keeps Floci's own, since no AWS answer to either operation is recorded.+ */+ private static void rejectShutDownCluster(EmrCluster cluster, Supplier<AwsException> shuttingDown) {+ if (EmrClusterLifecycle.TERMINATING.equals(cluster.getState())) {+ throw shuttingDown.get();+ }+ if (EmrClusterLifecycle.TERMINATED.equals(cluster.getState())) {throw invalid("Cluster " + cluster.getId() + " is terminated.");}}@@ -848,7 +962,8 @@ public class EmrService {*/private Optional<EmrCluster> findCluster(String id) {String region = regionResolver.getRegion();- return clusterStore.get(id).filter(c -> c.getRegion() == null || c.getRegion().equals(region));+ return clusterStore.get(id).filter(c -> c.getRegion() == null || c.getRegion().equals(region))+ .map(this::settle);}private void mutateClusters(List<String> ids, java.util.function.Consumer<EmrCluster> mutation) {@@ -860,68 +975,105 @@ public class EmrService {}}- private void advanceToWaiting(EmrCluster cluster) {- cluster.setState("WAITING");- cluster.setStateChangeReasonCode(cluster.getSteps().isEmpty() ? null : "ALL_STEPS_COMPLETED");- cluster.setStateChangeMessage("Cluster ready to run steps.");- cluster.setReadyDateTime(Instant.now());+ /** Shutting down or ended: "A job flow that is shutting down, terminated, or finished may not be modified." */+ private static boolean isShutDown(EmrCluster cluster) {+ return EmrClusterLifecycle.TERMINATING.equals(cluster.getState())+ || EmrClusterLifecycle.TERMINATED.equals(cluster.getState());+ }++ /** The only states in which a cluster's instance groups and fleets may be modified. */+ private static boolean runsOrWaits(EmrCluster cluster) {+ return EmrClusterLifecycle.RUNNING.equals(cluster.getState())+ || EmrClusterLifecycle.WAITING.equals(cluster.getState());}/**- * Runs a request's steps as the Management Guide describes ("Running steps to process data"): all- * are set PENDING, then each completes before the next starts. Here they complete at once. Each- * change queues its event as it is made, so the events follow the steps' recorded times.+ * Runs {@code body} holding the service, then publishes the events it queued once it no longer+ * does, so a rule's target that calls EMR never waits on the call that made the change. Every+ * call that reads a cluster runs this way, since reading one brings it up to date.*/- private void runSteps(EmrCluster cluster, List<EmrStep> steps, String region) {- for (EmrStep step : steps) {- initStep(step);- eventPublisher.queue(eventPublisher.stepStatusChange(cluster, step, "PENDING"), region);- }- for (EmrStep step : steps) {- completeStep(step);- eventPublisher.queue(eventPublisher.stepStatusChange(cluster, step, "COMPLETED"), region);+ private <T> T call(Supplier<T> body) {+ try {+ synchronized (this) {+ return body.get();+ }+ } finally {+ eventPublisher.publishQueued();}}- private void terminateAfterCompletedSteps(EmrCluster cluster) {- if (cluster.getSteps().isEmpty() || !cluster.isAutoTerminate()- || "TERMINATED".equals(cluster.getState())) {- return;- }- cluster.setState("TERMINATED");- releaseInstanceCapacity(cluster);- cluster.setStateChangeReasonCode("ALL_STEPS_COMPLETED");- cluster.setStateChangeMessage("All steps completed.");- cluster.setEndDateTime(Instant.now());+ private void run(Runnable body) {+ call(() -> {+ body.run();+ return null;+ });+ }++ /**+ * A copy of the cluster for a call to answer from: the timer goes on changing the stored one, and an+ * answer built from it after the service is let go could mix two of its states.+ */+ private static EmrCluster snapshotOf(EmrCluster cluster) {+ return SNAPSHOT_MAPPER.convertValue(cluster, EmrCluster.class);}- private void releaseInstanceCapacity(EmrCluster cluster) {- for (EmrInstanceGroup group : cluster.getInstanceGroups()) {- group.setState("TERMINATED");- group.setTerminatedInstanceCount(group.getRunningInstanceCount());- group.setRunningInstanceCount(0);+ /** The cluster as it stands now: every change due by now made, stored if there was one. */+ private EmrCluster settle(EmrCluster cluster) {+ if (lifecycle.advance(cluster, now(), regionOf(cluster))) {+ clusterStore.put(cluster.getId(), cluster);}- for (EmrInstanceFleet fleet : cluster.getInstanceFleets()) {- fleet.setState("TERMINATED");- fleet.setProvisionedOnDemandCapacity(0);- fleet.setProvisionedSpotCapacity(0);+ wakeForNextChange(cluster);+ return cluster;+ }++ /** Stores a cluster a call changed, once every change that falls due by now is made. */+ private void store(EmrCluster cluster) {+ lifecycle.advance(cluster, now(), regionOf(cluster));+ clusterStore.put(cluster.getId(), cluster);+ wakeForNextChange(cluster);+ }++ private void wakeForNextChange(EmrCluster cluster) {+ Instant due = lifecycle.nextChange(cluster);+ if (due != null) {+ wakeAt(regionResolver.getAccountId(), cluster, due);}}- private void initStep(EmrStep step) {- step.setId("s-" + randomId(13));- step.setState("PENDING");- step.setCreationDateTime(Instant.now());- if (step.getActionOnFailure() == null) {- step.setActionOnFailure("TERMINATE_CLUSTER");+ /**+ * Brings the cluster up to date at {@code due}, as {@code accountId} in its Region, unless an+ * earlier wakeup will: that one sets the next. Its events go out by {@link Wakeups#deliver}.+ */+ private void wakeAt(String accountId, EmrCluster cluster, Instant due) {+ String key = accountId + "/" + cluster.getId();+ Instant set = wakeupsDue.get(key);+ if (set != null && !set.isAfter(due)) {+ return;}+ wakeupsDue.put(key, due);+ String id = cluster.getId();+ String region = regionOf(cluster);+ wakeups.at(due, () -> {+ wakeupsDue.remove(key, due);+ try {+ RequestScopes.runAs(accountId, region, () -> {+ synchronized (this) {+ findCluster(id);+ }+ });+ } catch (RuntimeException e) {+ LOG.warnv(e, "EMR could not bring cluster {0} up to date", id);+ }+ wakeups.deliver(eventPublisher::publishQueued);+ });+ }++ private String regionOf(EmrCluster cluster) {+ return cluster.getRegion() != null ? cluster.getRegion() : regionResolver.getRegion();}- private void completeStep(EmrStep step) {- Instant now = Instant.now();- step.setStartDateTime(now);- step.setEndDateTime(now);- step.setState("COMPLETED");+ private Instant now() {+ return clock.instant().truncatedTo(ChronoUnit.MILLIS);}private String randomId(int length) {diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java b/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.javaindex a44f58de9c..c7c34fee6a 100644--- a/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java+++ b/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java@@ -10,7 +10,8 @@ import java.util.Map;/*** An EMR cluster (job flow). State machine:- * STARTING → BOOTSTRAPPING → RUNNING → WAITING, then → TERMINATING → TERMINATED.+ * STARTING → BOOTSTRAPPING → RUNNING → WAITING (and back to RUNNING for added steps), then+ * → TERMINATING → TERMINATED.* Steps, instance groups and fleets are stored as nested lists. Complex sub-structures* (ec2 attributes, applications, configurations) are stored as raw JSON strings.*/@@ -53,6 +54,8 @@ public class EmrCluster {private Instant creationDateTime;private Instant readyDateTime;private Instant endDateTime;+ // When the cluster began TERMINATING; Floci's own, as the API's Timeline has no such time.+ private Instant terminatingDateTime;private Map<String, String> tags = new LinkedHashMap<>();private List<EmrStep> steps = new ArrayList<>();private List<EmrInstanceGroup> instanceGroups = new ArrayList<>();@@ -180,6 +183,9 @@ public class EmrCluster {public Instant getEndDateTime() { return endDateTime; }public void setEndDateTime(Instant endDateTime) { this.endDateTime = endDateTime; }+ public Instant getTerminatingDateTime() { return terminatingDateTime; }+ public void setTerminatingDateTime(Instant terminatingDateTime) { this.terminatingDateTime = terminatingDateTime; }+public Map<String, String> getTags() { return tags; }public void setTags(Map<String, String> tags) { this.tags = tags; }diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex 492685bc89..ddc18f666a 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -578,6 +578,9 @@ floci:enabled: trueemr:enabled: true+ cluster-startup-delay-seconds: 0 # 0 = a cluster is ready as soon as it is created+ step-duration-seconds: 0 # 0 = a step completes as soon as it starts+ cluster-termination-delay-seconds: 0 # 0 = a cluster is TERMINATED as soon as it is TERMINATINGwafv2:enabled: truescheduler:diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventPublisherTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventPublisherTest.javaindex 6e0f2c4e48..4244ba9793 100644--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventPublisherTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventPublisherTest.java@@ -1,15 +1,9 @@package io.github.hectorvent.floci.services.emr;import com.fasterxml.jackson.databind.JsonNode;-import com.fasterxml.jackson.databind.ObjectMapper;-import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.AwsException;-import io.github.hectorvent.floci.core.common.RegionResolver;import io.github.hectorvent.floci.services.emr.model.EmrCluster;import io.github.hectorvent.floci.services.emr.model.EmrStep;-import io.github.hectorvent.floci.services.eventbridge.EventBridgeService;-import io.github.hectorvent.floci.testing.InMemoryStorageFactory;-import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;import java.time.Instant;@@ -19,69 +13,37 @@ import java.util.ArrayList;import java.util.List;import java.util.concurrent.atomic.AtomicReference;+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.REGION;+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.cluster;+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.step;import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertFalse;import static org.junit.jupiter.api.Assertions.assertNull;import static org.junit.jupiter.api.Assertions.assertThrows;-import static org.mockito.ArgumentMatchers.anyString;-import static org.mockito.ArgumentMatchers.eq;-import static org.mockito.ArgumentMatchers.isNull;-import static org.mockito.Mockito.RETURNS_DEEP_STUBS;-import static org.mockito.Mockito.doAnswer;-import static org.mockito.Mockito.mock;-import static org.mockito.Mockito.when;/*** The {@code aws.emr} events {@link EmrService} publishes, in the order it publishes them, each one- * checked against the Management Guide's detail and its tables' severity and message.+ * checked against the Management Guide's detail and its tables' severity and message. The durations+ * are 0, as by default; {@link EmrServiceTest} covers the timed lifecycle.*/class EmrEventPublisherTest {- private static final String REGION = "us-east-1";- private static final String ACCOUNT = "000000000000";private static final DateTimeFormatter MINUTE =DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm 'UTC'").withZone(ZoneOffset.UTC);- private final ObjectMapper objectMapper = new ObjectMapper();- private final List<JsonNode> published = new ArrayList<>();- private Target target = event -> { };- private EmrEventPublisher publisher;- private EmrService service;-- /** A rule's target, given each event as it is published, as EventBridge's first delivery is. */- @FunctionalInterface- private interface Target {- void deliver(JsonNode event) throws Exception;- }-- @BeforeEach- void setUp() {- EventBridgeService eventBridge = mock(EventBridgeService.class);- doAnswer(invocation -> {- // The call that made the change has let go of the service by now.- assertFalse(Thread.holdsLock(service), "published while EmrService was held");- JsonNode event = objectMapper.createObjectNode()- .put("detail-type", invocation.<String>getArgument(1))- .set("detail", objectMapper.readTree(invocation.<String>getArgument(2)));- published.add(event);- target.deliver(event);- return null;- }).when(eventBridge).publishServiceEvent(eq("aws.emr"), anyString(), anyString(), isNull(), eq(REGION));- EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);- when(config.services().emr().defaultReleaseLabel()).thenReturn("emr-7.5.0");- publisher = new EmrEventPublisher(eventBridge, objectMapper, new RegionResolver(REGION, ACCOUNT));- service = new EmrService(new InMemoryStorageFactory(ACCOUNT), config, new RegionResolver(REGION, ACCOUNT),- publisher);- }+ private final EmrServiceFixture fixture = new EmrServiceFixture();+ private final EmrService service = fixture.service;+ private final EmrEventPublisher publisher = fixture.publisher;+ private final List<JsonNode> published = fixture.published;@Test- void aClusterThatRunsItsStepsAndWaitsIsStartedAddsAndCompletesThemAndWaits() {+ void aClusterThatRunsItsStepsAndWaitsIsStartedAddsRunsAndCompletesThemAndWaits() {EmrCluster cluster = service.runJobFlow(cluster("nightly", true, step("load"), step("report")), REGION);EmrStep load = cluster.getSteps().get(0);EmrStep report = cluster.getSteps().get(1);- assertEquals(List.of("STARTING", "PENDING load", "PENDING report", "COMPLETED load", "COMPLETED report",- "WAITING"), states());+ assertEquals(List.of("STARTING", "PENDING load", "PENDING report", "RUNNING load", "COMPLETED load",+ "RUNNING report", "COMPLETED report", "WAITING"), fixture.states());// As the events tell it: every step was added before the first one ran.assertFalse(report.getCreationDateTime().isAfter(load.getStartDateTime()));assertCluster(published.get(0), cluster, "STARTING", "INFO", "{\"code\":\"\"}",@@ -90,22 +52,25 @@ class EmrEventPublisherTest {assertStep(published.get(1), cluster, load, "PENDING", "INFO","Step " + load.getId() + " (load) was added to Amazon EMR cluster " + cluster.getId()+ " (nightly) at " + minute(load.getCreationDateTime()) + " and is pending execution.");- assertStep(published.get(3), cluster, load, "COMPLETED", "INFO",+ assertStep(published.get(3), cluster, load, "RUNNING", "INFO",+ "Step " + load.getId() + " (load) in Amazon EMR cluster " + cluster.getId()+ + " (nightly) started running at " + minute(load.getStartDateTime()) + ".");+ assertStep(published.get(4), cluster, load, "COMPLETED", "INFO","Step " + load.getId() + " (load) in Amazon EMR cluster " + cluster.getId()+ " (nightly) completed execution at " + minute(load.getEndDateTime())+ ". The step started running at " + minute(load.getStartDateTime())+ " and took 0 minutes to complete.");- assertEquals(report.getId(), published.get(4).at("/detail/stepId").asText());- assertCluster(published.get(5), cluster, "WAITING", "INFO", "{\"code\":\"\"}",+ assertEquals(report.getId(), published.get(6).at("/detail/stepId").asText());+ assertCluster(published.get(7), cluster, "WAITING", "INFO", "{\"code\":\"\"}","Amazon EMR cluster " + cluster.getId() + " (nightly) finished running all pending steps at "- + minute(cluster.getReadyDateTime()) + ".");+ + minute(report.getEndDateTime()) + ".");}@Testvoid aClusterWithoutStepsIsReadyForUse() {EmrCluster cluster = service.runJobFlow(cluster("idle", true), REGION);- assertEquals(List.of("STARTING", "WAITING"), states());+ assertEquals(List.of("STARTING", "WAITING"), fixture.states());assertEquals("Amazon EMR cluster " + cluster.getId() + " (idle) was created at "+ minute(cluster.getReadyDateTime()) + " and is ready for use.",published.get(1).at("/detail/message").asText());@@ -115,21 +80,25 @@ class EmrEventPublisherTest {void aTransientClusterTerminatesAfterItsStepsWithoutWaiting() {EmrCluster cluster = service.runJobFlow(cluster("transient", false, step("only")), REGION);- assertEquals(List.of("STARTING", "PENDING only", "COMPLETED only", "TERMINATED"), states());- assertCluster(published.get(3), cluster, "TERMINATED", "INFO",+ assertEquals(List.of("STARTING", "PENDING only", "RUNNING only", "COMPLETED only", "TERMINATED"),+ fixture.states());+ assertCluster(published.get(4), cluster, "TERMINATED", "INFO","{\"code\":\"ALL_STEPS_COMPLETED\",\"message\":\"All steps completed.\"}","Amazon EMR Cluster " + cluster.getId() + " (transient) has terminated at "+ minute(cluster.getEndDateTime()) + " with a reason of ALL_STEPS_COMPLETED.");}@Test- void stepsAddedToAWaitingClusterLeaveItWaiting() {+ void stepsAddedToAWaitingClusterRunAndLeaveItWaitingAgain() {EmrCluster cluster = service.runJobFlow(cluster("kept", true), REGION);published.clear();- service.addJobFlowSteps(cluster.getId(), List.of(step("late")));+ EmrStep late = step("late");+ service.addJobFlowSteps(cluster.getId(), List.of(late));- assertEquals(List.of("PENDING late", "COMPLETED late"), states());+ assertEquals(List.of("PENDING late", "RUNNING late", "COMPLETED late", "WAITING"), fixture.states());+ assertEquals("Amazon EMR cluster " + cluster.getId() + " (kept) finished running all pending steps at "+ + minute(late.getEndDateTime()) + ".", published.get(3).at("/detail/message").asText());}@Test@@ -139,9 +108,9 @@ class EmrEventPublisherTest {service.addJobFlowSteps(cluster.getId(), List.of(step("late")));- assertEquals(List.of("PENDING late", "COMPLETED late", "TERMINATED"), states());+ assertEquals(List.of("PENDING late", "RUNNING late", "COMPLETED late", "TERMINATED"), fixture.states());assertEquals("{\"code\":\"ALL_STEPS_COMPLETED\",\"message\":\"All steps completed.\"}",- published.get(2).at("/detail/stateChangeReason").asText());+ published.get(3).at("/detail/stateChangeReason").asText());}@Test@@ -157,7 +126,7 @@ class EmrEventPublisherTest {List<String> ids = List.of(ended.getId(), gone.getId(), kept.getId());assertThrows(AwsException.class, () -> service.terminateJobFlows(ids));- assertEquals(List.of("TERMINATED"), states());+ assertEquals(List.of("TERMINATED"), fixture.states());assertCluster(published.get(0), service.describeCluster(ended.getId()), "TERMINATED", "INFO","{\"code\":\"USER_REQUEST\",\"message\":\"Terminated by user request\"}","Amazon EMR Cluster " + ended.getId() + " (ended) has terminated at "@@ -174,12 +143,12 @@ class EmrEventPublisherTest {assertThrows(AwsException.class, () -> service.addJobFlowSteps(cluster.getId(), List.of(step("late"))));assertThrows(AwsException.class, () -> service.runJobFlow(new EmrCluster(), REGION));- assertEquals(List.of(), states());+ assertEquals(List.of(), fixture.states());}@Testvoid aTargetThatEndsTheClusterOnItsStartingEventHearsOfItAfterTheRestOfTheCall() {- target = event -> {+ fixture.target = event -> {if ("STARTING".equals(event.at("/detail/state").asText())) {service.terminateJobFlows(List.of(event.at("/detail/clusterId").asText()));}@@ -187,13 +156,13 @@ class EmrEventPublisherTest {service.runJobFlow(cluster("answered", true), REGION);- assertEquals(List.of("STARTING", "WAITING", "TERMINATED"), states());+ assertEquals(List.of("STARTING", "WAITING", "TERMINATED"), fixture.states());}@Testvoid aTargetCallingBackFromAnotherThreadHearsOfItAfterTheRestOfTheCall() {AtomicReference<Throwable> failure = new AtomicReference<>();- target = event -> {+ fixture.target = event -> {if ("STARTING".equals(event.at("/detail/state").asText())) {// As an API destination would, from the thread serving its request, while this one waits.String clusterId = event.at("/detail/clusterId").asText();@@ -212,7 +181,7 @@ class EmrEventPublisherTest {service.runJobFlow(cluster("answered", true), REGION);assertNull(failure.get());- assertEquals(List.of("STARTING", "WAITING", "TERMINATED"), states());+ assertEquals(List.of("STARTING", "WAITING", "TERMINATED"), fixture.states());}@Test@@ -220,9 +189,9 @@ class EmrEventPublisherTest {EmrCluster cluster = service.runJobFlow(cluster("failing", true), REGION);cluster.setStateChangeReasonCode("STEP_FAILURE");cluster.setStateChangeMessage("Shut down as step failed");- cluster.setEndDateTime(Instant.parse("2026-10-24T09:41:30Z"));- JsonNode detail = objectMapper.readTree(publisher.clusterStateChange(cluster, "TERMINATED").detail());+ JsonNode detail = fixture.objectMapper.readTree(+ publisher.clusterStateChange(cluster, "TERMINATED", Instant.parse("2026-10-24T09:41:30Z")).detail());assertEquals("CRITICAL", detail.path("severity").asText());assertEquals("Amazon EMR Cluster " + cluster.getId() + " (failing) has terminated at 2026-10-24 09:41 UTC"@@ -238,7 +207,7 @@ class EmrEventPublisherTest {EmrEventPublisher.Event event = publisher.stepStatusChange(cluster, step, "CANCELLED");assertEquals("EMR Step Status Change", event.detailType());- JsonNode detail = objectMapper.readTree(event.detail());+ JsonNode detail = fixture.objectMapper.readTree(event.detail());assertEquals("WARN", detail.path("severity").asText());assertEquals("Cancellation request has succeeded for cluster step " + step.getId() + " (doomed) in Amazon EMR"+ " cluster " + cluster.getId() + " (cancelling) at 2026-10-24 09:41 UTC, and the step is now"@@ -274,16 +243,6 @@ class EmrEventPublisherTest {assertEquals(message, detail.path("message").asText());}- /** Each event's state, a step's followed by its name. */- private List<String> states() {- return published.stream().map(event -> {- JsonNode detail = event.path("detail");- return detail.has("stepId")- ? detail.path("state").asText() + " " + detail.path("name").asText()- : detail.path("state").asText();- }).toList();- }-private static List<String> fieldNames(JsonNode node) {List<String> names = new ArrayList<>();node.fieldNames().forEachRemaining(names::add);@@ -293,19 +252,4 @@ class EmrEventPublisherTest {private static String minute(Instant instant) {return MINUTE.format(instant);}-- private static EmrCluster cluster(String name, boolean keepAlive, EmrStep... steps) {- EmrCluster cluster = new EmrCluster();- cluster.setName(name);- cluster.setKeepJobFlowAliveWhenNoSteps(keepAlive);- cluster.setSteps(new ArrayList<>(List.of(steps)));- return cluster;- }-- private static EmrStep step(String name) {- EmrStep step = new EmrStep();- step.setName(name);- step.setActionOnFailure("CONTINUE");- return step;- }}diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventsIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventsIntegrationTest.javaindex 21db3427e6..0a141736d3 100644--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventsIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventsIntegrationTest.java@@ -13,9 +13,6 @@ import java.time.Instant;import java.time.ZoneOffset;import java.time.format.DateTimeFormatter;import java.util.List;-import java.util.Map;-import java.util.function.Function;-import java.util.stream.Collectors;import static io.github.hectorvent.floci.testing.EmrRequests.emr;import static org.hamcrest.MatcherAssert.assertThat;@@ -52,24 +49,32 @@ class EmrEventsIntegrationTest {+ "\"Steps\":[{\"Name\":\"load\",\"ActionOnFailure\":\"CONTINUE\","+ "\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}").then().statusCode(200).extract().path("JobFlowId");- emr(region, "AddJobFlowSteps", "{\"JobFlowId\":\"" + clusterId + "\",\"Steps\":[{\"Name\":\"report\","- + "\"ActionOnFailure\":\"CONTINUE\",\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")- .then().statusCode(200);+ String reportId = emr(region, "AddJobFlowSteps", "{\"JobFlowId\":\"" + clusterId + "\","+ + "\"Steps\":[{\"Name\":\"report\",\"ActionOnFailure\":\"CONTINUE\","+ + "\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")+ .then().statusCode(200).extract().path("StepIds[0]");+ Instant reportEnded = stepEnd(region, clusterId, reportId);emr(region, "TerminateJobFlows", "{\"JobFlowIds\":[\"" + clusterId + "\"]}").then().statusCode(200);- Map<String, JsonNode> events = byState(queue.drain(), region, clusterId);- assertThat(events.keySet(), containsInAnyOrder("STARTING", "PENDING load", "COMPLETED load", "WAITING",- "PENDING report", "COMPLETED report", "TERMINATED"));- JsonNode waiting = events.get("WAITING").path("detail");- assertEquals("{\"code\":\"\"}", waiting.path("stateChangeReason").asText());- assertEquals("Amazon EMR cluster " + clusterId + " (kept) finished running all pending steps at "- + minute(timeline(region, clusterId, "ReadyDateTime")) + ".", waiting.path("message").asText());- JsonNode completed = events.get("COMPLETED report").path("detail");+ List<JsonNode> events = checked(queue.drain(), region, clusterId);+ assertThat(events.stream().map(EmrEventsIntegrationTest::state).toList(), containsInAnyOrder(+ "STARTING", "PENDING load", "RUNNING load", "COMPLETED load", "WAITING",+ "PENDING report", "RUNNING report", "COMPLETED report", "WAITING", "TERMINATED"));+ // Once after the first step, and again after the one added later.+ List<JsonNode> waiting = withState(events, "WAITING");+ for (JsonNode event : waiting) {+ assertEquals("{\"code\":\"\"}", event.path("detail").path("stateChangeReason").asText());+ }+ String finished = "Amazon EMR cluster " + clusterId + " (kept) finished running all pending steps at ";+ assertThat(waiting.stream().map(event -> event.path("detail").path("message").asText()).toList(),+ containsInAnyOrder(finished + minute(timeline(region, clusterId, "ReadyDateTime")) + ".",+ finished + minute(reportEnded) + "."));+ JsonNode completed = withState(events, "COMPLETED report").get(0).path("detail");assertEquals("INFO", completed.path("severity").asText());assertEquals("CONTINUE", completed.path("actionOnFailure").asText());assertTrue(completed.path("message").asText().endsWith(" and took 0 minutes to complete."),completed.path("message").asText());- JsonNode terminated = events.get("TERMINATED").path("detail");+ JsonNode terminated = withState(events, "TERMINATED").get(0).path("detail");assertEquals("INFO", terminated.path("severity").asText());assertEquals("{\"code\":\"USER_REQUEST\",\"message\":\"Terminated by user request\"}",terminated.path("stateChangeReason").asText());@@ -87,24 +92,23 @@ class EmrEventsIntegrationTest {+ "\"Steps\":[{\"Name\":\"only\",\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}").then().statusCode(200).extract().path("JobFlowId");- Map<String, JsonNode> events = byState(queue.drain(), region, clusterId);- assertThat(events.keySet(),- containsInAnyOrder("STARTING", "PENDING only", "COMPLETED only", "TERMINATED"));- JsonNode terminated = events.get("TERMINATED").path("detail");+ List<JsonNode> events = checked(queue.drain(), region, clusterId);+ assertThat(events.stream().map(EmrEventsIntegrationTest::state).toList(), containsInAnyOrder(+ "STARTING", "PENDING only", "RUNNING only", "COMPLETED only", "TERMINATED"));+ JsonNode terminated = withState(events, "TERMINATED").get(0).path("detail");assertEquals("INFO", terminated.path("severity").asText());assertEquals("{\"code\":\"ALL_STEPS_COMPLETED\",\"message\":\"All steps completed.\"}",terminated.path("stateChangeReason").asText());assertEquals("TERMINATE_CLUSTER",- events.get("PENDING only").path("detail").path("actionOnFailure").asText());+ withState(events, "PENDING only").get(0).path("detail").path("actionOnFailure").asText());}}/**- * The events by state, a step's followed by its name, once each is checked to be one of- * {@code clusterId}'s, sent by EMR to {@code region}'s default bus with no resources. A state seen- * twice fails, so the keys are every event.+ * The events, once each is checked to be one of {@code clusterId}'s, sent by EMR to {@code region}'s+ * default bus with no resources.*/- private static Map<String, JsonNode> byState(List<JsonNode> events, String region, String clusterId) {+ private static List<JsonNode> checked(List<JsonNode> events, String region, String clusterId) {for (JsonNode event : events) {assertEquals("aws.emr", event.path("source").asText());assertEquals(ACCOUNT, event.path("account").asText());@@ -114,12 +118,19 @@ class EmrEventsIntegrationTest {assertEquals(event.path("detail").has("stepId") ? "EMR Step Status Change" : "EMR Cluster State Change",event.path("detail-type").asText());}- return events.stream().collect(Collectors.toMap(event -> {- JsonNode detail = event.path("detail");- return detail.has("stepId")- ? detail.path("state").asText() + " " + detail.path("name").asText()- : detail.path("state").asText();- }, Function.identity()));+ return events;+ }++ /** The event's state, a step's followed by its name. */+ private static String state(JsonNode event) {+ JsonNode detail = event.path("detail");+ return detail.has("stepId")+ ? detail.path("state").asText() + " " + detail.path("name").asText()+ : detail.path("state").asText();+ }++ private static List<JsonNode> withState(List<JsonNode> events, String state) {+ return events.stream().filter(event -> state.equals(state(event))).toList();}private static Instant timeline(String region, String clusterId, String field) {@@ -129,6 +140,13 @@ class EmrEventsIntegrationTest {return Instant.ofEpochMilli(seconds.movePointRight(3).longValue());}+ private static Instant stepEnd(String region, String clusterId, String stepId) {+ BigDecimal seconds = emr(region, "DescribeStep", "{\"ClusterId\":\"" + clusterId + "\",\"StepId\":\""+ + stepId + "\"}").then().statusCode(200).extract().response().jsonPath(DECIMAL_NUMBERS)+ .get("Step.Status.Timeline.EndDateTime");+ return Instant.ofEpochMilli(seconds.movePointRight(3).longValue());+ }+private static String minute(Instant instant) {return MINUTE.format(instant);}diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.javaindex f8380c9cd5..066f0c9e60 100644--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java@@ -127,22 +127,23 @@ class EmrIntegrationTest {.then().statusCode(200).body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());- EmrCluster cluster = emrService.describeCluster(clusterId);+ EmrCluster cluster = storedCluster("floci-emr-pinned");+ String pinnedId = cluster.getId();Instant originalCreationTime = cluster.getCreationDateTime();cluster.setCreationDateTime(Instant.ofEpochMilli(1_761_000_000_000L));try {emr("ListClusters", "{\"CreatedBefore\":1761000000.001}").then().statusCode(200)- .body("Clusters.find { it.Id == '" + clusterId + "' }.Name", equalTo("floci-emr"));+ .body("Clusters.find { it.Id == '" + pinnedId + "' }.Name", equalTo("floci-emr-pinned"));emr("ListClusters", "{\"CreatedAfter\":1760999999.999}").then().statusCode(200)- .body("Clusters.find { it.Id == '" + clusterId + "' }.Name", equalTo("floci-emr"));+ .body("Clusters.find { it.Id == '" + pinnedId + "' }.Name", equalTo("floci-emr-pinned"));emr("ListClusters", "{\"CreatedBefore\":1761000000.000000001}").then().statusCode(200)- .body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());+ .body("Clusters.find { it.Id == '" + pinnedId + "' }", nullValue());emr("ListClusters", "{\"CreatedAfter\":1760999999.999999999}").then().statusCode(200)- .body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());+ .body("Clusters.find { it.Id == '" + pinnedId + "' }", nullValue());} finally {cluster.setCreationDateTime(originalCreationTime);}@@ -153,7 +154,7 @@ class EmrIntegrationTest {.then().statusCode(200).extract().asString()).path("Clusters");JsonNode listedTime = null;for (JsonNode listed : clusters) {- if (clusterId.equals(listed.path("Id").asText())) {+ if (pinnedId.equals(listed.path("Id").asText())) {listedTime = listed.path("Status").path("Timeline").path("CreationDateTime");break;}@@ -163,7 +164,7 @@ class EmrIntegrationTest {assertEquals(0, new BigDecimal("1761000000.123").compareTo(listedTime.decimalValue()));JsonNode describedTime = objectMapper.readTree(emr("DescribeCluster",- "{\"ClusterId\":\"" + clusterId + "\"}")+ "{\"ClusterId\":\"" + pinnedId + "\"}").then().statusCode(200).extract().asString()).path("Cluster").path("Status").path("Timeline").path("CreationDateTime");assertTrue(describedTime.isNumber());@@ -171,10 +172,10 @@ class EmrIntegrationTest {emr("ListClusters", "{\"CreatedAfter\":" + listedTime.asText() + "}").then().statusCode(200)- .body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());+ .body("Clusters.find { it.Id == '" + pinnedId + "' }", nullValue());emr("ListClusters", "{\"CreatedBefore\":" + listedTime.asText() + "}").then().statusCode(200)- .body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());+ .body("Clusters.find { it.Id == '" + pinnedId + "' }", nullValue());} finally {cluster.setCreationDateTime(originalCreationTime);}@@ -441,14 +442,9 @@ class EmrIntegrationTest {// Catches: a cluster stored before the DNS fix keeps reporting its old us-east-1 master name.@Testvoid storedClusterReportsTheRegionalMasterDnsName() {- String id = emr("RunJobFlow",- "{\"Name\":\"stale-dns\",\"ReleaseLabel\":\"emr-7.5.0\","- + "\"Instances\":{\"KeepJobFlowAliveWhenNoSteps\":true,"- + "\"InstanceGroups\":[{\"Name\":\"master\",\"InstanceRole\":\"MASTER\","- + "\"InstanceType\":\"m5.xlarge\",\"InstanceCount\":1}]}}")- .then().statusCode(200)- .extract().path("JobFlowId");- emrService.describeCluster(id).setMasterPublicDnsName("ip-10-0-0-1.us-east-1.compute.internal");+ EmrCluster stored = storedCluster("stale-dns");+ stored.setMasterPublicDnsName("ip-10-0-0-1.us-east-1.compute.internal");+ String id = stored.getId();emr("DescribeCluster", "{\"ClusterId\":\"" + id + "\"}").then().statusCode(200)@@ -457,4 +453,15 @@ class EmrIntegrationTest {emr("TerminateJobFlows", "{\"JobFlowIds\":[\"" + id + "\"]}").then().statusCode(200);}++ /**+ * A kept-alive cluster as RunJobFlow stores it: the service's own object, which a test can change,+ * unlike the copy DescribeCluster answers from.+ */+ private EmrCluster storedCluster(String name) {+ EmrCluster cluster = new EmrCluster();+ cluster.setName(name);+ cluster.setKeepJobFlowAliveWhenNoSteps(true);+ return emrService.runJobFlow(cluster, "us-east-1");+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.javanew file mode 100644index 0000000000..a4d790fc8f--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.java@@ -0,0 +1,163 @@+package io.github.hectorvent.floci.services.emr;++import com.fasterxml.jackson.databind.JsonNode;+import com.fasterxml.jackson.databind.ObjectMapper;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.services.emr.model.EmrCluster;+import io.github.hectorvent.floci.services.emr.model.EmrStep;+import io.github.hectorvent.floci.services.eventbridge.EventBridgeService;+import io.github.hectorvent.floci.testing.InMemoryStorageFactory;+import io.github.hectorvent.floci.testing.MutableClock;++import java.time.Duration;+import java.time.Instant;+import java.util.ArrayList;+import java.util.List;++import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.ArgumentMatchers.isNull;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.doAnswer;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++/**+ * An {@link EmrService} on in-memory storage, with a clock the test moves and wakeups the test runs,+ * recording the {@code aws.emr} events it publishes. Each event is handed to {@link #target} as it is+ * published, as EventBridge's first delivery is, and checked to be published once the service is let go,+ * and, for a wakeup's, after it, as the timer's own thread never sends them.+ */+final class EmrServiceFixture {++ static final String REGION = "us-east-1";+ static final String ACCOUNT = "000000000000";++ /** A rule's target, given each event as it is published. */+ @FunctionalInterface+ interface Target {+ void deliver(JsonNode event) throws Exception;+ }++ private record Wakeup(Instant due, Runnable task) {}++ final ObjectMapper objectMapper = new ObjectMapper();+ final List<JsonNode> published = new ArrayList<>();+ final MutableClock clock;+ final EmrEventPublisher publisher;+ // Not final: the events' check reads it, from the mock set up before the service is built.+ EmrService service;+ Target target = event -> { };+ private final List<Wakeup> wakeups = new ArrayList<>();+ private final List<Runnable> deliveries = new ArrayList<>();+ private boolean waking;+ private final InMemoryStorageFactory storage;+ private final int[] durations;++ /** The service as configured by default, every duration 0. */+ EmrServiceFixture() {+ this(0, 0, 0);+ }++ EmrServiceFixture(int startupDelaySeconds, int stepDurationSeconds, int terminationDelaySeconds) {+ this(new InMemoryStorageFactory(ACCOUNT), new MutableClock(),+ startupDelaySeconds, stepDurationSeconds, terminationDelaySeconds);+ }++ private EmrServiceFixture(InMemoryStorageFactory storage, MutableClock clock, int... durations) {+ this.storage = storage;+ this.clock = clock;+ this.durations = durations;+ EventBridgeService eventBridge = mock(EventBridgeService.class);+ doAnswer(invocation -> {+ // The call that made the change has let go of the service by now.+ assertFalse(Thread.holdsLock(service), "published while EmrService was held");+ assertFalse(waking, "published by a wakeup itself");+ JsonNode event = objectMapper.createObjectNode()+ .put("detail-type", invocation.<String>getArgument(1))+ .set("detail", objectMapper.readTree(invocation.<String>getArgument(2)));+ published.add(event);+ target.deliver(event);+ return null;+ }).when(eventBridge).publishServiceEvent(eq("aws.emr"), anyString(), anyString(), isNull(), eq(REGION));+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.services().emr().defaultReleaseLabel()).thenReturn("emr-7.5.0");+ when(config.services().emr().clusterStartupDelaySeconds()).thenReturn(durations[0]);+ when(config.services().emr().stepDurationSeconds()).thenReturn(durations[1]);+ when(config.services().emr().clusterTerminationDelaySeconds()).thenReturn(durations[2]);+ RegionResolver regionResolver = new RegionResolver(REGION, ACCOUNT);+ publisher = new EmrEventPublisher(eventBridge, objectMapper, regionResolver);+ service = new EmrService(storage, config, regionResolver, publisher, clock, new EmrService.Wakeups() {+ @Override+ public void at(Instant due, Runnable task) {+ wakeups.add(new Wakeup(due, task));+ }++ @Override+ public void deliver(Runnable delivery) {+ deliveries.add(delivery);+ }+ });+ }++ /** A new service on the same storage and clock, as after a restart of Floci with persistent storage. */+ EmrServiceFixture restart() {+ return new EmrServiceFixture(storage, clock, durations);+ }++ /**+ * Moves the clock on by {@code duration} and runs the wakeups due by then, as the service's timer+ * would, the ones they set included, each followed by the delivery of its events.+ */+ void pass(Duration duration) {+ clock.advance(duration);+ Instant now = clock.instant();+ List<Wakeup> due;+ while (!(due = wakeups.stream().filter(wakeup -> !wakeup.due().isAfter(now)).toList()).isEmpty()) {+ wakeups.removeAll(due);+ for (Wakeup wakeup : due) {+ waking = true;+ try {+ wakeup.task().run();+ } finally {+ waking = false;+ }+ while (!deliveries.isEmpty()) {+ deliveries.remove(0).run();+ }+ }+ }+ }++ /** The times of the wakeups not yet run. */+ List<Instant> wakeupTimes() {+ return wakeups.stream().map(Wakeup::due).toList();+ }++ /** Each event's state, a step's followed by its name. */+ List<String> states() {+ return published.stream().map(event -> {+ JsonNode detail = event.path("detail");+ return detail.has("stepId")+ ? detail.path("state").asText() + " " + detail.path("name").asText()+ : detail.path("state").asText();+ }).toList();+ }++ static EmrCluster cluster(String name, boolean keepAlive, EmrStep... steps) {+ EmrCluster cluster = new EmrCluster();+ cluster.setName(name);+ cluster.setKeepJobFlowAliveWhenNoSteps(keepAlive);+ cluster.setSteps(new ArrayList<>(List.of(steps)));+ return cluster;+ }++ static EmrStep step(String name) {+ EmrStep step = new EmrStep();+ step.setName(name);+ step.setActionOnFailure("CONTINUE");+ return step;+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceTest.javanew file mode 100644index 0000000000..ab0a07fb61--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceTest.java@@ -0,0 +1,463 @@+package io.github.hectorvent.floci.services.emr;++import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.services.emr.model.EmrCluster;+import io.github.hectorvent.floci.services.emr.model.EmrInstanceFleet;+import io.github.hectorvent.floci.services.emr.model.EmrInstanceGroup;+import io.github.hectorvent.floci.services.emr.model.EmrStep;+import io.github.hectorvent.floci.testing.MutableClock;+import org.junit.jupiter.api.Test;++import java.time.Duration;+import java.time.Instant;+import java.util.List;+import java.util.Map;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.TimeUnit;++import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.REGION;+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.cluster;+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.step;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertNull;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;++/**+ * The timed lifecycle: the durations {@code cluster-startup-delay-seconds}, {@code step-duration-seconds}+ * and {@code cluster-termination-delay-seconds} hold a cluster and its steps in each state of the+ * Management Guide's lifecycle, a call finds them as they stand when it is made, and the service sends+ * each change's event when it falls due, without anyone asking.+ */+class EmrServiceTest {++ private static final String SHUT_DOWN =+ "A job flow that is shutting down, terminated, or finished may not be modified.";++ @Test+ void aClusterIsStartingThenBootstrappingUntilItIsReady() {+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 0);+ EmrService service = fixture.service;+ EmrCluster cluster = service.runJobFlow(cluster("slow", true), REGION);+ Instant created = cluster.getCreationDateTime();++ assertEquals("STARTING", service.describeCluster(cluster.getId()).getState());+ fixture.clock.advance(Duration.ofSeconds(5));+ assertEquals("BOOTSTRAPPING", service.describeCluster(cluster.getId()).getState());+ fixture.clock.advance(Duration.ofSeconds(5));++ EmrCluster ready = service.describeCluster(cluster.getId());+ assertEquals("WAITING", ready.getState());+ assertEquals(created.plusSeconds(10), ready.getReadyDateTime());+ assertEquals(List.of("STARTING", "WAITING"), fixture.states());+ assertTrue(fixture.published.get(1).at("/detail/message").asText().endsWith(" was created at "+ + "2026-01-01 00:00 UTC and is ready for use."));+ }++ @Test+ void aClusterNobodyReadsSendsEachEventWhenItFallsDue() {+ EmrServiceFixture fixture = new EmrServiceFixture(10, 60, 30);+ EmrCluster cluster = fixture.service.runJobFlow(cluster("unread", false, step("only")), REGION);+ Instant created = cluster.getCreationDateTime();++ assertEquals(List.of("STARTING", "PENDING only"), fixture.states());+ assertEquals(List.of(created.plusSeconds(5)), fixture.wakeupTimes());+ fixture.pass(Duration.ofSeconds(10));+ assertEquals(List.of("STARTING", "PENDING only", "RUNNING only"), fixture.states());+ assertEquals(List.of(created.plusSeconds(70)), fixture.wakeupTimes());+ fixture.pass(Duration.ofSeconds(60));+ assertEquals(List.of("STARTING", "PENDING only", "RUNNING only", "COMPLETED only"), fixture.states());+ fixture.pass(Duration.ofSeconds(30));++ assertEquals(List.of("STARTING", "PENDING only", "RUNNING only", "COMPLETED only", "TERMINATED"),+ fixture.states());+ assertEquals(List.of(), fixture.wakeupTimes());+ EmrCluster ended = fixture.service.describeCluster(cluster.getId());+ assertEquals("TERMINATED", ended.getState());+ assertEquals("ALL_STEPS_COMPLETED", ended.getStateChangeReasonCode());+ assertEquals(created.plusSeconds(100), ended.getEndDateTime());+ }++ @Test+ void stepsRunOneAtATimeInTheOrderTheyWereAdded() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);+ EmrService service = fixture.service;+ EmrCluster cluster = service.runJobFlow(cluster("serial", true, step("a"), step("b")), REGION);+ String id = cluster.getId();+ EmrStep a = cluster.getSteps().get(0);+ EmrStep b = cluster.getSteps().get(1);++ assertEquals("RUNNING", service.describeCluster(id).getState());+ assertEquals(List.of("PENDING", "RUNNING"), states(service.listSteps(id, null, null)));+ assertEquals(List.of(b.getId()), ids(service.listSteps(id, List.of("PENDING"), null)));+ fixture.clock.advance(Duration.ofSeconds(60));+ assertEquals(List.of("RUNNING", "COMPLETED"), states(service.listSteps(id, null, null)));+ assertEquals(a.getStartDateTime().plusSeconds(60), a.getEndDateTime());+ assertEquals(a.getEndDateTime(), b.getStartDateTime());+ fixture.clock.advance(Duration.ofSeconds(60));++ assertEquals("WAITING", service.describeCluster(id).getState());+ assertEquals(List.of("STARTING", "PENDING a", "PENDING b", "RUNNING a", "COMPLETED a", "RUNNING b",+ "COMPLETED b", "WAITING"), fixture.states());+ assertTrue(fixture.published.get(4).at("/detail/message").asText()+ .endsWith(" and took 1 minutes to complete."));+ }++ @Test+ void aClusterRunsAsManyStepsAtOnceAsItsConcurrencyLevelAllows() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);+ EmrService service = fixture.service;+ EmrCluster parallel = cluster("parallel", true, step("a"), step("b"), step("c"));+ parallel.setStepConcurrencyLevel(2);+ String id = service.runJobFlow(parallel, REGION).getId();++ assertEquals(List.of("PENDING", "RUNNING", "RUNNING"), states(service.listSteps(id, null, null)));+ fixture.clock.advance(Duration.ofSeconds(10));+ service.modifyCluster(id, 3);+ assertEquals(List.of("RUNNING", "RUNNING", "RUNNING"), states(service.listSteps(id, null, null)));+ // Lowering the level lets the steps already running complete.+ service.modifyCluster(id, 1);+ fixture.clock.advance(Duration.ofSeconds(60));++ assertEquals(List.of("COMPLETED", "COMPLETED", "COMPLETED"), states(service.listSteps(id, null, null)));+ assertEquals("WAITING", service.describeCluster(id).getState());+ }++ @Test+ void stepsAddedWhileAClusterStartsRunOnceItIsReady() {+ EmrServiceFixture fixture = new EmrServiceFixture(10, 60, 0);+ EmrService service = fixture.service;+ EmrCluster cluster = service.runJobFlow(cluster("early", true), REGION);+ EmrStep early = step("early");++ service.addJobFlowSteps(cluster.getId(), List.of(early));+ assertEquals("PENDING", early.getState());+ fixture.clock.advance(Duration.ofSeconds(10));++ assertEquals("RUNNING", service.describeStep(cluster.getId(), early.getId()).getState());+ assertEquals(cluster.getCreationDateTime().plusSeconds(10), early.getStartDateTime());+ assertEquals("RUNNING", service.describeCluster(cluster.getId()).getState());+ }++ @Test+ void aWaitingClusterIsRunningWhileItRunsAddedStepsAndWaitsAgainAfter() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);+ EmrService service = fixture.service;+ String id = service.runJobFlow(cluster("kept", true), REGION).getId();+ assertEquals("Cluster ready to run steps.", service.describeCluster(id).getStateChangeMessage());++ service.addJobFlowSteps(id, List.of(step("late")));+ EmrCluster running = service.describeCluster(id);+ assertEquals("RUNNING", running.getState());+ assertNull(running.getStateChangeReasonCode());+ assertNull(running.getStateChangeMessage());+ fixture.clock.advance(Duration.ofSeconds(60));++ EmrCluster waiting = service.describeCluster(id);+ assertEquals("WAITING", waiting.getState());+ assertEquals("ALL_STEPS_COMPLETED", waiting.getStateChangeReasonCode());+ assertEquals(List.of("STARTING", "WAITING", "PENDING late", "RUNNING late", "COMPLETED late", "WAITING"),+ fixture.states());+ }++ @Test+ void addingNoStepsLeavesAWaitingClusterAsItIs() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);+ EmrService service = fixture.service;+ String fresh = service.runJobFlow(cluster("fresh", true), REGION).getId();+ String done = service.runJobFlow(cluster("done", true, step("only")), REGION).getId();+ fixture.clock.advance(Duration.ofSeconds(60));+ // Set to end after its steps: a cluster that already waits ends only after the next ones it runs.+ service.setKeepJobFlowAliveWhenNoSteps(List.of(done), false);+ fixture.published.clear();++ assertEquals(List.of(), service.addJobFlowSteps(fresh, List.of()));+ assertEquals(List.of(), service.addJobFlowSteps(done, List.of()));++ assertEquals("Cluster ready to run steps.", service.describeCluster(fresh).getStateChangeMessage());+ EmrCluster waiting = service.describeCluster(done);+ assertEquals("WAITING", waiting.getState());+ assertEquals("ALL_STEPS_COMPLETED", waiting.getStateChangeReasonCode());+ assertEquals(List.of(), fixture.states());+ }++ @Test+ void cancellingAPendingStepLeavesTheRunningOneRunning() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);+ EmrService service = fixture.service;+ EmrCluster cluster = service.runJobFlow(cluster("cancel", true, step("a"), step("b")), REGION);+ EmrStep b = cluster.getSteps().get(1);++ List<EmrService.CancelStepInfo> infos = service.cancelSteps(cluster.getId(), List.of(b.getId()));++ assertEquals(List.of(new EmrService.CancelStepInfo(b.getId(), "SUBMITTED", null)), infos);+ assertEquals(List.of("CANCELLED", "RUNNING"), states(service.listSteps(cluster.getId(), null, null)));+ assertEquals("CANCELLED b", fixture.states().get(fixture.states().size() - 1));+ }++ @Test+ void cancellingARunningStepLetsTheNextOneRun() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);+ EmrService service = fixture.service;+ EmrCluster cluster = service.runJobFlow(cluster("cancel", true, step("a"), step("b")), REGION);+ EmrStep a = cluster.getSteps().get(0);+ fixture.clock.advance(Duration.ofSeconds(10));++ List<EmrService.CancelStepInfo> infos = service.cancelSteps(cluster.getId(), List.of(a.getId()));++ assertEquals(List.of(new EmrService.CancelStepInfo(a.getId(), "SUBMITTED", null)), infos);+ assertEquals(List.of("RUNNING", "CANCELLED"), states(service.listSteps(cluster.getId(), null, null)));+ assertEquals(a.getEndDateTime(), cluster.getSteps().get(1).getStartDateTime());+ assertEquals(List.of("CANCELLED a", "RUNNING b"),+ fixture.states().subList(fixture.states().size() - 2, fixture.states().size()));+ }++ @Test+ void aReleaseBefore528CancelsOnlyPendingSteps() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);+ EmrService service = fixture.service;+ EmrCluster old = cluster("old", true, step("a"), step("b"));+ old.setReleaseLabel("emr-5.27.0");+ EmrCluster cluster = service.runJobFlow(old, REGION);+ String a = cluster.getSteps().get(0).getId();+ String b = cluster.getSteps().get(1).getId();++ List<EmrService.CancelStepInfo> infos = service.cancelSteps(cluster.getId(), List.of(a, b));++ assertEquals(List.of(+ new EmrService.CancelStepInfo(a, "FAILED", "Step is not in a cancellable state: RUNNING"),+ new EmrService.CancelStepInfo(b, "SUBMITTED", null)), infos);+ assertTrue(EmrClusterLifecycle.cancelsRunningSteps("emr-5.28.0"));+ assertTrue(EmrClusterLifecycle.cancelsRunningSteps("emr-7.5.0"));+ assertFalse(EmrClusterLifecycle.cancelsRunningSteps("emr-4.9.6"));+ }++ @Test+ void terminatingCancelsTheStepsNotYetCompletedAndEndsAfterTheDelay() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 30);+ EmrService service = fixture.service;+ EmrCluster kept = cluster("ending", true, step("a"), step("b"));+ kept.getInstanceGroups().add(group("CORE", 2));+ EmrCluster cluster = service.runJobFlow(kept, REGION);+ String id = cluster.getId();+ fixture.clock.advance(Duration.ofSeconds(10));+ fixture.published.clear();++ service.terminateJobFlows(List.of(id));++ EmrCluster terminating = service.describeCluster(id);+ assertEquals("TERMINATING", terminating.getState());+ assertEquals("USER_REQUEST", terminating.getStateChangeReasonCode());+ assertNull(terminating.getEndDateTime());+ assertEquals(List.of("CANCELLED", "CANCELLED"), states(service.listSteps(id, null, null)));+ assertEquals("RUNNING", service.listInstanceGroups(id).get(0).getState());+ AwsException refused = assertThrows(AwsException.class,+ () -> service.addJobFlowSteps(id, List.of(step("late"))));+ assertEquals(SHUT_DOWN, refused.getMessage());+ service.terminateJobFlows(List.of(id));+ assertEquals(List.of("CANCELLED a", "CANCELLED b"), fixture.states());+ fixture.clock.advance(Duration.ofSeconds(30));++ EmrCluster ended = service.describeCluster(id);+ assertEquals("TERMINATED", ended.getState());+ assertEquals(terminating.getTerminatingDateTime().plusSeconds(30), ended.getEndDateTime());+ assertEquals("TERMINATED", service.listInstanceGroups(id).get(0).getState());+ assertEquals(0, service.listInstanceGroups(id).get(0).getRunningInstanceCount());+ assertEquals(List.of("CANCELLED a", "CANCELLED b", "TERMINATED"), fixture.states());+ }++ @Test+ void instanceGroupsChangeOnlyWhileTheClusterRunsOrWaits() {+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 30);+ EmrService service = fixture.service;+ EmrCluster withGroup = cluster("groups", true);+ withGroup.getInstanceGroups().add(group("CORE", 2));+ EmrCluster cluster = service.runJobFlow(withGroup, REGION);+ String groupId = cluster.getInstanceGroups().get(0).getId();+ List<EmrService.InstanceGroupModification> resize =+ List.of(new EmrService.InstanceGroupModification(groupId, 3, null));++ AwsException starting = assertThrows(AwsException.class,+ () -> service.modifyInstanceGroups(cluster.getId(), REGION, resize));+ assertEquals("An instance group may only be modified when the cluster is running or waiting.",+ starting.getMessage());+ fixture.clock.advance(Duration.ofSeconds(10));+ service.modifyInstanceGroups(null, REGION, resize);+ assertEquals(3, service.listInstanceGroups(cluster.getId()).get(0).getRequestedInstanceCount());+ service.terminateJobFlows(List.of(cluster.getId()));++ assertThrows(AwsException.class, () -> service.modifyInstanceGroups(cluster.getId(), REGION, resize));+ AwsException adding = assertThrows(AwsException.class,+ () -> service.addInstanceGroups(cluster.getId(), List.of(group("TASK", 1))));+ assertEquals("ValidationException", adding.getErrorCode());+ assertEquals(SHUT_DOWN, adding.getMessage());+ assertEquals(1, service.listInstanceGroups(cluster.getId()).size());+ }++ @Test+ void anInstanceFleetIsResizedOnlyOnceTheClusterRunsOrWaits() {+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 0);+ EmrService service = fixture.service;+ EmrCluster withFleet = cluster("starting", true);+ EmrInstanceFleet fleet = fleet("CORE", 2);+ withFleet.getInstanceFleets().add(fleet);+ String id = service.runJobFlow(withFleet, REGION).getId();++ AwsException starting = assertThrows(AwsException.class,+ () -> service.modifyInstanceFleet(id, fleet.getId(), 3, null));+ fixture.clock.advance(Duration.ofSeconds(5));+ AwsException bootstrapping = assertThrows(AwsException.class,+ () -> service.modifyInstanceFleet(id, fleet.getId(), 3, null));+ fixture.clock.advance(Duration.ofSeconds(5));+ service.modifyInstanceFleet(id, fleet.getId(), 3, null);++ assertNotRunningOrWaiting(starting);+ assertNotRunningOrWaiting(bootstrapping);+ assertEquals(3, service.listInstanceFleets(id).get(0).getTargetOnDemandCapacity());+ }++ @Test+ void anInstanceFleetIsNotAddedOrResizedWhileTheClusterShutsDown() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 0, 30);+ EmrService service = fixture.service;+ EmrCluster withFleet = cluster("fleets", true);+ EmrInstanceFleet fleet = fleet("CORE", 2);+ withFleet.getInstanceFleets().add(fleet);+ EmrCluster cluster = service.runJobFlow(withFleet, REGION);+ service.terminateJobFlows(List.of(cluster.getId()));++ AwsException resizing = assertThrows(AwsException.class,+ () -> service.modifyInstanceFleet(cluster.getId(), fleet.getId(), 3, null));+ AwsException adding = assertThrows(AwsException.class,+ () -> service.addInstanceFleet(cluster.getId(), new EmrInstanceFleet()));++ assertShutDown(resizing);+ assertShutDown(adding);+ assertEquals(1, service.listInstanceFleets(cluster.getId()).size());+ }++ @Test+ void listingClustersByStateFindsThemAsTheyStandNow() {+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 0);+ EmrService service = fixture.service;+ String id = service.runJobFlow(cluster("listed", true), REGION).getId();++ assertEquals(List.of(), service.listClusters(REGION, List.of("WAITING"), null, null));+ fixture.clock.advance(Duration.ofSeconds(10));++ assertEquals(List.of(id),+ service.listClusters(REGION, List.of("WAITING"), null, null).stream().map(EmrCluster::getId).toList());+ }++ @Test+ void aClusterLeftMidChangeByARestartMovesOnWhenItFallsDue() {+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 0);+ EmrCluster cluster = fixture.service.runJobFlow(cluster("restarted", true), REGION);++ EmrServiceFixture restarted = fixture.restart();++ assertEquals(List.of(cluster.getCreationDateTime().plusSeconds(5)), restarted.wakeupTimes());+ restarted.pass(Duration.ofSeconds(10));+ assertEquals(List.of("WAITING"), restarted.states());+ }++ @Test+ void whatACallReturnsStaysAsItWasWhenTheClusterMovesOn() {+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 30);+ EmrService service = fixture.service;+ EmrCluster read = cluster("read", true, step("only"));+ read.getInstanceGroups().add(group("CORE", 2));+ read.getInstanceFleets().add(fleet("TASK", 2));+ EmrCluster created = service.runJobFlow(read, REGION);+ String id = created.getId();+ String stepId = created.getSteps().get(0).getId();++ List<EmrCluster> clusters = List.of(service.describeCluster(id),+ service.listClusters(REGION, null, null, null).get(0), service.getClusterForInstances(id));+ List<EmrStep> steps = List.of(service.describeStep(id, stepId), service.listSteps(id, null, null).get(0));+ EmrInstanceGroup group = service.listInstanceGroups(id).get(0);+ EmrInstanceFleet fleet = service.listInstanceFleets(id).get(0);+ fixture.pass(Duration.ofSeconds(60));+ service.terminateJobFlows(List.of(id));+ fixture.pass(Duration.ofSeconds(30));++ assertEquals("TERMINATED", service.describeCluster(id).getState());+ assertEquals(List.of("RUNNING", "RUNNING", "RUNNING"), clusters.stream().map(EmrCluster::getState).toList());+ assertEquals(List.of("RUNNING", "RUNNING"), states(steps));+ assertNull(steps.get(0).getEndDateTime());+ assertEquals(2, group.getRunningInstanceCount());+ assertEquals(2, fleet.getProvisionedOnDemandCapacity());+ }++ @Test+ void aSlowDeliveryDoesNotHoldUpTheTimer() throws InterruptedException {+ EmrService.Wakeups timer = EmrService.Wakeups.timer(new MutableClock());+ CountDownLatch woken = new CountDownLatch(1);+ CountDownLatch delivered = new CountDownLatch(1);+ try {+ // Like a rule's target that answers only once the next cluster has changed.+ timer.deliver(() -> {+ try {+ if (woken.await(10, TimeUnit.SECONDS)) {+ delivered.countDown();+ }+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ }+ });+ timer.at(Instant.EPOCH, woken::countDown);++ assertTrue(delivered.await(10, TimeUnit.SECONDS));+ } finally {+ timer.close();+ }+ }++ @Test+ void aNegativeDurationIsRefused() {+ IllegalArgumentException refused = assertThrows(IllegalArgumentException.class,+ () -> new EmrServiceFixture(0, -1, 0));++ assertEquals("floci.services.emr.step-duration-seconds must not be negative: -1", refused.getMessage());+ }++ /** ModifyInstanceFleet's answer for a job flow that is shutting down, which AddInstanceFleet shares. */+ private static void assertShutDown(AwsException refused) {+ assertEquals("InvalidRequestException", refused.getErrorCode());+ assertEquals(SHUT_DOWN, refused.getMessage());+ assertEquals(Map.of("ErrorCode", "INVALID_JOB_SHUT_DOWN"), refused.getExtendedData());+ }++ /** ModifyInstanceFleet's answer for a cluster that is starting or bootstrapping. */+ private static void assertNotRunningOrWaiting(AwsException refused) {+ assertEquals("InvalidRequestException", refused.getErrorCode());+ assertEquals("An instance fleet may only be modified when the cluster is running or waiting.",+ refused.getMessage());+ assertEquals(Map.of("ErrorCode", "MODIFY_INSTANCE_FLEET_OF_RUNNING_JOBFLOW_ONLY"), refused.getExtendedData());+ }++ private static EmrInstanceGroup group(String type, int count) {+ EmrInstanceGroup group = new EmrInstanceGroup();+ group.setInstanceGroupType(type);+ group.setInstanceType("m5.xlarge");+ group.setRequestedInstanceCount(count);+ return group;+ }++ private static EmrInstanceFleet fleet(String type, int onDemandCapacity) {+ EmrInstanceFleet fleet = new EmrInstanceFleet();+ fleet.setInstanceFleetType(type);+ fleet.setTargetOnDemandCapacity(onDemandCapacity);+ return fleet;+ }++ /** ListSteps' states, newest step first, as it lists them. */+ private static List<String> states(List<EmrStep> steps) {+ return steps.stream().map(EmrStep::getState).toList();+ }++ private static List<String> ids(List<EmrStep> steps) {+ return steps.stream().map(EmrStep::getId).toList();+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrTimedLifecycleIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrTimedLifecycleIntegrationTest.javanew file mode 100644index 0000000000..4934972af4--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrTimedLifecycleIntegrationTest.java@@ -0,0 +1,77 @@+package io.github.hectorvent.floci.services.emr;++import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;+import io.quarkus.test.junit.QuarkusTest;+import io.quarkus.test.junit.QuarkusTestProfile;+import io.quarkus.test.junit.TestProfile;+import org.junit.jupiter.api.BeforeAll;+import org.junit.jupiter.api.Test;++import java.util.List;+import java.util.Map;++import static io.github.hectorvent.floci.testing.EmrRequests.emr;+import static org.hamcrest.Matchers.equalTo;+import static org.hamcrest.Matchers.hasItem;+import static org.hamcrest.Matchers.nullValue;++/**+ * With the lifecycle durations set, a cluster is seen STARTING and TERMINATING, and its steps PENDING,+ * over the wire. They are an hour long here, so nothing moves on while the test runs.+ */+@QuarkusTest+@TestProfile(EmrTimedLifecycleIntegrationTest.HourLong.class)+class EmrTimedLifecycleIntegrationTest {++ public static class HourLong implements QuarkusTestProfile {+ @Override+ public Map<String, String> getConfigOverrides() {+ return Map.of(+ "floci.services.emr.cluster-startup-delay-seconds", "3600",+ "floci.services.emr.step-duration-seconds", "3600",+ "floci.services.emr.cluster-termination-delay-seconds", "3600");+ }+ }++ @BeforeAll+ static void configure() {+ RestAssuredJsonUtils.configureAwsContentTypes();+ }++ @Test+ void aStartingClusterKeepsItsStepsPendingAndShutsDownWhenTerminated() {+ String clusterId = emr("RunJobFlow", "{\"Name\":\"hour-long\","+ + "\"Instances\":{\"KeepJobFlowAliveWhenNoSteps\":true},"+ + "\"Steps\":[{\"Name\":\"first\",\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}},"+ + "{\"Name\":\"second\",\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")+ .then().statusCode(200).extract().path("JobFlowId");+ String firstId = emr("ListSteps", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(200)+ .body("Steps.Status.State", equalTo(List.of("PENDING", "PENDING")))+ .extract().path("Steps[1].Id");++ emr("DescribeCluster", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(200)+ .body("Cluster.Status.State", equalTo("STARTING"))+ .body("Cluster.Status.Timeline.ReadyDateTime", nullValue());+ emr("ListClusters", "{\"ClusterStates\":[\"STARTING\"]}").then().statusCode(200)+ .body("Clusters.Id", hasItem(clusterId));+ emr("CancelSteps", "{\"ClusterId\":\"" + clusterId + "\",\"StepIds\":[\"" + firstId + "\"]}")+ .then().statusCode(200)+ .body("CancelStepsInfoList[0].Status", equalTo("SUBMITTED"));+ emr("DescribeStep", "{\"ClusterId\":\"" + clusterId + "\",\"StepId\":\"" + firstId + "\"}")+ .then().statusCode(200)+ .body("Step.Status.State", equalTo("CANCELLED"));+ emr("TerminateJobFlows", "{\"JobFlowIds\":[\"" + clusterId + "\"]}").then().statusCode(200);++ emr("DescribeCluster", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(200)+ .body("Cluster.Status.State", equalTo("TERMINATING"))+ .body("Cluster.Status.StateChangeReason.Code", equalTo("USER_REQUEST"));+ emr("ListSteps", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(200)+ .body("Steps.Status.State", equalTo(List.of("CANCELLED", "CANCELLED")));+ emr("AddJobFlowSteps", "{\"JobFlowId\":\"" + clusterId + "\",\"Steps\":[{\"Name\":\"late\","+ + "\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")+ .then().statusCode(400)+ .body("__type", equalTo("ValidationException"))+ .body("message", equalTo(+ "A job flow that is shutting down, terminated, or finished may not be modified."));+ }+}