Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(emr): add an optional timed lifecycle for clusters and steps (#6441)

Priamo Germosen55b060b891eb
diff --git a/docs/services/emr.md b/docs/services/emr.md
index 8790784736..fb4f93a16d 100644
--- a/docs/services/emr.md
+++ b/docs/services/emr.md
@@ -55,25 +55,45 @@ Floci emulates the Amazon EMR (Elastic MapReduce) management API. Clusters (job
## Cluster lifecycle
-Steps complete immediately in this metadata-only emulator, in the order the Management Guide gives:
-a request's steps are all created `PENDING`, then each completes before the next starts. A cluster
-created with `Instances.KeepJobFlowAliveWhenNoSteps` set to `false` terminates after its initial steps or
-steps added later complete, reporting `ALL_STEPS_COMPLETED`. A cluster with no steps remains
-`WAITING` until steps are added. Set `KeepJobFlowAliveWhenNoSteps` to `true` to keep the cluster
-`WAITING` after its steps complete. Automatic step-completion termination also applies when
-termination protection is enabled.
-`AddJobFlowSteps` rejects a terminated cluster without adding steps or changing its
+Nothing runs in this metadata-only emulator, but a cluster and its steps go through the states of the
+Management Guide's cluster lifecycle, each held for as long as the [configuration](#configuration)
+says. A cluster is `STARTING` for the first half of `cluster-startup-delay-seconds` and `BOOTSTRAPPING`
+for the second, then `RUNNING` while it runs steps and `WAITING` once it has none left. A request's
+steps are all created `PENDING`, then start in the order they were added, at most `StepConcurrencyLevel`
+at a time, each `RUNNING` for `step-duration-seconds`. Steps added while the cluster starts wait for it
+to be ready. Raising the level with `ModifyCluster` starts the steps that now fit at once, and lowering
+it lets the running ones complete. `TerminateJobFlows` makes a cluster `TERMINATING` for
+`cluster-termination-delay-seconds` and then `TERMINATED`, and cancels any step not yet completed, as its
+API reference says. Each call finds the clusters it reads as they stand at that moment. With every
+duration at its default of 0, a cluster is ready, and its steps complete, before the call that started
+them returns.
+
+`CancelSteps` cancels a `PENDING` step, and a `RUNNING` one from release `emr-5.28.0`, as its API
+reference says. A cancelled `RUNNING` step makes room for the next `PENDING` one. On an earlier release
+a `RUNNING` step answers `FAILED` with "Step is not in a cancellable state: RUNNING".
+
+A cluster created with `Instances.KeepJobFlowAliveWhenNoSteps` set to `false` terminates after its
+initial steps or steps added later complete, reporting `ALL_STEPS_COMPLETED`, without waiting first. A
+cluster with no steps remains `WAITING` until steps are added. Set `KeepJobFlowAliveWhenNoSteps` to
+`true` to keep the cluster `WAITING` after its steps complete. Automatic step-completion termination
+also applies when termination protection is enabled.
+`AddJobFlowSteps` rejects a `TERMINATING` or `TERMINATED` cluster without adding steps or changing its
termination status, and answers an unknown cluster with `ValidationException` "Specified job flow ID
does not exist.", as AWS does.
-After automatic or manual termination, instance groups and fleets report `TERMINATED` with zero
-running or provisioned capacity; their requested and target capacities remain recorded.
-`ListInstances` retains each group's instances with their original synthetic IDs and `TERMINATED`
-status for 30 days after termination. Capacity modifications and new groups or fleets are rejected
+Instance groups and fleets report `RUNNING` from the cluster's creation until it is `TERMINATED`. Then
+they report `TERMINATED` with zero running or provisioned capacity; their requested and target
+capacities remain recorded. `ListInstances` retains each group's instances with their original
+synthetic IDs and `TERMINATED` status for 30 days after termination. New groups or fleets are rejected
on terminated clusters. As on AWS, `ModifyInstanceGroups` answers `ValidationException` "An instance
-group may only be modified when the cluster is running or waiting.", and `ModifyInstanceFleet` answers
-`InvalidRequestException` "A job flow that is shutting down, terminated, or finished may not be
-modified." with `ErrorCode` `INVALID_JOB_SHUT_DOWN`. `InstanceStates` filters the modeled `RUNNING` and
-`TERMINATED` instances.
+group may only be modified when the cluster is running or waiting." for a cluster in any other state,
+and `ModifyInstanceFleet` answers `InvalidRequestException` "A job flow that is shutting down,
+terminated, or finished may not be modified." with `ErrorCode` `INVALID_JOB_SHUT_DOWN` for a
+`TERMINATING` or `TERMINATED` one, and "An instance fleet may only be modified when the cluster is
+running or waiting." with `ErrorCode` `MODIFY_INSTANCE_FLEET_OF_RUNNING_JOBFLOW_ONLY` for one in any
+other state, such as `STARTING` or `BOOTSTRAPPING`. `AddInstanceFleet` gives a `TERMINATING` cluster
+the "shutting down" answer, and `AddInstanceGroups` gives it as `ValidationException`, as
+`AddJobFlowSteps` does. `InstanceStates` filters the modeled `RUNNING` and `TERMINATED`
+instances.
`TerminateJobFlows`, the four `Set*` cluster setters and `CancelSteps` refuse a request without a
member their model requires (`JobFlowIds` and the setting, or `StepIds`) with `ValidationException`
@@ -95,29 +115,43 @@ examples. A cluster's detail holds `severity`, `stateChangeReason`, `name`, `clu
`message`. The severity and message are the ones the guide's "Monitoring Amazon EMR events" tables
give each state, with times to the minute (`2026-10-24 09:41 UTC`).
-Since steps complete at once, each call sends:
+Each change sends its events:
-| Call | Events, in order |
+| Change | Events, in order |
|---|---|
-| `RunJobFlow` | the cluster `STARTING`; each step `PENDING`, then each `COMPLETED`; then the cluster `WAITING`, or `TERMINATED` with `ALL_STEPS_COMPLETED` when it ends after its steps |
-| `AddJobFlowSteps` | each step `PENDING`, then each `COMPLETED`; then the cluster `TERMINATED` with `ALL_STEPS_COMPLETED` when it ends after them |
-| `TerminateJobFlows` | `TERMINATED` with `USER_REQUEST` for each cluster it ends, also when a protected cluster fails the call |
-| `CancelSteps` | `CANCELLED` for each step it cancels, none while steps complete at once |
+| `RunJobFlow` | the cluster `STARTING`, then each step `PENDING` |
+| `AddJobFlowSteps` | each step `PENDING` |
+| The cluster is ready, or a step completes | the step `COMPLETED`; each step it makes room for `RUNNING`; once none is left, the cluster `WAITING`, or `TERMINATED` with `ALL_STEPS_COMPLETED` when it ends after its steps |
+| `TerminateJobFlows` | `CANCELLED` for each step not yet completed; the cluster `TERMINATED` with `USER_REQUEST` once it has been `TERMINATING` for the delay, also when a protected cluster fails the call |
+| `CancelSteps` | `CANCELLED` for each step it cancels, and `RUNNING` for the step a cancelled `RUNNING` one makes room for |
+
+A change that falls due between calls, such as a step completing after `step-duration-seconds`, is
+made at that time and its events sent then, whether or not anything reads the cluster.
A `STARTING` or `WAITING` cluster's `stateChangeReason` is `{"code":""}`, as in the guide's example,
and a `TERMINATED` one's carries the reason's code and message, such as
`{"code":"USER_REQUEST","message":"Terminated by user request"}`. `TERMINATED` is `INFO` for
`USER_REQUEST` and `ALL_STEPS_COMPLETED` and `CRITICAL` for any other reason; a cancelled step is
`WARN`. The tables have no row for a cluster's `BOOTSTRAPPING`, `RUNNING` or `TERMINATING`, which
-send nothing. A step is never seen `RUNNING`, and a `WAITING` cluster that runs added steps stays
-`WAITING`, so neither sends an event for it. The guide gives two `WAITING` messages without saying
-when each applies: a cluster that ran steps reports it "finished running all pending steps", one that
-did not that it "was created ... and is ready for use".
+send nothing, so a `WAITING` cluster that runs added steps sends `WAITING` again when they are done.
+The guide gives two `WAITING` messages without saying when each applies: a cluster that ran steps
+reports it "finished running all pending steps", one that did not that it "was created ... and is
+ready for use".
Events go out in the order their changes were made, once the call that made them no longer holds the
cluster. A rule's target that calls EMR while an event is delivered, such as one ending the cluster on
its `STARTING` event, has its events sent after the rest of that call's.
+## Configuration
+
+| Variable | Default | Description |
+|---|---|---|
+| `FLOCI_SERVICES_EMR_ENABLED` | `true` | Enable or disable the service |
+| `FLOCI_SERVICES_EMR_DEFAULT_RELEASE_LABEL` | `emr-7.5.0` | The release label of a cluster created without one; see [Release catalogue](#release-catalogue) |
+| `FLOCI_SERVICES_EMR_CLUSTER_STARTUP_DELAY_SECONDS` | `0` | Seconds a new cluster takes to be ready to run steps, the first half `STARTING` and the second `BOOTSTRAPPING`; `0` makes it ready as soon as it is created |
+| `FLOCI_SERVICES_EMR_STEP_DURATION_SECONDS` | `0` | Seconds a step stays `RUNNING` before it completes; `0` completes it as soon as it starts |
+| `FLOCI_SERVICES_EMR_CLUSTER_TERMINATION_DELAY_SECONDS` | `0` | Seconds a cluster stays `TERMINATING` before it is `TERMINATED`; `0` ends it at once |
+
## Release catalogue
`ListReleaseLabels`, `DescribeReleaseLabel` and `ListSupportedInstanceTypes` read a curated catalogue,
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 34ec80ebf6..2ab9f25014 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -1937,9 +1937,20 @@ public interface EmulatorConfig {
@WithDefault("emr-7.5.0")
String defaultReleaseLabel();
- /** Delay before a cluster reaches WAITING; 0 = advance synchronously. */
+ /**
+ * How long a new cluster takes to be ready to run steps, the first half STARTING and the second
+ * BOOTSTRAPPING; 0 makes it ready as soon as it is created.
+ */
@WithDefault("0")
int clusterStartupDelaySeconds();
+
+ /** How long a step stays RUNNING before it completes; 0 completes it as soon as it starts. */
+ @WithDefault("0")
+ int stepDurationSeconds();
+
+ /** How long a cluster stays TERMINATING before it is TERMINATED; 0 ends it at once. */
+ @WithDefault("0")
+ int clusterTerminationDelaySeconds();
}
interface WafV2ServiceConfig {
diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.java
new file mode 100644
index 0000000000..8beb619342
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.java
@@ -0,0 +1,258 @@
+package io.github.hectorvent.floci.services.emr;
+
+import io.github.hectorvent.floci.services.emr.model.EmrCluster;
+import io.github.hectorvent.floci.services.emr.model.EmrInstanceFleet;
+import io.github.hectorvent.floci.services.emr.model.EmrInstanceGroup;
+import io.github.hectorvent.floci.services.emr.model.EmrStep;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Comparator;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * A cluster's and its steps' lifecycle, as the Management Guide gives it in "Understanding the cluster
+ * lifecycle" and "Running steps to process data". A cluster is STARTING, then BOOTSTRAPPING, then
+ * RUNNING while it runs its steps, in the order they were added and at most its StepConcurrencyLevel
+ * at a time, then WAITING, or TERMINATING and TERMINATED when it ends after them. Nothing runs: each
+ * change falls due at a time worked out from three durations and the times already recorded, and
+ * {@link #advance} makes every change due by a given time, in the order they fall due, queuing the
+ * event of each state the guide's event tables report. With the durations at 0, every change falls
+ * due at the time of the call that leads to it.
+ */
+final class EmrClusterLifecycle {
+
+ static final String STARTING = "STARTING";
+ static final String BOOTSTRAPPING = "BOOTSTRAPPING";
+ static final String RUNNING = "RUNNING";
+ static final String WAITING = "WAITING";
+ static final String TERMINATING = "TERMINATING";
+ static final String TERMINATED = "TERMINATED";
+ static final String PENDING = "PENDING";
+ static final String COMPLETED = "COMPLETED";
+ static final String CANCELLED = "CANCELLED";
+ private static final Pattern RELEASE_LABEL = Pattern.compile("emr-(\\d{1,6})\\.(\\d{1,6})(?:\\..*)?");
+
+ private final Duration startup;
+ private final Duration stepRun;
+ private final Duration termination;
+ private final EmrEventPublisher events;
+
+ EmrClusterLifecycle(int startupDelaySeconds, int stepDurationSeconds, int terminationDelaySeconds,
+ EmrEventPublisher events) {
+ this.startup = seconds("cluster-startup-delay-seconds", startupDelaySeconds);
+ this.stepRun = seconds("step-duration-seconds", stepDurationSeconds);
+ this.termination = seconds("cluster-termination-delay-seconds", terminationDelaySeconds);
+ this.events = events;
+ }
+
+ private static Duration seconds(String setting, int value) {
+ if (value < 0) {
+ throw new IllegalArgumentException("floci.services.emr." + setting + " must not be negative: " + value);
+ }
+ return Duration.ofSeconds(value);
+ }
+
+ /** A new cluster, STARTING at {@code now}, with its steps PENDING. */
+ void start(EmrCluster cluster, Instant now, String region) {
+ cluster.setState(STARTING);
+ cluster.setCreationDateTime(now);
+ events.queue(events.clusterStateChange(cluster, STARTING, now), region);
+ submit(cluster, cluster.getSteps(), now, region);
+ }
+
+ /**
+ * {@code steps}, already on the cluster's list, added at {@code now}: "The state of all steps is set
+ * to PENDING", and a cluster that is ready starts what it has room for. Adding none changes nothing.
+ */
+ void submit(EmrCluster cluster, List<EmrStep> steps, Instant now, String region) {
+ if (steps.isEmpty()) {
+ return;
+ }
+ for (EmrStep step : steps) {
+ step.setState(PENDING);
+ step.setCreationDateTime(now);
+ if (step.getActionOnFailure() == null) {
+ step.setActionOnFailure("TERMINATE_CLUSTER");
+ }
+ events.queue(events.stepStatusChange(cluster, step, PENDING), region);
+ }
+ if (RUNNING.equals(cluster.getState()) || WAITING.equals(cluster.getState())) {
+ runSteps(cluster, now, region);
+ }
+ }
+
+ /**
+ * Ends the cluster for {@code code}: TERMINATING from {@code now}, then TERMINATED. TerminateJobFlows:
+ * "When a job flow is shut down, any step not yet completed is canceled".
+ */
+ void terminate(EmrCluster cluster, String code, String message, Instant now, String region) {
+ cluster.setState(TERMINATING);
+ cluster.setStateChangeReasonCode(code);
+ cluster.setStateChangeMessage(message);
+ cluster.setTerminatingDateTime(now);
+ for (EmrStep step : cluster.getSteps()) {
+ if (PENDING.equals(step.getState()) || RUNNING.equals(step.getState())) {
+ cancelStep(cluster, step, now, region);
+ }
+ }
+ }
+
+ /**
+ * CancelSteps for one step: null once it is cancelled, or why it cannot be. CancelSteps: "When you
+ * use Amazon EMR releases 5.28.0 and later, you can cancel steps that are in a PENDING or RUNNING
+ * state." A cancelled RUNNING step makes room for the next PENDING one.
+ */
+ String cancel(EmrCluster cluster, EmrStep step, Instant now, String region) {
+ boolean running = RUNNING.equals(step.getState()) && cancelsRunningSteps(cluster.getReleaseLabel());
+ if (!running && !PENDING.equals(step.getState())) {
+ return "Step is not in a cancellable state: " + step.getState();
+ }
+ cancelStep(cluster, step, now, region);
+ if (running) {
+ runSteps(cluster, now, region);
+ }
+ return null;
+ }
+
+ /**
+ * The cluster's StepConcurrencyLevel changed: a RUNNING cluster starts what it now has room for. A
+ * lower level lets the steps already running complete, as the guide says EMR does.
+ */
+ void concurrencyChanged(EmrCluster cluster, Instant now, String region) {
+ if (RUNNING.equals(cluster.getState())) {
+ runSteps(cluster, now, region);
+ }
+ }
+
+ /** Makes every change due by {@code now}, in the order they fall due; true when there was one. */
+ boolean advance(EmrCluster cluster, Instant now, String region) {
+ boolean changed = false;
+ Instant due;
+ while ((due = nextChange(cluster)) != null && !due.isAfter(now)) {
+ change(cluster, due, region);
+ changed = true;
+ }
+ return changed;
+ }
+
+ /** When the cluster's next change falls due, or null while it waits for a call. */
+ Instant nextChange(EmrCluster cluster) {
+ return switch (cluster.getState()) {
+ case STARTING -> cluster.getCreationDateTime().plus(startup.dividedBy(2));
+ case BOOTSTRAPPING -> cluster.getCreationDateTime().plus(startup);
+ case RUNNING -> cluster.getSteps().stream()
+ .filter(step -> RUNNING.equals(step.getState()))
+ .map(step -> step.getStartDateTime().plus(stepRun))
+ .min(Comparator.naturalOrder())
+ .orElse(null);
+ case TERMINATING -> cluster.getTerminatingDateTime().plus(termination);
+ default -> null;
+ };
+ }
+
+ private void change(EmrCluster cluster, Instant at, String region) {
+ switch (cluster.getState()) {
+ case STARTING -> cluster.setState(BOOTSTRAPPING);
+ case BOOTSTRAPPING -> {
+ cluster.setReadyDateTime(at);
+ cluster.setState(RUNNING);
+ runSteps(cluster, at, region);
+ }
+ case RUNNING -> {
+ for (EmrStep step : cluster.getSteps()) {
+ if (RUNNING.equals(step.getState()) && !step.getStartDateTime().plus(stepRun).isAfter(at)) {
+ step.setState(COMPLETED);
+ step.setEndDateTime(at);
+ events.queue(events.stepStatusChange(cluster, step, COMPLETED), region);
+ }
+ }
+ runSteps(cluster, at, region);
+ }
+ case TERMINATING -> {
+ cluster.setState(TERMINATED);
+ cluster.setEndDateTime(at);
+ releaseInstanceCapacity(cluster);
+ events.queue(events.clusterStateChange(cluster, TERMINATED, at), region);
+ }
+ default -> throw new IllegalStateException("Nothing falls due for a " + cluster.getState() + " cluster");
+ }
+ }
+
+ /**
+ * Starts PENDING steps in the order they were added while the cluster has room: "pending steps in
+ * queue transition to running state in the order they were submitted". With none left, the
+ * cluster is done.
+ */
+ private void runSteps(EmrCluster cluster, Instant at, String region) {
+ long running = cluster.getSteps().stream().filter(step -> RUNNING.equals(step.getState())).count();
+ // The model allows 1 to 256; Floci stores any level it is given, and a lower one would never start a step.
+ int level = Math.max(1, cluster.getStepConcurrencyLevel());
+ for (EmrStep step : cluster.getSteps()) {
+ if (running >= level) {
+ break;
+ }
+ if (PENDING.equals(step.getState())) {
+ step.setState(RUNNING);
+ step.setStartDateTime(at);
+ events.queue(events.stepStatusChange(cluster, step, RUNNING), region);
+ running++;
+ }
+ }
+ if (running > 0) {
+ // A WAITING cluster's "ready to run steps" no longer holds; no source gives a RUNNING one's reason.
+ cluster.setState(RUNNING);
+ cluster.setStateChangeReasonCode(null);
+ cluster.setStateChangeMessage(null);
+ } else {
+ finish(cluster, at, region);
+ }
+ }
+
+ /**
+ * The cluster has no step left to run: it waits for more, or, set to end after its steps, ends
+ * without waiting. One created without steps waits either way.
+ */
+ private void finish(EmrCluster cluster, Instant at, String region) {
+ if (cluster.isAutoTerminate() && !cluster.getSteps().isEmpty()) {
+ terminate(cluster, "ALL_STEPS_COMPLETED", "All steps completed.", at, region);
+ return;
+ }
+ cluster.setState(WAITING);
+ cluster.setStateChangeReasonCode(cluster.getSteps().isEmpty() ? null : "ALL_STEPS_COMPLETED");
+ cluster.setStateChangeMessage("Cluster ready to run steps.");
+ events.queue(events.clusterStateChange(cluster, WAITING, at), region);
+ }
+
+ private void cancelStep(EmrCluster cluster, EmrStep step, Instant at, String region) {
+ step.setState(CANCELLED);
+ step.setEndDateTime(at);
+ events.queue(events.stepStatusChange(cluster, step, CANCELLED), region);
+ }
+
+ private static void releaseInstanceCapacity(EmrCluster cluster) {
+ for (EmrInstanceGroup group : cluster.getInstanceGroups()) {
+ group.setState(TERMINATED);
+ group.setTerminatedInstanceCount(group.getRunningInstanceCount());
+ group.setRunningInstanceCount(0);
+ }
+ for (EmrInstanceFleet fleet : cluster.getInstanceFleets()) {
+ fleet.setState(TERMINATED);
+ fleet.setProvisionedOnDemandCapacity(0);
+ fleet.setProvisionedSpotCapacity(0);
+ }
+ }
+
+ /** Releases from 5.28.0 cancel RUNNING steps; a label not of the emr-x.y form is taken as current. */
+ static boolean cancelsRunningSteps(String releaseLabel) {
+ Matcher matcher = RELEASE_LABEL.matcher(releaseLabel == null ? "" : releaseLabel);
+ if (!matcher.matches()) {
+ return true;
+ }
+ int major = Integer.parseInt(matcher.group(1));
+ int minor = Integer.parseInt(matcher.group(2));
+ return major > 5 || (major == 5 && minor >= 28);
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrEventPublisher.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrEventPublisher.java
index 7902a1642e..38d2db4b34 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrEventPublisher.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrEventPublisher.java
@@ -59,8 +59,8 @@ public class EmrEventPublisher {
this.regionResolver = regionResolver;
}
- /** {@code cluster} entering {@code state}: STARTING, WAITING or TERMINATED. */
- Event clusterStateChange(EmrCluster cluster, String state) {
+ /** {@code cluster} entering {@code state} at {@code at}: STARTING, WAITING or TERMINATED. */
+ Event clusterStateChange(EmrCluster cluster, String state, Instant at) {
boolean terminated = "TERMINATED".equals(state);
String code = cluster.getStateChangeReasonCode();
// The examples' reason is a JSON string: {"code":""} for a WAITING cluster, and the reason's code
@@ -76,11 +76,11 @@ public class EmrEventPublisher {
detail.put("name", cluster.getName());
detail.put("clusterId", cluster.getId());
detail.put("state", state);
- detail.put("message", clusterMessage(cluster, state));
+ detail.put("message", clusterMessage(cluster, state, at));
return new Event(CLUSTER_STATE_CHANGE, detail.toString());
}
- /** {@code step} of {@code cluster} entering {@code state}: PENDING, COMPLETED or CANCELLED. */
+ /** {@code step} of {@code cluster} entering {@code state}: PENDING, RUNNING, COMPLETED or CANCELLED. */
Event stepStatusChange(EmrCluster cluster, EmrStep step, String state) {
ObjectNode detail = objectMapper.createObjectNode();
detail.put("severity", "CANCELLED".equals(state) ? "WARN" : "INFO");
@@ -126,17 +126,15 @@ public class EmrEventPublisher {
SOURCE, queued.event().detailType(), queued.event().detail(), null, queued.region()));
}
- private static String clusterMessage(EmrCluster cluster, String state) {
+ private static String clusterMessage(EmrCluster cluster, String state, Instant at) {
String named = cluster.getId() + " (" + cluster.getName() + ")";
return switch (state) {
- case "STARTING" -> "Amazon EMR cluster " + named + " was requested at "
- + time(cluster.getCreationDateTime()) + " and is being created.";
+ case "STARTING" -> "Amazon EMR cluster " + named + " was requested at " + time(at)
+ + " and is being created.";
case "WAITING" -> cluster.getSteps().isEmpty()
- ? "Amazon EMR cluster " + named + " was created at " + time(cluster.getReadyDateTime())
- + " and is ready for use."
- : "Amazon EMR cluster " + named + " finished running all pending steps at "
- + time(cluster.getReadyDateTime()) + ".";
- case "TERMINATED" -> "Amazon EMR Cluster " + named + " has terminated at " + time(cluster.getEndDateTime())
+ ? "Amazon EMR cluster " + named + " was created at " + time(at) + " and is ready for use."
+ : "Amazon EMR cluster " + named + " finished running all pending steps at " + time(at) + ".";
+ case "TERMINATED" -> "Amazon EMR Cluster " + named + " has terminated at " + time(at)
+ " with a reason of " + cluster.getStateChangeReasonCode() + ".";
default -> throw new IllegalArgumentException("No cluster event for state " + state);
};
@@ -148,6 +146,8 @@ public class EmrEventPublisher {
return switch (state) {
case "PENDING" -> "Step " + stepNamed + " was added to Amazon EMR cluster " + clusterNamed + " at "
+ time(step.getCreationDateTime()) + " and is pending execution.";
+ case "RUNNING" -> "Step " + stepNamed + " in Amazon EMR cluster " + clusterNamed + " started running at "
+ + time(step.getStartDateTime()) + ".";
case "COMPLETED" -> "Step " + stepNamed + " in Amazon EMR cluster " + clusterNamed
+ " completed execution at " + time(step.getEndDateTime()) + ". The step started running at "
+ time(step.getStartDateTime())
diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java
index 5b4caf1844..c96d850ea9 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java
@@ -2,13 +2,17 @@ package io.github.hectorvent.floci.services.emr;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.core.common.AwsRegions;
import io.github.hectorvent.floci.core.common.PaginatedResult;
import io.github.hectorvent.floci.core.common.Pagination;
import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.common.RequestScopes;
import io.github.hectorvent.floci.core.common.SmithyValidationErrors;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.StorageBackend;
import io.github.hectorvent.floci.core.storage.StorageFactory;
import io.github.hectorvent.floci.services.emr.model.EmrBlockPublicAccess;
@@ -17,10 +21,16 @@ import io.github.hectorvent.floci.services.emr.model.EmrInstanceFleet;
import io.github.hectorvent.floci.services.emr.model.EmrInstanceGroup;
import io.github.hectorvent.floci.services.emr.model.EmrStep;
import io.github.hectorvent.floci.services.emr.model.SecurityConfiguration;
+import io.quarkus.runtime.StartupEvent;
+import jakarta.annotation.PreDestroy;
import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;
+import org.jboss.logging.Logger;
import java.security.SecureRandom;
+import java.time.Clock;
+import java.time.Duration;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
@@ -31,32 +41,46 @@ import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
import java.util.function.Function;
+import java.util.function.Supplier;
/**
- * EMR management-plane business logic. Cluster and step lifecycles are simulated
- * through immediate state transitions (no Hadoop runs): steps complete immediately,
- * leaving persistent clusters WAITING and transient clusters with steps TERMINATED.
- * The {@code clusterStartupDelaySeconds} config exists for
- * future deferred transitions; Phase 1 advances synchronously for deterministic tests.
- * Every state a cluster or step enters is published as an {@code aws.emr} event, once the call that
- * made the change no longer holds the service.
+ * EMR management-plane business logic. Nothing runs: clusters and steps go through the states of
+ * {@link EmrClusterLifecycle}, timed by {@code cluster-startup-delay-seconds},
+ * {@code step-duration-seconds} and {@code cluster-termination-delay-seconds}, all 0 by default, which
+ * makes a cluster ready, and a step complete, before the call that started it returns. A call brings
+ * each cluster it reads up to the time it is made, and the service wakes at each cluster's next
+ * change, so the change's event goes out without anyone asking. Events are published once the call
+ * that made the change no longer holds the service.
*/
@ApplicationScoped
public class EmrService {
+ private static final Logger LOG = Logger.getLogger(EmrService.class);
private static final String UPPER_ALNUM = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
private static final SecureRandom RANDOM = new SecureRandom();
- // AWS's answer to a change on a terminated cluster, for AddJobFlowSteps and ModifyInstanceFleet.
+ // AWS's answer to a change on a terminated cluster, for AddJobFlowSteps and ModifyInstanceFleet; Floci
+ // gives it to AddInstanceGroups and AddInstanceFleet too, for one shutting down.
private static final String JOB_FLOW_SHUT_DOWN =
"A job flow that is shutting down, terminated, or finished may not be modified.";
+ private static final ObjectMapper SNAPSHOT_MAPPER = new ObjectMapper().registerModule(new JavaTimeModule());
- private final StorageBackend<String, EmrCluster> clusterStore;
+ private final AccountAwareStorageBackend<EmrCluster> clusterStore;
private final StorageBackend<String, SecurityConfiguration> secConfigStore;
// Block public access is an account setting per Region, so this store is keyed by Region.
private final StorageBackend<String, EmrBlockPublicAccess> blockPublicAccessStore;
private final RegionResolver regionResolver;
private final EmrEventPublisher eventPublisher;
+ private final EmrClusterLifecycle lifecycle;
+ private final Clock clock;
+ private final Wakeups wakeups;
+ // Each cluster's next wakeup, by account and id, so a call that reads a cluster does not add another.
+ private final Map<String, Instant> wakeupsDue = new ConcurrentHashMap<>();
private static final Set<String> COMPUTE_LIMITS_UNIT_TYPES = Set.of("InstanceFleetUnits", "Instances", "VCPU");
private static final Set<String> SCALING_STRATEGIES = Set.of("DEFAULT", "ADVANCED");
@@ -69,9 +93,62 @@ public class EmrService {
+ "\"PermittedPublicSecurityGroupRuleRanges\":[{\"MinRange\":22,\"MaxRange\":22}]}";
private final String defaultReleaseLabel;
+ /**
+ * When the service next brings a cluster up to date by itself, and where it then sends the events
+ * that queued; a test runs both when it chooses.
+ */
+ interface Wakeups {
+
+ void at(Instant due, Runnable task);
+
+ /** Sends a wakeup's events, by default on the wakeup's own thread. */
+ default void deliver(Runnable delivery) {
+ delivery.run();
+ }
+
+ default void close() {
+ }
+
+ /**
+ * Wakeups on a daemon thread, each at its time by {@code clock}, and their events sent on another,
+ * in the order they queued: a rule's target can take seconds to answer, and no cluster's change
+ * waits for it.
+ */
+ static Wakeups timer(Clock clock) {
+ ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(
+ Thread.ofPlatform().daemon().name("emr-lifecycle").factory());
+ ExecutorService deliveries = Executors.newSingleThreadExecutor(
+ Thread.ofPlatform().daemon().name("emr-events").factory());
+ return new Wakeups() {
+ @Override
+ public void at(Instant due, Runnable task) {
+ long delay = Math.max(0, Duration.between(clock.instant(), due).toMillis());
+ executor.schedule(task, delay, TimeUnit.MILLISECONDS);
+ }
+
+ @Override
+ public void deliver(Runnable delivery) {
+ deliveries.execute(delivery);
+ }
+
+ @Override
+ public void close() {
+ executor.shutdownNow();
+ deliveries.shutdownNow();
+ }
+ };
+ }
+ }
+
@Inject
public EmrService(StorageFactory storageFactory, EmulatorConfig config, RegionResolver regionResolver,
EmrEventPublisher eventPublisher) {
+ this(storageFactory, config, regionResolver, eventPublisher, Clock.systemUTC(),
+ Wakeups.timer(Clock.systemUTC()));
+ }
+
+ EmrService(StorageFactory storageFactory, EmulatorConfig config, RegionResolver regionResolver,
+ EmrEventPublisher eventPublisher, Clock clock, Wakeups wakeups) {
this.clusterStore = storageFactory.create("emr", "emr-clusters.json",
new TypeReference<Map<String, EmrCluster>>() {});
this.secConfigStore = storageFactory.create("emr", "emr-security-configs.json",
@@ -81,55 +158,68 @@ public class EmrService {
this.regionResolver = regionResolver;
this.eventPublisher = eventPublisher;
this.defaultReleaseLabel = config.services().emr().defaultReleaseLabel();
+ this.lifecycle = new EmrClusterLifecycle(config.services().emr().clusterStartupDelaySeconds(),
+ config.services().emr().stepDurationSeconds(),
+ config.services().emr().clusterTerminationDelaySeconds(), eventPublisher);
+ this.clock = clock;
+ this.wakeups = wakeups;
+ // After a restart, a stored cluster with a change still to come wakes the service as before.
+ for (AccountAwareStorageBackend.AccountEntry<EmrCluster> entry
+ : clusterStore.scanAllAccountEntries(k -> true)) {
+ Instant due = lifecycle.nextChange(entry.value());
+ if (due != null) {
+ wakeAt(entry.accountId(), entry.value(), due);
+ }
+ }
}
- // ──────────────────────────── Cluster lifecycle ────────────────────────────
+ // A normal-scoped bean is built on first use, so without this a cluster left mid-change by a restart
+ // would not move on, nor send its events, until the next EMR call.
+ void onStart(@Observes StartupEvent ignored) {
+ LOG.debugv("EMR service ready");
+ }
- public EmrCluster runJobFlow(EmrCluster cluster, String region) {
- try {
- return runJobFlowLocked(cluster, region);
- } finally {
- eventPublisher.publishQueued();
- }
+ @PreDestroy
+ void shutdown() {
+ wakeups.close();
}
- private synchronized EmrCluster runJobFlowLocked(EmrCluster cluster, String region) {
- // Only a missing Name fails AWS's framework validation: the member is REQUIRED but its
- // constraint is len=[0..256], so an explicit empty string is accepted by real EMR.
- if (cluster.getName() == null) {
- throw SmithyValidationErrors.validationException(SmithyValidationErrors.notNull("name"));
- }
- String id = "j-" + randomId(13);
- cluster.setId(id);
- cluster.setRegion(region);
- cluster.setClusterArn(regionResolver.buildArn("elasticmapreduce", region, "cluster/" + id));
- if (cluster.getReleaseLabel() == null) {
- cluster.setReleaseLabel(defaultReleaseLabel);
- }
- cluster.setInstanceCollectionType(cluster.getInstanceFleets().isEmpty()
- ? "INSTANCE_GROUP" : "INSTANCE_FLEET");
- cluster.setAutoTerminate(!cluster.isKeepJobFlowAliveWhenNoSteps());
- cluster.setMasterPublicDnsName(masterDnsName(region));
- cluster.setCreationDateTime(Instant.now().truncatedTo(ChronoUnit.MILLIS));
- for (EmrInstanceGroup g : cluster.getInstanceGroups()) {
- g.setId("ig-" + randomId(13));
- g.setRunningInstanceCount(g.getRequestedInstanceCount());
- g.setState("RUNNING");
- }
- for (EmrInstanceFleet f : cluster.getInstanceFleets()) {
- f.setId("if-" + randomId(13));
- f.setProvisionedOnDemandCapacity(f.getTargetOnDemandCapacity());
- f.setProvisionedSpotCapacity(f.getTargetSpotCapacity());
- f.setState("RUNNING");
- }
- eventPublisher.queue(eventPublisher.clusterStateChange(cluster, "STARTING"), region);
- runSteps(cluster, cluster.getSteps(), region);
- advanceToWaiting(cluster);
- terminateAfterCompletedSteps(cluster);
- clusterStore.put(id, cluster);
- // The WAITING a transient cluster passes through here is never seen, so it has no event.
- eventPublisher.queue(eventPublisher.clusterStateChange(cluster, cluster.getState()), region);
- return cluster;
+ // ──────────────────────────── Cluster lifecycle ────────────────────────────
+
+ public EmrCluster runJobFlow(EmrCluster cluster, String region) {
+ return call(() -> {
+ // Only a missing Name fails AWS's framework validation: the member is REQUIRED but its
+ // constraint is len=[0..256], so an explicit empty string is accepted by real EMR.
+ if (cluster.getName() == null) {
+ throw SmithyValidationErrors.validationException(SmithyValidationErrors.notNull("name"));
+ }
+ String id = "j-" + randomId(13);
+ cluster.setId(id);
+ cluster.setRegion(region);
+ cluster.setClusterArn(regionResolver.buildArn("elasticmapreduce", region, "cluster/" + id));
+ if (cluster.getReleaseLabel() == null) {
+ cluster.setReleaseLabel(defaultReleaseLabel);
+ }
+ cluster.setInstanceCollectionType(cluster.getInstanceFleets().isEmpty()
+ ? "INSTANCE_GROUP" : "INSTANCE_FLEET");
+ cluster.setAutoTerminate(!cluster.isKeepJobFlowAliveWhenNoSteps());
+ cluster.setMasterPublicDnsName(masterDnsName(region));
+ for (EmrInstanceGroup g : cluster.getInstanceGroups()) {
+ g.setId("ig-" + randomId(13));
+ g.setRunningInstanceCount(g.getRequestedInstanceCount());
+ g.setState("RUNNING");
+ }
+ for (EmrInstanceFleet f : cluster.getInstanceFleets()) {
+ f.setId("if-" + randomId(13));
+ f.setProvisionedOnDemandCapacity(f.getTargetOnDemandCapacity());
+ f.setProvisionedSpotCapacity(f.getTargetSpotCapacity());
+ f.setState("RUNNING");
+ }
+ cluster.getSteps().forEach(step -> step.setId("s-" + randomId(13)));
+ lifecycle.start(cluster, now(), region);
+ store(cluster);
+ return cluster;
+ });
}
/** The master node's private DNS name, derived from the region so a stored name never goes stale. */
@@ -138,209 +228,180 @@ public class EmrService {
}
public EmrCluster describeCluster(String id) {
- return requireCluster(id);
+ return call(() -> snapshotOf(requireCluster(id)));
}
public List<EmrCluster> listClusters(String region, List<String> states,
Instant createdAfter, Instant createdBefore) {
- List<EmrCluster> result = new ArrayList<>();
- for (EmrCluster c : clusterStore.scan(k -> true)) {
- if (!region.equals(c.getRegion())) {
- continue;
- }
- if (states != null && !states.isEmpty() && !states.contains(c.getState())) {
- continue;
- }
- Instant createdAt = c.getCreationDateTime();
- if (createdAt != null) {
- createdAt = createdAt.truncatedTo(ChronoUnit.MILLIS);
- }
- if (createdAfter != null && (createdAt == null || !createdAt.isAfter(createdAfter))) {
- continue;
- }
- if (createdBefore != null && (createdAt == null || !createdAt.isBefore(createdBefore))) {
- continue;
+ return call(() -> {
+ List<EmrCluster> result = new ArrayList<>();
+ for (EmrCluster c : clusterStore.scan(k -> true)) {
+ if (!region.equals(c.getRegion())) {
+ continue;
+ }
+ settle(c);
+ if (states != null && !states.isEmpty() && !states.contains(c.getState())) {
+ continue;
+ }
+ Instant createdAt = c.getCreationDateTime();
+ if (createdAt != null) {
+ createdAt = createdAt.truncatedTo(ChronoUnit.MILLIS);
+ }
+ if (createdAfter != null && (createdAt == null || !createdAt.isAfter(createdAfter))) {
+ continue;
+ }
+ if (createdBefore != null && (createdAt == null || !createdAt.isBefore(createdBefore))) {
+ continue;
+ }
+ result.add(snapshotOf(c));
}
- result.add(c);
- }
- return result;
+ return result;
+ });
}
public void terminateJobFlows(List<String> ids) {
- try {
- terminateJobFlowsLocked(ids);
- } finally {
- // A request naming a protected cluster still terminates the others before it fails.
- eventPublisher.publishQueued();
- }
- }
-
- private synchronized void terminateJobFlowsLocked(List<String> ids) {
- // AWS terminates the unprotected clusters in the request and then fails it with a
- // ValidationException if any were termination protected.
- boolean anyProtected = false;
- for (String id : ids) {
- EmrCluster cluster = findCluster(id).orElse(null);
- if (cluster == null) {
- continue;
- }
- if ("TERMINATED".equals(cluster.getState())) {
- continue;
+ // A request naming a protected cluster still terminates the others before it fails.
+ run(() -> {
+ // AWS terminates the unprotected clusters in the request and then fails it with a
+ // ValidationException if any were termination protected.
+ boolean anyProtected = false;
+ for (String id : ids) {
+ EmrCluster cluster = findCluster(id).orElse(null);
+ if (cluster == null || isShutDown(cluster)) {
+ continue;
+ }
+ if (cluster.isTerminationProtected()) {
+ anyProtected = true;
+ continue;
+ }
+ lifecycle.terminate(cluster, "USER_REQUEST", "Terminated by user request", now(), regionOf(cluster));
+ store(cluster);
}
- if (cluster.isTerminationProtected()) {
- anyProtected = true;
- continue;
+ if (anyProtected) {
+ throw new AwsException("ValidationException",
+ "Could not shut down one or more job flows since they are termination protected.", 400);
}
- cluster.setState("TERMINATED");
- releaseInstanceCapacity(cluster);
- cluster.setStateChangeReasonCode("USER_REQUEST");
- cluster.setStateChangeMessage("Terminated by user request");
- cluster.setEndDateTime(Instant.now());
- clusterStore.put(id, cluster);
- eventPublisher.queue(eventPublisher.clusterStateChange(cluster, "TERMINATED"),
- regionResolver.getRegion());
- }
- if (anyProtected) {
- throw new AwsException("ValidationException",
- "Could not shut down one or more job flows since they are termination protected.", 400);
- }
+ });
}
- public synchronized void setTerminationProtection(List<String> ids, boolean value) {
- mutateClusters(ids, c -> c.setTerminationProtected(value));
+ public void setTerminationProtection(List<String> ids, boolean value) {
+ run(() -> mutateClusters(ids, c -> c.setTerminationProtected(value)));
}
- public synchronized void setVisibleToAllUsers(List<String> ids, boolean value) {
- mutateClusters(ids, c -> c.setVisibleToAllUsers(value));
+ public void setVisibleToAllUsers(List<String> ids, boolean value) {
+ run(() -> mutateClusters(ids, c -> c.setVisibleToAllUsers(value)));
}
- public synchronized void setKeepJobFlowAliveWhenNoSteps(List<String> ids, boolean value) {
- mutateClusters(ids, c -> {
+ public void setKeepJobFlowAliveWhenNoSteps(List<String> ids, boolean value) {
+ run(() -> mutateClusters(ids, c -> {
c.setKeepJobFlowAliveWhenNoSteps(value);
c.setAutoTerminate(!value);
- });
+ }));
}
- public synchronized void setUnhealthyNodeReplacement(List<String> ids, boolean value) {
- mutateClusters(ids, c -> c.setUnhealthyNodeReplacement(value));
+ public void setUnhealthyNodeReplacement(List<String> ids, boolean value) {
+ run(() -> mutateClusters(ids, c -> c.setUnhealthyNodeReplacement(value)));
}
- public synchronized int modifyCluster(String id, Integer stepConcurrencyLevel) {
- EmrCluster cluster = requireCluster(id);
- if (stepConcurrencyLevel != null) {
- cluster.setStepConcurrencyLevel(stepConcurrencyLevel);
- clusterStore.put(id, cluster);
- }
- return cluster.getStepConcurrencyLevel();
+ public int modifyCluster(String id, Integer stepConcurrencyLevel) {
+ return call(() -> {
+ EmrCluster cluster = requireCluster(id);
+ if (stepConcurrencyLevel != null) {
+ cluster.setStepConcurrencyLevel(stepConcurrencyLevel);
+ lifecycle.concurrencyChanged(cluster, now(), regionOf(cluster));
+ store(cluster);
+ }
+ return cluster.getStepConcurrencyLevel();
+ });
}
// ──────────────────────────── Steps ────────────────────────────
public List<String> addJobFlowSteps(String clusterId, List<EmrStep> steps) {
- try {
- return addJobFlowStepsLocked(clusterId, steps);
- } finally {
- eventPublisher.publishQueued();
- }
- }
-
- private synchronized List<String> addJobFlowStepsLocked(String clusterId, List<EmrStep> steps) {
- // AWS's own wording for an unknown job flow here, not DescribeCluster's: a botocore ClientError
- // (raster-foundry/raster-foundry#352) and a CloudTrail record on AWS re:Post both show it.
- EmrCluster cluster = clusterId == null ? requireCluster(null) : findCluster(clusterId).orElseThrow(
- () -> new AwsException("ValidationException", "Specified job flow ID does not exist.", 400));
- if ("TERMINATED".equals(cluster.getState())) {
- throw new AwsException("ValidationException", JOB_FLOW_SHUT_DOWN, 400);
- }
- String region = regionResolver.getRegion();
- runSteps(cluster, steps, region);
- cluster.getSteps().addAll(steps);
- List<String> ids = steps.stream().map(EmrStep::getId).toList();
- if (!steps.isEmpty()) {
- terminateAfterCompletedSteps(cluster);
- }
- clusterStore.put(clusterId, cluster);
- if ("TERMINATED".equals(cluster.getState())) {
- eventPublisher.queue(eventPublisher.clusterStateChange(cluster, "TERMINATED"), region);
- }
- return ids;
+ return call(() -> {
+ // AWS's own wording for an unknown job flow here, not DescribeCluster's: a botocore ClientError
+ // (raster-foundry/raster-foundry#352) and a CloudTrail record on AWS re:Post both show it.
+ EmrCluster cluster = clusterId == null ? requireCluster(null) : findCluster(clusterId).orElseThrow(
+ () -> new AwsException("ValidationException", "Specified job flow ID does not exist.", 400));
+ if (isShutDown(cluster)) {
+ throw jobFlowShutDownValidation();
+ }
+ steps.forEach(step -> step.setId("s-" + randomId(13)));
+ cluster.getSteps().addAll(steps);
+ lifecycle.submit(cluster, steps, now(), regionOf(cluster));
+ store(cluster);
+ return steps.stream().map(EmrStep::getId).toList();
+ });
}
public EmrStep describeStep(String clusterId, String stepId) {
- EmrCluster cluster = requireCluster(clusterId);
- return cluster.getSteps().stream()
- .filter(s -> s.getId().equals(stepId))
- .findFirst()
- .orElseThrow(() -> new AwsException("InvalidRequestException",
- "Step " + stepId + " does not exist.", 400));
+ return call(() -> {
+ EmrCluster cluster = snapshotOf(requireCluster(clusterId));
+ return cluster.getSteps().stream()
+ .filter(s -> s.getId().equals(stepId))
+ .findFirst()
+ .orElseThrow(() -> new AwsException("InvalidRequestException",
+ "Step " + stepId + " does not exist.", 400));
+ });
}
public List<EmrStep> listSteps(String clusterId, List<String> states, List<String> stepIds) {
- EmrCluster cluster = requireCluster(clusterId);
- List<EmrStep> result = new ArrayList<>();
- // AWS returns steps in reverse submission order (newest first).
- List<EmrStep> steps = cluster.getSteps();
- for (int i = steps.size() - 1; i >= 0; i--) {
- EmrStep step = steps.get(i);
- if (states != null && !states.isEmpty() && !states.contains(step.getState())) {
- continue;
- }
- if (stepIds != null && !stepIds.isEmpty() && !stepIds.contains(step.getId())) {
- continue;
+ return call(() -> {
+ EmrCluster cluster = snapshotOf(requireCluster(clusterId));
+ List<EmrStep> result = new ArrayList<>();
+ // AWS returns steps in reverse submission order (newest first).
+ List<EmrStep> steps = cluster.getSteps();
+ for (int i = steps.size() - 1; i >= 0; i--) {
+ EmrStep step = steps.get(i);
+ if (states != null && !states.isEmpty() && !states.contains(step.getState())) {
+ continue;
+ }
+ if (stepIds != null && !stepIds.isEmpty() && !stepIds.contains(step.getId())) {
+ continue;
+ }
+ result.add(step);
}
- result.add(step);
- }
- return result;
+ return result;
+ });
}
public record CancelStepInfo(String stepId, String status, String reason) {}
public List<CancelStepInfo> cancelSteps(String clusterId, List<String> stepIds) {
- try {
- return cancelStepsLocked(clusterId, stepIds);
- } finally {
- eventPublisher.publishQueued();
- }
- }
-
- private synchronized List<CancelStepInfo> cancelStepsLocked(String clusterId, List<String> stepIds) {
- EmrCluster cluster = requireCluster(clusterId);
- List<CancelStepInfo> result = new ArrayList<>();
- for (String stepId : stepIds) {
- EmrStep step = cluster.getSteps().stream()
- .filter(s -> s.getId().equals(stepId)).findFirst().orElse(null);
- if (step == null) {
- result.add(new CancelStepInfo(stepId, "FAILED", "Step does not exist."));
- } else if ("PENDING".equals(step.getState())) {
- step.setState("CANCELLED");
- step.setEndDateTime(Instant.now());
- result.add(new CancelStepInfo(stepId, "SUBMITTED", null));
- eventPublisher.queue(eventPublisher.stepStatusChange(cluster, step, "CANCELLED"),
- regionResolver.getRegion());
- } else {
- result.add(new CancelStepInfo(stepId, "FAILED",
- "Step is not in a cancellable state: " + step.getState()));
+ return call(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ List<CancelStepInfo> result = new ArrayList<>();
+ for (String stepId : stepIds) {
+ EmrStep step = cluster.getSteps().stream()
+ .filter(s -> s.getId().equals(stepId)).findFirst().orElse(null);
+ String refusal = step == null ? "Step does not exist."
+ : lifecycle.cancel(cluster, step, now(), regionOf(cluster));
+ result.add(refusal == null
+ ? new CancelStepInfo(stepId, "SUBMITTED", null)
+ : new CancelStepInfo(stepId, "FAILED", refusal));
}
- }
- clusterStore.put(clusterId, cluster);
- return result;
+ store(cluster);
+ return result;
+ });
}
// ──────────────────────────── Instance groups / fleets ────────────────────────────
- public synchronized List<String> addInstanceGroups(String clusterId, List<EmrInstanceGroup> groups) {
- EmrCluster cluster = requireCluster(clusterId);
- rejectTerminatedCluster(cluster);
- List<String> ids = new ArrayList<>();
- for (EmrInstanceGroup group : groups) {
- group.setId("ig-" + randomId(13));
- group.setRunningInstanceCount(group.getRequestedInstanceCount());
- group.setState("RUNNING");
- cluster.getInstanceGroups().add(group);
- ids.add(group.getId());
- }
- clusterStore.put(clusterId, cluster);
- return ids;
+ public List<String> addInstanceGroups(String clusterId, List<EmrInstanceGroup> groups) {
+ return call(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ rejectShutDownCluster(cluster, EmrService::jobFlowShutDownValidation);
+ List<String> ids = new ArrayList<>();
+ for (EmrInstanceGroup group : groups) {
+ group.setId("ig-" + randomId(13));
+ group.setRunningInstanceCount(group.getRequestedInstanceCount());
+ group.setState("RUNNING");
+ cluster.getInstanceGroups().add(group);
+ ids.add(group.getId());
+ }
+ clusterStore.put(clusterId, cluster);
+ return ids;
+ });
}
/** One InstanceGroupModifyConfig: a null count or configurations leaves that setting as it is. */
@@ -351,110 +412,123 @@ public class EmrService {
* without it each group is looked up across the account's clusters in the Region. Every modification is
* checked before any is applied.
*/
- public synchronized void modifyInstanceGroups(String clusterId, String region,
- List<InstanceGroupModification> modifications) {
- if (modifications.isEmpty()) {
- throw invalid("InstanceGroups must contain at least one instance group.");
- }
- Map<String, EmrCluster> touched = new LinkedHashMap<>();
- List<EmrInstanceGroup> groups = new ArrayList<>();
- for (InstanceGroupModification mod : modifications) {
- EmrCluster found = clusterId != null ? requireCluster(clusterId) : clusterOwning(region, mod.instanceGroupId());
- // AWS's answer here, recorded by Terraform users
- // (hashicorp/terraform-provider-aws#1355 and #9400).
- if ("TERMINATED".equals(found.getState())) {
- throw new AwsException("ValidationException",
- "An instance group may only be modified when the cluster is running or waiting.", 400);
+ public void modifyInstanceGroups(String clusterId, String region, List<InstanceGroupModification> modifications) {
+ run(() -> {
+ if (modifications.isEmpty()) {
+ throw invalid("InstanceGroups must contain at least one instance group.");
}
- // One copy per cluster, so two groups of the same cluster are changed and stored together.
- EmrCluster cluster = touched.computeIfAbsent(found.getId(), id -> found);
- EmrInstanceGroup group = requireInstanceGroup(cluster, mod.instanceGroupId());
- if (mod.instanceCount() != null) {
- if (mod.instanceCount() < 0) {
- throw invalid("InstanceCount must not be negative.");
+ Map<String, EmrCluster> touched = new LinkedHashMap<>();
+ List<EmrInstanceGroup> groups = new ArrayList<>();
+ for (InstanceGroupModification mod : modifications) {
+ EmrCluster found = clusterId != null
+ ? requireCluster(clusterId) : clusterOwning(region, mod.instanceGroupId());
+ // AWS's answer here, recorded by Terraform users
+ // (hashicorp/terraform-provider-aws#1355 and #9400).
+ if (!runsOrWaits(found)) {
+ throw new AwsException("ValidationException",
+ "An instance group may only be modified when the cluster is running or waiting.", 400);
}
- if ("MASTER".equals(group.getInstanceGroupType())
- && mod.instanceCount() != group.getRequestedInstanceCount()) {
- throw invalid("The instance count of a master instance group cannot be modified.");
+ // One copy per cluster, so two groups of the same cluster are changed and stored together.
+ EmrCluster cluster = touched.computeIfAbsent(found.getId(), id -> found);
+ EmrInstanceGroup group = requireInstanceGroup(cluster, mod.instanceGroupId());
+ if (mod.instanceCount() != null) {
+ if (mod.instanceCount() < 0) {
+ throw invalid("InstanceCount must not be negative.");
+ }
+ if ("MASTER".equals(group.getInstanceGroupType())
+ && mod.instanceCount() != group.getRequestedInstanceCount()) {
+ throw invalid("The instance count of a master instance group cannot be modified.");
+ }
}
+ groups.add(group);
}
- groups.add(group);
- }
- for (int i = 0; i < modifications.size(); i++) {
- InstanceGroupModification mod = modifications.get(i);
- EmrInstanceGroup group = groups.get(i);
- if (mod.instanceCount() != null) {
- group.setRequestedInstanceCount(mod.instanceCount());
- group.setRunningInstanceCount(mod.instanceCount());
- }
- if (mod.configurations() != null) {
- group.setConfigurations(mod.configurations());
- group.setConfigurationsVersion(group.getConfigurationsVersion() + 1);
+ for (int i = 0; i < modifications.size(); i++) {
+ InstanceGroupModification mod = modifications.get(i);
+ EmrInstanceGroup group = groups.get(i);
+ if (mod.instanceCount() != null) {
+ group.setRequestedInstanceCount(mod.instanceCount());
+ group.setRunningInstanceCount(mod.instanceCount());
+ }
+ if (mod.configurations() != null) {
+ group.setConfigurations(mod.configurations());
+ group.setConfigurationsVersion(group.getConfigurationsVersion() + 1);
+ }
}
- }
- touched.forEach(clusterStore::put);
+ touched.forEach(clusterStore::put);
+ });
}
public List<EmrInstanceGroup> listInstanceGroups(String clusterId) {
- return requireCluster(clusterId).getInstanceGroups();
- }
-
- public synchronized String addInstanceFleet(String clusterId, EmrInstanceFleet fleet) {
- EmrCluster cluster = requireCluster(clusterId);
- rejectTerminatedCluster(cluster);
- fleet.setId("if-" + randomId(13));
- fleet.setProvisionedOnDemandCapacity(fleet.getTargetOnDemandCapacity());
- fleet.setProvisionedSpotCapacity(fleet.getTargetSpotCapacity());
- fleet.setState("RUNNING");
- cluster.getInstanceFleets().add(fleet);
- clusterStore.put(clusterId, cluster);
- return fleet.getId();
+ return call(() -> snapshotOf(requireCluster(clusterId)).getInstanceGroups());
+ }
+
+ public String addInstanceFleet(String clusterId, EmrInstanceFleet fleet) {
+ return call(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ rejectShutDownCluster(cluster, EmrService::jobFlowShutDown);
+ fleet.setId("if-" + randomId(13));
+ fleet.setProvisionedOnDemandCapacity(fleet.getTargetOnDemandCapacity());
+ fleet.setProvisionedSpotCapacity(fleet.getTargetSpotCapacity());
+ fleet.setState("RUNNING");
+ cluster.getInstanceFleets().add(fleet);
+ clusterStore.put(clusterId, cluster);
+ return fleet.getId();
+ });
}
/** Resizes an instance fleet; a null target leaves that capacity as it is. */
- public synchronized void modifyInstanceFleet(String clusterId, String instanceFleetId,
- Integer targetOnDemandCapacity, Integer targetSpotCapacity) {
- EmrCluster cluster = requireCluster(clusterId);
- if (instanceFleetId == null) {
- throw invalid("InstanceFleetId is required.");
- }
- // AWS's answer for a cluster terminated a while before, ErrorCode member included
- // (hashicorp/terraform-provider-aws#31375, 2023).
- if ("TERMINATED".equals(cluster.getState())) {
- throw new AwsException("InvalidRequestException", JOB_FLOW_SHUT_DOWN, 400,
- Map.of("ErrorCode", "INVALID_JOB_SHUT_DOWN"));
- }
- EmrInstanceFleet fleet = cluster.getInstanceFleets().stream()
- .filter(f -> instanceFleetId.equals(f.getId()))
- .findFirst()
- .orElseThrow(() -> invalid("Instance fleet " + instanceFleetId + " is not in cluster " + clusterId + "."));
- int onDemand = targetOnDemandCapacity != null ? targetOnDemandCapacity : fleet.getTargetOnDemandCapacity();
- int spot = targetSpotCapacity != null ? targetSpotCapacity : fleet.getTargetSpotCapacity();
- if (onDemand < 0 || spot < 0) {
- throw invalid("Target capacities must not be negative.");
- }
- if ("MASTER".equals(fleet.getInstanceFleetType())
- && (onDemand != fleet.getTargetOnDemandCapacity() || spot != fleet.getTargetSpotCapacity())) {
- throw invalid("The target capacity of a master instance fleet cannot be modified.");
- }
- fleet.setTargetOnDemandCapacity(onDemand);
- fleet.setTargetSpotCapacity(spot);
- fleet.setProvisionedOnDemandCapacity(onDemand);
- fleet.setProvisionedSpotCapacity(spot);
- clusterStore.put(clusterId, cluster);
+ public void modifyInstanceFleet(String clusterId, String instanceFleetId,
+ Integer targetOnDemandCapacity, Integer targetSpotCapacity) {
+ run(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ if (instanceFleetId == null) {
+ throw invalid("InstanceFleetId is required.");
+ }
+ // AWS's answer for a cluster terminated a while before, ErrorCode member included
+ // (hashicorp/terraform-provider-aws#31375, 2023); its wording covers one shutting down.
+ if (isShutDown(cluster)) {
+ throw jobFlowShutDown();
+ }
+ // And for one in any other state but running or waiting, a STARTING or BOOTSTRAPPING one
+ // included, ErrorCode member too (hashicorp/terraform-provider-aws#15548, 2020).
+ if (!runsOrWaits(cluster)) {
+ throw new AwsException("InvalidRequestException",
+ "An instance fleet may only be modified when the cluster is running or waiting.", 400,
+ Map.of("ErrorCode", "MODIFY_INSTANCE_FLEET_OF_RUNNING_JOBFLOW_ONLY"));
+ }
+ EmrInstanceFleet fleet = cluster.getInstanceFleets().stream()
+ .filter(f -> instanceFleetId.equals(f.getId()))
+ .findFirst()
+ .orElseThrow(() -> invalid("Instance fleet " + instanceFleetId + " is not in cluster "
+ + clusterId + "."));
+ int onDemand = targetOnDemandCapacity != null ? targetOnDemandCapacity : fleet.getTargetOnDemandCapacity();
+ int spot = targetSpotCapacity != null ? targetSpotCapacity : fleet.getTargetSpotCapacity();
+ if (onDemand < 0 || spot < 0) {
+ throw invalid("Target capacities must not be negative.");
+ }
+ if ("MASTER".equals(fleet.getInstanceFleetType())
+ && (onDemand != fleet.getTargetOnDemandCapacity() || spot != fleet.getTargetSpotCapacity())) {
+ throw invalid("The target capacity of a master instance fleet cannot be modified.");
+ }
+ fleet.setTargetOnDemandCapacity(onDemand);
+ fleet.setTargetSpotCapacity(spot);
+ fleet.setProvisionedOnDemandCapacity(onDemand);
+ fleet.setProvisionedSpotCapacity(spot);
+ clusterStore.put(clusterId, cluster);
+ });
}
/** The cluster's bootstrap actions as given to RunJobFlow, raw JSON; null when it had none. */
public String listBootstrapActions(String clusterId) {
- return requireCluster(clusterId).getBootstrapActions();
+ return call(() -> requireCluster(clusterId).getBootstrapActions());
}
public List<EmrInstanceFleet> listInstanceFleets(String clusterId) {
- return requireCluster(clusterId).getInstanceFleets();
+ return call(() -> snapshotOf(requireCluster(clusterId)).getInstanceFleets());
}
public EmrCluster getClusterForInstances(String clusterId) {
- return requireCluster(clusterId);
+ return call(() -> snapshotOf(requireCluster(clusterId)));
}
// ──────────────────────────── Security configurations ────────────────────────────
@@ -585,73 +659,90 @@ public class EmrService {
// ──────────────────────────── Tags ────────────────────────────
- public synchronized void addTags(String resourceId, Map<String, String> tags) {
- EmrCluster cluster = requireCluster(resourceId);
- cluster.getTags().putAll(tags);
- clusterStore.put(resourceId, cluster);
+ public void addTags(String resourceId, Map<String, String> tags) {
+ run(() -> {
+ EmrCluster cluster = requireCluster(resourceId);
+ cluster.getTags().putAll(tags);
+ clusterStore.put(resourceId, cluster);
+ });
}
- public synchronized void removeTags(String resourceId, List<String> keys) {
- EmrCluster cluster = requireCluster(resourceId);
- keys.forEach(cluster.getTags()::remove);
- clusterStore.put(resourceId, cluster);
+ public void removeTags(String resourceId, List<String> keys) {
+ run(() -> {
+ EmrCluster cluster = requireCluster(resourceId);
+ keys.forEach(cluster.getTags()::remove);
+ clusterStore.put(resourceId, cluster);
+ });
}
// ──────────────────────────── Helpers ────────────────────────────
// ──────────────────────────── Scaling and termination policies ────────────────────────────
- public synchronized void putManagedScalingPolicy(String clusterId, JsonNode policy) {
- EmrCluster cluster = requireCluster(clusterId);
- validateManagedScalingPolicy(policy);
- cluster.setManagedScalingPolicy(policy.toString());
- clusterStore.put(clusterId, cluster);
+ public void putManagedScalingPolicy(String clusterId, JsonNode policy) {
+ run(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ validateManagedScalingPolicy(policy);
+ cluster.setManagedScalingPolicy(policy.toString());
+ clusterStore.put(clusterId, cluster);
+ });
}
/** The cluster's managed scaling policy as raw JSON, or null when it has none. */
public String getManagedScalingPolicy(String clusterId) {
- return requireCluster(clusterId).getManagedScalingPolicy();
+ return call(() -> requireCluster(clusterId).getManagedScalingPolicy());
}
- public synchronized void removeManagedScalingPolicy(String clusterId) {
- EmrCluster cluster = requireCluster(clusterId);
- cluster.setManagedScalingPolicy(null);
- clusterStore.put(clusterId, cluster);
+ public void removeManagedScalingPolicy(String clusterId) {
+ run(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ cluster.setManagedScalingPolicy(null);
+ clusterStore.put(clusterId, cluster);
+ });
}
/** AutoTerminationPolicy is optional in the request; an absent one is stored as an empty policy. */
- public synchronized void putAutoTerminationPolicy(String clusterId, JsonNode policy) {
- EmrCluster cluster = requireCluster(clusterId);
- validateAutoTerminationPolicy(policy);
- cluster.setAutoTerminationPolicy(policy == null || policy.isMissingNode() || policy.isNull() ? "{}" : policy.toString());
- clusterStore.put(clusterId, cluster);
+ public void putAutoTerminationPolicy(String clusterId, JsonNode policy) {
+ run(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ validateAutoTerminationPolicy(policy);
+ cluster.setAutoTerminationPolicy(
+ policy == null || policy.isMissingNode() || policy.isNull() ? "{}" : policy.toString());
+ clusterStore.put(clusterId, cluster);
+ });
}
/** The cluster's auto termination policy as raw JSON, or null when it has none. */
public String getAutoTerminationPolicy(String clusterId) {
- return requireCluster(clusterId).getAutoTerminationPolicy();
+ return call(() -> requireCluster(clusterId).getAutoTerminationPolicy());
}
- public synchronized void removeAutoTerminationPolicy(String clusterId) {
- EmrCluster cluster = requireCluster(clusterId);
- cluster.setAutoTerminationPolicy(null);
- clusterStore.put(clusterId, cluster);
+ public void removeAutoTerminationPolicy(String clusterId) {
+ run(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ cluster.setAutoTerminationPolicy(null);
+ clusterStore.put(clusterId, cluster);
+ });
}
/** Attaches an automatic scaling policy to a core or task instance group and returns the group. */
- public synchronized EmrInstanceGroup putAutoScalingPolicy(String clusterId, String instanceGroupId, JsonNode policy) {
- EmrCluster cluster = requireCluster(clusterId);
- EmrInstanceGroup group = requireInstanceGroup(cluster, instanceGroupId);
- validateAutoScalingPolicy(policy, group.getInstanceGroupType());
- group.setAutoScalingPolicy(policy.toString());
- clusterStore.put(clusterId, cluster);
- return group;
+ public EmrInstanceGroup putAutoScalingPolicy(String clusterId, String instanceGroupId, JsonNode policy) {
+ return call(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ EmrInstanceGroup group = requireInstanceGroup(cluster, instanceGroupId);
+ validateAutoScalingPolicy(policy, group.getInstanceGroupType());
+ group.setAutoScalingPolicy(policy.toString());
+ clusterStore.put(clusterId, cluster);
+ return group;
+ });
}
- public synchronized void removeAutoScalingPolicy(String clusterId, String instanceGroupId) {
- EmrCluster cluster = requireCluster(clusterId);
- requireInstanceGroup(cluster, instanceGroupId).setAutoScalingPolicy(null);
- clusterStore.put(clusterId, cluster);
+ public void removeAutoScalingPolicy(String clusterId, String instanceGroupId) {
+ run(() -> {
+ EmrCluster cluster = requireCluster(clusterId);
+ requireInstanceGroup(cluster, instanceGroupId).setAutoScalingPolicy(null);
+ clusterStore.put(clusterId, cluster);
+ });
}
// ──────────────────────────── Block public access ────────────────────────────
@@ -811,6 +902,7 @@ public class EmrService {
.filter(c -> region.equals(c.getRegion()))
.filter(c -> c.getInstanceGroups().stream().anyMatch(g -> instanceGroupId.equals(g.getId())))
.findFirst()
+ .map(this::settle)
.orElseThrow(() -> invalid("Instance group " + instanceGroupId + " is not valid."));
}
@@ -818,8 +910,30 @@ public class EmrService {
return new AwsException("InvalidRequestException", message, 400);
}
- private static void rejectTerminatedCluster(EmrCluster cluster) {
- if ("TERMINATED".equals(cluster.getState())) {
+ /** ModifyInstanceFleet's answer for a job flow that is shutting down or ended, ErrorCode member included. */
+ private static AwsException jobFlowShutDown() {
+ return new AwsException("InvalidRequestException", JOB_FLOW_SHUT_DOWN, 400,
+ Map.of("ErrorCode", "INVALID_JOB_SHUT_DOWN"));
+ }
+
+ /**
+ * AddJobFlowSteps' answer for a job flow that is shutting down or ended. AddInstanceGroups gives it too:
+ * like AddJobFlowSteps, its model declares no InvalidRequestException.
+ */
+ private static AwsException jobFlowShutDownValidation() {
+ return new AwsException("ValidationException", JOB_FLOW_SHUT_DOWN, 400);
+ }
+
+ /**
+ * AddInstanceGroups and AddInstanceFleet on a cluster that is shutting down or ended. A TERMINATING one
+ * gets {@code shuttingDown}, the operation's answer with AWS's wording, which covers it; a TERMINATED
+ * one keeps Floci's own, since no AWS answer to either operation is recorded.
+ */
+ private static void rejectShutDownCluster(EmrCluster cluster, Supplier<AwsException> shuttingDown) {
+ if (EmrClusterLifecycle.TERMINATING.equals(cluster.getState())) {
+ throw shuttingDown.get();
+ }
+ if (EmrClusterLifecycle.TERMINATED.equals(cluster.getState())) {
throw invalid("Cluster " + cluster.getId() + " is terminated.");
}
}
@@ -848,7 +962,8 @@ public class EmrService {
*/
private Optional<EmrCluster> findCluster(String id) {
String region = regionResolver.getRegion();
- return clusterStore.get(id).filter(c -> c.getRegion() == null || c.getRegion().equals(region));
+ return clusterStore.get(id).filter(c -> c.getRegion() == null || c.getRegion().equals(region))
+ .map(this::settle);
}
private void mutateClusters(List<String> ids, java.util.function.Consumer<EmrCluster> mutation) {
@@ -860,68 +975,105 @@ public class EmrService {
}
}
- private void advanceToWaiting(EmrCluster cluster) {
- cluster.setState("WAITING");
- cluster.setStateChangeReasonCode(cluster.getSteps().isEmpty() ? null : "ALL_STEPS_COMPLETED");
- cluster.setStateChangeMessage("Cluster ready to run steps.");
- cluster.setReadyDateTime(Instant.now());
+ /** Shutting down or ended: "A job flow that is shutting down, terminated, or finished may not be modified." */
+ private static boolean isShutDown(EmrCluster cluster) {
+ return EmrClusterLifecycle.TERMINATING.equals(cluster.getState())
+ || EmrClusterLifecycle.TERMINATED.equals(cluster.getState());
+ }
+
+ /** The only states in which a cluster's instance groups and fleets may be modified. */
+ private static boolean runsOrWaits(EmrCluster cluster) {
+ return EmrClusterLifecycle.RUNNING.equals(cluster.getState())
+ || EmrClusterLifecycle.WAITING.equals(cluster.getState());
}
/**
- * Runs a request's steps as the Management Guide describes ("Running steps to process data"): all
- * are set PENDING, then each completes before the next starts. Here they complete at once. Each
- * change queues its event as it is made, so the events follow the steps' recorded times.
+ * Runs {@code body} holding the service, then publishes the events it queued once it no longer
+ * does, so a rule's target that calls EMR never waits on the call that made the change. Every
+ * call that reads a cluster runs this way, since reading one brings it up to date.
*/
- private void runSteps(EmrCluster cluster, List<EmrStep> steps, String region) {
- for (EmrStep step : steps) {
- initStep(step);
- eventPublisher.queue(eventPublisher.stepStatusChange(cluster, step, "PENDING"), region);
- }
- for (EmrStep step : steps) {
- completeStep(step);
- eventPublisher.queue(eventPublisher.stepStatusChange(cluster, step, "COMPLETED"), region);
+ private <T> T call(Supplier<T> body) {
+ try {
+ synchronized (this) {
+ return body.get();
+ }
+ } finally {
+ eventPublisher.publishQueued();
}
}
- private void terminateAfterCompletedSteps(EmrCluster cluster) {
- if (cluster.getSteps().isEmpty() || !cluster.isAutoTerminate()
- || "TERMINATED".equals(cluster.getState())) {
- return;
- }
- cluster.setState("TERMINATED");
- releaseInstanceCapacity(cluster);
- cluster.setStateChangeReasonCode("ALL_STEPS_COMPLETED");
- cluster.setStateChangeMessage("All steps completed.");
- cluster.setEndDateTime(Instant.now());
+ private void run(Runnable body) {
+ call(() -> {
+ body.run();
+ return null;
+ });
+ }
+
+ /**
+ * A copy of the cluster for a call to answer from: the timer goes on changing the stored one, and an
+ * answer built from it after the service is let go could mix two of its states.
+ */
+ private static EmrCluster snapshotOf(EmrCluster cluster) {
+ return SNAPSHOT_MAPPER.convertValue(cluster, EmrCluster.class);
}
- private void releaseInstanceCapacity(EmrCluster cluster) {
- for (EmrInstanceGroup group : cluster.getInstanceGroups()) {
- group.setState("TERMINATED");
- group.setTerminatedInstanceCount(group.getRunningInstanceCount());
- group.setRunningInstanceCount(0);
+ /** The cluster as it stands now: every change due by now made, stored if there was one. */
+ private EmrCluster settle(EmrCluster cluster) {
+ if (lifecycle.advance(cluster, now(), regionOf(cluster))) {
+ clusterStore.put(cluster.getId(), cluster);
}
- for (EmrInstanceFleet fleet : cluster.getInstanceFleets()) {
- fleet.setState("TERMINATED");
- fleet.setProvisionedOnDemandCapacity(0);
- fleet.setProvisionedSpotCapacity(0);
+ wakeForNextChange(cluster);
+ return cluster;
+ }
+
+ /** Stores a cluster a call changed, once every change that falls due by now is made. */
+ private void store(EmrCluster cluster) {
+ lifecycle.advance(cluster, now(), regionOf(cluster));
+ clusterStore.put(cluster.getId(), cluster);
+ wakeForNextChange(cluster);
+ }
+
+ private void wakeForNextChange(EmrCluster cluster) {
+ Instant due = lifecycle.nextChange(cluster);
+ if (due != null) {
+ wakeAt(regionResolver.getAccountId(), cluster, due);
}
}
- private void initStep(EmrStep step) {
- step.setId("s-" + randomId(13));
- step.setState("PENDING");
- step.setCreationDateTime(Instant.now());
- if (step.getActionOnFailure() == null) {
- step.setActionOnFailure("TERMINATE_CLUSTER");
+ /**
+ * Brings the cluster up to date at {@code due}, as {@code accountId} in its Region, unless an
+ * earlier wakeup will: that one sets the next. Its events go out by {@link Wakeups#deliver}.
+ */
+ private void wakeAt(String accountId, EmrCluster cluster, Instant due) {
+ String key = accountId + "/" + cluster.getId();
+ Instant set = wakeupsDue.get(key);
+ if (set != null && !set.isAfter(due)) {
+ return;
}
+ wakeupsDue.put(key, due);
+ String id = cluster.getId();
+ String region = regionOf(cluster);
+ wakeups.at(due, () -> {
+ wakeupsDue.remove(key, due);
+ try {
+ RequestScopes.runAs(accountId, region, () -> {
+ synchronized (this) {
+ findCluster(id);
+ }
+ });
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "EMR could not bring cluster {0} up to date", id);
+ }
+ wakeups.deliver(eventPublisher::publishQueued);
+ });
+ }
+
+ private String regionOf(EmrCluster cluster) {
+ return cluster.getRegion() != null ? cluster.getRegion() : regionResolver.getRegion();
}
- private void completeStep(EmrStep step) {
- Instant now = Instant.now();
- step.setStartDateTime(now);
- step.setEndDateTime(now);
- step.setState("COMPLETED");
+ private Instant now() {
+ return clock.instant().truncatedTo(ChronoUnit.MILLIS);
}
private String randomId(int length) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java b/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java
index a44f58de9c..c7c34fee6a 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java
@@ -10,7 +10,8 @@ import java.util.Map;
/**
* An EMR cluster (job flow). State machine:
- * STARTING → BOOTSTRAPPING → RUNNING → WAITING, then → TERMINATING → TERMINATED.
+ * STARTING → BOOTSTRAPPING → RUNNING → WAITING (and back to RUNNING for added steps), then
+ * → TERMINATING → TERMINATED.
* Steps, instance groups and fleets are stored as nested lists. Complex sub-structures
* (ec2 attributes, applications, configurations) are stored as raw JSON strings.
*/
@@ -53,6 +54,8 @@ public class EmrCluster {
private Instant creationDateTime;
private Instant readyDateTime;
private Instant endDateTime;
+ // When the cluster began TERMINATING; Floci's own, as the API's Timeline has no such time.
+ private Instant terminatingDateTime;
private Map<String, String> tags = new LinkedHashMap<>();
private List<EmrStep> steps = new ArrayList<>();
private List<EmrInstanceGroup> instanceGroups = new ArrayList<>();
@@ -180,6 +183,9 @@ public class EmrCluster {
public Instant getEndDateTime() { return endDateTime; }
public void setEndDateTime(Instant endDateTime) { this.endDateTime = endDateTime; }
+ public Instant getTerminatingDateTime() { return terminatingDateTime; }
+ public void setTerminatingDateTime(Instant terminatingDateTime) { this.terminatingDateTime = terminatingDateTime; }
+
public Map<String, String> getTags() { return tags; }
public void setTags(Map<String, String> tags) { this.tags = tags; }
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 492685bc89..ddc18f666a 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -578,6 +578,9 @@ floci:
enabled: true
emr:
enabled: true
+ cluster-startup-delay-seconds: 0 # 0 = a cluster is ready as soon as it is created
+ step-duration-seconds: 0 # 0 = a step completes as soon as it starts
+ cluster-termination-delay-seconds: 0 # 0 = a cluster is TERMINATED as soon as it is TERMINATING
wafv2:
enabled: true
scheduler:
diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventPublisherTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventPublisherTest.java
index 6e0f2c4e48..4244ba9793 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventPublisherTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventPublisherTest.java
@@ -1,15 +1,9 @@
package io.github.hectorvent.floci.services.emr;
import com.fasterxml.jackson.databind.JsonNode;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.AwsException;
-import io.github.hectorvent.floci.core.common.RegionResolver;
import io.github.hectorvent.floci.services.emr.model.EmrCluster;
import io.github.hectorvent.floci.services.emr.model.EmrStep;
-import io.github.hectorvent.floci.services.eventbridge.EventBridgeService;
-import io.github.hectorvent.floci.testing.InMemoryStorageFactory;
-import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.time.Instant;
@@ -19,69 +13,37 @@ import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicReference;
+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.REGION;
+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.cluster;
+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.step;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
-import static org.mockito.ArgumentMatchers.anyString;
-import static org.mockito.ArgumentMatchers.eq;
-import static org.mockito.ArgumentMatchers.isNull;
-import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
-import static org.mockito.Mockito.doAnswer;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
/**
* The {@code aws.emr} events {@link EmrService} publishes, in the order it publishes them, each one
- * checked against the Management Guide's detail and its tables' severity and message.
+ * checked against the Management Guide's detail and its tables' severity and message. The durations
+ * are 0, as by default; {@link EmrServiceTest} covers the timed lifecycle.
*/
class EmrEventPublisherTest {
- private static final String REGION = "us-east-1";
- private static final String ACCOUNT = "000000000000";
private static final DateTimeFormatter MINUTE =
DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm 'UTC'").withZone(ZoneOffset.UTC);
- private final ObjectMapper objectMapper = new ObjectMapper();
- private final List<JsonNode> published = new ArrayList<>();
- private Target target = event -> { };
- private EmrEventPublisher publisher;
- private EmrService service;
-
- /** A rule's target, given each event as it is published, as EventBridge's first delivery is. */
- @FunctionalInterface
- private interface Target {
- void deliver(JsonNode event) throws Exception;
- }
-
- @BeforeEach
- void setUp() {
- EventBridgeService eventBridge = mock(EventBridgeService.class);
- doAnswer(invocation -> {
- // The call that made the change has let go of the service by now.
- assertFalse(Thread.holdsLock(service), "published while EmrService was held");
- JsonNode event = objectMapper.createObjectNode()
- .put("detail-type", invocation.<String>getArgument(1))
- .set("detail", objectMapper.readTree(invocation.<String>getArgument(2)));
- published.add(event);
- target.deliver(event);
- return null;
- }).when(eventBridge).publishServiceEvent(eq("aws.emr"), anyString(), anyString(), isNull(), eq(REGION));
- EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
- when(config.services().emr().defaultReleaseLabel()).thenReturn("emr-7.5.0");
- publisher = new EmrEventPublisher(eventBridge, objectMapper, new RegionResolver(REGION, ACCOUNT));
- service = new EmrService(new InMemoryStorageFactory(ACCOUNT), config, new RegionResolver(REGION, ACCOUNT),
- publisher);
- }
+ private final EmrServiceFixture fixture = new EmrServiceFixture();
+ private final EmrService service = fixture.service;
+ private final EmrEventPublisher publisher = fixture.publisher;
+ private final List<JsonNode> published = fixture.published;
@Test
- void aClusterThatRunsItsStepsAndWaitsIsStartedAddsAndCompletesThemAndWaits() {
+ void aClusterThatRunsItsStepsAndWaitsIsStartedAddsRunsAndCompletesThemAndWaits() {
EmrCluster cluster = service.runJobFlow(cluster("nightly", true, step("load"), step("report")), REGION);
EmrStep load = cluster.getSteps().get(0);
EmrStep report = cluster.getSteps().get(1);
- assertEquals(List.of("STARTING", "PENDING load", "PENDING report", "COMPLETED load", "COMPLETED report",
- "WAITING"), states());
+ assertEquals(List.of("STARTING", "PENDING load", "PENDING report", "RUNNING load", "COMPLETED load",
+ "RUNNING report", "COMPLETED report", "WAITING"), fixture.states());
// As the events tell it: every step was added before the first one ran.
assertFalse(report.getCreationDateTime().isAfter(load.getStartDateTime()));
assertCluster(published.get(0), cluster, "STARTING", "INFO", "{\"code\":\"\"}",
@@ -90,22 +52,25 @@ class EmrEventPublisherTest {
assertStep(published.get(1), cluster, load, "PENDING", "INFO",
"Step " + load.getId() + " (load) was added to Amazon EMR cluster " + cluster.getId()
+ " (nightly) at " + minute(load.getCreationDateTime()) + " and is pending execution.");
- assertStep(published.get(3), cluster, load, "COMPLETED", "INFO",
+ assertStep(published.get(3), cluster, load, "RUNNING", "INFO",
+ "Step " + load.getId() + " (load) in Amazon EMR cluster " + cluster.getId()
+ + " (nightly) started running at " + minute(load.getStartDateTime()) + ".");
+ assertStep(published.get(4), cluster, load, "COMPLETED", "INFO",
"Step " + load.getId() + " (load) in Amazon EMR cluster " + cluster.getId()
+ " (nightly) completed execution at " + minute(load.getEndDateTime())
+ ". The step started running at " + minute(load.getStartDateTime())
+ " and took 0 minutes to complete.");
- assertEquals(report.getId(), published.get(4).at("/detail/stepId").asText());
- assertCluster(published.get(5), cluster, "WAITING", "INFO", "{\"code\":\"\"}",
+ assertEquals(report.getId(), published.get(6).at("/detail/stepId").asText());
+ assertCluster(published.get(7), cluster, "WAITING", "INFO", "{\"code\":\"\"}",
"Amazon EMR cluster " + cluster.getId() + " (nightly) finished running all pending steps at "
- + minute(cluster.getReadyDateTime()) + ".");
+ + minute(report.getEndDateTime()) + ".");
}
@Test
void aClusterWithoutStepsIsReadyForUse() {
EmrCluster cluster = service.runJobFlow(cluster("idle", true), REGION);
- assertEquals(List.of("STARTING", "WAITING"), states());
+ assertEquals(List.of("STARTING", "WAITING"), fixture.states());
assertEquals("Amazon EMR cluster " + cluster.getId() + " (idle) was created at "
+ minute(cluster.getReadyDateTime()) + " and is ready for use.",
published.get(1).at("/detail/message").asText());
@@ -115,21 +80,25 @@ class EmrEventPublisherTest {
void aTransientClusterTerminatesAfterItsStepsWithoutWaiting() {
EmrCluster cluster = service.runJobFlow(cluster("transient", false, step("only")), REGION);
- assertEquals(List.of("STARTING", "PENDING only", "COMPLETED only", "TERMINATED"), states());
- assertCluster(published.get(3), cluster, "TERMINATED", "INFO",
+ assertEquals(List.of("STARTING", "PENDING only", "RUNNING only", "COMPLETED only", "TERMINATED"),
+ fixture.states());
+ assertCluster(published.get(4), cluster, "TERMINATED", "INFO",
"{\"code\":\"ALL_STEPS_COMPLETED\",\"message\":\"All steps completed.\"}",
"Amazon EMR Cluster " + cluster.getId() + " (transient) has terminated at "
+ minute(cluster.getEndDateTime()) + " with a reason of ALL_STEPS_COMPLETED.");
}
@Test
- void stepsAddedToAWaitingClusterLeaveItWaiting() {
+ void stepsAddedToAWaitingClusterRunAndLeaveItWaitingAgain() {
EmrCluster cluster = service.runJobFlow(cluster("kept", true), REGION);
published.clear();
- service.addJobFlowSteps(cluster.getId(), List.of(step("late")));
+ EmrStep late = step("late");
+ service.addJobFlowSteps(cluster.getId(), List.of(late));
- assertEquals(List.of("PENDING late", "COMPLETED late"), states());
+ assertEquals(List.of("PENDING late", "RUNNING late", "COMPLETED late", "WAITING"), fixture.states());
+ assertEquals("Amazon EMR cluster " + cluster.getId() + " (kept) finished running all pending steps at "
+ + minute(late.getEndDateTime()) + ".", published.get(3).at("/detail/message").asText());
}
@Test
@@ -139,9 +108,9 @@ class EmrEventPublisherTest {
service.addJobFlowSteps(cluster.getId(), List.of(step("late")));
- assertEquals(List.of("PENDING late", "COMPLETED late", "TERMINATED"), states());
+ assertEquals(List.of("PENDING late", "RUNNING late", "COMPLETED late", "TERMINATED"), fixture.states());
assertEquals("{\"code\":\"ALL_STEPS_COMPLETED\",\"message\":\"All steps completed.\"}",
- published.get(2).at("/detail/stateChangeReason").asText());
+ published.get(3).at("/detail/stateChangeReason").asText());
}
@Test
@@ -157,7 +126,7 @@ class EmrEventPublisherTest {
List<String> ids = List.of(ended.getId(), gone.getId(), kept.getId());
assertThrows(AwsException.class, () -> service.terminateJobFlows(ids));
- assertEquals(List.of("TERMINATED"), states());
+ assertEquals(List.of("TERMINATED"), fixture.states());
assertCluster(published.get(0), service.describeCluster(ended.getId()), "TERMINATED", "INFO",
"{\"code\":\"USER_REQUEST\",\"message\":\"Terminated by user request\"}",
"Amazon EMR Cluster " + ended.getId() + " (ended) has terminated at "
@@ -174,12 +143,12 @@ class EmrEventPublisherTest {
assertThrows(AwsException.class, () -> service.addJobFlowSteps(cluster.getId(), List.of(step("late"))));
assertThrows(AwsException.class, () -> service.runJobFlow(new EmrCluster(), REGION));
- assertEquals(List.of(), states());
+ assertEquals(List.of(), fixture.states());
}
@Test
void aTargetThatEndsTheClusterOnItsStartingEventHearsOfItAfterTheRestOfTheCall() {
- target = event -> {
+ fixture.target = event -> {
if ("STARTING".equals(event.at("/detail/state").asText())) {
service.terminateJobFlows(List.of(event.at("/detail/clusterId").asText()));
}
@@ -187,13 +156,13 @@ class EmrEventPublisherTest {
service.runJobFlow(cluster("answered", true), REGION);
- assertEquals(List.of("STARTING", "WAITING", "TERMINATED"), states());
+ assertEquals(List.of("STARTING", "WAITING", "TERMINATED"), fixture.states());
}
@Test
void aTargetCallingBackFromAnotherThreadHearsOfItAfterTheRestOfTheCall() {
AtomicReference<Throwable> failure = new AtomicReference<>();
- target = event -> {
+ fixture.target = event -> {
if ("STARTING".equals(event.at("/detail/state").asText())) {
// As an API destination would, from the thread serving its request, while this one waits.
String clusterId = event.at("/detail/clusterId").asText();
@@ -212,7 +181,7 @@ class EmrEventPublisherTest {
service.runJobFlow(cluster("answered", true), REGION);
assertNull(failure.get());
- assertEquals(List.of("STARTING", "WAITING", "TERMINATED"), states());
+ assertEquals(List.of("STARTING", "WAITING", "TERMINATED"), fixture.states());
}
@Test
@@ -220,9 +189,9 @@ class EmrEventPublisherTest {
EmrCluster cluster = service.runJobFlow(cluster("failing", true), REGION);
cluster.setStateChangeReasonCode("STEP_FAILURE");
cluster.setStateChangeMessage("Shut down as step failed");
- cluster.setEndDateTime(Instant.parse("2026-10-24T09:41:30Z"));
- JsonNode detail = objectMapper.readTree(publisher.clusterStateChange(cluster, "TERMINATED").detail());
+ JsonNode detail = fixture.objectMapper.readTree(
+ publisher.clusterStateChange(cluster, "TERMINATED", Instant.parse("2026-10-24T09:41:30Z")).detail());
assertEquals("CRITICAL", detail.path("severity").asText());
assertEquals("Amazon EMR Cluster " + cluster.getId() + " (failing) has terminated at 2026-10-24 09:41 UTC"
@@ -238,7 +207,7 @@ class EmrEventPublisherTest {
EmrEventPublisher.Event event = publisher.stepStatusChange(cluster, step, "CANCELLED");
assertEquals("EMR Step Status Change", event.detailType());
- JsonNode detail = objectMapper.readTree(event.detail());
+ JsonNode detail = fixture.objectMapper.readTree(event.detail());
assertEquals("WARN", detail.path("severity").asText());
assertEquals("Cancellation request has succeeded for cluster step " + step.getId() + " (doomed) in Amazon EMR"
+ " cluster " + cluster.getId() + " (cancelling) at 2026-10-24 09:41 UTC, and the step is now"
@@ -274,16 +243,6 @@ class EmrEventPublisherTest {
assertEquals(message, detail.path("message").asText());
}
- /** Each event's state, a step's followed by its name. */
- private List<String> states() {
- return published.stream().map(event -> {
- JsonNode detail = event.path("detail");
- return detail.has("stepId")
- ? detail.path("state").asText() + " " + detail.path("name").asText()
- : detail.path("state").asText();
- }).toList();
- }
-
private static List<String> fieldNames(JsonNode node) {
List<String> names = new ArrayList<>();
node.fieldNames().forEachRemaining(names::add);
@@ -293,19 +252,4 @@ class EmrEventPublisherTest {
private static String minute(Instant instant) {
return MINUTE.format(instant);
}
-
- private static EmrCluster cluster(String name, boolean keepAlive, EmrStep... steps) {
- EmrCluster cluster = new EmrCluster();
- cluster.setName(name);
- cluster.setKeepJobFlowAliveWhenNoSteps(keepAlive);
- cluster.setSteps(new ArrayList<>(List.of(steps)));
- return cluster;
- }
-
- private static EmrStep step(String name) {
- EmrStep step = new EmrStep();
- step.setName(name);
- step.setActionOnFailure("CONTINUE");
- return step;
- }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventsIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventsIntegrationTest.java
index 21db3427e6..0a141736d3 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventsIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrEventsIntegrationTest.java
@@ -13,9 +13,6 @@ import java.time.Instant;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.List;
-import java.util.Map;
-import java.util.function.Function;
-import java.util.stream.Collectors;
import static io.github.hectorvent.floci.testing.EmrRequests.emr;
import static org.hamcrest.MatcherAssert.assertThat;
@@ -52,24 +49,32 @@ class EmrEventsIntegrationTest {
+ "\"Steps\":[{\"Name\":\"load\",\"ActionOnFailure\":\"CONTINUE\","
+ "\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")
.then().statusCode(200).extract().path("JobFlowId");
- emr(region, "AddJobFlowSteps", "{\"JobFlowId\":\"" + clusterId + "\",\"Steps\":[{\"Name\":\"report\","
- + "\"ActionOnFailure\":\"CONTINUE\",\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")
- .then().statusCode(200);
+ String reportId = emr(region, "AddJobFlowSteps", "{\"JobFlowId\":\"" + clusterId + "\","
+ + "\"Steps\":[{\"Name\":\"report\",\"ActionOnFailure\":\"CONTINUE\","
+ + "\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")
+ .then().statusCode(200).extract().path("StepIds[0]");
+ Instant reportEnded = stepEnd(region, clusterId, reportId);
emr(region, "TerminateJobFlows", "{\"JobFlowIds\":[\"" + clusterId + "\"]}").then().statusCode(200);
- Map<String, JsonNode> events = byState(queue.drain(), region, clusterId);
- assertThat(events.keySet(), containsInAnyOrder("STARTING", "PENDING load", "COMPLETED load", "WAITING",
- "PENDING report", "COMPLETED report", "TERMINATED"));
- JsonNode waiting = events.get("WAITING").path("detail");
- assertEquals("{\"code\":\"\"}", waiting.path("stateChangeReason").asText());
- assertEquals("Amazon EMR cluster " + clusterId + " (kept) finished running all pending steps at "
- + minute(timeline(region, clusterId, "ReadyDateTime")) + ".", waiting.path("message").asText());
- JsonNode completed = events.get("COMPLETED report").path("detail");
+ List<JsonNode> events = checked(queue.drain(), region, clusterId);
+ assertThat(events.stream().map(EmrEventsIntegrationTest::state).toList(), containsInAnyOrder(
+ "STARTING", "PENDING load", "RUNNING load", "COMPLETED load", "WAITING",
+ "PENDING report", "RUNNING report", "COMPLETED report", "WAITING", "TERMINATED"));
+ // Once after the first step, and again after the one added later.
+ List<JsonNode> waiting = withState(events, "WAITING");
+ for (JsonNode event : waiting) {
+ assertEquals("{\"code\":\"\"}", event.path("detail").path("stateChangeReason").asText());
+ }
+ String finished = "Amazon EMR cluster " + clusterId + " (kept) finished running all pending steps at ";
+ assertThat(waiting.stream().map(event -> event.path("detail").path("message").asText()).toList(),
+ containsInAnyOrder(finished + minute(timeline(region, clusterId, "ReadyDateTime")) + ".",
+ finished + minute(reportEnded) + "."));
+ JsonNode completed = withState(events, "COMPLETED report").get(0).path("detail");
assertEquals("INFO", completed.path("severity").asText());
assertEquals("CONTINUE", completed.path("actionOnFailure").asText());
assertTrue(completed.path("message").asText().endsWith(" and took 0 minutes to complete."),
completed.path("message").asText());
- JsonNode terminated = events.get("TERMINATED").path("detail");
+ JsonNode terminated = withState(events, "TERMINATED").get(0).path("detail");
assertEquals("INFO", terminated.path("severity").asText());
assertEquals("{\"code\":\"USER_REQUEST\",\"message\":\"Terminated by user request\"}",
terminated.path("stateChangeReason").asText());
@@ -87,24 +92,23 @@ class EmrEventsIntegrationTest {
+ "\"Steps\":[{\"Name\":\"only\",\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")
.then().statusCode(200).extract().path("JobFlowId");
- Map<String, JsonNode> events = byState(queue.drain(), region, clusterId);
- assertThat(events.keySet(),
- containsInAnyOrder("STARTING", "PENDING only", "COMPLETED only", "TERMINATED"));
- JsonNode terminated = events.get("TERMINATED").path("detail");
+ List<JsonNode> events = checked(queue.drain(), region, clusterId);
+ assertThat(events.stream().map(EmrEventsIntegrationTest::state).toList(), containsInAnyOrder(
+ "STARTING", "PENDING only", "RUNNING only", "COMPLETED only", "TERMINATED"));
+ JsonNode terminated = withState(events, "TERMINATED").get(0).path("detail");
assertEquals("INFO", terminated.path("severity").asText());
assertEquals("{\"code\":\"ALL_STEPS_COMPLETED\",\"message\":\"All steps completed.\"}",
terminated.path("stateChangeReason").asText());
assertEquals("TERMINATE_CLUSTER",
- events.get("PENDING only").path("detail").path("actionOnFailure").asText());
+ withState(events, "PENDING only").get(0).path("detail").path("actionOnFailure").asText());
}
}
/**
- * The events by state, a step's followed by its name, once each is checked to be one of
- * {@code clusterId}'s, sent by EMR to {@code region}'s default bus with no resources. A state seen
- * twice fails, so the keys are every event.
+ * The events, once each is checked to be one of {@code clusterId}'s, sent by EMR to {@code region}'s
+ * default bus with no resources.
*/
- private static Map<String, JsonNode> byState(List<JsonNode> events, String region, String clusterId) {
+ private static List<JsonNode> checked(List<JsonNode> events, String region, String clusterId) {
for (JsonNode event : events) {
assertEquals("aws.emr", event.path("source").asText());
assertEquals(ACCOUNT, event.path("account").asText());
@@ -114,12 +118,19 @@ class EmrEventsIntegrationTest {
assertEquals(event.path("detail").has("stepId") ? "EMR Step Status Change" : "EMR Cluster State Change",
event.path("detail-type").asText());
}
- return events.stream().collect(Collectors.toMap(event -> {
- JsonNode detail = event.path("detail");
- return detail.has("stepId")
- ? detail.path("state").asText() + " " + detail.path("name").asText()
- : detail.path("state").asText();
- }, Function.identity()));
+ return events;
+ }
+
+ /** The event's state, a step's followed by its name. */
+ private static String state(JsonNode event) {
+ JsonNode detail = event.path("detail");
+ return detail.has("stepId")
+ ? detail.path("state").asText() + " " + detail.path("name").asText()
+ : detail.path("state").asText();
+ }
+
+ private static List<JsonNode> withState(List<JsonNode> events, String state) {
+ return events.stream().filter(event -> state.equals(state(event))).toList();
}
private static Instant timeline(String region, String clusterId, String field) {
@@ -129,6 +140,13 @@ class EmrEventsIntegrationTest {
return Instant.ofEpochMilli(seconds.movePointRight(3).longValue());
}
+ private static Instant stepEnd(String region, String clusterId, String stepId) {
+ BigDecimal seconds = emr(region, "DescribeStep", "{\"ClusterId\":\"" + clusterId + "\",\"StepId\":\""
+ + stepId + "\"}").then().statusCode(200).extract().response().jsonPath(DECIMAL_NUMBERS)
+ .get("Step.Status.Timeline.EndDateTime");
+ return Instant.ofEpochMilli(seconds.movePointRight(3).longValue());
+ }
+
private static String minute(Instant instant) {
return MINUTE.format(instant);
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java
index f8380c9cd5..066f0c9e60 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrIntegrationTest.java
@@ -127,22 +127,23 @@ class EmrIntegrationTest {
.then().statusCode(200)
.body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());
- EmrCluster cluster = emrService.describeCluster(clusterId);
+ EmrCluster cluster = storedCluster("floci-emr-pinned");
+ String pinnedId = cluster.getId();
Instant originalCreationTime = cluster.getCreationDateTime();
cluster.setCreationDateTime(Instant.ofEpochMilli(1_761_000_000_000L));
try {
emr("ListClusters", "{\"CreatedBefore\":1761000000.001}")
.then().statusCode(200)
- .body("Clusters.find { it.Id == '" + clusterId + "' }.Name", equalTo("floci-emr"));
+ .body("Clusters.find { it.Id == '" + pinnedId + "' }.Name", equalTo("floci-emr-pinned"));
emr("ListClusters", "{\"CreatedAfter\":1760999999.999}")
.then().statusCode(200)
- .body("Clusters.find { it.Id == '" + clusterId + "' }.Name", equalTo("floci-emr"));
+ .body("Clusters.find { it.Id == '" + pinnedId + "' }.Name", equalTo("floci-emr-pinned"));
emr("ListClusters", "{\"CreatedBefore\":1761000000.000000001}")
.then().statusCode(200)
- .body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());
+ .body("Clusters.find { it.Id == '" + pinnedId + "' }", nullValue());
emr("ListClusters", "{\"CreatedAfter\":1760999999.999999999}")
.then().statusCode(200)
- .body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());
+ .body("Clusters.find { it.Id == '" + pinnedId + "' }", nullValue());
} finally {
cluster.setCreationDateTime(originalCreationTime);
}
@@ -153,7 +154,7 @@ class EmrIntegrationTest {
.then().statusCode(200).extract().asString()).path("Clusters");
JsonNode listedTime = null;
for (JsonNode listed : clusters) {
- if (clusterId.equals(listed.path("Id").asText())) {
+ if (pinnedId.equals(listed.path("Id").asText())) {
listedTime = listed.path("Status").path("Timeline").path("CreationDateTime");
break;
}
@@ -163,7 +164,7 @@ class EmrIntegrationTest {
assertEquals(0, new BigDecimal("1761000000.123").compareTo(listedTime.decimalValue()));
JsonNode describedTime = objectMapper.readTree(emr("DescribeCluster",
- "{\"ClusterId\":\"" + clusterId + "\"}")
+ "{\"ClusterId\":\"" + pinnedId + "\"}")
.then().statusCode(200).extract().asString())
.path("Cluster").path("Status").path("Timeline").path("CreationDateTime");
assertTrue(describedTime.isNumber());
@@ -171,10 +172,10 @@ class EmrIntegrationTest {
emr("ListClusters", "{\"CreatedAfter\":" + listedTime.asText() + "}")
.then().statusCode(200)
- .body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());
+ .body("Clusters.find { it.Id == '" + pinnedId + "' }", nullValue());
emr("ListClusters", "{\"CreatedBefore\":" + listedTime.asText() + "}")
.then().statusCode(200)
- .body("Clusters.find { it.Id == '" + clusterId + "' }", nullValue());
+ .body("Clusters.find { it.Id == '" + pinnedId + "' }", nullValue());
} finally {
cluster.setCreationDateTime(originalCreationTime);
}
@@ -441,14 +442,9 @@ class EmrIntegrationTest {
// Catches: a cluster stored before the DNS fix keeps reporting its old us-east-1 master name.
@Test
void storedClusterReportsTheRegionalMasterDnsName() {
- String id = emr("RunJobFlow",
- "{\"Name\":\"stale-dns\",\"ReleaseLabel\":\"emr-7.5.0\","
- + "\"Instances\":{\"KeepJobFlowAliveWhenNoSteps\":true,"
- + "\"InstanceGroups\":[{\"Name\":\"master\",\"InstanceRole\":\"MASTER\","
- + "\"InstanceType\":\"m5.xlarge\",\"InstanceCount\":1}]}}")
- .then().statusCode(200)
- .extract().path("JobFlowId");
- emrService.describeCluster(id).setMasterPublicDnsName("ip-10-0-0-1.us-east-1.compute.internal");
+ EmrCluster stored = storedCluster("stale-dns");
+ stored.setMasterPublicDnsName("ip-10-0-0-1.us-east-1.compute.internal");
+ String id = stored.getId();
emr("DescribeCluster", "{\"ClusterId\":\"" + id + "\"}")
.then().statusCode(200)
@@ -457,4 +453,15 @@ class EmrIntegrationTest {
emr("TerminateJobFlows", "{\"JobFlowIds\":[\"" + id + "\"]}")
.then().statusCode(200);
}
+
+ /**
+ * A kept-alive cluster as RunJobFlow stores it: the service's own object, which a test can change,
+ * unlike the copy DescribeCluster answers from.
+ */
+ private EmrCluster storedCluster(String name) {
+ EmrCluster cluster = new EmrCluster();
+ cluster.setName(name);
+ cluster.setKeepJobFlowAliveWhenNoSteps(true);
+ return emrService.runJobFlow(cluster, "us-east-1");
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.java
new file mode 100644
index 0000000000..a4d790fc8f
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.java
@@ -0,0 +1,163 @@
+package io.github.hectorvent.floci.services.emr;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.services.emr.model.EmrCluster;
+import io.github.hectorvent.floci.services.emr.model.EmrStep;
+import io.github.hectorvent.floci.services.eventbridge.EventBridgeService;
+import io.github.hectorvent.floci.testing.InMemoryStorageFactory;
+import io.github.hectorvent.floci.testing.MutableClock;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.ArgumentMatchers.isNull;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * An {@link EmrService} on in-memory storage, with a clock the test moves and wakeups the test runs,
+ * recording the {@code aws.emr} events it publishes. Each event is handed to {@link #target} as it is
+ * published, as EventBridge's first delivery is, and checked to be published once the service is let go,
+ * and, for a wakeup's, after it, as the timer's own thread never sends them.
+ */
+final class EmrServiceFixture {
+
+ static final String REGION = "us-east-1";
+ static final String ACCOUNT = "000000000000";
+
+ /** A rule's target, given each event as it is published. */
+ @FunctionalInterface
+ interface Target {
+ void deliver(JsonNode event) throws Exception;
+ }
+
+ private record Wakeup(Instant due, Runnable task) {}
+
+ final ObjectMapper objectMapper = new ObjectMapper();
+ final List<JsonNode> published = new ArrayList<>();
+ final MutableClock clock;
+ final EmrEventPublisher publisher;
+ // Not final: the events' check reads it, from the mock set up before the service is built.
+ EmrService service;
+ Target target = event -> { };
+ private final List<Wakeup> wakeups = new ArrayList<>();
+ private final List<Runnable> deliveries = new ArrayList<>();
+ private boolean waking;
+ private final InMemoryStorageFactory storage;
+ private final int[] durations;
+
+ /** The service as configured by default, every duration 0. */
+ EmrServiceFixture() {
+ this(0, 0, 0);
+ }
+
+ EmrServiceFixture(int startupDelaySeconds, int stepDurationSeconds, int terminationDelaySeconds) {
+ this(new InMemoryStorageFactory(ACCOUNT), new MutableClock(),
+ startupDelaySeconds, stepDurationSeconds, terminationDelaySeconds);
+ }
+
+ private EmrServiceFixture(InMemoryStorageFactory storage, MutableClock clock, int... durations) {
+ this.storage = storage;
+ this.clock = clock;
+ this.durations = durations;
+ EventBridgeService eventBridge = mock(EventBridgeService.class);
+ doAnswer(invocation -> {
+ // The call that made the change has let go of the service by now.
+ assertFalse(Thread.holdsLock(service), "published while EmrService was held");
+ assertFalse(waking, "published by a wakeup itself");
+ JsonNode event = objectMapper.createObjectNode()
+ .put("detail-type", invocation.<String>getArgument(1))
+ .set("detail", objectMapper.readTree(invocation.<String>getArgument(2)));
+ published.add(event);
+ target.deliver(event);
+ return null;
+ }).when(eventBridge).publishServiceEvent(eq("aws.emr"), anyString(), anyString(), isNull(), eq(REGION));
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.services().emr().defaultReleaseLabel()).thenReturn("emr-7.5.0");
+ when(config.services().emr().clusterStartupDelaySeconds()).thenReturn(durations[0]);
+ when(config.services().emr().stepDurationSeconds()).thenReturn(durations[1]);
+ when(config.services().emr().clusterTerminationDelaySeconds()).thenReturn(durations[2]);
+ RegionResolver regionResolver = new RegionResolver(REGION, ACCOUNT);
+ publisher = new EmrEventPublisher(eventBridge, objectMapper, regionResolver);
+ service = new EmrService(storage, config, regionResolver, publisher, clock, new EmrService.Wakeups() {
+ @Override
+ public void at(Instant due, Runnable task) {
+ wakeups.add(new Wakeup(due, task));
+ }
+
+ @Override
+ public void deliver(Runnable delivery) {
+ deliveries.add(delivery);
+ }
+ });
+ }
+
+ /** A new service on the same storage and clock, as after a restart of Floci with persistent storage. */
+ EmrServiceFixture restart() {
+ return new EmrServiceFixture(storage, clock, durations);
+ }
+
+ /**
+ * Moves the clock on by {@code duration} and runs the wakeups due by then, as the service's timer
+ * would, the ones they set included, each followed by the delivery of its events.
+ */
+ void pass(Duration duration) {
+ clock.advance(duration);
+ Instant now = clock.instant();
+ List<Wakeup> due;
+ while (!(due = wakeups.stream().filter(wakeup -> !wakeup.due().isAfter(now)).toList()).isEmpty()) {
+ wakeups.removeAll(due);
+ for (Wakeup wakeup : due) {
+ waking = true;
+ try {
+ wakeup.task().run();
+ } finally {
+ waking = false;
+ }
+ while (!deliveries.isEmpty()) {
+ deliveries.remove(0).run();
+ }
+ }
+ }
+ }
+
+ /** The times of the wakeups not yet run. */
+ List<Instant> wakeupTimes() {
+ return wakeups.stream().map(Wakeup::due).toList();
+ }
+
+ /** Each event's state, a step's followed by its name. */
+ List<String> states() {
+ return published.stream().map(event -> {
+ JsonNode detail = event.path("detail");
+ return detail.has("stepId")
+ ? detail.path("state").asText() + " " + detail.path("name").asText()
+ : detail.path("state").asText();
+ }).toList();
+ }
+
+ static EmrCluster cluster(String name, boolean keepAlive, EmrStep... steps) {
+ EmrCluster cluster = new EmrCluster();
+ cluster.setName(name);
+ cluster.setKeepJobFlowAliveWhenNoSteps(keepAlive);
+ cluster.setSteps(new ArrayList<>(List.of(steps)));
+ return cluster;
+ }
+
+ static EmrStep step(String name) {
+ EmrStep step = new EmrStep();
+ step.setName(name);
+ step.setActionOnFailure("CONTINUE");
+ return step;
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceTest.java
new file mode 100644
index 0000000000..ab0a07fb61
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceTest.java
@@ -0,0 +1,463 @@
+package io.github.hectorvent.floci.services.emr;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.emr.model.EmrCluster;
+import io.github.hectorvent.floci.services.emr.model.EmrInstanceFleet;
+import io.github.hectorvent.floci.services.emr.model.EmrInstanceGroup;
+import io.github.hectorvent.floci.services.emr.model.EmrStep;
+import io.github.hectorvent.floci.testing.MutableClock;
+import org.junit.jupiter.api.Test;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.REGION;
+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.cluster;
+import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.step;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The timed lifecycle: the durations {@code cluster-startup-delay-seconds}, {@code step-duration-seconds}
+ * and {@code cluster-termination-delay-seconds} hold a cluster and its steps in each state of the
+ * Management Guide's lifecycle, a call finds them as they stand when it is made, and the service sends
+ * each change's event when it falls due, without anyone asking.
+ */
+class EmrServiceTest {
+
+ private static final String SHUT_DOWN =
+ "A job flow that is shutting down, terminated, or finished may not be modified.";
+
+ @Test
+ void aClusterIsStartingThenBootstrappingUntilItIsReady() {
+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 0);
+ EmrService service = fixture.service;
+ EmrCluster cluster = service.runJobFlow(cluster("slow", true), REGION);
+ Instant created = cluster.getCreationDateTime();
+
+ assertEquals("STARTING", service.describeCluster(cluster.getId()).getState());
+ fixture.clock.advance(Duration.ofSeconds(5));
+ assertEquals("BOOTSTRAPPING", service.describeCluster(cluster.getId()).getState());
+ fixture.clock.advance(Duration.ofSeconds(5));
+
+ EmrCluster ready = service.describeCluster(cluster.getId());
+ assertEquals("WAITING", ready.getState());
+ assertEquals(created.plusSeconds(10), ready.getReadyDateTime());
+ assertEquals(List.of("STARTING", "WAITING"), fixture.states());
+ assertTrue(fixture.published.get(1).at("/detail/message").asText().endsWith(" was created at "
+ + "2026-01-01 00:00 UTC and is ready for use."));
+ }
+
+ @Test
+ void aClusterNobodyReadsSendsEachEventWhenItFallsDue() {
+ EmrServiceFixture fixture = new EmrServiceFixture(10, 60, 30);
+ EmrCluster cluster = fixture.service.runJobFlow(cluster("unread", false, step("only")), REGION);
+ Instant created = cluster.getCreationDateTime();
+
+ assertEquals(List.of("STARTING", "PENDING only"), fixture.states());
+ assertEquals(List.of(created.plusSeconds(5)), fixture.wakeupTimes());
+ fixture.pass(Duration.ofSeconds(10));
+ assertEquals(List.of("STARTING", "PENDING only", "RUNNING only"), fixture.states());
+ assertEquals(List.of(created.plusSeconds(70)), fixture.wakeupTimes());
+ fixture.pass(Duration.ofSeconds(60));
+ assertEquals(List.of("STARTING", "PENDING only", "RUNNING only", "COMPLETED only"), fixture.states());
+ fixture.pass(Duration.ofSeconds(30));
+
+ assertEquals(List.of("STARTING", "PENDING only", "RUNNING only", "COMPLETED only", "TERMINATED"),
+ fixture.states());
+ assertEquals(List.of(), fixture.wakeupTimes());
+ EmrCluster ended = fixture.service.describeCluster(cluster.getId());
+ assertEquals("TERMINATED", ended.getState());
+ assertEquals("ALL_STEPS_COMPLETED", ended.getStateChangeReasonCode());
+ assertEquals(created.plusSeconds(100), ended.getEndDateTime());
+ }
+
+ @Test
+ void stepsRunOneAtATimeInTheOrderTheyWereAdded() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);
+ EmrService service = fixture.service;
+ EmrCluster cluster = service.runJobFlow(cluster("serial", true, step("a"), step("b")), REGION);
+ String id = cluster.getId();
+ EmrStep a = cluster.getSteps().get(0);
+ EmrStep b = cluster.getSteps().get(1);
+
+ assertEquals("RUNNING", service.describeCluster(id).getState());
+ assertEquals(List.of("PENDING", "RUNNING"), states(service.listSteps(id, null, null)));
+ assertEquals(List.of(b.getId()), ids(service.listSteps(id, List.of("PENDING"), null)));
+ fixture.clock.advance(Duration.ofSeconds(60));
+ assertEquals(List.of("RUNNING", "COMPLETED"), states(service.listSteps(id, null, null)));
+ assertEquals(a.getStartDateTime().plusSeconds(60), a.getEndDateTime());
+ assertEquals(a.getEndDateTime(), b.getStartDateTime());
+ fixture.clock.advance(Duration.ofSeconds(60));
+
+ assertEquals("WAITING", service.describeCluster(id).getState());
+ assertEquals(List.of("STARTING", "PENDING a", "PENDING b", "RUNNING a", "COMPLETED a", "RUNNING b",
+ "COMPLETED b", "WAITING"), fixture.states());
+ assertTrue(fixture.published.get(4).at("/detail/message").asText()
+ .endsWith(" and took 1 minutes to complete."));
+ }
+
+ @Test
+ void aClusterRunsAsManyStepsAtOnceAsItsConcurrencyLevelAllows() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);
+ EmrService service = fixture.service;
+ EmrCluster parallel = cluster("parallel", true, step("a"), step("b"), step("c"));
+ parallel.setStepConcurrencyLevel(2);
+ String id = service.runJobFlow(parallel, REGION).getId();
+
+ assertEquals(List.of("PENDING", "RUNNING", "RUNNING"), states(service.listSteps(id, null, null)));
+ fixture.clock.advance(Duration.ofSeconds(10));
+ service.modifyCluster(id, 3);
+ assertEquals(List.of("RUNNING", "RUNNING", "RUNNING"), states(service.listSteps(id, null, null)));
+ // Lowering the level lets the steps already running complete.
+ service.modifyCluster(id, 1);
+ fixture.clock.advance(Duration.ofSeconds(60));
+
+ assertEquals(List.of("COMPLETED", "COMPLETED", "COMPLETED"), states(service.listSteps(id, null, null)));
+ assertEquals("WAITING", service.describeCluster(id).getState());
+ }
+
+ @Test
+ void stepsAddedWhileAClusterStartsRunOnceItIsReady() {
+ EmrServiceFixture fixture = new EmrServiceFixture(10, 60, 0);
+ EmrService service = fixture.service;
+ EmrCluster cluster = service.runJobFlow(cluster("early", true), REGION);
+ EmrStep early = step("early");
+
+ service.addJobFlowSteps(cluster.getId(), List.of(early));
+ assertEquals("PENDING", early.getState());
+ fixture.clock.advance(Duration.ofSeconds(10));
+
+ assertEquals("RUNNING", service.describeStep(cluster.getId(), early.getId()).getState());
+ assertEquals(cluster.getCreationDateTime().plusSeconds(10), early.getStartDateTime());
+ assertEquals("RUNNING", service.describeCluster(cluster.getId()).getState());
+ }
+
+ @Test
+ void aWaitingClusterIsRunningWhileItRunsAddedStepsAndWaitsAgainAfter() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);
+ EmrService service = fixture.service;
+ String id = service.runJobFlow(cluster("kept", true), REGION).getId();
+ assertEquals("Cluster ready to run steps.", service.describeCluster(id).getStateChangeMessage());
+
+ service.addJobFlowSteps(id, List.of(step("late")));
+ EmrCluster running = service.describeCluster(id);
+ assertEquals("RUNNING", running.getState());
+ assertNull(running.getStateChangeReasonCode());
+ assertNull(running.getStateChangeMessage());
+ fixture.clock.advance(Duration.ofSeconds(60));
+
+ EmrCluster waiting = service.describeCluster(id);
+ assertEquals("WAITING", waiting.getState());
+ assertEquals("ALL_STEPS_COMPLETED", waiting.getStateChangeReasonCode());
+ assertEquals(List.of("STARTING", "WAITING", "PENDING late", "RUNNING late", "COMPLETED late", "WAITING"),
+ fixture.states());
+ }
+
+ @Test
+ void addingNoStepsLeavesAWaitingClusterAsItIs() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);
+ EmrService service = fixture.service;
+ String fresh = service.runJobFlow(cluster("fresh", true), REGION).getId();
+ String done = service.runJobFlow(cluster("done", true, step("only")), REGION).getId();
+ fixture.clock.advance(Duration.ofSeconds(60));
+ // Set to end after its steps: a cluster that already waits ends only after the next ones it runs.
+ service.setKeepJobFlowAliveWhenNoSteps(List.of(done), false);
+ fixture.published.clear();
+
+ assertEquals(List.of(), service.addJobFlowSteps(fresh, List.of()));
+ assertEquals(List.of(), service.addJobFlowSteps(done, List.of()));
+
+ assertEquals("Cluster ready to run steps.", service.describeCluster(fresh).getStateChangeMessage());
+ EmrCluster waiting = service.describeCluster(done);
+ assertEquals("WAITING", waiting.getState());
+ assertEquals("ALL_STEPS_COMPLETED", waiting.getStateChangeReasonCode());
+ assertEquals(List.of(), fixture.states());
+ }
+
+ @Test
+ void cancellingAPendingStepLeavesTheRunningOneRunning() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);
+ EmrService service = fixture.service;
+ EmrCluster cluster = service.runJobFlow(cluster("cancel", true, step("a"), step("b")), REGION);
+ EmrStep b = cluster.getSteps().get(1);
+
+ List<EmrService.CancelStepInfo> infos = service.cancelSteps(cluster.getId(), List.of(b.getId()));
+
+ assertEquals(List.of(new EmrService.CancelStepInfo(b.getId(), "SUBMITTED", null)), infos);
+ assertEquals(List.of("CANCELLED", "RUNNING"), states(service.listSteps(cluster.getId(), null, null)));
+ assertEquals("CANCELLED b", fixture.states().get(fixture.states().size() - 1));
+ }
+
+ @Test
+ void cancellingARunningStepLetsTheNextOneRun() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);
+ EmrService service = fixture.service;
+ EmrCluster cluster = service.runJobFlow(cluster("cancel", true, step("a"), step("b")), REGION);
+ EmrStep a = cluster.getSteps().get(0);
+ fixture.clock.advance(Duration.ofSeconds(10));
+
+ List<EmrService.CancelStepInfo> infos = service.cancelSteps(cluster.getId(), List.of(a.getId()));
+
+ assertEquals(List.of(new EmrService.CancelStepInfo(a.getId(), "SUBMITTED", null)), infos);
+ assertEquals(List.of("RUNNING", "CANCELLED"), states(service.listSteps(cluster.getId(), null, null)));
+ assertEquals(a.getEndDateTime(), cluster.getSteps().get(1).getStartDateTime());
+ assertEquals(List.of("CANCELLED a", "RUNNING b"),
+ fixture.states().subList(fixture.states().size() - 2, fixture.states().size()));
+ }
+
+ @Test
+ void aReleaseBefore528CancelsOnlyPendingSteps() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 0);
+ EmrService service = fixture.service;
+ EmrCluster old = cluster("old", true, step("a"), step("b"));
+ old.setReleaseLabel("emr-5.27.0");
+ EmrCluster cluster = service.runJobFlow(old, REGION);
+ String a = cluster.getSteps().get(0).getId();
+ String b = cluster.getSteps().get(1).getId();
+
+ List<EmrService.CancelStepInfo> infos = service.cancelSteps(cluster.getId(), List.of(a, b));
+
+ assertEquals(List.of(
+ new EmrService.CancelStepInfo(a, "FAILED", "Step is not in a cancellable state: RUNNING"),
+ new EmrService.CancelStepInfo(b, "SUBMITTED", null)), infos);
+ assertTrue(EmrClusterLifecycle.cancelsRunningSteps("emr-5.28.0"));
+ assertTrue(EmrClusterLifecycle.cancelsRunningSteps("emr-7.5.0"));
+ assertFalse(EmrClusterLifecycle.cancelsRunningSteps("emr-4.9.6"));
+ }
+
+ @Test
+ void terminatingCancelsTheStepsNotYetCompletedAndEndsAfterTheDelay() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 30);
+ EmrService service = fixture.service;
+ EmrCluster kept = cluster("ending", true, step("a"), step("b"));
+ kept.getInstanceGroups().add(group("CORE", 2));
+ EmrCluster cluster = service.runJobFlow(kept, REGION);
+ String id = cluster.getId();
+ fixture.clock.advance(Duration.ofSeconds(10));
+ fixture.published.clear();
+
+ service.terminateJobFlows(List.of(id));
+
+ EmrCluster terminating = service.describeCluster(id);
+ assertEquals("TERMINATING", terminating.getState());
+ assertEquals("USER_REQUEST", terminating.getStateChangeReasonCode());
+ assertNull(terminating.getEndDateTime());
+ assertEquals(List.of("CANCELLED", "CANCELLED"), states(service.listSteps(id, null, null)));
+ assertEquals("RUNNING", service.listInstanceGroups(id).get(0).getState());
+ AwsException refused = assertThrows(AwsException.class,
+ () -> service.addJobFlowSteps(id, List.of(step("late"))));
+ assertEquals(SHUT_DOWN, refused.getMessage());
+ service.terminateJobFlows(List.of(id));
+ assertEquals(List.of("CANCELLED a", "CANCELLED b"), fixture.states());
+ fixture.clock.advance(Duration.ofSeconds(30));
+
+ EmrCluster ended = service.describeCluster(id);
+ assertEquals("TERMINATED", ended.getState());
+ assertEquals(terminating.getTerminatingDateTime().plusSeconds(30), ended.getEndDateTime());
+ assertEquals("TERMINATED", service.listInstanceGroups(id).get(0).getState());
+ assertEquals(0, service.listInstanceGroups(id).get(0).getRunningInstanceCount());
+ assertEquals(List.of("CANCELLED a", "CANCELLED b", "TERMINATED"), fixture.states());
+ }
+
+ @Test
+ void instanceGroupsChangeOnlyWhileTheClusterRunsOrWaits() {
+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 30);
+ EmrService service = fixture.service;
+ EmrCluster withGroup = cluster("groups", true);
+ withGroup.getInstanceGroups().add(group("CORE", 2));
+ EmrCluster cluster = service.runJobFlow(withGroup, REGION);
+ String groupId = cluster.getInstanceGroups().get(0).getId();
+ List<EmrService.InstanceGroupModification> resize =
+ List.of(new EmrService.InstanceGroupModification(groupId, 3, null));
+
+ AwsException starting = assertThrows(AwsException.class,
+ () -> service.modifyInstanceGroups(cluster.getId(), REGION, resize));
+ assertEquals("An instance group may only be modified when the cluster is running or waiting.",
+ starting.getMessage());
+ fixture.clock.advance(Duration.ofSeconds(10));
+ service.modifyInstanceGroups(null, REGION, resize);
+ assertEquals(3, service.listInstanceGroups(cluster.getId()).get(0).getRequestedInstanceCount());
+ service.terminateJobFlows(List.of(cluster.getId()));
+
+ assertThrows(AwsException.class, () -> service.modifyInstanceGroups(cluster.getId(), REGION, resize));
+ AwsException adding = assertThrows(AwsException.class,
+ () -> service.addInstanceGroups(cluster.getId(), List.of(group("TASK", 1))));
+ assertEquals("ValidationException", adding.getErrorCode());
+ assertEquals(SHUT_DOWN, adding.getMessage());
+ assertEquals(1, service.listInstanceGroups(cluster.getId()).size());
+ }
+
+ @Test
+ void anInstanceFleetIsResizedOnlyOnceTheClusterRunsOrWaits() {
+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 0);
+ EmrService service = fixture.service;
+ EmrCluster withFleet = cluster("starting", true);
+ EmrInstanceFleet fleet = fleet("CORE", 2);
+ withFleet.getInstanceFleets().add(fleet);
+ String id = service.runJobFlow(withFleet, REGION).getId();
+
+ AwsException starting = assertThrows(AwsException.class,
+ () -> service.modifyInstanceFleet(id, fleet.getId(), 3, null));
+ fixture.clock.advance(Duration.ofSeconds(5));
+ AwsException bootstrapping = assertThrows(AwsException.class,
+ () -> service.modifyInstanceFleet(id, fleet.getId(), 3, null));
+ fixture.clock.advance(Duration.ofSeconds(5));
+ service.modifyInstanceFleet(id, fleet.getId(), 3, null);
+
+ assertNotRunningOrWaiting(starting);
+ assertNotRunningOrWaiting(bootstrapping);
+ assertEquals(3, service.listInstanceFleets(id).get(0).getTargetOnDemandCapacity());
+ }
+
+ @Test
+ void anInstanceFleetIsNotAddedOrResizedWhileTheClusterShutsDown() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 0, 30);
+ EmrService service = fixture.service;
+ EmrCluster withFleet = cluster("fleets", true);
+ EmrInstanceFleet fleet = fleet("CORE", 2);
+ withFleet.getInstanceFleets().add(fleet);
+ EmrCluster cluster = service.runJobFlow(withFleet, REGION);
+ service.terminateJobFlows(List.of(cluster.getId()));
+
+ AwsException resizing = assertThrows(AwsException.class,
+ () -> service.modifyInstanceFleet(cluster.getId(), fleet.getId(), 3, null));
+ AwsException adding = assertThrows(AwsException.class,
+ () -> service.addInstanceFleet(cluster.getId(), new EmrInstanceFleet()));
+
+ assertShutDown(resizing);
+ assertShutDown(adding);
+ assertEquals(1, service.listInstanceFleets(cluster.getId()).size());
+ }
+
+ @Test
+ void listingClustersByStateFindsThemAsTheyStandNow() {
+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 0);
+ EmrService service = fixture.service;
+ String id = service.runJobFlow(cluster("listed", true), REGION).getId();
+
+ assertEquals(List.of(), service.listClusters(REGION, List.of("WAITING"), null, null));
+ fixture.clock.advance(Duration.ofSeconds(10));
+
+ assertEquals(List.of(id),
+ service.listClusters(REGION, List.of("WAITING"), null, null).stream().map(EmrCluster::getId).toList());
+ }
+
+ @Test
+ void aClusterLeftMidChangeByARestartMovesOnWhenItFallsDue() {
+ EmrServiceFixture fixture = new EmrServiceFixture(10, 0, 0);
+ EmrCluster cluster = fixture.service.runJobFlow(cluster("restarted", true), REGION);
+
+ EmrServiceFixture restarted = fixture.restart();
+
+ assertEquals(List.of(cluster.getCreationDateTime().plusSeconds(5)), restarted.wakeupTimes());
+ restarted.pass(Duration.ofSeconds(10));
+ assertEquals(List.of("WAITING"), restarted.states());
+ }
+
+ @Test
+ void whatACallReturnsStaysAsItWasWhenTheClusterMovesOn() {
+ EmrServiceFixture fixture = new EmrServiceFixture(0, 60, 30);
+ EmrService service = fixture.service;
+ EmrCluster read = cluster("read", true, step("only"));
+ read.getInstanceGroups().add(group("CORE", 2));
+ read.getInstanceFleets().add(fleet("TASK", 2));
+ EmrCluster created = service.runJobFlow(read, REGION);
+ String id = created.getId();
+ String stepId = created.getSteps().get(0).getId();
+
+ List<EmrCluster> clusters = List.of(service.describeCluster(id),
+ service.listClusters(REGION, null, null, null).get(0), service.getClusterForInstances(id));
+ List<EmrStep> steps = List.of(service.describeStep(id, stepId), service.listSteps(id, null, null).get(0));
+ EmrInstanceGroup group = service.listInstanceGroups(id).get(0);
+ EmrInstanceFleet fleet = service.listInstanceFleets(id).get(0);
+ fixture.pass(Duration.ofSeconds(60));
+ service.terminateJobFlows(List.of(id));
+ fixture.pass(Duration.ofSeconds(30));
+
+ assertEquals("TERMINATED", service.describeCluster(id).getState());
+ assertEquals(List.of("RUNNING", "RUNNING", "RUNNING"), clusters.stream().map(EmrCluster::getState).toList());
+ assertEquals(List.of("RUNNING", "RUNNING"), states(steps));
+ assertNull(steps.get(0).getEndDateTime());
+ assertEquals(2, group.getRunningInstanceCount());
+ assertEquals(2, fleet.getProvisionedOnDemandCapacity());
+ }
+
+ @Test
+ void aSlowDeliveryDoesNotHoldUpTheTimer() throws InterruptedException {
+ EmrService.Wakeups timer = EmrService.Wakeups.timer(new MutableClock());
+ CountDownLatch woken = new CountDownLatch(1);
+ CountDownLatch delivered = new CountDownLatch(1);
+ try {
+ // Like a rule's target that answers only once the next cluster has changed.
+ timer.deliver(() -> {
+ try {
+ if (woken.await(10, TimeUnit.SECONDS)) {
+ delivered.countDown();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ });
+ timer.at(Instant.EPOCH, woken::countDown);
+
+ assertTrue(delivered.await(10, TimeUnit.SECONDS));
+ } finally {
+ timer.close();
+ }
+ }
+
+ @Test
+ void aNegativeDurationIsRefused() {
+ IllegalArgumentException refused = assertThrows(IllegalArgumentException.class,
+ () -> new EmrServiceFixture(0, -1, 0));
+
+ assertEquals("floci.services.emr.step-duration-seconds must not be negative: -1", refused.getMessage());
+ }
+
+ /** ModifyInstanceFleet's answer for a job flow that is shutting down, which AddInstanceFleet shares. */
+ private static void assertShutDown(AwsException refused) {
+ assertEquals("InvalidRequestException", refused.getErrorCode());
+ assertEquals(SHUT_DOWN, refused.getMessage());
+ assertEquals(Map.of("ErrorCode", "INVALID_JOB_SHUT_DOWN"), refused.getExtendedData());
+ }
+
+ /** ModifyInstanceFleet's answer for a cluster that is starting or bootstrapping. */
+ private static void assertNotRunningOrWaiting(AwsException refused) {
+ assertEquals("InvalidRequestException", refused.getErrorCode());
+ assertEquals("An instance fleet may only be modified when the cluster is running or waiting.",
+ refused.getMessage());
+ assertEquals(Map.of("ErrorCode", "MODIFY_INSTANCE_FLEET_OF_RUNNING_JOBFLOW_ONLY"), refused.getExtendedData());
+ }
+
+ private static EmrInstanceGroup group(String type, int count) {
+ EmrInstanceGroup group = new EmrInstanceGroup();
+ group.setInstanceGroupType(type);
+ group.setInstanceType("m5.xlarge");
+ group.setRequestedInstanceCount(count);
+ return group;
+ }
+
+ private static EmrInstanceFleet fleet(String type, int onDemandCapacity) {
+ EmrInstanceFleet fleet = new EmrInstanceFleet();
+ fleet.setInstanceFleetType(type);
+ fleet.setTargetOnDemandCapacity(onDemandCapacity);
+ return fleet;
+ }
+
+ /** ListSteps' states, newest step first, as it lists them. */
+ private static List<String> states(List<EmrStep> steps) {
+ return steps.stream().map(EmrStep::getState).toList();
+ }
+
+ private static List<String> ids(List<EmrStep> steps) {
+ return steps.stream().map(EmrStep::getId).toList();
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrTimedLifecycleIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrTimedLifecycleIntegrationTest.java
new file mode 100644
index 0000000000..4934972af4
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrTimedLifecycleIntegrationTest.java
@@ -0,0 +1,77 @@
+package io.github.hectorvent.floci.services.emr;
+
+import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
+import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.QuarkusTestProfile;
+import io.quarkus.test.junit.TestProfile;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+
+import static io.github.hectorvent.floci.testing.EmrRequests.emr;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.nullValue;
+
+/**
+ * With the lifecycle durations set, a cluster is seen STARTING and TERMINATING, and its steps PENDING,
+ * over the wire. They are an hour long here, so nothing moves on while the test runs.
+ */
+@QuarkusTest
+@TestProfile(EmrTimedLifecycleIntegrationTest.HourLong.class)
+class EmrTimedLifecycleIntegrationTest {
+
+ public static class HourLong implements QuarkusTestProfile {
+ @Override
+ public Map<String, String> getConfigOverrides() {
+ return Map.of(
+ "floci.services.emr.cluster-startup-delay-seconds", "3600",
+ "floci.services.emr.step-duration-seconds", "3600",
+ "floci.services.emr.cluster-termination-delay-seconds", "3600");
+ }
+ }
+
+ @BeforeAll
+ static void configure() {
+ RestAssuredJsonUtils.configureAwsContentTypes();
+ }
+
+ @Test
+ void aStartingClusterKeepsItsStepsPendingAndShutsDownWhenTerminated() {
+ String clusterId = emr("RunJobFlow", "{\"Name\":\"hour-long\","
+ + "\"Instances\":{\"KeepJobFlowAliveWhenNoSteps\":true},"
+ + "\"Steps\":[{\"Name\":\"first\",\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}},"
+ + "{\"Name\":\"second\",\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")
+ .then().statusCode(200).extract().path("JobFlowId");
+ String firstId = emr("ListSteps", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(200)
+ .body("Steps.Status.State", equalTo(List.of("PENDING", "PENDING")))
+ .extract().path("Steps[1].Id");
+
+ emr("DescribeCluster", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(200)
+ .body("Cluster.Status.State", equalTo("STARTING"))
+ .body("Cluster.Status.Timeline.ReadyDateTime", nullValue());
+ emr("ListClusters", "{\"ClusterStates\":[\"STARTING\"]}").then().statusCode(200)
+ .body("Clusters.Id", hasItem(clusterId));
+ emr("CancelSteps", "{\"ClusterId\":\"" + clusterId + "\",\"StepIds\":[\"" + firstId + "\"]}")
+ .then().statusCode(200)
+ .body("CancelStepsInfoList[0].Status", equalTo("SUBMITTED"));
+ emr("DescribeStep", "{\"ClusterId\":\"" + clusterId + "\",\"StepId\":\"" + firstId + "\"}")
+ .then().statusCode(200)
+ .body("Step.Status.State", equalTo("CANCELLED"));
+ emr("TerminateJobFlows", "{\"JobFlowIds\":[\"" + clusterId + "\"]}").then().statusCode(200);
+
+ emr("DescribeCluster", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(200)
+ .body("Cluster.Status.State", equalTo("TERMINATING"))
+ .body("Cluster.Status.StateChangeReason.Code", equalTo("USER_REQUEST"));
+ emr("ListSteps", "{\"ClusterId\":\"" + clusterId + "\"}").then().statusCode(200)
+ .body("Steps.Status.State", equalTo(List.of("CANCELLED", "CANCELLED")));
+ emr("AddJobFlowSteps", "{\"JobFlowId\":\"" + clusterId + "\",\"Steps\":[{\"Name\":\"late\","
+ + "\"HadoopJarStep\":{\"Jar\":\"command-runner.jar\"}}]}")
+ .then().statusCode(400)
+ .body("__type", equalTo("ValidationException"))
+ .body("message", equalTo(
+ "A job flow that is shutting down, terminated, or finished may not be modified."));
+ }
+}