Light, fluffy, and always free - The AWS Local Emulator alternative
feat(emr): run a cluster's bootstrap actions in the docker runner mode (#6878)
Priamo Germosen591d5f533b9f
diff --git a/docs/services/emr.md b/docs/services/emr.mdindex 2ade067950..d63c7512a2 100644--- a/docs/services/emr.md+++ b/docs/services/emr.md@@ -4,7 +4,7 @@**Endpoint:** `http://localhost:4566/`**Target prefix:** `X-Amz-Target: ElasticMapReduce.*`-Floci emulates the Amazon EMR (Elastic MapReduce) management API. Clusters (job flows), instance groups and fleets, steps, security configurations, and tags are tracked in Floci's storage backend so the AWS CLI and SDK clients can drive the full cluster lifecycle locally. EMR does not launch real Hadoop/Spark clusters. Clusters transition through their lifecycle states as a state machine. In the `docker` runner mode Spark steps, commands and scripts run for real, each in a container of its own (see [Running steps](#running-steps)).+Floci emulates the Amazon EMR (Elastic MapReduce) management API. Clusters (job flows), instance groups and fleets, steps, security configurations, and tags are tracked in Floci's storage backend so the AWS CLI and SDK clients can drive the full cluster lifecycle locally. EMR does not launch real Hadoop/Spark clusters. Clusters transition through their lifecycle states as a state machine. In the `docker` runner mode Spark steps, commands and scripts run for real, each in a container of its own, after the cluster's bootstrap actions (see [Running steps](#running-steps)).## Supported Actions@@ -111,7 +111,8 @@ With `runner-mode` set to `docker` (`FLOCI_SERVICES_EMR_RUNNER_MODE=docker`), aone container per step from the cluster's release image, the one EMR Serverless's `docker` mode uses(`public.ecr.aws/emr-serverless/spark/<release>`), and runs the step in it: a Spark step's`spark-submit` in local mode, the container holding the driver and its executors, or a command or a-script. The cluster itself gets no container, and goes through its lifecycle as in the `immediate` mode.+script. The cluster itself gets no container, and goes through its lifecycle as in the `immediate` mode,+but for its bootstrap actions.* **Which steps run**: a `command-runner.jar` step whose first argument is `spark-submit`, the way theManagement Guide writes Spark steps, runs the arguments after it, without its `--master` and@@ -130,6 +131,27 @@ script. The cluster itself gets no container, and goes through its lifecycle asits own. A `spark-submit` they start runs in local mode on Floci's S3 with the cluster's`spark-defaults`, which Floci puts in a Spark configuration of its own, ahead of the image's EMRServerless master; its own `--master yarn` or `--deploy-mode cluster` fails, as there is no YARN.+* **Bootstrap actions**: a cluster with `BootstrapActions` is `BOOTSTRAPPING` until they have run, one+ after the other, in one container of the release's image, whatever the second half of+ `cluster-startup-delay-seconds`. Each action's script is read from Floci's S3 (an `s3n://` or `s3a://`+ path names the same bucket, and another path is the image's own) and runs with its `Args` in+ `/home/hadoop`, with the credentials of the cluster's `JobFlowRole` instance profile, as a step would.+ "Bootstrap actions execute as the Hadoop user by default", and `sudo` gives them root, but the image+ has no `sudo`: an action runs as root, with a `sudo` that runs its command as it is, and what it leaves+ in `/home/hadoop` is the `hadoop` user's. Once every action has succeeded, the container is kept as an+ image named after the cluster, which its steps run in, so that what the actions installed or wrote is+ there for each of them; the image goes once the cluster has ended and its last step is done. An action+ that exits with another code than 0 ends the cluster: it cancels its steps and goes `TERMINATING`, then+ `TERMINATED_WITH_ERRORS`, with the reason `BOOTSTRAP_FAILURE` and AWS's message "On the master instance+ (i-00000000000000001), bootstrap action 1 returned a non-zero return code", the instance as+ `ListInstances` gives the master. A script that cannot be read ends it the same way, in Floci's own+ words, and a bootstrap that cannot run at all, an image that cannot be pulled say, with+ `INTERNAL_ERROR`. `TerminateJobFlows` stops the actions at once, and a cluster Floci was bootstrapping+ when it stopped ends with `INTERNAL_ERROR` once Floci is back. With a `LogUri`, each action's output+ goes to `stdout.gz` and `stderr.gz` under `<LogUri>/<cluster-id>/node/<instance-id>/bootstrap-actions/<n>/`,+ as the Management Guide's bootstrap failure pages give the folder. The actions run once, as on the+ master: there are no core or task nodes to run them on. Shutdown actions do not run, and+ `Status.ErrorDetails` is not given.* **States**: a step is `RUNNING` from when it starts until its driver or command exits; `step-duration-seconds`does not apply. The first step of a release also waits while its image is pulled (3.5 to 4.3 GBunpacked; see [`floci.docker.image-pull-timeout-seconds`](../configuration/docker.md)). It is then@@ -218,10 +240,10 @@ its `STARTING` event, has its events sent after the rest of that call's.| `FLOCI_SERVICES_EMR_CLUSTER_STARTUP_DELAY_SECONDS` | `0` | Seconds a new cluster takes to be ready to run steps, the first half `STARTING` and the second `BOOTSTRAPPING`; `0` makes it ready as soon as it is created || `FLOCI_SERVICES_EMR_STEP_DURATION_SECONDS` | `0` | Seconds a step stays `RUNNING` before it completes; `0` completes it as soon as it starts || `FLOCI_SERVICES_EMR_CLUSTER_TERMINATION_DELAY_SECONDS` | `0` | Seconds a cluster stays `TERMINATING` before it is `TERMINATED`; `0` ends it at once |-| `FLOCI_SERVICES_EMR_RUNNER_MODE` | `immediate` | `immediate` completes steps by time without running anything; `docker` runs each step in a container (see [Running steps](#running-steps)) |+| `FLOCI_SERVICES_EMR_RUNNER_MODE` | `immediate` | `immediate` completes steps by time without running anything; `docker` runs each step in a container, after the cluster's bootstrap actions (see [Running steps](#running-steps)) || `FLOCI_SERVICES_EMR_DOCKER_NETWORK` | the global default | Docker network for step containers || `FLOCI_SERVICES_EMR_IMAGE_TEMPLATE` | `public.ecr.aws/emr-serverless/spark/%s:latest`, or `spark4/` in place of `spark/` for an `emr-spark-*` release | Image for a `docker` step, `%s` standing for the cluster's release label |-| `FLOCI_SERVICES_EMR_RECONCILE_CONTAINERS_ON_STARTUP` | `true` | Remove the step drivers a previous run of this Floci left on the Docker daemon when the EMR service starts; turn it off when two Floci instances share a daemon with the same port and no resource namespace |+| `FLOCI_SERVICES_EMR_RECONCILE_CONTAINERS_ON_STARTUP` | `true` | Remove the step drivers and bootstrap containers a previous run of this Floci left on the Docker daemon, and the images bootstrap actions left for clusters that are gone or have ended, when the EMR service starts; turn it off when two Floci instances share a daemon with the same port and no resource namespace |## Release cataloguediff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex 6ed7923c09..17790d2c42 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -2014,10 +2014,11 @@ public interface EmulatorConfig {Optional<String> imageTemplate();/**- * When true, Floci removes, in the {@code docker} runner mode, every EMR step driver a previous run- * of <em>this same</em> Floci left on the daemon (matched by the {@code io.floci.owner} label) when- * the EMR service starts. Turn it off when two Floci instances share a daemon with the same port- * and no resource namespace, so one does not remove the other's drivers.+ * When true, Floci removes, in the {@code docker} runner mode, every EMR step driver and bootstrap+ * container a previous run of <em>this same</em> Floci left on the daemon (matched by the+ * {@code io.floci.owner} label), and the images bootstrap actions left for clusters that are gone or+ * have ended, when the EMR service starts. Turn it off when two Floci instances share a daemon with+ * the same port and no resource namespace, so one does not remove the other's drivers.** Env var: FLOCI_SERVICES_EMR_RECONCILE_CONTAINERS_ON_STARTUP*/diff --git a/src/main/java/io/github/hectorvent/floci/core/common/S3Uri.java b/src/main/java/io/github/hectorvent/floci/core/common/S3Uri.javaindex a3a0e30861..7a73fe9cca 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/S3Uri.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/S3Uri.java@@ -1,5 +1,7 @@package io.github.hectorvent.floci.core.common;+import java.util.regex.Pattern;+/*** An {@code s3://bucket/key} URI, split into its bucket and its key. The key is empty when the URI names* only a bucket, and keeps a trailing slash, as a key prefix does.@@ -7,6 +9,8 @@ package io.github.hectorvent.floci.core.common;public record S3Uri(String bucket, String key) {private static final String SCHEME = "s3://";+ // Hadoop's s3n:// and s3a://, in which an EMR cluster's paths are often written, name the same buckets.+ private static final Pattern ANY_S3_SCHEME = Pattern.compile("^s3[an]?://");/** Splits {@code uri}; one that is not {@code s3://} or names no bucket is an IllegalArgumentException. */public static S3Uri parse(String uri) {@@ -21,4 +25,14 @@ public record S3Uri(String bucket, String key) {}return new S3Uri(bucket, slash < 0 ? "" : rest.substring(slash + 1));}++ /** Whether {@code uri} is in {@code s3://}, or in Hadoop's {@code s3n://} or {@code s3a://}. */+ public static boolean isS3(String uri) {+ return uri != null && ANY_S3_SCHEME.matcher(uri).find();+ }++ /** As {@link #parse}, taking Hadoop's {@code s3n://} and {@code s3a://} for {@code s3://}. */+ public static S3Uri parseAnyScheme(String uri) {+ return parse(uri == null ? null : ANY_S3_SCHEME.matcher(uri).replaceFirst(SCHEME));+ }}diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.javaindex a77043e60f..ce1545281b 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.core.common.docker;import com.github.dockerjava.api.DockerClient;import com.github.dockerjava.api.async.ResultCallback;+import com.github.dockerjava.api.command.ExecCreateCmd;import com.github.dockerjava.api.model.Frame;import com.github.dockerjava.api.model.StreamType;import org.jboss.logging.Logger;@@ -10,6 +11,7 @@ import java.io.ByteArrayOutputStream;import java.io.IOException;import java.io.OutputStream;import java.nio.charset.StandardCharsets;+import java.util.List;import java.util.concurrent.TimeUnit;/**@@ -73,12 +75,24 @@ public final class ContainerExec {*/public static Result run(DockerClient dockerClient, String containerId, String[] cmd, int timeoutSeconds,OutputStream stdout, OutputStream stderr) {- String execId = dockerClient.execCreateCmd(containerId)+ return run(dockerClient, containerId, cmd, List.of(), timeoutSeconds, stdout, stderr);+ }++ /**+ * Runs {@code cmd} as {@link #run(DockerClient, String, String[], int, OutputStream, OutputStream)} does,+ * with {@code env} on top of the container's environment for this command alone, so a secret it holds+ * never becomes part of the container's configuration.+ */+ public static Result run(DockerClient dockerClient, String containerId, String[] cmd, List<String> env,+ int timeoutSeconds, OutputStream stdout, OutputStream stderr) {+ ExecCreateCmd create = dockerClient.execCreateCmd(containerId).withCmd(cmd).withAttachStdout(true)- .withAttachStderr(true)- .exec()- .getId();+ .withAttachStderr(true);+ if (!env.isEmpty()) {+ create.withEnv(env);+ }+ String execId = create.exec().getId();ResultCallback.Adapter<Frame> callback = new ResultCallback.Adapter<>() {@Overridediff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLaunches.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLaunches.javanew file mode 100644index 0000000000..f548e67fe9--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLaunches.java@@ -0,0 +1,79 @@+package io.github.hectorvent.floci.core.common.docker;++import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.atomic.AtomicReference;++/**+ * The containers of the runs a component has in progress, by run id, so a stop, a reset or a shutdown can+ * end them wherever they are: before the container is up, the run reads that it was asked to stop once it+ * is; after, the container is removed. Whoever claims a run's container removes it, and no one else.+ */+public final class ContainerLaunches {++ private final ConcurrentHashMap<String, Launch> launches = new ConcurrentHashMap<>();++ /** A run of {@code id} is starting: its launch, which {@link #end} ends. */+ public Launch begin(String id) {+ Launch launch = new Launch();+ launches.put(id, launch);+ return launch;+ }++ /** The launch of the run of {@code id} in progress, or null when there is none. */+ public Launch get(String id) {+ return launches.get(id);+ }++ /** The run is over: its container, for the caller to remove, unless someone has claimed it already. */+ public String end(String id, Launch launch) {+ launches.remove(id, launch);+ return launch.claim();+ }++ /** Stops every run in progress, as {@link Launch#stopNow} does, so a reset or a shutdown orphans none. */+ public void stopAll(ContainerLifecycleManager lifecycleManager, int stopTimeoutSeconds) {+ for (Launch launch : launches.values()) {+ launch.stopNow(lifecycleManager, stopTimeoutSeconds);+ }+ }++ /** A run in progress: its container, once up and until claimed, and whether it was asked to stop. */+ public static final class Launch {++ private final AtomicReference<String> container = new AtomicReference<>();+ private volatile boolean stopRequested;++ /** The run's container is up. */+ public void started(String containerId) {+ container.set(containerId);+ }++ /** The run's container, or null before it is up or once claimed. */+ public String container() {+ return container.get();+ }++ /** Takes the run's container, to remove it: null when it is not up, or someone else took it. */+ public String claim() {+ return container.getAndSet(null);+ }++ public boolean stopRequested() {+ return stopRequested;+ }++ /** Asks the run to stop, which it reads itself; nothing is removed. */+ public void requestStop() {+ stopRequested = true;+ }++ /** Asks the run to stop, and removes its container now when it is up and no one has claimed it. */+ public void stopNow(ContainerLifecycleManager lifecycleManager, int stopTimeoutSeconds) {+ stopRequested = true;+ String containerId = claim();+ if (containerId != null) {+ lifecycleManager.stopAndRemove(containerId, null, stopTimeoutSeconds);+ }+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.javaindex 57b5858d0e..2f135653b1 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java@@ -982,6 +982,38 @@ public class ContainerLifecycleManager {}}+ /**+ * Commits {@code containerId}, stopped or running, as the image {@code repository}:{@code tag}, run as+ * {@code user} and labelled {@code labels}, and returns its id. The image takes the container's own+ * configuration wherever the commit sets none, its environment, entrypoint and command among them.+ */+ public String commit(String containerId, String repository, String tag, String user, Map<String, String> labels) {+ return dockerClient.commitCmd(containerId)+ .withRepository(repository)+ .withTag(tag)+ .withUser(user)+ .withLabels(labels)+ .exec();+ }++ /**+ * Removes {@code image}, forcibly. True when it is gone, removed now or already absent; false when the+ * daemon refused, which is logged, so a caller that must find it again keeps its reference.+ */+ public boolean removeImage(String image) {+ try {+ dockerClient.removeImageCmd(image).withForce(true).exec();+ LOG.infov("Removed Docker image {0}", image);+ return true;+ } catch (NotFoundException e) {+ LOG.debugv("Docker image {0} was already absent", image);+ return true;+ } catch (RuntimeException e) {+ LOG.warnv("Could not remove Docker image {0}: {1}", image, e.getMessage());+ return false;+ }+ }+/** Whether a container exists and is running — distinguishing "gone" from "cannot tell". */public enum ContainerPresence {/** The container exists and is running. */diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.javaindex 619cf23b33..a7c5e36cc0 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamer.java@@ -14,6 +14,7 @@ import org.jboss.logging.Logger;import java.io.ByteArrayOutputStream;import java.io.Closeable;import java.io.IOException;+import java.io.OutputStream;import java.nio.charset.StandardCharsets;import java.time.Instant;import java.time.LocalDate;@@ -109,13 +110,11 @@ public class ContainerLogStreamer {String region, String logPrefix, Instant since) {ensureLogGroupAndStreamForAccount(accountId, logGroup, logStream, region);- // Trim trailing whitespace and drop blank lines, then fan each reassembled line out to the- // console logger and CloudWatch Logs.+ // Fan each reassembled line that is not blank out to the console logger and CloudWatch Logs.Consumer<String> emitter = line -> {- String trimmed = line.stripTrailing();- if (!trimmed.isEmpty()) {- LOG.infov("[{0}] {1}", logPrefix, trimmed);- forwardToCloudWatchLogs(accountId, logGroup, logStream, region, trimmed);+ String logged = logToConsole(logPrefix, line);+ if (logged != null) {+ forwardToCloudWatchLogs(accountId, logGroup, logStream, region, logged);}};return follow(containerId, since, emitter);@@ -136,13 +135,10 @@ public class ContainerLogStreamer {}}BiConsumer<StreamType, String> emitter = (stream, line) -> {- String trimmed = line.stripTrailing();- if (!trimmed.isEmpty()) {- LOG.infov("[{0}] {1}", logPrefix, trimmed);- String logStream = stream == StreamType.STDERR ? stderrStream : stdoutStream;- if (logStream != null) {- forwardToCloudWatchLogs(accountId, logGroup, logStream, region, trimmed);- }+ String logged = logToConsole(logPrefix, line);+ String logStream = stream == StreamType.STDERR ? stderrStream : stdoutStream;+ if (logged != null && logStream != null) {+ forwardToCloudWatchLogs(accountId, logGroup, logStream, region, logged);}};return follow(containerId, null,@@ -155,13 +151,46 @@ public class ContainerLogStreamer {* destination, such as an ECS container without an {@code awslogs} log configuration.*/public Closeable attachConsoleOnly(String containerId, String logPrefix) {- Consumer<String> emitter = line -> {- String trimmed = line.stripTrailing();- if (!trimmed.isEmpty()) {- LOG.infov("[{0}] {1}", logPrefix, trimmed);+ return follow(containerId, null, line -> logToConsole(logPrefix, line));+ }++ /**+ * A stream that logs each line written to it to the console logger as it arrives, as the attach methods+ * log a container's: for the output of a command run in a container. A line is held only until its+ * newline, or split at {@link LogLineBuffer}'s cap, so output without a newline holds no more than that.+ * Closing it logs a last line that has no newline.+ */+ public static OutputStream consoleLines(String logPrefix) {+ LogLineBuffer buffer = new LogLineBuffer();+ return new OutputStream() {+ @Override+ public synchronized void write(int b) {+ write(new byte[] {(byte) b}, 0, 1);+ }++ @Override+ public synchronized void write(byte[] bytes, int offset, int length) {+ byte[] payload = offset == 0 && length == bytes.length+ ? bytes : Arrays.copyOfRange(bytes, offset, offset + length);+ buffer.append(payload).forEach(line -> logToConsole(logPrefix, line));+ }++ @Override+ public synchronized void close() {+ buffer.flush().ifPresent(line -> logToConsole(logPrefix, line));}};- return follow(containerId, null, emitter);+ }++ // Logs a line of a container's output, prefixed, without its trailing whitespace; returns what it logged,+ // or null for a blank line, which it drops.+ private static String logToConsole(String logPrefix, String line) {+ String trimmed = line.stripTrailing();+ if (trimmed.isEmpty()) {+ return null;+ }+ LOG.infov("[{0}] {1}", logPrefix, trimmed);+ return trimmed;}private Closeable follow(String containerId, Instant since, Consumer<String> emitter) {diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/BootstrapRunner.java b/src/main/java/io/github/hectorvent/floci/services/emr/BootstrapRunner.javanew file mode 100644index 0000000000..cd07b93b1e--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/emr/BootstrapRunner.java@@ -0,0 +1,80 @@+package io.github.hectorvent.floci.services.emr;++import java.util.List;+import java.util.Set;+import java.util.function.BooleanSupplier;++/**+ * Runs a cluster's bootstrap actions in the EMR {@code docker} runner mode: one after the other, in one+ * container of the release's image, which is then kept as the image the cluster's steps run in. The+ * service calls it off the request thread; {@link EmrBootstrapRunner} is the Docker implementation, and a+ * test supplies its own.+ */+public interface BootstrapRunner {++ /**+ * Runs {@code bootstrap}'s actions in order, until one fails, and returns how they ended. Blocks until+ * then. A cluster {@code cancelled} reports as ending is stopped, and one it reports before the+ * container is up never starts.+ */+ Outcome run(Bootstrap bootstrap, BooleanSupplier cancelled);++ /** Stops the bootstrap of {@code clusterId}, now or as soon as its container is up. */+ void stop(String clusterId);++ /**+ * Removes {@code image}, which {@link #run} left, once its cluster no longer needs it: true when it is+ * gone, false when Docker refused, so the caller can try again.+ */+ boolean removeImage(String image);++ /**+ * Removes every image {@link #run} left in this deployment but those in {@code kept}, by id: after a restart,+ * the ones whose clusters are gone or have ended, and on a reset all of them.+ */+ void removeImages(Set<String> kept);++ /**+ * A cluster's bootstrap: its id, which names its container and its image; its account and Region; its+ * release, and the image template and Docker network to run it in (a null template is the release's+ * public EMR Serverless image, a null network the global default); the role of its EC2 instance profile,+ * whose credentials the actions get; its actions; and the S3 folder their logs go to, ending in a slash+ * (null for none), each action's in a folder of its own named by its number.+ */+ record Bootstrap(String clusterId, String accountId, String region, String releaseLabel, String imageTemplate,+ String dockerNetwork, String role, List<Action> actions, String logFolder) {+ }++ /** A bootstrap action: its name, the script it runs, and that script's arguments. */+ record Action(String name, String path, List<String> args) {+ }++ /**+ * How a bootstrap ended: every action succeeded, leaving the image of id {@code image}, which runs as it+ * is whatever registry Floci pulls images from; action {@code failedAction}, counted from 1, exited with+ * {@code exitCode}, or its script could not be read ({@code failure} says why); the bootstrap was+ * stopped; or it could not run at all ({@code failure} says why).+ */+ record Outcome(String image, Integer failedAction, Integer exitCode, boolean stopped, String failure) {++ public static Outcome succeeded(String image) {+ return new Outcome(image, null, null, false, null);+ }++ public static Outcome actionFailed(int action, int exitCode) {+ return new Outcome(null, action, exitCode, false, null);+ }++ public static Outcome notDownloaded(int action, String failure) {+ return new Outcome(null, action, null, false, failure);+ }++ public static Outcome halted() {+ return new Outcome(null, null, null, true, null);+ }++ public static Outcome couldNotRun(String failure) {+ return new Outcome(null, null, null, false, failure);+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrBootstrapRunner.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrBootstrapRunner.javanew file mode 100644index 0000000000..5039e7cdb8--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrBootstrapRunner.java@@ -0,0 +1,359 @@+package io.github.hectorvent.floci.services.emr;++import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.model.Image;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.core.common.ContainerTeardown;+import io.github.hectorvent.floci.core.common.RequestScopes;+import io.github.hectorvent.floci.core.common.S3Uri;+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import io.github.hectorvent.floci.core.common.docker.ContainerExec;+import io.github.hectorvent.floci.core.common.docker.ContainerLaunches;+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;+import io.github.hectorvent.floci.core.common.docker.ContainerSpec;+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;+import io.github.hectorvent.floci.core.common.docker.RetryingTarCopier;+import io.github.hectorvent.floci.services.emrserverless.EmrServerlessSparkRunner;+import io.github.hectorvent.floci.services.iam.model.SessionCreds;+import io.github.hectorvent.floci.services.lambda.launcher.LambdaExecutionRoleCredentials;+import io.github.hectorvent.floci.services.s3.S3Service;+import jakarta.enterprise.context.ApplicationScoped;+import org.jboss.logging.Logger;++import java.io.IOException;+import java.io.InputStream;+import java.io.OutputStream;+import java.io.UncheckedIOException;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.Set;+import java.util.function.BooleanSupplier;+import java.util.zip.GZIPOutputStream;++/**+ * Runs a cluster's bootstrap actions in a container of the release's EMR Serverless image, the one its+ * steps run in, then commits that container as the image the cluster's steps run in from then on, so what+ * the actions installed or wrote is there for every step. "Bootstrap actions execute as the Hadoop user by+ * default. You can execute a bootstrap action with root privileges by using sudo", but the image has no+ * {@code sudo}: an action runs as root instead, in the {@code hadoop} user's home, with a {@code sudo} that+ * runs its command as it is, and what it leaves in that home is the {@code hadoop} user's. Each action's+ * script is read from Floci's S3 as the cluster's account and runs through {@code docker exec}, with the+ * credentials of the cluster's EC2 instance profile in its environment alone, so that they never become+ * part of the image.+ */+@ApplicationScoped+public class EmrBootstrapRunner implements BootstrapRunner, ContainerTeardown {++ private static final Logger LOG = Logger.getLogger(EmrBootstrapRunner.class);+ private static final String SERVICE = "emr";+ private static final String COMPONENT = "emr-bootstrap";+ private static final int STOP_TIMEOUT_SECONDS = 0;+ // No source gives a bootstrap action a time limit: it runs until it ends, or its cluster does.+ private static final int NO_TIMEOUT = Integer.MAX_VALUE;+ // The sudo an action finds first on its PATH: its options are dropped, as the action already runs as root,+ // and its command, with any NAME=value before it, runs as it is.+ static final String SUDO = String.join("\n",+ "#!/bin/bash",+ "while [ $# -gt 0 ]; do",+ " case \"$1\" in",+ " -u|-g|-C|-D|-p|-r|-t|-T|-U) shift 2 ;;",+ " --) shift; break ;;",+ " -*) shift ;;",+ " *) break ;;",+ " esac",+ "done",+ "exec env \"$@\"");+ // What each action runs as root: $1 is its script, and the rest that script's arguments.+ private static final String ACTION_COMMAND = String.join("\n",+ "mkdir -p /tmp/floci-bin",+ "cat > /tmp/floci-bin/sudo <<'SUDO'",+ SUDO,+ "SUDO",+ "chmod 755 /tmp/floci-bin/sudo",+ "export PATH=\"/tmp/floci-bin:$PATH\" HOME=/home/hadoop",+ "cd /home/hadoop",+ "script=$1",+ "shift",+ "chmod u+x \"$script\"",+ "\"$script\" \"$@\"",+ "code=$?",+ "chown -R hadoop:hadoop /home/hadoop",+ "exit $code");+ // Run once every action has: what only the actions needed goes before the image is kept.+ private static final String CLEANUP_COMMAND = "rm -f /tmp/floci-bin/sudo /tmp/floci-bootstrap-action-*;"+ + " rmdir /tmp/floci-bin";++ // The bootstraps inside run(); a stop for one not here comes after its container has gone, and is dropped.+ private final ContainerLaunches launches = new ContainerLaunches();++ private final ContainerBuilder containerBuilder;+ private final ContainerLifecycleManager lifecycleManager;+ private final LaunchedContainerAwsEnv awsEnv;+ private final LambdaExecutionRoleCredentials roleCredentials;+ private final S3Service s3Service;+ private final EmulatorConfig config;++ public EmrBootstrapRunner(ContainerBuilder containerBuilder, ContainerLifecycleManager lifecycleManager,+ LaunchedContainerAwsEnv awsEnv, LambdaExecutionRoleCredentials roleCredentials,+ S3Service s3Service, EmulatorConfig config) {+ this.containerBuilder = containerBuilder;+ this.lifecycleManager = lifecycleManager;+ this.awsEnv = awsEnv;+ this.roleCredentials = roleCredentials;+ this.s3Service = s3Service;+ this.config = config;+ }++ @Override+ public Outcome run(Bootstrap bootstrap, BooleanSupplier cancelled) {+ ContainerLaunches.Launch launch = launches.begin(bootstrap.clusterId());+ Optional<SessionCreds> credentials = Optional.empty();+ try {+ if (cancelled.getAsBoolean()) {+ return Outcome.halted();+ }+ credentials = roleCredentials.forRole(bootstrap.role(), bootstrap.accountId(),+ bootstrap.clusterId() + "-bootstrap");+ String image = EmrServerlessSparkRunner.image(bootstrap.imageTemplate(), bootstrap.releaseLabel());+ Map<String, String> labels = labels(bootstrap);+ ContainerSpec spec = containerBuilder.newContainer(image)+ .withName(ContainerStorageHelper.dockerName(config,+ SERVICE + "-" + bootstrap.clusterId() + "-bootstrap"))+ .withUser("root")+ .withEntrypoint(List.of("sleep", "infinity"))+ .withDockerNetwork(Optional.ofNullable(bootstrap.dockerNetwork()))+ .withHostDockerInternalOnLinux()+ .withEmbeddedDns()+ .withLabels(labels)+ .build();+ String containerId = lifecycleManager.createAndStart(spec).containerId();+ launch.started(containerId);+ // A stop, a reset or a shutdown while the image was pulled is read now the container is up.+ if (launch.stopRequested() || cancelled.getAsBoolean()) {+ return Outcome.halted();+ }+ List<String> env = awsEnv.sdkBaselineEnv(bootstrap.region(), Optional.empty(), credentials,+ bootstrap.accountId());+ for (int number = 1; number <= bootstrap.actions().size(); number++) {+ Outcome failed = runAction(bootstrap, number, containerId, env);+ if (launch.stopRequested() || cancelled.getAsBoolean()) {+ return Outcome.halted();+ }+ if (failed != null) {+ return failed;+ }+ }+ DockerClient docker = lifecycleManager.getDockerClient();+ ContainerExec.run(docker, containerId, new String[] {"sh", "-c", CLEANUP_COMMAND}, 60);+ lifecycleManager.stop(containerId, STOP_TIMEOUT_SECONDS);+ return Outcome.succeeded(commit(docker, containerId, labels, bootstrap));+ } catch (RuntimeException e) {+ if (launch.stopRequested() || Thread.currentThread().isInterrupted()) {+ // A stop, a reset or a shutdown removed the container while an action ran.+ return Outcome.halted();+ }+ LOG.warnv(e, "The bootstrap actions of EMR cluster {0} could not run", bootstrap.clusterId());+ return Outcome.couldNotRun(e.getMessage());+ } finally {+ String unclaimed = launches.end(bootstrap.clusterId(), launch);+ if (unclaimed != null) {+ lifecycleManager.stopAndRemove(unclaimed, null, STOP_TIMEOUT_SECONDS);+ }+ credentials.ifPresent(session -> roleCredentials.unregister(bootstrap.accountId(),+ session.accessKeyId()));+ }+ }++ /**+ * Runs action {@code number} in the container, its output going to Floci's log and to the action's log+ * folder; null when it succeeded, or how it failed.+ */+ private Outcome runAction(Bootstrap bootstrap, int number, String containerId, List<String> env) {+ Action action = bootstrap.actions().get(number - 1);+ String script;+ try {+ script = placeScript(bootstrap, action, number, containerId);+ } catch (AwsException | IllegalArgumentException e) {+ return Outcome.notDownloaded(number, e.getMessage());+ }+ List<String> command = new ArrayList<>(List.of("bash", "-c", ACTION_COMMAND, "bootstrap-action", script));+ command.addAll(action.args());+ String prefix = SERVICE + ":" + bootstrap.clusterId() + ":bootstrap-action-" + number;+ boolean logged = bootstrap.logFolder() != null;+ ContainerExec.Result result;+ try (ActionOutput stdout = new ActionOutput(prefix, logged);+ ActionOutput stderr = new ActionOutput(prefix, logged)) {+ result = ContainerExec.run(lifecycleManager.getDockerClient(), containerId,+ command.toArray(String[]::new), env, NO_TIMEOUT, stdout, stderr);+ if (logged) {+ writeLogs(bootstrap, number, stdout, stderr);+ }+ } catch (IOException e) {+ throw new UncheckedIOException(e);+ }+ return result.exitCode() == 0 ? null : Outcome.actionFailed(number, (int) result.exitCode());+ }++ /**+ * Where in the container action {@code number}'s script is: one in S3 is read as the cluster's account+ * and copied in, and a path that is not in S3 is the image's own.+ */+ private String placeScript(Bootstrap bootstrap, Action action, int number, String containerId) {+ if (!S3Uri.isS3(action.path())) {+ return action.path().startsWith("file:") ? action.path().substring("file:".length()) : action.path();+ }+ S3Uri location = S3Uri.parseAnyScheme(action.path());+ if (location.key().isEmpty()) {+ throw new IllegalArgumentException("The path " + action.path() + " names no object.");+ }+ byte[] content = RequestScopes.callAs(bootstrap.accountId(), bootstrap.region(),+ () -> s3Service.getObject(location.bucket(), location.key()).getData());+ String name = "floci-bootstrap-action-" + number + "-"+ + location.key().substring(location.key().lastIndexOf('/') + 1);+ RetryingTarCopier.copyBytes(lifecycleManager.getDockerClient(), containerId, "/tmp", name, content, 0755);+ return "/tmp/" + name;+ }++ // An action's output goes to its own folder under the cluster's: bootstrap-actions/<number>/.+ private void writeLogs(Bootstrap bootstrap, int number, ActionOutput stdout, ActionOutput stderr) {+ String folder = bootstrap.logFolder() + number + "/";+ try (InputStream stdoutGzip = stdout.finish(); InputStream stderrGzip = stderr.finish()) {+ EmrServerlessSparkRunner.putLogs(s3Service, bootstrap.accountId(), bootstrap.region(), folder,+ stdoutGzip, stderrGzip);+ } catch (IOException | RuntimeException e) {+ LOG.warnv("Could not write the logs of bootstrap action {0} of EMR cluster {1} to {2}: {3}", number,+ bootstrap.clusterId(), folder, e.getMessage());+ }+ }++ /**+ * One of an action's output streams, as it arrives: each line to Floci's log and, when the cluster has a+ * log folder, gzipped into a file of its own, so that however much an action writes, Floci keeps no more+ * than a line of it in memory. Closing it deletes the file.+ */+ private static final class ActionOutput extends OutputStream {++ private final OutputStream console;+ private final Path file;+ private final GZIPOutputStream gzip;++ ActionOutput(String logPrefix, boolean kept) throws IOException {+ console = ContainerLogStreamer.consoleLines(logPrefix);+ file = kept ? Files.createTempFile("floci-emr-bootstrap-", ".gz") : null;+ try {+ gzip = kept ? new GZIPOutputStream(Files.newOutputStream(file)) : null;+ } catch (IOException e) {+ Files.delete(file);+ throw e;+ }+ }++ @Override+ public synchronized void write(int b) throws IOException {+ write(new byte[] {(byte) b}, 0, 1);+ }++ @Override+ public synchronized void write(byte[] bytes, int offset, int length) throws IOException {+ console.write(bytes, offset, length);+ if (gzip != null) {+ gzip.write(bytes, offset, length);+ }+ }++ /** The output has ended: its last line is logged, and its gzipped file is read from the start. */+ synchronized InputStream finish() throws IOException {+ console.close();+ gzip.close();+ return Files.newInputStream(file);+ }++ @Override+ public synchronized void close() throws IOException {+ try {+ console.close();+ if (gzip != null) {+ gzip.close();+ }+ } finally {+ if (file != null) {+ Files.deleteIfExists(file);+ }+ }+ }+ }++ /**+ * Commits the stopped container as the image the cluster's steps run in, and returns its id. The image+ * takes the container's configuration where the commit sets none, so its user, which the actions changed+ * to root, is set back to that of the image the container ran, the one the steps run as. The steps get+ * the id, which a container runs as it is, so the image-registry-base Floci prefixes image names with+ * does not send them to a registry for an image only this daemon has.+ */+ private String commit(DockerClient docker, String containerId, Map<String, String> labels, Bootstrap bootstrap) {+ String released = docker.inspectContainerCmd(containerId).exec().getImageId();+ String user = docker.inspectImageCmd(released).exec().getConfig().getUser();+ String repository = ContainerStorageHelper.dockerName(config, COMPONENT);+ lifecycleManager.commit(containerId, repository, bootstrap.clusterId(),+ user == null || user.isBlank() ? "root" : user, labels);+ String tag = repository + ":" + bootstrap.clusterId();+ String committed = docker.inspectImageCmd(tag).exec().getId();+ LOG.infov("Kept the bootstrap of EMR cluster {0} as image {1} ({2})", bootstrap.clusterId(), tag, committed);+ return committed;+ }++ @Override+ public void stop(String clusterId) {+ ContainerLaunches.Launch launch = launches.get(clusterId);+ if (launch != null) {+ // Removing the container ends the action running in it, which run() then reads as stopped.+ launch.stopNow(lifecycleManager, STOP_TIMEOUT_SECONDS);+ }+ }++ /** Stops every bootstrap inside {@link #run}, so a reset or a shutdown orphans none. */+ @Override+ public void stopManagedContainers() {+ launches.stopAll(lifecycleManager, STOP_TIMEOUT_SECONDS);+ }++ @Override+ public boolean removeImage(String image) {+ return lifecycleManager.removeImage(image);+ }++ @Override+ public void removeImages(Set<String> kept) {+ try {+ List<Image> images = lifecycleManager.getDockerClient().listImagesCmd()+ .withLabelFilter(Map.of(ContainerStorageHelper.COMPONENT_LABEL, COMPONENT,+ ContainerStorageHelper.OWNER_LABEL, ContainerStorageHelper.ownerIdentity(config)))+ .exec();+ for (Image image : images) {+ if (!kept.contains(image.getId())) {+ removeImage(image.getId());+ }+ }+ } catch (RuntimeException e) {+ LOG.warnv("Could not remove the bootstrap images of EMR clusters that have ended: {0}", e.getMessage());+ }+ }++ // The container's and the image's labels: the cluster's, and the deployment's that runs it.+ private Map<String, String> labels(Bootstrap bootstrap) {+ Map<String, String> labels = new LinkedHashMap<>(ContainerStorageHelper.resourceIdentityLabels(SERVICE,+ bootstrap.clusterId(), bootstrap.accountId(), bootstrap.region()));+ labels.put(ContainerStorageHelper.COMPONENT_LABEL, COMPONENT);+ labels.put(ContainerStorageHelper.OWNER_LABEL, ContainerStorageHelper.ownerIdentity(config));+ return labels;+ }++}diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.javaindex abc18e16fd..f8d42dbbc2 100644--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.java+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrClusterLifecycle.java@@ -10,6 +10,7 @@ import java.time.Instant;import java.util.ArrayList;import java.util.Comparator;import java.util.List;+import java.util.Set;import java.util.regex.Matcher;import java.util.regex.Pattern;@@ -22,7 +23,9 @@ import java.util.regex.Pattern;* every change due by a given time, in the order they fall due, queuing the event of each state the* guide's event tables report. With the durations at 0, every change falls due at the time of the call* that leads to it. Without a {@link StepExecution} nothing runs and a step completes by time; with one,- * a step runs for real, and ends, COMPLETED or FAILED, when {@link #stepEnded} says so.+ * a step runs for real, and ends, COMPLETED or FAILED, when {@link #stepEnded} says so. Likewise, with a+ * {@link BootstrapExecution} a cluster with bootstrap actions runs them for real, and is BOOTSTRAPPING until+ * {@link #bootstrapEnded} says they are done.*/final class EmrClusterLifecycle {@@ -41,6 +44,10 @@ final class EmrClusterLifecycle {// the failure, it returns "Unknown Error" as a reason." Floci never determines it.private static final String UNKNOWN_ERROR = "Unknown Error";private static final String STEP_FAILURE = "STEP_FAILURE";+ static final String BOOTSTRAP_FAILURE = "BOOTSTRAP_FAILURE";+ static final String INTERNAL_ERROR = "INTERNAL_ERROR";+ // "If a cluster terminates because of a failure, ... the cluster state is set to TERMINATED_WITH_ERRORS."+ private static final Set<String> FAILURES = Set.of(STEP_FAILURE, BOOTSTRAP_FAILURE, INTERNAL_ERROR);// The reason a cluster a failed step ends gives: the console's for a Snowplow cluster on its forum, and// mrjob's mock EMR.private static final String SHUT_DOWN_AS_STEP_FAILED = "Shut down as step failed";@@ -52,6 +59,8 @@ final class EmrClusterLifecycle {private final EmrEventPublisher events;// Null when steps complete by time.private final StepExecution execution;+ // Null when a cluster's bootstrap actions are done by time.+ private final BootstrapExecution bootstrapping;/*** Runs the steps a cluster starts, in the {@code docker} runner mode. Each call only sets work going@@ -66,13 +75,27 @@ final class EmrClusterLifecycle {void stop(EmrStep step);}+ /**+ * Runs the bootstrap actions of the clusters that have some, in the {@code docker} runner mode. Each call+ * only sets work going and returns: their end comes back later through {@link #bootstrapEnded}.+ */+ interface BootstrapExecution {++ /** {@code cluster}, which has bootstrap actions, has just gone BOOTSTRAPPING. */+ void start(EmrCluster cluster, String region);++ /** {@code cluster} is ending while its bootstrap actions still run. */+ void stop(EmrCluster cluster);+ }+EmrClusterLifecycle(int startupDelaySeconds, int stepDurationSeconds, int terminationDelaySeconds,- EmrEventPublisher events, StepExecution execution) {+ EmrEventPublisher events, StepExecution execution, BootstrapExecution bootstrapping) {this.startup = seconds("cluster-startup-delay-seconds", startupDelaySeconds);this.stepRun = seconds("step-duration-seconds", stepDurationSeconds);this.termination = seconds("cluster-termination-delay-seconds", terminationDelaySeconds);this.events = events;this.execution = execution;+ this.bootstrapping = bootstrapping;}private static Duration seconds(String setting, int value) {@@ -116,6 +139,9 @@ final class EmrClusterLifecycle {* "When a job flow is shut down, any step not yet completed is canceled".*/void terminate(EmrCluster cluster, String code, String message, Instant now, String region) {+ if (BOOTSTRAPPING.equals(cluster.getState()) && !bootstrapTimed(cluster)) {+ bootstrapping.stop(cluster);+ }cluster.setState(TERMINATING);cluster.setStateChangeReasonCode(code);cluster.setStateChangeMessage(message);@@ -169,7 +195,7 @@ final class EmrClusterLifecycle {Instant nextChange(EmrCluster cluster) {return switch (cluster.getState()) {case STARTING -> cluster.getCreationDateTime().plus(startup.dividedBy(2));- case BOOTSTRAPPING -> cluster.getCreationDateTime().plus(startup);+ case BOOTSTRAPPING -> bootstrapTimed(cluster) ? cluster.getCreationDateTime().plus(startup) : null;case RUNNING -> cluster.getSteps().stream().filter(step -> RUNNING.equals(step.getState()) && timed(step)).map(step -> step.getStartDateTime().plus(stepRun))@@ -182,12 +208,14 @@ final class EmrClusterLifecycle {private void change(EmrCluster cluster, Instant at, String region) {switch (cluster.getState()) {- case STARTING -> cluster.setState(BOOTSTRAPPING);- case BOOTSTRAPPING -> {- cluster.setReadyDateTime(at);- cluster.setState(RUNNING);- runSteps(cluster, at, region);+ case STARTING -> {+ cluster.setState(BOOTSTRAPPING);+ if (bootstrapping != null && cluster.getBootstrapActions() != null) {+ cluster.setBootstrapInContainer(true);+ bootstrapping.start(cluster, region);+ }}+ case BOOTSTRAPPING -> ready(cluster, at, region);case RUNNING -> {for (EmrStep step : cluster.getSteps()) {if (RUNNING.equals(step.getState()) && timed(step)@@ -200,9 +228,7 @@ final class EmrClusterLifecycle {runSteps(cluster, at, region);}case TERMINATING -> {- // "If a cluster terminates because of a failure, ... the cluster state is set to- // TERMINATED_WITH_ERRORS."- String ended = STEP_FAILURE.equals(cluster.getStateChangeReasonCode())+ String ended = FAILURES.contains(cluster.getStateChangeReasonCode())? TERMINATED_WITH_ERRORS : TERMINATED;cluster.setState(ended);cluster.setEndDateTime(at);@@ -213,6 +239,29 @@ final class EmrClusterLifecycle {}}+ /**+ * The bootstrap actions a cluster ran for real have ended: when {@code failureCode} is null they all+ * succeeded and left {@code image}, which its steps run in, and the cluster is ready to run them;+ * otherwise the cluster ends, for {@code failureCode} and {@code message}.+ */+ void bootstrapEnded(EmrCluster cluster, String image, String failureCode, String message, Instant at,+ String region) {+ cluster.setBootstrapInContainer(false);+ if (failureCode != null) {+ terminate(cluster, failureCode, message, at, region);+ return;+ }+ cluster.setBootstrapImage(image);+ ready(cluster, at, region);+ }++ // Bootstrapping is done: the cluster runs its steps.+ private void ready(EmrCluster cluster, Instant at, String region) {+ cluster.setReadyDateTime(at);+ cluster.setState(RUNNING);+ runSteps(cluster, at, region);+ }+/*** Starts PENDING steps in the order they were added while the cluster has room: "pending steps in* queue transition to running state in the order they were submitted". With none left, the@@ -336,6 +385,15 @@ final class EmrClusterLifecycle {return execution == null || !step.isRunInContainer();}+ /**+ * Whether a BOOTSTRAPPING cluster is done by time: every one without a {@link BootstrapExecution} or+ * without bootstrap actions, including one a {@code docker} runner mode started before a restart into the+ * {@code immediate} one.+ */+ private boolean bootstrapTimed(EmrCluster cluster) {+ return bootstrapping == null || !cluster.isBootstrapInContainer();+ }+/** TERMINATED, or TERMINATED_WITH_ERRORS: the cluster has ended. */static boolean isTerminated(String state) {return TERMINATED.equals(state) || TERMINATED_WITH_ERRORS.equals(state);diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrHandler.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrHandler.javaindex 342eefd5da..5cc7e24e4e 100644--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrHandler.java+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrHandler.java@@ -742,7 +742,7 @@ public class EmrHandler {boolean terminated) {ObjectNode node = objectMapper.createObjectNode();node.put("Id", "ci-" + index + cluster.getId());- node.put("Ec2InstanceId", "i-" + String.format("%017d", index));+ node.put("Ec2InstanceId", EmrService.ec2InstanceId(index));node.put("PublicDnsName", AwsEndpoints.ec2PublicDns("203.0.113." + index, cluster.getRegion()));String privateIp = "10.0.0." + index;node.put("PrivateDnsName", AwsRegions.ec2PrivateIpDnsName(privateIp, cluster.getRegion()));diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.javaindex d6453dc24b..51ebb25d29 100644--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java@@ -1,5 +1,6 @@package io.github.hectorvent.floci.services.emr;+import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.core.type.TypeReference;import com.fasterxml.jackson.databind.JsonNode;import com.fasterxml.jackson.databind.ObjectMapper;@@ -69,7 +70,8 @@ import java.util.function.Supplier;* <p>In the default {@code immediate} runner mode nothing runs. In the {@code docker} one each step* runs, in a container of its own, from when it goes RUNNING until its Spark driver exits, on a thread* of its own that ends the step under the service's lock: COMPLETED on exit code 0, FAILED otherwise,- * with its ActionOnFailure taken.+ * with its ActionOnFailure taken. A cluster's bootstrap actions run there too, while it is+ * BOOTSTRAPPING, and leave the image its steps then run in.*/@ApplicationScopedpublic class EmrService implements Resettable {@@ -85,6 +87,15 @@ public class EmrService implements Resettable {private static final String RUNNER_DOCKER = "docker";// The service a step's driver belongs to, which names it and labels it.private static final String SERVICE = "emr";+ // The reason of a cluster a failed bootstrap action ended, as Terraform reported a real one's+ // (hashicorp/terraform-provider-aws#4378) and the Management Guide's VPC errors page quotes it.+ private static final String BOOTSTRAP_ACTION_FAILED =+ "On the master instance (%s), bootstrap action %d returned a non-zero return code";+ // No source gives AWS's wording for these; they are Floci's.+ private static final String BOOTSTRAP_ACTION_NOT_READ =+ "On the master instance (%s), bootstrap action %d could not be read from %s: %s";+ private static final String BOOTSTRAP_NOT_RUN = "Floci could not run the bootstrap actions: %s";+ static final String BOOTSTRAP_INTERRUPTED = "Floci stopped while the cluster's bootstrap actions were running.";private final AccountAwareStorageBackend<EmrCluster> clusterStore;private final StorageBackend<String, SecurityConfiguration> secConfigStore;@@ -99,6 +110,7 @@ public class EmrService implements Resettable {private final Map<String, Instant> wakeupsDue = new ConcurrentHashMap<>();// The docker runner mode's; null in the immediate one.private final SparkJobRunner sparkRunner;+ private final BootstrapRunner bootstrapRunner;private final Executor stepExecutor;private final IamService iamService;// Where a step's driver runs: image-template, with the release label for %s, and docker-network; null when@@ -107,6 +119,11 @@ public class EmrService implements Resettable {private final String dockerNetwork;// The steps a thread is running, by account and id; a RUNNING container step not here was left by a restart.private final Set<String> executing = new HashSet<>();+ // Likewise the clusters a thread is bootstrapping.+ private final Set<String> bootstrapping = new HashSet<>();+ // The images of ended clusters a thread is removing. A cluster names its image until it is gone, so one+ // Docker refused to remove is tried again the next time the cluster is read or stored.+ private final Set<String> imagesBeingRemoved = new HashSet<>();// Raised by each reset, so a step's thread from before it writes nothing back after the wipe.private long generation;@@ -170,22 +187,25 @@ public class EmrService implements Resettable {@Injectpublic EmrService(StorageFactory storageFactory, EmulatorConfig config, RegionResolver regionResolver,- EmrEventPublisher eventPublisher, EmrServerlessSparkRunner sparkRunner, IamService iamService) {+ EmrEventPublisher eventPublisher, EmrServerlessSparkRunner sparkRunner,+ EmrBootstrapRunner bootstrapRunner, IamService iamService) {this(storageFactory, config, regionResolver, eventPublisher, Clock.systemUTC(),Wakeups.timer(Clock.systemUTC()),RUNNER_DOCKER.equalsIgnoreCase(config.services().emr().runnerMode()) ? sparkRunner : null,+ RUNNER_DOCKER.equalsIgnoreCase(config.services().emr().runnerMode()) ? bootstrapRunner : null,Executors.newCachedThreadPool(Thread.ofPlatform().daemon().name("emr-step-", 0).factory()),iamService);}EmrService(StorageFactory storageFactory, EmulatorConfig config, RegionResolver regionResolver,EmrEventPublisher eventPublisher, Clock clock, Wakeups wakeups) {- this(storageFactory, config, regionResolver, eventPublisher, clock, wakeups, null, Runnable::run, null);+ this(storageFactory, config, regionResolver, eventPublisher, clock, wakeups, null, null, Runnable::run,+ null);}EmrService(StorageFactory storageFactory, EmulatorConfig config, RegionResolver regionResolver,EmrEventPublisher eventPublisher, Clock clock, Wakeups wakeups, SparkJobRunner sparkRunner,- Executor stepExecutor, IamService iamService) {+ BootstrapRunner bootstrapRunner, Executor stepExecutor, IamService iamService) {this.clusterStore = storageFactory.create("emr", "emr-clusters.json",new TypeReference<Map<String, EmrCluster>>() {});this.secConfigStore = storageFactory.create("emr", "emr-security-configs.json",@@ -196,6 +216,7 @@ public class EmrService implements Resettable {this.eventPublisher = eventPublisher;this.defaultReleaseLabel = config.services().emr().defaultReleaseLabel();this.sparkRunner = sparkRunner;+ this.bootstrapRunner = bootstrapRunner;this.stepExecutor = stepExecutor;this.iamService = iamService;this.imageTemplate = config.services().emr().imageTemplate().orElse(null);@@ -203,7 +224,8 @@ public class EmrService implements Resettable {this.lifecycle = new EmrClusterLifecycle(config.services().emr().clusterStartupDelaySeconds(),config.services().emr().stepDurationSeconds(),config.services().emr().clusterTerminationDelaySeconds(), eventPublisher,- sparkRunner != null ? new ContainerSteps() : null);+ sparkRunner != null ? new ContainerSteps() : null,+ bootstrapRunner != null ? new ContainerBootstrap() : null);this.clock = clock;this.wakeups = wakeups;// Before any step of this process can start, the next step a restarted cluster's wakeup below starts@@ -213,15 +235,28 @@ public class EmrService implements Resettable {sparkRunner.removeLeftoverDrivers(SERVICE);}// After a restart, a stored cluster with a change still to come wakes the service as before, and one- // with a step a container ran, which no thread runs any more, wakes it at once to fail that step.+ // with a step a container ran, or bootstrap actions, which no thread runs any more, wakes it at once to+ // fail them. The images bootstrap left for clusters that are gone, or have ended, go first: a wakeup+ // runs on the timer's thread, and a cluster it moved on could keep an image the sweep never knew of.+ Set<String> imagesInUse = new HashSet<>();+ List<Runnable> wakeAtStart = new ArrayList<>();for (AccountAwareStorageBackend.AccountEntry<EmrCluster> entry: clusterStore.scanAllAccountEntries(k -> true)) {- Instant due = sparkRunner != null && hasContainerStepRunning(entry.value())- ? clock.instant() : lifecycle.nextChange(entry.value());+ EmrCluster cluster = entry.value();+ boolean interrupted = sparkRunner != null && hasContainerStepRunning(cluster)+ || bootstrapRunner != null && bootstrapsInContainer(cluster);+ Instant due = interrupted ? clock.instant() : lifecycle.nextChange(cluster);if (due != null) {- wakeAt(entry.accountId(), entry.value(), due);+ wakeAtStart.add(() -> wakeAt(entry.accountId(), cluster, due));}+ if (cluster.getBootstrapImage() != null && !EmrClusterLifecycle.isTerminated(cluster.getState())) {+ imagesInUse.add(cluster.getBootstrapImage());+ }+ }+ if (bootstrapRunner != null && config.services().emr().reconcileContainersOnStartup()) {+ bootstrapRunner.removeImages(imagesInUse);}+ wakeAtStart.forEach(Runnable::run);}// A normal-scoped bean is built on first use, so without this a cluster left mid-change by a restart@@ -239,13 +274,18 @@ public class EmrService implements Resettable {}/**- * A reset has begun: the steps' threads from before it write nothing more, so the storage wipe that- * follows leaves none of their clusters behind. Its teardown has already stopped their drivers.+ * A reset has begun: the steps' and bootstraps' threads from before it write nothing more, so the storage+ * wipe that follows leaves none of their clusters behind, and the images bootstrap left go with them. Its+ * teardown has already stopped their containers.*/@Overridepublic synchronized void beforeReset() {generation++;executing.clear();+ bootstrapping.clear();+ if (bootstrapRunner != null) {+ bootstrapRunner.removeImages(Set.of());+ }}@Override@@ -1091,14 +1131,66 @@ public class EmrService implements Resettable {* container ran that no thread runs any more, since Floci stopped while it ran, is FAILED.*/private EmrCluster settle(EmrCluster cluster) {- boolean interrupted = failInterruptedSteps(cluster);- if (lifecycle.advance(cluster, now(), regionOf(cluster)) || interrupted) {+ boolean interrupted = failInterruptedSteps(cluster) | failInterruptedBootstrap(cluster);+ boolean changed = lifecycle.advance(cluster, now(), regionOf(cluster)) || interrupted;+ if (changed) {clusterStore.put(cluster.getId(), cluster);}+ releaseBootstrapImage(cluster);wakeForNextChange(cluster);return cluster;}+ /** A cluster whose bootstrap no thread runs any more, since Floci stopped while it ran, ends. */+ private boolean failInterruptedBootstrap(EmrCluster cluster) {+ if (bootstrapRunner == null || !bootstrapsInContainer(cluster)+ || bootstrapping.contains(clusterKey(regionResolver.getAccountId(), cluster.getId()))) {+ return false;+ }+ lifecycle.bootstrapEnded(cluster, null, EmrClusterLifecycle.INTERNAL_ERROR, BOOTSTRAP_INTERRUPTED, now(),+ regionOf(cluster));+ return true;+ }++ private static boolean bootstrapsInContainer(EmrCluster cluster) {+ return EmrClusterLifecycle.BOOTSTRAPPING.equals(cluster.getState()) && cluster.isBootstrapInContainer();+ }++ /**+ * Once a cluster has ended and no step of it still runs in a container, the image its bootstrap left+ * goes, off the service's lock. The cluster names it until it is gone, and while a thread removes it, it+ * is not removed again.+ */+ private void releaseBootstrapImage(EmrCluster cluster) {+ String image = cluster.getBootstrapImage();+ if (image == null || bootstrapRunner == null || !EmrClusterLifecycle.isTerminated(cluster.getState())) {+ return;+ }+ String accountId = regionResolver.getAccountId();+ if (cluster.getSteps().stream().anyMatch(step -> executing.contains(stepKey(accountId, step.getId())))+ || !imagesBeingRemoved.add(image)) {+ return;+ }+ String clusterId = cluster.getId();+ String region = regionOf(cluster);+ stepExecutor.execute(() -> {+ boolean gone = bootstrapRunner.removeImage(image);+ RequestScopes.runAs(accountId, region, () -> run(() -> bootstrapImageRemoved(clusterId, image, gone)));+ });+ }++ // A thread's removal of a cluster's image has ended: the cluster names it no more once it is gone.+ private void bootstrapImageRemoved(String clusterId, String image, boolean gone) {+ imagesBeingRemoved.remove(image);+ if (!gone) {+ return;+ }+ clusterStore.get(clusterId).filter(cluster -> image.equals(cluster.getBootstrapImage())).ifPresent(cluster -> {+ cluster.setBootstrapImage(null);+ clusterStore.put(clusterId, cluster);+ });+ }+private boolean failInterruptedSteps(EmrCluster cluster) {if (sparkRunner == null) {return false;@@ -1125,6 +1217,10 @@ public class EmrService implements Resettable {return accountId + "/" + stepId;}+ private static String clusterKey(String accountId, String clusterId) {+ return accountId + "/" + clusterId;+ }+/*** The role a step's driver runs as: the step's runtime role, or else the role of the cluster's EC2* instance profile, its {@code JobFlowRole}, given by name or ARN; null when IAM knows neither, and the@@ -1133,9 +1229,12 @@ public class EmrService implements Resettable {* policy attached to the instance profile that you used to launch the cluster".*/private String executionRole(EmrCluster cluster, EmrStep step, String accountId) {- if (step.getExecutionRoleArn() != null) {- return step.getExecutionRoleArn();- }+ return step.getExecutionRoleArn() != null+ ? step.getExecutionRoleArn() : instanceProfileRole(cluster, accountId);+ }++ /** The role of the cluster's EC2 instance profile, its {@code JobFlowRole}, given by name or ARN, or null. */+ private String instanceProfileRole(EmrCluster cluster, String accountId) {String profile = cluster.getJobFlowRole();if (profile == null || iamService == null) {return null;@@ -1161,6 +1260,8 @@ public class EmrService implements Resettable {EmrStep step = found.flatMap(cluster -> cluster.getSteps().stream().filter(candidate -> launch.stepId().equals(candidate.getId())).findFirst()).orElse(null);if (step == null || !EmrClusterLifecycle.RUNNING.equals(step.getState())) {+ // A step of a cluster that has ended no longer runs: the cluster's image may go with it, its last one.+ found.ifPresent(this::releaseBootstrapImage);return;}EmrCluster cluster = found.get();@@ -1178,6 +1279,7 @@ public class EmrService implements Resettable {private void driverStopped(StepLaunch launch) {if (!stillRunning(launch)) {executing.remove(stepKey(launch.accountId(), launch.stepId()));+ clusterStore.get(launch.clusterId()).ifPresent(this::releaseBootstrapImage);}}@@ -1215,7 +1317,8 @@ public class EmrService implements Resettable {List<String> command = application == null ? EmrSparkSteps.command(step) : null;SparkJobRunner.Job job = application == null && command == null ? null: new SparkJobRunner.Job(SERVICE, step.getId(), accountId, region, cluster.getReleaseLabel(),- imageTemplate, dockerNetwork, executionRole(cluster, step, accountId),+ cluster.getBootstrapImage() != null ? cluster.getBootstrapImage() : imageTemplate,+ dockerNetwork, executionRole(cluster, step, accountId),EmrSparkSteps.sparkDefaults(cluster.getConfigurations(), SNAPSHOT_MAPPER),application != null ? application : List.of(), command, Duration.ZERO,EmrSparkSteps.logFolder(cluster, step), null);@@ -1252,6 +1355,136 @@ public class EmrService implements Resettable {RequestScopes.runAs(launch.accountId(), launch.region(), () -> run(() -> stepEnded(launch, failure)));}+ /** A cluster handed to a bootstrap thread: whose it is, where, and under which reset. */+ private record BootstrapLaunch(String accountId, String region, String clusterId, long launchedIn) {}++ /** Runs each cluster's bootstrap actions on a thread of its own, which ends its BOOTSTRAPPING. */+ private final class ContainerBootstrap implements EmrClusterLifecycle.BootstrapExecution {++ @Override+ public void start(EmrCluster cluster, String region) {+ String accountId = regionResolver.getAccountId();+ // "logs/<cluster-id>/node/<primary-instance-id>/bootstrap-actions/<action-number>/stdout.gz"+ String logFolder = EmrServerlessSparkRunner.logFolder(cluster.getLogUri(),+ cluster.getId() + "/node/" + masterInstanceId(cluster) + "/bootstrap-actions/");+ BootstrapRunner.Bootstrap bootstrap = new BootstrapRunner.Bootstrap(cluster.getId(), accountId, region,+ cluster.getReleaseLabel(), imageTemplate, dockerNetwork, instanceProfileRole(cluster, accountId),+ bootstrapActions(cluster), logFolder);+ BootstrapLaunch launch = new BootstrapLaunch(accountId, region, cluster.getId(), generation);+ bootstrapping.add(clusterKey(accountId, cluster.getId()));+ stepExecutor.execute(() -> runBootstrap(bootstrap, launch));+ }++ @Override+ public void stop(EmrCluster cluster) {+ // Off the service's lock, as stopping removes the container; the bootstrap reads the end itself too.+ String id = cluster.getId();+ stepExecutor.execute(() -> bootstrapRunner.stop(id));+ }+ }++ // A cluster's BootstrapActionConfig list, as RunJobFlow stored it once it had checked it.+ private static List<BootstrapRunner.Action> bootstrapActions(EmrCluster cluster) {+ List<BootstrapRunner.Action> actions = new ArrayList<>();+ try {+ for (JsonNode action : SNAPSHOT_MAPPER.readTree(cluster.getBootstrapActions())) {+ JsonNode script = action.path("ScriptBootstrapAction");+ List<String> args = new ArrayList<>();+ script.path("Args").forEach(arg -> args.add(arg.asText()));+ actions.add(new BootstrapRunner.Action(action.path("Name").asText(), script.path("Path").asText(),+ args));+ }+ } catch (JsonProcessingException ignored) {+ // RunJobFlow stored the list as it came, and checked that it is one.+ }+ return actions;+ }++ // A cluster's bootstrap thread: its actions run to their end, which then ends its BOOTSTRAPPING.+ private void runBootstrap(BootstrapRunner.Bootstrap bootstrap, BootstrapLaunch launch) {+ BootstrapRunner.Outcome outcome;+ try {+ outcome = bootstrapRunner.run(bootstrap, () -> !RequestScopes.callAs(launch.accountId(), launch.region(),+ () -> stillBootstrapping(launch)));+ } catch (RuntimeException e) {+ LOG.warnv(e, "The bootstrap actions of EMR cluster {0} could not run", launch.clusterId());+ outcome = BootstrapRunner.Outcome.couldNotRun(e.getMessage());+ }+ BootstrapRunner.Outcome ended = outcome;+ RequestScopes.runAs(launch.accountId(), launch.region(),+ () -> run(() -> bootstrapEnded(bootstrap, launch, ended)));+ }++ // Whether a cluster's bootstrap may still start, or go on: one that ended, or a reset removed, is past it.+ private synchronized boolean stillBootstrapping(BootstrapLaunch launch) {+ return launch.launchedIn() == generation+ && clusterStore.get(launch.clusterId()).filter(EmrService::bootstrapsInContainer).isPresent();+ }++ /**+ * Ends a cluster's BOOTSTRAPPING as its bootstrap ended: RUNNING when every action succeeded, or ended,+ * with errors, when one failed or they could not run. A cluster that ended meanwhile, or that a reset+ * since removed, stays as it is, and an image its bootstrap left goes. One Floci itself stopped, by a+ * reset's teardown or its shutdown, is left BOOTSTRAPPING and still counted as bootstrapping: the reset+ * wipes it, and a restart ends it.+ */+ private void bootstrapEnded(BootstrapRunner.Bootstrap bootstrap, BootstrapLaunch launch,+ BootstrapRunner.Outcome outcome) {+ Optional<EmrCluster> current = launch.launchedIn() == generation+ ? findCluster(launch.clusterId()) : Optional.empty();+ Optional<EmrCluster> found = current.filter(EmrService::bootstrapsInContainer);+ if (outcome.stopped() && found.isPresent()) {+ return;+ }+ bootstrapping.remove(clusterKey(launch.accountId(), launch.clusterId()));+ if (found.isEmpty()) {+ if (outcome.image() != null && current.isPresent()) {+ // The cluster ended meanwhile: the image goes as an ended cluster's does.+ current.get().setBootstrapImage(outcome.image());+ store(current.get());+ } else if (outcome.image() != null) {+ stepExecutor.execute(() -> bootstrapRunner.removeImage(outcome.image()));+ }+ return;+ }+ EmrCluster cluster = found.get();+ String master = masterInstanceId(cluster);+ String code = EmrClusterLifecycle.BOOTSTRAP_FAILURE;+ String message;+ if (outcome.image() != null) {+ code = null;+ message = null;+ } else if (outcome.exitCode() != null) {+ message = BOOTSTRAP_ACTION_FAILED.formatted(master, outcome.failedAction());+ } else if (outcome.failedAction() != null) {+ message = BOOTSTRAP_ACTION_NOT_READ.formatted(master, outcome.failedAction(),+ bootstrap.actions().get(outcome.failedAction() - 1).path(), outcome.failure());+ } else {+ code = EmrClusterLifecycle.INTERNAL_ERROR;+ message = BOOTSTRAP_NOT_RUN.formatted(outcome.failure());+ }+ lifecycle.bootstrapEnded(cluster, outcome.image(), code, message, now(), regionOf(cluster));+ store(cluster);+ LOG.infov("The bootstrap actions of EMR cluster {0} ended; it is {1}", cluster.getId(), cluster.getState());+ }++ /** The EC2 instance id ListInstances gives a cluster's instance {@code index}, counted from 1. */+ static String ec2InstanceId(int index) {+ return "i-" + String.format("%017d", index);+ }++ // The primary instance's: the first of the MASTER group in the order ListInstances lists them.+ static String masterInstanceId(EmrCluster cluster) {+ int index = 1;+ for (EmrInstanceGroup group : cluster.getInstanceGroups()) {+ if ("MASTER".equals(group.getInstanceGroupType())) {+ return ec2InstanceId(index);+ }+ index += group.getRunningInstanceCount();+ }+ return ec2InstanceId(1);+ }+/** What a step's thread asks the service while its driver goes: a step has no timeout to report. */private final class StepProgress implements SparkJobRunner.Progress {@@ -1280,6 +1513,7 @@ public class EmrService implements Resettable {/** Stores a cluster a call changed, once every change that falls due by now is made. */private void store(EmrCluster cluster) {lifecycle.advance(cluster, now(), regionOf(cluster));+ releaseBootstrapImage(cluster);clusterStore.put(cluster.getId(), cluster);wakeForNextChange(cluster);}diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java b/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.javaindex 474433a5cf..ba3c918250 100644--- a/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java+++ b/src/main/java/io/github/hectorvent/floci/services/emr/model/EmrCluster.java@@ -11,7 +11,7 @@ import java.util.Map;/*** An EMR cluster (job flow). State machine:* STARTING → BOOTSTRAPPING → RUNNING → WAITING (and back to RUNNING for added steps), then- * → TERMINATING → TERMINATED, or TERMINATED_WITH_ERRORS when a failed step ended it.+ * → TERMINATING → TERMINATED, or TERMINATED_WITH_ERRORS when a failed step or bootstrap action ended it.* Steps, instance groups and fleets are stored as nested lists. Complex sub-structures* (ec2 attributes, applications, configurations) are stored as raw JSON strings.*/@@ -47,6 +47,10 @@ public class EmrCluster {private String configurations;// The cluster's BootstrapActions (BootstrapActionConfig list) as given, raw JSON; null when none.private String bootstrapActions;+ // Whether the docker runner mode is running the cluster's bootstrap actions, and the image they left, which+ // its steps run in; Floci's own, as the API has neither.+ private boolean bootstrapInContainer;+ private String bootstrapImage;// The cluster's ManagedScalingPolicy and AutoTerminationPolicy as given, raw JSON; null when none.private String managedScalingPolicy;private String autoTerminationPolicy;@@ -165,6 +169,14 @@ public class EmrCluster {public String getBootstrapActions() { return bootstrapActions; }public void setBootstrapActions(String bootstrapActions) { this.bootstrapActions = bootstrapActions; }+ public boolean isBootstrapInContainer() { return bootstrapInContainer; }+ public void setBootstrapInContainer(boolean bootstrapInContainer) {+ this.bootstrapInContainer = bootstrapInContainer;+ }++ public String getBootstrapImage() { return bootstrapImage; }+ public void setBootstrapImage(String bootstrapImage) { this.bootstrapImage = bootstrapImage; }+public String getManagedScalingPolicy() { return managedScalingPolicy; }public void setManagedScalingPolicy(String managedScalingPolicy) { this.managedScalingPolicy = managedScalingPolicy; }diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.javaindex e3461299d3..c1d75fa2d8 100644--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java@@ -10,6 +10,7 @@ import io.github.hectorvent.floci.core.common.ContainerTeardown;import io.github.hectorvent.floci.core.common.RequestScopes;import io.github.hectorvent.floci.core.common.S3Uri;import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import io.github.hectorvent.floci.core.common.docker.ContainerLaunches;import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;import io.github.hectorvent.floci.core.common.docker.ContainerSpec;@@ -22,9 +23,11 @@ import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;+import java.io.ByteArrayInputStream;import java.io.ByteArrayOutputStream;import java.io.Closeable;import java.io.IOException;+import java.io.InputStream;import java.time.Clock;import java.time.Duration;import java.time.Instant;@@ -33,10 +36,7 @@ import java.util.LinkedHashMap;import java.util.List;import java.util.Map;import java.util.Optional;-import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.TimeUnit;-import java.util.concurrent.atomic.AtomicReference;-import java.util.regex.Pattern;import java.util.zip.GZIPOutputStream;/**@@ -80,14 +80,12 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardoprivate static final String DOUBLE_QUOTE_ESCAPES = "\"\\$`\n";// How often a running driver is looked at, as Batch's runner looks at its jobs.private static final Duration POLL_INTERVAL = Duration.ofMillis(250);- // A log folder may be given in a Hadoop scheme, as an EMR cluster's LogUri often is (s3n://).- private static final Pattern HADOOP_S3_SCHEME = Pattern.compile("^s3[an]://");private static final String GZIP = "application/x-gzip";// How long reading an ended driver's output may take; Docker's log rotation bounds its size.private static final Duration LOG_READ_TIMEOUT = Duration.ofSeconds(60);// The runs inside run(); a stop for a run not here comes after its driver has gone, and is dropped.- private final ConcurrentHashMap<String, Launch> launches = new ConcurrentHashMap<>();+ private final ContainerLaunches launches = new ContainerLaunches();private final ContainerBuilder containerBuilder;private final ContainerLifecycleManager lifecycleManager;@@ -125,8 +123,7 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo@Overridepublic Outcome run(Job job, Progress progress) {- Launch launch = new Launch();- launches.put(job.runId(), launch);+ ContainerLaunches.Launch launch = launches.begin(job.runId());Optional<SessionCreds> credentials = Optional.empty();Closeable logHandle = null;try {@@ -160,9 +157,9 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo.withLabels(labels).build();String containerId = lifecycleManager.createAndStart(spec).containerId();- launch.container.set(containerId);+ launch.started(containerId);// A stop, a reset or a shutdown while the image was pulled is read now the driver has a container.- if (launch.stopRequested || progress.cancelled()) {+ if (launch.stopRequested() || progress.cancelled()) {return Outcome.halted();}logHandle = attachLog(containerId, job);@@ -175,15 +172,14 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardoThread.currentThread().interrupt();return Outcome.halted();} catch (RuntimeException e) {- if (launch.stopRequested) {+ if (launch.stopRequested()) {// A reset removed the driver while it was being watched.return Outcome.halted();}LOG.warnv(e, "Spark run {0} of {1} could not run", job.runId(), job.service());return Outcome.couldNotStart(e.getMessage());} finally {- launches.remove(job.runId(), launch);- String unclaimed = launch.container.getAndSet(null);+ String unclaimed = launches.end(job.runId(), launch);if (unclaimed != null) {lifecycleManager.stopAndRemove(unclaimed, logHandle, STOP_TIMEOUT_SECONDS);} else if (logHandle != null) {@@ -196,11 +192,11 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo@Overridepublic void stop(String runId) {- Launch launch = launches.get(runId);+ ContainerLaunches.Launch launch = launches.get(runId);if (launch != null) {// run() reads it once the driver's container is up and at each look at the driver after, and// removes the driver itself: a stop makes no Docker call, so no Docker error can lose it.- launch.stopRequested = true;+ launch.requestStop();}}@@ -210,13 +206,7 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo*/@Overridepublic void stopManagedContainers() {- for (Launch launch : launches.values()) {- launch.stopRequested = true;- String containerId = launch.container.getAndSet(null);- if (containerId != null) {- lifecycleManager.stopAndRemove(containerId, null, STOP_TIMEOUT_SECONDS);- }- }+ launches.stopAll(lifecycleManager, STOP_TIMEOUT_SECONDS);}/**@@ -279,11 +269,32 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardoreturn logUri.substring(0, end) + "/" + path;}+ /**+ * Puts a process's gzipped standard output and error at {@code stdout.gz} and {@code stderr.gz} in+ * {@code folder}, an S3 folder ending in a slash, in any of the Hadoop schemes, as {@code accountId} in+ * {@code region}.+ */+ public static void putLogs(S3Service s3Service, String accountId, String region, String folder, byte[] stdout,+ byte[] stderr) {+ putLogs(s3Service, accountId, region, folder, new ByteArrayInputStream(stdout),+ new ByteArrayInputStream(stderr));+ }++ /** As the other {@code putLogs} does, with each file read from a stream, so it need not be held in memory. */+ public static void putLogs(S3Service s3Service, String accountId, String region, String folder,+ InputStream stdout, InputStream stderr) {+ S3Uri location = S3Uri.parseAnyScheme(folder);+ RequestScopes.runAs(accountId, region, () -> {+ s3Service.putObjectFromStream(location.bucket(), location.key() + "stdout.gz", stdout, GZIP, Map.of());+ s3Service.putObjectFromStream(location.bucket(), location.key() + "stderr.gz", stderr, GZIP, Map.of());+ });+ }+/*** The image a release's runs use: {@code imageTemplate}, with {@code %s} standing for the release* label, or, when it is null, the release's public EMR Serverless image.*/- static String image(String imageTemplate, String releaseLabel) {+ public static String image(String imageTemplate, String releaseLabel) {String template = imageTemplate != null ? imageTemplate: releaseLabel.startsWith("emr-spark-") ? SPARK_RUNTIME_DEFAULT_IMAGE : DEFAULT_IMAGE;return template.formatted(releaseLabel);@@ -426,14 +437,14 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo* while a reset's teardown was passing is stopped once the reset has removed its run. Each look at the* container is one short control-plane call, so a run holds no Docker connection while its driver runs.*/- private Outcome awaitEnd(String containerId, Job job, Launch launch, Progress progress)+ private Outcome awaitEnd(String containerId, Job job, ContainerLaunches.Launch launch, Progress progress)throws InterruptedException {Instant deadline = job.timeout().isZero() ? null : clock.instant().plus(job.timeout());- while (!launch.stopRequested && !progress.cancelled()) {+ while (!launch.stopRequested() && !progress.cancelled()) {InspectContainerResponse.ContainerState state =lifecycleManager.getDockerClient().inspectContainerCmd(containerId).exec().getState();if (!Boolean.TRUE.equals(state.getRunning())) {- if (launch.stopRequested) {+ if (launch.stopRequested()) {return Outcome.halted();}Long exitCode = state.getExitCodeLong();@@ -457,14 +468,13 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo* all it wrote; run() removes it after. One Floci's teardown took leaves nothing. Logs that cannot be* written, to a bucket that does not exist say, leave the run as it ended.*/- private void writeLogs(Job job, Launch launch) {- String containerId = launch.container.get();+ private void writeLogs(Job job, ContainerLaunches.Launch launch) {+ String containerId = launch.container();if (job.logFolder() == null || containerId == null) {return;}// Closed however the read ends, which ends Docker's stream and the compressors.try (DriverOutput output = new DriverOutput()) {- S3Uri folder = S3Uri.parse(HADOOP_S3_SCHEME.matcher(job.logFolder()).replaceFirst("s3://"));lifecycleManager.stop(containerId, STOP_TIMEOUT_SECONDS);lifecycleManager.getDockerClient().logContainerCmd(containerId).withStdOut(true)@@ -475,12 +485,8 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardoif (!output.awaitCompletion(LOG_READ_TIMEOUT.toMillis(), TimeUnit.MILLISECONDS)) {throw new IOException("reading the driver's output took longer than " + LOG_READ_TIMEOUT);}- byte[] stdout = output.gzipped(StreamType.STDOUT);- byte[] stderr = output.gzipped(StreamType.STDERR);- RequestScopes.runAs(job.accountId(), job.region(), () -> {- s3Service.putObject(folder.bucket(), folder.key() + "stdout.gz", stdout, GZIP, Map.of());- s3Service.putObject(folder.bucket(), folder.key() + "stderr.gz", stderr, GZIP, Map.of());- });+ putLogs(s3Service, job.accountId(), job.region(), job.logFolder(), output.gzipped(StreamType.STDOUT),+ output.gzipped(StreamType.STDERR));} catch (InterruptedException e) {// Floci is shutting down: the run ends as its driver did, without its logs.Thread.currentThread().interrupt();@@ -548,13 +554,4 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo}}}-- /**- * A run inside {@link #run}: whether it is to stop, and its driver's container once it has one, which- * whoever stops the run first claims, run() itself or Floci's teardown.- */- private static final class Launch {- final AtomicReference<String> container = new AtomicReference<>();- volatile boolean stopRequested;- }}diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex d58274e381..e474a5d353 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -601,11 +601,12 @@ floci:cluster-startup-delay-seconds: 0 # 0 = a cluster is ready as soon as it is createdstep-duration-seconds: 0 # 0 = a step completes as soon as it startscluster-termination-delay-seconds: 0 # 0 = a cluster is TERMINATED as soon as it is TERMINATING- runner-mode: immediate # docker = run each Spark step's spark-submit in a container+ runner-mode: immediate # docker = run bootstrap actions and steps in containers# docker-network: floci-network# image-template: "registry.internal/emr-serverless/spark/%s:20250425" # %s = the release label- # Remove the step drivers a previous run of this deployment left on the daemon when the EMR- # service starts. Scoped to this deployment's own containers.+ # Remove the step drivers a previous run of this deployment left on the daemon, and the images+ # bootstrap actions left for clusters that are gone or have ended, when the EMR service starts.+ # Scoped to this deployment's own containers and images.reconcile-containers-on-startup: truewafv2:enabled: truediff --git a/src/test/java/io/github/hectorvent/floci/core/common/S3UriTest.java b/src/test/java/io/github/hectorvent/floci/core/common/S3UriTest.javaindex 0cc0894fd8..0fae29afbe 100644--- a/src/test/java/io/github/hectorvent/floci/core/common/S3UriTest.java+++ b/src/test/java/io/github/hectorvent/floci/core/common/S3UriTest.java@@ -3,7 +3,9 @@ package io.github.hectorvent.floci.core.common;import org.junit.jupiter.api.Test;import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;class S3UriTest {@@ -30,4 +32,15 @@ class S3UriTest {assertThrows(IllegalArgumentException.class, () -> S3Uri.parse("s3:///emr/"));assertThrows(IllegalArgumentException.class, () -> S3Uri.parse(null));}++ @Test+ void hadoopsSchemesNameTheSameBuckets() {+ assertEquals(new S3Uri("logs", "emr/"), S3Uri.parseAnyScheme("s3n://logs/emr/"));+ assertEquals(new S3Uri("logs", "emr/"), S3Uri.parseAnyScheme("s3a://logs/emr/"));+ assertEquals(new S3Uri("logs", "emr/"), S3Uri.parseAnyScheme("s3://logs/emr/"));+ assertTrue(S3Uri.isS3("s3n://logs/emr/"));+ assertFalse(S3Uri.isS3("file:/usr/share/aws/emr/scripts/install"));+ assertFalse(S3Uri.isS3(null));+ assertThrows(IllegalArgumentException.class, () -> S3Uri.parseAnyScheme("hdfs://logs/emr/"));+ }}diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecStubs.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecStubs.javaindex 9b7b0f7329..966ec1989f 100644--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecStubs.java+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecStubs.java@@ -32,6 +32,15 @@ public final class ContainerExecStubs {/** Returns the commands run in {@code containerId}, in order, as the stubbed execs see them. */public static List<List<String>> completeEveryExec(DockerClient dockerClient, String containerId, long exitCode,String stdout, String stderr) {+ return completeEveryExec(dockerClient, containerId, List.of(exitCode), stdout, stderr);+ }++ /**+ * As {@link #completeEveryExec(DockerClient, String, long, String, String)}, the execs exiting with+ * {@code exitCodes} in turn, and every one after them with the last.+ */+ public static List<List<String>> completeEveryExec(DockerClient dockerClient, String containerId,+ List<Long> exitCodes, String stdout, String stderr) {String execId = "exec-" + containerId;List<List<String>> commands = new ArrayList<>();@@ -61,7 +70,8 @@ public final class ContainerExecStubs {InspectExecCmd inspectCmd = mock(InspectExecCmd.class);InspectExecResponse inspectResponse = mock(InspectExecResponse.class);- when(inspectResponse.getExitCodeLong()).thenReturn(exitCode);+ when(inspectResponse.getExitCodeLong())+ .thenReturn(exitCodes.getFirst(), exitCodes.subList(1, exitCodes.size()).toArray(Long[]::new));when(inspectCmd.exec()).thenReturn(inspectResponse);when(dockerClient.inspectExecCmd(execId)).thenReturn(inspectCmd);return commands;diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLaunchesTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLaunchesTest.javanew file mode 100644index 0000000000..3bc1cf9300--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLaunchesTest.java@@ -0,0 +1,55 @@+package io.github.hectorvent.floci.core.common.docker;++import org.junit.jupiter.api.Test;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertNull;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.verifyNoInteractions;++/** A run's container is removed by whoever claims it first, the run itself or a stop, and by no one else. */+class ContainerLaunchesTest {++ private final ContainerLifecycleManager lifecycle = mock(ContainerLifecycleManager.class);+ private final ContainerLaunches launches = new ContainerLaunches();++ @Test+ void aStopRemovesAStartedContainerAndLeavesTheRunNothingToRemove() {+ ContainerLaunches.Launch launch = launches.begin("run-1");+ launch.started("container-1");++ launches.stopAll(lifecycle, 0);++ verify(lifecycle).stopAndRemove("container-1", null, 0);+ assertTrue(launch.stopRequested());+ assertNull(launches.end("run-1", launch));+ assertNull(launches.get("run-1"));+ }++ @Test+ void aStopBeforeTheContainerIsUpOnlyAsksTheRunToStop() {+ ContainerLaunches.Launch launch = launches.begin("run-1");++ launches.get("run-1").stopNow(lifecycle, 0);+ launch.started("container-1");++ verifyNoInteractions(lifecycle);+ assertTrue(launch.stopRequested());+ assertEquals("container-1", launches.end("run-1", launch));+ }++ @Test+ void aRequestedStopRemovesNothing() {+ ContainerLaunches.Launch launch = launches.begin("run-1");+ launch.started("container-1");++ launch.requestStop();++ verify(lifecycle, never()).stopAndRemove("container-1", null, 0);+ assertEquals("container-1", launch.container());+ assertEquals("container-1", launches.end("run-1", launch));+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerImagesTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerImagesTest.javanew file mode 100644index 0000000000..1dd1f8ee4c--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerImagesTest.java@@ -0,0 +1,62 @@+package io.github.hectorvent.floci.core.common.docker;++import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.command.CommitCmd;+import com.github.dockerjava.api.command.RemoveImageCmd;+import com.github.dockerjava.api.exception.ConflictException;+import com.github.dockerjava.api.exception.NotFoundException;+import org.junit.jupiter.api.Test;++import java.util.Map;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.Mockito.RETURNS_SELF;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++/** A container kept as an image, and an image removed: gone counts as removed, a refusal does not. */+class ContainerLifecycleManagerImagesTest {++ private final DockerClient docker = mock(DockerClient.class);+ private final ContainerLifecycleManager manager = new ContainerLifecycleManager(docker, null, null, null);++ @Test+ void aCommitNamesTheImageAndSetsItsUserAndLabels() {+ CommitCmd commit = mock(CommitCmd.class, RETURNS_SELF);+ when(commit.exec()).thenReturn("sha256:kept");+ when(docker.commitCmd("container-1")).thenReturn(commit);++ assertEquals("sha256:kept", manager.commit("container-1", "floci-aws-emr-bootstrap", "j-1", "hadoop",+ Map.of("io.floci.component", "emr-bootstrap")));++ verify(commit).withRepository("floci-aws-emr-bootstrap");+ verify(commit).withTag("j-1");+ verify(commit).withUser("hadoop");+ verify(commit).withLabels(Map.of("io.floci.component", "emr-bootstrap"));+ }++ @Test+ void anImageIsRemovedByForceAndOneAlreadyGoneCountsAsRemoved() {+ RemoveImageCmd removed = mock(RemoveImageCmd.class, RETURNS_SELF);+ when(docker.removeImageCmd("kept:1")).thenReturn(removed);+ RemoveImageCmd missing = mock(RemoveImageCmd.class, RETURNS_SELF);+ when(missing.exec()).thenThrow(new NotFoundException("No such image"));+ when(docker.removeImageCmd("gone:1")).thenReturn(missing);++ assertTrue(manager.removeImage("kept:1"));+ assertTrue(manager.removeImage("gone:1"));+ verify(removed).withForce(true);+ }++ @Test+ void anImageTheDaemonRefusesToRemoveIsReported() {+ RemoveImageCmd refused = mock(RemoveImageCmd.class, RETURNS_SELF);+ when(refused.exec()).thenThrow(new ConflictException("image is being used by running container"));+ when(docker.removeImageCmd("in-use:1")).thenReturn(refused);++ assertFalse(manager.removeImage("in-use:1"));+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamerTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamerTest.javaindex c7c7f3b869..81f88b08e6 100644--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamerTest.java+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLogStreamerTest.java@@ -14,6 +14,9 @@ import org.junit.jupiter.params.provider.ValueSource;import org.mockito.ArgumentCaptor;import java.io.Closeable;+import java.io.IOException;+import java.io.OutputStream;+import java.io.UncheckedIOException;import java.nio.charset.StandardCharsets;import java.time.Instant;import java.util.ArrayList;@@ -46,6 +49,27 @@ class ContainerLogStreamerTest {return s.getBytes(StandardCharsets.UTF_8);}+ private static void write(OutputStream stream, String text) {+ try {+ stream.write(utf8(text));+ } catch (IOException e) {+ throw new UncheckedIOException(e);+ }+ }++ private static void close(OutputStream stream) {+ try {+ stream.close();+ } catch (IOException e) {+ throw new UncheckedIOException(e);+ }+ }++ // The container lines that records logged, without their prefix.+ private static List<Object> loggedLines(List<LogRecord> records) {+ return records.stream().map(record -> record.getParameters()[1]).toList();+ }+@Testvoid ensureLogGroupAndStreamUsesOwningAccount() {CloudWatchLogsService cloudWatchLogsService = mock(CloudWatchLogsService.class);@@ -208,6 +232,22 @@ class ContainerLogStreamerTest {eq("us-east-1"));}+ @Test+ void consoleLinesLogsEachLineAsItArrivesAndALastOneWithoutANewlineWhenClosed() {+ OutputStream console = ContainerLogStreamer.consoleLines("emr:j-1:bootstrap-action-1");++ List<LogRecord> asWritten = LogCapture.capture(ContainerLogStreamer.class, () -> {+ write(console, "Collecting pan");+ write(console, "das\n\nInstalled pandas\nDone");+ });+ List<LogRecord> asClosed = LogCapture.capture(ContainerLogStreamer.class, () -> close(console));++ // A blank line is dropped, as the attach methods drop one.+ assertEquals(List.of("Collecting pandas", "Installed pandas"), loggedLines(asWritten));+ assertEquals(List.of("Done"), loggedLines(asClosed));+ assertEquals("emr:j-1:bootstrap-action-1", asClosed.getFirst().getParameters()[0]);+ }+@Testvoid reassemblesSingleLineSplitAcrossFrames() {// A ~1.5 KB stdout line that Docker delivers as multiple frames must arrivediff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrBootstrapRunnerTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrBootstrapRunnerTest.javanew file mode 100644index 0000000000..bed3b3eb1f--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrBootstrapRunnerTest.java@@ -0,0 +1,258 @@+package io.github.hectorvent.floci.services.emr;++import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.command.ExecCreateCmd;+import com.github.dockerjava.api.model.Image;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import io.github.hectorvent.floci.core.common.docker.ContainerExecStubs;+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;+import io.github.hectorvent.floci.services.lambda.launcher.BootstrapProcess;+import io.github.hectorvent.floci.services.lambda.launcher.LambdaExecutionRoleCredentials;+import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.s3.model.S3Object;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.io.TempDir;++import java.io.IOException;+import java.io.InputStream;+import java.nio.charset.StandardCharsets;+import java.nio.file.Files;+import java.nio.file.Path;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.Set;+import java.util.concurrent.atomic.AtomicBoolean;+import java.util.stream.Collectors;+import java.util.stream.IntStream;+import java.util.stream.Stream;+import java.util.zip.GZIPInputStream;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyMap;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.ArgumentMatchers.argThat;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.RETURNS_SELF;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.times;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++/**+ * A cluster's bootstrap actions run one after the other through {@code docker exec}, as root, in one+ * container of the release's image, which is then kept as the image the cluster's steps run in.+ */+class EmrBootstrapRunnerTest {++ private static final String ACCOUNT = "123456789012";+ private static final String REGION = "eu-west-1";+ private static final String CONTAINER = "bootstrap";+ private static final String RELEASE_IMAGE = "public.ecr.aws/emr-serverless/spark/emr-7.5.0:latest";+ private static final String RELEASE_IMAGE_ID = "sha256:" + "1".repeat(64);+ private static final String IMAGE_ID = "sha256:" + "2".repeat(64);+ private static final List<String> ENV = List.of("AWS_ACCESS_KEY_ID=ASIAEXAMPLE", "AWS_ENDPOINT_URL=http://floci");++ private final EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ private final ContainerLifecycleManager lifecycle = mock(ContainerLifecycleManager.class);+ private final DockerClient docker = mock(DockerClient.class, RETURNS_DEEP_STUBS);+ private final ContainerBuilder builder = mock(ContainerBuilder.class);+ private final ContainerBuilder.Builder spec = mock(ContainerBuilder.Builder.class, RETURNS_SELF);+ private final LaunchedContainerAwsEnv awsEnv = mock(LaunchedContainerAwsEnv.class);+ private final LambdaExecutionRoleCredentials roleCredentials = mock(LambdaExecutionRoleCredentials.class);+ private final S3Service s3 = mock(S3Service.class);+ private final EmrBootstrapRunner runner =+ new EmrBootstrapRunner(builder, lifecycle, awsEnv, roleCredentials, s3, config);+ // What the runner put in S3, by bucket and key, gunzipped: read as it is put, as its files then go.+ private final Map<String, String> uploaded = new LinkedHashMap<>();++ @BeforeEach+ void setUp() {+ when(builder.newContainer(anyString())).thenReturn(spec);+ when(lifecycle.getDockerClient()).thenReturn(docker);+ when(lifecycle.createAndStart(any()))+ .thenReturn(new ContainerLifecycleManager.ContainerInfo(CONTAINER, Map.of()));+ when(roleCredentials.forRole(any(), any(), any())).thenReturn(Optional.empty());+ when(awsEnv.sdkBaselineEnv(REGION, Optional.empty(), Optional.empty(), ACCOUNT)).thenReturn(ENV);+ // The container runs the release image, under whatever name an image-registry-base gave it.+ when(docker.inspectContainerCmd(CONTAINER).exec().getImageId()).thenReturn(RELEASE_IMAGE_ID);+ when(docker.inspectImageCmd(RELEASE_IMAGE_ID).exec().getConfig().getUser()).thenReturn("hadoop");+ when(docker.inspectImageCmd(ContainerStorageHelper.dockerName(config, "emr-bootstrap") + ":j-1").exec()+ .getId()).thenReturn(IMAGE_ID);+ script("bucket", "bootstrap/install.sh");+ script("bucket", "bootstrap/jars.sh");+ when(s3.putObjectFromStream(anyString(), anyString(), any(InputStream.class), eq("application/x-gzip"),+ anyMap())).thenAnswer(put -> {+ try (GZIPInputStream gunzip = new GZIPInputStream(put.<InputStream>getArgument(2))) {+ uploaded.put(put.getArgument(0) + "/" + put.getArgument(1),+ new String(gunzip.readAllBytes(), StandardCharsets.UTF_8));+ }+ return null;+ });+ }++ @Test+ void eachActionRunsInTurnAsRootThenTheContainerIsKeptAsTheImageTheStepsRunIn() {+ List<List<String>> commands = ContainerExecStubs.completeEveryExec(docker, CONTAINER, 0, "installed\n", "");++ BootstrapRunner.Outcome outcome = runner.run(bootstrap("s3://logs/j-1/node/i-1/bootstrap-actions/",+ action("s3://bucket/bootstrap/install.sh", "pandas", "2.2"),+ action("s3n://bucket/bootstrap/jars.sh")), () -> false);++ // The steps get the kept image's id, which runs as it is, whatever registry Floci pulls images from.+ assertEquals(BootstrapRunner.Outcome.succeeded(IMAGE_ID), outcome);+ verify(builder).newContainer(RELEASE_IMAGE);+ verify(spec).withUser("root");+ verify(spec).withEntrypoint(List.of("sleep", "infinity"));+ verify(spec).withLabels(argThat(labels -> "emr".equals(labels.get(ContainerStorageHelper.SERVICE_LABEL))+ && "j-1".equals(labels.get(ContainerStorageHelper.RESOURCE_ID_LABEL))+ && "emr-bootstrap".equals(labels.get(ContainerStorageHelper.COMPONENT_LABEL))));+ // Each script is read as the cluster's account and copied in, then run with its arguments.+ verify(s3).getObject("bucket", "bootstrap/install.sh");+ verify(s3).getObject("bucket", "bootstrap/jars.sh");+ assertEquals(List.of("bootstrap-action", "/tmp/floci-bootstrap-action-1-install.sh", "pandas", "2.2"),+ commands.get(0).subList(3, commands.get(0).size()));+ assertEquals(List.of("bootstrap-action", "/tmp/floci-bootstrap-action-2-jars.sh"),+ commands.get(1).subList(3, commands.get(1).size()));+ assertEquals(List.of("bash", "-c"), commands.get(0).subList(0, 2));+ assertTrue(commands.get(0).get(2).contains("chown -R hadoop:hadoop /home/hadoop"));+ assertEquals(3, commands.size());+ // The credentials go to each action's exec alone, never to the container the image is made of.+ ExecCreateCmd exec = docker.execCreateCmd(CONTAINER);+ verify(exec, times(2)).withEnv(ENV);+ verify(spec, never()).withEnv(any(List.class));+ // The image runs as the user of the image the container ran, as the steps do.+ String repository = ContainerStorageHelper.dockerName(config, "emr-bootstrap");+ verify(lifecycle).commit(eq(CONTAINER), eq(repository), eq("j-1"), eq("hadoop"),+ argThat(labels -> "emr-bootstrap".equals(labels.get(ContainerStorageHelper.COMPONENT_LABEL))));+ assertEquals(Map.of("logs/j-1/node/i-1/bootstrap-actions/1/stdout.gz", "installed\n",+ "logs/j-1/node/i-1/bootstrap-actions/1/stderr.gz", "",+ "logs/j-1/node/i-1/bootstrap-actions/2/stdout.gz", "installed\n",+ "logs/j-1/node/i-1/bootstrap-actions/2/stderr.gz", ""), uploaded);+ verify(lifecycle).stopAndRemove(CONTAINER, null, 0);+ }++ @Test+ void aFailingActionEndsTheBootstrapThereAndNothingIsKept() {+ List<List<String>> commands = ContainerExecStubs.completeEveryExec(docker, CONTAINER, List.of(0L, 3L), "", "");++ BootstrapRunner.Outcome outcome = runner.run(bootstrap(null, action("s3://bucket/bootstrap/install.sh"),+ action("s3://bucket/bootstrap/jars.sh"), action("s3://bucket/bootstrap/never.sh")), () -> false);++ assertEquals(BootstrapRunner.Outcome.actionFailed(2, 3), outcome);+ assertEquals(2, commands.size());+ verify(lifecycle, never()).commit(any(), any(), any(), any(), any());+ assertTrue(uploaded.isEmpty());+ verify(lifecycle).stopAndRemove(CONTAINER, null, 0);+ }++ @Test+ void anActionsOutputReachesItsLogsWholeHoweverLongAndLeavesNoFileBehind() throws IOException {+ // Megabytes of output, as a chatty pip install gives, which is kept on disk as it arrives, not in memory.+ String output = IntStream.range(0, 2_000).mapToObj(line -> "Collecting package-" + line + " "+ + "x".repeat(1_000) + "\n").collect(Collectors.joining());+ ContainerExecStubs.completeEveryExec(docker, CONTAINER, 0, output, "WARNING: no newline at the end");+ Set<Path> filesBefore = bootstrapFiles();++ runner.run(bootstrap("s3://logs/j-1/node/i-1/bootstrap-actions/", action("s3://bucket/bootstrap/install.sh")),+ () -> false);++ assertEquals(output, uploaded.get("logs/j-1/node/i-1/bootstrap-actions/1/stdout.gz"));+ assertEquals("WARNING: no newline at the end", uploaded.get("logs/j-1/node/i-1/bootstrap-actions/1/stderr.gz"));+ assertEquals(filesBefore, bootstrapFiles());+ }++ @Test+ void aScriptThatCannotBeReadFailsItsActionWithoutRunningIt() {+ List<List<String>> commands = ContainerExecStubs.completeEveryExec(docker, CONTAINER, 0, "", "");+ when(s3.getObject("bucket", "bootstrap/missing.sh"))+ .thenThrow(new AwsException("NoSuchKey", "The specified key does not exist.", 404));++ BootstrapRunner.Outcome outcome = runner.run(bootstrap(null, action("s3://bucket/bootstrap/missing.sh")),+ () -> false);++ assertEquals(BootstrapRunner.Outcome.notDownloaded(1, "The specified key does not exist."), outcome);+ assertTrue(commands.isEmpty());+ }++ @Test+ void aClusterThatEndsWhileAnActionRunsStopsTheBootstrapThere() {+ List<List<String>> commands = ContainerExecStubs.completeEveryExec(docker, CONTAINER, 0, "", "");+ AtomicBoolean ended = new AtomicBoolean();+ when(s3.getObject("bucket", "bootstrap/install.sh")).thenAnswer(read -> {+ ended.set(true);+ return content("#!/bin/bash\n");+ });++ BootstrapRunner.Outcome outcome = runner.run(bootstrap(null, action("s3://bucket/bootstrap/install.sh"),+ action("s3://bucket/bootstrap/jars.sh")), ended::get);++ assertEquals(BootstrapRunner.Outcome.halted(), outcome);+ assertEquals(1, commands.size());+ verify(lifecycle, never()).commit(any(), any(), any(), any(), any());+ }++ @Test+ void imagesOfClustersThatAreGoneAreRemovedAndTheOthersKept() {+ String goneId = "sha256:" + "3".repeat(64);+ Image kept = mock(Image.class);+ when(kept.getId()).thenReturn(IMAGE_ID);+ Image gone = mock(Image.class);+ when(gone.getId()).thenReturn(goneId);+ when(docker.listImagesCmd().withLabelFilter(anyMap()).exec()).thenReturn(List.of(kept, gone));++ runner.removeImages(Set.of(IMAGE_ID));++ verify(lifecycle).removeImage(goneId);+ verify(lifecycle, never()).removeImage(IMAGE_ID);+ }++ @Test+ void sudoDropsItsOptionsAndRunsItsCommandAsItIs(@TempDir Path directory) throws Exception {+ Path sudo = directory.resolve("sudo");+ Files.writeString(sudo, EmrBootstrapRunner.SUDO + "\n");++ BootstrapProcess.Result ended = BootstrapProcess.run(new ProcessBuilder("bash", sudo.toString(), "-E", "-u",+ "root", "--", "GREETING=hello", "sh", "-c", "printf '%s %s|' \"$GREETING\" \"$1\"", "sh", "two words"));++ assertEquals(new BootstrapProcess.Result(0, "hello two words|"), ended);+ }++ // The files an action's output is gzipped into while it runs, in the temporary directory.+ private static Set<Path> bootstrapFiles() throws IOException {+ try (Stream<Path> files = Files.list(Path.of(System.getProperty("java.io.tmpdir")))) {+ return files.filter(file -> file.getFileName().toString().startsWith("floci-emr-bootstrap-"))+ .collect(Collectors.toSet());+ }+ }++ private void script(String bucket, String key) {+ S3Object object = content("#!/bin/bash\necho installed\n");+ when(s3.getObject(bucket, key)).thenReturn(object);+ }++ private static S3Object content(String script) {+ S3Object object = mock(S3Object.class);+ when(object.getData()).thenReturn(script.getBytes(StandardCharsets.UTF_8));+ return object;+ }++ private static BootstrapRunner.Action action(String path, String... args) {+ return new BootstrapRunner.Action(path.substring(path.lastIndexOf('/') + 1), path, List.of(args));+ }++ private static BootstrapRunner.Bootstrap bootstrap(String logFolder, BootstrapRunner.Action... actions) {+ return new BootstrapRunner.Bootstrap("j-1", ACCOUNT, REGION, "emr-7.5.0", null, null,+ "arn:aws:iam::" + ACCOUNT + ":role/emr-ec2", List.of(actions), logFolder);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.javaindex 174b28ff36..6f4b8a809e 100644--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java@@ -5,6 +5,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.RegionResolver;import io.github.hectorvent.floci.services.emr.model.EmrCluster;+import io.github.hectorvent.floci.services.emr.model.EmrInstanceGroup;import io.github.hectorvent.floci.services.emr.model.EmrStep;import io.github.hectorvent.floci.services.emrserverless.EmrServerlessSparkRunner;import io.github.hectorvent.floci.services.emrserverless.SparkJobRunner;@@ -23,12 +24,14 @@ import java.util.List;import java.util.Map;import java.util.Optional;import java.util.Set;+import java.util.function.BooleanSupplier;import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.ACCOUNT;import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.REGION;import static io.github.hectorvent.floci.services.emr.EmrServiceFixture.cluster;import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertNull;+import static org.junit.jupiter.api.Assertions.assertTrue;import static org.mockito.ArgumentMatchers.any;import static org.mockito.Mockito.RETURNS_DEEP_STUBS;import static org.mockito.Mockito.inOrder;@@ -40,14 +43,19 @@ import static org.mockito.Mockito.when;/*** The {@code docker} runner mode: each step runs in a container from when it goes RUNNING until its Spark* driver ends, and then is COMPLETED, or FAILED with its ActionOnFailure taken, as the API reference and- * the Management Guide give it.+ * the Management Guide give it. A cluster's bootstrap actions run while it is BOOTSTRAPPING, and its steps+ * then in the image they left.*/class EmrServiceContainerStepsTest {private static final String REPORT = "s3://bucket/jobs/report.py";private static final String DRIVER_FAILED = "The Spark driver exited with code 1.";+ private static final String INSTALL = "s3://bucket/bootstrap/install.sh";+ // The id of the image a bootstrap kept, which the cluster's steps run in.+ private static final String IMAGE = "sha256:" + "2".repeat(64);private final FakeRunner runner = new FakeRunner();+ private final FakeBootstrap bootstrap = new FakeBootstrap();private final EmrServiceFixture fixture = EmrServiceFixture.docker(runner, null);private final EmrService service = fixture.service;@@ -385,6 +393,21 @@ class EmrServiceContainerStepsTest {order.verify(wakeups).at(any(), any());}+ @Test+ void leftoverBootstrapImagesAreRemovedBeforeARestartedClusterCanMoveOn() {+ BootstrapRunner bootstrapRunner = mock(BootstrapRunner.class);+ EmrServiceFixture bootstrapping = EmrServiceFixture.docker(runner, bootstrapRunner, null);+ bootstrapping.service.runJobFlow(bootstrapped(cluster("restarted", true)), REGION);+ EmrService.Wakeups wakeups = mock(EmrService.Wakeups.class);++ bootstrapping.restartReconciling(runner, wakeups);++ // A wakeup runs on the timer's thread: a cluster it moved on could keep an image the sweep never knew of.+ InOrder order = inOrder(bootstrapRunner, wakeups);+ order.verify(bootstrapRunner).removeImages(Set.of());+ order.verify(wakeups).at(any(), any());+ }+@Testvoid aStepRunsAsItsRuntimeRoleOrElseAsTheClustersInstanceProfileRole() {IamService iam = mock(IamService.class);@@ -418,22 +441,217 @@ class EmrServiceContainerStepsTest {}@Test- void leftoverDriversAreRemovedWhenTheServiceStartsUnlessReconcilingIsOff() {+ void leftoverDriversAndBootstrapImagesAreRemovedWhenTheServiceStartsUnlessReconcilingIsOff() {for (boolean reconcile : List.of(true, false)) {EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);when(config.services().emr().runnerMode()).thenReturn("docker");when(config.services().emr().reconcileContainersOnStartup()).thenReturn(reconcile);EmrServerlessSparkRunner sparkRunner = mock(EmrServerlessSparkRunner.class);+ EmrBootstrapRunner bootstrapRunner = mock(EmrBootstrapRunner.class);RegionResolver regionResolver = new RegionResolver(REGION, ACCOUNT);new EmrService(new InMemoryStorageFactory(ACCOUNT), config, regionResolver,new EmrEventPublisher(mock(EventBridgeService.class), new ObjectMapper(), regionResolver),- sparkRunner, mock(IamService.class)).shutdown();+ sparkRunner, bootstrapRunner, mock(IamService.class)).shutdown();verify(sparkRunner, times(reconcile ? 1 : 0)).removeLeftoverDrivers("emr");+ verify(bootstrapRunner, times(reconcile ? 1 : 0)).removeImages(Set.of());}}+ @Test+ void aClustersBootstrapActionsRunWhileItIsBootstrappingThenItsStepsRunInTheImageTheyLeft() {+ IamService iam = mock(IamService.class);+ InstanceProfile profile = new InstanceProfile();+ profile.setRoleNames(List.of("emr-ec2"));+ IamRole role = new IamRole();+ role.setArn("arn:aws:iam::" + ACCOUNT + ":role/emr-ec2");+ when(iam.findInstanceProfile(ACCOUNT, "EMR_EC2_DefaultRole")).thenReturn(Optional.of(profile));+ when(iam.findRole(ACCOUNT, "emr-ec2")).thenReturn(Optional.of(role));+ EmrServiceFixture bootstrapping = EmrServiceFixture.docker(runner, bootstrap, iam);+ EmrCluster requested = bootstrapped(cluster("bootstrapped", true, sparkStep("report", "CONTINUE")));+ requested.setJobFlowRole("EMR_EC2_DefaultRole");+ requested.setLogUri("s3://logs/emr/");+ // ListInstances lists the core group's two instances before the master's.+ requested.setInstanceGroups(new ArrayList<>(List.of(group("CORE", 2), group("MASTER", 1))));+ EmrCluster cluster = bootstrapping.service.runJobFlow(requested, REGION);+ String stepId = cluster.getSteps().get(0).getId();++ assertEquals("BOOTSTRAPPING", bootstrapping.service.describeCluster(cluster.getId()).getState());+ assertEquals("PENDING", bootstrapping.service.describeStep(cluster.getId(), stepId).getState());+ bootstrapping.runStepThread();++ // "logs/<cluster-id>/node/<primary-instance-id>/bootstrap-actions/<action-number>/stdout.gz"+ assertEquals(new BootstrapRunner.Bootstrap(cluster.getId(), ACCOUNT, REGION, "emr-7.5.0", null, null,+ "arn:aws:iam::" + ACCOUNT + ":role/emr-ec2", List.of(+ new BootstrapRunner.Action("pandas", INSTALL, List.of("pandas", "2.2")),+ new BootstrapRunner.Action("jars", "s3://bucket/bootstrap/jars.sh", List.of())),+ "s3://logs/emr/" + cluster.getId() + "/node/i-00000000000000003/bootstrap-actions/"),+ bootstrap.bootstraps.get(0));+ assertEquals("RUNNING", bootstrapping.service.describeCluster(cluster.getId()).getState());+ assertEquals("RUNNING", bootstrapping.service.describeStep(cluster.getId(), stepId).getState());+ bootstrapping.runStepThread();++ assertEquals(IMAGE, runner.jobs.get(0).imageTemplate());+ assertEquals("COMPLETED", bootstrapping.service.describeStep(cluster.getId(), stepId).getState());+ assertEquals(List.of("STARTING", "PENDING report", "RUNNING report", "COMPLETED report", "WAITING"),+ bootstrapping.states());+ }++ @Test+ void aClusterWithoutBootstrapActionsRunsItsStepsInTheReleasesImage() {+ EmrServiceFixture bootstrapping = EmrServiceFixture.docker(runner, bootstrap, null);+ EmrCluster cluster = bootstrapping.service.runJobFlow(cluster("plain", true, sparkStep("report", "CONTINUE")),+ REGION);++ assertEquals("RUNNING", bootstrapping.service.describeCluster(cluster.getId()).getState());+ bootstrapping.runStepThread();++ assertTrue(bootstrap.bootstraps.isEmpty());+ assertNull(runner.jobs.get(0).imageTemplate());+ }++ @Test+ void aFailedBootstrapActionEndsTheClusterWithErrorsBeforeAnyStepRuns() {+ bootstrap.outcome = BootstrapRunner.Outcome.actionFailed(2, 1);+ EmrServiceFixture bootstrapping = EmrServiceFixture.docker(runner, bootstrap, null);+ EmrCluster cluster = bootstrapping.service.runJobFlow(+ bootstrapped(cluster("failing", true, sparkStep("report", "CONTINUE"))), REGION);++ bootstrapping.runStepThread();++ EmrCluster ended = bootstrapping.service.describeCluster(cluster.getId());+ assertEquals("TERMINATED_WITH_ERRORS", ended.getState());+ assertEquals("BOOTSTRAP_FAILURE", ended.getStateChangeReasonCode());+ assertEquals("On the master instance (i-00000000000000001), bootstrap action 2 returned a non-zero return code",+ ended.getStateChangeMessage());+ assertEquals("CANCELLED", bootstrapping.service.describeStep(cluster.getId(),+ cluster.getSteps().get(0).getId()).getState());+ assertTrue(runner.jobs.isEmpty());+ assertEquals(0, bootstrapping.stepThreadsWaiting());+ }++ @Test+ void aScriptThatCannotBeReadOrABootstrapThatCannotRunEndsTheClusterWithErrorsToo() {+ Map<BootstrapRunner.Outcome, List<String>> reasons = Map.of(+ BootstrapRunner.Outcome.notDownloaded(1, "The specified key does not exist."),+ List.of("BOOTSTRAP_FAILURE", "On the master instance (i-00000000000000001), bootstrap action 1 could"+ + " not be read from " + INSTALL + ": The specified key does not exist."),+ BootstrapRunner.Outcome.couldNotRun("the image could not be pulled"),+ List.of("INTERNAL_ERROR", "Floci could not run the bootstrap actions: the image could not be pulled"));+ reasons.forEach((outcome, reason) -> {+ bootstrap.outcome = outcome;+ EmrServiceFixture bootstrapping = EmrServiceFixture.docker(runner, bootstrap, null);+ EmrCluster cluster = bootstrapping.service.runJobFlow(bootstrapped(cluster("broken", true)), REGION);++ bootstrapping.runStepThread();++ EmrCluster ended = bootstrapping.service.describeCluster(cluster.getId());+ assertEquals("TERMINATED_WITH_ERRORS", ended.getState());+ assertEquals(reason, List.of(ended.getStateChangeReasonCode(), ended.getStateChangeMessage()));+ });+ }++ @Test+ void aClusterTerminatedWhileBootstrappingStopsItsBootstrapAndTheImageItLeftGoes() {+ EmrServiceFixture bootstrapping = EmrServiceFixture.docker(runner, bootstrap, null);+ EmrCluster cluster = bootstrapping.service.runJobFlow(+ bootstrapped(cluster("cut short", true, sparkStep("report", "CONTINUE"))), REGION);+ // As a bootstrap that had kept its image just as its cluster ended.+ bootstrap.whileRunning = () -> bootstrapping.service.terminateJobFlows(List.of(cluster.getId()));++ while (bootstrapping.stepThreadsWaiting() > 0) {+ bootstrapping.runStepThread();+ }++ EmrCluster ended = bootstrapping.service.describeCluster(cluster.getId());+ assertEquals("TERMINATED", ended.getState());+ assertEquals("USER_REQUEST", ended.getStateChangeReasonCode());+ assertEquals(List.of(cluster.getId()), bootstrap.stopped);+ assertEquals(List.of(IMAGE), bootstrap.removed);+ assertTrue(runner.jobs.isEmpty());+ }++ @Test+ void theImageABootstrapLeftGoesOnceItsClusterHasEndedAndItsLastStepIsDone() {+ EmrServiceFixture bootstrapping = EmrServiceFixture.docker(runner, bootstrap, null);+ EmrCluster cluster = bootstrapping.service.runJobFlow(+ bootstrapped(cluster("transient", false, sparkStep("report", "CONTINUE"))), REGION);+ bootstrapping.runStepThread();+ // The step's driver, still running as its cluster ends, still needs the image.+ List<Integer> threadsAsTheClusterEnded = new ArrayList<>();+ runner.whileRunning = () -> {+ bootstrapping.service.terminateJobFlows(List.of(cluster.getId()));+ threadsAsTheClusterEnded.add(bootstrapping.stepThreadsWaiting());+ };++ bootstrapping.runStepThread();+ assertEquals(List.of(0), threadsAsTheClusterEnded);+ assertEquals("TERMINATED", bootstrapping.service.describeCluster(cluster.getId()).getState());+ bootstrapping.runStepThread();++ assertEquals(List.of(IMAGE), bootstrap.removed);+ assertNull(bootstrapping.service.describeCluster(cluster.getId()).getBootstrapImage());+ }++ @Test+ void anImageDockerRefusesToRemoveIsTriedAgainTheNextTimeItsClusterIsRead() {+ EmrServiceFixture bootstrapping = EmrServiceFixture.docker(runner, bootstrap, null);+ EmrCluster cluster = bootstrapping.service.runJobFlow(bootstrapped(cluster("refused", true)), REGION);+ bootstrapping.runStepThread();+ bootstrap.removes = false;+ bootstrapping.service.terminateJobFlows(List.of(cluster.getId()));++ // One removal at a time, however often the cluster is read meanwhile.+ bootstrapping.service.describeCluster(cluster.getId());+ assertEquals(1, bootstrapping.stepThreadsWaiting());+ bootstrapping.runStepThread();+ // Refused: the cluster still names its image, and reading it tries again.+ assertEquals(IMAGE, bootstrapping.service.describeCluster(cluster.getId()).getBootstrapImage());+ bootstrap.removes = true;+ bootstrapping.runStepThread();++ assertEquals(List.of(IMAGE, IMAGE), bootstrap.removed);+ assertNull(bootstrapping.service.describeCluster(cluster.getId()).getBootstrapImage());+ assertEquals(0, bootstrapping.stepThreadsWaiting());+ }++ @Test+ void aClusterARestartLeftBootstrappingEndsWithErrors() {+ EmrServiceFixture bootstrapping = EmrServiceFixture.docker(runner, bootstrap, null);+ EmrCluster cluster = bootstrapping.service.runJobFlow(bootstrapped(cluster("interrupted", true)), REGION);++ EmrCluster ended = bootstrapping.restart().service.describeCluster(cluster.getId());++ assertEquals("TERMINATED_WITH_ERRORS", ended.getState());+ assertEquals("INTERNAL_ERROR", ended.getStateChangeReasonCode());+ assertEquals(EmrService.BOOTSTRAP_INTERRUPTED, ended.getStateChangeMessage());+ }++ @Test+ void aResetRemovesEveryImageABootstrapLeft() {+ EmrServiceFixture.docker(runner, bootstrap, null).service.beforeReset();++ assertEquals(List.of(Set.of()), bootstrap.kept);+ }++ // Two bootstrap actions, as RunJobFlow stores them.+ private static EmrCluster bootstrapped(EmrCluster cluster) {+ cluster.setBootstrapActions("""+ [{"Name": "pandas", "ScriptBootstrapAction": {"Path": "%s", "Args": ["pandas", "2.2"]}},+ {"Name": "jars", "ScriptBootstrapAction": {"Path": "s3://bucket/bootstrap/jars.sh"}}]"""+ .formatted(INSTALL));+ return cluster;+ }++ private static EmrInstanceGroup group(String type, int count) {+ EmrInstanceGroup group = new EmrInstanceGroup();+ group.setInstanceGroupType(type);+ group.setInstanceType("m5.xlarge");+ group.setRequestedInstanceCount(count);+ return group;+ }+// A spark-submit step through command-runner.jar, written for a cluster with YARN.private static EmrStep sparkStep(String name, String actionOnFailure) {EmrStep step = new EmrStep();@@ -448,6 +666,43 @@ class EmrServiceContainerStepsTest {* Runs a step's driver on the test's thread: it starts, does what the test asks, and ends with* {@code outcome}, or stopped once {@link #stop} has been called for it.*/+ private static final class FakeBootstrap implements BootstrapRunner {+ final List<Bootstrap> bootstraps = new ArrayList<>();+ final List<String> stopped = new ArrayList<>();+ final List<String> removed = new ArrayList<>();+ final List<Set<String>> kept = new ArrayList<>();+ Outcome outcome = Outcome.succeeded(IMAGE);+ Runnable whileRunning = () -> { };+ // Whether Docker removes an image it is asked to.+ boolean removes = true;++ @Override+ public Outcome run(Bootstrap bootstrap, BooleanSupplier cancelled) {+ bootstraps.add(bootstrap);+ if (cancelled.getAsBoolean()) {+ return Outcome.halted();+ }+ whileRunning.run();+ return outcome;+ }++ @Override+ public void stop(String clusterId) {+ stopped.add(clusterId);+ }++ @Override+ public boolean removeImage(String image) {+ removed.add(image);+ return removes;+ }++ @Override+ public void removeImages(Set<String> kept) {+ this.kept.add(kept);+ }+ }+private static final class FakeRunner implements SparkJobRunner {final List<Job> jobs = new ArrayList<>();final Set<String> stopped = new HashSet<>();diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.javaindex 39fec978d1..5ddb80bedd 100644--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.java+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceFixture.java@@ -32,7 +32,7 @@ import static org.mockito.Mockito.when;* recording the {@code aws.emr} events it publishes. Each event is handed to {@link #target} as it is* published, as EventBridge's first delivery is, and checked to be published once the service is let go,* and, for a wakeup's, after it, as the timer's own thread never sends them. In the {@code docker} runner- * mode a step's thread is one the test runs, with {@link #runStepThread}.+ * mode a step's thread, and a bootstrap's, is one the test runs, with {@link #runStepThread}.*/final class EmrServiceFixture {@@ -61,6 +61,7 @@ final class EmrServiceFixture {private boolean waking;private final InMemoryStorageFactory storage;private final SparkJobRunner runner;+ private final BootstrapRunner bootstrap;private final IamService iam;private final int[] durations;@@ -70,20 +71,27 @@ final class EmrServiceFixture {}EmrServiceFixture(int startupDelaySeconds, int stepDurationSeconds, int terminationDelaySeconds) {- this(new InMemoryStorageFactory(ACCOUNT), new MutableClock(), null, null,+ this(new InMemoryStorageFactory(ACCOUNT), new MutableClock(), null, null, null,startupDelaySeconds, stepDurationSeconds, terminationDelaySeconds);}/** The service in the {@code docker} runner mode, every duration 0, its steps run by {@code runner}. */static EmrServiceFixture docker(SparkJobRunner runner, IamService iam) {- return new EmrServiceFixture(new InMemoryStorageFactory(ACCOUNT), new MutableClock(), runner, iam, 0, 0, 0);+ return docker(runner, null, iam);+ }++ /** As {@link #docker(SparkJobRunner, IamService)}, its clusters' bootstrap actions run by {@code bootstrap}. */+ static EmrServiceFixture docker(SparkJobRunner runner, BootstrapRunner bootstrap, IamService iam) {+ return new EmrServiceFixture(new InMemoryStorageFactory(ACCOUNT), new MutableClock(), runner, bootstrap, iam,+ 0, 0, 0);}private EmrServiceFixture(InMemoryStorageFactory storage, MutableClock clock, SparkJobRunner runner,- IamService iam, int... durations) {+ BootstrapRunner bootstrap, IamService iam, int... durations) {this.storage = storage;this.clock = clock;this.runner = runner;+ this.bootstrap = bootstrap;this.iam = iam;this.durations = durations;EventBridgeService eventBridge = mock(EventBridgeService.class);@@ -110,12 +118,12 @@ final class EmrServiceFixture {public void deliver(Runnable delivery) {deliveries.add(delivery);}- }, runner, stepThreads::add, iam);+ }, runner, bootstrap, stepThreads::add, iam);}/** A new service on the same storage and clock, as after a restart of Floci with persistent storage. */EmrServiceFixture restart() {- return new EmrServiceFixture(storage, clock, runner, iam, durations);+ return new EmrServiceFixture(storage, clock, runner, bootstrap, iam, durations);}/**@@ -124,7 +132,7 @@ final class EmrServiceFixture {*/EmrService restartReconciling(SparkJobRunner runner, EmrService.Wakeups wakeups) {return new EmrService(storage, config(true), new RegionResolver(REGION, ACCOUNT), publisher, clock, wakeups,- runner, stepThreads::add, iam);+ runner, bootstrap, stepThreads::add, iam);}private EmulatorConfig config(boolean reconcile) {diff --git a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.javaindex 9fc88dc5c3..9af0e299e9 100644--- a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java@@ -20,9 +20,9 @@ import org.junit.jupiter.api.Test;import org.mockito.ArgumentCaptor;import org.mockito.InOrder;-import java.io.ByteArrayInputStream;import java.io.Closeable;import java.io.IOException;+import java.io.InputStream;import java.io.StringReader;import java.nio.charset.StandardCharsets;import java.time.Duration;@@ -121,6 +121,9 @@ class EmrServerlessSparkRunnerTest {EmrServerlessSparkRunner.image(null, "emr-spark-8.1.0"));assertEquals("registry.internal/emr-serverless/spark/emr-7.5.0:20250425",EmrServerlessSparkRunner.image("registry.internal/emr-serverless/spark/%s:20250425", "emr-7.5.0"));+ // The image a cluster's bootstrap left, which its steps get by id.+ String bootstrapped = "sha256:" + "2".repeat(64);+ assertEquals(bootstrapped, EmrServerlessSparkRunner.image(bootstrapped, "emr-7.5.0"));}@Test@@ -359,7 +362,8 @@ class EmrServerlessSparkRunnerTest {InOrder order = inOrder(lifecycle, docker, s3);order.verify(lifecycle).stop("driver", 0);order.verify(docker).logContainerCmd("driver");- order.verify(s3).putObject(eq("logs"), eq("emr/j-1/steps/s-1/stdout.gz"), any(), anyString(), anyMap());+ order.verify(s3).putObjectFromStream(eq("logs"), eq("emr/j-1/steps/s-1/stdout.gz"), any(), anyString(),+ anyMap());order.verify(lifecycle).stopAndRemove("driver", null, 0);}@@ -382,7 +386,7 @@ class EmrServerlessSparkRunnerTest {assertEquals(0, outcome.exitCode());verify(stream).close();- verify(s3, never()).putObject(anyString(), anyString(), any(), anyString(), anyMap());+ verify(s3, never()).putObjectFromStream(anyString(), anyString(), any(), anyString(), anyMap());} finally {assertTrue(Thread.interrupted());}@@ -392,7 +396,7 @@ class EmrServerlessSparkRunnerTest {void logsThatCannotBeWrittenLeaveTheRunAsItEnded() {EmrServerlessSparkRunner runner = runner();driverWrote(new Frame(StreamType.STDOUT, bytes("done\n")));- when(s3.putObject(anyString(), anyString(), any(), anyString(), anyMap()))+ when(s3.putObjectFromStream(anyString(), anyString(), any(), anyString(), anyMap())).thenThrow(new AwsException("NoSuchBucket", "The specified bucket does not exist", 404));SparkJobRunner.Outcome outcome = runner.run(job("s3://missing/emr/j-1/steps/s-1/"), progress);@@ -413,7 +417,7 @@ class EmrServerlessSparkRunnerTest {assertTrue(outcome.stopped());verify(docker, never()).logContainerCmd(anyString());- verify(s3, never()).putObject(anyString(), anyString(), any(), anyString(), anyMap());+ verify(s3, never()).putObjectFromStream(anyString(), anyString(), any(), anyString(), anyMap());}@Test@@ -499,9 +503,9 @@ class EmrServerlessSparkRunnerTest {// The object the runner put at key, gunzipped.private String written(String bucket, String key) throws IOException {- ArgumentCaptor<byte[]> data = ArgumentCaptor.forClass(byte[].class);- verify(s3).putObject(eq(bucket), eq(key), data.capture(), eq("application/x-gzip"), eq(Map.of()));- try (GZIPInputStream gunzip = new GZIPInputStream(new ByteArrayInputStream(data.getValue()))) {+ ArgumentCaptor<InputStream> data = ArgumentCaptor.forClass(InputStream.class);+ verify(s3).putObjectFromStream(eq(bucket), eq(key), data.capture(), eq("application/x-gzip"), eq(Map.of()));+ try (GZIPInputStream gunzip = new GZIPInputStream(data.getValue())) {return new String(gunzip.readAllBytes(), StandardCharsets.UTF_8);}}