Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(emr): run command-runner.jar commands and script-runner.jar scripts (#6747)

Priamo Germosen897667f4e632
diff --git a/docs/services/emr.md b/docs/services/emr.md
index faa441a776..2ade067950 100644
--- a/docs/services/emr.md
+++ b/docs/services/emr.md
@@ -4,7 +4,7 @@
**Endpoint:** `http://localhost:4566/`
**Target prefix:** `X-Amz-Target: ElasticMapReduce.*`
-Floci emulates the Amazon EMR (Elastic MapReduce) management API. Clusters (job flows), instance groups and fleets, steps, security configurations, and tags are tracked in Floci's storage backend so the AWS CLI and SDK clients can drive the full cluster lifecycle locally. EMR does not launch real Hadoop/Spark clusters. Clusters transition through their lifecycle states as a state machine. In the `docker` runner mode Spark steps run for real, each in a container of its own (see [Running Spark steps](#running-spark-steps)).
+Floci emulates the Amazon EMR (Elastic MapReduce) management API. Clusters (job flows), instance groups and fleets, steps, security configurations, and tags are tracked in Floci's storage backend so the AWS CLI and SDK clients can drive the full cluster lifecycle locally. EMR does not launch real Hadoop/Spark clusters. Clusters transition through their lifecycle states as a state machine. In the `docker` runner mode Spark steps, commands and scripts run for real, each in a container of its own (see [Running steps](#running-steps)).
## Supported Actions
@@ -105,25 +105,38 @@ still accepted.
A state machine drives clusters and steps through the `elasticmapreduce:` integrations, `.sync`
included; see [Step Functions: Amazon EMR tasks](step-functions.md#amazon-emr-tasks).
-## Running Spark steps
+## Running steps
-With `runner-mode` set to `docker` (`FLOCI_SERVICES_EMR_RUNNER_MODE=docker`), a Spark step runs. Floci
-starts one container per step from the cluster's release image, the one EMR Serverless's `docker` mode
-uses (`public.ecr.aws/emr-serverless/spark/<release>`), and runs `spark-submit` in local mode in it: the
-container holds the driver and its executors. The cluster itself gets no container, and goes through
-its lifecycle as in the `immediate` mode.
+With `runner-mode` set to `docker` (`FLOCI_SERVICES_EMR_RUNNER_MODE=docker`), a step runs. Floci starts
+one container per step from the cluster's release image, the one EMR Serverless's `docker` mode uses
+(`public.ecr.aws/emr-serverless/spark/<release>`), and runs the step in it: a Spark step's
+`spark-submit` in local mode, the container holding the driver and its executors, or a command or a
+script. The cluster itself gets no container, and goes through its lifecycle as in the `immediate` mode.
* **Which steps run**: a `command-runner.jar` step whose first argument is `spark-submit`, the way the
Management Guide writes Spark steps, runs the arguments after it, without its `--master` and
`--deploy-mode`. A custom JAR step runs its `MainClass`, or else the one its manifest names, through
- `spark-submit`, with the JAR and the step's `Args`. Any other step, such as Hive, Pig,
- `script-runner.jar` or another `command-runner.jar` command, is `FAILED` without running.
-* **States**: a step is `RUNNING` from when it starts until its driver exits; `step-duration-seconds`
+ `spark-submit`, with the JAR and the step's `Args`. Any other `command-runner.jar` step runs its
+ command, its first argument, with the rest, and a `script-runner.jar` step (by name, or by its
+ `s3://<region>.elasticmapreduce/libs/script-runner/script-runner.jar` URI) its script, as the Release
+ Guide's "Run commands and scripts on an Amazon EMR cluster" describes them. `hive-script`,
+ `pig-script`, `hadoop-streaming`, `hadoop-lzo` and `s3-dist-cp`, which the image does not have, are
+ `FAILED` without running.
+* **Commands and scripts**: they run as the image's `hadoop` user in `/home/hadoop`, with `bash`, Python
+ 3, Java, `curl` and the AWS CLI v2, which reaches Floci through `AWS_ENDPOINT_URL`. A
+ `script-runner.jar` step's script is fetched from S3 with the AWS CLI and the step's credentials, then
+ run with the step's other arguments; a path not in S3 is run as it is, a bare name from `/home/hadoop`
+ when a file there has it, though nothing a former step wrote is there, each step having a container of
+ its own. A `spark-submit` they start runs in local mode on Floci's S3 with the cluster's
+ `spark-defaults`, which Floci puts in a Spark configuration of its own, ahead of the image's EMR
+ Serverless master; its own `--master yarn` or `--deploy-mode cluster` fails, as there is no YARN.
+* **States**: a step is `RUNNING` from when it starts until its driver or command exits; `step-duration-seconds`
does not apply. The first step of a release also waits while its image is pulled (3.5 to 4.3 GB
unpacked; see [`floci.docker.image-pull-timeout-seconds`](../configuration/docker.md)). It is then
`COMPLETED` on exit code 0, and `FAILED` otherwise. A `FAILED` step's `Status.FailureDetails` has the `Reason`
"Unknown Error", which the API reference says EMR gives when it cannot find the root cause, and a
- `Message` in Floci's own words, such as "The Spark driver exited with code 1.".
+ `Message` in Floci's own words, such as "The Spark driver exited with code 1." or "The step's command
+ exited with code 1.".
* **ActionOnFailure**: a failed step's action is taken as the API reference and the Management Guide's
termination protection table give it. `CONTINUE` runs the next step. `TERMINATE_CLUSTER` (or
`TERMINATE_JOB_FLOW`) ends the cluster, unless it waits after its steps
@@ -205,7 +218,7 @@ its `STARTING` event, has its events sent after the rest of that call's.
| `FLOCI_SERVICES_EMR_CLUSTER_STARTUP_DELAY_SECONDS` | `0` | Seconds a new cluster takes to be ready to run steps, the first half `STARTING` and the second `BOOTSTRAPPING`; `0` makes it ready as soon as it is created |
| `FLOCI_SERVICES_EMR_STEP_DURATION_SECONDS` | `0` | Seconds a step stays `RUNNING` before it completes; `0` completes it as soon as it starts |
| `FLOCI_SERVICES_EMR_CLUSTER_TERMINATION_DELAY_SECONDS` | `0` | Seconds a cluster stays `TERMINATING` before it is `TERMINATED`; `0` ends it at once |
-| `FLOCI_SERVICES_EMR_RUNNER_MODE` | `immediate` | `immediate` completes steps by time without running anything; `docker` runs each Spark step in a container (see [Running Spark steps](#running-spark-steps)) |
+| `FLOCI_SERVICES_EMR_RUNNER_MODE` | `immediate` | `immediate` completes steps by time without running anything; `docker` runs each step in a container (see [Running steps](#running-steps)) |
| `FLOCI_SERVICES_EMR_DOCKER_NETWORK` | the global default | Docker network for step containers |
| `FLOCI_SERVICES_EMR_IMAGE_TEMPLATE` | `public.ecr.aws/emr-serverless/spark/%s:latest`, or `spark4/` in place of `spark/` for an `emr-spark-*` release | Image for a `docker` step, `%s` standing for the cluster's release label |
| `FLOCI_SERVICES_EMR_RECONCILE_CONTAINERS_ON_STARTUP` | `true` | Remove the step drivers a previous run of this Floci left on the Docker daemon when the EMR service starts; turn it off when two Floci instances share a daemon with the same port and no resource namespace |
diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java
index 75470e50fd..d6453dc24b 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrService.java
@@ -1185,9 +1185,10 @@ public class EmrService implements Resettable {
* Why a step whose driver ended this way, by itself or before it could start, failed, or null when it
* did not. No source gives AWS's wording for these; they are Floci's.
*/
- private static String failureOf(SparkJobRunner.Outcome outcome) {
+ private static String failureOf(SparkJobRunner.Job job, SparkJobRunner.Outcome outcome) {
if (outcome.exitCode() != null) {
- return outcome.exitCode() == 0 ? null : "The Spark driver exited with code " + outcome.exitCode() + ".";
+ String ended = job.command() != null ? "The step's command" : "The Spark driver";
+ return outcome.exitCode() == 0 ? null : ended + " exited with code " + outcome.exitCode() + ".";
}
return "The step could not start: " + outcome.failure();
}
@@ -1211,11 +1212,13 @@ public class EmrService implements Resettable {
public void start(EmrCluster cluster, EmrStep step, String region) {
String accountId = regionResolver.getAccountId();
List<String> application = EmrSparkSteps.application(step);
- SparkJobRunner.Job job = application == null ? null : new SparkJobRunner.Job(SERVICE, step.getId(),
- accountId, region, cluster.getReleaseLabel(), imageTemplate, dockerNetwork,
- executionRole(cluster, step, accountId),
- EmrSparkSteps.sparkDefaults(cluster.getConfigurations(), SNAPSHOT_MAPPER), application,
- Duration.ZERO, EmrSparkSteps.logFolder(cluster, step), null);
+ List<String> command = application == null ? EmrSparkSteps.command(step) : null;
+ SparkJobRunner.Job job = application == null && command == null ? null
+ : new SparkJobRunner.Job(SERVICE, step.getId(), accountId, region, cluster.getReleaseLabel(),
+ imageTemplate, dockerNetwork, executionRole(cluster, step, accountId),
+ EmrSparkSteps.sparkDefaults(cluster.getConfigurations(), SNAPSHOT_MAPPER),
+ application != null ? application : List.of(), command, Duration.ZERO,
+ EmrSparkSteps.logFolder(cluster, step), null);
StepLaunch launch = new StepLaunch(accountId, region, cluster.getId(), step.getId(), generation);
executing.add(stepKey(accountId, step.getId()));
stepExecutor.execute(() -> runStep(job, launch));
@@ -1244,7 +1247,7 @@ public class EmrService implements Resettable {
RequestScopes.runAs(launch.accountId(), launch.region(), () -> run(() -> driverStopped(launch)));
return;
}
- failure = failureOf(outcome);
+ failure = failureOf(job, outcome);
}
RequestScopes.runAs(launch.accountId(), launch.region(), () -> run(() -> stepEnded(launch, failure)));
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/emr/EmrSparkSteps.java b/src/main/java/io/github/hectorvent/floci/services/emr/EmrSparkSteps.java
index 39588c1d80..0ad0a56e54 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emr/EmrSparkSteps.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emr/EmrSparkSteps.java
@@ -14,16 +14,18 @@ import java.util.Map;
import java.util.Set;
/**
- * What a step runs in the {@code docker} runner mode, which runs Spark steps only, each through
- * {@code spark-submit} in local mode: a {@code command-runner.jar} step running {@code spark-submit}, as
- * the Management Guide's Spark steps are written, and a custom JAR step, whose main class
- * {@code spark-submit} runs with the JAR on its class path.
+ * What a step runs in the {@code docker} runner mode, in a container of the release's EMR Serverless image.
+ * A Spark step runs through {@code spark-submit} in local mode: a {@code command-runner.jar} step running
+ * {@code spark-submit}, as the Management Guide's Spark steps are written, and a custom JAR step, whose main
+ * class {@code spark-submit} runs with the JAR on its class path. Any other {@code command-runner.jar} step
+ * runs its command, and a {@code script-runner.jar} step its script, as the Release Guide's "Run commands
+ * and scripts on an Amazon EMR cluster" describes them.
*/
final class EmrSparkSteps {
// No source gives AWS's wording for these; they are Floci's.
- static final String NOT_RUN = "Floci runs Spark steps only: spark-submit through command-runner.jar, or a"
- + " custom JAR.";
+ static final String NOT_RUN = "Floci does not run this step: hive-script, pig-script, hadoop-streaming,"
+ + " hadoop-lzo and s3-dist-cp are not in the image steps run in.";
static final String INTERRUPTED = "Floci stopped while the step was running.";
private static final String COMMAND_RUNNER = "command-runner.jar";
@@ -35,6 +37,31 @@ final class EmrSparkSteps {
private static final Set<String> SWITCHES = Set.of("--help", "-h", "--supervise", "--usage-error", "--verbose",
"-v", "--version");
private static final String SPARK_DEFAULTS = "spark-defaults";
+ // The Release Guide's command-runner.jar tools that the EMR Serverless image does not have; its hadoop
+ // has no HDFS to run streaming or the LZO indexer against.
+ private static final Set<String> UNAVAILABLE_TOOLS = Set.of("hive-script", "pig-script", "hadoop-streaming",
+ "hadoop-lzo", "s3-dist-cp");
+ // script-runner.jar fetches its script from S3 with the step's credentials, the image's AWS CLI reaching
+ // Floci's S3, and runs it with its arguments; a path that is not in S3 is run as it is, a bare name from
+ // the working directory when a file there has it, as exec would otherwise look for it on the PATH only.
+ private static final String SCRIPT_RUNNER_COMMAND = String.join("\n",
+ "script=$1",
+ "shift",
+ "case \"$script\" in",
+ " s3://*|s3n://*|s3a://*)",
+ " mkdir -p /tmp/floci-script-runner",
+ " local_copy=\"/tmp/floci-script-runner/${script##*/}\"",
+ " aws s3 cp --only-show-errors \"s3://${script#*://}\" \"$local_copy\" || exit $?",
+ " script=$local_copy",
+ " ;;",
+ " */*)",
+ " ;;",
+ " *)",
+ " if [ -f \"$script\" ]; then script=\"./$script\"; fi",
+ " ;;",
+ "esac",
+ "chmod u+x \"$script\"",
+ "exec \"$script\" \"$@\"");
private EmrSparkSteps() {
}
@@ -67,6 +94,29 @@ final class EmrSparkSteps {
return application;
}
+ /**
+ * The command a step that is not a Spark step runs, or null for one Floci does not run: a
+ * {@code command-runner.jar} step's arguments, its command first, unless it names a tool the image does
+ * not have; a {@code script-runner.jar} step's script, the first of its arguments, with the rest.
+ */
+ static List<String> command(EmrStep step) {
+ String jar = step.getJar();
+ List<String> args = step.getArgs() != null ? step.getArgs() : List.of();
+ if (jar == null || args.isEmpty()) {
+ return null;
+ }
+ if (named(jar, COMMAND_RUNNER)) {
+ return named(args.getFirst(), SPARK_SUBMIT) || UNAVAILABLE_TOOLS.contains(args.getFirst())
+ ? null : List.copyOf(args);
+ }
+ if (named(jar, SCRIPT_RUNNER)) {
+ List<String> command = new ArrayList<>(List.of("bash", "-c", SCRIPT_RUNNER_COMMAND, "script-runner"));
+ command.addAll(args);
+ return command;
+ }
+ return null;
+ }
+
/**
* The S3 folder a step's logs go to, or null when its cluster has no {@code LogUri}: "If a value is not
* provided, logs are not created". The Management Guide's "Log locations" puts step logs at
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java
index bc48304773..b47961a3dc 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java
@@ -464,7 +464,7 @@ public class EmrServerlessJobRunService implements Resettable {
? Duration.ZERO : Duration.ofMinutes(minutes);
SparkJobRunner.Job job = new SparkJobRunner.Job(SERVICE, run.getJobRunId(), arn.accountId(), arn.region(),
run.getReleaseLabel(), imageTemplate, dockerNetwork, run.getExecutionRole(), sparkDefaults(app, run),
- application(run.getJobDriver().getSparkSubmit()), timeout, driverLogFolder(app, run),
+ application(run.getJobDriver().getSparkSubmit()), null, timeout, driverLogFolder(app, run),
driverCloudWatchLogs(app, run));
executing.add(run.getJobRunId());
long launchedIn = generation;
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java
index 6948aa9cf0..e3461299d3 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java
@@ -54,6 +54,21 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo
private static final Logger LOG = Logger.getLogger(EmrServerlessSparkRunner.class);
private static final String SPARK_SUBMIT = "/usr/lib/spark/bin/spark-submit";
+ private static final String SPARK_DEFAULTS_VARIABLE = "FLOCI_SPARK_DEFAULTS";
+ // A command runs after this sets up a Spark configuration of its own, so a spark-submit it starts runs
+ // locally on Floci's S3: the image's spark-defaults.conf names EMR Serverless's resource manager as its
+ // master, which a system property cannot override, so its master and that manager's lines give way to
+ // local mode, Floci's settings and the job's spark-defaults, passed in SPARK_DEFAULTS_VARIABLE.
+ private static final String COMMAND_WRAPPER = String.join("\n",
+ "conf=/tmp/floci-spark-conf",
+ "mkdir -p \"$conf\"",
+ "cp -r /usr/lib/spark/conf/. \"$conf\" 2>/dev/null",
+ "touch \"$conf/spark-defaults.conf\"",
+ "sed -i -e '/^spark\\.master[[:space:]]/d' -e '/^spark\\.submit\\.deployMode[[:space:]]/d'"
+ + " -e '/^spark\\.submit\\.customResourceManager/d' \"$conf/spark-defaults.conf\"",
+ "printf '%s\\n' \"$" + SPARK_DEFAULTS_VARIABLE + "\" >> \"$conf/spark-defaults.conf\"",
+ "export SPARK_CONF_DIR=\"$conf\"",
+ "exec \"$@\"");
private static final String DEFAULT_IMAGE =
"public.ecr.aws/emr-serverless/spark/%s:latest"; // partition-literal: ECR Public's registry host, where AWS publishes the base images; image-template overrides it
// The AWS runtime for Apache Spark (emr-spark-8.0.0 on) publishes its images under spark4/ instead.
@@ -125,11 +140,19 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo
// The deployment that started the driver, which removeLeftoverDrivers() goes by.
labels.put(ContainerStorageHelper.OWNER_LABEL, ContainerStorageHelper.ownerIdentity(config));
String endpoint = awsEnv.flociEndpoint();
- ContainerSpec spec = containerBuilder.newContainer(image(job.imageTemplate(), job.releaseLabel()))
+ ContainerBuilder.Builder container = containerBuilder.newContainer(
+ image(job.imageTemplate(), job.releaseLabel()))
.withName(ContainerStorageHelper.dockerName(config, job.service() + "-" + job.runId()))
- .withEntrypoint(List.of(SPARK_SUBMIT))
- .withCmd(sparkSubmitArguments(job, endpoint))
- .withEnv(awsEnv.sdkBaselineEnv(job.region(), Optional.empty(), credentials, job.accountId()))
+ .withEnv(awsEnv.sdkBaselineEnv(job.region(), Optional.empty(), credentials, job.accountId()));
+ if (job.command() == null) {
+ container.withEntrypoint(List.of(SPARK_SUBMIT)).withCmd(sparkSubmitArguments(job, endpoint));
+ } else {
+ // bash -c's script gets "floci" as its $0, and the command as its arguments, "$@".
+ container.withEntrypoint(List.of("/bin/bash", "-c", COMMAND_WRAPPER, "floci"))
+ .withCmd(job.command())
+ .withEnv(SPARK_DEFAULTS_VARIABLE, sparkDefaultsFile(job, endpoint));
+ }
+ ContainerSpec spec = container
.withDockerNetwork(Optional.ofNullable(job.dockerNetwork()))
.withHostDockerInternalOnLinux()
.withEmbeddedDns()
@@ -276,19 +299,64 @@ public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardo
*/
static List<String> sparkSubmitArguments(Job job, String flociEndpoint) {
List<String> arguments = new ArrayList<>(List.of("--master", "local[*]"));
- conf(arguments, "spark.hadoop.fs.s3a.endpoint", flociEndpoint);
- conf(arguments, "spark.hadoop.fs.s3a.endpoint.region", job.region());
- conf(arguments, "spark.hadoop.fs.s3a.path.style.access", "true");
- conf(arguments, "spark.hadoop.fs.s3a.committer.magic.track.commits.in.memory.enabled", "true");
- conf(arguments, "spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
- conf(arguments, "spark.hadoop.fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3A");
- conf(arguments, "spark.hadoop.aws.glue.endpoint", flociEndpoint);
- conf(arguments, "spark.hadoop.aws.region", job.region());
- job.sparkDefaults().forEach((key, value) -> conf(arguments, key, value));
+ for (SparkSetting setting : sparkSettings(job, flociEndpoint)) {
+ conf(arguments, setting.key(), setting.value());
+ }
arguments.addAll(job.application());
return arguments;
}
+ /**
+ * The {@code spark-defaults.conf} lines a command's own {@code spark-submit} runs with: local mode, then
+ * the settings {@link #sparkSubmitArguments} passes, a later line winning over an earlier one, and the
+ * submit's own options over them all. Spark reads the file as a {@code Properties} file, so each
+ * key and value is escaped for it, to reach Spark as the job gave it.
+ */
+ static String sparkDefaultsFile(Job job, String flociEndpoint) {
+ StringBuilder file = new StringBuilder("spark.master local[*]");
+ for (SparkSetting setting : sparkSettings(job, flociEndpoint)) {
+ file.append('\n').append(propertiesEscaped(setting.key(), true)).append(' ')
+ .append(propertiesEscaped(String.valueOf(setting.value()), false));
+ }
+ return file.toString();
+ }
+
+ // A backslash starts an escape and a line break ends a setting; in a key, whitespace, '=' and ':' end it,
+ // and a leading '#' or '!' makes the line a comment.
+ private static String propertiesEscaped(String text, boolean key) {
+ StringBuilder escaped = new StringBuilder();
+ for (char c : text.toCharArray()) {
+ switch (c) {
+ case '\\' -> escaped.append("\\\\");
+ case '\n' -> escaped.append("\\n");
+ case '\r' -> escaped.append("\\r");
+ case '\t' -> escaped.append("\\t");
+ case '\f' -> escaped.append("\\f");
+ case ' ', '=', ':', '#', '!' -> escaped.append(key ? "\\" : "").append(c);
+ default -> escaped.append(c);
+ }
+ }
+ return escaped.toString();
+ }
+
+ // Floci's settings, then the job's spark-defaults, in the order a later one wins.
+ private static List<SparkSetting> sparkSettings(Job job, String flociEndpoint) {
+ List<SparkSetting> settings = new ArrayList<>(List.of(
+ new SparkSetting("spark.hadoop.fs.s3a.endpoint", flociEndpoint),
+ new SparkSetting("spark.hadoop.fs.s3a.endpoint.region", job.region()),
+ new SparkSetting("spark.hadoop.fs.s3a.path.style.access", "true"),
+ new SparkSetting("spark.hadoop.fs.s3a.committer.magic.track.commits.in.memory.enabled", "true"),
+ new SparkSetting("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem"),
+ new SparkSetting("spark.hadoop.fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3A"),
+ new SparkSetting("spark.hadoop.aws.glue.endpoint", flociEndpoint),
+ new SparkSetting("spark.hadoop.aws.region", job.region())));
+ job.sparkDefaults().forEach((key, value) -> settings.add(new SparkSetting(key, value)));
+ return settings;
+ }
+
+ private record SparkSetting(String key, String value) {
+ }
+
/**
* {@code spark-submit} parameters given as one string, as EMR Serverless's
* {@code sparkSubmitParameters}, split as a POSIX shell would split them: on whitespace, except inside single
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java
index 2677bea5a2..48dbb2d6ec 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java
@@ -53,14 +53,18 @@ public interface SparkJobRunner {
* the release's public EMR Serverless image, a null network the global default); the execution role
* whose credentials it gets; the {@code spark-defaults} properties that apply to it;
* {@code application}, what follows Floci's own settings on {@code spark-submit}'s command line (the
- * run's options, then its application and that application's arguments); how long it may run
+ * run's options, then its application and that application's arguments); or {@code command}, when not
+ * null, what the container runs in place of {@code spark-submit}, where a {@code spark-submit} it starts
+ * gets Floci's settings and the {@code spark-defaults} as defaults (its {@code application} is then
+ * empty); how long it may run
* ({@link Duration#ZERO} for no limit); the S3 folder its driver's {@code stdout.gz} and
* {@code stderr.gz} go to once it ends, ending in a slash (null for none); and where in CloudWatch Logs
* its driver's output goes while it runs (null for nowhere but Floci's log).
*/
record Job(String service, String runId, String accountId, String region, String releaseLabel,
String imageTemplate, String dockerNetwork, String executionRole, Map<String, String> sparkDefaults,
- List<String> application, Duration timeout, String logFolder, CloudWatchLogs cloudWatchLogs) {
+ List<String> application, List<String> command, Duration timeout, String logFolder,
+ CloudWatchLogs cloudWatchLogs) {
}
/**
diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java
index 552a0581f0..174b28ff36 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrServiceContainerStepsTest.java
@@ -79,6 +79,7 @@ class EmrServiceContainerStepsTest {
assertNull(job.imageTemplate());
assertNull(job.dockerNetwork());
assertEquals(List.of(REPORT, "s3://bucket/out/"), job.application());
+ assertNull(job.command());
assertEquals(Map.of("spark.executor.cores", "2"), job.sparkDefaults());
assertEquals(Duration.ZERO, job.timeout());
assertNull(job.logFolder());
@@ -215,6 +216,38 @@ class EmrServiceContainerStepsTest {
assertEquals("RUNNING", service.describeStep(cluster.getId(), cluster.getSteps().get(1).getId()).getState());
}
+ @Test
+ void aCommandStepRunsItsCommandAndFailsWithItsExitCode() {
+ EmrStep copy = sparkStep("copy", "CONTINUE");
+ copy.setArgs(List.of("bash", "-c", "aws s3 cp s3://bucket/in/ s3://bucket/out/ --recursive"));
+ EmrCluster cluster = service.runJobFlow(cluster("commands", true, copy), REGION);
+ runner.outcome = SparkJobRunner.Outcome.exitedWith(1);
+
+ fixture.runStepThread();
+
+ SparkJobRunner.Job job = runner.jobs.get(0);
+ assertEquals(copy.getArgs(), job.command());
+ assertEquals(List.of(), job.application());
+ EmrStep failed = service.describeStep(cluster.getId(), cluster.getSteps().get(0).getId());
+ assertEquals("FAILED", failed.getState());
+ assertEquals("The step's command exited with code 1.", failed.getFailureMessage());
+ }
+
+ @Test
+ void aScriptRunnerStepRunsItsScriptAsACommand() {
+ EmrStep script = sparkStep("script", "CONTINUE");
+ script.setJar("s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar");
+ script.setArgs(List.of("s3://bucket/scripts/load.sh", "2026-10-26"));
+ EmrCluster cluster = service.runJobFlow(cluster("scripts", true, script), REGION);
+
+ fixture.runStepThread();
+
+ List<String> command = runner.jobs.get(0).command();
+ assertEquals(List.of("script-runner", "s3://bucket/scripts/load.sh", "2026-10-26"),
+ command.subList(3, command.size()));
+ assertEquals("COMPLETED", service.describeStep(cluster.getId(), cluster.getSteps().get(0).getId()).getState());
+ }
+
@Test
void aStepFlociDoesNotRunFailsWithoutRunning() {
EmrStep hive = sparkStep("query", "CONTINUE");
diff --git a/src/test/java/io/github/hectorvent/floci/services/emr/EmrSparkStepsTest.java b/src/test/java/io/github/hectorvent/floci/services/emr/EmrSparkStepsTest.java
index 8e72979703..7e6daa6d6b 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emr/EmrSparkStepsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emr/EmrSparkStepsTest.java
@@ -2,13 +2,18 @@ package io.github.hectorvent.floci.services.emr;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.github.hectorvent.floci.services.emr.model.EmrStep;
+import io.github.hectorvent.floci.services.lambda.launcher.BootstrapProcess;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import java.nio.file.Files;
+import java.nio.file.Path;
import java.util.List;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
class EmrSparkStepsTest {
@@ -48,6 +53,56 @@ class EmrSparkStepsTest {
assertNull(EmrSparkSteps.application(step(null, null)));
}
+ @Test
+ void aCommandRunnerStepThatIsNotSparkRunsItsCommand() {
+ // The Release Guide's own example: download a script, make it executable, run it.
+ List<String> bash = List.of("bash", "-c",
+ "aws s3 cp s3://bucket/my-script.sh /home/hadoop; chmod u+x /home/hadoop/my-script.sh;"
+ + " cd /home/hadoop; ./my-script.sh");
+
+ assertEquals(bash, EmrSparkSteps.command(step("command-runner.jar", null, bash.toArray(String[]::new))));
+ assertEquals(List.of("python3", "-c", "print(1)"),
+ EmrSparkSteps.command(step("command-runner.jar", null, "python3", "-c", "print(1)")));
+ assertNull(EmrSparkSteps.command(
+ step("command-runner.jar", null, "spark-submit", "s3://bucket/jobs/report.py")));
+ }
+
+ @Test
+ void aScriptRunnerStepFetchesItsScriptAndRunsItWithItsArguments() {
+ for (String jar : List.of("script-runner.jar",
+ "s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar")) {
+ List<String> command = EmrSparkSteps.command(step(jar, null, "s3://bucket/my-script.sh", "--day", "1"));
+
+ assertEquals(List.of("bash", "-c"), command.subList(0, 2));
+ assertTrue(command.get(2).contains("aws s3 cp --only-show-errors"));
+ assertEquals(List.of("script-runner", "s3://bucket/my-script.sh", "--day", "1"),
+ command.subList(3, command.size()));
+ }
+ }
+
+ @Test
+ void aScriptRunnerScriptNamedWithoutAPathRunsFromTheWorkingDirectory(@TempDir Path workingDirectory)
+ throws Exception {
+ // Not executable yet: script-runner makes it so, as it does a script it fetched from S3.
+ Files.writeString(workingDirectory.resolve("my-script.sh"), "#!/bin/sh\nprintf 'ran:%s|' \"$@\"\n");
+ List<String> command = EmrSparkSteps.command(step("script-runner.jar", null, "my-script.sh", "--day", "1 2"));
+
+ BootstrapProcess.Result ended =
+ BootstrapProcess.run(new ProcessBuilder(command).directory(workingDirectory.toFile()));
+
+ assertEquals(new BootstrapProcess.Result(0, "ran:--day|ran:1 2|"), ended);
+ }
+
+ @Test
+ void toolsTheImageDoesNotHaveAreNotRun() {
+ for (String tool : List.of("hive-script", "pig-script", "hadoop-streaming", "hadoop-lzo", "s3-dist-cp")) {
+ assertNull(EmrSparkSteps.command(step("command-runner.jar", null, tool, "--src", "s3://bucket/in/")));
+ }
+ assertNull(EmrSparkSteps.command(step("command-runner.jar", null)));
+ assertNull(EmrSparkSteps.command(step("script-runner.jar", null)));
+ assertNull(EmrSparkSteps.command(step("s3://bucket/jars/app.jar", null, "1")));
+ }
+
@Test
void theClustersSparkDefaultsComeFromItsConfigurations() {
String configurations = """
diff --git a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java
index 50ef0de074..9fc88dc5c3 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java
@@ -23,12 +23,14 @@ import org.mockito.InOrder;
import java.io.ByteArrayInputStream;
import java.io.Closeable;
import java.io.IOException;
+import java.io.StringReader;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Properties;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.zip.GZIPInputStream;
@@ -130,12 +132,71 @@ class EmrServerlessSparkRunnerTest {
runner(builder).run(job(Duration.ZERO), progress);
verify(builder).newContainer("public.ecr.aws/emr-serverless/spark/emr-7.5.0:latest");
+ verify(spec).withEntrypoint(List.of("/usr/lib/spark/bin/spark-submit"));
verify(spec).withName(ContainerStorageHelper.dockerName(config, "emr-serverless-00f0000000000001"));
verify(spec).withDockerNetwork(Optional.of("floci-network"));
verify(spec).withLabels(argThat(labels ->
"emr-serverless".equals(labels.get(ContainerStorageHelper.SERVICE_LABEL))));
}
+ @Test
+ void aCommandRunsThroughFlocisWrapperWithTheJobsSparkSettingsInItsEnvironment() {
+ ContainerBuilder builder = mock(ContainerBuilder.class);
+ ContainerBuilder.Builder spec = mock(ContainerBuilder.Builder.class, RETURNS_SELF);
+ when(builder.newContainer(anyString())).thenReturn(spec);
+ List<String> command = List.of("bash", "-c", "aws s3 ls s3://bucket/");
+
+ runner(builder).run(commandJob(command, Map.of("spark.executor.cores", "4")), progress);
+
+ // bash -c runs Floci's wrapper with "floci" as its $0 and the command as "$@".
+ verify(spec).withEntrypoint(argThat(entrypoint -> entrypoint.size() == 4
+ && entrypoint.subList(0, 2).equals(List.of("/bin/bash", "-c"))
+ && entrypoint.get(2).contains("export SPARK_CONF_DIR=") && entrypoint.get(2).endsWith("exec \"$@\"")
+ && "floci".equals(entrypoint.get(3))));
+ verify(spec).withCmd(command);
+ verify(spec).withEnv(eq("FLOCI_SPARK_DEFAULTS"),
+ argThat(file -> file.startsWith("spark.master local[*]\n")
+ && file.endsWith("\nspark.executor.cores 4")));
+ verify(spec, never()).withEntrypoint(List.of("/usr/lib/spark/bin/spark-submit"));
+ }
+
+ @Test
+ void aCommandsOwnSparkSubmitRunsLocallyOnFlocisS3WithTheJobsSparkDefaults() {
+ assertEquals(String.join("\n",
+ "spark.master local[*]",
+ "spark.hadoop.fs.s3a.endpoint http://floci:4566",
+ "spark.hadoop.fs.s3a.endpoint.region eu-west-1",
+ "spark.hadoop.fs.s3a.path.style.access true",
+ "spark.hadoop.fs.s3a.committer.magic.track.commits.in.memory.enabled true",
+ "spark.hadoop.fs.s3.impl org.apache.hadoop.fs.s3a.S3AFileSystem",
+ "spark.hadoop.fs.AbstractFileSystem.s3.impl org.apache.hadoop.fs.s3a.S3A",
+ "spark.hadoop.aws.glue.endpoint http://floci:4566",
+ "spark.hadoop.aws.region eu-west-1",
+ "spark.sql.shuffle.partitions 8"),
+ EmrServerlessSparkRunner.sparkDefaultsFile(
+ commandJob(List.of("bash", "-c", "spark-submit s3://bucket/jobs/app.py"),
+ Map.of("spark.sql.shuffle.partitions", "8")), ENDPOINT));
+ }
+
+ @Test
+ void aCommandsSparkDefaultsReachSparkAsTheJobGaveThem() throws IOException {
+ Map<String, String> sparkDefaults = new LinkedHashMap<>();
+ sparkDefaults.put("spark.driver.extraJavaOptions", "-Dpattern=\\d+ -Dpath=C:\\tmp");
+ sparkDefaults.put("spark.odd key=with:separators", "#not a comment\nnor a second line\t!");
+ sparkDefaults.put("!spark.leading", "kept");
+
+ // Spark's Utils.getPropertiesFromFile loads spark-defaults.conf with Properties.load.
+ Properties loaded = new Properties();
+ loaded.load(new StringReader(EmrServerlessSparkRunner.sparkDefaultsFile(
+ commandJob(List.of("bash", "-c", "spark-submit s3://bucket/jobs/app.py"), sparkDefaults), ENDPOINT)));
+
+ assertEquals("local[*]", loaded.getProperty("spark.master"));
+ assertEquals("-Dpattern=\\d+ -Dpath=C:\\tmp", loaded.getProperty("spark.driver.extraJavaOptions"));
+ assertEquals("#not a comment\nnor a second line\t!", loaded.getProperty("spark.odd key=with:separators"));
+ assertEquals("kept", loaded.getProperty("!spark.leading"));
+ assertEquals(ENDPOINT, loaded.getProperty("spark.hadoop.fs.s3a.endpoint"));
+ }
+
@Test
void aTemplateThatNamesNoImageFailsTheRunWithItsReason() {
EmrServerlessSparkRunner runner = runner();
@@ -488,6 +549,12 @@ class EmrServerlessSparkRunnerTest {
SparkJobRunner.CloudWatchLogs cloudWatchLogs) {
return new SparkJobRunner.Job("emr-serverless", "00f0000000000001", "123456789012", "eu-west-1", "emr-7.5.0",
imageTemplate, "floci-network", "arn:aws:iam::123456789012:role/job", sparkDefaults, application,
- timeout, logFolder, cloudWatchLogs);
+ null, timeout, logFolder, cloudWatchLogs);
+ }
+
+ private static SparkJobRunner.Job commandJob(List<String> command, Map<String, String> sparkDefaults) {
+ return new SparkJobRunner.Job("emr", "s-1A2B3C4D5E6F7", "123456789012", "eu-west-1", "emr-7.5.0", null,
+ "floci-network", "arn:aws:iam::123456789012:role/job", sparkDefaults, List.of(), command,
+ Duration.ZERO, null, null);
}
}