Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(emr-serverless): run Spark job runs in a container (#6469)

Priamo Germosenedbcac33c770
diff --git a/docs/services/emr-serverless.md b/docs/services/emr-serverless.md
index da246d4c75..98d3f5fc0e 100644
--- a/docs/services/emr-serverless.md
+++ b/docs/services/emr-serverless.md
@@ -9,6 +9,10 @@ EMR Serverless is an AWS service that provides serverless data analytics environ
| `floci.emrserverless.enabled` | Whether the EMR Serverless API is enabled | `true` |
| `floci.emrserverless.port` | The port the service is exposed on | `4566` |
| `floci.services.emrserverless.job-run-duration-seconds` | How long a `BATCH` job run stays `RUNNING` before it succeeds; `0` makes it `SUCCESS` as soon as it starts | `0` |
+| `floci.services.emrserverless.runner-mode` | `immediate` moves job runs through their states without running anything; `docker` runs each Spark job run in a container (see [Running Spark job runs](#running-spark-job-runs)) | `immediate` |
+| `floci.services.emrserverless.docker-network` | Docker network for job run containers | the global default |
+| `floci.services.emrserverless.image-template` | Image for a `docker` job run, `%s` standing for the release label | `public.ecr.aws/emr-serverless/spark/%s:latest`, or `spark4/` in place of `spark/` for an `emr-spark-*` release |
+| `floci.services.emrserverless.reconcile-containers-on-startup` | Remove the drivers a previous run of this Floci left on the Docker daemon before the first job run starts; turn it off when two Floci instances share a daemon with the same port and no resource namespace | `true` |
## Endpoints
@@ -57,8 +61,9 @@ omits the terminated application. A terminated application cannot be started or
## Job runs
-Job runs are metadata only: nothing executes. A run follows the job run states and `GetJobRun`'s
-response shape, and settles when it is read.
+In the default `immediate` runner mode, job runs are metadata only: nothing executes. A run follows
+the job run states and `GetJobRun`'s response shape, and settles when it is read. In the `docker`
+runner mode a Spark run executes; see [Running Spark job runs](#running-spark-job-runs).
* **Lifecycle**: a run is `RUNNING` from the moment it is submitted. With the default
`job-run-duration-seconds` of `0`, a `BATCH` run is `SUCCESS` as soon as it starts; a positive
@@ -82,7 +87,7 @@ response shape, and settles when it is read.
state in the order the filter names them. A state or mode that does not exist, or more than 8
states, is refused with `ValidationException`. An unknown application has no runs. A `nextToken`
resumes only the listing it came from, with the same application and filters.
-* **Attempts**: nothing runs, so no run fails and none is retried: every run is a single attempt,
+* **Attempts**: no run is retried, and in the `immediate` mode none fails: every run is a single attempt,
number 1. `GetJobRun`, `ListJobRuns` and `ListJobRunAttempts` report it, with the run's own
creation and update times. `GetJobRun` with `attempt` 1, or without one, returns the run; an
`attempt` of 2 or more answers `ResourceNotFoundException`, and one below 1 or not an integer
@@ -93,11 +98,44 @@ response shape, and settles when it is read.
A state machine drives applications and job runs through the `emr-serverless:` integrations, `.sync`
included; see [Step Functions: Amazon EMR Serverless tasks](step-functions.md#amazon-emr-serverless-tasks).
+## Running Spark job runs
+
+With `runner-mode` set to `docker` (`FLOCI_SERVICES_EMRSERVERLESS_RUNNER_MODE=docker`), a Spark job
+run executes. Floci starts one container per run from the release's image, the base image AWS
+publishes for customizing EMR Serverless images (`public.ecr.aws/emr-serverless/spark/<release>`, and
+`spark4/<release>` for the AWS runtime for Apache Spark releases, `emr-spark-8.0.0` on), and runs
+`spark-submit` in local mode in it: the container holds the driver and its executors.
+
+* **States**: a run is `SCHEDULED` while its image is pulled and its container starts (the first run
+ of a release pulls its image, 3.5 to 4.3 GB unpacked; Floci gives a pull three attempts of five
+ minutes, so on a slow connection run `docker pull` on the image first, or the run is `FAILED` with
+ "Could not pull image"), and `RUNNING` once the Spark driver runs, which sets `startedAt`. It
+ is `SUCCESS` when the driver exits with code 0 and `FAILED` otherwise, with the exit code in
+ `stateDetails`. `CancelJobRun`, or a `BATCH` run passing its `executionTimeoutMinutes`, makes it
+ `CANCELLING` until the driver is stopped, then `CANCELLED`, as the job run states describe. A run
+ Floci was running when it stopped is `FAILED` once it is read again. Floci's shutdown and a state
+ reset stop every driver, one whose image is still being pulled included, and a driver a hard stop
+ left running is removed the first time job runs are used after Floci restarts, unless
+ `reconcile-containers-on-startup` is off.
+* **What `spark-submit` gets**: the `entryPoint`, its `entryPointArguments` and the
+ `sparkSubmitParameters`, after the `spark-defaults` properties of the application's
+ `runtimeConfiguration`, or the run's own in `configurationOverrides.applicationConfiguration`,
+ which replace the application's whole `spark-defaults` classification. From `emr-spark-8.1.0` the
+ two are merged by property instead, the run's winning. `s3://` and `s3a://` paths reach Floci's S3 through S3A, on releases before 7.10.0
+ too, where `s3://` would otherwise be EMRFS; the entry point and its files can be read from there.
+ The Glue Data Catalog client is pointed at Floci's Glue.
+* **Credentials**: the container gets a session for the run's execution role when IAM knows the role,
+ otherwise the account's placeholder credentials, and `AWS_ENDPOINT_URL` pointing at Floci, so the
+ job's own SDK clients reach Floci too.
+* **Output**: the driver's output goes to Floci's log, prefixed `emr-serverless:<jobRunId>`.
+* **Image**: `image-template` names another image, such as a mirror or a dated tag of the release's
+ image: `registry.internal/emr-serverless/spark/%s:20250425`.
+
## Limitations and Differences from AWS
-* **Jobs do not execute**: `StartJobRun` records a job run and moves it through its states, but no Spark or Hive job runs.
+* **Jobs do not execute by default**: in the `immediate` runner mode `StartJobRun` records a job run and moves it through its states, but nothing runs. In the `docker` mode a Spark run executes in local mode in one container, so worker counts and sizes, initial and maximum capacity are not applied; a Hive run is `FAILED` without running, in Floci's own wording, as is a driver's failure. The monitoring configuration's S3 and CloudWatch log destinations are not written to, and Floci's CA is not in the job's Java truststore, so a job reaches Floci over its HTTP endpoint only.
* **Unsourced errors**: AWS documents neither the error for a run submitted to an application whose auto-start is off nor the one for a run on a terminated application; Floci answers both with `ValidationException`. The same holds for a `BATCH` timeout outside 1 to 10080 minutes on `emr-7.11.0` or later (`ValidationException`) and for `GetJobRun` naming an attempt a run never made (`ResourceNotFoundException`), both in Floci's own wording. A run needs a `jobDriver`, as every `JobRun` carries one, and `jobDriver` is a union, so a run without a driver, or with one that sets both `sparkSubmit` and `hive` or neither, is refused with `ValidationException` too, in Floci's own wording.
* **StopApplication with running jobs**: AWS requires running jobs to finish or be cancelled before an application stops. Floci does not enforce it; with the default duration of `0` no `BATCH` run is ever running.
-* **Not emulated**: retries (`retryPolicy` is stored and returned, but no run fails, so none is retried), queuing (`QUEUED`), `totalResourceUtilization` and `billedResourceUtilization`, and `shutdownGracePeriodInSeconds` (accepted and ignored).
+* **Not emulated**: retries (`retryPolicy` is stored and returned, but no run is retried), queuing (`QUEUED`), `totalResourceUtilization` and `billedResourceUtilization`, and `shutdownGracePeriodInSeconds` (accepted and ignored: a cancelled run's driver is stopped at once).
* **Instant Start/Stop**: Floci marks the application `STARTED` or `STOPPED` immediately without provisioning actual compute capacity in the background.
-* **Data Plane**: Floci does not implement the data-plane API or any execution environments.
+* **Data Plane**: Floci does not implement the data-plane API, such as interactive sessions.
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 2ab9f25014..2dbc3567ba 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -1104,6 +1104,34 @@ public interface EmulatorConfig {
/** How long a BATCH job run stays RUNNING before it succeeds; 0 = a run succeeds as soon as it starts. */
@WithDefault("0")
int jobRunDurationSeconds();
+
+ /**
+ * {@code immediate} follows the job run states without running anything; {@code docker} runs
+ * each Spark job run's {@code spark-submit} in a container of the release's EMR Serverless image.
+ */
+ @WithDefault("immediate")
+ String runnerMode();
+
+ Optional<String> dockerNetwork();
+
+ /**
+ * The image a {@code docker} job run uses, with {@code %s} standing for the application's
+ * release label, for example "registry.internal/emr-serverless/spark/%s:20250425". If omitted,
+ * Floci uses the release's public image on Amazon ECR Public.
+ */
+ Optional<String> imageTemplate();
+
+ /**
+ * When true, Floci removes, in the {@code docker} runner mode, every EMR Serverless driver a
+ * previous run of <em>this same</em> Floci left on the daemon (matched by the
+ * {@code io.floci.owner} label) before its first job run starts. Turn it off when two Floci
+ * instances share a daemon with the same port and no resource namespace, so one does not
+ * remove the other's drivers.
+ *
+ * Env var: FLOCI_SERVICES_EMRSERVERLESS_RECONCILE_CONTAINERS_ON_STARTUP
+ */
+ @WithDefault("true")
+ boolean reconcileContainersOnStartup();
}
interface Route53ResolverServiceConfig {
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java
index 82bb33fb4a..efbf66f303 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java
@@ -525,6 +525,10 @@ public class ContainerLifecycleManager {
} catch (NotFoundException e) {
LOG.debugv("Container {0} not found (already removed)", containerId);
stoppedOrMissing = true;
+ } catch (NotModifiedException e) {
+ // HTTP 304: the container had already exited, which is the outcome the stop wanted.
+ LOG.debugv("Container {0} had already stopped", containerId);
+ stoppedOrMissing = true;
} catch (Exception e) {
LOG.warnv("Error stopping container {0}: {1}", containerId, e.getMessage());
}
@@ -586,6 +590,9 @@ public class ContainerLifecycleManager {
LOG.debugv("Container {0} not found (already removed)", containerId);
closeLogStreamIfPresent(logStream);
return;
+ } catch (NotModifiedException e) {
+ // HTTP 304: the container had already exited, which is the outcome the stop wanted.
+ LOG.debugv("Container {0} had already stopped", containerId);
} catch (Exception e) {
stopFailure = e;
LOG.warnv("Error stopping container {0}: {1}", containerId, e.getMessage());
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java
index d074d2f2ba..38f599bcc4 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java
@@ -12,6 +12,8 @@ import io.github.hectorvent.floci.core.common.PaginatedResult;
import io.github.hectorvent.floci.core.common.Pagination;
import io.github.hectorvent.floci.core.common.RegionResolver;
import io.github.hectorvent.floci.core.common.RequestContext;
+import io.github.hectorvent.floci.core.common.RequestScopes;
+import io.github.hectorvent.floci.core.common.Resettable;
import io.github.hectorvent.floci.core.common.SmithyValidationErrors;
import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
import io.github.hectorvent.floci.core.storage.StorageFactory;
@@ -21,8 +23,10 @@ import io.github.hectorvent.floci.services.emrserverless.model.JobRun;
import io.github.hectorvent.floci.services.emrserverless.model.JobRunAttemptSummary;
import io.github.hectorvent.floci.services.emrserverless.model.JobRunRecord;
import io.github.hectorvent.floci.services.emrserverless.model.JobRunSummary;
+import io.github.hectorvent.floci.services.emrserverless.model.RuntimeConfiguration;
import io.github.hectorvent.floci.services.emrserverless.model.StartJobRunRequest;
import io.github.hectorvent.floci.services.iam.IamService;
+import jakarta.annotation.PreDestroy;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import org.jboss.logging.Logger;
@@ -35,35 +39,54 @@ import java.time.format.DateTimeParseException;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
- * EMR Serverless job runs. Nothing executes: a run follows the job run states and GetJobRun's
- * shape, and settles lazily when read. Times are kept to the millisecond, as AWS reports them. With the default {@code job-run-duration-seconds} of 0 a
+ * EMR Serverless job runs. In the default {@code immediate} runner mode nothing executes: a run
+ * follows the job run states and GetJobRun's shape, and settles lazily when read. Times are kept to
+ * the millisecond, as AWS reports them. With the default {@code job-run-duration-seconds} of 0 a
* BATCH run is SUCCESS as soon as it starts; a positive duration keeps it RUNNING that long, which
* makes CancelJobRun and the execution timeout observable. A STREAMING run, which EMR Serverless
* "expects to run for a long time", stays RUNNING until it is cancelled.
*
+ * <p>In the {@code docker} runner mode a Spark run executes: it is SCHEDULED while its container is
+ * prepared, RUNNING once its Spark driver runs, and ends as the driver does, SUCCESS on exit code 0
+ * and FAILED otherwise. Cancelling it, or its execution timeout, makes it CANCELLING until the driver
+ * is stopped, then CANCELLED.
+ *
* <p>Every public method is synchronized: admitting a run (look up its client token, then write
- * it), settling one and cancelling one each read and then write a run, and must not interleave.
+ * it), settling one and cancelling one each read and then write a run, and must not interleave. A
+ * container run's changes come from the thread that runs it, under the same lock.
*/
@ApplicationScoped
-public class EmrServerlessJobRunService {
+public class EmrServerlessJobRunService implements Resettable {
private static final Logger LOG = Logger.getLogger(EmrServerlessJobRunService.class);
+ static final String STATE_SCHEDULED = "SCHEDULED";
static final String STATE_RUNNING = "RUNNING";
static final String STATE_SUCCESS = "SUCCESS";
+ static final String STATE_FAILED = "FAILED";
+ static final String STATE_CANCELLING = "CANCELLING";
static final String STATE_CANCELLED = "CANCELLED";
private static final String MODE_BATCH = "BATCH";
private static final String MODE_STREAMING = "STREAMING";
+ private static final String RUNNER_DOCKER = "docker";
+ private static final String SPARK_DEFAULTS = "spark-defaults";
+ // No source gives AWS's wording for these; they are Floci's.
+ static final String HIVE_NOT_RUN = "Floci runs Spark job runs only, and this is a Hive job run.";
+ static final String INTERRUPTED = "Floci stopped while the job run was in progress.";
- private static final Set<String> TERMINAL_STATES = Set.of(STATE_SUCCESS, "FAILED", STATE_CANCELLED);
+ private static final Set<String> TERMINAL_STATES = Set.of(STATE_SUCCESS, STATE_FAILED, STATE_CANCELLED);
private static final List<String> STATES = List.of("SUBMITTED", "PENDING", "SCHEDULED", STATE_RUNNING,
STATE_SUCCESS, "FAILED", "CANCELLING", STATE_CANCELLED, "QUEUED");
// JobRunStateSet holds at most 8 states.
@@ -77,7 +100,10 @@ public class EmrServerlessJobRunService {
// onwards", and the timeout "can no longer be set to 0".
private static final long MAX_BATCH_EXECUTION_TIMEOUT_MINUTES = 10_080;
private static final Pattern RELEASE_LABEL = Pattern.compile("emr-(\\d{1,6})\\.(\\d{1,6})(?:\\..*)?");
- // Nothing runs, so no run fails and none is retried: every run is its own one attempt.
+ // The AWS runtime for Apache Spark, emr-spark-8.0.0 on (and emr-spark-8.0-preview).
+ private static final Pattern SPARK_RUNTIME_RELEASE_LABEL =
+ Pattern.compile("emr-spark-(\\d{1,6})\\.(\\d{1,6})(?:[.-].*)?");
+ // No run is retried: every run is its own one attempt.
private static final int ATTEMPT = 1;
private static final int MAX_LIST_RESULTS = 50;
// IAMRoleArn's pattern, as the model and AWS's validation message spell it.
@@ -93,20 +119,41 @@ public class EmrServerlessJobRunService {
private final ObjectMapper objectMapper;
private final int runDurationSeconds;
private final Clock clock;
+ // Null in the immediate runner mode.
+ private final SparkJobRunner sparkRunner;
+ private final Executor runExecutor;
+ // Container runs a thread is running; one not here that has not ended was left by a restart.
+ private final Set<String> executing = new HashSet<>();
+ // Raised by each reset, so a run's thread from before it writes nothing back after the wipe.
+ private long generation;
@Inject
public EmrServerlessJobRunService(StorageFactory storageFactory, EmrServerlessService applications,
IamService iamService, RegionResolver regionResolver,
RequestContext requestContext, ObjectMapper objectMapper,
- EmulatorConfig config, Clock clock) {
+ EmulatorConfig config, Clock clock, EmrServerlessSparkRunner sparkRunner) {
this(storageFactory.create("emrserverless", "emr-serverless-job-runs.json", new TypeReference<>() {}),
applications, iamService, regionResolver, requestContext, objectMapper,
- config.services().emrserverless().jobRunDurationSeconds(), clock);
+ config.services().emrserverless().jobRunDurationSeconds(), clock,
+ RUNNER_DOCKER.equalsIgnoreCase(config.services().emrserverless().runnerMode()) ? sparkRunner : null,
+ Executors.newCachedThreadPool(Thread.ofPlatform().daemon().name("emr-serverless-run-", 0).factory()));
+ // Before any run of this process starts: a driver an earlier one left belongs to a run read as FAILED.
+ if (this.sparkRunner != null && config.services().emrserverless().reconcileContainersOnStartup()) {
+ sparkRunner.removeLeftoverDrivers();
+ }
}
EmrServerlessJobRunService(AccountAwareStorageBackend<JobRunRecord> store, EmrServerlessService applications,
IamService iamService, RegionResolver regionResolver, RequestContext requestContext,
ObjectMapper objectMapper, int runDurationSeconds, Clock clock) {
+ this(store, applications, iamService, regionResolver, requestContext, objectMapper, runDurationSeconds, clock,
+ null, Runnable::run);
+ }
+
+ EmrServerlessJobRunService(AccountAwareStorageBackend<JobRunRecord> store, EmrServerlessService applications,
+ IamService iamService, RegionResolver regionResolver, RequestContext requestContext,
+ ObjectMapper objectMapper, int runDurationSeconds, Clock clock,
+ SparkJobRunner sparkRunner, Executor runExecutor) {
if (runDurationSeconds < 0) {
throw new IllegalArgumentException(
"floci.services.emrserverless.job-run-duration-seconds must not be negative: " + runDurationSeconds);
@@ -119,6 +166,15 @@ public class EmrServerlessJobRunService {
this.objectMapper = objectMapper;
this.runDurationSeconds = runDurationSeconds;
this.clock = clock;
+ this.sparkRunner = sparkRunner;
+ this.runExecutor = runExecutor;
+ }
+
+ @PreDestroy
+ void shutdown() {
+ if (runExecutor instanceof ExecutorService executor) {
+ executor.shutdownNow();
+ }
}
/**
@@ -142,8 +198,13 @@ public class EmrServerlessJobRunService {
if (body.getTags() != null) {
EmrServerlessService.validateTags(body.getTags());
}
- JobRunRecord record = applications.admitJobRun(applicationId, app -> checkBatchTimeout(app, body),
- app -> newRun(app, body, original));
+ JobRunRecord record = applications.admitJobRun(applicationId, app -> checkBatchTimeout(app, body), app -> {
+ JobRunRecord admitted = newRun(app, body, original);
+ if (sparkRunner != null) {
+ launch(admitted, app);
+ }
+ return admitted;
+ });
LOG.infov("Started EMR Serverless job run {0} on application {1}", record.getJobRun().getJobRunId(),
applicationId);
return settle(record).getJobRun();
@@ -167,7 +228,8 @@ public class EmrServerlessJobRunService {
throw new AwsException("ResourceNotFoundException",
"Job run " + jobRunId + " has no attempt " + attempt, 404);
}
- return settle(record).getJobRun();
+ // A copy: a container run's thread may change the stored run while the response is written.
+ return objectMapper.convertValue(settle(record).getJobRun(), JobRun.class);
}
/** A run's attempts, which are its one attempt. */
@@ -246,13 +308,25 @@ public class EmrServerlessJobRunService {
page.nextToken());
}
- /** Cancels a run that has not finished. A finished run is refused, with AWS's wording. */
+ /**
+ * Cancels a run that has not finished. A finished run is refused, with AWS's wording. A container
+ * run is CANCELLING until its driver has stopped.
+ */
public synchronized JobRun cancelJobRun(String applicationId, String jobRunId) {
applications.getApplication(applicationId);
JobRunRecord record = settle(findRun(applicationId, jobRunId));
if (isTerminal(record.getJobRun())) {
throw new AwsException("ValidationException", "Job run is not in a cancellable state", 400);
}
+ if (record.isContainerRun()) {
+ JobRun run = record.getJobRun();
+ if (!STATE_CANCELLING.equals(run.getState())) {
+ markCancelling(record);
+ runExecutor.execute(() -> sparkRunner.stop(jobRunId));
+ LOG.infov("Cancelling EMR Serverless job run {0} on application {1}", jobRunId, applicationId);
+ }
+ return run;
+ }
finish(record, STATE_CANCELLED, clock.instant().truncatedTo(ChronoUnit.MILLIS));
LOG.infov("Cancelled EMR Serverless job run {0} on application {1}", jobRunId, applicationId);
return record.getJobRun();
@@ -322,7 +396,7 @@ public class EmrServerlessJobRunService {
run.setUpdatedAt(now);
run.setExecutionRole(request.getExecutionRoleArn());
run.setExecutionIamPolicy(request.getExecutionIamPolicy());
- run.setState(STATE_RUNNING);
+ run.setState(sparkRunner != null ? STATE_SCHEDULED : STATE_RUNNING);
run.setStateDetails("");
run.setReleaseLabel(app.getReleaseLabel());
run.setConfigurationOverrides(request.getConfigurationOverrides());
@@ -336,17 +410,177 @@ public class EmrServerlessJobRunService {
run.setAttempt(ATTEMPT);
run.setAttemptCreatedAt(now);
run.setAttemptUpdatedAt(now);
- run.setStartedAt(now);
+ if (sparkRunner == null) {
+ run.setStartedAt(now);
+ }
JobRunRecord record = new JobRunRecord();
record.setJobRun(run);
record.setClientToken(request.getClientToken());
record.setRequest(original);
record.setApplicationType(app.getType());
+ record.setContainerRun(sparkRunner != null);
store.put(jobRunId, record);
return record;
}
+ /** Hands a container run to a thread of its own, which runs it to its end. */
+ private void launch(JobRunRecord record, Application app) {
+ JobRun run = record.getJobRun();
+ if (run.getJobDriver().getSparkSubmit() == null) {
+ run.setStateDetails(HIVE_NOT_RUN);
+ finish(record, STATE_FAILED, run.getCreatedAt());
+ return;
+ }
+ AwsArnUtils.Arn arn = AwsArnUtils.parse(run.getArn());
+ // A STREAMING run "expects to run for a long time", so only a BATCH run's timeout ends it.
+ Long minutes = run.getExecutionTimeoutMinutes();
+ Duration timeout = MODE_STREAMING.equals(run.getMode()) || minutes == null || minutes == 0
+ ? Duration.ZERO : Duration.ofMinutes(minutes);
+ SparkJobRunner.Job job = new SparkJobRunner.Job(run.getJobRunId(), run.getApplicationId(), arn.accountId(),
+ arn.region(), run.getReleaseLabel(), run.getExecutionRole(), run.getJobDriver().getSparkSubmit(),
+ sparkDefaults(app, run), timeout);
+ executing.add(run.getJobRunId());
+ long launchedIn = generation;
+ runExecutor.execute(() -> execute(job, launchedIn));
+ }
+
+ private void execute(SparkJobRunner.Job job, long launchedIn) {
+ SparkJobRunner.Outcome outcome;
+ try {
+ outcome = sparkRunner.run(job, new RunProgress(job, launchedIn));
+ } catch (RuntimeException e) {
+ LOG.warnv(e, "EMR Serverless job run {0} could not run", job.jobRunId());
+ outcome = SparkJobRunner.Outcome.couldNotStart(e.getMessage());
+ }
+ SparkJobRunner.Outcome ended = outcome;
+ RequestScopes.runAs(job.accountId(), job.region(), () -> ended(job.jobRunId(), ended, launchedIn));
+ }
+
+ /**
+ * A reset has begun: the runs' threads from before it write nothing more, so the storage wipe that
+ * follows leaves none of their runs behind. Its teardown has already stopped their drivers.
+ */
+ @Override
+ public synchronized void beforeReset() {
+ generation++;
+ executing.clear();
+ }
+
+ @Override
+ public void clear() {
+ // The runs live in storage, which the reset has already wiped.
+ }
+
+ // Whether a run's driver may still start: a cancelled run is past it, and so is one a reset removed.
+ private synchronized boolean waitingToStart(String jobRunId, long launchedIn) {
+ return launchedIn == generation && store.get(jobRunId)
+ .map(record -> STATE_SCHEDULED.equals(record.getJobRun().getState())).orElse(false);
+ }
+
+ private synchronized void driverStarted(String jobRunId, long launchedIn) {
+ if (launchedIn != generation) {
+ return;
+ }
+ store.get(jobRunId).filter(record -> STATE_SCHEDULED.equals(record.getJobRun().getState()))
+ .ifPresent(record -> {
+ JobRun run = record.getJobRun();
+ Instant now = clock.instant().truncatedTo(ChronoUnit.MILLIS);
+ run.setState(STATE_RUNNING);
+ run.setStartedAt(now);
+ run.setUpdatedAt(now);
+ run.setAttemptUpdatedAt(now);
+ store.put(jobRunId, record);
+ LOG.infov("EMR Serverless job run {0} is running", jobRunId);
+ });
+ }
+
+ // A run past its timeout is CANCELLING while its driver is stopped: "the job run has timed out".
+ private synchronized void driverTimedOut(String jobRunId, long launchedIn) {
+ if (launchedIn != generation) {
+ return;
+ }
+ store.get(jobRunId).filter(record -> STATE_RUNNING.equals(record.getJobRun().getState()))
+ .ifPresent(this::markCancelling);
+ }
+
+ private void markCancelling(JobRunRecord record) {
+ JobRun run = record.getJobRun();
+ Instant now = clock.instant().truncatedTo(ChronoUnit.MILLIS);
+ run.setState(STATE_CANCELLING);
+ run.setUpdatedAt(now);
+ run.setAttemptUpdatedAt(now);
+ store.put(run.getJobRunId(), record);
+ }
+
+ /**
+ * Ends a container run as its driver did. A cancelled run, or one past its timeout ("the job run
+ * has timed out" is a reason to be CANCELLING), is CANCELLED; a driver stopped otherwise was
+ * stopped by Floci's own shutdown.
+ */
+ private synchronized void ended(String jobRunId, SparkJobRunner.Outcome outcome, long launchedIn) {
+ if (launchedIn != generation) {
+ return;
+ }
+ executing.remove(jobRunId);
+ JobRunRecord record = store.get(jobRunId).orElse(null);
+ if (record == null || isTerminal(record.getJobRun())) {
+ return;
+ }
+ JobRun run = record.getJobRun();
+ Instant now = clock.instant().truncatedTo(ChronoUnit.MILLIS);
+ if (STATE_CANCELLING.equals(run.getState()) || outcome.timedOut()) {
+ finish(record, STATE_CANCELLED, now);
+ } else if (outcome.exitCode() != null && outcome.exitCode() == 0) {
+ finish(record, STATE_SUCCESS, now);
+ } else {
+ // No source gives AWS's wording for a driver that fails; this is Floci's.
+ run.setStateDetails(outcome.stopped() ? INTERRUPTED
+ : outcome.failure() != null ? "Job run could not start: " + outcome.failure()
+ : "Job run failed: the Spark driver exited with code " + outcome.exitCode() + ".");
+ finish(record, STATE_FAILED, now);
+ }
+ LOG.infov("EMR Serverless job run {0} ended {1}", jobRunId, run.getState());
+ }
+
+ /**
+ * The {@code spark-defaults} properties for a run: the application's runtime configuration, then the
+ * run's own {@code applicationConfiguration}. The two merge by classification, so a run's
+ * {@code spark-defaults} replaces the application's: "Only the properties included in StartJobRun is
+ * considered for this classification." From emr-spark-8.1.0 they merge by property, the run's winning.
+ */
+ private static Map<String, String> sparkDefaults(Application app, JobRun run) {
+ Map<String, String> properties = new LinkedHashMap<>();
+ if (app.getRuntimeConfiguration() != null) {
+ for (RuntimeConfiguration configuration : app.getRuntimeConfiguration()) {
+ addSparkDefaults(properties, configuration.getMembers());
+ }
+ }
+ Object overrides = run.getConfigurationOverrides() != null
+ ? run.getConfigurationOverrides().get("applicationConfiguration") : null;
+ if (overrides instanceof List<?> configurations) {
+ boolean replaces = !releaseAtLeast(SPARK_RUNTIME_RELEASE_LABEL, run.getReleaseLabel(), 8, 1);
+ for (Object configuration : configurations) {
+ if (configuration instanceof Map<?, ?> members
+ && SPARK_DEFAULTS.equals(members.get("classification"))) {
+ if (replaces) {
+ properties.clear();
+ replaces = false;
+ }
+ addSparkDefaults(properties, members);
+ }
+ }
+ }
+ return properties;
+ }
+
+ private static void addSparkDefaults(Map<String, String> properties, Map<?, ?> configuration) {
+ if (SPARK_DEFAULTS.equals(configuration.get("classification"))
+ && configuration.get("properties") instanceof Map<?, ?> values) {
+ values.forEach((key, value) -> properties.put(String.valueOf(key), String.valueOf(value)));
+ }
+ }
+
/**
* Finishes a BATCH run that has been RUNNING for the configured duration, or cancels it at its
* execution timeout when that comes first: "If the job run exceeds this duration, Amazon EMR
@@ -354,6 +588,14 @@ public class EmrServerlessJobRunService {
*/
private JobRunRecord settle(JobRunRecord record) {
JobRun run = record.getJobRun();
+ if (record.isContainerRun()) {
+ // Its thread ends it; without one, Floci stopped while it ran and nothing is left to.
+ if (!isTerminal(run) && !executing.contains(run.getJobRunId())) {
+ run.setStateDetails(INTERRUPTED);
+ finish(record, STATE_FAILED, clock.instant().truncatedTo(ChronoUnit.MILLIS));
+ }
+ return record;
+ }
if (isTerminal(run) || MODE_STREAMING.equals(run.getMode())) {
return record;
}
@@ -379,7 +621,9 @@ public class EmrServerlessJobRunService {
run.setEndedAt(endedAt);
run.setUpdatedAt(endedAt);
run.setAttemptUpdatedAt(endedAt);
- run.setTotalExecutionDurationSeconds((int) Duration.between(run.getStartedAt(), endedAt).toSeconds());
+ // A container run cancelled or failed before its driver ran never started.
+ run.setTotalExecutionDurationSeconds(run.getStartedAt() != null
+ ? (int) Duration.between(run.getStartedAt(), endedAt).toSeconds() : 0);
store.put(run.getJobRunId(), record);
}
@@ -500,12 +744,17 @@ public class EmrServerlessJobRunService {
}
private static boolean boundsBatchTimeouts(String releaseLabel) {
- Matcher matcher = RELEASE_LABEL.matcher(releaseLabel != null ? releaseLabel : "");
+ return releaseAtLeast(RELEASE_LABEL, releaseLabel, 7, 11);
+ }
+
+ // Whether releaseLabel is of the family labelPattern matches, at major.minor or later.
+ private static boolean releaseAtLeast(Pattern labelPattern, String releaseLabel, int major, int minor) {
+ Matcher matcher = labelPattern.matcher(releaseLabel != null ? releaseLabel : "");
if (!matcher.matches()) {
return false;
}
- int major = Integer.parseInt(matcher.group(1));
- return major > 7 || (major == 7 && Integer.parseInt(matcher.group(2)) >= 11);
+ int labelMajor = Integer.parseInt(matcher.group(1));
+ return labelMajor > major || (labelMajor == major && Integer.parseInt(matcher.group(2)) >= minor);
}
private static JobRunSummary toSummary(JobRunRecord record) {
@@ -550,4 +799,32 @@ public class EmrServerlessJobRunService {
summary.setAttempt(ATTEMPT);
return summary;
}
+
+ /** What the thread running a container run tells the service, and asks it, as that run. */
+ private final class RunProgress implements SparkJobRunner.Progress {
+
+ private final SparkJobRunner.Job job;
+ private final long launchedIn;
+
+ RunProgress(SparkJobRunner.Job job, long launchedIn) {
+ this.job = job;
+ this.launchedIn = launchedIn;
+ }
+
+ @Override
+ public void started() {
+ RequestScopes.runAs(job.accountId(), job.region(), () -> driverStarted(job.jobRunId(), launchedIn));
+ }
+
+ @Override
+ public void timedOut() {
+ RequestScopes.runAs(job.accountId(), job.region(), () -> driverTimedOut(job.jobRunId(), launchedIn));
+ }
+
+ @Override
+ public boolean cancelled() {
+ return !RequestScopes.callAs(job.accountId(), job.region(),
+ () -> waitingToStart(job.jobRunId(), launchedIn));
+ }
+ }
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java
new file mode 100644
index 0000000000..eee225b28d
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java
@@ -0,0 +1,352 @@
+package io.github.hectorvent.floci.services.emrserverless;
+
+import com.github.dockerjava.api.command.InspectContainerResponse;
+import com.github.dockerjava.api.exception.NotFoundException;
+import com.github.dockerjava.api.exception.NotModifiedException;
+import com.github.dockerjava.api.model.Container;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.ContainerTeardown;
+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;
+import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;
+import io.github.hectorvent.floci.services.emrserverless.model.JobDriver;
+import io.github.hectorvent.floci.services.iam.model.SessionCreds;
+import io.github.hectorvent.floci.services.lambda.launcher.LambdaExecutionRoleCredentials;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+import org.jboss.logging.Logger;
+
+import java.io.Closeable;
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Runs a Spark job run's {@code spark-submit} in a container of the release's EMR Serverless image.
+ * The image's own entrypoint hands over to a script EMR Serverless supplies at run time, which the
+ * public image lacks, so the runner starts {@code spark-submit} itself, in local mode: the container
+ * holds the driver and its executors. The container reaches Floci as the other containers Floci
+ * launches do, with the run's execution role as its credentials, and S3A, which takes {@code s3://}
+ * on every release, is pointed at Floci's S3.
+ */
+@ApplicationScoped
+public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardown {
+
+ private static final Logger LOG = Logger.getLogger(EmrServerlessSparkRunner.class);
+ private static final String SPARK_SUBMIT = "/usr/lib/spark/bin/spark-submit";
+ private static final String DEFAULT_IMAGE =
+ "public.ecr.aws/emr-serverless/spark/%s:latest"; // partition-literal: ECR Public's registry host, where AWS publishes the base images; image-template overrides it
+ // The AWS runtime for Apache Spark (emr-spark-8.0.0 on) publishes its images under spark4/ instead.
+ private static final String SPARK_RUNTIME_DEFAULT_IMAGE =
+ "public.ecr.aws/emr-serverless/spark4/%s:latest"; // partition-literal: ECR Public's registry host, where AWS publishes the base images; image-template overrides it
+ // "The default behavior for shutdown is to have no grace period": immediate termination.
+ private static final int STOP_TIMEOUT_SECONDS = 0;
+ private static final String SERVICE = "emr-serverless";
+ // The characters a backslash escapes inside double quotes; before any other it is kept.
+ private static final String DOUBLE_QUOTE_ESCAPES = "\"\\$`\n";
+ // How often a running driver is looked at, as Batch's runner looks at its jobs.
+ private static final Duration POLL_INTERVAL = Duration.ofMillis(250);
+
+ // The runs inside run(); a stop for a run not here comes after its driver has gone, and is dropped.
+ private final ConcurrentHashMap<String, Launch> launches = new ConcurrentHashMap<>();
+
+ private final ContainerBuilder containerBuilder;
+ private final ContainerLifecycleManager lifecycleManager;
+ private final ContainerLogStreamer logStreamer;
+ private final LaunchedContainerAwsEnv awsEnv;
+ private final LambdaExecutionRoleCredentials roleCredentials;
+ private final EmulatorConfig config;
+ private final Clock clock;
+ private final Duration pollInterval;
+
+ @Inject
+ public EmrServerlessSparkRunner(ContainerBuilder containerBuilder, ContainerLifecycleManager lifecycleManager,
+ ContainerLogStreamer logStreamer, LaunchedContainerAwsEnv awsEnv,
+ LambdaExecutionRoleCredentials roleCredentials, EmulatorConfig config,
+ Clock clock) {
+ this(containerBuilder, lifecycleManager, logStreamer, awsEnv, roleCredentials, config, clock, POLL_INTERVAL);
+ }
+
+ EmrServerlessSparkRunner(ContainerBuilder containerBuilder, ContainerLifecycleManager lifecycleManager,
+ ContainerLogStreamer logStreamer, LaunchedContainerAwsEnv awsEnv,
+ LambdaExecutionRoleCredentials roleCredentials, EmulatorConfig config, Clock clock,
+ Duration pollInterval) {
+ this.containerBuilder = containerBuilder;
+ this.lifecycleManager = lifecycleManager;
+ this.logStreamer = logStreamer;
+ this.awsEnv = awsEnv;
+ this.roleCredentials = roleCredentials;
+ this.config = config;
+ this.clock = clock;
+ this.pollInterval = pollInterval;
+ }
+
+ @Override
+ public Outcome run(Job job, Progress progress) {
+ Launch launch = new Launch();
+ launches.put(job.jobRunId(), launch);
+ Optional<SessionCreds> credentials = Optional.empty();
+ Closeable logHandle = null;
+ try {
+ // A cancel before the run was in launches had nothing to stop, so it is read here.
+ if (progress.cancelled()) {
+ return Outcome.halted();
+ }
+ credentials = roleCredentials.forRole(job.executionRole(), job.accountId(), job.jobRunId());
+ Map<String, String> labels = new LinkedHashMap<>(ContainerStorageHelper.resourceIdentityLabels(
+ SERVICE, job.jobRunId(), job.accountId(), job.region()));
+ // The deployment that started the driver, which removeLeftoverDrivers() goes by.
+ labels.put(ContainerStorageHelper.OWNER_LABEL, ContainerStorageHelper.ownerIdentity(config));
+ String endpoint = awsEnv.flociEndpoint();
+ ContainerSpec spec = containerBuilder.newContainer(image(job.releaseLabel()))
+ .withName(ContainerStorageHelper.dockerName(config, SERVICE + "-" + job.jobRunId()))
+ .withEntrypoint(List.of(SPARK_SUBMIT))
+ .withCmd(sparkSubmitArguments(job, endpoint))
+ .withEnv(awsEnv.sdkBaselineEnv(job.region(), Optional.empty(), credentials, job.accountId()))
+ .withDockerNetwork(config.services().emrserverless().dockerNetwork())
+ .withHostDockerInternalOnLinux()
+ .withEmbeddedDns()
+ .withLogRotation()
+ .withLabels(labels)
+ .build();
+ String containerId = lifecycleManager.createAndStart(spec).containerId();
+ launch.container.set(containerId);
+ // A stop, a reset or a shutdown while the image was pulled found no container to stop.
+ if (launch.stopRequested || progress.cancelled()) {
+ return Outcome.halted();
+ }
+ logHandle = logStreamer.attachConsoleOnly(containerId, SERVICE + ":" + job.jobRunId());
+ progress.started();
+ Integer exitCode = awaitExit(containerId, job, progress);
+ if (launch.stopRequested) {
+ return Outcome.halted();
+ }
+ return exitCode == null ? Outcome.outOfTime() : Outcome.exitedWith(exitCode);
+ } catch (InterruptedException e) {
+ // Floci is shutting down: its teardown stops the driver.
+ Thread.currentThread().interrupt();
+ return Outcome.halted();
+ } catch (RuntimeException e) {
+ if (launch.stopRequested) {
+ // A reset removed the driver while it was being watched.
+ return Outcome.halted();
+ }
+ LOG.warnv(e, "EMR Serverless job run {0} could not run", job.jobRunId());
+ return Outcome.couldNotStart(e.getMessage());
+ } finally {
+ launches.remove(job.jobRunId(), launch);
+ String unclaimed = launch.container.getAndSet(null);
+ if (unclaimed != null) {
+ lifecycleManager.stopAndRemove(unclaimed, logHandle, STOP_TIMEOUT_SECONDS);
+ } else if (logHandle != null) {
+ // Floci's teardown claimed the driver, but the log stream is still this run's to close.
+ lifecycleManager.closeLogStreamAfterContainerStop(logHandle);
+ }
+ credentials.ifPresent(session -> roleCredentials.unregister(job.accountId(), session.accessKeyId()));
+ }
+ }
+
+ @Override
+ public void stop(String jobRunId) {
+ Launch launch = launches.get(jobRunId);
+ if (launch == null) {
+ return;
+ }
+ launch.stopRequested = true;
+ String containerId = launch.container.get();
+ if (containerId == null) {
+ // Still starting: run() reads the request once its container is up.
+ return;
+ }
+ // run() sees it stopped at its next look, and removes it.
+ try {
+ lifecycleManager.getDockerClient().stopContainerCmd(containerId).withTimeout(STOP_TIMEOUT_SECONDS).exec();
+ } catch (NotFoundException | NotModifiedException alreadyEnded) {
+ // Gone already (404), or exited and not yet removed (304).
+ LOG.debugv("EMR Serverless job run {0} had ended before it was stopped", jobRunId);
+ }
+ }
+
+ /**
+ * Stops every run inside {@link #run}, so a reset or a shutdown orphans none: a running driver is
+ * stopped and removed now, and one whose image is still being pulled as soon as its container is up.
+ */
+ @Override
+ public void stopManagedContainers() {
+ for (Launch launch : launches.values()) {
+ launch.stopRequested = true;
+ String containerId = launch.container.getAndSet(null);
+ if (containerId != null) {
+ lifecycleManager.stopAndRemove(containerId, null, STOP_TIMEOUT_SECONDS);
+ }
+ }
+ }
+
+ /**
+ * Removes the drivers an earlier Floci process left running, which no thread follows any more: their
+ * runs are {@code FAILED} once read, so nothing would ever stop them. The job run service calls this
+ * before it hands a run to {@link #run}, so every driver of this deployment it finds is a leftover.
+ */
+ public void removeLeftoverDrivers() {
+ String owner = ContainerStorageHelper.ownerIdentity(config);
+ ContainerStorageHelper.LabelAliases aliases = ContainerStorageHelper.CONTAINER_LABEL_ALIASES;
+ try {
+ List<Container> containers = aliases.listByLabels(
+ Map.of(ContainerStorageHelper.SERVICE_LABEL, SERVICE, ContainerStorageHelper.OWNER_LABEL, owner),
+ filter -> lifecycleManager.getDockerClient().listContainersCmd()
+ .withShowAll(true)
+ .withLabelFilter(filter)
+ .exec(),
+ Container::getId);
+ for (Container container : containers) {
+ Map<String, String> labels = container.getLabels() != null ? container.getLabels() : Map.of();
+ if (aliases.consistent(container.getId(), labels)
+ && aliases.matches(labels, ContainerStorageHelper.OWNER_LABEL, owner)) {
+ lifecycleManager.removeIfExists(container.getId());
+ LOG.infov("Removed EMR Serverless driver {0}, left running by an earlier Floci process",
+ container.getId());
+ }
+ }
+ } catch (RuntimeException e) {
+ LOG.warnv("Could not remove the EMR Serverless drivers an earlier Floci process left: {0}",
+ e.getMessage());
+ }
+ }
+
+ String image(String releaseLabel) {
+ return config.services().emrserverless().imageTemplate()
+ .orElse(releaseLabel.startsWith("emr-spark-") ? SPARK_RUNTIME_DEFAULT_IMAGE : DEFAULT_IMAGE)
+ .formatted(releaseLabel);
+ }
+
+ /**
+ * {@code spark-submit}'s arguments: local mode, S3A on Floci's S3 for {@code s3://} and {@code s3a://}
+ * paths, then the run's {@code spark-defaults} and its own {@code sparkSubmitParameters}, which win
+ * over what comes before them, then the entry point and its arguments. Before release 7.10.0
+ * {@code s3://} is EMRFS, which ignores the endpoint setting, so it is switched to S3A. S3A writes
+ * are committed through commit data the writer keeps in memory, without which EMR's committer finds
+ * none and a write is silently lost.
+ */
+ static List<String> sparkSubmitArguments(Job job, String flociEndpoint) {
+ List<String> arguments = new ArrayList<>(List.of("--master", "local[*]"));
+ conf(arguments, "spark.hadoop.fs.s3a.endpoint", flociEndpoint);
+ conf(arguments, "spark.hadoop.fs.s3a.endpoint.region", job.region());
+ conf(arguments, "spark.hadoop.fs.s3a.path.style.access", "true");
+ conf(arguments, "spark.hadoop.fs.s3a.committer.magic.track.commits.in.memory.enabled", "true");
+ conf(arguments, "spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
+ conf(arguments, "spark.hadoop.fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3A");
+ conf(arguments, "spark.hadoop.aws.glue.endpoint", flociEndpoint);
+ conf(arguments, "spark.hadoop.aws.region", job.region());
+ job.sparkDefaults().forEach((key, value) -> conf(arguments, key, value));
+ JobDriver.SparkSubmit sparkSubmit = job.sparkSubmit();
+ arguments.addAll(splitParameters(sparkSubmit.getSparkSubmitParameters()));
+ arguments.add(sparkSubmit.getEntryPoint());
+ if (sparkSubmit.getEntryPointArguments() != null) {
+ arguments.addAll(sparkSubmit.getEntryPointArguments());
+ }
+ return arguments;
+ }
+
+ /**
+ * {@code sparkSubmitParameters} as a POSIX shell would split them: on whitespace, except inside single
+ * or double quotes. Outside quotes a backslash takes the next character as it is; inside double quotes
+ * only a {@code "}, {@code \}, {@code $}, {@code `} or newline, and before any other character it is
+ * kept. A backslash before a newline continues the line, and both go.
+ */
+ static List<String> splitParameters(String parameters) {
+ List<String> words = new ArrayList<>();
+ if (parameters == null) {
+ return words;
+ }
+ StringBuilder word = new StringBuilder();
+ boolean inWord = false;
+ char quote = 0;
+ for (int i = 0; i < parameters.length(); i++) {
+ char c = parameters.charAt(i);
+ if (quote == '\'') {
+ if (c == '\'') {
+ quote = 0;
+ } else {
+ word.append(c);
+ }
+ } else if (c == '\\' && i + 1 < parameters.length()
+ && (quote == 0 || DOUBLE_QUOTE_ESCAPES.indexOf(parameters.charAt(i + 1)) >= 0)) {
+ char escaped = parameters.charAt(++i);
+ if (escaped != '\n') {
+ word.append(escaped);
+ inWord = true;
+ }
+ } else if (quote == '"') {
+ if (c == '"') {
+ quote = 0;
+ } else {
+ word.append(c);
+ }
+ } else if (c == '\'' || c == '"') {
+ quote = c;
+ inWord = true;
+ } else if (Character.isWhitespace(c)) {
+ if (inWord) {
+ words.add(word.toString());
+ word.setLength(0);
+ inWord = false;
+ }
+ } else {
+ word.append(c);
+ inWord = true;
+ }
+ }
+ if (inWord) {
+ words.add(word.toString());
+ }
+ return words;
+ }
+
+ private static void conf(List<String> arguments, String key, String value) {
+ arguments.add("--conf");
+ arguments.add(key + "=" + value);
+ }
+
+ /**
+ * The driver's exit code, or null once the run's timeout has passed with the driver still running,
+ * which {@code progress} hears before the driver is stopped. Each look at the container is one short
+ * control-plane call, so a run holds no Docker connection while its driver runs.
+ */
+ private Integer awaitExit(String containerId, Job job, Progress progress) throws InterruptedException {
+ Instant deadline = job.timeout().isZero() ? null : clock.instant().plus(job.timeout());
+ while (true) {
+ InspectContainerResponse.ContainerState state =
+ lifecycleManager.getDockerClient().inspectContainerCmd(containerId).exec().getState();
+ if (!Boolean.TRUE.equals(state.getRunning())) {
+ Long exitCode = state.getExitCodeLong();
+ return exitCode != null ? exitCode.intValue() : -1;
+ }
+ if (deadline != null && !clock.instant().isBefore(deadline)) {
+ LOG.infov("EMR Serverless job run {0} passed its execution timeout of {1}", job.jobRunId(),
+ job.timeout());
+ progress.timedOut();
+ return null;
+ }
+ Thread.sleep(pollInterval.toMillis());
+ }
+ }
+
+ /**
+ * A run inside {@link #run}: whether it is to stop, and its driver's container once it has one, which
+ * whoever stops the run first claims, run() itself or Floci's teardown.
+ */
+ private static final class Launch {
+ final AtomicReference<String> container = new AtomicReference<>();
+ volatile boolean stopRequested;
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java
new file mode 100644
index 0000000000..45e2543afd
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java
@@ -0,0 +1,73 @@
+package io.github.hectorvent.floci.services.emrserverless;
+
+import io.github.hectorvent.floci.services.emrserverless.model.JobDriver;
+
+import java.time.Duration;
+import java.util.Map;
+
+/**
+ * Runs a Spark job run to its end, in the {@code docker} runner mode. The job run service holds the
+ * run's state and calls this off the request thread; {@link EmrServerlessSparkRunner} is the Docker
+ * implementation, and a test supplies its own.
+ */
+interface SparkJobRunner {
+
+ /**
+ * Runs {@code job} and returns how it ended, telling {@code progress} once its Spark driver runs and
+ * when it passes its timeout. Blocks until then. A run {@code progress} reports as cancelled before
+ * its driver runs never starts it.
+ */
+ Outcome run(Job job, Progress progress);
+
+ /**
+ * Stops the run's driver, now or as soon as it starts, so {@link #run} returns {@link Outcome#halted};
+ * a run {@link #run} has already returned from is left as it is.
+ */
+ void stop(String jobRunId);
+
+ /** What a run tells the job run service, and asks it, while it goes. */
+ interface Progress {
+
+ /** The run's Spark driver runs. */
+ void started();
+
+ /** The run has passed its timeout, and its driver is about to be stopped. */
+ void timedOut();
+
+ /** Whether the run was cancelled, or is gone, so that its driver must not start. */
+ boolean cancelled();
+ }
+
+ /**
+ * What a run needs: its identity, the application's release, the run's execution role, its Spark
+ * submission, the {@code spark-defaults} properties that apply to it, and how long it may run
+ * ({@link Duration#ZERO} for no limit).
+ */
+ record Job(String jobRunId, String applicationId, String accountId, String region, String releaseLabel,
+ String executionRole, JobDriver.SparkSubmit sparkSubmit, Map<String, String> sparkDefaults,
+ Duration timeout) {
+ }
+
+ /**
+ * How a run ended: its driver's exit code, or that it ran out of time, was stopped, or could not
+ * start ({@code failure} says why).
+ */
+ record Outcome(Integer exitCode, boolean timedOut, boolean stopped, String failure) {
+
+ static Outcome exitedWith(int exitCode) {
+ return new Outcome(exitCode, false, false, null);
+ }
+
+ static Outcome outOfTime() {
+ return new Outcome(null, true, false, null);
+ }
+
+ static Outcome halted() {
+ return new Outcome(null, false, true, null);
+ }
+
+ static Outcome couldNotStart(String failure) {
+ return new Outcome(null, false, false, failure);
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/model/JobRunRecord.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/model/JobRunRecord.java
index e55acce436..9f6ecda61d 100644
--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/model/JobRunRecord.java
+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/model/JobRunRecord.java
@@ -6,7 +6,8 @@ import io.quarkus.runtime.annotations.RegisterForReflection;
/**
* A stored job run: the run as GetJobRun reports it, plus what only Floci needs. The client token
* and the request it came with decide whether a repeated StartJobRun is a retry or a conflict; the
- * application type fills ListJobRuns' {@code type}.
+ * application type fills ListJobRuns' {@code type}. A container run is one the {@code docker} runner
+ * mode runs, whose state its container decides.
*/
@RegisterForReflection
public class JobRunRecord {
@@ -14,6 +15,7 @@ public class JobRunRecord {
private String clientToken;
private JsonNode request;
private String applicationType;
+ private boolean containerRun;
public JobRun getJobRun() { return jobRun; }
public void setJobRun(JobRun jobRun) { this.jobRun = jobRun; }
@@ -23,4 +25,6 @@ public class JobRunRecord {
public void setRequest(JsonNode request) { this.request = request; }
public String getApplicationType() { return applicationType; }
public void setApplicationType(String applicationType) { this.applicationType = applicationType; }
+ public boolean isContainerRun() { return containerRun; }
+ public void setContainerRun(boolean containerRun) { this.containerRun = containerRun; }
}
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index ddc18f666a..d95b3fe489 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -972,6 +972,12 @@ floci:
emrserverless:
enabled: true
job-run-duration-seconds: 0 # 0 = BATCH job runs succeed as soon as they start
+ runner-mode: immediate # docker = run each Spark job run's spark-submit in a container
+ # docker-network: floci-network
+ # image-template: "registry.internal/emr-serverless/spark/%s:20250425" # %s = the release label
+ # Remove the drivers a previous run of this deployment left on the daemon before the first
+ # job run starts. Scoped to this deployment's own containers.
+ reconcile-containers-on-startup: true
ssoadmin:
enabled: true
ssooidc:
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerTeardownTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerTeardownTest.java
index 40ba5c9363..e34a245665 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerTeardownTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerTeardownTest.java
@@ -3,8 +3,10 @@ package io.github.hectorvent.floci.core.common.docker;
import com.github.dockerjava.api.DockerClient;
import com.github.dockerjava.api.command.RemoveContainerCmd;
import com.github.dockerjava.api.command.StopContainerCmd;
+import com.github.dockerjava.api.exception.NotModifiedException;
import org.junit.jupiter.api.Test;
+import java.io.Closeable;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CountDownLatch;
@@ -15,6 +17,7 @@ import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import static io.github.hectorvent.floci.testing.ThreadStates.awaitBlockedOrDone;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -142,4 +145,27 @@ class ContainerLifecycleManagerTeardownTest {
}
}
+ @Test
+ void aContainerThatHadAlreadyExitedCountsAsStopped() throws Exception {
+ DockerClient docker = mock(DockerClient.class);
+ ContainerLifecycleManager manager = new ContainerLifecycleManager(docker, null, null, null);
+ StopContainerCmd stop = mock(StopContainerCmd.class);
+ when(docker.stopContainerCmd(anyString())).thenReturn(stop);
+ when(stop.withTimeout(anyInt())).thenReturn(stop);
+ // Docker answers a stop of an exited container with HTTP 304.
+ when(stop.exec()).thenThrow(new NotModifiedException("Container already stopped"));
+ RemoveContainerCmd remove = mock(RemoveContainerCmd.class);
+ when(docker.removeContainerCmd(anyString())).thenReturn(remove);
+ when(remove.withForce(true)).thenReturn(remove);
+ when(remove.exec()).thenThrow(new IllegalStateException("daemon refused removal"));
+ Closeable logStream = mock(Closeable.class);
+
+ manager.stopAndRemove("exited", logStream, 0);
+
+ verify(logStream).close();
+ IllegalStateException failure = assertThrows(IllegalStateException.class,
+ () -> manager.stopAndRemoveStrict("exited-strict", null));
+ assertEquals(0, failure.getSuppressed().length);
+ }
+
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java
index 32a88d5f96..99ed0117bf 100644
--- a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java
@@ -6,10 +6,12 @@ import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.core.common.RegionResolver;
import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
import io.github.hectorvent.floci.services.emrserverless.model.CreateApplicationRequest;
import io.github.hectorvent.floci.services.emrserverless.model.JobDriver;
import io.github.hectorvent.floci.services.emrserverless.model.JobRun;
import io.github.hectorvent.floci.services.emrserverless.model.JobRunRecord;
+import io.github.hectorvent.floci.services.emrserverless.model.RuntimeConfiguration;
import io.github.hectorvent.floci.services.emrserverless.model.StartJobRunRequest;
import io.github.hectorvent.floci.services.iam.IamService;
import io.github.hectorvent.floci.testing.InMemoryStorageFactory;
@@ -19,15 +21,22 @@ import org.junit.jupiter.api.Test;
import java.time.Duration;
import java.time.Instant;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
import java.util.UUID;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
class EmrServerlessJobRunServiceTest {
@@ -42,6 +51,8 @@ class EmrServerlessJobRunServiceTest {
private AccountAwareStorageBackend<JobRunRecord> store;
private MutableClock clock;
private String applicationId;
+ // The docker runner mode's thread pool, run by hand: a container run's thread is the test's.
+ private final List<Runnable> tasks = new ArrayList<>();
@BeforeEach
void setUp() {
@@ -203,4 +214,273 @@ class EmrServerlessJobRunServiceTest {
() -> EmrServerlessJobRunService.parseTimestamp("yesterday", "createdAtAfter"));
assertEquals("ValidationException", invalid.getErrorCode());
}
+
+ @Test
+ void aContainerRunIsScheduledUntilItsDriverRunsThenSucceedsWhenItExitsZero() {
+ FakeRunner runner = new FakeRunner();
+ EmrServerlessJobRunService service = dockerService(runner);
+ JobRun submitted = service.startJobRun(applicationId, request(null, null));
+ String jobRunId = submitted.getJobRunId();
+ List<JobRun> whileRunning = new ArrayList<>();
+ runner.whileRunning = () -> {
+ whileRunning.add(service.getJobRun(applicationId, jobRunId, null));
+ clock.advance(Duration.ofSeconds(42));
+ };
+
+ assertEquals("SCHEDULED", service.getJobRun(applicationId, jobRunId, null).getState());
+ assertNull(service.getJobRun(applicationId, jobRunId, null).getStartedAt());
+ clock.advance(Duration.ofSeconds(3));
+ runNext();
+
+ assertEquals("RUNNING", whileRunning.get(0).getState());
+ assertEquals(3, Duration.between(submitted.getCreatedAt(), whileRunning.get(0).getStartedAt()).toSeconds());
+ JobRun finished = service.getJobRun(applicationId, jobRunId, null);
+ assertEquals("SUCCESS", finished.getState());
+ assertEquals(42, Duration.between(finished.getStartedAt(), finished.getEndedAt()).toSeconds());
+ assertEquals(42, finished.getTotalExecutionDurationSeconds());
+ }
+
+ @Test
+ void theRunnerGetsTheRunsAccountReleaseRoleAndBatchTimeout() {
+ FakeRunner runner = new FakeRunner();
+ EmrServerlessJobRunService service = dockerService(runner);
+ String batch = service.startJobRun(applicationId, request(null, 30L)).getJobRunId();
+ service.startJobRun(applicationId, request("STREAMING", 30L));
+ runNext();
+ runNext();
+
+ SparkJobRunner.Job job = runner.jobs.get(0);
+ assertEquals(batch, job.jobRunId());
+ assertEquals(applicationId, job.applicationId());
+ assertEquals(ACCOUNT_ID, job.accountId());
+ assertEquals(REGION, job.region());
+ assertEquals("emr-7.5.0", job.releaseLabel());
+ assertEquals(ROLE, job.executionRole());
+ assertEquals("s3://bucket/jobs/app.jar", job.sparkSubmit().getEntryPoint());
+ assertEquals(Duration.ofMinutes(30), job.timeout());
+ assertEquals(Duration.ZERO, runner.jobs.get(1).timeout());
+ }
+
+ @Test
+ void aRunsSparkDefaultsReplaceTheApplicationsUntilEmrSpark810MergesThem() {
+ FakeRunner runner = new FakeRunner();
+ EmrServerlessJobRunService service = dockerService(runner);
+
+ for (String release : List.of("emr-7.10.0", "emr-spark-8.0.0", "emr-spark-8.1.0")) {
+ String configured = applicationWithSparkDefaults(release);
+ StartJobRunRequest overriding = request(null, null);
+ overriding.setConfigurationOverrides(Map.of("applicationConfiguration", List.of(Map.of(
+ "classification", "spark-defaults", "properties", Map.of("spark.executor.cores", "2")))));
+ service.startJobRun(configured, request(null, null));
+ service.startJobRun(configured, overriding);
+ runNext();
+ runNext();
+ }
+
+ Map<String, String> fromApp = Map.of("spark.executor.cores", "4", "spark.app.name", "from-app");
+ List<Map<String, String>> sparkDefaults = runner.jobs.stream().map(SparkJobRunner.Job::sparkDefaults).toList();
+ assertEquals(List.of(fromApp, Map.of("spark.executor.cores", "2"), fromApp, Map.of("spark.executor.cores", "2"),
+ fromApp, Map.of("spark.executor.cores", "2", "spark.app.name", "from-app")), sparkDefaults);
+ assertEquals("emr-spark-8.1.0", runner.jobs.get(5).releaseLabel());
+ }
+
+ private String applicationWithSparkDefaults(String releaseLabel) {
+ RuntimeConfiguration defaults = new RuntimeConfiguration();
+ defaults.setMember("classification", "spark-defaults");
+ defaults.setMember("properties", Map.of("spark.executor.cores", "4", "spark.app.name", "from-app"));
+ CreateApplicationRequest create = new CreateApplicationRequest();
+ create.setName("configured-" + releaseLabel);
+ create.setReleaseLabel(releaseLabel);
+ create.setType("SPARK");
+ create.setClientToken(UUID.randomUUID().toString());
+ create.setRuntimeConfiguration(List.of(defaults));
+ return applications.createApplication(create).getApplicationId();
+ }
+
+ @Test
+ void aDriverThatExitsNonZeroOrNeverStartsFailsTheRun() {
+ FakeRunner runner = new FakeRunner();
+ EmrServerlessJobRunService service = dockerService(runner);
+ String exited = service.startJobRun(applicationId, request(null, null)).getJobRunId();
+ String unstarted = service.startJobRun(applicationId, request(null, null)).getJobRunId();
+
+ runner.outcome = SparkJobRunner.Outcome.exitedWith(1);
+ runNext();
+ runner.outcome = SparkJobRunner.Outcome.couldNotStart("no such image");
+ runNext();
+
+ JobRun failed = service.getJobRun(applicationId, exited, null);
+ assertEquals("FAILED", failed.getState());
+ assertEquals("Job run failed: the Spark driver exited with code 1.", failed.getStateDetails());
+ assertEquals("FAILED", service.getJobRun(applicationId, unstarted, null).getState());
+ assertEquals("Job run could not start: no such image",
+ service.getJobRun(applicationId, unstarted, null).getStateDetails());
+ }
+
+ @Test
+ void cancellingARunningContainerRunStopsItsDriver() {
+ FakeRunner runner = new FakeRunner();
+ EmrServerlessJobRunService service = dockerService(runner);
+ String jobRunId = service.startJobRun(applicationId, request(null, null)).getJobRunId();
+ List<String> whileStopping = new ArrayList<>();
+ runner.whileRunning = () -> {
+ clock.advance(Duration.ofSeconds(10));
+ service.cancelJobRun(applicationId, jobRunId);
+ whileStopping.add(service.getJobRun(applicationId, jobRunId, null).getState());
+ runNext();
+ };
+
+ runNext();
+
+ assertEquals(List.of("CANCELLING"), whileStopping);
+ assertEquals(Set.of(jobRunId), runner.stopped);
+ JobRun cancelled = service.getJobRun(applicationId, jobRunId, null);
+ assertEquals("CANCELLED", cancelled.getState());
+ assertEquals(10, cancelled.getTotalExecutionDurationSeconds());
+ }
+
+ @Test
+ void aRunCancelledBeforeItsDriverRunsNeverStarts() {
+ FakeRunner runner = new FakeRunner();
+ EmrServerlessJobRunService service = dockerService(runner);
+ String jobRunId = service.startJobRun(applicationId, request(null, null)).getJobRunId();
+
+ service.cancelJobRun(applicationId, jobRunId);
+ tasks.remove(1).run();
+ runNext();
+
+ JobRun cancelled = service.getJobRun(applicationId, jobRunId, null);
+ assertEquals("CANCELLED", cancelled.getState());
+ assertNull(cancelled.getStartedAt());
+ assertEquals(0, cancelled.getTotalExecutionDurationSeconds());
+ assertEquals(List.of(), runner.driversStarted);
+ }
+
+ @Test
+ void aContainerRunPastItsTimeoutIsCancelled() {
+ FakeRunner runner = new FakeRunner();
+ runner.outcome = SparkJobRunner.Outcome.outOfTime();
+ EmrServerlessJobRunService service = dockerService(runner);
+ String jobRunId = service.startJobRun(applicationId, request(null, 5L)).getJobRunId();
+ List<String> whileStopping = new ArrayList<>();
+ runner.whileTimingOut = () -> whileStopping.add(service.getJobRun(applicationId, jobRunId, null).getState());
+
+ runNext();
+
+ assertEquals(List.of("CANCELLING"), whileStopping);
+ assertEquals("CANCELLED", service.getJobRun(applicationId, jobRunId, null).getState());
+ }
+
+ @Test
+ void aHiveRunFailsWithoutRunning() {
+ CreateApplicationRequest create = new CreateApplicationRequest();
+ create.setName("hive");
+ create.setReleaseLabel("emr-7.5.0");
+ create.setType("HIVE");
+ create.setClientToken(UUID.randomUUID().toString());
+ String hive = applications.createApplication(create).getApplicationId();
+ JobDriver.Hive query = new JobDriver.Hive();
+ query.setQuery("s3://bucket/queries/report.sql");
+ StartJobRunRequest request = request(null, null);
+ request.getJobDriver().setSparkSubmit(null);
+ request.getJobDriver().setHive(query);
+
+ JobRun run = dockerService(new FakeRunner()).startJobRun(hive, request);
+
+ assertEquals("FAILED", run.getState());
+ assertEquals(EmrServerlessJobRunService.HIVE_NOT_RUN, run.getStateDetails());
+ assertEquals(List.of(), tasks);
+ }
+
+ @Test
+ void aContainerRunNothingRunsAnyMoreFails() {
+ FakeRunner runner = new FakeRunner();
+ String leftByRestart = dockerService(runner).startJobRun(applicationId, request(null, null)).getJobRunId();
+ EmrServerlessJobRunService service = dockerService(runner);
+ String stoppedByShutdown = service.startJobRun(applicationId, request(null, null)).getJobRunId();
+ runner.whileRunning = () -> runner.stop(stoppedByShutdown);
+ tasks.remove(1).run();
+
+ for (String jobRunId : List.of(leftByRestart, stoppedByShutdown)) {
+ JobRun run = service.getJobRun(applicationId, jobRunId, null);
+ assertEquals("FAILED", run.getState());
+ assertEquals(EmrServerlessJobRunService.INTERRUPTED, run.getStateDetails());
+ }
+ }
+
+ private EmrServerlessJobRunService dockerService(SparkJobRunner runner) {
+ return new EmrServerlessJobRunService(store, applications, iamService, new RegionResolver(REGION, ACCOUNT_ID),
+ null, objectMapper, 0, clock, runner, tasks::add);
+ }
+
+ private void runNext() {
+ tasks.remove(0).run();
+ }
+
+ @Test
+ void aRunsThreadWritesNothingOnceAResetHasBegun() {
+ FakeRunner runner = new FakeRunner();
+ EmrServerlessJobRunService service = dockerService(runner);
+ String beforeReset = service.startJobRun(applicationId, request(null, null)).getJobRunId();
+ // The reset's teardown stops the driver while it runs; beforeReset() comes before the storage wipe.
+ runner.whileRunning = service::beforeReset;
+
+ runNext();
+
+ assertEquals("RUNNING", store.get(beforeReset).orElseThrow().getJobRun().getState());
+ runner.whileRunning = () -> { };
+ String afterReset = service.startJobRun(applicationId, request(null, null)).getJobRunId();
+ runNext();
+ assertEquals("SUCCESS", store.get(afterReset).orElseThrow().getJobRun().getState());
+ }
+
+ @Test
+ void leftoverDriversAreRemovedWhenTheServiceStartsUnlessReconcilingIsOff() {
+ for (boolean reconcile : List.of(true, false)) {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.services().emrserverless().runnerMode()).thenReturn("docker");
+ when(config.services().emrserverless().reconcileContainersOnStartup()).thenReturn(reconcile);
+ EmrServerlessSparkRunner sparkRunner = mock(EmrServerlessSparkRunner.class);
+
+ new EmrServerlessJobRunService(mock(StorageFactory.class), applications, iamService,
+ new RegionResolver(REGION, ACCOUNT_ID), null, objectMapper, config, clock, sparkRunner).shutdown();
+
+ verify(sparkRunner, times(reconcile ? 1 : 0)).removeLeftoverDrivers();
+ }
+ }
+
+ /**
+ * Runs a container run on the test's thread: its driver starts, does what the test asks, and ends
+ * with {@code outcome}, or stopped once {@link #stop} has been called for it.
+ */
+ private static final class FakeRunner implements SparkJobRunner {
+ final List<Job> jobs = new ArrayList<>();
+ final Set<String> stopped = new HashSet<>();
+ final List<String> driversStarted = new ArrayList<>();
+ Outcome outcome = Outcome.exitedWith(0);
+ Runnable whileRunning = () -> { };
+ Runnable whileTimingOut = () -> { };
+
+ @Override
+ public Outcome run(Job job, Progress progress) {
+ jobs.add(job);
+ // As the Docker runner, a stop counts only for a run in progress, so a cancel before then is read here.
+ if (progress.cancelled()) {
+ return Outcome.halted();
+ }
+ driversStarted.add(job.jobRunId());
+ progress.started();
+ whileRunning.run();
+ if (outcome.timedOut()) {
+ progress.timedOut();
+ whileTimingOut.run();
+ }
+ return stopped.contains(job.jobRunId()) ? Outcome.halted() : outcome;
+ }
+
+ @Override
+ public void stop(String jobRunId) {
+ stopped.add(jobRunId);
+ }
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java
new file mode 100644
index 0000000000..792b6ee239
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java
@@ -0,0 +1,242 @@
+package io.github.hectorvent.floci.services.emrserverless;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.exception.NotModifiedException;
+import com.github.dockerjava.api.model.Container;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;
+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;
+import io.github.hectorvent.floci.services.emrserverless.model.JobDriver;
+import io.github.hectorvent.floci.services.lambda.launcher.LambdaExecutionRoleCredentials;
+import io.github.hectorvent.floci.testing.MutableClock;
+import org.junit.jupiter.api.Test;
+import org.mockito.InOrder;
+
+import java.time.Duration;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyMap;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.inOrder;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+class EmrServerlessSparkRunnerTest {
+
+ private static final String ENDPOINT = "http://floci:4566";
+
+ private final EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ private final ContainerLifecycleManager lifecycle = mock(ContainerLifecycleManager.class);
+ private final DockerClient docker = mock(DockerClient.class, RETURNS_DEEP_STUBS);
+ private final SparkJobRunner.Progress progress = mock(SparkJobRunner.Progress.class);
+
+ @Test
+ void sparkSubmitRunsLocallyOnFlocisS3ThenTheRunsOwnSettings() {
+ JobDriver.SparkSubmit sparkSubmit = new JobDriver.SparkSubmit();
+ sparkSubmit.setEntryPoint("s3://bucket/jobs/report.py");
+ sparkSubmit.setEntryPointArguments(List.of("s3://bucket/input/", "s3://bucket/output/"));
+ sparkSubmit.setSparkSubmitParameters("--conf spark.executor.cores=2 --py-files s3://bucket/jobs/lib.zip");
+ Map<String, String> defaults = new LinkedHashMap<>();
+ defaults.put("spark.executor.cores", "4");
+ defaults.put("spark.sql.shuffle.partitions", "8");
+
+ List<String> arguments = EmrServerlessSparkRunner.sparkSubmitArguments(job(sparkSubmit, defaults), ENDPOINT);
+
+ assertEquals(List.of("--master", "local[*]",
+ "--conf", "spark.hadoop.fs.s3a.endpoint=http://floci:4566",
+ "--conf", "spark.hadoop.fs.s3a.endpoint.region=eu-west-1",
+ "--conf", "spark.hadoop.fs.s3a.path.style.access=true",
+ "--conf", "spark.hadoop.fs.s3a.committer.magic.track.commits.in.memory.enabled=true",
+ "--conf", "spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem",
+ "--conf", "spark.hadoop.fs.AbstractFileSystem.s3.impl=org.apache.hadoop.fs.s3a.S3A",
+ "--conf", "spark.hadoop.aws.glue.endpoint=http://floci:4566",
+ "--conf", "spark.hadoop.aws.region=eu-west-1",
+ "--conf", "spark.executor.cores=4",
+ "--conf", "spark.sql.shuffle.partitions=8",
+ "--conf", "spark.executor.cores=2",
+ "--py-files", "s3://bucket/jobs/lib.zip",
+ "s3://bucket/jobs/report.py", "s3://bucket/input/", "s3://bucket/output/"), arguments);
+ }
+
+ @Test
+ void anEntryPointWithoutParametersOrArgumentsFollowsFlocisSettings() {
+ JobDriver.SparkSubmit sparkSubmit = new JobDriver.SparkSubmit();
+ sparkSubmit.setEntryPoint("s3://bucket/jobs/app.jar");
+
+ List<String> arguments = EmrServerlessSparkRunner.sparkSubmitArguments(job(sparkSubmit, Map.of()), ENDPOINT);
+
+ assertEquals("--conf", arguments.get(arguments.size() - 3));
+ assertEquals("spark.hadoop.aws.region=eu-west-1", arguments.get(arguments.size() - 2));
+ assertEquals("s3://bucket/jobs/app.jar", arguments.get(arguments.size() - 1));
+ }
+
+ @Test
+ void sparkSubmitParametersSplitAsAShellWould() {
+ assertEquals(List.of("--conf", "spark.driver.extraJavaOptions=-Da=1 -Db=2", "--conf", "x=it's", "a b", ""),
+ EmrServerlessSparkRunner.splitParameters(
+ " --conf \"spark.driver.extraJavaOptions=-Da=1 -Db=2\"\t--conf 'x=it'\\''s' a\\ b '' "));
+ // Inside double quotes a backslash escapes only ", \, $, ` and a newline, and is kept before anything else.
+ assertEquals(List.of("--conf", "spark.driver.extraJavaOptions=-Dpattern=\\d+ -Dq=\"x\" -Dp=\\ $HOME", "onetwo"),
+ EmrServerlessSparkRunner.splitParameters(
+ "--conf \"spark.driver.extraJavaOptions=-Dpattern=\\d+ -Dq=\\\"x\\\" -Dp=\\\\ \\$HOME\" one\\\ntwo"));
+ assertEquals(List.of(), EmrServerlessSparkRunner.splitParameters(null));
+ assertEquals(List.of(), EmrServerlessSparkRunner.splitParameters(" "));
+ }
+
+ @Test
+ void theImageIsTheReleasesPublicImageUnlessATemplateNamesAnother() {
+ EmrServerlessSparkRunner runner = runner();
+
+ assertEquals("public.ecr.aws/emr-serverless/spark/emr-7.5.0:latest", runner.image("emr-7.5.0"));
+ assertEquals("public.ecr.aws/emr-serverless/spark4/emr-spark-8.1.0:latest", runner.image("emr-spark-8.1.0"));
+ when(config.services().emrserverless().imageTemplate())
+ .thenReturn(Optional.of("registry.internal/emr-serverless/spark/%s:20250425"));
+ assertEquals("registry.internal/emr-serverless/spark/emr-7.5.0:20250425", runner.image("emr-7.5.0"));
+ }
+
+ @Test
+ void aCancelThatFindsTheDriverAlreadyExitedStillStopsTheRun() {
+ EmrServerlessSparkRunner runner = runner();
+ // Docker answers a stop of an exited container with HTTP 304.
+ when(docker.stopContainerCmd("driver").withTimeout(0).exec())
+ .thenThrow(new NotModifiedException("Container already stopped"));
+ SparkJobRunner.Job job = job(Duration.ZERO);
+ doAnswer(started -> {
+ runner.stop(job.jobRunId());
+ return null;
+ }).when(progress).started();
+
+ SparkJobRunner.Outcome outcome = runner.run(job, progress);
+
+ assertTrue(outcome.stopped());
+ verify(lifecycle).stopAndRemove("driver", null, 0);
+ }
+
+ @Test
+ void aResetWhileTheImageIsPulledStopsTheRunOnceItsContainerIsUp() {
+ EmrServerlessSparkRunner runner = runner();
+ when(lifecycle.createAndStart(any())).thenAnswer(pulling -> {
+ runner.stopManagedContainers();
+ return new ContainerLifecycleManager.ContainerInfo("driver", Map.of());
+ });
+
+ SparkJobRunner.Outcome outcome = runner.run(job(Duration.ZERO), progress);
+
+ assertTrue(outcome.stopped());
+ verify(progress, never()).started();
+ verify(lifecycle).stopAndRemove("driver", null, 0);
+ }
+
+ @Test
+ void aRunCancelledWhileItsImageIsPulledNeverStartsItsDriver() {
+ EmrServerlessSparkRunner runner = runner();
+ // Not cancelled when the run begins, cancelled by the time its container is up.
+ when(progress.cancelled()).thenReturn(false, true);
+
+ SparkJobRunner.Outcome outcome = runner.run(job(Duration.ZERO), progress);
+
+ assertTrue(outcome.stopped());
+ verify(progress, never()).started();
+ verify(lifecycle).stopAndRemove("driver", null, 0);
+ }
+
+ @Test
+ void aStopAfterTheRunHasEndedIsDropped() {
+ EmrServerlessSparkRunner runner = runner();
+ SparkJobRunner.Job job = job(Duration.ZERO);
+ assertEquals(0, runner.run(job, progress).exitCode());
+
+ runner.stop(job.jobRunId());
+
+ // A stop kept past its run would halt the next run of this id before it started.
+ assertEquals(0, runner.run(job, progress).exitCode());
+ verify(docker, never()).stopContainerCmd(anyString());
+ }
+
+ @Test
+ void aRunPastItsTimeoutIsToldSoBeforeItsDriverIsRemoved() {
+ EmrServerlessSparkRunner runner = runner();
+ when(docker.inspectContainerCmd("driver").exec().getState().getRunning()).thenReturn(true);
+
+ SparkJobRunner.Outcome outcome = runner.run(job(Duration.ofMillis(5)), progress);
+
+ assertTrue(outcome.timedOut());
+ InOrder order = inOrder(progress, lifecycle);
+ order.verify(progress).timedOut();
+ order.verify(lifecycle).stopAndRemove("driver", null, 0);
+ }
+
+ @Test
+ void theDriverIsWatchedWithoutHoldingADockerConnection() {
+ EmrServerlessSparkRunner runner = runner();
+
+ runner.run(job(Duration.ZERO), progress);
+
+ // A look at the container is one short call; no wait or other stream is held while the driver runs.
+ verify(docker, atLeastOnce()).inspectContainerCmd("driver");
+ verify(docker, never()).waitContainerCmd(anyString());
+ }
+
+ @Test
+ void driversAnEarlierProcessOfThisDeploymentLeftAreRemoved() {
+ EmrServerlessSparkRunner runner = runner();
+ when(config.port()).thenReturn(4566);
+ List<Container> drivers = List.of(driver("left-behind", "4566"), driver("another-deployments", "4567"));
+ when(docker.listContainersCmd().withShowAll(true).withLabelFilter(anyMap()).exec()).thenReturn(drivers);
+
+ runner.removeLeftoverDrivers();
+
+ verify(lifecycle).removeIfExists("left-behind");
+ verify(lifecycle, never()).removeIfExists("another-deployments");
+ }
+
+ // A runner whose driver container, "driver", starts at once and exits with code 0 unless a test says otherwise.
+ private EmrServerlessSparkRunner runner() {
+ when(config.services().emrserverless().imageTemplate()).thenReturn(Optional.empty());
+ when(lifecycle.getDockerClient()).thenReturn(docker);
+ when(lifecycle.createAndStart(any())).thenReturn(new ContainerLifecycleManager.ContainerInfo("driver", Map.of()));
+ when(docker.inspectContainerCmd("driver").exec().getState().getRunning()).thenReturn(false);
+ when(docker.inspectContainerCmd("driver").exec().getState().getExitCodeLong()).thenReturn(0L);
+ return new EmrServerlessSparkRunner(mock(ContainerBuilder.class, RETURNS_DEEP_STUBS), lifecycle,
+ mock(ContainerLogStreamer.class), mock(LaunchedContainerAwsEnv.class),
+ mock(LambdaExecutionRoleCredentials.class), config, new MutableClock(), Duration.ZERO);
+ }
+
+ private static Container driver(String id, String owner) {
+ Container container = mock(Container.class);
+ when(container.getId()).thenReturn(id);
+ when(container.getLabels()).thenReturn(Map.of(ContainerStorageHelper.SERVICE_LABEL, "emr-serverless",
+ ContainerStorageHelper.OWNER_LABEL, owner));
+ return container;
+ }
+
+ private static SparkJobRunner.Job job(Duration timeout) {
+ JobDriver.SparkSubmit sparkSubmit = new JobDriver.SparkSubmit();
+ sparkSubmit.setEntryPoint("s3://bucket/jobs/app.jar");
+ return job(sparkSubmit, Map.of(), timeout);
+ }
+
+ private static SparkJobRunner.Job job(JobDriver.SparkSubmit sparkSubmit, Map<String, String> sparkDefaults) {
+ return job(sparkSubmit, sparkDefaults, Duration.ZERO);
+ }
+
+ private static SparkJobRunner.Job job(JobDriver.SparkSubmit sparkSubmit, Map<String, String> sparkDefaults,
+ Duration timeout) {
+ return new SparkJobRunner.Job("00f0000000000001", "00f0000000000000", "123456789012", "eu-west-1",
+ "emr-7.5.0", "arn:aws:iam::123456789012:role/job", sparkSubmit, sparkDefaults, timeout);
+ }
+}
diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml
index 642bc49535..b52da0422b 100644
--- a/src/test/resources/application.yml
+++ b/src/test/resources/application.yml
@@ -501,6 +501,7 @@ floci:
emrserverless:
enabled: true
job-run-duration-seconds: 0 # 0 = BATCH job runs succeed as soon as they start
+ runner-mode: immediate
ssoadmin:
enabled: true
ssooidc: