Light, fluffy, and always free - The AWS Local Emulator alternative
feat(emr-serverless): run Spark job runs in a container (#6469)
Priamo Germosenedbcac33c770
diff --git a/docs/services/emr-serverless.md b/docs/services/emr-serverless.mdindex da246d4c75..98d3f5fc0e 100644--- a/docs/services/emr-serverless.md+++ b/docs/services/emr-serverless.md@@ -9,6 +9,10 @@ EMR Serverless is an AWS service that provides serverless data analytics environ| `floci.emrserverless.enabled` | Whether the EMR Serverless API is enabled | `true` || `floci.emrserverless.port` | The port the service is exposed on | `4566` || `floci.services.emrserverless.job-run-duration-seconds` | How long a `BATCH` job run stays `RUNNING` before it succeeds; `0` makes it `SUCCESS` as soon as it starts | `0` |+| `floci.services.emrserverless.runner-mode` | `immediate` moves job runs through their states without running anything; `docker` runs each Spark job run in a container (see [Running Spark job runs](#running-spark-job-runs)) | `immediate` |+| `floci.services.emrserverless.docker-network` | Docker network for job run containers | the global default |+| `floci.services.emrserverless.image-template` | Image for a `docker` job run, `%s` standing for the release label | `public.ecr.aws/emr-serverless/spark/%s:latest`, or `spark4/` in place of `spark/` for an `emr-spark-*` release |+| `floci.services.emrserverless.reconcile-containers-on-startup` | Remove the drivers a previous run of this Floci left on the Docker daemon before the first job run starts; turn it off when two Floci instances share a daemon with the same port and no resource namespace | `true` |## Endpoints@@ -57,8 +61,9 @@ omits the terminated application. A terminated application cannot be started or## Job runs-Job runs are metadata only: nothing executes. A run follows the job run states and `GetJobRun`'s-response shape, and settles when it is read.+In the default `immediate` runner mode, job runs are metadata only: nothing executes. A run follows+the job run states and `GetJobRun`'s response shape, and settles when it is read. In the `docker`+runner mode a Spark run executes; see [Running Spark job runs](#running-spark-job-runs).* **Lifecycle**: a run is `RUNNING` from the moment it is submitted. With the default`job-run-duration-seconds` of `0`, a `BATCH` run is `SUCCESS` as soon as it starts; a positive@@ -82,7 +87,7 @@ response shape, and settles when it is read.state in the order the filter names them. A state or mode that does not exist, or more than 8states, is refused with `ValidationException`. An unknown application has no runs. A `nextToken`resumes only the listing it came from, with the same application and filters.-* **Attempts**: nothing runs, so no run fails and none is retried: every run is a single attempt,+* **Attempts**: no run is retried, and in the `immediate` mode none fails: every run is a single attempt,number 1. `GetJobRun`, `ListJobRuns` and `ListJobRunAttempts` report it, with the run's owncreation and update times. `GetJobRun` with `attempt` 1, or without one, returns the run; an`attempt` of 2 or more answers `ResourceNotFoundException`, and one below 1 or not an integer@@ -93,11 +98,44 @@ response shape, and settles when it is read.A state machine drives applications and job runs through the `emr-serverless:` integrations, `.sync`included; see [Step Functions: Amazon EMR Serverless tasks](step-functions.md#amazon-emr-serverless-tasks).+## Running Spark job runs++With `runner-mode` set to `docker` (`FLOCI_SERVICES_EMRSERVERLESS_RUNNER_MODE=docker`), a Spark job+run executes. Floci starts one container per run from the release's image, the base image AWS+publishes for customizing EMR Serverless images (`public.ecr.aws/emr-serverless/spark/<release>`, and+`spark4/<release>` for the AWS runtime for Apache Spark releases, `emr-spark-8.0.0` on), and runs+`spark-submit` in local mode in it: the container holds the driver and its executors.++* **States**: a run is `SCHEDULED` while its image is pulled and its container starts (the first run+ of a release pulls its image, 3.5 to 4.3 GB unpacked; Floci gives a pull three attempts of five+ minutes, so on a slow connection run `docker pull` on the image first, or the run is `FAILED` with+ "Could not pull image"), and `RUNNING` once the Spark driver runs, which sets `startedAt`. It+ is `SUCCESS` when the driver exits with code 0 and `FAILED` otherwise, with the exit code in+ `stateDetails`. `CancelJobRun`, or a `BATCH` run passing its `executionTimeoutMinutes`, makes it+ `CANCELLING` until the driver is stopped, then `CANCELLED`, as the job run states describe. A run+ Floci was running when it stopped is `FAILED` once it is read again. Floci's shutdown and a state+ reset stop every driver, one whose image is still being pulled included, and a driver a hard stop+ left running is removed the first time job runs are used after Floci restarts, unless+ `reconcile-containers-on-startup` is off.+* **What `spark-submit` gets**: the `entryPoint`, its `entryPointArguments` and the+ `sparkSubmitParameters`, after the `spark-defaults` properties of the application's+ `runtimeConfiguration`, or the run's own in `configurationOverrides.applicationConfiguration`,+ which replace the application's whole `spark-defaults` classification. From `emr-spark-8.1.0` the+ two are merged by property instead, the run's winning. `s3://` and `s3a://` paths reach Floci's S3 through S3A, on releases before 7.10.0+ too, where `s3://` would otherwise be EMRFS; the entry point and its files can be read from there.+ The Glue Data Catalog client is pointed at Floci's Glue.+* **Credentials**: the container gets a session for the run's execution role when IAM knows the role,+ otherwise the account's placeholder credentials, and `AWS_ENDPOINT_URL` pointing at Floci, so the+ job's own SDK clients reach Floci too.+* **Output**: the driver's output goes to Floci's log, prefixed `emr-serverless:<jobRunId>`.+* **Image**: `image-template` names another image, such as a mirror or a dated tag of the release's+ image: `registry.internal/emr-serverless/spark/%s:20250425`.+## Limitations and Differences from AWS-* **Jobs do not execute**: `StartJobRun` records a job run and moves it through its states, but no Spark or Hive job runs.+* **Jobs do not execute by default**: in the `immediate` runner mode `StartJobRun` records a job run and moves it through its states, but nothing runs. In the `docker` mode a Spark run executes in local mode in one container, so worker counts and sizes, initial and maximum capacity are not applied; a Hive run is `FAILED` without running, in Floci's own wording, as is a driver's failure. The monitoring configuration's S3 and CloudWatch log destinations are not written to, and Floci's CA is not in the job's Java truststore, so a job reaches Floci over its HTTP endpoint only.* **Unsourced errors**: AWS documents neither the error for a run submitted to an application whose auto-start is off nor the one for a run on a terminated application; Floci answers both with `ValidationException`. The same holds for a `BATCH` timeout outside 1 to 10080 minutes on `emr-7.11.0` or later (`ValidationException`) and for `GetJobRun` naming an attempt a run never made (`ResourceNotFoundException`), both in Floci's own wording. A run needs a `jobDriver`, as every `JobRun` carries one, and `jobDriver` is a union, so a run without a driver, or with one that sets both `sparkSubmit` and `hive` or neither, is refused with `ValidationException` too, in Floci's own wording.* **StopApplication with running jobs**: AWS requires running jobs to finish or be cancelled before an application stops. Floci does not enforce it; with the default duration of `0` no `BATCH` run is ever running.-* **Not emulated**: retries (`retryPolicy` is stored and returned, but no run fails, so none is retried), queuing (`QUEUED`), `totalResourceUtilization` and `billedResourceUtilization`, and `shutdownGracePeriodInSeconds` (accepted and ignored).+* **Not emulated**: retries (`retryPolicy` is stored and returned, but no run is retried), queuing (`QUEUED`), `totalResourceUtilization` and `billedResourceUtilization`, and `shutdownGracePeriodInSeconds` (accepted and ignored: a cancelled run's driver is stopped at once).* **Instant Start/Stop**: Floci marks the application `STARTED` or `STOPPED` immediately without provisioning actual compute capacity in the background.-* **Data Plane**: Floci does not implement the data-plane API or any execution environments.+* **Data Plane**: Floci does not implement the data-plane API, such as interactive sessions.diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex 2ab9f25014..2dbc3567ba 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -1104,6 +1104,34 @@ public interface EmulatorConfig {/** How long a BATCH job run stays RUNNING before it succeeds; 0 = a run succeeds as soon as it starts. */@WithDefault("0")int jobRunDurationSeconds();++ /**+ * {@code immediate} follows the job run states without running anything; {@code docker} runs+ * each Spark job run's {@code spark-submit} in a container of the release's EMR Serverless image.+ */+ @WithDefault("immediate")+ String runnerMode();++ Optional<String> dockerNetwork();++ /**+ * The image a {@code docker} job run uses, with {@code %s} standing for the application's+ * release label, for example "registry.internal/emr-serverless/spark/%s:20250425". If omitted,+ * Floci uses the release's public image on Amazon ECR Public.+ */+ Optional<String> imageTemplate();++ /**+ * When true, Floci removes, in the {@code docker} runner mode, every EMR Serverless driver a+ * previous run of <em>this same</em> Floci left on the daemon (matched by the+ * {@code io.floci.owner} label) before its first job run starts. Turn it off when two Floci+ * instances share a daemon with the same port and no resource namespace, so one does not+ * remove the other's drivers.+ *+ * Env var: FLOCI_SERVICES_EMRSERVERLESS_RECONCILE_CONTAINERS_ON_STARTUP+ */+ @WithDefault("true")+ boolean reconcileContainersOnStartup();}interface Route53ResolverServiceConfig {diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.javaindex 82bb33fb4a..efbf66f303 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java@@ -525,6 +525,10 @@ public class ContainerLifecycleManager {} catch (NotFoundException e) {LOG.debugv("Container {0} not found (already removed)", containerId);stoppedOrMissing = true;+ } catch (NotModifiedException e) {+ // HTTP 304: the container had already exited, which is the outcome the stop wanted.+ LOG.debugv("Container {0} had already stopped", containerId);+ stoppedOrMissing = true;} catch (Exception e) {LOG.warnv("Error stopping container {0}: {1}", containerId, e.getMessage());}@@ -586,6 +590,9 @@ public class ContainerLifecycleManager {LOG.debugv("Container {0} not found (already removed)", containerId);closeLogStreamIfPresent(logStream);return;+ } catch (NotModifiedException e) {+ // HTTP 304: the container had already exited, which is the outcome the stop wanted.+ LOG.debugv("Container {0} had already stopped", containerId);} catch (Exception e) {stopFailure = e;LOG.warnv("Error stopping container {0}: {1}", containerId, e.getMessage());diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.javaindex d074d2f2ba..38f599bcc4 100644--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunService.java@@ -12,6 +12,8 @@ import io.github.hectorvent.floci.core.common.PaginatedResult;import io.github.hectorvent.floci.core.common.Pagination;import io.github.hectorvent.floci.core.common.RegionResolver;import io.github.hectorvent.floci.core.common.RequestContext;+import io.github.hectorvent.floci.core.common.RequestScopes;+import io.github.hectorvent.floci.core.common.Resettable;import io.github.hectorvent.floci.core.common.SmithyValidationErrors;import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;import io.github.hectorvent.floci.core.storage.StorageFactory;@@ -21,8 +23,10 @@ import io.github.hectorvent.floci.services.emrserverless.model.JobRun;import io.github.hectorvent.floci.services.emrserverless.model.JobRunAttemptSummary;import io.github.hectorvent.floci.services.emrserverless.model.JobRunRecord;import io.github.hectorvent.floci.services.emrserverless.model.JobRunSummary;+import io.github.hectorvent.floci.services.emrserverless.model.RuntimeConfiguration;import io.github.hectorvent.floci.services.emrserverless.model.StartJobRunRequest;import io.github.hectorvent.floci.services.iam.IamService;+import jakarta.annotation.PreDestroy;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;@@ -35,35 +39,54 @@ import java.time.format.DateTimeParseException;import java.time.temporal.ChronoUnit;import java.util.ArrayList;import java.util.HashMap;+import java.util.HashSet;import java.util.LinkedHashMap;import java.util.List;import java.util.Map;import java.util.Set;+import java.util.concurrent.Executor;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;import java.util.regex.Matcher;import java.util.regex.Pattern;/**- * EMR Serverless job runs. Nothing executes: a run follows the job run states and GetJobRun's- * shape, and settles lazily when read. Times are kept to the millisecond, as AWS reports them. With the default {@code job-run-duration-seconds} of 0 a+ * EMR Serverless job runs. In the default {@code immediate} runner mode nothing executes: a run+ * follows the job run states and GetJobRun's shape, and settles lazily when read. Times are kept to+ * the millisecond, as AWS reports them. With the default {@code job-run-duration-seconds} of 0 a* BATCH run is SUCCESS as soon as it starts; a positive duration keeps it RUNNING that long, which* makes CancelJobRun and the execution timeout observable. A STREAMING run, which EMR Serverless* "expects to run for a long time", stays RUNNING until it is cancelled.*+ * <p>In the {@code docker} runner mode a Spark run executes: it is SCHEDULED while its container is+ * prepared, RUNNING once its Spark driver runs, and ends as the driver does, SUCCESS on exit code 0+ * and FAILED otherwise. Cancelling it, or its execution timeout, makes it CANCELLING until the driver+ * is stopped, then CANCELLED.+ ** <p>Every public method is synchronized: admitting a run (look up its client token, then write- * it), settling one and cancelling one each read and then write a run, and must not interleave.+ * it), settling one and cancelling one each read and then write a run, and must not interleave. A+ * container run's changes come from the thread that runs it, under the same lock.*/@ApplicationScoped-public class EmrServerlessJobRunService {+public class EmrServerlessJobRunService implements Resettable {private static final Logger LOG = Logger.getLogger(EmrServerlessJobRunService.class);+ static final String STATE_SCHEDULED = "SCHEDULED";static final String STATE_RUNNING = "RUNNING";static final String STATE_SUCCESS = "SUCCESS";+ static final String STATE_FAILED = "FAILED";+ static final String STATE_CANCELLING = "CANCELLING";static final String STATE_CANCELLED = "CANCELLED";private static final String MODE_BATCH = "BATCH";private static final String MODE_STREAMING = "STREAMING";+ private static final String RUNNER_DOCKER = "docker";+ private static final String SPARK_DEFAULTS = "spark-defaults";+ // No source gives AWS's wording for these; they are Floci's.+ static final String HIVE_NOT_RUN = "Floci runs Spark job runs only, and this is a Hive job run.";+ static final String INTERRUPTED = "Floci stopped while the job run was in progress.";- private static final Set<String> TERMINAL_STATES = Set.of(STATE_SUCCESS, "FAILED", STATE_CANCELLED);+ private static final Set<String> TERMINAL_STATES = Set.of(STATE_SUCCESS, STATE_FAILED, STATE_CANCELLED);private static final List<String> STATES = List.of("SUBMITTED", "PENDING", "SCHEDULED", STATE_RUNNING,STATE_SUCCESS, "FAILED", "CANCELLING", STATE_CANCELLED, "QUEUED");// JobRunStateSet holds at most 8 states.@@ -77,7 +100,10 @@ public class EmrServerlessJobRunService {// onwards", and the timeout "can no longer be set to 0".private static final long MAX_BATCH_EXECUTION_TIMEOUT_MINUTES = 10_080;private static final Pattern RELEASE_LABEL = Pattern.compile("emr-(\\d{1,6})\\.(\\d{1,6})(?:\\..*)?");- // Nothing runs, so no run fails and none is retried: every run is its own one attempt.+ // The AWS runtime for Apache Spark, emr-spark-8.0.0 on (and emr-spark-8.0-preview).+ private static final Pattern SPARK_RUNTIME_RELEASE_LABEL =+ Pattern.compile("emr-spark-(\\d{1,6})\\.(\\d{1,6})(?:[.-].*)?");+ // No run is retried: every run is its own one attempt.private static final int ATTEMPT = 1;private static final int MAX_LIST_RESULTS = 50;// IAMRoleArn's pattern, as the model and AWS's validation message spell it.@@ -93,20 +119,41 @@ public class EmrServerlessJobRunService {private final ObjectMapper objectMapper;private final int runDurationSeconds;private final Clock clock;+ // Null in the immediate runner mode.+ private final SparkJobRunner sparkRunner;+ private final Executor runExecutor;+ // Container runs a thread is running; one not here that has not ended was left by a restart.+ private final Set<String> executing = new HashSet<>();+ // Raised by each reset, so a run's thread from before it writes nothing back after the wipe.+ private long generation;@Injectpublic EmrServerlessJobRunService(StorageFactory storageFactory, EmrServerlessService applications,IamService iamService, RegionResolver regionResolver,RequestContext requestContext, ObjectMapper objectMapper,- EmulatorConfig config, Clock clock) {+ EmulatorConfig config, Clock clock, EmrServerlessSparkRunner sparkRunner) {this(storageFactory.create("emrserverless", "emr-serverless-job-runs.json", new TypeReference<>() {}),applications, iamService, regionResolver, requestContext, objectMapper,- config.services().emrserverless().jobRunDurationSeconds(), clock);+ config.services().emrserverless().jobRunDurationSeconds(), clock,+ RUNNER_DOCKER.equalsIgnoreCase(config.services().emrserverless().runnerMode()) ? sparkRunner : null,+ Executors.newCachedThreadPool(Thread.ofPlatform().daemon().name("emr-serverless-run-", 0).factory()));+ // Before any run of this process starts: a driver an earlier one left belongs to a run read as FAILED.+ if (this.sparkRunner != null && config.services().emrserverless().reconcileContainersOnStartup()) {+ sparkRunner.removeLeftoverDrivers();+ }}EmrServerlessJobRunService(AccountAwareStorageBackend<JobRunRecord> store, EmrServerlessService applications,IamService iamService, RegionResolver regionResolver, RequestContext requestContext,ObjectMapper objectMapper, int runDurationSeconds, Clock clock) {+ this(store, applications, iamService, regionResolver, requestContext, objectMapper, runDurationSeconds, clock,+ null, Runnable::run);+ }++ EmrServerlessJobRunService(AccountAwareStorageBackend<JobRunRecord> store, EmrServerlessService applications,+ IamService iamService, RegionResolver regionResolver, RequestContext requestContext,+ ObjectMapper objectMapper, int runDurationSeconds, Clock clock,+ SparkJobRunner sparkRunner, Executor runExecutor) {if (runDurationSeconds < 0) {throw new IllegalArgumentException("floci.services.emrserverless.job-run-duration-seconds must not be negative: " + runDurationSeconds);@@ -119,6 +166,15 @@ public class EmrServerlessJobRunService {this.objectMapper = objectMapper;this.runDurationSeconds = runDurationSeconds;this.clock = clock;+ this.sparkRunner = sparkRunner;+ this.runExecutor = runExecutor;+ }++ @PreDestroy+ void shutdown() {+ if (runExecutor instanceof ExecutorService executor) {+ executor.shutdownNow();+ }}/**@@ -142,8 +198,13 @@ public class EmrServerlessJobRunService {if (body.getTags() != null) {EmrServerlessService.validateTags(body.getTags());}- JobRunRecord record = applications.admitJobRun(applicationId, app -> checkBatchTimeout(app, body),- app -> newRun(app, body, original));+ JobRunRecord record = applications.admitJobRun(applicationId, app -> checkBatchTimeout(app, body), app -> {+ JobRunRecord admitted = newRun(app, body, original);+ if (sparkRunner != null) {+ launch(admitted, app);+ }+ return admitted;+ });LOG.infov("Started EMR Serverless job run {0} on application {1}", record.getJobRun().getJobRunId(),applicationId);return settle(record).getJobRun();@@ -167,7 +228,8 @@ public class EmrServerlessJobRunService {throw new AwsException("ResourceNotFoundException","Job run " + jobRunId + " has no attempt " + attempt, 404);}- return settle(record).getJobRun();+ // A copy: a container run's thread may change the stored run while the response is written.+ return objectMapper.convertValue(settle(record).getJobRun(), JobRun.class);}/** A run's attempts, which are its one attempt. */@@ -246,13 +308,25 @@ public class EmrServerlessJobRunService {page.nextToken());}- /** Cancels a run that has not finished. A finished run is refused, with AWS's wording. */+ /**+ * Cancels a run that has not finished. A finished run is refused, with AWS's wording. A container+ * run is CANCELLING until its driver has stopped.+ */public synchronized JobRun cancelJobRun(String applicationId, String jobRunId) {applications.getApplication(applicationId);JobRunRecord record = settle(findRun(applicationId, jobRunId));if (isTerminal(record.getJobRun())) {throw new AwsException("ValidationException", "Job run is not in a cancellable state", 400);}+ if (record.isContainerRun()) {+ JobRun run = record.getJobRun();+ if (!STATE_CANCELLING.equals(run.getState())) {+ markCancelling(record);+ runExecutor.execute(() -> sparkRunner.stop(jobRunId));+ LOG.infov("Cancelling EMR Serverless job run {0} on application {1}", jobRunId, applicationId);+ }+ return run;+ }finish(record, STATE_CANCELLED, clock.instant().truncatedTo(ChronoUnit.MILLIS));LOG.infov("Cancelled EMR Serverless job run {0} on application {1}", jobRunId, applicationId);return record.getJobRun();@@ -322,7 +396,7 @@ public class EmrServerlessJobRunService {run.setUpdatedAt(now);run.setExecutionRole(request.getExecutionRoleArn());run.setExecutionIamPolicy(request.getExecutionIamPolicy());- run.setState(STATE_RUNNING);+ run.setState(sparkRunner != null ? STATE_SCHEDULED : STATE_RUNNING);run.setStateDetails("");run.setReleaseLabel(app.getReleaseLabel());run.setConfigurationOverrides(request.getConfigurationOverrides());@@ -336,17 +410,177 @@ public class EmrServerlessJobRunService {run.setAttempt(ATTEMPT);run.setAttemptCreatedAt(now);run.setAttemptUpdatedAt(now);- run.setStartedAt(now);+ if (sparkRunner == null) {+ run.setStartedAt(now);+ }JobRunRecord record = new JobRunRecord();record.setJobRun(run);record.setClientToken(request.getClientToken());record.setRequest(original);record.setApplicationType(app.getType());+ record.setContainerRun(sparkRunner != null);store.put(jobRunId, record);return record;}+ /** Hands a container run to a thread of its own, which runs it to its end. */+ private void launch(JobRunRecord record, Application app) {+ JobRun run = record.getJobRun();+ if (run.getJobDriver().getSparkSubmit() == null) {+ run.setStateDetails(HIVE_NOT_RUN);+ finish(record, STATE_FAILED, run.getCreatedAt());+ return;+ }+ AwsArnUtils.Arn arn = AwsArnUtils.parse(run.getArn());+ // A STREAMING run "expects to run for a long time", so only a BATCH run's timeout ends it.+ Long minutes = run.getExecutionTimeoutMinutes();+ Duration timeout = MODE_STREAMING.equals(run.getMode()) || minutes == null || minutes == 0+ ? Duration.ZERO : Duration.ofMinutes(minutes);+ SparkJobRunner.Job job = new SparkJobRunner.Job(run.getJobRunId(), run.getApplicationId(), arn.accountId(),+ arn.region(), run.getReleaseLabel(), run.getExecutionRole(), run.getJobDriver().getSparkSubmit(),+ sparkDefaults(app, run), timeout);+ executing.add(run.getJobRunId());+ long launchedIn = generation;+ runExecutor.execute(() -> execute(job, launchedIn));+ }++ private void execute(SparkJobRunner.Job job, long launchedIn) {+ SparkJobRunner.Outcome outcome;+ try {+ outcome = sparkRunner.run(job, new RunProgress(job, launchedIn));+ } catch (RuntimeException e) {+ LOG.warnv(e, "EMR Serverless job run {0} could not run", job.jobRunId());+ outcome = SparkJobRunner.Outcome.couldNotStart(e.getMessage());+ }+ SparkJobRunner.Outcome ended = outcome;+ RequestScopes.runAs(job.accountId(), job.region(), () -> ended(job.jobRunId(), ended, launchedIn));+ }++ /**+ * A reset has begun: the runs' threads from before it write nothing more, so the storage wipe that+ * follows leaves none of their runs behind. Its teardown has already stopped their drivers.+ */+ @Override+ public synchronized void beforeReset() {+ generation++;+ executing.clear();+ }++ @Override+ public void clear() {+ // The runs live in storage, which the reset has already wiped.+ }++ // Whether a run's driver may still start: a cancelled run is past it, and so is one a reset removed.+ private synchronized boolean waitingToStart(String jobRunId, long launchedIn) {+ return launchedIn == generation && store.get(jobRunId)+ .map(record -> STATE_SCHEDULED.equals(record.getJobRun().getState())).orElse(false);+ }++ private synchronized void driverStarted(String jobRunId, long launchedIn) {+ if (launchedIn != generation) {+ return;+ }+ store.get(jobRunId).filter(record -> STATE_SCHEDULED.equals(record.getJobRun().getState()))+ .ifPresent(record -> {+ JobRun run = record.getJobRun();+ Instant now = clock.instant().truncatedTo(ChronoUnit.MILLIS);+ run.setState(STATE_RUNNING);+ run.setStartedAt(now);+ run.setUpdatedAt(now);+ run.setAttemptUpdatedAt(now);+ store.put(jobRunId, record);+ LOG.infov("EMR Serverless job run {0} is running", jobRunId);+ });+ }++ // A run past its timeout is CANCELLING while its driver is stopped: "the job run has timed out".+ private synchronized void driverTimedOut(String jobRunId, long launchedIn) {+ if (launchedIn != generation) {+ return;+ }+ store.get(jobRunId).filter(record -> STATE_RUNNING.equals(record.getJobRun().getState()))+ .ifPresent(this::markCancelling);+ }++ private void markCancelling(JobRunRecord record) {+ JobRun run = record.getJobRun();+ Instant now = clock.instant().truncatedTo(ChronoUnit.MILLIS);+ run.setState(STATE_CANCELLING);+ run.setUpdatedAt(now);+ run.setAttemptUpdatedAt(now);+ store.put(run.getJobRunId(), record);+ }++ /**+ * Ends a container run as its driver did. A cancelled run, or one past its timeout ("the job run+ * has timed out" is a reason to be CANCELLING), is CANCELLED; a driver stopped otherwise was+ * stopped by Floci's own shutdown.+ */+ private synchronized void ended(String jobRunId, SparkJobRunner.Outcome outcome, long launchedIn) {+ if (launchedIn != generation) {+ return;+ }+ executing.remove(jobRunId);+ JobRunRecord record = store.get(jobRunId).orElse(null);+ if (record == null || isTerminal(record.getJobRun())) {+ return;+ }+ JobRun run = record.getJobRun();+ Instant now = clock.instant().truncatedTo(ChronoUnit.MILLIS);+ if (STATE_CANCELLING.equals(run.getState()) || outcome.timedOut()) {+ finish(record, STATE_CANCELLED, now);+ } else if (outcome.exitCode() != null && outcome.exitCode() == 0) {+ finish(record, STATE_SUCCESS, now);+ } else {+ // No source gives AWS's wording for a driver that fails; this is Floci's.+ run.setStateDetails(outcome.stopped() ? INTERRUPTED+ : outcome.failure() != null ? "Job run could not start: " + outcome.failure()+ : "Job run failed: the Spark driver exited with code " + outcome.exitCode() + ".");+ finish(record, STATE_FAILED, now);+ }+ LOG.infov("EMR Serverless job run {0} ended {1}", jobRunId, run.getState());+ }++ /**+ * The {@code spark-defaults} properties for a run: the application's runtime configuration, then the+ * run's own {@code applicationConfiguration}. The two merge by classification, so a run's+ * {@code spark-defaults} replaces the application's: "Only the properties included in StartJobRun is+ * considered for this classification." From emr-spark-8.1.0 they merge by property, the run's winning.+ */+ private static Map<String, String> sparkDefaults(Application app, JobRun run) {+ Map<String, String> properties = new LinkedHashMap<>();+ if (app.getRuntimeConfiguration() != null) {+ for (RuntimeConfiguration configuration : app.getRuntimeConfiguration()) {+ addSparkDefaults(properties, configuration.getMembers());+ }+ }+ Object overrides = run.getConfigurationOverrides() != null+ ? run.getConfigurationOverrides().get("applicationConfiguration") : null;+ if (overrides instanceof List<?> configurations) {+ boolean replaces = !releaseAtLeast(SPARK_RUNTIME_RELEASE_LABEL, run.getReleaseLabel(), 8, 1);+ for (Object configuration : configurations) {+ if (configuration instanceof Map<?, ?> members+ && SPARK_DEFAULTS.equals(members.get("classification"))) {+ if (replaces) {+ properties.clear();+ replaces = false;+ }+ addSparkDefaults(properties, members);+ }+ }+ }+ return properties;+ }++ private static void addSparkDefaults(Map<String, String> properties, Map<?, ?> configuration) {+ if (SPARK_DEFAULTS.equals(configuration.get("classification"))+ && configuration.get("properties") instanceof Map<?, ?> values) {+ values.forEach((key, value) -> properties.put(String.valueOf(key), String.valueOf(value)));+ }+ }+/*** Finishes a BATCH run that has been RUNNING for the configured duration, or cancels it at its* execution timeout when that comes first: "If the job run exceeds this duration, Amazon EMR@@ -354,6 +588,14 @@ public class EmrServerlessJobRunService {*/private JobRunRecord settle(JobRunRecord record) {JobRun run = record.getJobRun();+ if (record.isContainerRun()) {+ // Its thread ends it; without one, Floci stopped while it ran and nothing is left to.+ if (!isTerminal(run) && !executing.contains(run.getJobRunId())) {+ run.setStateDetails(INTERRUPTED);+ finish(record, STATE_FAILED, clock.instant().truncatedTo(ChronoUnit.MILLIS));+ }+ return record;+ }if (isTerminal(run) || MODE_STREAMING.equals(run.getMode())) {return record;}@@ -379,7 +621,9 @@ public class EmrServerlessJobRunService {run.setEndedAt(endedAt);run.setUpdatedAt(endedAt);run.setAttemptUpdatedAt(endedAt);- run.setTotalExecutionDurationSeconds((int) Duration.between(run.getStartedAt(), endedAt).toSeconds());+ // A container run cancelled or failed before its driver ran never started.+ run.setTotalExecutionDurationSeconds(run.getStartedAt() != null+ ? (int) Duration.between(run.getStartedAt(), endedAt).toSeconds() : 0);store.put(run.getJobRunId(), record);}@@ -500,12 +744,17 @@ public class EmrServerlessJobRunService {}private static boolean boundsBatchTimeouts(String releaseLabel) {- Matcher matcher = RELEASE_LABEL.matcher(releaseLabel != null ? releaseLabel : "");+ return releaseAtLeast(RELEASE_LABEL, releaseLabel, 7, 11);+ }++ // Whether releaseLabel is of the family labelPattern matches, at major.minor or later.+ private static boolean releaseAtLeast(Pattern labelPattern, String releaseLabel, int major, int minor) {+ Matcher matcher = labelPattern.matcher(releaseLabel != null ? releaseLabel : "");if (!matcher.matches()) {return false;}- int major = Integer.parseInt(matcher.group(1));- return major > 7 || (major == 7 && Integer.parseInt(matcher.group(2)) >= 11);+ int labelMajor = Integer.parseInt(matcher.group(1));+ return labelMajor > major || (labelMajor == major && Integer.parseInt(matcher.group(2)) >= minor);}private static JobRunSummary toSummary(JobRunRecord record) {@@ -550,4 +799,32 @@ public class EmrServerlessJobRunService {summary.setAttempt(ATTEMPT);return summary;}++ /** What the thread running a container run tells the service, and asks it, as that run. */+ private final class RunProgress implements SparkJobRunner.Progress {++ private final SparkJobRunner.Job job;+ private final long launchedIn;++ RunProgress(SparkJobRunner.Job job, long launchedIn) {+ this.job = job;+ this.launchedIn = launchedIn;+ }++ @Override+ public void started() {+ RequestScopes.runAs(job.accountId(), job.region(), () -> driverStarted(job.jobRunId(), launchedIn));+ }++ @Override+ public void timedOut() {+ RequestScopes.runAs(job.accountId(), job.region(), () -> driverTimedOut(job.jobRunId(), launchedIn));+ }++ @Override+ public boolean cancelled() {+ return !RequestScopes.callAs(job.accountId(), job.region(),+ () -> waitingToStart(job.jobRunId(), launchedIn));+ }+ }}diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.javanew file mode 100644index 0000000000..eee225b28d--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunner.java@@ -0,0 +1,352 @@+package io.github.hectorvent.floci.services.emrserverless;++import com.github.dockerjava.api.command.InspectContainerResponse;+import com.github.dockerjava.api.exception.NotFoundException;+import com.github.dockerjava.api.exception.NotModifiedException;+import com.github.dockerjava.api.model.Container;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.ContainerTeardown;+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;+import io.github.hectorvent.floci.core.common.docker.ContainerSpec;+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;+import io.github.hectorvent.floci.services.emrserverless.model.JobDriver;+import io.github.hectorvent.floci.services.iam.model.SessionCreds;+import io.github.hectorvent.floci.services.lambda.launcher.LambdaExecutionRoleCredentials;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.inject.Inject;+import org.jboss.logging.Logger;++import java.io.Closeable;+import java.time.Clock;+import java.time.Duration;+import java.time.Instant;+import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.atomic.AtomicReference;++/**+ * Runs a Spark job run's {@code spark-submit} in a container of the release's EMR Serverless image.+ * The image's own entrypoint hands over to a script EMR Serverless supplies at run time, which the+ * public image lacks, so the runner starts {@code spark-submit} itself, in local mode: the container+ * holds the driver and its executors. The container reaches Floci as the other containers Floci+ * launches do, with the run's execution role as its credentials, and S3A, which takes {@code s3://}+ * on every release, is pointed at Floci's S3.+ */+@ApplicationScoped+public class EmrServerlessSparkRunner implements SparkJobRunner, ContainerTeardown {++ private static final Logger LOG = Logger.getLogger(EmrServerlessSparkRunner.class);+ private static final String SPARK_SUBMIT = "/usr/lib/spark/bin/spark-submit";+ private static final String DEFAULT_IMAGE =+ "public.ecr.aws/emr-serverless/spark/%s:latest"; // partition-literal: ECR Public's registry host, where AWS publishes the base images; image-template overrides it+ // The AWS runtime for Apache Spark (emr-spark-8.0.0 on) publishes its images under spark4/ instead.+ private static final String SPARK_RUNTIME_DEFAULT_IMAGE =+ "public.ecr.aws/emr-serverless/spark4/%s:latest"; // partition-literal: ECR Public's registry host, where AWS publishes the base images; image-template overrides it+ // "The default behavior for shutdown is to have no grace period": immediate termination.+ private static final int STOP_TIMEOUT_SECONDS = 0;+ private static final String SERVICE = "emr-serverless";+ // The characters a backslash escapes inside double quotes; before any other it is kept.+ private static final String DOUBLE_QUOTE_ESCAPES = "\"\\$`\n";+ // How often a running driver is looked at, as Batch's runner looks at its jobs.+ private static final Duration POLL_INTERVAL = Duration.ofMillis(250);++ // The runs inside run(); a stop for a run not here comes after its driver has gone, and is dropped.+ private final ConcurrentHashMap<String, Launch> launches = new ConcurrentHashMap<>();++ private final ContainerBuilder containerBuilder;+ private final ContainerLifecycleManager lifecycleManager;+ private final ContainerLogStreamer logStreamer;+ private final LaunchedContainerAwsEnv awsEnv;+ private final LambdaExecutionRoleCredentials roleCredentials;+ private final EmulatorConfig config;+ private final Clock clock;+ private final Duration pollInterval;++ @Inject+ public EmrServerlessSparkRunner(ContainerBuilder containerBuilder, ContainerLifecycleManager lifecycleManager,+ ContainerLogStreamer logStreamer, LaunchedContainerAwsEnv awsEnv,+ LambdaExecutionRoleCredentials roleCredentials, EmulatorConfig config,+ Clock clock) {+ this(containerBuilder, lifecycleManager, logStreamer, awsEnv, roleCredentials, config, clock, POLL_INTERVAL);+ }++ EmrServerlessSparkRunner(ContainerBuilder containerBuilder, ContainerLifecycleManager lifecycleManager,+ ContainerLogStreamer logStreamer, LaunchedContainerAwsEnv awsEnv,+ LambdaExecutionRoleCredentials roleCredentials, EmulatorConfig config, Clock clock,+ Duration pollInterval) {+ this.containerBuilder = containerBuilder;+ this.lifecycleManager = lifecycleManager;+ this.logStreamer = logStreamer;+ this.awsEnv = awsEnv;+ this.roleCredentials = roleCredentials;+ this.config = config;+ this.clock = clock;+ this.pollInterval = pollInterval;+ }++ @Override+ public Outcome run(Job job, Progress progress) {+ Launch launch = new Launch();+ launches.put(job.jobRunId(), launch);+ Optional<SessionCreds> credentials = Optional.empty();+ Closeable logHandle = null;+ try {+ // A cancel before the run was in launches had nothing to stop, so it is read here.+ if (progress.cancelled()) {+ return Outcome.halted();+ }+ credentials = roleCredentials.forRole(job.executionRole(), job.accountId(), job.jobRunId());+ Map<String, String> labels = new LinkedHashMap<>(ContainerStorageHelper.resourceIdentityLabels(+ SERVICE, job.jobRunId(), job.accountId(), job.region()));+ // The deployment that started the driver, which removeLeftoverDrivers() goes by.+ labels.put(ContainerStorageHelper.OWNER_LABEL, ContainerStorageHelper.ownerIdentity(config));+ String endpoint = awsEnv.flociEndpoint();+ ContainerSpec spec = containerBuilder.newContainer(image(job.releaseLabel()))+ .withName(ContainerStorageHelper.dockerName(config, SERVICE + "-" + job.jobRunId()))+ .withEntrypoint(List.of(SPARK_SUBMIT))+ .withCmd(sparkSubmitArguments(job, endpoint))+ .withEnv(awsEnv.sdkBaselineEnv(job.region(), Optional.empty(), credentials, job.accountId()))+ .withDockerNetwork(config.services().emrserverless().dockerNetwork())+ .withHostDockerInternalOnLinux()+ .withEmbeddedDns()+ .withLogRotation()+ .withLabels(labels)+ .build();+ String containerId = lifecycleManager.createAndStart(spec).containerId();+ launch.container.set(containerId);+ // A stop, a reset or a shutdown while the image was pulled found no container to stop.+ if (launch.stopRequested || progress.cancelled()) {+ return Outcome.halted();+ }+ logHandle = logStreamer.attachConsoleOnly(containerId, SERVICE + ":" + job.jobRunId());+ progress.started();+ Integer exitCode = awaitExit(containerId, job, progress);+ if (launch.stopRequested) {+ return Outcome.halted();+ }+ return exitCode == null ? Outcome.outOfTime() : Outcome.exitedWith(exitCode);+ } catch (InterruptedException e) {+ // Floci is shutting down: its teardown stops the driver.+ Thread.currentThread().interrupt();+ return Outcome.halted();+ } catch (RuntimeException e) {+ if (launch.stopRequested) {+ // A reset removed the driver while it was being watched.+ return Outcome.halted();+ }+ LOG.warnv(e, "EMR Serverless job run {0} could not run", job.jobRunId());+ return Outcome.couldNotStart(e.getMessage());+ } finally {+ launches.remove(job.jobRunId(), launch);+ String unclaimed = launch.container.getAndSet(null);+ if (unclaimed != null) {+ lifecycleManager.stopAndRemove(unclaimed, logHandle, STOP_TIMEOUT_SECONDS);+ } else if (logHandle != null) {+ // Floci's teardown claimed the driver, but the log stream is still this run's to close.+ lifecycleManager.closeLogStreamAfterContainerStop(logHandle);+ }+ credentials.ifPresent(session -> roleCredentials.unregister(job.accountId(), session.accessKeyId()));+ }+ }++ @Override+ public void stop(String jobRunId) {+ Launch launch = launches.get(jobRunId);+ if (launch == null) {+ return;+ }+ launch.stopRequested = true;+ String containerId = launch.container.get();+ if (containerId == null) {+ // Still starting: run() reads the request once its container is up.+ return;+ }+ // run() sees it stopped at its next look, and removes it.+ try {+ lifecycleManager.getDockerClient().stopContainerCmd(containerId).withTimeout(STOP_TIMEOUT_SECONDS).exec();+ } catch (NotFoundException | NotModifiedException alreadyEnded) {+ // Gone already (404), or exited and not yet removed (304).+ LOG.debugv("EMR Serverless job run {0} had ended before it was stopped", jobRunId);+ }+ }++ /**+ * Stops every run inside {@link #run}, so a reset or a shutdown orphans none: a running driver is+ * stopped and removed now, and one whose image is still being pulled as soon as its container is up.+ */+ @Override+ public void stopManagedContainers() {+ for (Launch launch : launches.values()) {+ launch.stopRequested = true;+ String containerId = launch.container.getAndSet(null);+ if (containerId != null) {+ lifecycleManager.stopAndRemove(containerId, null, STOP_TIMEOUT_SECONDS);+ }+ }+ }++ /**+ * Removes the drivers an earlier Floci process left running, which no thread follows any more: their+ * runs are {@code FAILED} once read, so nothing would ever stop them. The job run service calls this+ * before it hands a run to {@link #run}, so every driver of this deployment it finds is a leftover.+ */+ public void removeLeftoverDrivers() {+ String owner = ContainerStorageHelper.ownerIdentity(config);+ ContainerStorageHelper.LabelAliases aliases = ContainerStorageHelper.CONTAINER_LABEL_ALIASES;+ try {+ List<Container> containers = aliases.listByLabels(+ Map.of(ContainerStorageHelper.SERVICE_LABEL, SERVICE, ContainerStorageHelper.OWNER_LABEL, owner),+ filter -> lifecycleManager.getDockerClient().listContainersCmd()+ .withShowAll(true)+ .withLabelFilter(filter)+ .exec(),+ Container::getId);+ for (Container container : containers) {+ Map<String, String> labels = container.getLabels() != null ? container.getLabels() : Map.of();+ if (aliases.consistent(container.getId(), labels)+ && aliases.matches(labels, ContainerStorageHelper.OWNER_LABEL, owner)) {+ lifecycleManager.removeIfExists(container.getId());+ LOG.infov("Removed EMR Serverless driver {0}, left running by an earlier Floci process",+ container.getId());+ }+ }+ } catch (RuntimeException e) {+ LOG.warnv("Could not remove the EMR Serverless drivers an earlier Floci process left: {0}",+ e.getMessage());+ }+ }++ String image(String releaseLabel) {+ return config.services().emrserverless().imageTemplate()+ .orElse(releaseLabel.startsWith("emr-spark-") ? SPARK_RUNTIME_DEFAULT_IMAGE : DEFAULT_IMAGE)+ .formatted(releaseLabel);+ }++ /**+ * {@code spark-submit}'s arguments: local mode, S3A on Floci's S3 for {@code s3://} and {@code s3a://}+ * paths, then the run's {@code spark-defaults} and its own {@code sparkSubmitParameters}, which win+ * over what comes before them, then the entry point and its arguments. Before release 7.10.0+ * {@code s3://} is EMRFS, which ignores the endpoint setting, so it is switched to S3A. S3A writes+ * are committed through commit data the writer keeps in memory, without which EMR's committer finds+ * none and a write is silently lost.+ */+ static List<String> sparkSubmitArguments(Job job, String flociEndpoint) {+ List<String> arguments = new ArrayList<>(List.of("--master", "local[*]"));+ conf(arguments, "spark.hadoop.fs.s3a.endpoint", flociEndpoint);+ conf(arguments, "spark.hadoop.fs.s3a.endpoint.region", job.region());+ conf(arguments, "spark.hadoop.fs.s3a.path.style.access", "true");+ conf(arguments, "spark.hadoop.fs.s3a.committer.magic.track.commits.in.memory.enabled", "true");+ conf(arguments, "spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");+ conf(arguments, "spark.hadoop.fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3A");+ conf(arguments, "spark.hadoop.aws.glue.endpoint", flociEndpoint);+ conf(arguments, "spark.hadoop.aws.region", job.region());+ job.sparkDefaults().forEach((key, value) -> conf(arguments, key, value));+ JobDriver.SparkSubmit sparkSubmit = job.sparkSubmit();+ arguments.addAll(splitParameters(sparkSubmit.getSparkSubmitParameters()));+ arguments.add(sparkSubmit.getEntryPoint());+ if (sparkSubmit.getEntryPointArguments() != null) {+ arguments.addAll(sparkSubmit.getEntryPointArguments());+ }+ return arguments;+ }++ /**+ * {@code sparkSubmitParameters} as a POSIX shell would split them: on whitespace, except inside single+ * or double quotes. Outside quotes a backslash takes the next character as it is; inside double quotes+ * only a {@code "}, {@code \}, {@code $}, {@code `} or newline, and before any other character it is+ * kept. A backslash before a newline continues the line, and both go.+ */+ static List<String> splitParameters(String parameters) {+ List<String> words = new ArrayList<>();+ if (parameters == null) {+ return words;+ }+ StringBuilder word = new StringBuilder();+ boolean inWord = false;+ char quote = 0;+ for (int i = 0; i < parameters.length(); i++) {+ char c = parameters.charAt(i);+ if (quote == '\'') {+ if (c == '\'') {+ quote = 0;+ } else {+ word.append(c);+ }+ } else if (c == '\\' && i + 1 < parameters.length()+ && (quote == 0 || DOUBLE_QUOTE_ESCAPES.indexOf(parameters.charAt(i + 1)) >= 0)) {+ char escaped = parameters.charAt(++i);+ if (escaped != '\n') {+ word.append(escaped);+ inWord = true;+ }+ } else if (quote == '"') {+ if (c == '"') {+ quote = 0;+ } else {+ word.append(c);+ }+ } else if (c == '\'' || c == '"') {+ quote = c;+ inWord = true;+ } else if (Character.isWhitespace(c)) {+ if (inWord) {+ words.add(word.toString());+ word.setLength(0);+ inWord = false;+ }+ } else {+ word.append(c);+ inWord = true;+ }+ }+ if (inWord) {+ words.add(word.toString());+ }+ return words;+ }++ private static void conf(List<String> arguments, String key, String value) {+ arguments.add("--conf");+ arguments.add(key + "=" + value);+ }++ /**+ * The driver's exit code, or null once the run's timeout has passed with the driver still running,+ * which {@code progress} hears before the driver is stopped. Each look at the container is one short+ * control-plane call, so a run holds no Docker connection while its driver runs.+ */+ private Integer awaitExit(String containerId, Job job, Progress progress) throws InterruptedException {+ Instant deadline = job.timeout().isZero() ? null : clock.instant().plus(job.timeout());+ while (true) {+ InspectContainerResponse.ContainerState state =+ lifecycleManager.getDockerClient().inspectContainerCmd(containerId).exec().getState();+ if (!Boolean.TRUE.equals(state.getRunning())) {+ Long exitCode = state.getExitCodeLong();+ return exitCode != null ? exitCode.intValue() : -1;+ }+ if (deadline != null && !clock.instant().isBefore(deadline)) {+ LOG.infov("EMR Serverless job run {0} passed its execution timeout of {1}", job.jobRunId(),+ job.timeout());+ progress.timedOut();+ return null;+ }+ Thread.sleep(pollInterval.toMillis());+ }+ }++ /**+ * A run inside {@link #run}: whether it is to stop, and its driver's container once it has one, which+ * whoever stops the run first claims, run() itself or Floci's teardown.+ */+ private static final class Launch {+ final AtomicReference<String> container = new AtomicReference<>();+ volatile boolean stopRequested;+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.javanew file mode 100644index 0000000000..45e2543afd--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/SparkJobRunner.java@@ -0,0 +1,73 @@+package io.github.hectorvent.floci.services.emrserverless;++import io.github.hectorvent.floci.services.emrserverless.model.JobDriver;++import java.time.Duration;+import java.util.Map;++/**+ * Runs a Spark job run to its end, in the {@code docker} runner mode. The job run service holds the+ * run's state and calls this off the request thread; {@link EmrServerlessSparkRunner} is the Docker+ * implementation, and a test supplies its own.+ */+interface SparkJobRunner {++ /**+ * Runs {@code job} and returns how it ended, telling {@code progress} once its Spark driver runs and+ * when it passes its timeout. Blocks until then. A run {@code progress} reports as cancelled before+ * its driver runs never starts it.+ */+ Outcome run(Job job, Progress progress);++ /**+ * Stops the run's driver, now or as soon as it starts, so {@link #run} returns {@link Outcome#halted};+ * a run {@link #run} has already returned from is left as it is.+ */+ void stop(String jobRunId);++ /** What a run tells the job run service, and asks it, while it goes. */+ interface Progress {++ /** The run's Spark driver runs. */+ void started();++ /** The run has passed its timeout, and its driver is about to be stopped. */+ void timedOut();++ /** Whether the run was cancelled, or is gone, so that its driver must not start. */+ boolean cancelled();+ }++ /**+ * What a run needs: its identity, the application's release, the run's execution role, its Spark+ * submission, the {@code spark-defaults} properties that apply to it, and how long it may run+ * ({@link Duration#ZERO} for no limit).+ */+ record Job(String jobRunId, String applicationId, String accountId, String region, String releaseLabel,+ String executionRole, JobDriver.SparkSubmit sparkSubmit, Map<String, String> sparkDefaults,+ Duration timeout) {+ }++ /**+ * How a run ended: its driver's exit code, or that it ran out of time, was stopped, or could not+ * start ({@code failure} says why).+ */+ record Outcome(Integer exitCode, boolean timedOut, boolean stopped, String failure) {++ static Outcome exitedWith(int exitCode) {+ return new Outcome(exitCode, false, false, null);+ }++ static Outcome outOfTime() {+ return new Outcome(null, true, false, null);+ }++ static Outcome halted() {+ return new Outcome(null, false, true, null);+ }++ static Outcome couldNotStart(String failure) {+ return new Outcome(null, false, false, failure);+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/emrserverless/model/JobRunRecord.java b/src/main/java/io/github/hectorvent/floci/services/emrserverless/model/JobRunRecord.javaindex e55acce436..9f6ecda61d 100644--- a/src/main/java/io/github/hectorvent/floci/services/emrserverless/model/JobRunRecord.java+++ b/src/main/java/io/github/hectorvent/floci/services/emrserverless/model/JobRunRecord.java@@ -6,7 +6,8 @@ import io.quarkus.runtime.annotations.RegisterForReflection;/*** A stored job run: the run as GetJobRun reports it, plus what only Floci needs. The client token* and the request it came with decide whether a repeated StartJobRun is a retry or a conflict; the- * application type fills ListJobRuns' {@code type}.+ * application type fills ListJobRuns' {@code type}. A container run is one the {@code docker} runner+ * mode runs, whose state its container decides.*/@RegisterForReflectionpublic class JobRunRecord {@@ -14,6 +15,7 @@ public class JobRunRecord {private String clientToken;private JsonNode request;private String applicationType;+ private boolean containerRun;public JobRun getJobRun() { return jobRun; }public void setJobRun(JobRun jobRun) { this.jobRun = jobRun; }@@ -23,4 +25,6 @@ public class JobRunRecord {public void setRequest(JsonNode request) { this.request = request; }public String getApplicationType() { return applicationType; }public void setApplicationType(String applicationType) { this.applicationType = applicationType; }+ public boolean isContainerRun() { return containerRun; }+ public void setContainerRun(boolean containerRun) { this.containerRun = containerRun; }}diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex ddc18f666a..d95b3fe489 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -972,6 +972,12 @@ floci:emrserverless:enabled: truejob-run-duration-seconds: 0 # 0 = BATCH job runs succeed as soon as they start+ runner-mode: immediate # docker = run each Spark job run's spark-submit in a container+ # docker-network: floci-network+ # image-template: "registry.internal/emr-serverless/spark/%s:20250425" # %s = the release label+ # Remove the drivers a previous run of this deployment left on the daemon before the first+ # job run starts. Scoped to this deployment's own containers.+ reconcile-containers-on-startup: truessoadmin:enabled: truessooidc:diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerTeardownTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerTeardownTest.javaindex 40ba5c9363..e34a245665 100644--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerTeardownTest.java+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerTeardownTest.java@@ -3,8 +3,10 @@ package io.github.hectorvent.floci.core.common.docker;import com.github.dockerjava.api.DockerClient;import com.github.dockerjava.api.command.RemoveContainerCmd;import com.github.dockerjava.api.command.StopContainerCmd;+import com.github.dockerjava.api.exception.NotModifiedException;import org.junit.jupiter.api.Test;+import java.io.Closeable;import java.util.ArrayList;import java.util.List;import java.util.concurrent.CountDownLatch;@@ -15,6 +17,7 @@ import java.util.concurrent.atomic.AtomicBoolean;import java.util.concurrent.atomic.AtomicReference;import static io.github.hectorvent.floci.testing.ThreadStates.awaitBlockedOrDone;+import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertFalse;import static org.junit.jupiter.api.Assertions.assertNull;import static org.junit.jupiter.api.Assertions.assertThrows;@@ -142,4 +145,27 @@ class ContainerLifecycleManagerTeardownTest {}}+ @Test+ void aContainerThatHadAlreadyExitedCountsAsStopped() throws Exception {+ DockerClient docker = mock(DockerClient.class);+ ContainerLifecycleManager manager = new ContainerLifecycleManager(docker, null, null, null);+ StopContainerCmd stop = mock(StopContainerCmd.class);+ when(docker.stopContainerCmd(anyString())).thenReturn(stop);+ when(stop.withTimeout(anyInt())).thenReturn(stop);+ // Docker answers a stop of an exited container with HTTP 304.+ when(stop.exec()).thenThrow(new NotModifiedException("Container already stopped"));+ RemoveContainerCmd remove = mock(RemoveContainerCmd.class);+ when(docker.removeContainerCmd(anyString())).thenReturn(remove);+ when(remove.withForce(true)).thenReturn(remove);+ when(remove.exec()).thenThrow(new IllegalStateException("daemon refused removal"));+ Closeable logStream = mock(Closeable.class);++ manager.stopAndRemove("exited", logStream, 0);++ verify(logStream).close();+ IllegalStateException failure = assertThrows(IllegalStateException.class,+ () -> manager.stopAndRemoveStrict("exited-strict", null));+ assertEquals(0, failure.getSuppressed().length);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.javaindex 32a88d5f96..99ed0117bf 100644--- a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessJobRunServiceTest.java@@ -6,10 +6,12 @@ import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.core.common.RegionResolver;import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.StorageFactory;import io.github.hectorvent.floci.services.emrserverless.model.CreateApplicationRequest;import io.github.hectorvent.floci.services.emrserverless.model.JobDriver;import io.github.hectorvent.floci.services.emrserverless.model.JobRun;import io.github.hectorvent.floci.services.emrserverless.model.JobRunRecord;+import io.github.hectorvent.floci.services.emrserverless.model.RuntimeConfiguration;import io.github.hectorvent.floci.services.emrserverless.model.StartJobRunRequest;import io.github.hectorvent.floci.services.iam.IamService;import io.github.hectorvent.floci.testing.InMemoryStorageFactory;@@ -19,15 +21,22 @@ import org.junit.jupiter.api.Test;import java.time.Duration;import java.time.Instant;+import java.util.ArrayList;+import java.util.HashSet;+import java.util.List;import java.util.Map;import java.util.Optional;+import java.util.Set;import java.util.UUID;import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertNull;import static org.junit.jupiter.api.Assertions.assertThrows;import static org.mockito.ArgumentMatchers.any;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.times;+import static org.mockito.Mockito.verify;import static org.mockito.Mockito.when;class EmrServerlessJobRunServiceTest {@@ -42,6 +51,8 @@ class EmrServerlessJobRunServiceTest {private AccountAwareStorageBackend<JobRunRecord> store;private MutableClock clock;private String applicationId;+ // The docker runner mode's thread pool, run by hand: a container run's thread is the test's.+ private final List<Runnable> tasks = new ArrayList<>();@BeforeEachvoid setUp() {@@ -203,4 +214,273 @@ class EmrServerlessJobRunServiceTest {() -> EmrServerlessJobRunService.parseTimestamp("yesterday", "createdAtAfter"));assertEquals("ValidationException", invalid.getErrorCode());}++ @Test+ void aContainerRunIsScheduledUntilItsDriverRunsThenSucceedsWhenItExitsZero() {+ FakeRunner runner = new FakeRunner();+ EmrServerlessJobRunService service = dockerService(runner);+ JobRun submitted = service.startJobRun(applicationId, request(null, null));+ String jobRunId = submitted.getJobRunId();+ List<JobRun> whileRunning = new ArrayList<>();+ runner.whileRunning = () -> {+ whileRunning.add(service.getJobRun(applicationId, jobRunId, null));+ clock.advance(Duration.ofSeconds(42));+ };++ assertEquals("SCHEDULED", service.getJobRun(applicationId, jobRunId, null).getState());+ assertNull(service.getJobRun(applicationId, jobRunId, null).getStartedAt());+ clock.advance(Duration.ofSeconds(3));+ runNext();++ assertEquals("RUNNING", whileRunning.get(0).getState());+ assertEquals(3, Duration.between(submitted.getCreatedAt(), whileRunning.get(0).getStartedAt()).toSeconds());+ JobRun finished = service.getJobRun(applicationId, jobRunId, null);+ assertEquals("SUCCESS", finished.getState());+ assertEquals(42, Duration.between(finished.getStartedAt(), finished.getEndedAt()).toSeconds());+ assertEquals(42, finished.getTotalExecutionDurationSeconds());+ }++ @Test+ void theRunnerGetsTheRunsAccountReleaseRoleAndBatchTimeout() {+ FakeRunner runner = new FakeRunner();+ EmrServerlessJobRunService service = dockerService(runner);+ String batch = service.startJobRun(applicationId, request(null, 30L)).getJobRunId();+ service.startJobRun(applicationId, request("STREAMING", 30L));+ runNext();+ runNext();++ SparkJobRunner.Job job = runner.jobs.get(0);+ assertEquals(batch, job.jobRunId());+ assertEquals(applicationId, job.applicationId());+ assertEquals(ACCOUNT_ID, job.accountId());+ assertEquals(REGION, job.region());+ assertEquals("emr-7.5.0", job.releaseLabel());+ assertEquals(ROLE, job.executionRole());+ assertEquals("s3://bucket/jobs/app.jar", job.sparkSubmit().getEntryPoint());+ assertEquals(Duration.ofMinutes(30), job.timeout());+ assertEquals(Duration.ZERO, runner.jobs.get(1).timeout());+ }++ @Test+ void aRunsSparkDefaultsReplaceTheApplicationsUntilEmrSpark810MergesThem() {+ FakeRunner runner = new FakeRunner();+ EmrServerlessJobRunService service = dockerService(runner);++ for (String release : List.of("emr-7.10.0", "emr-spark-8.0.0", "emr-spark-8.1.0")) {+ String configured = applicationWithSparkDefaults(release);+ StartJobRunRequest overriding = request(null, null);+ overriding.setConfigurationOverrides(Map.of("applicationConfiguration", List.of(Map.of(+ "classification", "spark-defaults", "properties", Map.of("spark.executor.cores", "2")))));+ service.startJobRun(configured, request(null, null));+ service.startJobRun(configured, overriding);+ runNext();+ runNext();+ }++ Map<String, String> fromApp = Map.of("spark.executor.cores", "4", "spark.app.name", "from-app");+ List<Map<String, String>> sparkDefaults = runner.jobs.stream().map(SparkJobRunner.Job::sparkDefaults).toList();+ assertEquals(List.of(fromApp, Map.of("spark.executor.cores", "2"), fromApp, Map.of("spark.executor.cores", "2"),+ fromApp, Map.of("spark.executor.cores", "2", "spark.app.name", "from-app")), sparkDefaults);+ assertEquals("emr-spark-8.1.0", runner.jobs.get(5).releaseLabel());+ }++ private String applicationWithSparkDefaults(String releaseLabel) {+ RuntimeConfiguration defaults = new RuntimeConfiguration();+ defaults.setMember("classification", "spark-defaults");+ defaults.setMember("properties", Map.of("spark.executor.cores", "4", "spark.app.name", "from-app"));+ CreateApplicationRequest create = new CreateApplicationRequest();+ create.setName("configured-" + releaseLabel);+ create.setReleaseLabel(releaseLabel);+ create.setType("SPARK");+ create.setClientToken(UUID.randomUUID().toString());+ create.setRuntimeConfiguration(List.of(defaults));+ return applications.createApplication(create).getApplicationId();+ }++ @Test+ void aDriverThatExitsNonZeroOrNeverStartsFailsTheRun() {+ FakeRunner runner = new FakeRunner();+ EmrServerlessJobRunService service = dockerService(runner);+ String exited = service.startJobRun(applicationId, request(null, null)).getJobRunId();+ String unstarted = service.startJobRun(applicationId, request(null, null)).getJobRunId();++ runner.outcome = SparkJobRunner.Outcome.exitedWith(1);+ runNext();+ runner.outcome = SparkJobRunner.Outcome.couldNotStart("no such image");+ runNext();++ JobRun failed = service.getJobRun(applicationId, exited, null);+ assertEquals("FAILED", failed.getState());+ assertEquals("Job run failed: the Spark driver exited with code 1.", failed.getStateDetails());+ assertEquals("FAILED", service.getJobRun(applicationId, unstarted, null).getState());+ assertEquals("Job run could not start: no such image",+ service.getJobRun(applicationId, unstarted, null).getStateDetails());+ }++ @Test+ void cancellingARunningContainerRunStopsItsDriver() {+ FakeRunner runner = new FakeRunner();+ EmrServerlessJobRunService service = dockerService(runner);+ String jobRunId = service.startJobRun(applicationId, request(null, null)).getJobRunId();+ List<String> whileStopping = new ArrayList<>();+ runner.whileRunning = () -> {+ clock.advance(Duration.ofSeconds(10));+ service.cancelJobRun(applicationId, jobRunId);+ whileStopping.add(service.getJobRun(applicationId, jobRunId, null).getState());+ runNext();+ };++ runNext();++ assertEquals(List.of("CANCELLING"), whileStopping);+ assertEquals(Set.of(jobRunId), runner.stopped);+ JobRun cancelled = service.getJobRun(applicationId, jobRunId, null);+ assertEquals("CANCELLED", cancelled.getState());+ assertEquals(10, cancelled.getTotalExecutionDurationSeconds());+ }++ @Test+ void aRunCancelledBeforeItsDriverRunsNeverStarts() {+ FakeRunner runner = new FakeRunner();+ EmrServerlessJobRunService service = dockerService(runner);+ String jobRunId = service.startJobRun(applicationId, request(null, null)).getJobRunId();++ service.cancelJobRun(applicationId, jobRunId);+ tasks.remove(1).run();+ runNext();++ JobRun cancelled = service.getJobRun(applicationId, jobRunId, null);+ assertEquals("CANCELLED", cancelled.getState());+ assertNull(cancelled.getStartedAt());+ assertEquals(0, cancelled.getTotalExecutionDurationSeconds());+ assertEquals(List.of(), runner.driversStarted);+ }++ @Test+ void aContainerRunPastItsTimeoutIsCancelled() {+ FakeRunner runner = new FakeRunner();+ runner.outcome = SparkJobRunner.Outcome.outOfTime();+ EmrServerlessJobRunService service = dockerService(runner);+ String jobRunId = service.startJobRun(applicationId, request(null, 5L)).getJobRunId();+ List<String> whileStopping = new ArrayList<>();+ runner.whileTimingOut = () -> whileStopping.add(service.getJobRun(applicationId, jobRunId, null).getState());++ runNext();++ assertEquals(List.of("CANCELLING"), whileStopping);+ assertEquals("CANCELLED", service.getJobRun(applicationId, jobRunId, null).getState());+ }++ @Test+ void aHiveRunFailsWithoutRunning() {+ CreateApplicationRequest create = new CreateApplicationRequest();+ create.setName("hive");+ create.setReleaseLabel("emr-7.5.0");+ create.setType("HIVE");+ create.setClientToken(UUID.randomUUID().toString());+ String hive = applications.createApplication(create).getApplicationId();+ JobDriver.Hive query = new JobDriver.Hive();+ query.setQuery("s3://bucket/queries/report.sql");+ StartJobRunRequest request = request(null, null);+ request.getJobDriver().setSparkSubmit(null);+ request.getJobDriver().setHive(query);++ JobRun run = dockerService(new FakeRunner()).startJobRun(hive, request);++ assertEquals("FAILED", run.getState());+ assertEquals(EmrServerlessJobRunService.HIVE_NOT_RUN, run.getStateDetails());+ assertEquals(List.of(), tasks);+ }++ @Test+ void aContainerRunNothingRunsAnyMoreFails() {+ FakeRunner runner = new FakeRunner();+ String leftByRestart = dockerService(runner).startJobRun(applicationId, request(null, null)).getJobRunId();+ EmrServerlessJobRunService service = dockerService(runner);+ String stoppedByShutdown = service.startJobRun(applicationId, request(null, null)).getJobRunId();+ runner.whileRunning = () -> runner.stop(stoppedByShutdown);+ tasks.remove(1).run();++ for (String jobRunId : List.of(leftByRestart, stoppedByShutdown)) {+ JobRun run = service.getJobRun(applicationId, jobRunId, null);+ assertEquals("FAILED", run.getState());+ assertEquals(EmrServerlessJobRunService.INTERRUPTED, run.getStateDetails());+ }+ }++ private EmrServerlessJobRunService dockerService(SparkJobRunner runner) {+ return new EmrServerlessJobRunService(store, applications, iamService, new RegionResolver(REGION, ACCOUNT_ID),+ null, objectMapper, 0, clock, runner, tasks::add);+ }++ private void runNext() {+ tasks.remove(0).run();+ }++ @Test+ void aRunsThreadWritesNothingOnceAResetHasBegun() {+ FakeRunner runner = new FakeRunner();+ EmrServerlessJobRunService service = dockerService(runner);+ String beforeReset = service.startJobRun(applicationId, request(null, null)).getJobRunId();+ // The reset's teardown stops the driver while it runs; beforeReset() comes before the storage wipe.+ runner.whileRunning = service::beforeReset;++ runNext();++ assertEquals("RUNNING", store.get(beforeReset).orElseThrow().getJobRun().getState());+ runner.whileRunning = () -> { };+ String afterReset = service.startJobRun(applicationId, request(null, null)).getJobRunId();+ runNext();+ assertEquals("SUCCESS", store.get(afterReset).orElseThrow().getJobRun().getState());+ }++ @Test+ void leftoverDriversAreRemovedWhenTheServiceStartsUnlessReconcilingIsOff() {+ for (boolean reconcile : List.of(true, false)) {+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.services().emrserverless().runnerMode()).thenReturn("docker");+ when(config.services().emrserverless().reconcileContainersOnStartup()).thenReturn(reconcile);+ EmrServerlessSparkRunner sparkRunner = mock(EmrServerlessSparkRunner.class);++ new EmrServerlessJobRunService(mock(StorageFactory.class), applications, iamService,+ new RegionResolver(REGION, ACCOUNT_ID), null, objectMapper, config, clock, sparkRunner).shutdown();++ verify(sparkRunner, times(reconcile ? 1 : 0)).removeLeftoverDrivers();+ }+ }++ /**+ * Runs a container run on the test's thread: its driver starts, does what the test asks, and ends+ * with {@code outcome}, or stopped once {@link #stop} has been called for it.+ */+ private static final class FakeRunner implements SparkJobRunner {+ final List<Job> jobs = new ArrayList<>();+ final Set<String> stopped = new HashSet<>();+ final List<String> driversStarted = new ArrayList<>();+ Outcome outcome = Outcome.exitedWith(0);+ Runnable whileRunning = () -> { };+ Runnable whileTimingOut = () -> { };++ @Override+ public Outcome run(Job job, Progress progress) {+ jobs.add(job);+ // As the Docker runner, a stop counts only for a run in progress, so a cancel before then is read here.+ if (progress.cancelled()) {+ return Outcome.halted();+ }+ driversStarted.add(job.jobRunId());+ progress.started();+ whileRunning.run();+ if (outcome.timedOut()) {+ progress.timedOut();+ whileTimingOut.run();+ }+ return stopped.contains(job.jobRunId()) ? Outcome.halted() : outcome;+ }++ @Override+ public void stop(String jobRunId) {+ stopped.add(jobRunId);+ }+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.javanew file mode 100644index 0000000000..792b6ee239--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/emrserverless/EmrServerlessSparkRunnerTest.java@@ -0,0 +1,242 @@+package io.github.hectorvent.floci.services.emrserverless;++import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.exception.NotModifiedException;+import com.github.dockerjava.api.model.Container;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;+import io.github.hectorvent.floci.services.emrserverless.model.JobDriver;+import io.github.hectorvent.floci.services.lambda.launcher.LambdaExecutionRoleCredentials;+import io.github.hectorvent.floci.testing.MutableClock;+import org.junit.jupiter.api.Test;+import org.mockito.InOrder;++import java.time.Duration;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyMap;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.atLeastOnce;+import static org.mockito.Mockito.doAnswer;+import static org.mockito.Mockito.inOrder;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++class EmrServerlessSparkRunnerTest {++ private static final String ENDPOINT = "http://floci:4566";++ private final EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ private final ContainerLifecycleManager lifecycle = mock(ContainerLifecycleManager.class);+ private final DockerClient docker = mock(DockerClient.class, RETURNS_DEEP_STUBS);+ private final SparkJobRunner.Progress progress = mock(SparkJobRunner.Progress.class);++ @Test+ void sparkSubmitRunsLocallyOnFlocisS3ThenTheRunsOwnSettings() {+ JobDriver.SparkSubmit sparkSubmit = new JobDriver.SparkSubmit();+ sparkSubmit.setEntryPoint("s3://bucket/jobs/report.py");+ sparkSubmit.setEntryPointArguments(List.of("s3://bucket/input/", "s3://bucket/output/"));+ sparkSubmit.setSparkSubmitParameters("--conf spark.executor.cores=2 --py-files s3://bucket/jobs/lib.zip");+ Map<String, String> defaults = new LinkedHashMap<>();+ defaults.put("spark.executor.cores", "4");+ defaults.put("spark.sql.shuffle.partitions", "8");++ List<String> arguments = EmrServerlessSparkRunner.sparkSubmitArguments(job(sparkSubmit, defaults), ENDPOINT);++ assertEquals(List.of("--master", "local[*]",+ "--conf", "spark.hadoop.fs.s3a.endpoint=http://floci:4566",+ "--conf", "spark.hadoop.fs.s3a.endpoint.region=eu-west-1",+ "--conf", "spark.hadoop.fs.s3a.path.style.access=true",+ "--conf", "spark.hadoop.fs.s3a.committer.magic.track.commits.in.memory.enabled=true",+ "--conf", "spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem",+ "--conf", "spark.hadoop.fs.AbstractFileSystem.s3.impl=org.apache.hadoop.fs.s3a.S3A",+ "--conf", "spark.hadoop.aws.glue.endpoint=http://floci:4566",+ "--conf", "spark.hadoop.aws.region=eu-west-1",+ "--conf", "spark.executor.cores=4",+ "--conf", "spark.sql.shuffle.partitions=8",+ "--conf", "spark.executor.cores=2",+ "--py-files", "s3://bucket/jobs/lib.zip",+ "s3://bucket/jobs/report.py", "s3://bucket/input/", "s3://bucket/output/"), arguments);+ }++ @Test+ void anEntryPointWithoutParametersOrArgumentsFollowsFlocisSettings() {+ JobDriver.SparkSubmit sparkSubmit = new JobDriver.SparkSubmit();+ sparkSubmit.setEntryPoint("s3://bucket/jobs/app.jar");++ List<String> arguments = EmrServerlessSparkRunner.sparkSubmitArguments(job(sparkSubmit, Map.of()), ENDPOINT);++ assertEquals("--conf", arguments.get(arguments.size() - 3));+ assertEquals("spark.hadoop.aws.region=eu-west-1", arguments.get(arguments.size() - 2));+ assertEquals("s3://bucket/jobs/app.jar", arguments.get(arguments.size() - 1));+ }++ @Test+ void sparkSubmitParametersSplitAsAShellWould() {+ assertEquals(List.of("--conf", "spark.driver.extraJavaOptions=-Da=1 -Db=2", "--conf", "x=it's", "a b", ""),+ EmrServerlessSparkRunner.splitParameters(+ " --conf \"spark.driver.extraJavaOptions=-Da=1 -Db=2\"\t--conf 'x=it'\\''s' a\\ b '' "));+ // Inside double quotes a backslash escapes only ", \, $, ` and a newline, and is kept before anything else.+ assertEquals(List.of("--conf", "spark.driver.extraJavaOptions=-Dpattern=\\d+ -Dq=\"x\" -Dp=\\ $HOME", "onetwo"),+ EmrServerlessSparkRunner.splitParameters(+ "--conf \"spark.driver.extraJavaOptions=-Dpattern=\\d+ -Dq=\\\"x\\\" -Dp=\\\\ \\$HOME\" one\\\ntwo"));+ assertEquals(List.of(), EmrServerlessSparkRunner.splitParameters(null));+ assertEquals(List.of(), EmrServerlessSparkRunner.splitParameters(" "));+ }++ @Test+ void theImageIsTheReleasesPublicImageUnlessATemplateNamesAnother() {+ EmrServerlessSparkRunner runner = runner();++ assertEquals("public.ecr.aws/emr-serverless/spark/emr-7.5.0:latest", runner.image("emr-7.5.0"));+ assertEquals("public.ecr.aws/emr-serverless/spark4/emr-spark-8.1.0:latest", runner.image("emr-spark-8.1.0"));+ when(config.services().emrserverless().imageTemplate())+ .thenReturn(Optional.of("registry.internal/emr-serverless/spark/%s:20250425"));+ assertEquals("registry.internal/emr-serverless/spark/emr-7.5.0:20250425", runner.image("emr-7.5.0"));+ }++ @Test+ void aCancelThatFindsTheDriverAlreadyExitedStillStopsTheRun() {+ EmrServerlessSparkRunner runner = runner();+ // Docker answers a stop of an exited container with HTTP 304.+ when(docker.stopContainerCmd("driver").withTimeout(0).exec())+ .thenThrow(new NotModifiedException("Container already stopped"));+ SparkJobRunner.Job job = job(Duration.ZERO);+ doAnswer(started -> {+ runner.stop(job.jobRunId());+ return null;+ }).when(progress).started();++ SparkJobRunner.Outcome outcome = runner.run(job, progress);++ assertTrue(outcome.stopped());+ verify(lifecycle).stopAndRemove("driver", null, 0);+ }++ @Test+ void aResetWhileTheImageIsPulledStopsTheRunOnceItsContainerIsUp() {+ EmrServerlessSparkRunner runner = runner();+ when(lifecycle.createAndStart(any())).thenAnswer(pulling -> {+ runner.stopManagedContainers();+ return new ContainerLifecycleManager.ContainerInfo("driver", Map.of());+ });++ SparkJobRunner.Outcome outcome = runner.run(job(Duration.ZERO), progress);++ assertTrue(outcome.stopped());+ verify(progress, never()).started();+ verify(lifecycle).stopAndRemove("driver", null, 0);+ }++ @Test+ void aRunCancelledWhileItsImageIsPulledNeverStartsItsDriver() {+ EmrServerlessSparkRunner runner = runner();+ // Not cancelled when the run begins, cancelled by the time its container is up.+ when(progress.cancelled()).thenReturn(false, true);++ SparkJobRunner.Outcome outcome = runner.run(job(Duration.ZERO), progress);++ assertTrue(outcome.stopped());+ verify(progress, never()).started();+ verify(lifecycle).stopAndRemove("driver", null, 0);+ }++ @Test+ void aStopAfterTheRunHasEndedIsDropped() {+ EmrServerlessSparkRunner runner = runner();+ SparkJobRunner.Job job = job(Duration.ZERO);+ assertEquals(0, runner.run(job, progress).exitCode());++ runner.stop(job.jobRunId());++ // A stop kept past its run would halt the next run of this id before it started.+ assertEquals(0, runner.run(job, progress).exitCode());+ verify(docker, never()).stopContainerCmd(anyString());+ }++ @Test+ void aRunPastItsTimeoutIsToldSoBeforeItsDriverIsRemoved() {+ EmrServerlessSparkRunner runner = runner();+ when(docker.inspectContainerCmd("driver").exec().getState().getRunning()).thenReturn(true);++ SparkJobRunner.Outcome outcome = runner.run(job(Duration.ofMillis(5)), progress);++ assertTrue(outcome.timedOut());+ InOrder order = inOrder(progress, lifecycle);+ order.verify(progress).timedOut();+ order.verify(lifecycle).stopAndRemove("driver", null, 0);+ }++ @Test+ void theDriverIsWatchedWithoutHoldingADockerConnection() {+ EmrServerlessSparkRunner runner = runner();++ runner.run(job(Duration.ZERO), progress);++ // A look at the container is one short call; no wait or other stream is held while the driver runs.+ verify(docker, atLeastOnce()).inspectContainerCmd("driver");+ verify(docker, never()).waitContainerCmd(anyString());+ }++ @Test+ void driversAnEarlierProcessOfThisDeploymentLeftAreRemoved() {+ EmrServerlessSparkRunner runner = runner();+ when(config.port()).thenReturn(4566);+ List<Container> drivers = List.of(driver("left-behind", "4566"), driver("another-deployments", "4567"));+ when(docker.listContainersCmd().withShowAll(true).withLabelFilter(anyMap()).exec()).thenReturn(drivers);++ runner.removeLeftoverDrivers();++ verify(lifecycle).removeIfExists("left-behind");+ verify(lifecycle, never()).removeIfExists("another-deployments");+ }++ // A runner whose driver container, "driver", starts at once and exits with code 0 unless a test says otherwise.+ private EmrServerlessSparkRunner runner() {+ when(config.services().emrserverless().imageTemplate()).thenReturn(Optional.empty());+ when(lifecycle.getDockerClient()).thenReturn(docker);+ when(lifecycle.createAndStart(any())).thenReturn(new ContainerLifecycleManager.ContainerInfo("driver", Map.of()));+ when(docker.inspectContainerCmd("driver").exec().getState().getRunning()).thenReturn(false);+ when(docker.inspectContainerCmd("driver").exec().getState().getExitCodeLong()).thenReturn(0L);+ return new EmrServerlessSparkRunner(mock(ContainerBuilder.class, RETURNS_DEEP_STUBS), lifecycle,+ mock(ContainerLogStreamer.class), mock(LaunchedContainerAwsEnv.class),+ mock(LambdaExecutionRoleCredentials.class), config, new MutableClock(), Duration.ZERO);+ }++ private static Container driver(String id, String owner) {+ Container container = mock(Container.class);+ when(container.getId()).thenReturn(id);+ when(container.getLabels()).thenReturn(Map.of(ContainerStorageHelper.SERVICE_LABEL, "emr-serverless",+ ContainerStorageHelper.OWNER_LABEL, owner));+ return container;+ }++ private static SparkJobRunner.Job job(Duration timeout) {+ JobDriver.SparkSubmit sparkSubmit = new JobDriver.SparkSubmit();+ sparkSubmit.setEntryPoint("s3://bucket/jobs/app.jar");+ return job(sparkSubmit, Map.of(), timeout);+ }++ private static SparkJobRunner.Job job(JobDriver.SparkSubmit sparkSubmit, Map<String, String> sparkDefaults) {+ return job(sparkSubmit, sparkDefaults, Duration.ZERO);+ }++ private static SparkJobRunner.Job job(JobDriver.SparkSubmit sparkSubmit, Map<String, String> sparkDefaults,+ Duration timeout) {+ return new SparkJobRunner.Job("00f0000000000001", "00f0000000000000", "123456789012", "eu-west-1",+ "emr-7.5.0", "arn:aws:iam::123456789012:role/job", sparkSubmit, sparkDefaults, timeout);+ }+}diff --git a/src/test/resources/application.yml b/src/test/resources/application.ymlindex 642bc49535..b52da0422b 100644--- a/src/test/resources/application.yml+++ b/src/test/resources/application.yml@@ -501,6 +501,7 @@ floci:emrserverless:enabled: truejob-run-duration-seconds: 0 # 0 = BATCH job runs succeed as soon as they start+ runner-mode: immediatessoadmin:enabled: truessooidc: