Light, fluffy, and always free - The AWS Local Emulator alternative
feat(batch): support canceling and terminating jobs (#3922)
XuJian9c4cd8c9e13b
diff --git a/docs/services/batch.md b/docs/services/batch.mdindex 7b42eb0523..1c98a02a60 100644--- a/docs/services/batch.md+++ b/docs/services/batch.md@@ -21,6 +21,8 @@ Floci Batch implements the AWS Batch control plane for local integration tests.| `DeregisterJobDefinition` | `POST /v1/deregisterjobdefinition` | Mark a job definition revision inactive || `DescribeJobDefinitions` | `POST /v1/describejobdefinitions` | List job definitions by name, ARN, revision, and status || `SubmitJob` | `POST /v1/submitjob` | Submit a local Batch job |+| `CancelJob` | `POST /v1/canceljob` | Cancel a job that has not started running |+| `TerminateJob` | `POST /v1/terminatejob` | Terminate a nonterminal job and stop its running container or nodes || `DescribeJobs` | `POST /v1/describejobs` | Describe jobs by job ID || `ListJobs` | `POST /v1/listjobs` | List jobs by queue, status, AWS `filters`, and pagination |@@ -61,6 +63,19 @@ SUBMITTED -> PENDING -> RUNNABLE -> STARTING -> RUNNING -> SUCCEEDED|FAILEDDriving every immediate-mode job through `PENDING` is a local simplification for tests. AWS may skip that state when there are no dependency or capacity waits.+## Job Control++`CancelJob` accepts jobs in `SUBMITTED`, `PENDING`, or `RUNNABLE` and moves them to `FAILED` with+the requested reason. Jobs that already reached `STARTING` or `RUNNING` are left unchanged, matching+AWS behavior. `TerminateJob` moves any nonterminal job to `FAILED`; in Docker mode it also stops and+removes the running container. Both operations are idempotent for terminal jobs and require `jobId`+and `reason`.++For an array parent, job control cascades to its children. Cancellation affects only children that+have not started, while termination also stops running children. Terminating an MNP job stops every+running node container. The stored terminal reason wins over a concurrent container exit, so the+background runner cannot overwrite a canceled or terminated job with a later result.+### Array jobsSubmitting with `arrayProperties.size` (2-10,000) fans out one child job per index, sharing the@@ -69,8 +84,8 @@ its own Docker container in `docker` mode). The parent's `status` and `arrayPropare computed live from its children on every `DescribeJobs`/`ListJobs` call rather than stored andupdated incrementally, so there is nothing to race when many children finish concurrently. Aqueue-level `ListJobs` shows one entry per array job (the parent); pass `arrayJobId` to list its-children. `TerminateJob`/`CancelJob` cascade is not applicable since neither operation is-implemented (see Limitations).+children. `CancelJob` and `TerminateJob` on the parent cascade to the children using the job-control+rules above.### Multi-node parallel (MNP) jobs@@ -80,8 +95,9 @@ A job definition registered with `"type": "multinode"` and `nodeProperties` (`nu`AWS_BATCH_JOB_NODE_INDEX`, `AWS_BATCH_JOB_MAIN_NODE_INDEX`, and `AWS_BATCH_JOB_NUM_NODES` set. Thejob's status, retry decision, and `startedAt`/`stoppedAt`/`statusReason` are all determined solely bythe **main node** (matching AWS: a child node failing does not fail or retry the job). Every node-still runs to completion rather than being stopped early (see Limitations below for the two cases-this affects). `DescribeJobs`/queue-level `ListJobs` return one entry for the whole+normally runs to completion, while `TerminateJob` stops all running nodes. A natural main-node exit+does not yet preempt the remaining nodes (see Limitations). `DescribeJobs`/queue-level `ListJobs`+return one entry for the wholejob (with `nodeProperties`, no per-node `container`); pass `multiNodeJobId` to `ListJobs` for aone-entry-per-node breakdown with each node's own `container` and `nodeProperties.nodeIndex`.`containerOverrides` and array jobs are not supported on MNP job definitions; use `nodeOverrides`@@ -132,15 +148,12 @@ IAM roles, VPC fields, Fargate declarations, log configuration, storage, and res- No VPC/subnet/security-group simulation.- No AWS-faithful capacity scheduling.- No job dependencies (`dependsOn`).-- `CancelJob` and `TerminateJob` are not implemented, so there is no way to cancel/terminate an- array job's children or an MNP job's nodes either — this is the same pre-existing gap as for a- regular job, not something array/MNP jobs make worse.- EventBridge input transformers work through the existing EventBridge target input path; full Batch-specific input-transformer parity is not implemented.- An MNP job does not finish the instant its main node exits. AWS stops the job (and its remainingchild nodes) as soon as the main node exits; Floci instead waits for every node's container toexit before scoring the attempt, so a child that outlives the main node keeps the job `RUNNING`- longer than AWS would. Closing this needs a way for `BatchDockerRunner` to stop one node's- container from outside that node's own run loop, which does not exist yet.+ longer than AWS would. `TerminateJob` can stop the whole job, but main-node completion does not+ yet trigger that same preemption path automatically.**Known follow-ups on the array/MNP surface** (not required for `SubmitJob`/`DescribeJobs`/`ListJobs`correctness, deliberately deferred):@@ -153,7 +166,7 @@ correctness, deliberately deferred):definitions; there is no way to provision a `multinode` job definition via CloudFormation, onlyvia a direct `RegisterJobDefinition` call.- `ListJobs` with `arrayJobId` ignores the `filters` parameter (only `jobStatus` is honored for- children) — matches AWS's own note that filters don't apply to child jobs, but it means array+ children). This matches AWS's own note that filters don't apply to child jobs, but it means arraychildren can't be filtered by `JOB_NAME`/`JOB_DEFINITION`/etc.- `ListJobs` with `multiNodeJobId` has no pagination (`maxResults`/`nextToken`); it always returnsevery node in one response.diff --git a/src/main/java/io/github/hectorvent/floci/services/batch/BatchController.java b/src/main/java/io/github/hectorvent/floci/services/batch/BatchController.javaindex 509b8b6c6e..d057c8449a 100644--- a/src/main/java/io/github/hectorvent/floci/services/batch/BatchController.java+++ b/src/main/java/io/github/hectorvent/floci/services/batch/BatchController.java@@ -119,6 +119,20 @@ public class BatchController {Response.ok(service.submitJob(request, region)).build());}+ @POST+ @Path("/v1/canceljob")+ public Response cancelJob(@Context HttpHeaders headers, String body) {+ return handle(headers, body, (request, region) ->+ Response.ok(service.cancelJob(request)).build());+ }++ @POST+ @Path("/v1/terminatejob")+ public Response terminateJob(@Context HttpHeaders headers, String body) {+ return handle(headers, body, (request, region) ->+ Response.ok(service.terminateJob(request)).build());+ }+@POST@Path("/v1/describejobs")public Response describeJobs(@Context HttpHeaders headers, String body) {diff --git a/src/main/java/io/github/hectorvent/floci/services/batch/BatchDockerRunner.java b/src/main/java/io/github/hectorvent/floci/services/batch/BatchDockerRunner.javaindex 2157c27d9e..4fb502e960 100644--- a/src/main/java/io/github/hectorvent/floci/services/batch/BatchDockerRunner.java+++ b/src/main/java/io/github/hectorvent/floci/services/batch/BatchDockerRunner.java@@ -25,6 +25,7 @@ import java.time.Duration;import java.util.ArrayList;import java.util.List;import java.util.Map;+import java.util.Set;import java.util.concurrent.ConcurrentHashMap;@ApplicationScoped@@ -36,6 +37,7 @@ public class BatchDockerRunner implements ContainerTeardown {// Containers of jobs currently inside run(); drained on emulator shutdown so a// SIGTERM mid-job does not orphan the container.private final ConcurrentHashMap<String, String> inFlightContainers = new ConcurrentHashMap<>();+ private final Set<String> stopRequestedJobs = ConcurrentHashMap.newKeySet();private final ContainerBuilder containerBuilder;private final ContainerLifecycleManager lifecycleManager;@@ -88,6 +90,9 @@ public class BatchDockerRunner implements ContainerTeardown {String containerId = null;try {+ if (stopRequestedJobs.contains(job.getJobId())) {+ return stopped(startedAt, logStreamName);+ }if (image == null || image.isBlank()) {return failed(startedAt, logStreamName, missingImageMessage);}@@ -110,7 +115,15 @@ public class BatchDockerRunner implements ContainerTeardown {ContainerSpec spec = builder.build();containerId = lifecycleManager.createAndStart(spec).containerId();inFlightContainers.put(inFlightKey, containerId);+ if (stopRequestedJobs.contains(job.getJobId())) {+ releaseAndStop(inFlightKey, containerId, null);+ return stopped(startedAt, logStreamName);+ }logHandle = logStreamer.attach(containerId, LOG_GROUP, logStreamName, job.getRegion(), logSourceLabel);+ if (stopRequestedJobs.contains(job.getJobId())) {+ releaseAndStop(inFlightKey, containerId, logHandle);+ return stopped(startedAt, logStreamName);+ }Integer exitCode = waitForExit(containerId, timeout(job));long stoppedAt = System.currentTimeMillis();@@ -129,7 +142,30 @@ public class BatchDockerRunner implements ContainerTeardown {}}- // Whoever wins the map removal owns the stop — stopManagedContainers() may have+ public void requestStop(String jobId) {+ stopRequestedJobs.add(jobId);+ }++ public void stopJob(String jobId) {+ for (Map.Entry<String, String> entry : new ConcurrentHashMap<>(inFlightContainers).entrySet()) {+ if (!entry.getKey().equals(jobId) && !entry.getKey().startsWith(jobId + "#node")) {+ continue;+ }+ if (inFlightContainers.remove(entry.getKey(), entry.getValue())) {+ try {+ lifecycleManager.stopAndRemove(entry.getValue(), null);+ } catch (Exception e) {+ LOG.warnv("Failed to stop Batch container for job {0}: {1}", jobId, e.getMessage());+ }+ }+ }+ }++ public void clearStopRequest(String jobId) {+ stopRequestedJobs.remove(jobId);+ }++ // Whoever wins the map removal owns the stop. stopManagedContainers() may have// claimed the container first during shutdown, but the log stream is still ours.private void releaseAndStop(String jobId, String containerId, Closeable logHandle) {if (inFlightContainers.remove(jobId, containerId)) {@@ -165,6 +201,11 @@ public class BatchDockerRunner implements ContainerTeardown {return new BatchRunResult(1, reason, logStreamName, startedAt, System.currentTimeMillis(), false);}+ private BatchRunResult stopped(long startedAt, String logStreamName) {+ return new BatchRunResult(137, "Job terminated", logStreamName,+ startedAt, System.currentTimeMillis(), false);+ }+private List<String> buildEnvironment(BatchJob job, int attemptNumber) {List<String> env = baseEnvironment(job, attemptNumber);appendEnvironment(env, job.getResolvedEnvironment());diff --git a/src/main/java/io/github/hectorvent/floci/services/batch/BatchService.java b/src/main/java/io/github/hectorvent/floci/services/batch/BatchService.javaindex 79e0fa8d05..aae27817b7 100644--- a/src/main/java/io/github/hectorvent/floci/services/batch/BatchService.java+++ b/src/main/java/io/github/hectorvent/floci/services/batch/BatchService.java@@ -56,6 +56,7 @@ public class BatchService {private static final int DEFAULT_MAX_RESULTS = 1000;private static final int FILTERED_MAX_RESULTS = 100;private static final int MAX_TAGS = 50;+ private static final int MAX_JOB_CONTROL_REASON_LENGTH = 1024;private static final Pattern JOB_NAME_PATTERN = Pattern.compile("[A-Za-z0-9][A-Za-z0-9_-]{0,127}");private static final Set<String> SUPPORTED_JOB_DEFINITION_TYPES = Set.of("container", "multinode");private static final int MIN_ARRAY_SIZE = 2;@@ -64,6 +65,8 @@ public class BatchService {private static final Set<String> SUPPORTED_COMPUTE_ENVIRONMENT_STATES = Set.of("ENABLED", "DISABLED");private static final Set<String> SUPPORTED_LIST_FILTERS = Set.of("JOB_NAME", "JOB_DEFINITION", "BEFORE_CREATED_AT", "AFTER_CREATED_AT", "SHARE_IDENTIFIER");+ private static final Set<String> TERMINAL_JOB_STATUSES = Set.of(+ BatchStatus.SUCCEEDED.name(), BatchStatus.FAILED.name());private final StorageBackend<String, BatchJobDefinition> jobDefinitionStore;private final StorageBackend<String, BatchJobQueue> jobQueueStore;@@ -456,6 +459,71 @@ public class BatchService {return submitJobResponse(job);}+ public ObjectNode cancelJob(JsonNode request) {+ return controlJob(request, false);+ }++ public ObjectNode terminateJob(JsonNode request) {+ return controlJob(request, true);+ }++ private ObjectNode controlJob(JsonNode request, boolean terminate) {+ String jobId = requiredText(request, "jobId");+ String reason = requiredText(request, "reason");+ if (reason.length() > MAX_JOB_CONTROL_REASON_LENGTH) {+ throw client("reason must be at most " + MAX_JOB_CONTROL_REASON_LENGTH + " characters");+ }+ List<String> jobsToStop = new ArrayList<>();++ synchronized (this) {+ BatchJob job = getJob(jobId).orElseThrow(() -> client("Job not found: " + jobId));+ if (job.getArraySize() != null) {+ boolean changed = false;+ List<BatchJob> children = jobStore.scan(k -> true).stream()+ .filter(child -> jobId.equals(child.getArrayJobId()))+ .toList();+ for (BatchJob child : children) {+ boolean wasRunning = BatchStatus.RUNNING.name().equals(child.getStatus());+ if (controlSingleJob(child, reason, terminate)) {+ changed = true;+ if (terminate && wasRunning) {+ dockerRunner.requestStop(child.getJobId());+ jobsToStop.add(child.getJobId());+ }+ }+ }+ if (changed) {+ job.setStatusReason(reason);+ putJob(job);+ }+ } else {+ boolean wasRunning = BatchStatus.RUNNING.name().equals(job.getStatus());+ if (controlSingleJob(job, reason, terminate) && terminate && wasRunning) {+ dockerRunner.requestStop(job.getJobId());+ jobsToStop.add(job.getJobId());+ }+ }+ }++ jobsToStop.forEach(dockerRunner::stopJob);+ return objectMapper.createObjectNode();+ }++ private boolean controlSingleJob(BatchJob job, String reason, boolean terminate) {+ if (isTerminalStatus(job.getStatus())) {+ return false;+ }+ if (!terminate && (BatchStatus.STARTING.name().equals(job.getStatus())+ || BatchStatus.RUNNING.name().equals(job.getStatus()))) {+ return false;+ }+ job.setStatus(BatchStatus.FAILED.name());+ job.setStatusReason(reason);+ job.setStoppedAt(now());+ putJob(job);+ return true;+ }+private ObjectNode submitArrayJob(JsonNode request, String region, String jobName, BatchJobQueue queue,BatchJobDefinition def, Map<String, String> parameters, int size) {BatchJob parent = new BatchJob();@@ -800,13 +868,21 @@ public class BatchService {}int maxAttempts = maxAttempts(job);for (int attemptNumber = 1; attemptNumber <= maxAttempts; attemptNumber++) {- transition(accountId, jobId, BatchStatus.PENDING, null, false);+ if (!transition(accountId, jobId, BatchStatus.PENDING, null, false)) {+ return;+ }sleepQuietly();- transition(accountId, jobId, BatchStatus.RUNNABLE, null, false);+ if (!transition(accountId, jobId, BatchStatus.RUNNABLE, null, false)) {+ return;+ }sleepQuietly();- transition(accountId, jobId, BatchStatus.STARTING, null, false);+ if (!transition(accountId, jobId, BatchStatus.STARTING, null, false)) {+ return;+ }sleepQuietly();- transition(accountId, jobId, BatchStatus.RUNNING, null, true);+ if (!transition(accountId, jobId, BatchStatus.RUNNING, null, true)) {+ return;+ }BatchJob attemptJob = getJobForAccount(accountId, jobId).orElse(null);if (attemptJob == null) {return;@@ -821,6 +897,8 @@ public class BatchService {} catch (Exception e) {LOG.warnv("Batch job {0} failed while running: {1}", jobId, e.getMessage());failJob(accountId, jobId, e.getMessage());+ } finally {+ dockerRunner.clearStopRequest(jobId);}}@@ -840,6 +918,7 @@ public class BatchService {if (job == null) {return true;}+ boolean controlledTerminal = isTerminalStatus(job.getStatus());BatchAttemptContainer container = new BatchAttemptContainer();container.setExitCode(result.exitCode());container.setReason(result.reason());@@ -853,6 +932,10 @@ public class BatchService {job.getAttempts().add(attempt);job.setContainer(container);+ if (controlledTerminal) {+ putJobForAccount(accountId, job);+ return true;+ }if (result.exitCode() == 0) {job.setStoppedAt(result.stoppedAt());job.setStatus(BatchStatus.SUCCEEDED.name());@@ -881,13 +964,21 @@ public class BatchService {}int maxAttempts = maxAttempts(job);for (int attemptNumber = 1; attemptNumber <= maxAttempts; attemptNumber++) {- transition(accountId, jobId, BatchStatus.PENDING, null, false);+ if (!transition(accountId, jobId, BatchStatus.PENDING, null, false)) {+ return;+ }sleepQuietly();- transition(accountId, jobId, BatchStatus.RUNNABLE, null, false);+ if (!transition(accountId, jobId, BatchStatus.RUNNABLE, null, false)) {+ return;+ }sleepQuietly();- transition(accountId, jobId, BatchStatus.STARTING, null, false);+ if (!transition(accountId, jobId, BatchStatus.STARTING, null, false)) {+ return;+ }sleepQuietly();- transition(accountId, jobId, BatchStatus.RUNNING, null, true);+ if (!transition(accountId, jobId, BatchStatus.RUNNING, null, true)) {+ return;+ }BatchJob attemptJob = getJobForAccount(accountId, jobId).orElse(null);if (attemptJob == null) {return;@@ -900,6 +991,8 @@ public class BatchService {} catch (Exception e) {LOG.warnv("Batch multi-node job {0} failed while running: {1}", jobId, e.getMessage());failJob(accountId, jobId, e.getMessage());+ } finally {+ dockerRunner.clearStopRequest(jobId);}}@@ -947,6 +1040,7 @@ public class BatchService {if (job == null) {return true;}+ boolean controlledTerminal = isTerminalStatus(job.getStatus());List<BatchNodeExecution> executions = job.getNodeExecutions();BatchRunResult mainResult = null;for (int i = 0; i < executions.size(); i++) {@@ -970,6 +1064,11 @@ public class BatchService {attempt.setStatusReason(mainResult.reason());job.getAttempts().add(attempt);+ if (controlledTerminal) {+ putJobForAccount(accountId, job);+ return true;+ }+if (mainResult.exitCode() == 0) {job.setStoppedAt(mainResult.stoppedAt());job.setStatus(BatchStatus.SUCCEEDED.name());@@ -999,7 +1098,7 @@ public class BatchService {private synchronized void failJob(String accountId, String jobId, String reason) {BatchJob job = getJobForAccount(accountId, jobId).orElse(null);- if (job == null) {+ if (job == null || isTerminalStatus(job.getStatus())) {return;}job.setStatus(BatchStatus.FAILED.name());@@ -1008,10 +1107,11 @@ public class BatchService {putJobForAccount(accountId, job);}- private synchronized void transition(String accountId, String jobId, BatchStatus status, String reason, boolean started) {+ private synchronized boolean transition(String accountId, String jobId, BatchStatus status,+ String reason, boolean started) {BatchJob job = getJobForAccount(accountId, jobId).orElse(null);- if (job == null) {- return;+ if (job == null || isTerminalStatus(job.getStatus())) {+ return false;}job.setStatus(status.name());if (reason != null) {@@ -1021,6 +1121,11 @@ public class BatchService {job.setStartedAt(now());}putJobForAccount(accountId, job);+ return true;+ }++ private boolean isTerminalStatus(String status) {+ return TERMINAL_JOB_STATUSES.contains(status);}private void sleepQuietly() {@@ -1419,6 +1524,9 @@ public class BatchService {if (job.getArraySize() != null) {ArrayRollup rollup = arrayRollup(job.getJobId());summary.put("status", rollup.status());+ if (job.getStatusReason() != null) {+ summary.put("statusReason", job.getStatusReason());+ }if (rollup.startedAt() != null) {summary.put("startedAt", rollup.startedAt());}@@ -1512,11 +1620,13 @@ public class BatchService {for (BatchJob child : children) {statusSummary.merge(child.getStatus(), 1, Integer::sum);}+ boolean allTerminal = !children.isEmpty()+ && children.stream().allMatch(child -> isTerminalStatus(child.getStatus()));boolean anyFailed = children.stream().anyMatch(c -> BatchStatus.FAILED.name().equals(c.getStatus()));boolean allSucceeded = !children.isEmpty()&& children.stream().allMatch(c -> BatchStatus.SUCCEEDED.name().equals(c.getStatus()));String status;- if (anyFailed) {+ if (allTerminal && anyFailed) {status = BatchStatus.FAILED.name();} else if (allSucceeded) {status = BatchStatus.SUCCEEDED.name();@@ -1616,6 +1726,9 @@ public class BatchService {if (job.getArraySize() != null) {ArrayRollup rollup = arrayRollup(job.getJobId());detail.put("status", rollup.status());+ if (job.getStatusReason() != null) {+ detail.put("statusReason", job.getStatusReason());+ }if (rollup.startedAt() != null) {detail.put("startedAt", rollup.startedAt());}diff --git a/src/test/java/io/github/hectorvent/floci/services/batch/BatchDockerRunnerTest.java b/src/test/java/io/github/hectorvent/floci/services/batch/BatchDockerRunnerTest.javaindex c9421b4325..b4ea194609 100644--- a/src/test/java/io/github/hectorvent/floci/services/batch/BatchDockerRunnerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/batch/BatchDockerRunnerTest.java@@ -1,5 +1,6 @@package io.github.hectorvent.floci.services.batch;+import com.github.dockerjava.api.DockerClient;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.dns.EmbeddedDnsServer;import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;@@ -8,15 +9,28 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;import io.github.hectorvent.floci.core.common.docker.ContainerSpec;import io.github.hectorvent.floci.services.batch.model.BatchJob;-import com.github.dockerjava.api.DockerClient;+import io.github.hectorvent.floci.services.batch.model.BatchNodeExecution;+import io.github.hectorvent.floci.services.batch.model.BatchRunResult;import org.junit.jupiter.api.Test;import java.util.Map;import java.util.Optional;+import java.util.Set;+import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.atomic.AtomicBoolean;+import java.util.concurrent.atomic.AtomicInteger;+import java.util.concurrent.atomic.AtomicReference;import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;import static org.mockito.ArgumentMatchers.any;import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.ArgumentMatchers.isNull;+import static org.mockito.Mockito.doAnswer;import static org.mockito.Mockito.RETURNS_DEEP_STUBS;import static org.mockito.Mockito.RETURNS_SELF;import static org.mockito.Mockito.mock;@@ -99,6 +113,165 @@ class BatchDockerRunnerTest {"io.floci.region", "us-east-1"));}+ @Test+ void stopJobStopsAnInFlightContainer() throws Exception {+ EmulatorConfig config = mock(EmulatorConfig.class);+ EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);+ EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);+ when(config.services()).thenReturn(services);+ when(services.batch()).thenReturn(batch);+ when(batch.dockerNetwork()).thenReturn(Optional.empty());+ when(config.port()).thenReturn(4566);+ when(config.hostname()).thenReturn(Optional.empty());++ CountDownLatch containerStarted = new CountDownLatch(1);+ AtomicBoolean stopped = new AtomicBoolean();+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);+ when(lifecycleManager.createAndStart(any())).thenAnswer(invocation -> {+ containerStarted.countDown();+ return new ContainerLifecycleManager.ContainerInfo("container-id", Map.of());+ });+ DockerClient dockerClient = mock(DockerClient.class, RETURNS_DEEP_STUBS);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);+ when(dockerClient.inspectContainerCmd("container-id").exec().getState().getRunning())+ .thenAnswer(invocation -> !stopped.get());+ when(dockerClient.inspectContainerCmd("container-id").exec().getState().getExitCodeLong()).thenReturn(137L);+ doAnswer(invocation -> {+ stopped.set(true);+ return null;+ }).when(lifecycleManager).stopAndRemove(eq("container-id"), isNull());++ ContainerBuilder containerBuilder = mock(ContainerBuilder.class);+ ContainerBuilder.Builder builder = mock(ContainerBuilder.Builder.class, RETURNS_SELF);+ when(containerBuilder.newContainer(anyString())).thenReturn(builder);+ when(builder.build()).thenReturn(mock(ContainerSpec.class));+ BatchDockerRunner runner = new BatchDockerRunner(containerBuilder, lifecycleManager,+ mock(ContainerLogStreamer.class), config, mock(ContainerDetector.class));+ BatchJob job = job();+ AtomicReference<BatchRunResult> result = new AtomicReference<>();++ Thread runThread = Thread.startVirtualThread(() -> result.set(runner.run(job, 1)));+ assertTrue(containerStarted.await(2, TimeUnit.SECONDS));+ runner.requestStop(job.getJobId());+ runner.stopJob(job.getJobId());+ runThread.join(2000);++ assertFalse(runThread.isAlive());+ assertEquals(137, result.get().exitCode());+ verify(lifecycleManager).stopAndRemove("container-id", null);+ }++ @Test+ void stopJobAfterRequestIsClearedDoesNotRecreateStopMarker() {+ EmulatorConfig config = config(Optional.empty());+ EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);+ EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);+ when(config.services()).thenReturn(services);+ when(services.batch()).thenReturn(batch);+ when(batch.dockerNetwork()).thenReturn(Optional.empty());+ when(config.port()).thenReturn(4566);++ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);+ when(lifecycleManager.createAndStart(any())).thenReturn(+ new ContainerLifecycleManager.ContainerInfo("container-id", Map.of()));+ DockerClient dockerClient = mock(DockerClient.class, RETURNS_DEEP_STUBS);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);+ when(dockerClient.inspectContainerCmd("container-id").exec().getState().getRunning()).thenReturn(false);+ when(dockerClient.inspectContainerCmd("container-id").exec().getState().getExitCodeLong()).thenReturn(0L);+ ContainerBuilder containerBuilder = mock(ContainerBuilder.class);+ ContainerBuilder.Builder builder = mock(ContainerBuilder.Builder.class, RETURNS_SELF);+ when(containerBuilder.newContainer(anyString())).thenReturn(builder);+ when(builder.build()).thenReturn(mock(ContainerSpec.class));+ BatchDockerRunner runner = new BatchDockerRunner(containerBuilder, lifecycleManager,+ mock(ContainerLogStreamer.class), config, mock(ContainerDetector.class));++ runner.requestStop("job-1");+ runner.clearStopRequest("job-1");+ runner.stopJob("job-1");+ BatchRunResult result = runner.run(job(), 1);++ assertEquals(0, result.exitCode());+ verify(lifecycleManager).createAndStart(any());+ }++ @Test+ void stopJobStopsEveryInFlightNodeContainer() throws Exception {+ EmulatorConfig config = config(Optional.empty());+ EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);+ EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);+ when(config.services()).thenReturn(services);+ when(services.batch()).thenReturn(batch);+ when(batch.dockerNetwork()).thenReturn(Optional.empty());+ when(config.port()).thenReturn(4566);++ CountDownLatch containersStarted = new CountDownLatch(2);+ AtomicInteger containerSequence = new AtomicInteger();+ Set<String> stoppedContainers = ConcurrentHashMap.newKeySet();+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);+ when(lifecycleManager.createAndStart(any())).thenAnswer(invocation -> {+ String containerId = "container-" + containerSequence.getAndIncrement();+ containersStarted.countDown();+ return new ContainerLifecycleManager.ContainerInfo(containerId, Map.of());+ });+ DockerClient dockerClient = mock(DockerClient.class, RETURNS_DEEP_STUBS);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);+ for (int i = 0; i < 2; i++) {+ String containerId = "container-" + i;+ when(dockerClient.inspectContainerCmd(containerId).exec().getState().getRunning())+ .thenAnswer(invocation -> !stoppedContainers.contains(containerId));+ when(dockerClient.inspectContainerCmd(containerId).exec().getState().getExitCodeLong()).thenReturn(137L);+ doAnswer(invocation -> {+ stoppedContainers.add(containerId);+ return null;+ }).when(lifecycleManager).stopAndRemove(eq(containerId), isNull());+ }+ ContainerBuilder containerBuilder = mock(ContainerBuilder.class);+ ContainerBuilder.Builder builder = mock(ContainerBuilder.Builder.class, RETURNS_SELF);+ when(containerBuilder.newContainer(anyString())).thenReturn(builder);+ when(builder.build()).thenReturn(mock(ContainerSpec.class));+ BatchDockerRunner runner = new BatchDockerRunner(containerBuilder, lifecycleManager,+ mock(ContainerLogStreamer.class), config, mock(ContainerDetector.class));+ BatchJob job = job();+ BatchNodeExecution firstNode = node(0);+ BatchNodeExecution secondNode = node(1);+ AtomicReference<BatchRunResult> firstResult = new AtomicReference<>();+ AtomicReference<BatchRunResult> secondResult = new AtomicReference<>();++ Thread firstThread = Thread.startVirtualThread(() -> firstResult.set(runner.run(job, 1, firstNode)));+ Thread secondThread = Thread.startVirtualThread(() -> secondResult.set(runner.run(job, 1, secondNode)));+ assertTrue(containersStarted.await(2, TimeUnit.SECONDS));+ runner.requestStop(job.getJobId());+ runner.stopJob(job.getJobId());+ firstThread.join(2000);+ secondThread.join(2000);++ assertFalse(firstThread.isAlive());+ assertFalse(secondThread.isAlive());+ assertEquals(137, firstResult.get().exitCode());+ assertEquals(137, secondResult.get().exitCode());+ verify(lifecycleManager).stopAndRemove("container-0", null);+ verify(lifecycleManager).stopAndRemove("container-1", null);+ }++ private BatchJob job() {+ BatchJob job = new BatchJob();+ job.setJobId("job-1");+ job.setJobName("my-job");+ job.setJobQueueName("my-queue");+ job.setJobDefinitionName("my-def");+ job.setContainerImage("busybox:stable");+ job.setRegion("us-east-1");+ job.setAccountId("000000000000");+ return job;+ }++ private BatchNodeExecution node(int nodeIndex) {+ BatchNodeExecution node = new BatchNodeExecution();+ node.setNodeIndex(nodeIndex);+ node.setContainerImage("busybox:stable");+ return node;+ }+private BatchDockerRunner runner(EmulatorConfig config, ContainerDetector detector) {return new BatchDockerRunner(mock(ContainerBuilder.class),diff --git a/src/test/java/io/github/hectorvent/floci/services/batch/BatchIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/batch/BatchIntegrationTest.javaindex e03e7a973a..fc4d33fd24 100644--- a/src/test/java/io/github/hectorvent/floci/services/batch/BatchIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/batch/BatchIntegrationTest.java@@ -80,6 +80,49 @@ class BatchIntegrationTest {.body("jobSummaryList.find { it.jobId == '%s' }.container.taskArn".formatted(jobId), nullValue());}+ @Test+ void cancelAndTerminateJobExposeAwsRoutes() {+ String suffix = uniqueSuffix();+ String queueArn = createQueue("batch-control-queue-" + suffix,+ createComputeEnvironment("batch-control-ce-" + suffix));+ String definitionArn = registerJobDefinition("batch-control-job-" + suffix, "[]", "[]");+ String jobId = submit(queueArn, definitionArn, "batch-control-submit-" + suffix);++ givenJson("""+ {"jobId":"%s","reason":"No longer needed"}+ """.formatted(jobId))+ .when()+ .post("/v1/canceljob")+ .then()+ .statusCode(200);++ givenJson("""+ {"jobId":"%s","reason":"Stop requested"}+ """.formatted(jobId))+ .when()+ .post("/v1/terminatejob")+ .then()+ .statusCode(200);++ givenJson("""+ {"jobId":"%s"}+ """.formatted(jobId))+ .when()+ .post("/v1/canceljob")+ .then()+ .statusCode(400)+ .body("__type", equalTo("ClientException"));++ givenJson("""+ {"jobId":"missing-job","reason":"Stop requested"}+ """)+ .when()+ .post("/v1/terminatejob")+ .then()+ .statusCode(400)+ .body("__type", equalTo("ClientException"));+ }+@Testvoid jobDefinitionRevisionsAndDeregister() {String suffix = uniqueSuffix();diff --git a/src/test/java/io/github/hectorvent/floci/services/batch/BatchServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/batch/BatchServiceTest.javaindex 1558979e3e..5767245170 100644--- a/src/test/java/io/github/hectorvent/floci/services/batch/BatchServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/batch/BatchServiceTest.java@@ -14,6 +14,7 @@ import io.github.hectorvent.floci.services.batch.model.BatchJobDefinition;import io.github.hectorvent.floci.services.batch.model.BatchJobQueue;import io.github.hectorvent.floci.services.batch.model.BatchNodeExecution;import io.github.hectorvent.floci.services.batch.model.BatchRunResult;+import io.github.hectorvent.floci.services.batch.model.BatchStatus;import org.junit.jupiter.api.Test;import java.util.ArrayList;@@ -23,14 +24,19 @@ import java.util.List;import java.util.Optional;import java.util.Set;import java.util.UUID;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.TimeUnit;import java.util.function.Predicate;import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertNotNull;import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;import static org.mockito.ArgumentMatchers.any;import static org.mockito.ArgumentMatchers.anyInt;+import static org.mockito.Mockito.doAnswer;import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;import static org.mockito.Mockito.times;import static org.mockito.Mockito.verify;import static org.mockito.Mockito.when;@@ -779,6 +785,265 @@ class BatchServiceTest {assertEquals("ClientException", e.getErrorCode());}+ @Test+ void cancelJobFailsRunnableJobWithRequestedReason() throws Exception {+ InMemoryStorage<String, BatchJob> jobStore = new InMemoryStorage<>();+ BatchService service = service("immediate", jobStore, mock(BatchDockerRunner.class));+ BatchJob job = storedJob("cancel-runnable", BatchStatus.RUNNABLE);+ jobStore.put(job.getJobId(), job);++ service.cancelJob(json("""+ {"jobId":"cancel-runnable","reason":"No longer needed"}+ """));++ JsonNode detail = service.describeJobs(json("""+ {"jobs":["cancel-runnable"]}+ """)).path("jobs").get(0);+ assertEquals("FAILED", detail.path("status").asText());+ assertEquals("No longer needed", detail.path("statusReason").asText());+ assertTrue(detail.path("stoppedAt").asLong() > 0);+ }++ @Test+ void cancelJobDoesNotStopRunningJob() throws Exception {+ BatchDockerRunner runner = mock(BatchDockerRunner.class);+ CountDownLatch runStarted = new CountDownLatch(1);+ CountDownLatch releaseRun = new CountDownLatch(1);+ when(runner.run(any(BatchJob.class), anyInt())).thenAnswer(invocation -> {+ runStarted.countDown();+ assertTrue(releaseRun.await(2, TimeUnit.SECONDS));+ return new BatchRunResult(0, null, "log-stream", 1L, 2L, false);+ });+ BatchService service = dockerService(runner);+ String queueArn = arrayReadyQueue(service);+ String definitionArn = service.registerJobDefinition(json("""+ {+ "jobDefinitionName":"cancel-running-job",+ "type":"container",+ "containerProperties":{"image":"job:latest"}+ }+ """), REGION).path("jobDefinitionArn").asText();+ String jobId = service.submitJob(json("""+ {"jobName":"cancel-running","jobQueue":"%s","jobDefinition":"%s"}+ """.formatted(queueArn, definitionArn)), REGION).path("jobId").asText();++ assertTrue(runStarted.await(2, TimeUnit.SECONDS));+ try {+ service.cancelJob(json("""+ {"jobId":"%s","reason":"Too late to cancel"}+ """.formatted(jobId)));+ JsonNode running = service.describeJobs(json("""+ {"jobs":["%s"]}+ """.formatted(jobId))).path("jobs").get(0);+ assertEquals("RUNNING", running.path("status").asText());+ verify(runner, never()).requestStop(jobId);+ verify(runner, never()).stopJob(jobId);+ } finally {+ releaseRun.countDown();+ }+ assertNotNull(waitForJobStatus(service, jobId, "SUCCEEDED"));+ }++ @Test+ void terminateJobStopsRunningContainerAndPreservesRequestedReason() throws Exception {+ BatchDockerRunner runner = mock(BatchDockerRunner.class);+ CountDownLatch runStarted = new CountDownLatch(1);+ CountDownLatch stopRequested = new CountDownLatch(1);+ when(runner.run(any(BatchJob.class), anyInt())).thenAnswer(invocation -> {+ runStarted.countDown();+ assertTrue(stopRequested.await(2, TimeUnit.SECONDS));+ return new BatchRunResult(137, "Job terminated", "log-stream", 1L, 2L, false);+ });+ doAnswer(invocation -> {+ stopRequested.countDown();+ return null;+ }).when(runner).requestStop(any(String.class));+ BatchService service = dockerService(runner);+ String queueArn = arrayReadyQueue(service);+ String definitionArn = service.registerJobDefinition(json("""+ {+ "jobDefinitionName":"terminate-running-job",+ "type":"container",+ "containerProperties":{"image":"job:latest"}+ }+ """), REGION).path("jobDefinitionArn").asText();+ String jobId = service.submitJob(json("""+ {"jobName":"terminate-running","jobQueue":"%s","jobDefinition":"%s"}+ """.formatted(queueArn, definitionArn)), REGION).path("jobId").asText();++ assertTrue(runStarted.await(2, TimeUnit.SECONDS));+ service.terminateJob(json("""+ {"jobId":"%s","reason":"Operator requested shutdown"}+ """.formatted(jobId)));++ JsonNode failed = waitForJobStatus(service, jobId, "FAILED");+ assertNotNull(failed);+ assertEquals("Operator requested shutdown", failed.path("statusReason").asText());+ JsonNode failedWithAttempt = waitForJobAttemptCount(service, jobId, 1);+ assertNotNull(failedWithAttempt);+ assertEquals(137, failedWithAttempt.path("attempts").get(0).path("container").path("exitCode").asInt());+ assertEquals("Operator requested shutdown", failedWithAttempt.path("statusReason").asText());+ verify(runner).requestStop(jobId);+ verify(runner).stopJob(jobId);+ }++ @Test+ void jobControlValidatesRequiredFieldsReasonLimitAndUnknownJobs() throws Exception {+ InMemoryStorage<String, BatchJob> jobStore = new InMemoryStorage<>();+ BatchService service = immediateService(jobStore);++ AwsException missingJobId = assertThrows(AwsException.class, () -> service.cancelJob(json("""+ {"reason":"No longer needed"}+ """)));+ assertEquals("ClientException", missingJobId.getErrorCode());+ assertEquals("jobId is required", missingJobId.getMessage());++ AwsException missingReason = assertThrows(AwsException.class, () -> service.terminateJob(json("""+ {"jobId":"job-1"}+ """)));+ assertEquals("ClientException", missingReason.getErrorCode());+ assertEquals("reason is required", missingReason.getMessage());++ BatchJob boundaryJob = storedJob("boundary-job", BatchStatus.RUNNABLE);+ jobStore.put(boundaryJob.getJobId(), boundaryJob);+ String boundaryReason = "x".repeat(1024);+ service.cancelJob(json("""+ {"jobId":"boundary-job","reason":"%s"}+ """.formatted(boundaryReason)));+ JsonNode boundaryDetail = service.describeJobs(json("""+ {"jobs":["boundary-job"]}+ """)).path("jobs").get(0);+ assertEquals(boundaryReason, boundaryDetail.path("statusReason").asText());++ BatchJob overLimitJob = storedJob("over-limit-job", BatchStatus.RUNNABLE);+ jobStore.put(overLimitJob.getJobId(), overLimitJob);+ AwsException overLimit = assertThrows(AwsException.class, () -> service.cancelJob(json("""+ {"jobId":"over-limit-job","reason":"%s"}+ """.formatted("x".repeat(1025)))));+ assertEquals("ClientException", overLimit.getErrorCode());++ AwsException unknownJob = assertThrows(AwsException.class, () -> service.terminateJob(json("""+ {"jobId":"unknown-job","reason":"Stop requested"}+ """)));+ assertEquals("ClientException", unknownJob.getErrorCode());+ assertEquals("Job not found: unknown-job", unknownJob.getMessage());+ }++ @Test+ void jobControlIsIdempotentForTerminalJobs() throws Exception {+ InMemoryStorage<String, BatchJob> jobStore = new InMemoryStorage<>();+ BatchDockerRunner runner = mock(BatchDockerRunner.class);+ BatchService service = service("docker", jobStore, runner);+ BatchJob succeeded = storedJob("succeeded-job", BatchStatus.SUCCEEDED);+ succeeded.setStatusReason("Job completed successfully");+ BatchJob failed = storedJob("failed-job", BatchStatus.FAILED);+ failed.setStatusReason("Original failure");+ jobStore.put(succeeded.getJobId(), succeeded);+ jobStore.put(failed.getJobId(), failed);++ service.cancelJob(json("""+ {"jobId":"succeeded-job","reason":"Cancel again"}+ """));+ service.terminateJob(json("""+ {"jobId":"failed-job","reason":"Terminate again"}+ """));++ JsonNode jobs = service.describeJobs(json("""+ {"jobs":["succeeded-job","failed-job"]}+ """)).path("jobs");+ assertEquals("SUCCEEDED", jobs.get(0).path("status").asText());+ assertEquals("Job completed successfully", jobs.get(0).path("statusReason").asText());+ assertEquals("FAILED", jobs.get(1).path("status").asText());+ assertEquals("Original failure", jobs.get(1).path("statusReason").asText());+ verify(runner, never()).requestStop(any(String.class));+ verify(runner, never()).stopJob(any(String.class));+ }++ @Test+ void terminateMultiNodeJobStopsNodesAndPreservesRequestedReason() throws Exception {+ BatchDockerRunner runner = mock(BatchDockerRunner.class);+ CountDownLatch nodesStarted = new CountDownLatch(2);+ CountDownLatch stopRequested = new CountDownLatch(1);+ when(runner.run(any(BatchJob.class), anyInt(), any(BatchNodeExecution.class)))+ .thenAnswer(invocation -> {+ BatchNodeExecution node = invocation.getArgument(2);+ nodesStarted.countDown();+ assertTrue(stopRequested.await(2, TimeUnit.SECONDS));+ return new BatchRunResult(137, "Job terminated", "log-" + node.getNodeIndex(),+ 1L, 2L, false);+ });+ doAnswer(invocation -> {+ stopRequested.countDown();+ return null;+ }).when(runner).requestStop(any(String.class));+ BatchService service = dockerService(runner);+ String queueArn = arrayReadyQueue(service);+ String definitionArn = service.registerJobDefinition(json("""+ {+ "jobDefinitionName":"terminate-mnp-job",+ "type":"multinode",+ "nodeProperties":{+ "numNodes":2,+ "mainNode":0,+ "nodeRangeProperties":[{"targetNodes":"0:1","container":{"image":"worker:latest"}}]+ }+ }+ """), REGION).path("jobDefinitionArn").asText();+ String jobId = service.submitJob(json("""+ {"jobName":"terminate-mnp","jobQueue":"%s","jobDefinition":"%s"}+ """.formatted(queueArn, definitionArn)), REGION).path("jobId").asText();++ assertTrue(nodesStarted.await(2, TimeUnit.SECONDS));+ service.terminateJob(json("""+ {"jobId":"%s","reason":"Stop every node"}+ """.formatted(jobId)));++ JsonNode failedWithAttempt = waitForJobAttemptCount(service, jobId, 1);+ assertNotNull(failedWithAttempt);+ assertEquals("FAILED", failedWithAttempt.path("status").asText());+ assertEquals("Stop every node", failedWithAttempt.path("statusReason").asText());+ JsonNode nodes = service.listJobs(json("""+ {"multiNodeJobId":"%s"}+ """.formatted(jobId))).path("jobSummaryList");+ assertEquals(137, nodes.get(0).path("container").path("exitCode").asInt());+ assertEquals(137, nodes.get(1).path("container").path("exitCode").asInt());+ verify(runner).requestStop(jobId);+ verify(runner).stopJob(jobId);+ }++ @Test+ void terminateArrayParentFailsAndStopsEveryNonterminalChild() throws Exception {+ InMemoryStorage<String, BatchJob> jobStore = new InMemoryStorage<>();+ BatchDockerRunner runner = mock(BatchDockerRunner.class);+ BatchService service = service("docker", jobStore, runner);+ BatchJob parent = storedJob("array-parent", BatchStatus.SUBMITTED);+ parent.setArraySize(2);+ BatchJob first = storedJob("array-parent:0", BatchStatus.RUNNABLE);+ first.setArrayJobId(parent.getJobId());+ first.setArrayIndex(0);+ BatchJob second = storedJob("array-parent:1", BatchStatus.RUNNING);+ second.setArrayJobId(parent.getJobId());+ second.setArrayIndex(1);+ jobStore.put(parent.getJobId(), parent);+ jobStore.put(first.getJobId(), first);+ jobStore.put(second.getJobId(), second);++ service.terminateJob(json("""+ {"jobId":"array-parent","reason":"Stop the array"}+ """));++ JsonNode detail = service.describeJobs(json("""+ {"jobs":["array-parent","array-parent:0","array-parent:1"]}+ """)).path("jobs");+ assertEquals("FAILED", detail.get(0).path("status").asText());+ assertEquals("Stop the array", detail.get(0).path("statusReason").asText());+ assertEquals("FAILED", detail.get(1).path("status").asText());+ assertEquals("FAILED", detail.get(2).path("status").asText());+ verify(runner).requestStop("array-parent:1");+ verify(runner, never()).stopJob("array-parent:0");+ verify(runner).stopJob("array-parent:1");+ }+private String arrayReadyQueue(BatchService service) throws Exception {String suffix = UUID.randomUUID().toString();String computeArn = service.createComputeEnvironment(json("""@@ -794,18 +1059,23 @@ class BatchServiceTest {}private BatchService dockerService(BatchDockerRunner runner) {+ return service("docker", new InMemoryStorage<String, BatchJob>(), runner);+ }++ private BatchService service(String runnerMode, StorageBackend<String, BatchJob> jobStore,+ BatchDockerRunner runner) {EmulatorConfig config = mock(EmulatorConfig.class);EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);when(config.services()).thenReturn(services);when(services.batch()).thenReturn(batch);- when(batch.runnerMode()).thenReturn("docker");+ when(batch.runnerMode()).thenReturn(runnerMode);return new BatchService(new InMemoryStorage<String, BatchJobDefinition>(),new InMemoryStorage<String, BatchJobQueue>(),new InMemoryStorage<String, BatchComputeEnvironment>(),- new InMemoryStorage<String, BatchJob>(),+ jobStore,new RegionResolver(REGION, ACCOUNT),config,objectMapper,@@ -813,22 +1083,21 @@ class BatchServiceTest {}private BatchService immediateService(StorageBackend<String, BatchJob> jobStore) {- EmulatorConfig config = mock(EmulatorConfig.class);- EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);- EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);- when(config.services()).thenReturn(services);- when(services.batch()).thenReturn(batch);- when(batch.runnerMode()).thenReturn("immediate");+ return service("immediate", jobStore, mock(BatchDockerRunner.class));+ }- return new BatchService(- new InMemoryStorage<String, BatchJobDefinition>(),- new InMemoryStorage<String, BatchJobQueue>(),- new InMemoryStorage<String, BatchComputeEnvironment>(),- jobStore,- new RegionResolver(REGION, ACCOUNT),- config,- objectMapper,- mock(BatchDockerRunner.class));+ private BatchJob storedJob(String jobId, BatchStatus status) {+ BatchJob job = new BatchJob();+ job.setJobId(jobId);+ job.setJobArn("arn:aws:batch:us-east-1:" + ACCOUNT + ":job/" + jobId);+ job.setJobName(jobId);+ job.setJobQueue("queue");+ job.setJobDefinition("definition");+ job.setStatus(status.name());+ job.setCreatedAt(1L);+ job.setRegion(REGION);+ job.setAccountId(ACCOUNT);+ return job;}private ObjectNode json(String body) throws Exception {@@ -848,6 +1117,19 @@ class BatchServiceTest {return null;}+ private JsonNode waitForJobAttemptCount(BatchService service, String jobId, int attemptCount) throws Exception {+ ObjectNode request = objectMapper.createObjectNode();+ request.putArray("jobs").add(jobId);+ for (int i = 0; i < 100; i++) {+ JsonNode job = service.describeJobs(request).path("jobs").get(0);+ if (job != null && job.path("attempts").size() == attemptCount) {+ return job;+ }+ Thread.sleep(10);+ }+ return null;+ }+private static final class ReverseScanJobStorage implements StorageBackend<String, BatchJob> {private final LinkedHashMap<String, BatchJob> store = new LinkedHashMap<>();