Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(batch): support canceling and terminating jobs (#3922)

XuJian9c4cd8c9e13b
diff --git a/docs/services/batch.md b/docs/services/batch.md
index 7b42eb0523..1c98a02a60 100644
--- a/docs/services/batch.md
+++ b/docs/services/batch.md
@@ -21,6 +21,8 @@ Floci Batch implements the AWS Batch control plane for local integration tests.
| `DeregisterJobDefinition` | `POST /v1/deregisterjobdefinition` | Mark a job definition revision inactive |
| `DescribeJobDefinitions` | `POST /v1/describejobdefinitions` | List job definitions by name, ARN, revision, and status |
| `SubmitJob` | `POST /v1/submitjob` | Submit a local Batch job |
+| `CancelJob` | `POST /v1/canceljob` | Cancel a job that has not started running |
+| `TerminateJob` | `POST /v1/terminatejob` | Terminate a nonterminal job and stop its running container or nodes |
| `DescribeJobs` | `POST /v1/describejobs` | Describe jobs by job ID |
| `ListJobs` | `POST /v1/listjobs` | List jobs by queue, status, AWS `filters`, and pagination |
@@ -61,6 +63,19 @@ SUBMITTED -> PENDING -> RUNNABLE -> STARTING -> RUNNING -> SUCCEEDED|FAILED
Driving every immediate-mode job through `PENDING` is a local simplification for tests. AWS may skip that state when there are no dependency or capacity waits.
+## Job Control
+
+`CancelJob` accepts jobs in `SUBMITTED`, `PENDING`, or `RUNNABLE` and moves them to `FAILED` with
+the requested reason. Jobs that already reached `STARTING` or `RUNNING` are left unchanged, matching
+AWS behavior. `TerminateJob` moves any nonterminal job to `FAILED`; in Docker mode it also stops and
+removes the running container. Both operations are idempotent for terminal jobs and require `jobId`
+and `reason`.
+
+For an array parent, job control cascades to its children. Cancellation affects only children that
+have not started, while termination also stops running children. Terminating an MNP job stops every
+running node container. The stored terminal reason wins over a concurrent container exit, so the
+background runner cannot overwrite a canceled or terminated job with a later result.
+
### Array jobs
Submitting with `arrayProperties.size` (2-10,000) fans out one child job per index, sharing the
@@ -69,8 +84,8 @@ its own Docker container in `docker` mode). The parent's `status` and `arrayProp
are computed live from its children on every `DescribeJobs`/`ListJobs` call rather than stored and
updated incrementally, so there is nothing to race when many children finish concurrently. A
queue-level `ListJobs` shows one entry per array job (the parent); pass `arrayJobId` to list its
-children. `TerminateJob`/`CancelJob` cascade is not applicable since neither operation is
-implemented (see Limitations).
+children. `CancelJob` and `TerminateJob` on the parent cascade to the children using the job-control
+rules above.
### Multi-node parallel (MNP) jobs
@@ -80,8 +95,9 @@ A job definition registered with `"type": "multinode"` and `nodeProperties` (`nu
`AWS_BATCH_JOB_NODE_INDEX`, `AWS_BATCH_JOB_MAIN_NODE_INDEX`, and `AWS_BATCH_JOB_NUM_NODES` set. The
job's status, retry decision, and `startedAt`/`stoppedAt`/`statusReason` are all determined solely by
the **main node** (matching AWS: a child node failing does not fail or retry the job). Every node
-still runs to completion rather than being stopped early (see Limitations below for the two cases
-this affects). `DescribeJobs`/queue-level `ListJobs` return one entry for the whole
+normally runs to completion, while `TerminateJob` stops all running nodes. A natural main-node exit
+does not yet preempt the remaining nodes (see Limitations). `DescribeJobs`/queue-level `ListJobs`
+return one entry for the whole
job (with `nodeProperties`, no per-node `container`); pass `multiNodeJobId` to `ListJobs` for a
one-entry-per-node breakdown with each node's own `container` and `nodeProperties.nodeIndex`.
`containerOverrides` and array jobs are not supported on MNP job definitions; use `nodeOverrides`
@@ -132,15 +148,12 @@ IAM roles, VPC fields, Fargate declarations, log configuration, storage, and res
- No VPC/subnet/security-group simulation.
- No AWS-faithful capacity scheduling.
- No job dependencies (`dependsOn`).
-- `CancelJob` and `TerminateJob` are not implemented, so there is no way to cancel/terminate an
- array job's children or an MNP job's nodes either — this is the same pre-existing gap as for a
- regular job, not something array/MNP jobs make worse.
- EventBridge input transformers work through the existing EventBridge target input path; full Batch-specific input-transformer parity is not implemented.
- An MNP job does not finish the instant its main node exits. AWS stops the job (and its remaining
child nodes) as soon as the main node exits; Floci instead waits for every node's container to
exit before scoring the attempt, so a child that outlives the main node keeps the job `RUNNING`
- longer than AWS would. Closing this needs a way for `BatchDockerRunner` to stop one node's
- container from outside that node's own run loop, which does not exist yet.
+ longer than AWS would. `TerminateJob` can stop the whole job, but main-node completion does not
+ yet trigger that same preemption path automatically.
**Known follow-ups on the array/MNP surface** (not required for `SubmitJob`/`DescribeJobs`/`ListJobs`
correctness, deliberately deferred):
@@ -153,7 +166,7 @@ correctness, deliberately deferred):
definitions; there is no way to provision a `multinode` job definition via CloudFormation, only
via a direct `RegisterJobDefinition` call.
- `ListJobs` with `arrayJobId` ignores the `filters` parameter (only `jobStatus` is honored for
- children) — matches AWS's own note that filters don't apply to child jobs, but it means array
+ children). This matches AWS's own note that filters don't apply to child jobs, but it means array
children can't be filtered by `JOB_NAME`/`JOB_DEFINITION`/etc.
- `ListJobs` with `multiNodeJobId` has no pagination (`maxResults`/`nextToken`); it always returns
every node in one response.
diff --git a/src/main/java/io/github/hectorvent/floci/services/batch/BatchController.java b/src/main/java/io/github/hectorvent/floci/services/batch/BatchController.java
index 509b8b6c6e..d057c8449a 100644
--- a/src/main/java/io/github/hectorvent/floci/services/batch/BatchController.java
+++ b/src/main/java/io/github/hectorvent/floci/services/batch/BatchController.java
@@ -119,6 +119,20 @@ public class BatchController {
Response.ok(service.submitJob(request, region)).build());
}
+ @POST
+ @Path("/v1/canceljob")
+ public Response cancelJob(@Context HttpHeaders headers, String body) {
+ return handle(headers, body, (request, region) ->
+ Response.ok(service.cancelJob(request)).build());
+ }
+
+ @POST
+ @Path("/v1/terminatejob")
+ public Response terminateJob(@Context HttpHeaders headers, String body) {
+ return handle(headers, body, (request, region) ->
+ Response.ok(service.terminateJob(request)).build());
+ }
+
@POST
@Path("/v1/describejobs")
public Response describeJobs(@Context HttpHeaders headers, String body) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/batch/BatchDockerRunner.java b/src/main/java/io/github/hectorvent/floci/services/batch/BatchDockerRunner.java
index 2157c27d9e..4fb502e960 100644
--- a/src/main/java/io/github/hectorvent/floci/services/batch/BatchDockerRunner.java
+++ b/src/main/java/io/github/hectorvent/floci/services/batch/BatchDockerRunner.java
@@ -25,6 +25,7 @@ import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
@ApplicationScoped
@@ -36,6 +37,7 @@ public class BatchDockerRunner implements ContainerTeardown {
// Containers of jobs currently inside run(); drained on emulator shutdown so a
// SIGTERM mid-job does not orphan the container.
private final ConcurrentHashMap<String, String> inFlightContainers = new ConcurrentHashMap<>();
+ private final Set<String> stopRequestedJobs = ConcurrentHashMap.newKeySet();
private final ContainerBuilder containerBuilder;
private final ContainerLifecycleManager lifecycleManager;
@@ -88,6 +90,9 @@ public class BatchDockerRunner implements ContainerTeardown {
String containerId = null;
try {
+ if (stopRequestedJobs.contains(job.getJobId())) {
+ return stopped(startedAt, logStreamName);
+ }
if (image == null || image.isBlank()) {
return failed(startedAt, logStreamName, missingImageMessage);
}
@@ -110,7 +115,15 @@ public class BatchDockerRunner implements ContainerTeardown {
ContainerSpec spec = builder.build();
containerId = lifecycleManager.createAndStart(spec).containerId();
inFlightContainers.put(inFlightKey, containerId);
+ if (stopRequestedJobs.contains(job.getJobId())) {
+ releaseAndStop(inFlightKey, containerId, null);
+ return stopped(startedAt, logStreamName);
+ }
logHandle = logStreamer.attach(containerId, LOG_GROUP, logStreamName, job.getRegion(), logSourceLabel);
+ if (stopRequestedJobs.contains(job.getJobId())) {
+ releaseAndStop(inFlightKey, containerId, logHandle);
+ return stopped(startedAt, logStreamName);
+ }
Integer exitCode = waitForExit(containerId, timeout(job));
long stoppedAt = System.currentTimeMillis();
@@ -129,7 +142,30 @@ public class BatchDockerRunner implements ContainerTeardown {
}
}
- // Whoever wins the map removal owns the stop — stopManagedContainers() may have
+ public void requestStop(String jobId) {
+ stopRequestedJobs.add(jobId);
+ }
+
+ public void stopJob(String jobId) {
+ for (Map.Entry<String, String> entry : new ConcurrentHashMap<>(inFlightContainers).entrySet()) {
+ if (!entry.getKey().equals(jobId) && !entry.getKey().startsWith(jobId + "#node")) {
+ continue;
+ }
+ if (inFlightContainers.remove(entry.getKey(), entry.getValue())) {
+ try {
+ lifecycleManager.stopAndRemove(entry.getValue(), null);
+ } catch (Exception e) {
+ LOG.warnv("Failed to stop Batch container for job {0}: {1}", jobId, e.getMessage());
+ }
+ }
+ }
+ }
+
+ public void clearStopRequest(String jobId) {
+ stopRequestedJobs.remove(jobId);
+ }
+
+ // Whoever wins the map removal owns the stop. stopManagedContainers() may have
// claimed the container first during shutdown, but the log stream is still ours.
private void releaseAndStop(String jobId, String containerId, Closeable logHandle) {
if (inFlightContainers.remove(jobId, containerId)) {
@@ -165,6 +201,11 @@ public class BatchDockerRunner implements ContainerTeardown {
return new BatchRunResult(1, reason, logStreamName, startedAt, System.currentTimeMillis(), false);
}
+ private BatchRunResult stopped(long startedAt, String logStreamName) {
+ return new BatchRunResult(137, "Job terminated", logStreamName,
+ startedAt, System.currentTimeMillis(), false);
+ }
+
private List<String> buildEnvironment(BatchJob job, int attemptNumber) {
List<String> env = baseEnvironment(job, attemptNumber);
appendEnvironment(env, job.getResolvedEnvironment());
diff --git a/src/main/java/io/github/hectorvent/floci/services/batch/BatchService.java b/src/main/java/io/github/hectorvent/floci/services/batch/BatchService.java
index 79e0fa8d05..aae27817b7 100644
--- a/src/main/java/io/github/hectorvent/floci/services/batch/BatchService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/batch/BatchService.java
@@ -56,6 +56,7 @@ public class BatchService {
private static final int DEFAULT_MAX_RESULTS = 1000;
private static final int FILTERED_MAX_RESULTS = 100;
private static final int MAX_TAGS = 50;
+ private static final int MAX_JOB_CONTROL_REASON_LENGTH = 1024;
private static final Pattern JOB_NAME_PATTERN = Pattern.compile("[A-Za-z0-9][A-Za-z0-9_-]{0,127}");
private static final Set<String> SUPPORTED_JOB_DEFINITION_TYPES = Set.of("container", "multinode");
private static final int MIN_ARRAY_SIZE = 2;
@@ -64,6 +65,8 @@ public class BatchService {
private static final Set<String> SUPPORTED_COMPUTE_ENVIRONMENT_STATES = Set.of("ENABLED", "DISABLED");
private static final Set<String> SUPPORTED_LIST_FILTERS = Set.of(
"JOB_NAME", "JOB_DEFINITION", "BEFORE_CREATED_AT", "AFTER_CREATED_AT", "SHARE_IDENTIFIER");
+ private static final Set<String> TERMINAL_JOB_STATUSES = Set.of(
+ BatchStatus.SUCCEEDED.name(), BatchStatus.FAILED.name());
private final StorageBackend<String, BatchJobDefinition> jobDefinitionStore;
private final StorageBackend<String, BatchJobQueue> jobQueueStore;
@@ -456,6 +459,71 @@ public class BatchService {
return submitJobResponse(job);
}
+ public ObjectNode cancelJob(JsonNode request) {
+ return controlJob(request, false);
+ }
+
+ public ObjectNode terminateJob(JsonNode request) {
+ return controlJob(request, true);
+ }
+
+ private ObjectNode controlJob(JsonNode request, boolean terminate) {
+ String jobId = requiredText(request, "jobId");
+ String reason = requiredText(request, "reason");
+ if (reason.length() > MAX_JOB_CONTROL_REASON_LENGTH) {
+ throw client("reason must be at most " + MAX_JOB_CONTROL_REASON_LENGTH + " characters");
+ }
+ List<String> jobsToStop = new ArrayList<>();
+
+ synchronized (this) {
+ BatchJob job = getJob(jobId).orElseThrow(() -> client("Job not found: " + jobId));
+ if (job.getArraySize() != null) {
+ boolean changed = false;
+ List<BatchJob> children = jobStore.scan(k -> true).stream()
+ .filter(child -> jobId.equals(child.getArrayJobId()))
+ .toList();
+ for (BatchJob child : children) {
+ boolean wasRunning = BatchStatus.RUNNING.name().equals(child.getStatus());
+ if (controlSingleJob(child, reason, terminate)) {
+ changed = true;
+ if (terminate && wasRunning) {
+ dockerRunner.requestStop(child.getJobId());
+ jobsToStop.add(child.getJobId());
+ }
+ }
+ }
+ if (changed) {
+ job.setStatusReason(reason);
+ putJob(job);
+ }
+ } else {
+ boolean wasRunning = BatchStatus.RUNNING.name().equals(job.getStatus());
+ if (controlSingleJob(job, reason, terminate) && terminate && wasRunning) {
+ dockerRunner.requestStop(job.getJobId());
+ jobsToStop.add(job.getJobId());
+ }
+ }
+ }
+
+ jobsToStop.forEach(dockerRunner::stopJob);
+ return objectMapper.createObjectNode();
+ }
+
+ private boolean controlSingleJob(BatchJob job, String reason, boolean terminate) {
+ if (isTerminalStatus(job.getStatus())) {
+ return false;
+ }
+ if (!terminate && (BatchStatus.STARTING.name().equals(job.getStatus())
+ || BatchStatus.RUNNING.name().equals(job.getStatus()))) {
+ return false;
+ }
+ job.setStatus(BatchStatus.FAILED.name());
+ job.setStatusReason(reason);
+ job.setStoppedAt(now());
+ putJob(job);
+ return true;
+ }
+
private ObjectNode submitArrayJob(JsonNode request, String region, String jobName, BatchJobQueue queue,
BatchJobDefinition def, Map<String, String> parameters, int size) {
BatchJob parent = new BatchJob();
@@ -800,13 +868,21 @@ public class BatchService {
}
int maxAttempts = maxAttempts(job);
for (int attemptNumber = 1; attemptNumber <= maxAttempts; attemptNumber++) {
- transition(accountId, jobId, BatchStatus.PENDING, null, false);
+ if (!transition(accountId, jobId, BatchStatus.PENDING, null, false)) {
+ return;
+ }
sleepQuietly();
- transition(accountId, jobId, BatchStatus.RUNNABLE, null, false);
+ if (!transition(accountId, jobId, BatchStatus.RUNNABLE, null, false)) {
+ return;
+ }
sleepQuietly();
- transition(accountId, jobId, BatchStatus.STARTING, null, false);
+ if (!transition(accountId, jobId, BatchStatus.STARTING, null, false)) {
+ return;
+ }
sleepQuietly();
- transition(accountId, jobId, BatchStatus.RUNNING, null, true);
+ if (!transition(accountId, jobId, BatchStatus.RUNNING, null, true)) {
+ return;
+ }
BatchJob attemptJob = getJobForAccount(accountId, jobId).orElse(null);
if (attemptJob == null) {
return;
@@ -821,6 +897,8 @@ public class BatchService {
} catch (Exception e) {
LOG.warnv("Batch job {0} failed while running: {1}", jobId, e.getMessage());
failJob(accountId, jobId, e.getMessage());
+ } finally {
+ dockerRunner.clearStopRequest(jobId);
}
}
@@ -840,6 +918,7 @@ public class BatchService {
if (job == null) {
return true;
}
+ boolean controlledTerminal = isTerminalStatus(job.getStatus());
BatchAttemptContainer container = new BatchAttemptContainer();
container.setExitCode(result.exitCode());
container.setReason(result.reason());
@@ -853,6 +932,10 @@ public class BatchService {
job.getAttempts().add(attempt);
job.setContainer(container);
+ if (controlledTerminal) {
+ putJobForAccount(accountId, job);
+ return true;
+ }
if (result.exitCode() == 0) {
job.setStoppedAt(result.stoppedAt());
job.setStatus(BatchStatus.SUCCEEDED.name());
@@ -881,13 +964,21 @@ public class BatchService {
}
int maxAttempts = maxAttempts(job);
for (int attemptNumber = 1; attemptNumber <= maxAttempts; attemptNumber++) {
- transition(accountId, jobId, BatchStatus.PENDING, null, false);
+ if (!transition(accountId, jobId, BatchStatus.PENDING, null, false)) {
+ return;
+ }
sleepQuietly();
- transition(accountId, jobId, BatchStatus.RUNNABLE, null, false);
+ if (!transition(accountId, jobId, BatchStatus.RUNNABLE, null, false)) {
+ return;
+ }
sleepQuietly();
- transition(accountId, jobId, BatchStatus.STARTING, null, false);
+ if (!transition(accountId, jobId, BatchStatus.STARTING, null, false)) {
+ return;
+ }
sleepQuietly();
- transition(accountId, jobId, BatchStatus.RUNNING, null, true);
+ if (!transition(accountId, jobId, BatchStatus.RUNNING, null, true)) {
+ return;
+ }
BatchJob attemptJob = getJobForAccount(accountId, jobId).orElse(null);
if (attemptJob == null) {
return;
@@ -900,6 +991,8 @@ public class BatchService {
} catch (Exception e) {
LOG.warnv("Batch multi-node job {0} failed while running: {1}", jobId, e.getMessage());
failJob(accountId, jobId, e.getMessage());
+ } finally {
+ dockerRunner.clearStopRequest(jobId);
}
}
@@ -947,6 +1040,7 @@ public class BatchService {
if (job == null) {
return true;
}
+ boolean controlledTerminal = isTerminalStatus(job.getStatus());
List<BatchNodeExecution> executions = job.getNodeExecutions();
BatchRunResult mainResult = null;
for (int i = 0; i < executions.size(); i++) {
@@ -970,6 +1064,11 @@ public class BatchService {
attempt.setStatusReason(mainResult.reason());
job.getAttempts().add(attempt);
+ if (controlledTerminal) {
+ putJobForAccount(accountId, job);
+ return true;
+ }
+
if (mainResult.exitCode() == 0) {
job.setStoppedAt(mainResult.stoppedAt());
job.setStatus(BatchStatus.SUCCEEDED.name());
@@ -999,7 +1098,7 @@ public class BatchService {
private synchronized void failJob(String accountId, String jobId, String reason) {
BatchJob job = getJobForAccount(accountId, jobId).orElse(null);
- if (job == null) {
+ if (job == null || isTerminalStatus(job.getStatus())) {
return;
}
job.setStatus(BatchStatus.FAILED.name());
@@ -1008,10 +1107,11 @@ public class BatchService {
putJobForAccount(accountId, job);
}
- private synchronized void transition(String accountId, String jobId, BatchStatus status, String reason, boolean started) {
+ private synchronized boolean transition(String accountId, String jobId, BatchStatus status,
+ String reason, boolean started) {
BatchJob job = getJobForAccount(accountId, jobId).orElse(null);
- if (job == null) {
- return;
+ if (job == null || isTerminalStatus(job.getStatus())) {
+ return false;
}
job.setStatus(status.name());
if (reason != null) {
@@ -1021,6 +1121,11 @@ public class BatchService {
job.setStartedAt(now());
}
putJobForAccount(accountId, job);
+ return true;
+ }
+
+ private boolean isTerminalStatus(String status) {
+ return TERMINAL_JOB_STATUSES.contains(status);
}
private void sleepQuietly() {
@@ -1419,6 +1524,9 @@ public class BatchService {
if (job.getArraySize() != null) {
ArrayRollup rollup = arrayRollup(job.getJobId());
summary.put("status", rollup.status());
+ if (job.getStatusReason() != null) {
+ summary.put("statusReason", job.getStatusReason());
+ }
if (rollup.startedAt() != null) {
summary.put("startedAt", rollup.startedAt());
}
@@ -1512,11 +1620,13 @@ public class BatchService {
for (BatchJob child : children) {
statusSummary.merge(child.getStatus(), 1, Integer::sum);
}
+ boolean allTerminal = !children.isEmpty()
+ && children.stream().allMatch(child -> isTerminalStatus(child.getStatus()));
boolean anyFailed = children.stream().anyMatch(c -> BatchStatus.FAILED.name().equals(c.getStatus()));
boolean allSucceeded = !children.isEmpty()
&& children.stream().allMatch(c -> BatchStatus.SUCCEEDED.name().equals(c.getStatus()));
String status;
- if (anyFailed) {
+ if (allTerminal && anyFailed) {
status = BatchStatus.FAILED.name();
} else if (allSucceeded) {
status = BatchStatus.SUCCEEDED.name();
@@ -1616,6 +1726,9 @@ public class BatchService {
if (job.getArraySize() != null) {
ArrayRollup rollup = arrayRollup(job.getJobId());
detail.put("status", rollup.status());
+ if (job.getStatusReason() != null) {
+ detail.put("statusReason", job.getStatusReason());
+ }
if (rollup.startedAt() != null) {
detail.put("startedAt", rollup.startedAt());
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/batch/BatchDockerRunnerTest.java b/src/test/java/io/github/hectorvent/floci/services/batch/BatchDockerRunnerTest.java
index c9421b4325..b4ea194609 100644
--- a/src/test/java/io/github/hectorvent/floci/services/batch/BatchDockerRunnerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/batch/BatchDockerRunnerTest.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.batch;
+import com.github.dockerjava.api.DockerClient;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.dns.EmbeddedDnsServer;
import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
@@ -8,15 +9,28 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;
import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
import io.github.hectorvent.floci.services.batch.model.BatchJob;
-import com.github.dockerjava.api.DockerClient;
+import io.github.hectorvent.floci.services.batch.model.BatchNodeExecution;
+import io.github.hectorvent.floci.services.batch.model.BatchRunResult;
import org.junit.jupiter.api.Test;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.ArgumentMatchers.isNull;
+import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
import static org.mockito.Mockito.RETURNS_SELF;
import static org.mockito.Mockito.mock;
@@ -99,6 +113,165 @@ class BatchDockerRunnerTest {
"io.floci.region", "us-east-1"));
}
+ @Test
+ void stopJobStopsAnInFlightContainer() throws Exception {
+ EmulatorConfig config = mock(EmulatorConfig.class);
+ EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);
+ EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);
+ when(config.services()).thenReturn(services);
+ when(services.batch()).thenReturn(batch);
+ when(batch.dockerNetwork()).thenReturn(Optional.empty());
+ when(config.port()).thenReturn(4566);
+ when(config.hostname()).thenReturn(Optional.empty());
+
+ CountDownLatch containerStarted = new CountDownLatch(1);
+ AtomicBoolean stopped = new AtomicBoolean();
+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
+ when(lifecycleManager.createAndStart(any())).thenAnswer(invocation -> {
+ containerStarted.countDown();
+ return new ContainerLifecycleManager.ContainerInfo("container-id", Map.of());
+ });
+ DockerClient dockerClient = mock(DockerClient.class, RETURNS_DEEP_STUBS);
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
+ when(dockerClient.inspectContainerCmd("container-id").exec().getState().getRunning())
+ .thenAnswer(invocation -> !stopped.get());
+ when(dockerClient.inspectContainerCmd("container-id").exec().getState().getExitCodeLong()).thenReturn(137L);
+ doAnswer(invocation -> {
+ stopped.set(true);
+ return null;
+ }).when(lifecycleManager).stopAndRemove(eq("container-id"), isNull());
+
+ ContainerBuilder containerBuilder = mock(ContainerBuilder.class);
+ ContainerBuilder.Builder builder = mock(ContainerBuilder.Builder.class, RETURNS_SELF);
+ when(containerBuilder.newContainer(anyString())).thenReturn(builder);
+ when(builder.build()).thenReturn(mock(ContainerSpec.class));
+ BatchDockerRunner runner = new BatchDockerRunner(containerBuilder, lifecycleManager,
+ mock(ContainerLogStreamer.class), config, mock(ContainerDetector.class));
+ BatchJob job = job();
+ AtomicReference<BatchRunResult> result = new AtomicReference<>();
+
+ Thread runThread = Thread.startVirtualThread(() -> result.set(runner.run(job, 1)));
+ assertTrue(containerStarted.await(2, TimeUnit.SECONDS));
+ runner.requestStop(job.getJobId());
+ runner.stopJob(job.getJobId());
+ runThread.join(2000);
+
+ assertFalse(runThread.isAlive());
+ assertEquals(137, result.get().exitCode());
+ verify(lifecycleManager).stopAndRemove("container-id", null);
+ }
+
+ @Test
+ void stopJobAfterRequestIsClearedDoesNotRecreateStopMarker() {
+ EmulatorConfig config = config(Optional.empty());
+ EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);
+ EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);
+ when(config.services()).thenReturn(services);
+ when(services.batch()).thenReturn(batch);
+ when(batch.dockerNetwork()).thenReturn(Optional.empty());
+ when(config.port()).thenReturn(4566);
+
+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
+ when(lifecycleManager.createAndStart(any())).thenReturn(
+ new ContainerLifecycleManager.ContainerInfo("container-id", Map.of()));
+ DockerClient dockerClient = mock(DockerClient.class, RETURNS_DEEP_STUBS);
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
+ when(dockerClient.inspectContainerCmd("container-id").exec().getState().getRunning()).thenReturn(false);
+ when(dockerClient.inspectContainerCmd("container-id").exec().getState().getExitCodeLong()).thenReturn(0L);
+ ContainerBuilder containerBuilder = mock(ContainerBuilder.class);
+ ContainerBuilder.Builder builder = mock(ContainerBuilder.Builder.class, RETURNS_SELF);
+ when(containerBuilder.newContainer(anyString())).thenReturn(builder);
+ when(builder.build()).thenReturn(mock(ContainerSpec.class));
+ BatchDockerRunner runner = new BatchDockerRunner(containerBuilder, lifecycleManager,
+ mock(ContainerLogStreamer.class), config, mock(ContainerDetector.class));
+
+ runner.requestStop("job-1");
+ runner.clearStopRequest("job-1");
+ runner.stopJob("job-1");
+ BatchRunResult result = runner.run(job(), 1);
+
+ assertEquals(0, result.exitCode());
+ verify(lifecycleManager).createAndStart(any());
+ }
+
+ @Test
+ void stopJobStopsEveryInFlightNodeContainer() throws Exception {
+ EmulatorConfig config = config(Optional.empty());
+ EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);
+ EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);
+ when(config.services()).thenReturn(services);
+ when(services.batch()).thenReturn(batch);
+ when(batch.dockerNetwork()).thenReturn(Optional.empty());
+ when(config.port()).thenReturn(4566);
+
+ CountDownLatch containersStarted = new CountDownLatch(2);
+ AtomicInteger containerSequence = new AtomicInteger();
+ Set<String> stoppedContainers = ConcurrentHashMap.newKeySet();
+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
+ when(lifecycleManager.createAndStart(any())).thenAnswer(invocation -> {
+ String containerId = "container-" + containerSequence.getAndIncrement();
+ containersStarted.countDown();
+ return new ContainerLifecycleManager.ContainerInfo(containerId, Map.of());
+ });
+ DockerClient dockerClient = mock(DockerClient.class, RETURNS_DEEP_STUBS);
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
+ for (int i = 0; i < 2; i++) {
+ String containerId = "container-" + i;
+ when(dockerClient.inspectContainerCmd(containerId).exec().getState().getRunning())
+ .thenAnswer(invocation -> !stoppedContainers.contains(containerId));
+ when(dockerClient.inspectContainerCmd(containerId).exec().getState().getExitCodeLong()).thenReturn(137L);
+ doAnswer(invocation -> {
+ stoppedContainers.add(containerId);
+ return null;
+ }).when(lifecycleManager).stopAndRemove(eq(containerId), isNull());
+ }
+ ContainerBuilder containerBuilder = mock(ContainerBuilder.class);
+ ContainerBuilder.Builder builder = mock(ContainerBuilder.Builder.class, RETURNS_SELF);
+ when(containerBuilder.newContainer(anyString())).thenReturn(builder);
+ when(builder.build()).thenReturn(mock(ContainerSpec.class));
+ BatchDockerRunner runner = new BatchDockerRunner(containerBuilder, lifecycleManager,
+ mock(ContainerLogStreamer.class), config, mock(ContainerDetector.class));
+ BatchJob job = job();
+ BatchNodeExecution firstNode = node(0);
+ BatchNodeExecution secondNode = node(1);
+ AtomicReference<BatchRunResult> firstResult = new AtomicReference<>();
+ AtomicReference<BatchRunResult> secondResult = new AtomicReference<>();
+
+ Thread firstThread = Thread.startVirtualThread(() -> firstResult.set(runner.run(job, 1, firstNode)));
+ Thread secondThread = Thread.startVirtualThread(() -> secondResult.set(runner.run(job, 1, secondNode)));
+ assertTrue(containersStarted.await(2, TimeUnit.SECONDS));
+ runner.requestStop(job.getJobId());
+ runner.stopJob(job.getJobId());
+ firstThread.join(2000);
+ secondThread.join(2000);
+
+ assertFalse(firstThread.isAlive());
+ assertFalse(secondThread.isAlive());
+ assertEquals(137, firstResult.get().exitCode());
+ assertEquals(137, secondResult.get().exitCode());
+ verify(lifecycleManager).stopAndRemove("container-0", null);
+ verify(lifecycleManager).stopAndRemove("container-1", null);
+ }
+
+ private BatchJob job() {
+ BatchJob job = new BatchJob();
+ job.setJobId("job-1");
+ job.setJobName("my-job");
+ job.setJobQueueName("my-queue");
+ job.setJobDefinitionName("my-def");
+ job.setContainerImage("busybox:stable");
+ job.setRegion("us-east-1");
+ job.setAccountId("000000000000");
+ return job;
+ }
+
+ private BatchNodeExecution node(int nodeIndex) {
+ BatchNodeExecution node = new BatchNodeExecution();
+ node.setNodeIndex(nodeIndex);
+ node.setContainerImage("busybox:stable");
+ return node;
+ }
+
private BatchDockerRunner runner(EmulatorConfig config, ContainerDetector detector) {
return new BatchDockerRunner(
mock(ContainerBuilder.class),
diff --git a/src/test/java/io/github/hectorvent/floci/services/batch/BatchIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/batch/BatchIntegrationTest.java
index e03e7a973a..fc4d33fd24 100644
--- a/src/test/java/io/github/hectorvent/floci/services/batch/BatchIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/batch/BatchIntegrationTest.java
@@ -80,6 +80,49 @@ class BatchIntegrationTest {
.body("jobSummaryList.find { it.jobId == '%s' }.container.taskArn".formatted(jobId), nullValue());
}
+ @Test
+ void cancelAndTerminateJobExposeAwsRoutes() {
+ String suffix = uniqueSuffix();
+ String queueArn = createQueue("batch-control-queue-" + suffix,
+ createComputeEnvironment("batch-control-ce-" + suffix));
+ String definitionArn = registerJobDefinition("batch-control-job-" + suffix, "[]", "[]");
+ String jobId = submit(queueArn, definitionArn, "batch-control-submit-" + suffix);
+
+ givenJson("""
+ {"jobId":"%s","reason":"No longer needed"}
+ """.formatted(jobId))
+ .when()
+ .post("/v1/canceljob")
+ .then()
+ .statusCode(200);
+
+ givenJson("""
+ {"jobId":"%s","reason":"Stop requested"}
+ """.formatted(jobId))
+ .when()
+ .post("/v1/terminatejob")
+ .then()
+ .statusCode(200);
+
+ givenJson("""
+ {"jobId":"%s"}
+ """.formatted(jobId))
+ .when()
+ .post("/v1/canceljob")
+ .then()
+ .statusCode(400)
+ .body("__type", equalTo("ClientException"));
+
+ givenJson("""
+ {"jobId":"missing-job","reason":"Stop requested"}
+ """)
+ .when()
+ .post("/v1/terminatejob")
+ .then()
+ .statusCode(400)
+ .body("__type", equalTo("ClientException"));
+ }
+
@Test
void jobDefinitionRevisionsAndDeregister() {
String suffix = uniqueSuffix();
diff --git a/src/test/java/io/github/hectorvent/floci/services/batch/BatchServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/batch/BatchServiceTest.java
index 1558979e3e..5767245170 100644
--- a/src/test/java/io/github/hectorvent/floci/services/batch/BatchServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/batch/BatchServiceTest.java
@@ -14,6 +14,7 @@ import io.github.hectorvent.floci.services.batch.model.BatchJobDefinition;
import io.github.hectorvent.floci.services.batch.model.BatchJobQueue;
import io.github.hectorvent.floci.services.batch.model.BatchNodeExecution;
import io.github.hectorvent.floci.services.batch.model.BatchRunResult;
+import io.github.hectorvent.floci.services.batch.model.BatchStatus;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
@@ -23,14 +24,19 @@ import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
import java.util.function.Predicate;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@@ -779,6 +785,265 @@ class BatchServiceTest {
assertEquals("ClientException", e.getErrorCode());
}
+ @Test
+ void cancelJobFailsRunnableJobWithRequestedReason() throws Exception {
+ InMemoryStorage<String, BatchJob> jobStore = new InMemoryStorage<>();
+ BatchService service = service("immediate", jobStore, mock(BatchDockerRunner.class));
+ BatchJob job = storedJob("cancel-runnable", BatchStatus.RUNNABLE);
+ jobStore.put(job.getJobId(), job);
+
+ service.cancelJob(json("""
+ {"jobId":"cancel-runnable","reason":"No longer needed"}
+ """));
+
+ JsonNode detail = service.describeJobs(json("""
+ {"jobs":["cancel-runnable"]}
+ """)).path("jobs").get(0);
+ assertEquals("FAILED", detail.path("status").asText());
+ assertEquals("No longer needed", detail.path("statusReason").asText());
+ assertTrue(detail.path("stoppedAt").asLong() > 0);
+ }
+
+ @Test
+ void cancelJobDoesNotStopRunningJob() throws Exception {
+ BatchDockerRunner runner = mock(BatchDockerRunner.class);
+ CountDownLatch runStarted = new CountDownLatch(1);
+ CountDownLatch releaseRun = new CountDownLatch(1);
+ when(runner.run(any(BatchJob.class), anyInt())).thenAnswer(invocation -> {
+ runStarted.countDown();
+ assertTrue(releaseRun.await(2, TimeUnit.SECONDS));
+ return new BatchRunResult(0, null, "log-stream", 1L, 2L, false);
+ });
+ BatchService service = dockerService(runner);
+ String queueArn = arrayReadyQueue(service);
+ String definitionArn = service.registerJobDefinition(json("""
+ {
+ "jobDefinitionName":"cancel-running-job",
+ "type":"container",
+ "containerProperties":{"image":"job:latest"}
+ }
+ """), REGION).path("jobDefinitionArn").asText();
+ String jobId = service.submitJob(json("""
+ {"jobName":"cancel-running","jobQueue":"%s","jobDefinition":"%s"}
+ """.formatted(queueArn, definitionArn)), REGION).path("jobId").asText();
+
+ assertTrue(runStarted.await(2, TimeUnit.SECONDS));
+ try {
+ service.cancelJob(json("""
+ {"jobId":"%s","reason":"Too late to cancel"}
+ """.formatted(jobId)));
+ JsonNode running = service.describeJobs(json("""
+ {"jobs":["%s"]}
+ """.formatted(jobId))).path("jobs").get(0);
+ assertEquals("RUNNING", running.path("status").asText());
+ verify(runner, never()).requestStop(jobId);
+ verify(runner, never()).stopJob(jobId);
+ } finally {
+ releaseRun.countDown();
+ }
+ assertNotNull(waitForJobStatus(service, jobId, "SUCCEEDED"));
+ }
+
+ @Test
+ void terminateJobStopsRunningContainerAndPreservesRequestedReason() throws Exception {
+ BatchDockerRunner runner = mock(BatchDockerRunner.class);
+ CountDownLatch runStarted = new CountDownLatch(1);
+ CountDownLatch stopRequested = new CountDownLatch(1);
+ when(runner.run(any(BatchJob.class), anyInt())).thenAnswer(invocation -> {
+ runStarted.countDown();
+ assertTrue(stopRequested.await(2, TimeUnit.SECONDS));
+ return new BatchRunResult(137, "Job terminated", "log-stream", 1L, 2L, false);
+ });
+ doAnswer(invocation -> {
+ stopRequested.countDown();
+ return null;
+ }).when(runner).requestStop(any(String.class));
+ BatchService service = dockerService(runner);
+ String queueArn = arrayReadyQueue(service);
+ String definitionArn = service.registerJobDefinition(json("""
+ {
+ "jobDefinitionName":"terminate-running-job",
+ "type":"container",
+ "containerProperties":{"image":"job:latest"}
+ }
+ """), REGION).path("jobDefinitionArn").asText();
+ String jobId = service.submitJob(json("""
+ {"jobName":"terminate-running","jobQueue":"%s","jobDefinition":"%s"}
+ """.formatted(queueArn, definitionArn)), REGION).path("jobId").asText();
+
+ assertTrue(runStarted.await(2, TimeUnit.SECONDS));
+ service.terminateJob(json("""
+ {"jobId":"%s","reason":"Operator requested shutdown"}
+ """.formatted(jobId)));
+
+ JsonNode failed = waitForJobStatus(service, jobId, "FAILED");
+ assertNotNull(failed);
+ assertEquals("Operator requested shutdown", failed.path("statusReason").asText());
+ JsonNode failedWithAttempt = waitForJobAttemptCount(service, jobId, 1);
+ assertNotNull(failedWithAttempt);
+ assertEquals(137, failedWithAttempt.path("attempts").get(0).path("container").path("exitCode").asInt());
+ assertEquals("Operator requested shutdown", failedWithAttempt.path("statusReason").asText());
+ verify(runner).requestStop(jobId);
+ verify(runner).stopJob(jobId);
+ }
+
+ @Test
+ void jobControlValidatesRequiredFieldsReasonLimitAndUnknownJobs() throws Exception {
+ InMemoryStorage<String, BatchJob> jobStore = new InMemoryStorage<>();
+ BatchService service = immediateService(jobStore);
+
+ AwsException missingJobId = assertThrows(AwsException.class, () -> service.cancelJob(json("""
+ {"reason":"No longer needed"}
+ """)));
+ assertEquals("ClientException", missingJobId.getErrorCode());
+ assertEquals("jobId is required", missingJobId.getMessage());
+
+ AwsException missingReason = assertThrows(AwsException.class, () -> service.terminateJob(json("""
+ {"jobId":"job-1"}
+ """)));
+ assertEquals("ClientException", missingReason.getErrorCode());
+ assertEquals("reason is required", missingReason.getMessage());
+
+ BatchJob boundaryJob = storedJob("boundary-job", BatchStatus.RUNNABLE);
+ jobStore.put(boundaryJob.getJobId(), boundaryJob);
+ String boundaryReason = "x".repeat(1024);
+ service.cancelJob(json("""
+ {"jobId":"boundary-job","reason":"%s"}
+ """.formatted(boundaryReason)));
+ JsonNode boundaryDetail = service.describeJobs(json("""
+ {"jobs":["boundary-job"]}
+ """)).path("jobs").get(0);
+ assertEquals(boundaryReason, boundaryDetail.path("statusReason").asText());
+
+ BatchJob overLimitJob = storedJob("over-limit-job", BatchStatus.RUNNABLE);
+ jobStore.put(overLimitJob.getJobId(), overLimitJob);
+ AwsException overLimit = assertThrows(AwsException.class, () -> service.cancelJob(json("""
+ {"jobId":"over-limit-job","reason":"%s"}
+ """.formatted("x".repeat(1025)))));
+ assertEquals("ClientException", overLimit.getErrorCode());
+
+ AwsException unknownJob = assertThrows(AwsException.class, () -> service.terminateJob(json("""
+ {"jobId":"unknown-job","reason":"Stop requested"}
+ """)));
+ assertEquals("ClientException", unknownJob.getErrorCode());
+ assertEquals("Job not found: unknown-job", unknownJob.getMessage());
+ }
+
+ @Test
+ void jobControlIsIdempotentForTerminalJobs() throws Exception {
+ InMemoryStorage<String, BatchJob> jobStore = new InMemoryStorage<>();
+ BatchDockerRunner runner = mock(BatchDockerRunner.class);
+ BatchService service = service("docker", jobStore, runner);
+ BatchJob succeeded = storedJob("succeeded-job", BatchStatus.SUCCEEDED);
+ succeeded.setStatusReason("Job completed successfully");
+ BatchJob failed = storedJob("failed-job", BatchStatus.FAILED);
+ failed.setStatusReason("Original failure");
+ jobStore.put(succeeded.getJobId(), succeeded);
+ jobStore.put(failed.getJobId(), failed);
+
+ service.cancelJob(json("""
+ {"jobId":"succeeded-job","reason":"Cancel again"}
+ """));
+ service.terminateJob(json("""
+ {"jobId":"failed-job","reason":"Terminate again"}
+ """));
+
+ JsonNode jobs = service.describeJobs(json("""
+ {"jobs":["succeeded-job","failed-job"]}
+ """)).path("jobs");
+ assertEquals("SUCCEEDED", jobs.get(0).path("status").asText());
+ assertEquals("Job completed successfully", jobs.get(0).path("statusReason").asText());
+ assertEquals("FAILED", jobs.get(1).path("status").asText());
+ assertEquals("Original failure", jobs.get(1).path("statusReason").asText());
+ verify(runner, never()).requestStop(any(String.class));
+ verify(runner, never()).stopJob(any(String.class));
+ }
+
+ @Test
+ void terminateMultiNodeJobStopsNodesAndPreservesRequestedReason() throws Exception {
+ BatchDockerRunner runner = mock(BatchDockerRunner.class);
+ CountDownLatch nodesStarted = new CountDownLatch(2);
+ CountDownLatch stopRequested = new CountDownLatch(1);
+ when(runner.run(any(BatchJob.class), anyInt(), any(BatchNodeExecution.class)))
+ .thenAnswer(invocation -> {
+ BatchNodeExecution node = invocation.getArgument(2);
+ nodesStarted.countDown();
+ assertTrue(stopRequested.await(2, TimeUnit.SECONDS));
+ return new BatchRunResult(137, "Job terminated", "log-" + node.getNodeIndex(),
+ 1L, 2L, false);
+ });
+ doAnswer(invocation -> {
+ stopRequested.countDown();
+ return null;
+ }).when(runner).requestStop(any(String.class));
+ BatchService service = dockerService(runner);
+ String queueArn = arrayReadyQueue(service);
+ String definitionArn = service.registerJobDefinition(json("""
+ {
+ "jobDefinitionName":"terminate-mnp-job",
+ "type":"multinode",
+ "nodeProperties":{
+ "numNodes":2,
+ "mainNode":0,
+ "nodeRangeProperties":[{"targetNodes":"0:1","container":{"image":"worker:latest"}}]
+ }
+ }
+ """), REGION).path("jobDefinitionArn").asText();
+ String jobId = service.submitJob(json("""
+ {"jobName":"terminate-mnp","jobQueue":"%s","jobDefinition":"%s"}
+ """.formatted(queueArn, definitionArn)), REGION).path("jobId").asText();
+
+ assertTrue(nodesStarted.await(2, TimeUnit.SECONDS));
+ service.terminateJob(json("""
+ {"jobId":"%s","reason":"Stop every node"}
+ """.formatted(jobId)));
+
+ JsonNode failedWithAttempt = waitForJobAttemptCount(service, jobId, 1);
+ assertNotNull(failedWithAttempt);
+ assertEquals("FAILED", failedWithAttempt.path("status").asText());
+ assertEquals("Stop every node", failedWithAttempt.path("statusReason").asText());
+ JsonNode nodes = service.listJobs(json("""
+ {"multiNodeJobId":"%s"}
+ """.formatted(jobId))).path("jobSummaryList");
+ assertEquals(137, nodes.get(0).path("container").path("exitCode").asInt());
+ assertEquals(137, nodes.get(1).path("container").path("exitCode").asInt());
+ verify(runner).requestStop(jobId);
+ verify(runner).stopJob(jobId);
+ }
+
+ @Test
+ void terminateArrayParentFailsAndStopsEveryNonterminalChild() throws Exception {
+ InMemoryStorage<String, BatchJob> jobStore = new InMemoryStorage<>();
+ BatchDockerRunner runner = mock(BatchDockerRunner.class);
+ BatchService service = service("docker", jobStore, runner);
+ BatchJob parent = storedJob("array-parent", BatchStatus.SUBMITTED);
+ parent.setArraySize(2);
+ BatchJob first = storedJob("array-parent:0", BatchStatus.RUNNABLE);
+ first.setArrayJobId(parent.getJobId());
+ first.setArrayIndex(0);
+ BatchJob second = storedJob("array-parent:1", BatchStatus.RUNNING);
+ second.setArrayJobId(parent.getJobId());
+ second.setArrayIndex(1);
+ jobStore.put(parent.getJobId(), parent);
+ jobStore.put(first.getJobId(), first);
+ jobStore.put(second.getJobId(), second);
+
+ service.terminateJob(json("""
+ {"jobId":"array-parent","reason":"Stop the array"}
+ """));
+
+ JsonNode detail = service.describeJobs(json("""
+ {"jobs":["array-parent","array-parent:0","array-parent:1"]}
+ """)).path("jobs");
+ assertEquals("FAILED", detail.get(0).path("status").asText());
+ assertEquals("Stop the array", detail.get(0).path("statusReason").asText());
+ assertEquals("FAILED", detail.get(1).path("status").asText());
+ assertEquals("FAILED", detail.get(2).path("status").asText());
+ verify(runner).requestStop("array-parent:1");
+ verify(runner, never()).stopJob("array-parent:0");
+ verify(runner).stopJob("array-parent:1");
+ }
+
private String arrayReadyQueue(BatchService service) throws Exception {
String suffix = UUID.randomUUID().toString();
String computeArn = service.createComputeEnvironment(json("""
@@ -794,18 +1059,23 @@ class BatchServiceTest {
}
private BatchService dockerService(BatchDockerRunner runner) {
+ return service("docker", new InMemoryStorage<String, BatchJob>(), runner);
+ }
+
+ private BatchService service(String runnerMode, StorageBackend<String, BatchJob> jobStore,
+ BatchDockerRunner runner) {
EmulatorConfig config = mock(EmulatorConfig.class);
EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);
EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);
when(config.services()).thenReturn(services);
when(services.batch()).thenReturn(batch);
- when(batch.runnerMode()).thenReturn("docker");
+ when(batch.runnerMode()).thenReturn(runnerMode);
return new BatchService(
new InMemoryStorage<String, BatchJobDefinition>(),
new InMemoryStorage<String, BatchJobQueue>(),
new InMemoryStorage<String, BatchComputeEnvironment>(),
- new InMemoryStorage<String, BatchJob>(),
+ jobStore,
new RegionResolver(REGION, ACCOUNT),
config,
objectMapper,
@@ -813,22 +1083,21 @@ class BatchServiceTest {
}
private BatchService immediateService(StorageBackend<String, BatchJob> jobStore) {
- EmulatorConfig config = mock(EmulatorConfig.class);
- EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);
- EmulatorConfig.BatchServiceConfig batch = mock(EmulatorConfig.BatchServiceConfig.class);
- when(config.services()).thenReturn(services);
- when(services.batch()).thenReturn(batch);
- when(batch.runnerMode()).thenReturn("immediate");
+ return service("immediate", jobStore, mock(BatchDockerRunner.class));
+ }
- return new BatchService(
- new InMemoryStorage<String, BatchJobDefinition>(),
- new InMemoryStorage<String, BatchJobQueue>(),
- new InMemoryStorage<String, BatchComputeEnvironment>(),
- jobStore,
- new RegionResolver(REGION, ACCOUNT),
- config,
- objectMapper,
- mock(BatchDockerRunner.class));
+ private BatchJob storedJob(String jobId, BatchStatus status) {
+ BatchJob job = new BatchJob();
+ job.setJobId(jobId);
+ job.setJobArn("arn:aws:batch:us-east-1:" + ACCOUNT + ":job/" + jobId);
+ job.setJobName(jobId);
+ job.setJobQueue("queue");
+ job.setJobDefinition("definition");
+ job.setStatus(status.name());
+ job.setCreatedAt(1L);
+ job.setRegion(REGION);
+ job.setAccountId(ACCOUNT);
+ return job;
}
private ObjectNode json(String body) throws Exception {
@@ -848,6 +1117,19 @@ class BatchServiceTest {
return null;
}
+ private JsonNode waitForJobAttemptCount(BatchService service, String jobId, int attemptCount) throws Exception {
+ ObjectNode request = objectMapper.createObjectNode();
+ request.putArray("jobs").add(jobId);
+ for (int i = 0; i < 100; i++) {
+ JsonNode job = service.describeJobs(request).path("jobs").get(0);
+ if (job != null && job.path("attempts").size() == attemptCount) {
+ return job;
+ }
+ Thread.sleep(10);
+ }
+ return null;
+ }
+
private static final class ReverseScanJobStorage implements StorageBackend<String, BatchJob> {
private final LinkedHashMap<String, BatchJob> store = new LinkedHashMap<>();