Light, fluffy, and always free - The AWS Local Emulator alternative
feat(sagemaker): support GPU-backed training jobs (#4015)
Austin We14221fe621b
diff --git a/docs/services/sagemaker.md b/docs/services/sagemaker.mdindex ddba8dd45a..f276e80fe5 100644--- a/docs/services/sagemaker.md+++ b/docs/services/sagemaker.md@@ -44,10 +44,48 @@ Floci emulates SageMaker control-plane APIs and runs real Docker containers for`CreateTrainingJob` starts `AlgorithmSpecification.TrainingImage` with command `train` unless `ContainerEntrypoint`/`ContainerArguments` are supplied. Floci writes SageMaker config files under `/opt/ml/input/config`, downloads channel data from S3 into `/opt/ml/input/data/<channel>`, waits for container exit, and uploads `/opt/ml/model` as `model.tar.gz` under `OutputDataConfig.S3OutputPath/<TrainingJobName>/output/`.+## GPU-backed training++On AWS, `ResourceConfig.InstanceType` *is* the hardware: `ml.g5.xlarge` provisions a machine that physically has one A10G. Floci has one host with whatever cards it has, so a GPU instance type is served by substituting a local device. That substitution is off by default and the SDK request is unchanged either way.++How many accelerators each `ml.*` type carries is a property of AWS, so it ships with Floci in `sagemaker/instance-type-catalog.yaml`. Configuration covers only the local decision of which of this machine's devices may be handed out:++```yaml+floci:+ services:+ sagemaker:+ gpu:+ enabled: true+ mode: cdi # cdi | device-ids | count+ devices:+ - nvidia.com/gpu=GPU-0e1f2a3b+```++| Setting | Meaning |+| --- | --- |+| `enabled` | Off by default. While off, every training container is CPU-only, exactly as before. |+| `mode` | How the request reaches the daemon. `cdi` names [Container Device Interface](https://github.com/cncf-tags/container-device-interface) devices, `device-ids` uses daemon device ids, and `count` asks for a number and lets the daemon choose. |+| `devices` | The devices Floci may use, for `cdi` and `device-ids`. Unset allows none, so a shared machine does not hand over every GPU by default. Ignored by `count`. |++`mode` defaults to `cdi` because daemons differ: Podman resolves only CDI and will accept a count request while attaching no device ([containers/podman#22645](https://github.com/containers/podman/issues/22645)). Choosing CDI means a misconfiguration fails the container start rather than quietly training on CPU. Use `count` or `device-ids` against Docker.++Floci does not reserve configured devices across concurrent training jobs. In `cdi` and `device-ids` modes, each job takes the first devices it needs from the allowlist, so two simultaneous one-GPU jobs can target the same local card and contend for memory.++A training job fails with an explanatory `FailureReason`, rather than running on CPU, when:++- the instance type belongs to a GPU family (`ml.g*`, `ml.p*`, `ml.inf*`, `ml.trn*`) that the catalog does not list, so its accelerator count is unknown+- fewer devices are allowed than the instance type needs+- `cdi` or `device-ids` is selected with no `devices` configured+- `InstanceCount` is above one, since Floci runs a job as a single container++Only the accelerator is substituted. An instance type also implies vCPU and memory, which Floci does not emulate for any service: EC2 instance types are metadata for `DescribeInstanceTypes` and do not size containers, and RDS records `DBInstanceClass` without acting on it. Resource limits are applied only where an API supplies an explicit number, as Lambda's `MemorySize` does.+## Endpoint hosting`CreateEndpoint` starts the model image as a long-lived Docker container with command `serve`, port `8080`, `/ping` health checks, and `/invocations` runtime proxying. `ModelDataUrl` artifacts are downloaded from S3 and placed in `/opt/ml/model`.+Endpoint containers are CPU-only for now; the `gpu` settings above apply to training jobs.+## Examples```pythondiff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex fde2bc5881..12fe844f1f 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -1138,6 +1138,57 @@ public interface EmulatorConfig {boolean enabled();Optional<String> dockerNetwork();++ GpuConfig gpu();++ /**+ * Whether, and how, this host lends its accelerators to training containers.+ *+ * <p>How many GPUs an instance type has is an AWS fact and lives in the shipped+ * catalog; this covers only the local decision of which devices Floci may use.+ */+ interface GpuConfig {+ /**+ * Off by default, so an existing deployment keeps launching CPU-only+ * containers exactly as before.+ */+ @WithDefault("false")+ boolean enabled();++ /**+ * How the device request is expressed on the wire, because daemons disagree.+ *+ * <p>Defaults to {@code cdi}: Podman resolves only that form, and accepts the+ * {@code count} form while attaching no device+ * (containers/podman#22645). Preferring CDI means a misconfiguration fails+ * the container start instead of silently training on CPU. Use {@code count}+ * or {@code device-ids} against Docker.+ */+ @WithDefault("cdi")+ GpuRequestMode mode();++ /**+ * The devices Floci may hand out: CDI names such as+ * {@code nvidia.com/gpu=GPU-<uuid>} for {@code cdi} mode, or daemon device ids+ * for {@code device-ids} mode.+ *+ * <p>Unset allows no device, so a training job in those modes fails rather than+ * starting. Defaulting to "every device the daemon exposes" would be the wrong+ * behaviour on a machine sharing GPUs with other workloads.+ *+ * <p>Ignored in {@code count} mode, where the daemon does the choosing.+ *+ * <p>Optional rather than a defaulted list because SmallRye rejects an empty+ * string as a collection default: unset simply means no device is allowed.+ */+ Optional<List<String>> devices();+ }++ enum GpuRequestMode {+ CDI,+ DEVICE_IDS,+ COUNT+ }}interface CodeDeployServiceConfig {diff --git a/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuResolver.java b/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuResolver.javanew file mode 100644index 0000000000..80449ac964--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuResolver.java@@ -0,0 +1,130 @@+package io.github.hectorvent.floci.services.sagemaker;++import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.config.EmulatorConfig.SageMakerServiceConfig.GpuRequestMode;+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.inject.Inject;++import java.util.List;+import java.util.Optional;++/**+ * Turns a SageMaker {@code InstanceType} into a device request on a container, or into a+ * clear refusal.+ *+ * <p>Currently used by training jobs. It is kept separate from the training runner so the+ * same rules can be reused if GPU-backed endpoint hosting is added later.+ *+ * <p>The design rule throughout: never answer a GPU request with a CPU container. A job+ * that asks for {@code ml.g5.xlarge}, trains on CPU and reports {@code Completed} leaves+ * behind an artifact that looks legitimate, which is worse than a failure. Every path that+ * cannot honour the request raises {@link UnsupportedResourceException} instead.+ *+ * <p>Only the accelerator is mapped. An instance type also implies vCPU and memory, and+ * Floci does not emulate those: the EC2 catalog is metadata for DescribeInstanceTypes and+ * never sizes a container, and RDS keeps DBInstanceClass without acting on it. Limits are+ * applied only where an AWS API hands over an explicit number, as Lambda's MemorySize and+ * ECS task memory do.+ */+@ApplicationScoped+public class SageMakerGpuResolver {++ private final EmulatorConfig config;+ private final SageMakerInstanceTypeCatalog catalog;++ @Inject+ public SageMakerGpuResolver(EmulatorConfig config, SageMakerInstanceTypeCatalog catalog) {+ this.config = config;+ this.catalog = catalog;+ }++ /**+ * Raised when the request names hardware this host cannot stand in for. Callers surface+ * the message as the AWS failure reason, so it has to say what is wrong and what would+ * fix it.+ */+ public static class UnsupportedResourceException extends RuntimeException {+ public UnsupportedResourceException(String message) {+ super(message);+ }+ }++ /**+ * Applies a device request to {@code builder} when the instance type calls for one.+ *+ * @param instanceType the requested {@code ml.*} type, as given in the AWS request+ * @param instanceCount instances requested; anything above one is refused, since a+ * single container is not distributed training+ * @throws UnsupportedResourceException when the request cannot be honoured+ */+ public void applyTo(ContainerBuilder.Builder builder, String instanceType, int instanceCount) {+ EmulatorConfig.SageMakerServiceConfig.GpuConfig gpu = config.services().sagemaker().gpu();+ if (!gpu.enabled()) {+ // Feature off: behave exactly as before, including for GPU instance types.+ // Turning it on is what opts a deployment into the stricter rules below.+ return;+ }++ Optional<Integer> catalogued = catalog.gpuCount(instanceType);+ if (catalogued.isEmpty()) {+ if (SageMakerInstanceTypeCatalog.isAcceleratorFamily(instanceType)) {+ throw new UnsupportedResourceException(+ "Instance type " + instanceType + " is a GPU family but is not in Floci's "+ + "SageMaker instance type catalog, so the number of accelerators it "+ + "should receive is unknown. Add it to the catalog or choose a "+ + "listed type.");+ }+ // An unlisted non-accelerator type is a CPU job, which needs nothing from us.+ return;+ }++ int requested = catalogued.get();+ if (requested == 0) {+ return;+ }+ if (instanceCount > 1) {+ throw new UnsupportedResourceException(+ "Floci runs a training job as a single container, so InstanceCount " + instanceCount+ + " with GPU instance type " + instanceType + " cannot be honoured. Use "+ + "InstanceCount 1.");+ }++ List<String> devices = gpu.devices().orElse(List.of());+ GpuRequestMode mode = gpu.mode();+ if (mode != GpuRequestMode.COUNT && !devices.isEmpty() && devices.size() < requested) {+ throw new UnsupportedResourceException(+ "Instance type " + instanceType + " needs " + requested + " GPUs but only "+ + devices.size() + " device(s) are allowed by "+ + "floci.services.sagemaker.gpu.devices.");+ }++ switch (mode) {+ case CDI -> {+ requireDevicesConfigured(devices, "cdi", instanceType);+ builder.withCdiDevices(devices.subList(0, requested));+ }+ case DEVICE_IDS -> {+ requireDevicesConfigured(devices, "device-ids", instanceType);+ builder.withGpuDeviceIds(devices.subList(0, requested));+ }+ // The daemon picks the devices, so no allowlist applies.+ case COUNT -> builder.withGpuCount(requested);+ }+ }++ /**+ * Both explicit-device modes need to be told which devices exist. Defaulting to "all+ * of them" would quietly hand Floci every GPU on a machine that is usually sharing+ * them with something else.+ */+ private static void requireDevicesConfigured(List<String> devices, String mode, String instanceType) {+ if (devices.isEmpty()) {+ throw new UnsupportedResourceException(+ "GPU support is enabled in " + mode + " mode but "+ + "floci.services.sagemaker.gpu.devices is empty, so there is no device to "+ + "give " + instanceType + ". List the devices Floci may use, or switch to "+ + "count mode to let the daemon choose.");+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerInstanceTypeCatalog.java b/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerInstanceTypeCatalog.javanew file mode 100644index 0000000000..268f818540--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerInstanceTypeCatalog.java@@ -0,0 +1,137 @@+package io.github.hectorvent.floci.services.sagemaker;++import com.fasterxml.jackson.annotation.JsonIgnoreProperties;+import com.fasterxml.jackson.databind.ObjectMapper;+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;+import io.quarkus.runtime.annotations.RegisterForReflection;+import jakarta.enterprise.context.ApplicationScoped;++import java.io.IOException;+import java.io.InputStream;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.Set;++/**+ * How many accelerators each SageMaker {@code ml.*} instance type carries.+ *+ * <p>This is shipped data rather than configuration for the same reason+ * {@code Ec2InstanceTypeCatalog} is: the GPU count of an {@code ml.g5.xlarge} is a fact+ * about AWS, identical for everyone, and not something an operator should have to+ * restate. What the local machine is willing to hand over is the separate, genuinely+ * local decision, and lives under {@code floci.services.sagemaker.gpu}.+ *+ * <p>Deliberately not merged into the EC2 catalog: {@code ml.*} is its own namespace and+ * these types must never appear in {@code DescribeInstanceTypes}.+ */+@ApplicationScoped+public class SageMakerInstanceTypeCatalog {++ private static final String CATALOG_RESOURCE_NAME = "sagemaker/instance-type-catalog.yaml";+ private static final ObjectMapper YAML_MAPPER = new ObjectMapper(new YAMLFactory());++ /**+ * Families whose members carry accelerators. Used only to decide whether an unknown+ * type should be refused rather than run on CPU: a caller naming an unlisted+ * {@code ml.g*} clearly wanted a GPU, and silently giving them CPU would produce a+ * job that reports Completed with a legitimate-looking artifact.+ *+ * <p>Never used to infer a count. Counts vary within a family, which is why the+ * catalog is a table.+ */+ private static final Set<String> ACCELERATOR_FAMILY_PREFIXES =+ Set.of("ml.g", "ml.p", "ml.inf", "ml.trn");++ private volatile Loaded loaded;++ public SageMakerInstanceTypeCatalog() {+ // Load lazily, matching Ec2InstanceTypeCatalog: a deployment that never touches+ // SageMaker should not pay for resource loading at bean construction.+ }++ SageMakerInstanceTypeCatalog(Catalog catalog) {+ this.loaded = new Loaded(catalog);+ }++ /**+ * The accelerator count for a type, or empty when the catalog does not describe it.+ * An empty result is not "no GPU": see {@link #isAcceleratorFamily(String)}.+ */+ public Optional<Integer> gpuCount(String instanceType) {+ if (instanceType == null || instanceType.isBlank()) {+ return Optional.empty();+ }+ CatalogInstanceType found = loaded().byName.get(instanceType.trim());+ return found == null ? Optional.empty() : Optional.of(found.gpuCount);+ }++ /**+ * Whether the type belongs to a family that carries accelerators on AWS, regardless of+ * whether this catalog lists the specific size.+ *+ * <p>Case sensitive, like {@link #gpuCount(String)} and like EC2's {@code find} and+ * {@code familyOf}: AWS instance types are exact tokens, so folding case here would make+ * this disagree with the catalog lookup and answer "GPU family" for a spelling the lookup+ * then reports as missing from the catalog.+ */+ public static boolean isAcceleratorFamily(String instanceType) {+ if (instanceType == null || instanceType.isBlank()) {+ return false;+ }+ String normalized = instanceType.trim();+ return ACCELERATOR_FAMILY_PREFIXES.stream().anyMatch(normalized::startsWith);+ }++ private Loaded loaded() {+ Loaded current = loaded;+ if (current == null) {+ synchronized (this) {+ current = loaded;+ if (current == null) {+ current = new Loaded(readCatalog());+ loaded = current;+ }+ }+ }+ return current;+ }++ private static Catalog readCatalog() {+ try (InputStream in = Thread.currentThread().getContextClassLoader()+ .getResourceAsStream(CATALOG_RESOURCE_NAME)) {+ if (in == null) {+ throw new IllegalStateException("Missing catalog resource " + CATALOG_RESOURCE_NAME);+ }+ return YAML_MAPPER.readValue(in, Catalog.class);+ } catch (IOException e) {+ throw new IllegalStateException("Could not read " + CATALOG_RESOURCE_NAME, e);+ }+ }++ private static final class Loaded {+ private final Map<String, CatalogInstanceType> byName;++ private Loaded(Catalog catalog) {+ Map<String, CatalogInstanceType> index = new LinkedHashMap<>();+ for (CatalogInstanceType type : catalog.instanceTypes) {+ index.put(type.instanceType, type);+ }+ this.byName = Map.copyOf(index);+ }+ }++ @RegisterForReflection+ @JsonIgnoreProperties(ignoreUnknown = true)+ public static final class Catalog {+ public List<CatalogInstanceType> instanceTypes = List.of();+ }++ @RegisterForReflection+ @JsonIgnoreProperties(ignoreUnknown = true)+ public static final class CatalogInstanceType {+ public String instanceType;+ public int gpuCount;+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerTrainingRunner.java b/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerTrainingRunner.javaindex 76c979dc3f..b7b696fee7 100644--- a/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerTrainingRunner.java+++ b/src/main/java/io/github/hectorvent/floci/services/sagemaker/SageMakerTrainingRunner.java@@ -54,6 +54,7 @@ public class SageMakerTrainingRunner implements ContainerTeardown, Resettable {private final ContainerDetector containerDetector;private final S3Service s3Service;private final ObjectMapper mapper;+ private final SageMakerGpuResolver gpuResolver;// Replaced by afterReset() after a state reset, whose container teardown shuts this pool down.private volatile ExecutorService executor = Executors.newCachedThreadPool();private final ConcurrentHashMap<String, String> containers = new ConcurrentHashMap<>();@@ -66,7 +67,8 @@ public class SageMakerTrainingRunner implements ContainerTeardown, Resettable {@Injectpublic SageMakerTrainingRunner(ContainerBuilder containerBuilder, ContainerLifecycleManager lifecycleManager,ContainerLogStreamer logStreamer, EmulatorConfig config,- ContainerDetector containerDetector, S3Service s3Service, ObjectMapper mapper) {+ ContainerDetector containerDetector, S3Service s3Service, ObjectMapper mapper,+ SageMakerGpuResolver gpuResolver) {this.containerBuilder = containerBuilder;this.lifecycleManager = lifecycleManager;this.logStreamer = logStreamer;@@ -74,6 +76,7 @@ public class SageMakerTrainingRunner implements ContainerTeardown, Resettable {this.containerDetector = containerDetector;this.s3Service = s3Service;this.mapper = mapper;+ this.gpuResolver = gpuResolver;}public void runAsync(TrainingJobResource job, SageMakerService service) {@@ -95,6 +98,10 @@ public class SageMakerTrainingRunner implements ContainerTeardown, Resettable {.withEmbeddedDns().withLogRotation();SageMakerEndpointManager.applyEntrypoint(builder, job.algorithmSpecification, "train");+ // Throws when the requested instance type asks for hardware this host cannot+ // stand in for, so the job fails with a reason rather than quietly training+ // on CPU and producing an artifact that looks legitimate.+ gpuResolver.applyTo(builder, instanceType(job), instanceCount(job));ContainerSpec spec = builder.build();containerId = lifecycleManager.create(spec);containers.put(job.trainingJobName, containerId);@@ -324,6 +331,16 @@ public class SageMakerTrainingRunner implements ContainerTeardown, Resettable {}}+ private static String instanceType(TrainingJobResource job) {+ return SageMakerEndpointManager.string(job.resourceConfig.get("InstanceType"));+ }++ /** Absent or unparseable means one, matching how a single-instance job is described. */+ private static int instanceCount(TrainingJobResource job) {+ Object value = job.resourceConfig.get("InstanceCount");+ return value instanceof Number n ? n.intValue() : 1;+ }+private Duration timeout(TrainingJobResource job) {Object value = job.stoppingCondition.get("MaxRuntimeInSeconds");if (value instanceof Number n) {diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex ca95bf4c36..673803bd4c 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -41,12 +41,13 @@ quarkus:add-all-charsets: trueresources:# The Pricing service reads its bundled snapshot, EC2 reads its image- # catalog (ec2/image-catalog.yaml), AppSync's GraphQL parser reads its- # message bundles, and API Gateway's Velocity engine reads its default- # properties at runtime via ClassLoader#getResourceAsStream. Quarkus's- # static resource analysis cannot detect these dynamically-built paths,- # so they are registered explicitly to ensure GraalVM includes every- # file in the native image.+ # catalog (ec2/image-catalog.yaml), SageMaker reads its instance type+ # catalog (sagemaker/instance-type-catalog.yaml), AppSync's GraphQL+ # parser reads its message bundles, and API Gateway's Velocity engine+ # reads its default properties at runtime via+ # ClassLoader#getResourceAsStream. Quarkus's static resource analysis+ # cannot detect these dynamically-built paths, so they are registered+ # explicitly to ensure GraalVM includes every file in the native image.# The IAM managed-policy catalog (iam/*) is not listed on purpose.# AwsManagedPolicies parses it while the image is built and the parsed# catalog lives in the image heap, so the 3.5 MB of source files would@@ -55,6 +56,7 @@ quarkus:- pricing-snapshots/**- ui/*.html- ec2/*.yaml+ - sagemaker/*.yaml- i18n/*.properties- org/apache/velocity/runtime/defaults/*.properties# DriverManager finds the JDBC drivers through ServiceLoader, and the MySQL@@ -623,6 +625,18 @@ floci:sagemaker:enabled: true# docker-network: floci-network+ gpu:+ # Off by default: an existing deployment keeps launching CPU-only+ # containers. How many GPUs an ml.* type has is shipped in+ # sagemaker/instance-type-catalog.yaml; this is only about which of+ # this machine's devices Floci may hand out.+ enabled: false+ # cdi | device-ids | count. Podman resolves only cdi, and accepts+ # count while attaching no device, so cdi fails loudly instead.+ mode: cdi+ # Required for cdi and device-ids modes; unset allows no device.+ # devices:+ # - nvidia.com/gpu=GPU-<uuid>batch:enabled: truerunner-mode: immediatediff --git a/src/main/resources/sagemaker/instance-type-catalog.yaml b/src/main/resources/sagemaker/instance-type-catalog.yamlnew file mode 100644index 0000000000..44559eb12b--- /dev/null+++ b/src/main/resources/sagemaker/instance-type-catalog.yaml@@ -0,0 +1,83 @@+# Accelerator facts for SageMaker ml.* instance types.+#+# Kept separate from ec2/instance-type-catalog.yaml because ml.* is its own+# namespace: these types are not EC2 instance types and must never surface in+# DescribeInstanceTypes.+#+# gpuCount is a property of the AWS instance type, not of the machine Floci+# runs on, which is why it is shipped data rather than configuration. What the+# local host is willing to hand over is a separate operator decision under+# floci.services.sagemaker.gpu.+#+# Like the EC2 catalog, this ships the sizes worth supporting rather than the+# full AWS inventory. A GPU-family type that is missing here is refused rather+# than guessed at, so adding a family means adding its rows.+#+# The counts below happen to be 1, 4 and 8 because that is what the g4dn, g5,+# p3 and p4d families offer. That is not a rule to extrapolate from: other AWS+# accelerated families carry exactly two (g3.8xlarge, g5g.16xlarge,+# g7.12xlarge), and a size is not a multiplier either, since g4dn.16xlarge has+# one GPU while the smaller g4dn.12xlarge has four. Hence a table.+instanceTypes:+ # CPU families. Listed so the resolver can tell "this type has no+ # accelerator" apart from "this type is unknown to Floci".+ - instanceType: ml.m5.large+ gpuCount: 0+ - instanceType: ml.m5.xlarge+ gpuCount: 0+ - instanceType: ml.m5.2xlarge+ gpuCount: 0+ - instanceType: ml.c5.large+ gpuCount: 0+ - instanceType: ml.c5.xlarge+ gpuCount: 0+ - instanceType: ml.c5.2xlarge+ gpuCount: 0+ - instanceType: ml.t3.medium+ gpuCount: 0+ - instanceType: ml.t3.large+ gpuCount: 0++ # G4dn: 1x NVIDIA T4, except the 12xlarge (4) and metal (8).+ - instanceType: ml.g4dn.xlarge+ gpuCount: 1+ - instanceType: ml.g4dn.2xlarge+ gpuCount: 1+ - instanceType: ml.g4dn.4xlarge+ gpuCount: 1+ - instanceType: ml.g4dn.8xlarge+ gpuCount: 1+ - instanceType: ml.g4dn.12xlarge+ gpuCount: 4+ - instanceType: ml.g4dn.16xlarge+ gpuCount: 1++ # G5: 1x NVIDIA A10G, except the 12xlarge (4) and 48xlarge (8).+ - instanceType: ml.g5.xlarge+ gpuCount: 1+ - instanceType: ml.g5.2xlarge+ gpuCount: 1+ - instanceType: ml.g5.4xlarge+ gpuCount: 1+ - instanceType: ml.g5.8xlarge+ gpuCount: 1+ - instanceType: ml.g5.12xlarge+ gpuCount: 4+ - instanceType: ml.g5.16xlarge+ gpuCount: 1+ - instanceType: ml.g5.24xlarge+ gpuCount: 4+ - instanceType: ml.g5.48xlarge+ gpuCount: 8++ # P3: NVIDIA V100.+ - instanceType: ml.p3.2xlarge+ gpuCount: 1+ - instanceType: ml.p3.8xlarge+ gpuCount: 4+ - instanceType: ml.p3.16xlarge+ gpuCount: 8++ # P4d: 8x NVIDIA A100.+ - instanceType: ml.p4d.24xlarge+ gpuCount: 8diff --git a/src/test/java/io/github/hectorvent/floci/config/NativeImageResourceIncludesTest.java b/src/test/java/io/github/hectorvent/floci/config/NativeImageResourceIncludesTest.javanew file mode 100644index 0000000000..8be96e96a6--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/config/NativeImageResourceIncludesTest.java@@ -0,0 +1,143 @@+package io.github.hectorvent.floci.config;++import org.junit.jupiter.api.Test;++import java.io.IOException;+import java.io.UncheckedIOException;+import java.nio.file.FileSystems;+import java.nio.file.Files;+import java.nio.file.Path;+import java.nio.file.PathMatcher;+import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import java.util.Set;+import java.util.regex.Matcher;+import java.util.regex.Pattern;+import java.util.stream.Stream;++import static org.junit.jupiter.api.Assertions.assertTrue;++/**+ * Guards the GraalVM native-image build against a classpath resource that is read at runtime but+ * never packaged.+ *+ * <p>Quarkus's static analysis finds an inline literal such as+ * {@code getResourceAsStream("/redshift/bootstrap-catalog.sql")}, so those need no registration.+ * It cannot follow a constant, so {@code getResourceAsStream(CATALOG_RESOURCE_NAME)} leaves the+ * file out of the image and the first read fails at runtime with a missing-resource error. The+ * remedy is a matching glob under {@code quarkus.native.resources.includes} in+ * {@code application.yml}.+ *+ * <p>Like {@link NativeImageRuntimeInitializationTest}, this failure only surfaces in the slow+ * native build and never in the JVM CI jobs, so it moves the check into ordinary CI. Regression+ * context: the SageMaker instance type catalog shipped reading+ * {@code sagemaker/instance-type-catalog.yaml} through a constant without the matching include,+ * which would have failed every GPU training job in the native image while passing every JVM test.+ */+class NativeImageResourceIncludesTest {++ private static final Path MAIN_SOURCE_ROOT = Path.of("src", "main", "java");+ private static final Path APPLICATION_YML = Path.of("src", "main", "resources", "application.yml");+ private static final Path MAIN_RESOURCE_ROOT = Path.of("src", "main", "resources");++ private static final String JAVA_SUFFIX = ".java";++ /** A {@code static final String NAME = "some/path.ext";} declaration holding a resource path. */+ private static final Pattern RESOURCE_CONSTANT = Pattern.compile(+ "static\\s+final\\s+String\\s+(\\w+)\\s*=\\s*\"(/?[\\w.\\-]+(?:/[\\w.\\-]+)+\\.[A-Za-z0-9]+)\"");++ /**+ * Read while the image is built, not at runtime. {@code AwsManagedPolicies} parses the catalog+ * during the build and keeps the parsed form in the image heap, so including the 3.5 MB of+ * source files would only be a second copy. {@code application.yml} says the same.+ */+ private static final Set<String> BUILD_TIME_ONLY = Set.of(+ "iam/managed-policies.yaml",+ "iam/managed-policy-documents.json",+ "iam/managed-policy-versions.json");++ @Test+ void everyResourceReadThroughAConstantIsIncludedInTheNativeImage() throws IOException {+ List<PathMatcher> includes = parseIncludes();+ Map<String, String> offenders = new LinkedHashMap<>();++ try (Stream<Path> sources = Files.walk(MAIN_SOURCE_ROOT)) {+ sources.filter(p -> p.toString().endsWith(JAVA_SUFFIX)).forEach(source -> {+ for (String resource : resourcesReadThroughAConstant(source)) {+ if (BUILD_TIME_ONLY.contains(resource)) {+ continue;+ }+ // Only a file this repository actually ships can be registered. A constant+ // naming something from a dependency is that dependency's problem.+ if (!Files.exists(MAIN_RESOURCE_ROOT.resolve(resource))) {+ continue;+ }+ if (includes.stream().noneMatch(m -> m.matches(Path.of(resource)))) {+ offenders.put(resource, source.toString());+ }+ }+ });+ }++ assertTrue(offenders.isEmpty(),+ "These resources are read at runtime through a constant, which Quarkus's static analysis "+ + "cannot follow, but no glob in " + APPLICATION_YML + " under "+ + "quarkus.native.resources.includes matches them. The native image will not contain "+ + "them and the first read will fail at runtime. Add an include for each:\n "+ + offenders.entrySet().stream()+ .map(e -> e.getKey() + " (read by " + e.getValue() + ")")+ .reduce((a, b) -> a + "\n " + b).orElse(""));+ }++ /** Resource paths declared as a constant in {@code javaFile} and passed to getResourceAsStream there. */+ private List<String> resourcesReadThroughAConstant(Path javaFile) {+ String source;+ try {+ source = Files.readString(javaFile);+ } catch (IOException e) {+ throw new UncheckedIOException("Unable to read source file " + javaFile, e);+ }+ List<String> found = new ArrayList<>();+ if (!source.contains("getResourceAsStream")) {+ return found;+ }+ Matcher constants = RESOURCE_CONSTANT.matcher(source);+ while (constants.find()) {+ String name = constants.group(1);+ String value = constants.group(2);+ if (source.contains("getResourceAsStream(" + name + ")")+ || source.contains("getResourceAsStream(\n")+ && source.contains(name + ")")) {+ found.add(value.startsWith("/") ? value.substring(1) : value);+ }+ }+ return found;+ }++ /** The {@code quarkus.native.resources.includes} globs, as matchers over classpath-relative paths. */+ private List<PathMatcher> parseIncludes() throws IOException {+ List<String> lines = Files.readAllLines(APPLICATION_YML);+ List<PathMatcher> matchers = new ArrayList<>();+ boolean inIncludes = false;+ for (String raw : lines) {+ String line = raw.strip();+ if (line.startsWith("includes:")) {+ inIncludes = true;+ continue;+ }+ if (inIncludes) {+ if (!line.startsWith("- ")) {+ if (!line.isEmpty() && !line.startsWith("#")) {+ break;+ }+ continue;+ }+ matchers.add(FileSystems.getDefault().getPathMatcher("glob:" + line.substring(2).strip()));+ }+ }+ assertTrue(!matchers.isEmpty(), "Found no quarkus.native.resources.includes entries in " + APPLICATION_YML);+ return matchers;+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuDockerIntegrationTest.javanew file mode 100644index 0000000000..95ebfbedff--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuDockerIntegrationTest.java@@ -0,0 +1,181 @@+package io.github.hectorvent.floci.services.sagemaker;++import com.github.dockerjava.api.DockerClient;+import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;+import io.quarkus.test.junit.QuarkusTest;+import io.quarkus.test.junit.QuarkusTestProfile;+import io.quarkus.test.junit.TestProfile;+import io.restassured.response.Response;+import jakarta.inject.Inject;+import org.jboss.logging.Logger;+import org.junit.jupiter.api.Assumptions;+import org.junit.jupiter.api.BeforeAll;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.time.Duration;+import java.util.HashMap;+import java.util.Map;++import static io.restassured.RestAssured.given;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertTrue;++/**+ * Proves an {@code ml.*} GPU instance type reaches a real accelerator, end to end through+ * {@code CreateTrainingJob} rather than through the container layer directly.+ *+ * <p>Opt-in for the same reason as {@link+ * io.github.hectorvent.floci.core.common.docker.ContainerGpuDockerIntegrationTest}: it needs+ * hardware no CI runner has. Run it with {@code -Dgpu.integration.enabled=true} on a host with a+ * GPU, naming the device the same way the feature's own configuration does:+ *+ * <pre>+ * ./mvnw test -Dtest=SageMakerGpuDockerIntegrationTest \+ * -Dgpu.integration.enabled=true \+ * -Dgpu.integration.cdi-device=nvidia.com/gpu=GPU-<uuid>+ * </pre>+ *+ * <p>Without the flag it skips, leaving {@link SageMakerGpuResolverTest} and {@link+ * SageMakerGpuIntegrationTest} as the coverage that runs by default. Those pin every path that+ * refuses a request; this pins the one that honours it.+ *+ * <p>The assertion is the container's exit code, not a log line. The image runs {@code nvidia-smi},+ * which the runtime injects alongside the device, so it is absent when no device was attached: a+ * job that quietly fell back to CPU exits nonzero and reports Failed. Naming a device also makes+ * the check exact, since the daemon must hand over that card rather than any card.+ *+ * <p>With no device named the test asks for every GPU using the count form, which is the shape to+ * run against Docker. Podman accepts that form and attaches nothing+ * (containers/podman#22645), so that combination fails here by design, which is the same reason+ * {@code mode} defaults to {@code cdi}.+ */+@QuarkusTest+@TestProfile(SageMakerGpuDockerIntegrationTest.GpuHardwareProfile.class)+class SageMakerGpuDockerIntegrationTest {++ private static final Logger LOG = Logger.getLogger(SageMakerGpuDockerIntegrationTest.class);++ private static final String ENABLED_PROPERTY = "gpu.integration.enabled";+ private static final String CDI_DEVICE_PROPERTY = "gpu.integration.cdi-device";+ private static final String IMAGE_PROPERTY = "gpu.integration.image";+ private static final String DEFAULT_IMAGE = "docker.io/nvidia/cuda:12.8.1-base-ubuntu24.04";++ /**+ * Mirrors the deployment-side configuration the feature documents. A named device selects+ * {@code cdi} mode and the allowlist; with none, {@code count} lets the daemon choose.+ */+ public static class GpuHardwareProfile implements QuarkusTestProfile {+ @Override+ public Map<String, String> getConfigOverrides() {+ Map<String, String> overrides = new HashMap<>();+ overrides.put("floci.services.sagemaker.gpu.enabled", "true");+ String device = System.getProperty(CDI_DEVICE_PROPERTY);+ if (device != null && !device.isBlank()) {+ overrides.put("floci.services.sagemaker.gpu.mode", "cdi");+ overrides.put("floci.services.sagemaker.gpu.devices", device);+ } else {+ overrides.put("floci.services.sagemaker.gpu.mode", "count");+ }+ return overrides;+ }+ }++ private static final String AUTH =+ "AWS4-HMAC-SHA256 Credential=test/20260205/us-east-1/sagemaker/aws4_request";++ @Inject+ DockerClient dockerClient;++ @Inject+ S3Service s3Service;++ @BeforeAll+ static void configureRestAssured() {+ RestAssuredJsonUtils.configureAwsContentTypes();+ }++ @BeforeEach+ void requireGpuEnabledDaemon() {+ Assumptions.assumeTrue(Boolean.getBoolean(ENABLED_PROPERTY),+ "GPU hardware tests are opt-in: run with -D" + ENABLED_PROPERTY + "=true on a host with a GPU");+ Assumptions.assumeTrue(isDockerAvailable(),+ "A container daemon must be available for GPU integration tests");+ }++ @Test+ void gpuInstanceTypeTrainsOnTheRequestedDevice() throws Exception {+ String suffix = Long.toString(System.nanoTime(), 36);+ String bucket = "sm-gpu-" + suffix;+ s3Service.createBucket(bucket, "us-east-1");+ String job = "sm-gpu-" + suffix;+ String image = System.getProperty(IMAGE_PROPERTY, DEFAULT_IMAGE);++ // nvidia-smi is injected with the device, so its absence is the clearest signal that+ // nothing was attached. Writing the UUID into /opt/ml/model makes Floci upload the+ // evidence rather than leaving it in a log.+ //+ // Redirect rather than pipe: a pipeline exits with the status of its last command, so+ // `nvidia-smi -L | tee` reports success even when nvidia-smi could not reach the driver.+ // That would be this test passing on a CPU container, which is the exact failure it exists+ // to catch.+ post("SageMaker.CreateTrainingJob", """+ {+ "TrainingJobName":"%s",+ "AlgorithmSpecification":{+ "TrainingImage":"%s",+ "ContainerEntrypoint":["/bin/sh","-c"],+ "ContainerArguments":["mkdir -p /opt/ml/model && nvidia-smi -L > /opt/ml/model/devices.txt && cat /opt/ml/model/devices.txt"]+ },+ "OutputDataConfig":{"S3OutputPath":"s3://%s/output"},+ "ResourceConfig":{"InstanceType":"ml.g5.xlarge","InstanceCount":1,"VolumeSizeInGB":1},+ "StoppingCondition":{"MaxRuntimeInSeconds":300}+ }+ """.formatted(job, image, bucket)).then().statusCode(200);++ Response describe = awaitTerminal(job);+ String status = describe.path("TrainingJobStatus");+ String reason = describe.path("FailureReason");+ assertEquals("Completed", status,+ "ml.g5.xlarge did not reach a GPU: " + (reason == null ? "no FailureReason" : reason));++ String artifact = describe.path("ModelArtifacts.S3ModelArtifacts");+ assertTrue(artifact != null && artifact.endsWith("model.tar.gz"),+ "no model artifact was uploaded: " + artifact);+ LOG.infov("GPU training job completed, artifact at {0}", artifact);+ }++ private Response awaitTerminal(String job) throws Exception {+ long deadline = System.currentTimeMillis() + Duration.ofSeconds(300).toMillis();+ while (System.currentTimeMillis() < deadline) {+ Response describe = post("SageMaker.DescribeTrainingJob",+ "{\"TrainingJobName\":\"%s\"}".formatted(job));+ describe.then().statusCode(200);+ String status = describe.path("TrainingJobStatus");+ if ("Failed".equals(status) || "Completed".equals(status) || "Stopped".equals(status)) {+ return describe;+ }+ Thread.sleep(1000);+ }+ throw new AssertionError("Timed out waiting for training job " + job);+ }++ private Response post(String target, String body) {+ return given().header("Authorization", AUTH)+ .header("X-Amz-Target", target)+ .contentType("application/x-amz-json-1.1")+ .body(body)+ .when().post("/");+ }++ private boolean isDockerAvailable() {+ try {+ dockerClient.pingCmd().exec();+ return true;+ } catch (Exception e) {+ LOG.warn("No container daemon available for the SageMaker GPU integration test", e);+ return false;+ }+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuIntegrationTest.javanew file mode 100644index 0000000000..4ac0719ba7--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuIntegrationTest.java@@ -0,0 +1,140 @@+package io.github.hectorvent.floci.services.sagemaker;++import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;+import io.quarkus.test.junit.QuarkusTest;+import io.quarkus.test.junit.TestProfile;+import io.quarkus.test.junit.QuarkusTestProfile;+import io.restassured.response.Response;+import org.junit.jupiter.api.BeforeAll;+import org.junit.jupiter.api.Test;++import java.time.Duration;+import java.util.Map;++import static io.restassured.RestAssured.given;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;++/**+ * Covers the GPU refusal paths, which need no GPU and no container daemon.+ *+ * <p>{@link SageMakerGpuResolver} raises before the runner asks the daemon for anything, and+ * {@code removeIfExists} on the line above it swallows a missing daemon, so every case here+ * reaches its verdict on a plain CI runner. The hardware path is the opposite and lives in+ * {@link SageMakerGpuDockerIntegrationTest}, which skips without a GPU.+ *+ * <p>What is being pinned is the contract that a request Floci cannot honour fails with a+ * reason naming the fix, rather than falling back to CPU. A job that reports {@code Completed}+ * after training on the wrong hardware leaves an artifact that looks legitimate.+ *+ * <p>The profile turns the feature on because {@code application.yml} ships it off. Devices are+ * deliberately left unset, which is what makes the third case reachable.+ */+@QuarkusTest+@TestProfile(SageMakerGpuIntegrationTest.GpuEnabledProfile.class)+class SageMakerGpuIntegrationTest {++ public static class GpuEnabledProfile implements QuarkusTestProfile {+ @Override+ public Map<String, String> getConfigOverrides() {+ return Map.of("floci.services.sagemaker.gpu.enabled", "true");+ }+ }++ private static final String AUTH =+ "AWS4-HMAC-SHA256 Credential=test/20260205/us-east-1/sagemaker/aws4_request";++ @BeforeAll+ static void configureRestAssured() {+ RestAssuredJsonUtils.configureAwsContentTypes();+ }++ @Test+ void gpuFamilyMissingFromTheCatalogFailsNamingTheCatalog() throws Exception {+ String job = submit("ml.p5.48xlarge", 1);+ String reason = awaitFailureReason(job);++ assertTrue(reason.contains("ml.p5.48xlarge"), reason);+ assertTrue(reason.contains("catalog"), reason);+ }++ @Test+ void instanceCountAboveOneFailsRatherThanTrainingOnOneContainer() throws Exception {+ String job = submit("ml.g5.xlarge", 2);+ String reason = awaitFailureReason(job);++ assertTrue(reason.contains("InstanceCount 2"), reason);+ assertTrue(reason.contains("ml.g5.xlarge"), reason);+ }++ @Test+ void cdiModeWithNoConfiguredDevicesFailsRatherThanTakingEveryGpu() throws Exception {+ String job = submit("ml.g5.xlarge", 1);+ String reason = awaitFailureReason(job);++ assertTrue(reason.contains("floci.services.sagemaker.gpu.devices"), reason);+ }++ /**+ * Enabling the feature must not change what a CPU instance type does. The job's outcome+ * depends on whether a daemon is present, so only the reason is asserted: whatever happens,+ * it must not be the resolver refusing the request.+ */+ @Test+ void cpuInstanceTypeIsNotRefusedWhenGpuSupportIsEnabled() throws Exception {+ String job = submit("ml.m5.large", 1);+ Response describe = awaitTerminal(job);+ String reason = describe.path("FailureReason") == null ? "" : describe.path("FailureReason");++ assertFalse(reason.contains("floci.services.sagemaker.gpu"), reason);+ assertFalse(reason.contains("catalog"), reason);+ }++ private String submit(String instanceType, int instanceCount) {+ String job = "sm-gpu-" + Long.toString(System.nanoTime(), 36);+ post("SageMaker.CreateTrainingJob", """+ {+ "TrainingJobName":"%s",+ "AlgorithmSpecification":{"TrainingImage":"busybox:stable","TrainingInputMode":"File"},+ "OutputDataConfig":{"S3OutputPath":"s3://sm-gpu-out/output"},+ "ResourceConfig":{"InstanceType":"%s","InstanceCount":%d,"VolumeSizeInGB":1},+ "StoppingCondition":{"MaxRuntimeInSeconds":60}+ }+ """.formatted(job, instanceType, instanceCount))+ .then().statusCode(200);+ return job;+ }++ private String awaitFailureReason(String job) throws Exception {+ Response describe = awaitTerminal(job);+ assertEquals("Failed", describe.path("TrainingJobStatus"),+ "expected the request to be refused, not honoured");+ String reason = describe.path("FailureReason");+ assertTrue(reason != null && !reason.isBlank(), "Failed with no FailureReason");+ return reason;+ }++ private Response awaitTerminal(String job) throws Exception {+ long deadline = System.currentTimeMillis() + Duration.ofSeconds(60).toMillis();+ while (System.currentTimeMillis() < deadline) {+ Response describe = post("SageMaker.DescribeTrainingJob",+ "{\"TrainingJobName\":\"%s\"}".formatted(job));+ describe.then().statusCode(200);+ String status = describe.path("TrainingJobStatus");+ if ("Failed".equals(status) || "Completed".equals(status) || "Stopped".equals(status)) {+ return describe;+ }+ Thread.sleep(200);+ }+ throw new AssertionError("Timed out waiting for training job " + job + " to reach a terminal state");+ }++ private Response post(String target, String body) {+ return given().header("Authorization", AUTH)+ .header("X-Amz-Target", target)+ .contentType("application/x-amz-json-1.1")+ .body(body)+ .when().post("/");+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuResolverTest.java b/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuResolverTest.javanew file mode 100644index 0000000000..1016944521--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerGpuResolverTest.java@@ -0,0 +1,187 @@+package io.github.hectorvent.floci.services.sagemaker;++import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.config.EmulatorConfig.SageMakerServiceConfig.GpuRequestMode;+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import io.github.hectorvent.floci.core.common.dns.EmbeddedDnsServer;+import io.github.hectorvent.floci.core.common.docker.ContainerSpec;+import io.github.hectorvent.floci.core.common.docker.CurrentContainerNetworkResolver;+import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;+import com.github.dockerjava.api.model.DeviceRequest;+import org.junit.jupiter.api.Test;++import java.util.List;+import java.util.Optional;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++class SageMakerGpuResolverTest {++ private static final String UUID_A = "nvidia.com/gpu=GPU-aaaaaaaa";+ private static final String UUID_B = "nvidia.com/gpu=GPU-bbbbbbbb";+ private static final String UUID_C = "nvidia.com/gpu=GPU-cccccccc";+ private static final String UUID_D = "nvidia.com/gpu=GPU-dddddddd";++ @Test+ void disabledByDefaultLeavesEvenGpuInstanceTypesAlone() {+ Fixture fixture = new Fixture(false, GpuRequestMode.CDI, List.of(UUID_A));++ ContainerSpec spec = fixture.resolve("ml.g5.xlarge", 1);++ // The guard on every existing deployment: opting out means nothing changes,+ // including for instance types that would otherwise ask for hardware.+ assertEquals(List.of(), spec.deviceRequests());+ }++ @Test+ void cpuInstanceTypeGetsNoDevice() {+ Fixture fixture = new Fixture(true, GpuRequestMode.CDI, List.of(UUID_A));++ assertEquals(List.of(), fixture.resolve("ml.m5.large", 1).deviceRequests());+ }++ @Test+ void unknownNonAcceleratorTypeIsTreatedAsCpu() {+ Fixture fixture = new Fixture(true, GpuRequestMode.CDI, List.of(UUID_A));++ // Not in the catalog and not a GPU family: a CPU job Floci simply does not+ // have metadata for, which is no reason to fail it.+ assertEquals(List.of(), fixture.resolve("ml.c7i.12xlarge", 1).deviceRequests());+ }++ @Test+ void gpuInstanceTypeRequestsItsCataloguedDeviceCount() {+ Fixture fixture = new Fixture(true, GpuRequestMode.CDI, List.of(UUID_A, UUID_B));++ ContainerSpec spec = fixture.resolve("ml.g5.xlarge", 1);++ DeviceRequest request = spec.deviceRequests().get(0);+ assertEquals("cdi", request.getDriver());+ assertEquals(List.of(UUID_A), request.getDeviceIds());+ }++ @Test+ void multiGpuTypeTakesAsManyDevicesAsTheCatalogSays() {+ Fixture fixture = new Fixture(true, GpuRequestMode.CDI,+ List.of(UUID_A, UUID_B, UUID_C, UUID_D));++ // ml.g5.12xlarge is four devices on AWS, so a host offering four gets all four+ // in one container. Counts come from the catalog rather than from matching+ // ml.g*, because they vary within a family.+ ContainerSpec spec = fixture.resolve("ml.g5.12xlarge", 1);++ assertEquals(List.of(UUID_A, UUID_B, UUID_C, UUID_D),+ spec.deviceRequests().get(0).getDeviceIds());+ }++ @Test+ void devicesAreTakenInOrderSoASmallerTypeDoesNotClaimEverything() {+ Fixture fixture = new Fixture(true, GpuRequestMode.CDI,+ List.of(UUID_A, UUID_B, UUID_C, UUID_D));++ // A one-GPU type on a four-GPU host takes one device, not the whole allowlist.+ assertEquals(List.of(UUID_A), fixture.resolve("ml.g5.xlarge", 1)+ .deviceRequests().get(0).getDeviceIds());+ }++ @Test+ void multiGpuTypeIsRefusedWhenTheHostAllowsFewerDevices() {+ Fixture fixture = new Fixture(true, GpuRequestMode.CDI, List.of(UUID_A, UUID_B));++ SageMakerGpuResolver.UnsupportedResourceException thrown = assertThrows(+ SageMakerGpuResolver.UnsupportedResourceException.class,+ () -> fixture.resolve("ml.g5.12xlarge", 1));+ assertTrue(thrown.getMessage().contains("needs 4 GPUs"), thrown.getMessage());+ }++ @Test+ void deviceIdsModeUsesTheDaemonIdForm() {+ Fixture fixture = new Fixture(true, GpuRequestMode.DEVICE_IDS, List.of("0", "1"));++ DeviceRequest request = fixture.resolve("ml.g5.xlarge", 1).deviceRequests().get(0);++ assertEquals(List.of("0"), request.getDeviceIds());+ assertEquals(List.of(List.of("gpu")), request.getCapabilities());+ }++ @Test+ void countModeLetsTheDaemonChooseAndIgnoresTheAllowlist() {+ Fixture fixture = new Fixture(true, GpuRequestMode.COUNT, List.of());++ DeviceRequest request = fixture.resolve("ml.g5.xlarge", 1).deviceRequests().get(0);++ assertEquals(1, request.getCount());+ }++ @Test+ void acceleratorTypeMissingFromTheCatalogIsRefused() {+ Fixture fixture = new Fixture(true, GpuRequestMode.CDI, List.of(UUID_A));++ // The case this whole class exists to prevent: the caller asked for a GPU, so+ // running on CPU and reporting Completed would hand back an artifact that looks+ // legitimate.+ SageMakerGpuResolver.UnsupportedResourceException thrown = assertThrows(+ SageMakerGpuResolver.UnsupportedResourceException.class,+ () -> fixture.resolve("ml.p5.48xlarge", 1));+ assertTrue(thrown.getMessage().contains("not in Floci's"), thrown.getMessage());+ }++ @Test+ void multipleInstancesAreRefusedRatherThanPretended() {+ Fixture fixture = new Fixture(true, GpuRequestMode.CDI, List.of(UUID_A, UUID_B));++ SageMakerGpuResolver.UnsupportedResourceException thrown = assertThrows(+ SageMakerGpuResolver.UnsupportedResourceException.class,+ () -> fixture.resolve("ml.g5.xlarge", 2));+ assertTrue(thrown.getMessage().contains("InstanceCount 2"), thrown.getMessage());+ }++ @Test+ void explicitModeWithNoDevicesConfiguredIsRefused() {+ Fixture fixture = new Fixture(true, GpuRequestMode.CDI, List.of());++ SageMakerGpuResolver.UnsupportedResourceException thrown = assertThrows(+ SageMakerGpuResolver.UnsupportedResourceException.class,+ () -> fixture.resolve("ml.g5.xlarge", 1));+ assertTrue(thrown.getMessage().contains("devices is empty"), thrown.getMessage());+ }++ /** Wires a real ContainerBuilder so the assertions see the request actually built. */+ private static final class Fixture {+ private final SageMakerGpuResolver resolver;+ private final ContainerBuilder containerBuilder;++ private Fixture(boolean enabled, GpuRequestMode mode, List<String> devices) {+ EmulatorConfig config = mock(EmulatorConfig.class);+ EmulatorConfig.ServicesConfig services = mock(EmulatorConfig.ServicesConfig.class);+ EmulatorConfig.SageMakerServiceConfig sagemaker =+ mock(EmulatorConfig.SageMakerServiceConfig.class);+ EmulatorConfig.SageMakerServiceConfig.GpuConfig gpu =+ mock(EmulatorConfig.SageMakerServiceConfig.GpuConfig.class);+ EmulatorConfig.DockerConfig docker = mock(EmulatorConfig.DockerConfig.class);++ when(config.services()).thenReturn(services);+ when(services.sagemaker()).thenReturn(sagemaker);+ when(sagemaker.gpu()).thenReturn(gpu);+ when(gpu.enabled()).thenReturn(enabled);+ when(gpu.mode()).thenReturn(mode);+ when(gpu.devices()).thenReturn(Optional.of(devices));+ when(config.docker()).thenReturn(docker);+ when(docker.imageRegistryBase()).thenReturn(Optional.empty());++ this.containerBuilder = new ContainerBuilder(config, mock(DockerHostResolver.class),+ mock(EmbeddedDnsServer.class), mock(CurrentContainerNetworkResolver.class));+ this.resolver = new SageMakerGpuResolver(config, new SageMakerInstanceTypeCatalog());+ }++ private ContainerSpec resolve(String instanceType, int instanceCount) {+ ContainerBuilder.Builder builder = containerBuilder.newContainer("alpine");+ resolver.applyTo(builder, instanceType, instanceCount);+ return builder.build();+ }+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerTrainingRunnerTest.java b/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerTrainingRunnerTest.javaindex 16da79984d..70136b4777 100644--- a/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerTrainingRunnerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/sagemaker/SageMakerTrainingRunnerTest.java@@ -74,6 +74,7 @@ class SageMakerTrainingRunnerTest {private static SageMakerTrainingRunner runner() {return new SageMakerTrainingRunner(mock(ContainerBuilder.class), mock(ContainerLifecycleManager.class),mock(ContainerLogStreamer.class), mock(EmulatorConfig.class, RETURNS_DEEP_STUBS),- mock(ContainerDetector.class), mock(S3Service.class), new ObjectMapper());+ mock(ContainerDetector.class), mock(S3Service.class), new ObjectMapper(),+ mock(SageMakerGpuResolver.class));}}diff --git a/src/test/resources/application.yml b/src/test/resources/application.ymlindex df1e01a654..53bbd06470 100644--- a/src/test/resources/application.yml+++ b/src/test/resources/application.yml@@ -332,6 +332,18 @@ floci:sagemaker:enabled: true# docker-network: floci-network+ gpu:+ # Off by default: an existing deployment keeps launching CPU-only+ # containers. How many GPUs an ml.* type has is shipped in+ # sagemaker/instance-type-catalog.yaml; this is only about which of+ # this machine's devices Floci may hand out.+ enabled: false+ # cdi | device-ids | count. Podman resolves only cdi, and accepts+ # count while attaching no device, so cdi fails loudly instead.+ mode: cdi+ # Required for cdi and device-ids modes; unset allows no device.+ # devices:+ # - nvidia.com/gpu=GPU-<uuid>batch:enabled: truerunner-mode: immediate