Light, fluffy, and always free - The AWS Local Emulator alternative
feat(eks): bound cluster containers and derive kubelet reservations (#4590)
Simone Primarosa047d0d848cdf
diff --git a/docs/services/eks.md b/docs/services/eks.mdindex 2392dc9820..e1a7332748 100644--- a/docs/services/eks.md+++ b/docs/services/eks.md@@ -369,6 +369,57 @@ A restored cluster reports `CREATING` until its API server answers again, then rback (for example Docker is unavailable), the cluster is marked `FAILED` instead of appearing`ACTIVE` while unreachable.+#### Cluster node capacity++Floci runs one k3s container per cluster. Before a node group exists, its node uses the+`m5.large` entry in the EC2 instance type catalog. Creating the first node group selects the+group's first `instanceTypes` entry. If that changes the type, Floci recreates the container,+keeps its named k3s data volume, and reuses its published API port so existing kubeconfigs stay+valid. Later node groups share this node and cannot change its type. An unknown type falls back to+`m5.large` with a warning. The synthesized EC2 instance reports the selected type.+First-group selection is serialized per cluster. If the replacement fails, Floci recreates the+previous node, marks that node group `CREATE_FAILED`, and keeps its data volume. If recovery also+fails, the cluster becomes `FAILED` instead of waiting indefinitely for readiness.+Groups remain `CREATING` while launch-template user data runs. Later groups wait for the first+group's result before accepting its node type as the shared capacity.++The container receives a hard CPU quota equal to the selected type's vCPUs and a memory limit+equal to its catalog memory plus 10% headroom (at least 128 MiB). Both are capped by Docker's host+capacity. Memory is capped at 80% of the Docker host's memory so other processes can continue to+run. `floci.services.eks.max-memory-mib` and `floci.services.eks.max-vcpus` provide optional lower+ceilings; zero means no extra ceiling. Their environment variables are `FLOCI_SERVICES_EKS_MAX_MEMORY_MIB`+and `FLOCI_SERVICES_EKS_MAX_VCPUS`.++The same calculation sets kubelet's `kube-reserved`, `system-reserved`, and `eviction-hard` flags.+The EKS AMI reserves `255 + 11 * maxPods` MiB and a tiered CPU fraction for kubelet and the+container runtime. Floci derives `maxPods` from the catalog's primary network card and IPv4+addresses per interface. Its `system-reserved` memory holds the container headroom, and its CPU+reservation excludes the Docker host CPUs outside the container quota. Its absolute+`memory.available` threshold includes the difference between Docker host memory and the container+limit, plus a 100 MiB eviction buffer. This is necessary because kubelet in the nested container+can report host memory while the kernel enforces the smaller cgroup limit.+The EKS node filesystem eviction defaults remain at 10% available space and 5% free inodes.+The pod-density calculation uses EKS managed node group caps of 110 pods up to 30 vCPUs and+250 pods above 30 vCPUs; it does not change k3s `maxPods`.++If Docker cannot report host capacity, or the host is too small to leave 256 MiB for pods after+reservations, Floci warns and launches the cluster without the derived limits or kubelet arguments.+This preserves startup on small hosts. An explicit memory or CPU ceiling remains a hard bound even+when Docker host information is unavailable; in that case Floci cannot calculate kubelet arguments.+When an explicit memory ceiling is smaller than the default reservation budget, Floci reduces the+memory reservations to leave some room for pods and warns that they no longer match the EKS AMI+amount.++The reservation formulas follow the [EKS AMI nodeadm source](https://github.com/awslabs/amazon-eks-ami/blob/main/nodeadm/internal/kubelet/config.go) and [AWS's node memory calculation](https://docs.aws.amazon.com/batch/latest/userguide/memory-cpu-batch-eks.html).+On restore, a surviving container whose capacity label differs from the current calculation is+recreated against the retained data volume, reusing its published API port. Floci keeps the old+container until the replacement starts and restarts it if replacement fails. This also upgrades+containers started before resource limits were available. If Docker cannot remove the stopped+backup after a successful replacement, the new node keeps running and cluster deletion retries+the backup cleanup after stopping the live node. If cleanup still fails, deletion reports the+error and can be retried before the data volume is removed. Backup container names end in+`.capacity-backup`, which cannot match an EKS cluster name.+#### Cluster node provider ID and topology labelsIn real mode, cluster nodes carry a Kubernetes `spec.providerID` matching the AWS format:@@ -394,6 +445,8 @@ The derived availability zone and instance ID match the synthetic EC2 node insta|---|---|---|| `FLOCI_SERVICES_EKS_ENABLED` | `true` | Enable the EKS service || `FLOCI_SERVICES_EKS_MOCK` | `false` | Metadata-only mode (no Docker) |+| `FLOCI_SERVICES_EKS_MAX_MEMORY_MIB` | `0` | Optional k3s container memory ceiling in MiB |+| `FLOCI_SERVICES_EKS_MAX_VCPUS` | `0` | Optional k3s container vCPU ceiling || `FLOCI_SERVICES_EKS_DEFAULT_IMAGE` | `rancher/k3s:latest` | k3s Docker image fallback || `FLOCI_SERVICES_EKS_IMAGE_TEMPLATE` | *(unset)* | Format string for custom k3s images (e.g. `myregistry.io/k3s:v%s`), taking cluster version || `FLOCI_SERVICES_EKS_API_SERVER_BASE_PORT` | `6500` | First port in the k3s API server range |@@ -521,7 +574,7 @@ services:```## Instance Metadata Service (IMDS)-The cluster container is registered as a synthesized EC2 instance node (type `m5.large`, image `ami-eks-k3s`) regardless of whether IMDS is enabled, making it visible to the EC2 service (`DescribeInstances`, `DescribeInstanceStatus`, volume attachments, tagging).+The cluster container is registered as a synthesized EC2 instance node (image `ami-eks-k3s`) regardless of whether IMDS is enabled, making it visible to the EC2 service (`DescribeInstances`, `DescribeInstanceStatus`, volume attachments, tagging). Its instance type is the same type used for the container limits.When IMDS is enabled (`FLOCI_SERVICES_EKS_IMDS=true`), each k3s cluster container exposes the AWS Instance Metadata Service on the link-local address `169.254.169.254:80`. Inside the container, Floci adds `169.254.169.254/32` to the loopback interface (`lo`) and runs a lightweight `socat` TCP relay forwarding metadata requests to Floci's IMDS server.diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex 2a33312296..c08024429f 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -3086,6 +3086,14 @@ public interface EmulatorConfig {@WithDefault("false")boolean mock();+ /** Optional ceiling for the single k3s node container, in MiB. Zero uses its instance type. */+ @WithDefault("0")+ int maxMemoryMib();++ /** Optional vCPU ceiling for the single k3s node container. Zero uses its instance type. */+ @WithDefault("0")+ int maxVcpus();+@WithDefault("k3s")String provider();diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.javaindex fb2378b21e..a77043e60f 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java@@ -101,6 +101,7 @@ public final class ContainerExec {dockerClient.execStartCmd(execId).exec(callback);try {+ // docker-java closes the attach stream on completion, timeout, or interruption.if (!callback.awaitCompletion(timeoutSeconds, TimeUnit.SECONDS)) {return new Result(-1, "", "Timed out after " + timeoutSeconds + "s", true);}diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.javaindex 6b01dd352c..7fe1d86473 100644--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java@@ -4,6 +4,7 @@ import com.github.dockerjava.api.DockerClient;import com.github.dockerjava.api.command.ExecCreateCmdResponse;import com.github.dockerjava.api.command.InspectContainerResponse;import com.github.dockerjava.api.model.ContainerNetwork;+import com.github.dockerjava.api.model.Info;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.config.FlociCertificateAuthority;import io.github.hectorvent.floci.core.common.AwsRegions;@@ -22,6 +23,8 @@ import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;import io.github.hectorvent.floci.core.common.docker.PortAllocator;import io.github.hectorvent.floci.core.common.docker.UserDataPipeline;import io.github.hectorvent.floci.services.ec2.ClusterNodeInstanceProvider;+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog;+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogInstanceType;import io.github.hectorvent.floci.services.ec2.Ec2MetadataProxy;import io.github.hectorvent.floci.services.ec2.Ec2MetadataServer;import io.github.hectorvent.floci.services.ec2.Ec2Service;@@ -64,6 +67,7 @@ import java.time.Duration;import java.time.Instant;import java.util.ArrayList;import java.util.Base64;+import java.util.LinkedHashMap;import java.util.LinkedHashSet;import java.util.List;import java.util.Map;@@ -85,6 +89,8 @@ public class EksClusterManagerprivate static final Logger LOG = Logger.getLogger(EksClusterManager.class);private static final int K3S_API_SERVER_PORT = 6443;+ static final String DEFAULT_NODE_INSTANCE_TYPE = "m5.large";+ private static final String NODE_CAPACITY_LABEL = "io.floci.eks.node-capacity";private static final String WEBHOOK_CONFIG_DIR = "/etc";private static final String WEBHOOK_CONFIG_FILE = "token-webhook.yaml";@@ -140,6 +146,7 @@ public class EksClusterManagerprivate final EksOidcService oidcService;private final FlociCertificateAuthority certificateAuthority;private final ContainerLogStreamer logStreamer;+ private final Ec2InstanceTypeCatalog instanceTypeCatalog = new Ec2InstanceTypeCatalog();private final Map<String, ClusterNodeRecord> clusterNodeInstances = new ConcurrentHashMap<>();private final Map<String, Closeable> clusterLogHandles = new ConcurrentHashMap<>();private final List<Consumer<Instance>> nodeRegistrationListeners = new CopyOnWriteArrayList<>();@@ -373,6 +380,10 @@ public class EksClusterManager* {@link #isReady(Cluster)} returns true and {@link #finalizeCluster(Cluster)} is called.*/public void startCluster(Cluster cluster) {+ startCluster(cluster, null);+ }++ private void startCluster(Cluster cluster, Integer retainedPort) {String image = resolveClusterImage(cluster);if (cluster.getDockerName() == null) {cluster.setDockerName(accountQualifiedName(cluster));@@ -383,7 +394,7 @@ public class EksClusterManagercluster.getName(), image);// Allocate host port for the k3s API server- int hostPort = portAllocator.allocate(+ int hostPort = retainedPort != null ? retainedPort : portAllocator.allocate(config.services().eks().apiServerBasePort(),config.services().eks().apiServerMaxPort());@@ -413,6 +424,11 @@ public class EksClusterManagerList<String> serverArgs = buildServerArgs(config.services().eks().disableCni(), serviceCidr, clusterCidr);+ EksNodeCapacity.Limits nodeLimits = resolveNodeCapacity(cluster);+ if (nodeLimits != null) {+ nodeLimits.addKubeletArgs(serverArgs);+ }+try {String providerId = deriveClusterNodeProviderId(cluster);serverArgs.add("--kubelet-arg=provider-id=" + providerId);@@ -436,6 +452,9 @@ public class EksClusterManager// The account label comes from the cluster record when set (restore runs with no request// context); regionResolver is the fallback for the create path.String labelAccountId = resolveClusterAccountId(cluster);+ Map<String, String> labels = new LinkedHashMap<>(ContainerStorageHelper.resourceIdentityLabels(+ "eks", cluster.getName(), labelAccountId, clusterRegion(cluster)));+ labels.put(NODE_CAPACITY_LABEL, capacityLabel(cluster, nodeLimits));ContainerBuilder.Builder specBuilder = containerBuilder.newContainer(image).withName(containerName).withEnv("K3S_KUBECONFIG_MODE", "644")@@ -444,8 +463,16 @@ public class EksClusterManager.withDockerNetwork(config.services().eks().dockerNetwork()).withPrivileged(true).withLogRotation()- .withLabels(ContainerStorageHelper.resourceIdentityLabels(- "eks", cluster.getName(), labelAccountId, clusterRegion(cluster)));+ .withLabels(labels);++ if (nodeLimits != null) {+ if (nodeLimits.memoryBytes() > 0) {+ specBuilder.withMemoryBytes(nodeLimits.memoryBytes());+ }+ if (nodeLimits.vcpus() > 0) {+ specBuilder.withCpuUnits(nodeLimits.vcpus() * 1024);+ }+ }if (config.services().eks().embeddedDns()) {specBuilder.withEmbeddedDns();@@ -572,25 +599,38 @@ public class EksClusterManagerreturn;}+ String desiredCapacity = capacityLabel(cluster, resolveNodeCapacity(cluster));+ Map<String, String> existingLabels = existing.get().getLabels();+ boolean capacityChanged = existingLabels == null+ || !desiredCapacity.equals(existingLabels.get(NODE_CAPACITY_LABEL));+ if (!adoptSurvivingCluster(cluster, existing.get().getId())) {+ startCluster(cluster);+ return;+ }+ if (capacityChanged) {+ replaceClusterContainer(cluster, null);+ }+ }++ private boolean adoptSurvivingCluster(Cluster cluster, String containerId) {if (config.services().eks().irsaSigningKey() && oidcService != null) {- reinjectSigningKeys(existing.get().getId(), cluster);+ reinjectSigningKeys(containerId, cluster);}ContainerInfo info;try {- info = lifecycleManager.adopt(existing.get().getId(), List.of(K3S_API_SERVER_PORT));+ info = lifecycleManager.adopt(containerId, List.of(K3S_API_SERVER_PORT));} catch (Exception e) {LOG.warnv("Could not adopt surviving k3s container {0} for EKS cluster {1} ({2}); recreating it",- containerName, cluster.getName(), e.getMessage());- startCluster(cluster);- return;+ cluster.getDockerName(), cluster.getName(), e.getMessage());+ return false;}var publishedPort = info.publishedHostPort(K3S_API_SERVER_PORT);if (publishedPort.isEmpty()) {- LOG.warnv("Surviving k3s container {0} publishes no API server port; recreating it", containerName);- startCluster(cluster);- return;+ LOG.warnv("Surviving k3s container {0} publishes no API server port; recreating it",+ cluster.getDockerName());+ return false;}int hostPort = publishedPort.getAsInt();@@ -598,7 +638,7 @@ public class EksClusterManagerportAllocator.markReserved(hostPort);cluster.setContainerId(info.containerId());cluster.setHostPort(hostPort);- applyEndpoints(cluster, containerName, hostPort, info);+ applyEndpoints(cluster, cluster.getDockerName(), hostPort, info);registerClusterNodeInstance(cluster, info.containerId());configureLinkLocalMetadataEndpoint(cluster, info.containerId());configurePodIdentityRelay(cluster, info.containerId());@@ -608,6 +648,71 @@ public class EksClusterManagerLOG.infov("Adopted surviving k3s container {0} for EKS cluster {1} on port {2} (internal: {3})",info.containerId(), cluster.getName(), String.valueOf(hostPort), cluster.getInternalEndpoint());+ return true;+ }++ /** Keep the old container stopped but recoverable until its replacement has started. */+ private boolean replaceClusterContainer(Cluster cluster, String previousType) {+ String oldId = cluster.getContainerId();+ int oldPort = cluster.getHostPort();+ String containerName = cluster.getDockerName();+ String backupName = capacityBackupName(cluster);+ DockerClient docker = lifecycleManager.getDockerClient();+ try {+ lifecycleManager.removeIfExistsStrict(backupName);+ } catch (RuntimeException cleanup) {+ LOG.warnv("Could not clear prior EKS capacity backup for cluster {0}: {1}; keeping current node",+ cluster.getName(), cleanup.getMessage());+ restorePreviousNodeType(cluster, previousType);+ return false;+ }+ boolean renamed = false;+ try {+ docker.renameContainerCmd(oldId).withName(backupName).exec();+ renamed = true;+ docker.stopContainerCmd(oldId).exec();+ unregisterMetadataEndpoint(cluster);+ closeQuietly(clusterLogHandles.remove(clusterResourceName(cluster)));+ cluster.setContainerId(null);+ startCluster(cluster, oldPort);+ LOG.infov("Replaced EKS cluster {0} to apply current node capacity limits", cluster.getName());+ } catch (RuntimeException replacement) {+ LOG.warnv("Could not replace EKS cluster {0} for node capacity: {1}; restoring surviving node",+ cluster.getName(), replacement.getMessage());+ restorePreviousNodeType(cluster, previousType);+ if (!renamed) {+ return false;+ }+ try {+ lifecycleManager.removeIfExistsStrict(containerName);+ docker.renameContainerCmd(oldId).withName(containerName).exec();+ if (!adoptSurvivingCluster(cluster, oldId)) {+ throw new IllegalStateException("Could not adopt previous EKS container " + oldId);+ }+ } catch (RuntimeException rollback) {+ throw new IllegalStateException("Could not restore EKS cluster " + cluster.getName()+ + " after node capacity replacement failed", rollback);+ }+ return false;+ }+ try {+ lifecycleManager.removeIfExistsStrict(backupName);+ } catch (RuntimeException cleanup) {+ LOG.warnv("EKS cluster {0} replacement is running, but its stopped capacity backup"+ + " could not be removed: {1}; cluster deletion will retry",+ cluster.getName(), cleanup.getMessage());+ }+ return true;+ }++ private static void restorePreviousNodeType(Cluster cluster, String previousType) {+ if (previousType != null) {+ cluster.setNodeInstanceType(previousType);+ }+ }++ private static String capacityBackupName(Cluster cluster) {+ return cluster.getDockerName() + ".capacity-backup";}/**@@ -705,10 +810,88 @@ public class EksClusterManagerreturn;}lifecycleManager.stopAndRemove(cluster.getContainerId(), logStream);+ if (cluster.getDockerName() != null) {+ // A failed backup cleanup must not leave the live node running. Keep the cluster+ // record so explicit deletion can be retried once Docker accepts the removal.+ lifecycleManager.removeIfExistsStrict(capacityBackupName(cluster));+ }ContainerStorageHelper.removeNamedVolume(config, lifecycleManager, clusterResourceName(cluster));LOG.infov("Stopped k3s container for cluster {0}", cluster.getName());}+ String selectNodeInstanceType(String requestedType) {+ if (instanceTypeCatalog.find(requestedType).isPresent()) {+ return requestedType;+ }+ LOG.warnv("EKS node instance type {0} is absent from the EC2 catalog; using {1}",+ requestedType, DEFAULT_NODE_INSTANCE_TYPE);+ return DEFAULT_NODE_INSTANCE_TYPE;+ }++ private EksNodeCapacity.Limits resolveNodeCapacity(Cluster cluster) {+ try {+ Info host = lifecycleManager.getDockerClient().infoCmd().exec();+ CatalogInstanceType type = instanceTypeCatalog.find(nodeInstanceType(cluster)).orElseThrow();+ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(type, host.getMemTotal(), host.getNCPU(),+ config.services().eks().maxMemoryMib(), config.services().eks().maxVcpus());+ if (limits == null) {+ EksNodeCapacity.Limits configured = explicitCeilingWithoutHostInfo(cluster);+ if (configured != null) {+ return configured;+ }+ LOG.warnv("EKS cluster {0} cannot fit node type {1} and its kubelet reservations"+ + " within the Docker host; starting without resource limits",+ cluster.getName(), type.instanceType);+ } else if (limits.reducedReservations()) {+ LOG.warnv("EKS cluster {0} memory ceiling reduces kubelet memory reservations below"+ + " the EKS AMI defaults", cluster.getName());+ }+ return limits;+ } catch (Exception e) {+ LOG.warnv("EKS cluster {0} Docker host capacity unavailable: {1}",+ cluster.getName(), e.getMessage());+ return explicitCeilingWithoutHostInfo(cluster);+ }+ }++ private EksNodeCapacity.Limits explicitCeilingWithoutHostInfo(Cluster cluster) {+ int maxMemoryMib = config.services().eks().maxMemoryMib();+ int maxVcpus = config.services().eks().maxVcpus();+ if (maxMemoryMib <= 0 && maxVcpus <= 0) {+ return null;+ }+ int vcpus = instanceTypeCatalog.find(nodeInstanceType(cluster))+ .map(type -> maxVcpus > 0 ? Math.min(type.vcpu, maxVcpus) : 0)+ .orElse(maxVcpus);+ LOG.warnv("Applying explicit EKS cluster {0} ceiling without kubelet reservations"+ + " because Docker host capacity is unavailable", cluster.getName());+ return EksNodeCapacity.explicitCeilingWithoutHostInfo(maxMemoryMib, vcpus);+ }++ private static String capacityLabel(Cluster cluster, EksNodeCapacity.Limits limits) {+ if (limits == null) {+ return nodeInstanceType(cluster) + ":unbounded";+ }+ return nodeInstanceType(cluster) + ":" + limits.memoryBytes() + ":" + limits.vcpus()+ + ":" + limits.systemMemoryMib() + ":" + limits.kubeMemoryMib()+ + ":" + limits.evictionMemoryMib() + ":" + limits.systemCpuMilli()+ + ":" + limits.kubeCpuMilli() + ":" + limits.reducedReservations()+ + ":" + limits.kubeletArgsEnabled();+ }++ static String nodeInstanceType(Cluster cluster) {+ return cluster != null && cluster.getNodeInstanceType() != null+ ? cluster.getNodeInstanceType() : DEFAULT_NODE_INSTANCE_TYPE;+ }++ /** Recreate the shared node with its named data volume intact, restoring its old type on failure. */+ boolean restartForNodeCapacity(Cluster cluster, String previousType) {+ if (cluster.getContainerId() == null) {+ return true;+ }+ return replaceClusterContainer(cluster, previousType);+ }+/*** Releases Floci's hold on the given cluster (its metadata endpoint and log stream) and leaves* the k3s container running, so a later Floci start can re-latch it through@@ -2071,7 +2254,7 @@ public class EksClusterManagerinst.setInstanceId(instanceId);inst.setImageId("ami-eks-k3s");- inst.setInstanceType("m5.large");+ inst.setInstanceType(nodeInstanceType(cluster));inst.setPlacement(new Placement(az));inst.setRegion(safeRegion);inst.setState(InstanceState.running());diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksController.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksController.javaindex 03903d494c..9e41493265 100644--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksController.java+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksController.java@@ -92,11 +92,12 @@ public class EksController {* causes Jackson to omit the property due to @JsonInclude(NON_DEFAULT).*/private Cluster toClusterResponse(Cluster cluster) {- if (cluster == null || !cluster.isExplicitVersion()) {+ if (cluster == null || (!cluster.isExplicitVersion() && cluster.getNodeInstanceType() == null)) {return cluster;}Cluster response = cluster.copy();response.setExplicitVersion(false);+ response.setNodeInstanceType(null);return response;}diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksNodeCapacity.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksNodeCapacity.javanew file mode 100644index 0000000000..2a94d628f8--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksNodeCapacity.java@@ -0,0 +1,102 @@+package io.github.hectorvent.floci.services.eks;++import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogInstanceType;++import java.util.List;++/** The one calculation shared by the Docker limit and the kubelet's node budget. */+final class EksNodeCapacity {+ private static final long MIB = 1024L * 1024L;++ private EksNodeCapacity() {}++ record Limits(long memoryBytes, int vcpus, long systemMemoryMib, long kubeMemoryMib,+ long evictionMemoryMib, int systemCpuMilli, int kubeCpuMilli,+ boolean reducedReservations, boolean kubeletArgsEnabled) {+ void addKubeletArgs(List<String> args) {+ if (!kubeletArgsEnabled) {+ return;+ }+ args.add("--kubelet-arg=system-reserved=cpu=" + systemCpuMilli + "m,memory="+ + systemMemoryMib + "Mi");+ args.add("--kubelet-arg=kube-reserved=cpu=" + kubeCpuMilli + "m,memory="+ + kubeMemoryMib + "Mi");+ args.add("--kubelet-arg=eviction-hard=memory.available<" + evictionMemoryMib+ + "Mi,nodefs.available<10%,nodefs.inodesFree<5%");+ }+ }++ static Limits explicitCeilingWithoutHostInfo(int maxMemoryMib, int vcpus) {+ return new Limits((long) Math.max(0, maxMemoryMib) * MIB, vcpus,+ 0, 0, 0, 0, 0, false, false);+ }++ /**+ * Docker's host totals are also what kubelet sees in this nested setup. Its eviction threshold+ * includes the host-to-cgroup gap; otherwise the cgroup can OOM before kubelet sees pressure.+ * The EKS AMI kube reservation is 11 MiB per pod plus 255 MiB, and its CPU tiers are 6%,+ * 1%, 0.5%, then 0.25% of successive cores. The additional system memory is Floci's+ * container headroom, not an EKS AMI default.+ */+ static Limits calculate(CatalogInstanceType type, long hostMemoryBytes, int hostCpus,+ int maxMemoryMib, int maxVcpus) {+ if (hostMemoryBytes <= 0 || hostCpus <= 0 || maxMemoryMib < 0 || maxVcpus < 0) {+ return null;+ }+ long hostMib = hostMemoryBytes / MIB;+ long requestedMib = type.memoryMib + Math.max(128L, type.memoryMib / 10L);+ long hostLimitMib = Math.min(requestedMib, hostMib * 4 / 5);+ long memoryMib = hostLimitMib;+ if (maxMemoryMib > 0) {+ memoryMib = Math.min(memoryMib, maxMemoryMib);+ }+ int vcpus = Math.min(type.vcpu, hostCpus);+ if (maxVcpus > 0) {+ vcpus = Math.min(vcpus, maxVcpus);+ }+ if (vcpus <= 0) {+ return null;+ }++ int interfaces = type.networkCards.stream()+ .filter(card -> card.networkCardIndex != null && card.networkCardIndex == 0+ && card.maximumNetworkInterfaces != null)+ .mapToInt(card -> card.maximumNetworkInterfaces)+ .findFirst().orElse(0);+ int addresses = type.ipv4AddressesPerInterface == null ? 0 : type.ipv4AddressesPerInterface;+ int podCap = type.vcpu > 30 ? 250 : 110;+ int maxPods = interfaces > 0 && addresses > 0+ ? Math.min(podCap, interfaces * (addresses - 1) + 2) : podCap;+ long kubeMemoryMib = 11L * maxPods + 255;+ long systemMemoryMib = Math.max(128, memoryMib / 10);+ long evictionBufferMib = Math.min(100, memoryMib / 4);+ // Only an undersized host with no explicit ceiling falls back to the old unbounded mode.+ // An operator's ceiling must remain a hard bound, even if it leaves few resources for pods.+ if (maxMemoryMib == 0 && memoryMib < kubeMemoryMib + systemMemoryMib + evictionBufferMib + 256) {+ return null;+ }+ if (memoryMib <= 0) {+ return null;+ }+ boolean reducedReservations = maxMemoryMib > 0+ && memoryMib < kubeMemoryMib + systemMemoryMib + evictionBufferMib + 256;+ if (reducedReservations) {+ long reservableMib = Math.max(0, memoryMib - evictionBufferMib - 1);+ kubeMemoryMib = Math.min(kubeMemoryMib, reservableMib / 2);+ systemMemoryMib = Math.min(systemMemoryMib, reservableMib - kubeMemoryMib);+ }+ int kubeCpuMilli = cpuReservationMilli(vcpus);+ int systemCpuMilli = (hostCpus - vcpus) * 1000;+ long evictionMemoryMib = hostMib - memoryMib + evictionBufferMib;+ return new Limits(memoryMib * MIB, vcpus, systemMemoryMib, kubeMemoryMib,+ evictionMemoryMib, systemCpuMilli, kubeCpuMilli, reducedReservations, true);+ }++ private static int cpuReservationMilli(int vcpus) {+ int first = Math.min(vcpus, 1) * 60;+ int second = Math.min(Math.max(vcpus - 1, 0), 1) * 10;+ int third = Math.min(Math.max(vcpus - 2, 0), 2) * 5;+ int rest = Math.max(vcpus - 4, 0) * 25 / 10;+ return first + second + third + rest;+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksService.javaindex 9fd6315082..412e55733c 100644--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java@@ -48,6 +48,7 @@ import org.jboss.logging.Logger;import java.time.Instant;import java.util.ArrayList;+import java.util.Comparator;import java.util.HashMap;import java.util.HashSet;import java.util.HexFormat;@@ -102,6 +103,8 @@ public class EksService implements TagHandler, ResourceProvider {private final EksAddonService addons;private final ScheduledExecutorService poller = Executors.newSingleThreadScheduledExecutor();private final Map<String, UserDataClaim> appliedClusterUserData = new ConcurrentHashMap<>();+ private final Map<String, Object> nodeGroupCapacityLocks = new ConcurrentHashMap<>();+ private final Map<String, CompletableFuture<Boolean>> pendingFirstNodeGroups = new ConcurrentHashMap<>();@Injectpublic EksService(StorageFactory storageFactory, EmulatorConfig config,@@ -204,6 +207,10 @@ public class EksService implements TagHandler, ResourceProvider {LOG.infov("Restoring k3s container for persisted EKS cluster {0}", cluster.getName());cluster.setStatus(ClusterStatus.CREATING);cluster.setPodCidr(EksClusterManager.DEFAULT_POD_CIDR);+ if (cluster.getNodeInstanceType() == null) {+ firstNodeGroup(cluster.getName(), entry.accountId()).ifPresent(group ->+ cluster.setNodeInstanceType(selectedNodeInstanceType(group)));+ }clusterManager.restoreCluster(cluster);} catch (Exception e) {if (!clusterManager.isDockerReachable()) {@@ -233,6 +240,22 @@ public class EksService implements TagHandler, ResourceProvider {.toList();}+ private Optional<Nodegroup> firstNodeGroup(String clusterName, String accountId) {+ String prefix = clusterName + "/";+ List<Nodegroup> groups;+ if (nodeGroupStorage instanceof AccountAwareStorageBackend<Nodegroup> aware) {+ groups = aware.scanAllAccountEntries(key -> key.startsWith(prefix)).stream()+ .filter(entry -> accountId.equals(entry.accountId()))+ .map(AccountAwareStorageBackend.AccountEntry::value).toList();+ } else {+ groups = nodeGroupStorage.scan(key -> key.startsWith(prefix));+ }+ return groups.stream().filter(group -> group.getStatus() == NodegroupStatus.ACTIVE)+ .min(Comparator.comparing(Nodegroup::getCreatedAt,+ Comparator.nullsLast(Comparator.naturalOrder()))+ .thenComparing(Nodegroup::getNodegroupName));+ }+/*** Gives clusters persisted before IRSA support an OIDC issuer and signing key. Without this,* a cluster restored from {@code eks-clusters.json} would report no@@ -440,10 +463,15 @@ public class EksService implements TagHandler, ResourceProvider {if (!config.services().eks().mock()) {boolean keepRunning = config.services().eks().keepRunningOnShutdown();for (Cluster cluster : allClusters()) {- if (keepRunning) {- clusterManager.detachCluster(cluster);- } else {- clusterManager.stopCluster(cluster);+ try {+ if (keepRunning) {+ clusterManager.detachCluster(cluster);+ } else {+ clusterManager.stopCluster(cluster);+ }+ } catch (RuntimeException e) {+ LOG.warnv("Could not clean up EKS cluster {0} during shutdown: {1}",+ cluster.getName(), e.getMessage());}}}@@ -713,15 +741,130 @@ public class EksService implements TagHandler, ResourceProvider {nodeGroup.setLabels(request.getLabels() != null ? new HashMap<>(request.getLabels()) : null);nodeGroup.setTags(request.getTags() != null ? new HashMap<>(request.getTags()) : new HashMap<>());- if (launchTemplateData != null && launchTemplateData.getUserData() != null- && !launchTemplateData.getUserData().isBlank()) {- applyNodeGroupUserData(cluster, nodegroupName, launchTemplateData.getUserData(), nodeGroup);- }+ String capacityKey = accountId + "/" + clusterName;+ Object capacityLock = nodeGroupCapacityLocks.computeIfAbsent(capacityKey,+ ignored -> new Object());+ boolean hasUserData = launchTemplateData != null && launchTemplateData.getUserData() != null+ && !launchTemplateData.getUserData().isBlank();+ Cluster currentCluster;+ CompletableFuture<Boolean> pendingFirst;+ boolean ownsPendingFirst = false;+ synchronized (capacityLock) {+ if (nodeGroupStorage.get(storageKey).isPresent()) {+ throw new AwsException("ResourceInUseException",+ "Nodegroup already exists: " + nodegroupName, 409);+ }+ currentCluster = describeCluster(clusterName);+ pendingFirst = pendingFirstNodeGroups.get(capacityKey);+ boolean firstGroup = firstNodeGroup(clusterName, accountId).isEmpty() && pendingFirst == null;+ if (firstGroup && !config.services().eks().mock() && clusterManager != null) {+ applyFirstNodeGroupCapacity(clusterName, nodegroupName, currentCluster, nodeGroup);+ } else if (!firstGroup && pendingFirst == null+ && !config.services().eks().mock() && clusterManager != null) {+ if (currentCluster.getNodeInstanceType() == null) {+ firstNodeGroup(clusterName, accountId).ifPresent(group ->+ currentCluster.setNodeInstanceType(selectedNodeInstanceType(group)));+ storage.put(clusterName, currentCluster);+ }+ if (!nodeGroup.getInstanceTypes().isEmpty()+ && !nodeGroup.getInstanceTypes().contains(currentCluster.getNodeInstanceType())) {+ LOG.warnv("EKS cluster {0} has one shared node; nodegroup {1} cannot change its capacity from {2}",+ clusterName, nodegroupName, currentCluster.getNodeInstanceType());+ }+ }- nodeGroupStorage.put(storageKey, nodeGroup);+ if (nodeGroup.getStatus() == NodegroupStatus.ACTIVE && (hasUserData || pendingFirst != null)) {+ nodeGroup.setStatus(NodegroupStatus.CREATING);+ if (firstGroup && hasUserData) {+ pendingFirst = new CompletableFuture<>();+ ownsPendingFirst = true;+ }+ }+ nodeGroupStorage.put(storageKey, nodeGroup);+ if (ownsPendingFirst) {+ pendingFirstNodeGroups.put(capacityKey, pendingFirst);+ }+ }+ if (nodeGroup.getStatus() == NodegroupStatus.CREATING && hasUserData) {+ try {+ applyNodeGroupUserData(currentCluster, nodegroupName, launchTemplateData.getUserData(), nodeGroup);+ } catch (Throwable e) {+ synchronized (capacityLock) {+ if (nodeGroupStorage.get(storageKey)+ .filter(saved -> arn.equals(saved.getNodegroupArn())).isPresent()) {+ nodeGroupStorage.delete(storageKey);+ }+ if (ownsPendingFirst) {+ pendingFirstNodeGroups.remove(capacityKey, pendingFirst);+ pendingFirst.complete(false);+ }+ }+ throw e;+ }+ }+ if (pendingFirst != null && !ownsPendingFirst) {+ pendingFirst.join();+ }+ synchronized (capacityLock) {+ boolean stored = false;+ try {+ if (nodeGroupStorage.get(storageKey)+ .filter(saved -> arn.equals(saved.getNodegroupArn())).isPresent()) {+ if (nodeGroup.getStatus() == NodegroupStatus.CREATING) {+ if (!ownsPendingFirst && firstNodeGroup(clusterName, accountId).isEmpty()+ && !config.services().eks().mock() && clusterManager != null) {+ Cluster refreshedCluster = describeCluster(clusterName);+ applyFirstNodeGroupCapacity(clusterName, nodegroupName, refreshedCluster, nodeGroup);+ }+ if (nodeGroup.getStatus() == NodegroupStatus.CREATING) {+ nodeGroup.setStatus(NodegroupStatus.ACTIVE);+ }+ }+ nodeGroupStorage.put(storageKey, nodeGroup);+ stored = true;+ }+ } finally {+ if (ownsPendingFirst) {+ pendingFirstNodeGroups.remove(capacityKey, pendingFirst);+ pendingFirst.complete(stored && nodeGroup.getStatus() == NodegroupStatus.ACTIVE);+ }+ }+ }return nodeGroup;}+ private void applyFirstNodeGroupCapacity(String clusterName, String nodegroupName,+ Cluster currentCluster, Nodegroup nodeGroup) {+ String selectedType = selectedNodeInstanceType(nodeGroup);+ String previousType = EksClusterManager.nodeInstanceType(currentCluster);+ currentCluster.setNodeInstanceType(selectedType);+ if (currentCluster.getContainerId() != null && !selectedType.equals(previousType)) {+ currentCluster.setStatus(ClusterStatus.CREATING);+ try {+ if (!clusterManager.restartForNodeCapacity(currentCluster, previousType)) {+ nodeGroup.setStatus(NodegroupStatus.CREATE_FAILED);+ nodeGroup.setHealth(failedNodeGroupHealth(nodegroupName,+ "Could not apply node capacity; the previous node was restored"));+ }+ } catch (RuntimeException e) {+ currentCluster.setNodeInstanceType(previousType);+ currentCluster.setStatus(ClusterStatus.FAILED);+ nodeGroup.setStatus(NodegroupStatus.CREATE_FAILED);+ nodeGroup.setHealth(failedNodeGroupHealth(nodegroupName, e.getMessage()));+ LOG.errorv("EKS cluster {0} could not recover after node capacity change: {1}",+ clusterName, e.getMessage());+ }+ }+ storage.put(clusterName, currentCluster);+ }++ private String selectedNodeInstanceType(Nodegroup group) {+ String requestedType = group.getInstanceTypes() == null || group.getInstanceTypes().isEmpty()+ ? null : group.getInstanceTypes().getFirst();+ String selectedType = clusterManager.selectNodeInstanceType(requestedType);+ return selectedType != null ? selectedType : EksClusterManager.DEFAULT_NODE_INSTANCE_TYPE;+ }+private LaunchTemplateData validateLaunchTemplate(String region, Object launchTemplateObj) {if (!(launchTemplateObj instanceof Map<?, ?> map)) {return null;@@ -775,11 +918,15 @@ public class EksService implements TagHandler, ResourceProvider {}public Nodegroup deleteNodeGroup(String clusterName, String nodegroupName) {- Nodegroup nodeGroup = describeNodeGroup(clusterName, nodegroupName);- nodeGroup.setStatus(NodegroupStatus.DELETING);- nodeGroup.setModifiedAt(Instant.now());- nodeGroupStorage.delete(nodeGroupKey(clusterName, nodegroupName));- return nodeGroup;+ Object capacityLock = nodeGroupCapacityLocks.computeIfAbsent(+ regionResolver.getAccountId() + "/" + clusterName, ignored -> new Object());+ synchronized (capacityLock) {+ Nodegroup nodeGroup = describeNodeGroup(clusterName, nodegroupName);+ nodeGroup.setStatus(NodegroupStatus.DELETING);+ nodeGroup.setModifiedAt(Instant.now());+ nodeGroupStorage.delete(nodeGroupKey(clusterName, nodegroupName));+ return nodeGroup;+ }}public FargateProfile createFargateProfile(String clusterName, CreateFargateProfileRequest request) {diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/model/Cluster.java b/src/main/java/io/github/hectorvent/floci/services/eks/model/Cluster.javaindex 2843690687..38243c63fe 100644--- a/src/main/java/io/github/hectorvent/floci/services/eks/model/Cluster.java+++ b/src/main/java/io/github/hectorvent/floci/services/eks/model/Cluster.java@@ -96,6 +96,11 @@ public class Cluster {@JsonIgnoreprivate String dockerName;+ /** Instance type selected for the single shared node container. Internal persisted metadata. */+ @JsonProperty("nodeInstanceType")+ @JsonInclude(JsonInclude.Include.NON_NULL)+ private String nodeInstanceType;+private AccessConfig accessConfig;public AccessConfig getAccessConfig() { return accessConfig; }@@ -145,6 +150,9 @@ public class Cluster {public String getContainerId() { return containerId; }public void setContainerId(String containerId) { this.containerId = containerId; }+ public String getNodeInstanceType() { return nodeInstanceType; }+ public void setNodeInstanceType(String nodeInstanceType) { this.nodeInstanceType = nodeInstanceType; }+public String getAccountId() { return accountId; }public void setAccountId(String accountId) { this.accountId = accountId; }@@ -193,6 +201,7 @@ public class Cluster {c.podCidr = this.podCidr;c.explicitVersion = this.explicitVersion;c.dockerName = this.dockerName;+ c.nodeInstanceType = this.nodeInstanceType;c.accessConfig = this.accessConfig;return c;}diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex 60a78fb133..1ccadd0247 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -647,6 +647,8 @@ floci:eks:enabled: truemock: false+ max-memory-mib: 0 # optional k3s container memory ceiling; 0 follows the node type+ max-vcpus: 0 # optional k3s container CPU ceiling; 0 follows the node typeprovider: k3sdefault-image: "rancher/k3s:latest"# image-template: "myregistry.io/k3s:v%s" # optional custom image format template taking cluster versiondiff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecTest.javaindex 88c316dbb5..bfac749398 100644--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecTest.java+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecTest.java@@ -12,6 +12,8 @@ import com.github.dockerjava.api.model.StreamType;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;+import java.io.Closeable;+import java.io.IOException;import java.nio.charset.StandardCharsets;import java.util.List;import java.util.function.Consumer;@@ -106,8 +108,10 @@ class ContainerExecTest {}@Test- void aCommandThatOutlivesItsTimeoutIsReportedWithWhatItPrintedSoFar() {+ void aCommandThatOutlivesItsTimeoutIsReportedWithWhatItPrintedSoFar() throws IOException {+ Closeable attach = mock(Closeable.class);startWith(callback -> {+ callback.onStart(attach);callback.onNext(frame(StreamType.STDOUT, "partial"));callback.onNext(frame(StreamType.STDERR, "warning"));});@@ -118,6 +122,7 @@ class ContainerExecTest {assertEquals(-1L, result.exitCode());assertEquals("partial", result.stdout());assertEquals("warning\nTimed out after 0s", result.stderr());+ verify(attach).close();verify(dockerClient, never()).inspectExecCmd(EXEC_ID);}@@ -172,13 +177,14 @@ class ContainerExecTest {}@Test- void anInterruptIsRaisedAndTheInterruptFlagKept() {- startWith(callback -> {- });+ void anInterruptIsRaisedAndTheInterruptFlagKept() throws IOException {+ Closeable attach = mock(Closeable.class);+ startWith(callback -> callback.onStart(attach));Thread.currentThread().interrupt();try {assertThrows(IllegalStateException.class, () -> ContainerExec.run(dockerClient, CONTAINER_ID, CMD, 5));assertTrue(Thread.currentThread().isInterrupted());+ verify(attach).close();} finally {Thread.interrupted();}diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.javaindex fa3a9c2d69..5ad7b7cf3f 100644--- a/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java@@ -39,7 +39,9 @@ import com.github.dockerjava.api.command.InspectExecResponse;import com.github.dockerjava.api.command.InspectVolumeCmd;import com.github.dockerjava.api.model.Container;import com.github.dockerjava.api.model.Frame;+import com.github.dockerjava.api.model.Info;import com.github.dockerjava.api.model.NetworkSettings;+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog;import io.github.hectorvent.floci.services.ec2.Ec2MetadataServer;import io.github.hectorvent.floci.services.ec2.model.Instance;import org.junit.jupiter.api.BeforeEach;@@ -319,7 +321,8 @@ class EksClusterManagerTest {"io.floci.service", "eks","io.floci.resource-id", "my-cluster","io.floci.account", "000000000000",- "io.floci.region", "us-east-1"));+ "io.floci.region", "us-east-1",+ "io.floci.eks.node-capacity", "m5.large:unbounded"));}@ParameterizedTest@@ -362,6 +365,7 @@ class EksClusterManagerTest {private EmulatorConfig.StorageConfig storage;private ContainerLifecycleManager lifecycleManager;private PortAllocator portAllocator;+ private DockerClient dockerClient;private EksClusterManager manager;@BeforeEach@@ -388,6 +392,8 @@ class EksClusterManagerTest {when(containerBuilder.newContainer(anyString())).thenReturn(builder);when(builder.build()).thenReturn(Mockito.mock(ContainerSpec.class));portAllocator = Mockito.mock(PortAllocator.class);+ dockerClient = Mockito.mock(DockerClient.class, Mockito.RETURNS_DEEP_STUBS);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);manager = new EksClusterManager(containerBuilder, lifecycleManager,Mockito.mock(ContainerDetector.class), portAllocator,@@ -397,12 +403,14 @@ class EksClusterManagerTest {// Container's getters are final, so survivors are built from JSON instead of mocked.private Container survivingContainer(String id) {- return containerFromJson("{\"Id\":\"" + id + "\"}");+ return containerFromJson("{\"Id\":\"" + id + "\","+ + "\"Labels\":{\"io.floci.eks.node-capacity\":\"m5.large:unbounded\"}}");}private Container survivingContainerOwnedBy(String id, String accountId) {return containerFromJson("{\"Id\":\"" + id + "\","- + "\"Labels\":{\"io.floci.account\":\"" + accountId + "\"}}");+ + "\"Labels\":{\"io.floci.account\":\"" + accountId + "\","+ + "\"io.floci.eks.node-capacity\":\"m5.large:unbounded\"}}");}private Container containerFromJson(String json) {@@ -446,6 +454,136 @@ class EksClusterManagerTest {verify(lifecycleManager, never()).create(any());}+ @Test+ void recreatesAnOldSurvivorWithoutCapacityLimits() {+ when(lifecycleManager.findByName("floci-eks-demo"))+ .thenReturn(Optional.of(containerFromJson("{\"Id\":\"cid-old\"}")));+ when(lifecycleManager.adopt("cid-old", List.of(6443)))+ .thenReturn(new ContainerInfo("cid-old", Map.of(), Map.of(6443, 6440)));+ stubFreshStart("cid-new", 6440);++ Cluster cluster = cluster();+ manager.restoreCluster(cluster);++ assertEquals("cid-new", cluster.getContainerId());+ verify(lifecycleManager).create(any());+ verify(portAllocator, never()).allocate(6440, 6499);+ verify(dockerClient).stopContainerCmd("cid-old");+ }++ @Test+ void restoresOldSurvivorWhenCapacityReplacementFails() {+ when(lifecycleManager.findByName("floci-eks-demo"))+ .thenReturn(Optional.of(containerFromJson("{\"Id\":\"cid-old\"}")));+ when(lifecycleManager.adopt("cid-old", List.of(6443)))+ .thenReturn(new ContainerInfo("cid-old", Map.of(), Map.of(6443, 6440)));+ when(lifecycleManager.create(any())).thenReturn("cid-new");+ when(lifecycleManager.startCreated(any(), any()))+ .thenThrow(new RuntimeException("replacement failed"));++ Cluster cluster = cluster();+ manager.restoreCluster(cluster);++ assertEquals("cid-old", cluster.getContainerId());+ assertEquals(6440, cluster.getHostPort());+ verify(lifecycleManager).removeIfExistsStrict("floci-eks-demo");+ verify(dockerClient).stopContainerCmd("cid-old");+ verify(portAllocator, never()).allocate(6440, 6499);+ }++ @Test+ void firstNodeGroupReplacementKeepsTheClusterEndpoint() {+ when(lifecycleManager.create(any())).thenReturn("cid-new");+ when(lifecycleManager.startCreated(any(), any()))+ .thenReturn(new ContainerInfo("cid-new", Map.of()));+ Cluster cluster = cluster();+ cluster.setDockerName("floci-eks-demo");+ cluster.setContainerId("cid-old");+ cluster.setHostPort(6441);+ cluster.setNodeInstanceType("t3.medium");++ assertTrue(manager.restartForNodeCapacity(cluster, "m5.large"));++ assertEquals("cid-new", cluster.getContainerId());+ assertEquals(6441, cluster.getHostPort());+ assertEquals("https://localhost:6441", cluster.getEndpoint());+ verify(portAllocator, never()).allocate(6440, 6499);+ verify(portAllocator, never()).release(6441);+ verify(dockerClient).stopContainerCmd("cid-old");+ verify(lifecycleManager, Mockito.times(2))+ .removeIfExistsStrict("floci-eks-demo.capacity-backup");+ }++ @Test+ void capacityBackupDoesNotCollideWithAnotherClusterName() {+ when(lifecycleManager.create(any())).thenReturn("cid-new");+ when(lifecycleManager.startCreated(any(), any()))+ .thenReturn(new ContainerInfo("cid-new", Map.of()));+ Cluster cluster = cluster();+ cluster.setName("foo");+ cluster.setDockerName("floci-eks-foo");+ cluster.setContainerId("cid-foo");+ cluster.setHostPort(6441);+ cluster.setNodeInstanceType("t3.medium");+ Cluster other = cluster();+ other.setName("foo-capacity-backup");+ other.setDockerName("floci-eks-foo-capacity-backup");+ other.setContainerId("cid-other");++ assertTrue(manager.restartForNodeCapacity(cluster, "m5.large"));+ manager.stopCluster(cluster);++ assertTrue(other.getName().matches(EksService.CLUSTER_NAME_REGEX));+ assertFalse("foo.capacity-backup".matches(EksService.CLUSTER_NAME_REGEX));+ assertEquals("floci-eks-foo-capacity-backup", manager.clusterResourceName(other));+ verify(lifecycleManager, Mockito.times(3))+ .removeIfExistsStrict("floci-eks-foo.capacity-backup");+ verify(lifecycleManager, never()).removeIfExistsStrict(other.getDockerName());+ verify(lifecycleManager, never()).stopAndRemove("cid-other", null);+ }++ @Test+ void failedFirstNodeGroupReplacementRestoresPreviousContainerAndType() {+ when(lifecycleManager.create(any())).thenReturn("cid-new");+ when(lifecycleManager.startCreated(any(), any()))+ .thenThrow(new IllegalStateException("replacement failed"));+ when(lifecycleManager.adopt("cid-old", List.of(6443)))+ .thenReturn(new ContainerInfo("cid-old", Map.of(), Map.of(6443, 6441)));+ Cluster cluster = cluster();+ cluster.setDockerName("floci-eks-demo");+ cluster.setContainerId("cid-old");+ cluster.setHostPort(6441);+ cluster.setNodeInstanceType("t3.medium");++ assertFalse(manager.restartForNodeCapacity(cluster, "m5.large"));++ assertEquals("m5.large", cluster.getNodeInstanceType());+ assertEquals("cid-old", cluster.getContainerId());+ assertEquals(6441, cluster.getHostPort());+ assertEquals("https://localhost:6441", cluster.getEndpoint());+ verify(portAllocator, never()).allocate(6440, 6499);+ verify(portAllocator, never()).release(6441);+ verify(lifecycleManager, never()).removeVolume(anyString());+ }++ @Test+ void backupCleanupFailureKeepsTheNewNodeRunning() {+ when(lifecycleManager.findByName("floci-eks-demo"))+ .thenReturn(Optional.of(containerFromJson("{\"Id\":\"cid-old\"}")));+ when(lifecycleManager.adopt("cid-old", List.of(6443)))+ .thenReturn(new ContainerInfo("cid-old", Map.of(), Map.of(6443, 6440)));+ stubFreshStart("cid-new", 6440);+ Mockito.doNothing().doThrow(new IllegalStateException("Docker cleanup failed"))+ .when(lifecycleManager).removeIfExistsStrict("floci-eks-demo.capacity-backup");++ Cluster cluster = cluster();+ manager.restoreCluster(cluster);++ assertEquals("cid-new", cluster.getContainerId());+ verify(lifecycleManager, Mockito.times(1)).adopt("cid-old", List.of(6443));+ verify(lifecycleManager, never()).removeIfExistsStrict("floci-eks-demo");+ }+@Testvoid recreatesTheContainerWhenNoneSurvives() {when(lifecycleManager.findByName("floci-eks-demo")).thenReturn(Optional.empty());@@ -504,6 +642,20 @@ class EksClusterManagerTest {verify(lifecycleManager, never()).removeVolume(anyString());}+ @Test+ void backupCleanupFailureDoesNotLeaveTheLiveNodeRunning() {+ Cluster cluster = cluster();+ cluster.setDockerName("floci-eks-demo");+ cluster.setContainerId("cid-new");+ Mockito.doThrow(new IllegalStateException("Docker cleanup failed"))+ .when(lifecycleManager).removeIfExistsStrict("floci-eks-demo.capacity-backup");++ assertThrows(IllegalStateException.class, () -> manager.stopCluster(cluster));++ verify(lifecycleManager).stopAndRemove("cid-new", null);+ verify(lifecycleManager, never()).removeVolume(anyString());+ }+@Testvoid stopClusterRemovesTheDataVolumeInMemoryStorageMode() {when(storage.mode()).thenReturn("memory");@@ -1454,6 +1606,8 @@ class EksClusterManagerTest {Container container = Mockito.mock(Container.class);when(container.getId()).thenReturn("surviving-container-id");+ when(container.getLabels()).thenReturn(Map.of(+ "io.floci.eks.node-capacity", "m5.large:unbounded"));when(lifecycleManager.findByName("floci-eks-my-cluster")).thenReturn(Optional.of(container));when(lifecycleManager.adopt(anyString(), any())).thenReturn(new ContainerInfo("surviving-container-id", Map.of(6443, new ContainerLifecycleManager.EndpointInfo("localhost", 6500)), Map.of(6443, 6500)));@@ -1745,6 +1899,74 @@ class EksClusterManagerTest {config, regionResolver, null, Mockito.mock(EksOidcService.class));}+ @Test+ void nodeGroupTypeControlsContainerAndSynthesizedInstance() {+ DockerClient dockerClient = Mockito.mock(DockerClient.class, Mockito.RETURNS_DEEP_STUBS);+ Info host = Mockito.mock(Info.class);+ when(host.getMemTotal()).thenReturn(16L * 1024 * 1024 * 1024);+ when(host.getNCPU()).thenReturn(8);+ when(dockerClient.infoCmd().exec()).thenReturn(host);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);++ Cluster cluster = new Cluster();+ cluster.setName("capacity-cluster");+ cluster.setNodeInstanceType(manager.selectNodeInstanceType("t3.medium"));+ manager.startCluster(cluster);++ Ec2InstanceTypeCatalog catalog = new Ec2InstanceTypeCatalog();+ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(+ catalog.find("t3.medium").orElseThrow(), host.getMemTotal(), host.getNCPU(), 0, 0);+ verify(builder).withMemoryBytes(limits.memoryBytes());+ verify(builder).withCpuUnits(limits.vcpus() * 1024);+ @SuppressWarnings("unchecked")+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);+ verify(builder).withCmd(cmdCaptor.capture());+ List<String> expectedArgs = new ArrayList<>();+ limits.addKubeletArgs(expectedArgs);+ assertTrue(cmdCaptor.getValue().containsAll(expectedArgs));+ assertEquals("t3.medium", manager.synthesizeClusterNodeInstance(+ cluster, "172.17.0.2", "us-east-1", "000000000000").getInstanceType());+ }++ @Test+ void hostTooSmallLeavesClusterUnboundedAndStarting() {+ DockerClient dockerClient = Mockito.mock(DockerClient.class, Mockito.RETURNS_DEEP_STUBS);+ Info host = Mockito.mock(Info.class);+ when(host.getMemTotal()).thenReturn(1024L * 1024 * 1024);+ when(host.getNCPU()).thenReturn(2);+ when(dockerClient.infoCmd().exec()).thenReturn(host);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);++ Cluster cluster = new Cluster();+ cluster.setName("small-host");+ assertDoesNotThrow(() -> manager.startCluster(cluster));+ verify(builder, never()).withMemoryBytes(Mockito.anyLong());+ verify(builder, never()).withCpuUnits(Mockito.anyInt());+ @SuppressWarnings("unchecked")+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);+ verify(builder).withCmd(cmdCaptor.capture());+ assertFalse(cmdCaptor.getValue().stream().anyMatch(arg -> arg.contains("reserved=")+ || arg.contains("eviction-hard=")));+ }++ @Test+ void explicitCeilingSurvivesMissingDockerHostInformation() {+ when(eks.maxMemoryMib()).thenReturn(512);+ when(eks.maxVcpus()).thenReturn(1);+ Cluster cluster = new Cluster();+ cluster.setName("capped-cluster");++ manager.startCluster(cluster);++ verify(builder).withMemoryBytes(512L * 1024 * 1024);+ verify(builder).withCpuUnits(1024);+ @SuppressWarnings("unchecked")+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);+ verify(builder).withCmd(cmdCaptor.capture());+ assertFalse(cmdCaptor.getValue().stream().anyMatch(arg -> arg.contains("reserved=")+ || arg.contains("eviction-hard=")));+ }+@Testvoid resolveClusterImageMapsSupportedVersions() {Cluster cluster = new Cluster();@@ -2208,6 +2430,8 @@ class EksClusterManagerTest {void restoreClusterAttachesLogsForAdoptedContainer() {Container container = Mockito.mock(Container.class);when(container.getId()).thenReturn("adopted-container-id-12345678901234567890");+ when(container.getLabels()).thenReturn(Map.of(+ "io.floci.eks.node-capacity", "m5.large:unbounded"));when(lifecycleManager.findByName("floci-eks-prod-cluster")).thenReturn(Optional.of(container));when(lifecycleManager.adopt(anyString(), any())).thenReturn(new ContainerInfo("adopted-container-id-12345678901234567890",diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksLaunchTemplateUserDataTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksLaunchTemplateUserDataTest.javaindex 8684b1ccc8..e39ca3c573 100644--- a/src/test/java/io/github/hectorvent/floci/services/eks/EksLaunchTemplateUserDataTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksLaunchTemplateUserDataTest.java@@ -342,6 +342,52 @@ class EksLaunchTemplateUserDataTest {}}+ @Test+ void laterGroupWaitsForFirstBootstrapBeforeBecomingActive() throws InterruptedException {+ String script = "#!/bin/bash\nexit 1\n";+ String encoded = Base64.getEncoder().encodeToString(script.getBytes(StandardCharsets.UTF_8));+ String ltId = createLaunchTemplate("lt-pending-first", encoded);+ CountDownLatch enteredExecution = new CountDownLatch(1);+ CountDownLatch releaseExecution = new CountDownLatch(1);+ when(clusterManager.executeUserData(any(Cluster.class), eq("ng-1"), eq(encoded)))+ .thenAnswer(invocation -> {+ enteredExecution.countDown();+ assertTrue(releaseExecution.await(5, TimeUnit.SECONDS));+ return UserDataPipeline.ExecutionResult.failed(1L, 1, 1, "bootstrap failed");+ });++ CreateNodeGroupRequest firstRequest = nodeGroupRequest("ng-1");+ firstRequest.setLaunchTemplate(Map.of("id", ltId));+ AtomicReference<Nodegroup> firstResult = new AtomicReference<>();+ Thread first = new Thread(() -> firstResult.set(eksService.createNodeGroup(CLUSTER_NAME, firstRequest)));+ first.start();+ try {+ assertTrue(enteredExecution.await(5, TimeUnit.SECONDS));+ assertEquals(NodegroupStatus.CREATING,+ eksService.describeNodeGroup(CLUSTER_NAME, "ng-1").getStatus());++ AtomicReference<Nodegroup> secondResult = new AtomicReference<>();+ Thread second = new Thread(() -> secondResult.set(+ eksService.createNodeGroup(CLUSTER_NAME, nodeGroupRequest("ng-2"))));+ second.start();+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);+ while (!eksService.listNodeGroups(CLUSTER_NAME).contains("ng-2")+ && System.nanoTime() < deadline) {+ Thread.sleep(10);+ }+ assertEquals(NodegroupStatus.CREATING,+ eksService.describeNodeGroup(CLUSTER_NAME, "ng-2").getStatus());++ releaseExecution.countDown();+ first.join(5000);+ second.join(5000);+ assertEquals(NodegroupStatus.CREATE_FAILED, firstResult.get().getStatus());+ assertEquals(NodegroupStatus.ACTIVE, secondResult.get().getStatus());+ } finally {+ releaseExecution.countDown();+ }+ }+@Testvoid unstubbedClusterManagerDoesNotThrowNpe() {String script = "#!/bin/bash\necho 'unstubbed'\n";diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksNodeCapacityTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksNodeCapacityTest.javanew file mode 100644index 0000000000..508a763236--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksNodeCapacityTest.java@@ -0,0 +1,84 @@+package io.github.hectorvent.floci.services.eks;++import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog;+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogInstanceType;+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogNetworkCard;+import org.junit.jupiter.api.Test;++import java.util.ArrayList;+import java.util.List;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertNotNull;+import static org.junit.jupiter.api.Assertions.assertNull;+import static org.junit.jupiter.api.Assertions.assertTrue;++class EksNodeCapacityTest {+ private static final long MIB = 1024L * 1024L;+ private final Ec2InstanceTypeCatalog catalog = new Ec2InstanceTypeCatalog();++ @Test+ void reservationsFollowTheAppliedLimitAndTheInstancePodDensity() {+ CatalogInstanceType type = catalog.find("m5.large").orElseThrow();+ long hostMib = 16384;+ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(type, hostMib * MIB, 8, 0, 0);+ long limitMib = limits.memoryBytes() / MIB;+ int maxPods = type.networkCards.getFirst().maximumNetworkInterfaces+ * (type.ipv4AddressesPerInterface - 1) + 2;++ assertEquals(type.memoryMib + Math.max(128, type.memoryMib / 10), limitMib);+ assertEquals(type.vcpu, limits.vcpus());+ assertEquals(255 + 11L * maxPods, limits.kubeMemoryMib());+ assertEquals(Math.max(128, limitMib / 10), limits.systemMemoryMib());+ assertEquals(hostMib - limitMib + 100, limits.evictionMemoryMib());+ assertEquals((8 - limits.vcpus()) * 1000, limits.systemCpuMilli());+ assertEquals(70, limits.kubeCpuMilli());++ List<String> args = new ArrayList<>();+ limits.addKubeletArgs(args);+ assertTrue(args.contains("--kubelet-arg=system-reserved=cpu=" + limits.systemCpuMilli()+ + "m,memory=" + limits.systemMemoryMib() + "Mi"));+ assertTrue(args.contains("--kubelet-arg=kube-reserved=cpu=" + limits.kubeCpuMilli()+ + "m,memory=" + limits.kubeMemoryMib() + "Mi"));+ assertTrue(args.contains("--kubelet-arg=eviction-hard=memory.available<"+ + limits.evictionMemoryMib() + "Mi,nodefs.available<10%,nodefs.inodesFree<5%"));+ }++ @Test+ void hostAndUserCeilingsBoundTheContainerAndReservations() {+ CatalogInstanceType type = catalog.find("m5.large").orElseThrow();+ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(type, 8192 * MIB, 4, 4096, 1);+ assertEquals(4096 * MIB, limits.memoryBytes());+ assertEquals(1, limits.vcpus());+ assertEquals(60, limits.kubeCpuMilli());+ assertEquals(3000, limits.systemCpuMilli());+ assertEquals(8192 - 4096 + 100, limits.evictionMemoryMib());+ }++ @Test+ void largeManagedNodeUsesThe250PodCapForMemoryReservation() {+ CatalogInstanceType type = new CatalogInstanceType();+ type.vcpu = 32;+ type.memoryMib = 131072;+ type.ipv4AddressesPerInterface = 50;+ CatalogNetworkCard card = new CatalogNetworkCard();+ card.networkCardIndex = 0;+ card.maximumNetworkInterfaces = 15;+ type.networkCards = List.of(card);++ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(type, 262144 * MIB, 64, 0, 0);+ assertEquals(255 + 11 * 250, limits.kubeMemoryMib());+ }++ @Test+ void tooSmallHostOrMissingHostInformationFallsBackToUnbounded() {+ CatalogInstanceType type = catalog.find("m5.large").orElseThrow();+ assertNull(EksNodeCapacity.calculate(type, 1024 * MIB, 2, 0, 0));+ assertNull(EksNodeCapacity.calculate(type, 0, 0, 0, 0));+ EksNodeCapacity.Limits capped = EksNodeCapacity.calculate(type, 8192 * MIB, 2, 512, 0);+ assertNotNull(capped);+ assertEquals(512 * MIB, capped.memoryBytes());+ assertTrue(capped.kubeMemoryMib() + capped.systemMemoryMib()+ + 100 < capped.memoryBytes() / MIB);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.javaindex 090370b885..2ecca6c458 100644--- a/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java@@ -52,14 +52,21 @@ import java.nio.file.Path;import java.util.List;import java.util.Map;import java.util.Optional;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.Future;+import java.util.concurrent.TimeUnit;import java.util.stream.Collectors;import static org.junit.jupiter.api.Assertions.*;import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyString;import static org.mockito.Mockito.doThrow;import static org.mockito.Mockito.mock;import static org.mockito.Mockito.never;import static org.mockito.Mockito.spy;+import static org.mockito.Mockito.times;import static org.mockito.Mockito.verify;import static org.mockito.Mockito.when;@@ -901,6 +908,69 @@ class EksServiceTest {assertEquals("1.30-eks-1", nodeGroup.getReleaseVersion());}+ @Test+ void firstNodeGroupSelectsAndRestartsTheSharedNodeOnce() {+ EksClusterManager clusterManager = mock(EksClusterManager.class);+ when(clusterManager.tryStartCluster(any())).thenReturn(true);+ when(clusterManager.selectNodeInstanceType("t3.medium")).thenReturn("t3.medium");+ when(clusterManager.restartForNodeCapacity(any(), anyString())).thenReturn(true);+ EksService service = newService(clusterManager, false);+ CreateClusterRequest request = new CreateClusterRequest();+ request.setName("capacity-cluster");+ request.setRoleArn("arn:aws:iam::000000000000:role/eks-role");+ Cluster cluster = service.createCluster(request);+ cluster.setContainerId("existing-container");++ service.createNodeGroup("capacity-cluster", nodeGroupRequest("first"));+ assertEquals("t3.medium", cluster.getNodeInstanceType());+ verify(clusterManager).restartForNodeCapacity(cluster, "m5.large");++ service.createNodeGroup("capacity-cluster", nodeGroupRequest("second"));+ verify(clusterManager).restartForNodeCapacity(cluster, "m5.large");+ }++ @Test+ void concurrentFirstNodeGroupsChooseOnlyOneCapacity() throws Exception {+ EksClusterManager clusterManager = mock(EksClusterManager.class);+ when(clusterManager.tryStartCluster(any())).thenReturn(true);+ when(clusterManager.selectNodeInstanceType("t3.medium")).thenReturn("t3.medium");+ CountDownLatch restartEntered = new CountDownLatch(1);+ CountDownLatch releaseRestart = new CountDownLatch(1);+ when(clusterManager.restartForNodeCapacity(any(), anyString())).thenAnswer(invocation -> {+ restartEntered.countDown();+ if (!releaseRestart.await(10, TimeUnit.SECONDS)) {+ throw new IllegalStateException("Timed out waiting to finish the first restart");+ }+ return true;+ });+ EksService service = newService(clusterManager, false);+ CreateClusterRequest request = new CreateClusterRequest();+ request.setName("concurrent-capacity");+ request.setRoleArn("arn:aws:iam::000000000000:role/eks-role");+ Cluster cluster = service.createCluster(request);+ cluster.setContainerId("existing-container");++ ExecutorService executor = Executors.newFixedThreadPool(2);+ try {+ Future<Nodegroup> first = executor.submit(() -> service.createNodeGroup(+ "concurrent-capacity", nodeGroupRequest("first")));+ assertTrue(restartEntered.await(10, TimeUnit.SECONDS));+ CreateNodeGroupRequest secondRequest = nodeGroupRequest("second");+ secondRequest.setInstanceTypes(List.of("m5.large"));+ Future<Nodegroup> second = executor.submit(() -> service.createNodeGroup(+ "concurrent-capacity", secondRequest));+ releaseRestart.countDown();+ first.get(10, TimeUnit.SECONDS);+ second.get(10, TimeUnit.SECONDS);++ assertEquals("t3.medium", cluster.getNodeInstanceType());+ verify(clusterManager, times(1)).restartForNodeCapacity(cluster, "m5.large");+ } finally {+ releaseRestart.countDown();+ executor.shutdownNow();+ }+ }+@Testvoid nodeGroupLifecycleDescribeListDelete() {createTestCluster("my-eks-cluster");diff --git a/src/test/resources/application.yml b/src/test/resources/application.ymlindex cf67c9d5d6..9ba51826f8 100644--- a/src/test/resources/application.yml+++ b/src/test/resources/application.yml@@ -332,6 +332,8 @@ floci:eks:enabled: truemock: true+ max-memory-mib: 0+ max-vcpus: 0ecr-registry-mirror: trueimds: falseirsa-signing-key: true