Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(eks): bound cluster containers and derive kubelet reservations (#4590)

Simone Primarosa047d0d848cdf
diff --git a/docs/services/eks.md b/docs/services/eks.md
index 2392dc9820..e1a7332748 100644
--- a/docs/services/eks.md
+++ b/docs/services/eks.md
@@ -369,6 +369,57 @@ A restored cluster reports `CREATING` until its API server answers again, then r
back (for example Docker is unavailable), the cluster is marked `FAILED` instead of appearing
`ACTIVE` while unreachable.
+#### Cluster node capacity
+
+Floci runs one k3s container per cluster. Before a node group exists, its node uses the
+`m5.large` entry in the EC2 instance type catalog. Creating the first node group selects the
+group's first `instanceTypes` entry. If that changes the type, Floci recreates the container,
+keeps its named k3s data volume, and reuses its published API port so existing kubeconfigs stay
+valid. Later node groups share this node and cannot change its type. An unknown type falls back to
+`m5.large` with a warning. The synthesized EC2 instance reports the selected type.
+First-group selection is serialized per cluster. If the replacement fails, Floci recreates the
+previous node, marks that node group `CREATE_FAILED`, and keeps its data volume. If recovery also
+fails, the cluster becomes `FAILED` instead of waiting indefinitely for readiness.
+Groups remain `CREATING` while launch-template user data runs. Later groups wait for the first
+group's result before accepting its node type as the shared capacity.
+
+The container receives a hard CPU quota equal to the selected type's vCPUs and a memory limit
+equal to its catalog memory plus 10% headroom (at least 128 MiB). Both are capped by Docker's host
+capacity. Memory is capped at 80% of the Docker host's memory so other processes can continue to
+run. `floci.services.eks.max-memory-mib` and `floci.services.eks.max-vcpus` provide optional lower
+ceilings; zero means no extra ceiling. Their environment variables are `FLOCI_SERVICES_EKS_MAX_MEMORY_MIB`
+and `FLOCI_SERVICES_EKS_MAX_VCPUS`.
+
+The same calculation sets kubelet's `kube-reserved`, `system-reserved`, and `eviction-hard` flags.
+The EKS AMI reserves `255 + 11 * maxPods` MiB and a tiered CPU fraction for kubelet and the
+container runtime. Floci derives `maxPods` from the catalog's primary network card and IPv4
+addresses per interface. Its `system-reserved` memory holds the container headroom, and its CPU
+reservation excludes the Docker host CPUs outside the container quota. Its absolute
+`memory.available` threshold includes the difference between Docker host memory and the container
+limit, plus a 100 MiB eviction buffer. This is necessary because kubelet in the nested container
+can report host memory while the kernel enforces the smaller cgroup limit.
+The EKS node filesystem eviction defaults remain at 10% available space and 5% free inodes.
+The pod-density calculation uses EKS managed node group caps of 110 pods up to 30 vCPUs and
+250 pods above 30 vCPUs; it does not change k3s `maxPods`.
+
+If Docker cannot report host capacity, or the host is too small to leave 256 MiB for pods after
+reservations, Floci warns and launches the cluster without the derived limits or kubelet arguments.
+This preserves startup on small hosts. An explicit memory or CPU ceiling remains a hard bound even
+when Docker host information is unavailable; in that case Floci cannot calculate kubelet arguments.
+When an explicit memory ceiling is smaller than the default reservation budget, Floci reduces the
+memory reservations to leave some room for pods and warns that they no longer match the EKS AMI
+amount.
+
+The reservation formulas follow the [EKS AMI nodeadm source](https://github.com/awslabs/amazon-eks-ami/blob/main/nodeadm/internal/kubelet/config.go) and [AWS's node memory calculation](https://docs.aws.amazon.com/batch/latest/userguide/memory-cpu-batch-eks.html).
+On restore, a surviving container whose capacity label differs from the current calculation is
+recreated against the retained data volume, reusing its published API port. Floci keeps the old
+container until the replacement starts and restarts it if replacement fails. This also upgrades
+containers started before resource limits were available. If Docker cannot remove the stopped
+backup after a successful replacement, the new node keeps running and cluster deletion retries
+the backup cleanup after stopping the live node. If cleanup still fails, deletion reports the
+error and can be retried before the data volume is removed. Backup container names end in
+`.capacity-backup`, which cannot match an EKS cluster name.
+
#### Cluster node provider ID and topology labels
In real mode, cluster nodes carry a Kubernetes `spec.providerID` matching the AWS format:
@@ -394,6 +445,8 @@ The derived availability zone and instance ID match the synthetic EC2 node insta
|---|---|---|
| `FLOCI_SERVICES_EKS_ENABLED` | `true` | Enable the EKS service |
| `FLOCI_SERVICES_EKS_MOCK` | `false` | Metadata-only mode (no Docker) |
+| `FLOCI_SERVICES_EKS_MAX_MEMORY_MIB` | `0` | Optional k3s container memory ceiling in MiB |
+| `FLOCI_SERVICES_EKS_MAX_VCPUS` | `0` | Optional k3s container vCPU ceiling |
| `FLOCI_SERVICES_EKS_DEFAULT_IMAGE` | `rancher/k3s:latest` | k3s Docker image fallback |
| `FLOCI_SERVICES_EKS_IMAGE_TEMPLATE` | *(unset)* | Format string for custom k3s images (e.g. `myregistry.io/k3s:v%s`), taking cluster version |
| `FLOCI_SERVICES_EKS_API_SERVER_BASE_PORT` | `6500` | First port in the k3s API server range |
@@ -521,7 +574,7 @@ services:
```
## Instance Metadata Service (IMDS)
-The cluster container is registered as a synthesized EC2 instance node (type `m5.large`, image `ami-eks-k3s`) regardless of whether IMDS is enabled, making it visible to the EC2 service (`DescribeInstances`, `DescribeInstanceStatus`, volume attachments, tagging).
+The cluster container is registered as a synthesized EC2 instance node (image `ami-eks-k3s`) regardless of whether IMDS is enabled, making it visible to the EC2 service (`DescribeInstances`, `DescribeInstanceStatus`, volume attachments, tagging). Its instance type is the same type used for the container limits.
When IMDS is enabled (`FLOCI_SERVICES_EKS_IMDS=true`), each k3s cluster container exposes the AWS Instance Metadata Service on the link-local address `169.254.169.254:80`. Inside the container, Floci adds `169.254.169.254/32` to the loopback interface (`lo`) and runs a lightweight `socat` TCP relay forwarding metadata requests to Floci's IMDS server.
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 2a33312296..c08024429f 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -3086,6 +3086,14 @@ public interface EmulatorConfig {
@WithDefault("false")
boolean mock();
+ /** Optional ceiling for the single k3s node container, in MiB. Zero uses its instance type. */
+ @WithDefault("0")
+ int maxMemoryMib();
+
+ /** Optional vCPU ceiling for the single k3s node container. Zero uses its instance type. */
+ @WithDefault("0")
+ int maxVcpus();
+
@WithDefault("k3s")
String provider();
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java
index fb2378b21e..a77043e60f 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerExec.java
@@ -101,6 +101,7 @@ public final class ContainerExec {
dockerClient.execStartCmd(execId).exec(callback);
try {
+ // docker-java closes the attach stream on completion, timeout, or interruption.
if (!callback.awaitCompletion(timeoutSeconds, TimeUnit.SECONDS)) {
return new Result(-1, "", "Timed out after " + timeoutSeconds + "s", true);
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java
index 6b01dd352c..7fe1d86473 100644
--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java
@@ -4,6 +4,7 @@ import com.github.dockerjava.api.DockerClient;
import com.github.dockerjava.api.command.ExecCreateCmdResponse;
import com.github.dockerjava.api.command.InspectContainerResponse;
import com.github.dockerjava.api.model.ContainerNetwork;
+import com.github.dockerjava.api.model.Info;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.config.FlociCertificateAuthority;
import io.github.hectorvent.floci.core.common.AwsRegions;
@@ -22,6 +23,8 @@ import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
import io.github.hectorvent.floci.core.common.docker.PortAllocator;
import io.github.hectorvent.floci.core.common.docker.UserDataPipeline;
import io.github.hectorvent.floci.services.ec2.ClusterNodeInstanceProvider;
+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog;
+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogInstanceType;
import io.github.hectorvent.floci.services.ec2.Ec2MetadataProxy;
import io.github.hectorvent.floci.services.ec2.Ec2MetadataServer;
import io.github.hectorvent.floci.services.ec2.Ec2Service;
@@ -64,6 +67,7 @@ import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Base64;
+import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
@@ -85,6 +89,8 @@ public class EksClusterManager
private static final Logger LOG = Logger.getLogger(EksClusterManager.class);
private static final int K3S_API_SERVER_PORT = 6443;
+ static final String DEFAULT_NODE_INSTANCE_TYPE = "m5.large";
+ private static final String NODE_CAPACITY_LABEL = "io.floci.eks.node-capacity";
private static final String WEBHOOK_CONFIG_DIR = "/etc";
private static final String WEBHOOK_CONFIG_FILE = "token-webhook.yaml";
@@ -140,6 +146,7 @@ public class EksClusterManager
private final EksOidcService oidcService;
private final FlociCertificateAuthority certificateAuthority;
private final ContainerLogStreamer logStreamer;
+ private final Ec2InstanceTypeCatalog instanceTypeCatalog = new Ec2InstanceTypeCatalog();
private final Map<String, ClusterNodeRecord> clusterNodeInstances = new ConcurrentHashMap<>();
private final Map<String, Closeable> clusterLogHandles = new ConcurrentHashMap<>();
private final List<Consumer<Instance>> nodeRegistrationListeners = new CopyOnWriteArrayList<>();
@@ -373,6 +380,10 @@ public class EksClusterManager
* {@link #isReady(Cluster)} returns true and {@link #finalizeCluster(Cluster)} is called.
*/
public void startCluster(Cluster cluster) {
+ startCluster(cluster, null);
+ }
+
+ private void startCluster(Cluster cluster, Integer retainedPort) {
String image = resolveClusterImage(cluster);
if (cluster.getDockerName() == null) {
cluster.setDockerName(accountQualifiedName(cluster));
@@ -383,7 +394,7 @@ public class EksClusterManager
cluster.getName(), image);
// Allocate host port for the k3s API server
- int hostPort = portAllocator.allocate(
+ int hostPort = retainedPort != null ? retainedPort : portAllocator.allocate(
config.services().eks().apiServerBasePort(),
config.services().eks().apiServerMaxPort());
@@ -413,6 +424,11 @@ public class EksClusterManager
List<String> serverArgs = buildServerArgs(config.services().eks().disableCni(), serviceCidr, clusterCidr);
+ EksNodeCapacity.Limits nodeLimits = resolveNodeCapacity(cluster);
+ if (nodeLimits != null) {
+ nodeLimits.addKubeletArgs(serverArgs);
+ }
+
try {
String providerId = deriveClusterNodeProviderId(cluster);
serverArgs.add("--kubelet-arg=provider-id=" + providerId);
@@ -436,6 +452,9 @@ public class EksClusterManager
// The account label comes from the cluster record when set (restore runs with no request
// context); regionResolver is the fallback for the create path.
String labelAccountId = resolveClusterAccountId(cluster);
+ Map<String, String> labels = new LinkedHashMap<>(ContainerStorageHelper.resourceIdentityLabels(
+ "eks", cluster.getName(), labelAccountId, clusterRegion(cluster)));
+ labels.put(NODE_CAPACITY_LABEL, capacityLabel(cluster, nodeLimits));
ContainerBuilder.Builder specBuilder = containerBuilder.newContainer(image)
.withName(containerName)
.withEnv("K3S_KUBECONFIG_MODE", "644")
@@ -444,8 +463,16 @@ public class EksClusterManager
.withDockerNetwork(config.services().eks().dockerNetwork())
.withPrivileged(true)
.withLogRotation()
- .withLabels(ContainerStorageHelper.resourceIdentityLabels(
- "eks", cluster.getName(), labelAccountId, clusterRegion(cluster)));
+ .withLabels(labels);
+
+ if (nodeLimits != null) {
+ if (nodeLimits.memoryBytes() > 0) {
+ specBuilder.withMemoryBytes(nodeLimits.memoryBytes());
+ }
+ if (nodeLimits.vcpus() > 0) {
+ specBuilder.withCpuUnits(nodeLimits.vcpus() * 1024);
+ }
+ }
if (config.services().eks().embeddedDns()) {
specBuilder.withEmbeddedDns();
@@ -572,25 +599,38 @@ public class EksClusterManager
return;
}
+ String desiredCapacity = capacityLabel(cluster, resolveNodeCapacity(cluster));
+ Map<String, String> existingLabels = existing.get().getLabels();
+ boolean capacityChanged = existingLabels == null
+ || !desiredCapacity.equals(existingLabels.get(NODE_CAPACITY_LABEL));
+ if (!adoptSurvivingCluster(cluster, existing.get().getId())) {
+ startCluster(cluster);
+ return;
+ }
+ if (capacityChanged) {
+ replaceClusterContainer(cluster, null);
+ }
+ }
+
+ private boolean adoptSurvivingCluster(Cluster cluster, String containerId) {
if (config.services().eks().irsaSigningKey() && oidcService != null) {
- reinjectSigningKeys(existing.get().getId(), cluster);
+ reinjectSigningKeys(containerId, cluster);
}
ContainerInfo info;
try {
- info = lifecycleManager.adopt(existing.get().getId(), List.of(K3S_API_SERVER_PORT));
+ info = lifecycleManager.adopt(containerId, List.of(K3S_API_SERVER_PORT));
} catch (Exception e) {
LOG.warnv("Could not adopt surviving k3s container {0} for EKS cluster {1} ({2}); recreating it",
- containerName, cluster.getName(), e.getMessage());
- startCluster(cluster);
- return;
+ cluster.getDockerName(), cluster.getName(), e.getMessage());
+ return false;
}
var publishedPort = info.publishedHostPort(K3S_API_SERVER_PORT);
if (publishedPort.isEmpty()) {
- LOG.warnv("Surviving k3s container {0} publishes no API server port; recreating it", containerName);
- startCluster(cluster);
- return;
+ LOG.warnv("Surviving k3s container {0} publishes no API server port; recreating it",
+ cluster.getDockerName());
+ return false;
}
int hostPort = publishedPort.getAsInt();
@@ -598,7 +638,7 @@ public class EksClusterManager
portAllocator.markReserved(hostPort);
cluster.setContainerId(info.containerId());
cluster.setHostPort(hostPort);
- applyEndpoints(cluster, containerName, hostPort, info);
+ applyEndpoints(cluster, cluster.getDockerName(), hostPort, info);
registerClusterNodeInstance(cluster, info.containerId());
configureLinkLocalMetadataEndpoint(cluster, info.containerId());
configurePodIdentityRelay(cluster, info.containerId());
@@ -608,6 +648,71 @@ public class EksClusterManager
LOG.infov("Adopted surviving k3s container {0} for EKS cluster {1} on port {2} (internal: {3})",
info.containerId(), cluster.getName(), String.valueOf(hostPort), cluster.getInternalEndpoint());
+ return true;
+ }
+
+ /** Keep the old container stopped but recoverable until its replacement has started. */
+ private boolean replaceClusterContainer(Cluster cluster, String previousType) {
+ String oldId = cluster.getContainerId();
+ int oldPort = cluster.getHostPort();
+ String containerName = cluster.getDockerName();
+ String backupName = capacityBackupName(cluster);
+ DockerClient docker = lifecycleManager.getDockerClient();
+ try {
+ lifecycleManager.removeIfExistsStrict(backupName);
+ } catch (RuntimeException cleanup) {
+ LOG.warnv("Could not clear prior EKS capacity backup for cluster {0}: {1}; keeping current node",
+ cluster.getName(), cleanup.getMessage());
+ restorePreviousNodeType(cluster, previousType);
+ return false;
+ }
+ boolean renamed = false;
+ try {
+ docker.renameContainerCmd(oldId).withName(backupName).exec();
+ renamed = true;
+ docker.stopContainerCmd(oldId).exec();
+ unregisterMetadataEndpoint(cluster);
+ closeQuietly(clusterLogHandles.remove(clusterResourceName(cluster)));
+ cluster.setContainerId(null);
+ startCluster(cluster, oldPort);
+ LOG.infov("Replaced EKS cluster {0} to apply current node capacity limits", cluster.getName());
+ } catch (RuntimeException replacement) {
+ LOG.warnv("Could not replace EKS cluster {0} for node capacity: {1}; restoring surviving node",
+ cluster.getName(), replacement.getMessage());
+ restorePreviousNodeType(cluster, previousType);
+ if (!renamed) {
+ return false;
+ }
+ try {
+ lifecycleManager.removeIfExistsStrict(containerName);
+ docker.renameContainerCmd(oldId).withName(containerName).exec();
+ if (!adoptSurvivingCluster(cluster, oldId)) {
+ throw new IllegalStateException("Could not adopt previous EKS container " + oldId);
+ }
+ } catch (RuntimeException rollback) {
+ throw new IllegalStateException("Could not restore EKS cluster " + cluster.getName()
+ + " after node capacity replacement failed", rollback);
+ }
+ return false;
+ }
+ try {
+ lifecycleManager.removeIfExistsStrict(backupName);
+ } catch (RuntimeException cleanup) {
+ LOG.warnv("EKS cluster {0} replacement is running, but its stopped capacity backup"
+ + " could not be removed: {1}; cluster deletion will retry",
+ cluster.getName(), cleanup.getMessage());
+ }
+ return true;
+ }
+
+ private static void restorePreviousNodeType(Cluster cluster, String previousType) {
+ if (previousType != null) {
+ cluster.setNodeInstanceType(previousType);
+ }
+ }
+
+ private static String capacityBackupName(Cluster cluster) {
+ return cluster.getDockerName() + ".capacity-backup";
}
/**
@@ -705,10 +810,88 @@ public class EksClusterManager
return;
}
lifecycleManager.stopAndRemove(cluster.getContainerId(), logStream);
+ if (cluster.getDockerName() != null) {
+ // A failed backup cleanup must not leave the live node running. Keep the cluster
+ // record so explicit deletion can be retried once Docker accepts the removal.
+ lifecycleManager.removeIfExistsStrict(capacityBackupName(cluster));
+ }
ContainerStorageHelper.removeNamedVolume(config, lifecycleManager, clusterResourceName(cluster));
LOG.infov("Stopped k3s container for cluster {0}", cluster.getName());
}
+ String selectNodeInstanceType(String requestedType) {
+ if (instanceTypeCatalog.find(requestedType).isPresent()) {
+ return requestedType;
+ }
+ LOG.warnv("EKS node instance type {0} is absent from the EC2 catalog; using {1}",
+ requestedType, DEFAULT_NODE_INSTANCE_TYPE);
+ return DEFAULT_NODE_INSTANCE_TYPE;
+ }
+
+ private EksNodeCapacity.Limits resolveNodeCapacity(Cluster cluster) {
+ try {
+ Info host = lifecycleManager.getDockerClient().infoCmd().exec();
+ CatalogInstanceType type = instanceTypeCatalog.find(nodeInstanceType(cluster)).orElseThrow();
+ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(type, host.getMemTotal(), host.getNCPU(),
+ config.services().eks().maxMemoryMib(), config.services().eks().maxVcpus());
+ if (limits == null) {
+ EksNodeCapacity.Limits configured = explicitCeilingWithoutHostInfo(cluster);
+ if (configured != null) {
+ return configured;
+ }
+ LOG.warnv("EKS cluster {0} cannot fit node type {1} and its kubelet reservations"
+ + " within the Docker host; starting without resource limits",
+ cluster.getName(), type.instanceType);
+ } else if (limits.reducedReservations()) {
+ LOG.warnv("EKS cluster {0} memory ceiling reduces kubelet memory reservations below"
+ + " the EKS AMI defaults", cluster.getName());
+ }
+ return limits;
+ } catch (Exception e) {
+ LOG.warnv("EKS cluster {0} Docker host capacity unavailable: {1}",
+ cluster.getName(), e.getMessage());
+ return explicitCeilingWithoutHostInfo(cluster);
+ }
+ }
+
+ private EksNodeCapacity.Limits explicitCeilingWithoutHostInfo(Cluster cluster) {
+ int maxMemoryMib = config.services().eks().maxMemoryMib();
+ int maxVcpus = config.services().eks().maxVcpus();
+ if (maxMemoryMib <= 0 && maxVcpus <= 0) {
+ return null;
+ }
+ int vcpus = instanceTypeCatalog.find(nodeInstanceType(cluster))
+ .map(type -> maxVcpus > 0 ? Math.min(type.vcpu, maxVcpus) : 0)
+ .orElse(maxVcpus);
+ LOG.warnv("Applying explicit EKS cluster {0} ceiling without kubelet reservations"
+ + " because Docker host capacity is unavailable", cluster.getName());
+ return EksNodeCapacity.explicitCeilingWithoutHostInfo(maxMemoryMib, vcpus);
+ }
+
+ private static String capacityLabel(Cluster cluster, EksNodeCapacity.Limits limits) {
+ if (limits == null) {
+ return nodeInstanceType(cluster) + ":unbounded";
+ }
+ return nodeInstanceType(cluster) + ":" + limits.memoryBytes() + ":" + limits.vcpus()
+ + ":" + limits.systemMemoryMib() + ":" + limits.kubeMemoryMib()
+ + ":" + limits.evictionMemoryMib() + ":" + limits.systemCpuMilli()
+ + ":" + limits.kubeCpuMilli() + ":" + limits.reducedReservations()
+ + ":" + limits.kubeletArgsEnabled();
+ }
+
+ static String nodeInstanceType(Cluster cluster) {
+ return cluster != null && cluster.getNodeInstanceType() != null
+ ? cluster.getNodeInstanceType() : DEFAULT_NODE_INSTANCE_TYPE;
+ }
+
+ /** Recreate the shared node with its named data volume intact, restoring its old type on failure. */
+ boolean restartForNodeCapacity(Cluster cluster, String previousType) {
+ if (cluster.getContainerId() == null) {
+ return true;
+ }
+ return replaceClusterContainer(cluster, previousType);
+ }
+
/**
* Releases Floci's hold on the given cluster (its metadata endpoint and log stream) and leaves
* the k3s container running, so a later Floci start can re-latch it through
@@ -2071,7 +2254,7 @@ public class EksClusterManager
inst.setInstanceId(instanceId);
inst.setImageId("ami-eks-k3s");
- inst.setInstanceType("m5.large");
+ inst.setInstanceType(nodeInstanceType(cluster));
inst.setPlacement(new Placement(az));
inst.setRegion(safeRegion);
inst.setState(InstanceState.running());
diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksController.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksController.java
index 03903d494c..9e41493265 100644
--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksController.java
+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksController.java
@@ -92,11 +92,12 @@ public class EksController {
* causes Jackson to omit the property due to @JsonInclude(NON_DEFAULT).
*/
private Cluster toClusterResponse(Cluster cluster) {
- if (cluster == null || !cluster.isExplicitVersion()) {
+ if (cluster == null || (!cluster.isExplicitVersion() && cluster.getNodeInstanceType() == null)) {
return cluster;
}
Cluster response = cluster.copy();
response.setExplicitVersion(false);
+ response.setNodeInstanceType(null);
return response;
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksNodeCapacity.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksNodeCapacity.java
new file mode 100644
index 0000000000..2a94d628f8
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksNodeCapacity.java
@@ -0,0 +1,102 @@
+package io.github.hectorvent.floci.services.eks;
+
+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogInstanceType;
+
+import java.util.List;
+
+/** The one calculation shared by the Docker limit and the kubelet's node budget. */
+final class EksNodeCapacity {
+ private static final long MIB = 1024L * 1024L;
+
+ private EksNodeCapacity() {}
+
+ record Limits(long memoryBytes, int vcpus, long systemMemoryMib, long kubeMemoryMib,
+ long evictionMemoryMib, int systemCpuMilli, int kubeCpuMilli,
+ boolean reducedReservations, boolean kubeletArgsEnabled) {
+ void addKubeletArgs(List<String> args) {
+ if (!kubeletArgsEnabled) {
+ return;
+ }
+ args.add("--kubelet-arg=system-reserved=cpu=" + systemCpuMilli + "m,memory="
+ + systemMemoryMib + "Mi");
+ args.add("--kubelet-arg=kube-reserved=cpu=" + kubeCpuMilli + "m,memory="
+ + kubeMemoryMib + "Mi");
+ args.add("--kubelet-arg=eviction-hard=memory.available<" + evictionMemoryMib
+ + "Mi,nodefs.available<10%,nodefs.inodesFree<5%");
+ }
+ }
+
+ static Limits explicitCeilingWithoutHostInfo(int maxMemoryMib, int vcpus) {
+ return new Limits((long) Math.max(0, maxMemoryMib) * MIB, vcpus,
+ 0, 0, 0, 0, 0, false, false);
+ }
+
+ /**
+ * Docker's host totals are also what kubelet sees in this nested setup. Its eviction threshold
+ * includes the host-to-cgroup gap; otherwise the cgroup can OOM before kubelet sees pressure.
+ * The EKS AMI kube reservation is 11 MiB per pod plus 255 MiB, and its CPU tiers are 6%,
+ * 1%, 0.5%, then 0.25% of successive cores. The additional system memory is Floci's
+ * container headroom, not an EKS AMI default.
+ */
+ static Limits calculate(CatalogInstanceType type, long hostMemoryBytes, int hostCpus,
+ int maxMemoryMib, int maxVcpus) {
+ if (hostMemoryBytes <= 0 || hostCpus <= 0 || maxMemoryMib < 0 || maxVcpus < 0) {
+ return null;
+ }
+ long hostMib = hostMemoryBytes / MIB;
+ long requestedMib = type.memoryMib + Math.max(128L, type.memoryMib / 10L);
+ long hostLimitMib = Math.min(requestedMib, hostMib * 4 / 5);
+ long memoryMib = hostLimitMib;
+ if (maxMemoryMib > 0) {
+ memoryMib = Math.min(memoryMib, maxMemoryMib);
+ }
+ int vcpus = Math.min(type.vcpu, hostCpus);
+ if (maxVcpus > 0) {
+ vcpus = Math.min(vcpus, maxVcpus);
+ }
+ if (vcpus <= 0) {
+ return null;
+ }
+
+ int interfaces = type.networkCards.stream()
+ .filter(card -> card.networkCardIndex != null && card.networkCardIndex == 0
+ && card.maximumNetworkInterfaces != null)
+ .mapToInt(card -> card.maximumNetworkInterfaces)
+ .findFirst().orElse(0);
+ int addresses = type.ipv4AddressesPerInterface == null ? 0 : type.ipv4AddressesPerInterface;
+ int podCap = type.vcpu > 30 ? 250 : 110;
+ int maxPods = interfaces > 0 && addresses > 0
+ ? Math.min(podCap, interfaces * (addresses - 1) + 2) : podCap;
+ long kubeMemoryMib = 11L * maxPods + 255;
+ long systemMemoryMib = Math.max(128, memoryMib / 10);
+ long evictionBufferMib = Math.min(100, memoryMib / 4);
+ // Only an undersized host with no explicit ceiling falls back to the old unbounded mode.
+ // An operator's ceiling must remain a hard bound, even if it leaves few resources for pods.
+ if (maxMemoryMib == 0 && memoryMib < kubeMemoryMib + systemMemoryMib + evictionBufferMib + 256) {
+ return null;
+ }
+ if (memoryMib <= 0) {
+ return null;
+ }
+ boolean reducedReservations = maxMemoryMib > 0
+ && memoryMib < kubeMemoryMib + systemMemoryMib + evictionBufferMib + 256;
+ if (reducedReservations) {
+ long reservableMib = Math.max(0, memoryMib - evictionBufferMib - 1);
+ kubeMemoryMib = Math.min(kubeMemoryMib, reservableMib / 2);
+ systemMemoryMib = Math.min(systemMemoryMib, reservableMib - kubeMemoryMib);
+ }
+ int kubeCpuMilli = cpuReservationMilli(vcpus);
+ int systemCpuMilli = (hostCpus - vcpus) * 1000;
+ long evictionMemoryMib = hostMib - memoryMib + evictionBufferMib;
+ return new Limits(memoryMib * MIB, vcpus, systemMemoryMib, kubeMemoryMib,
+ evictionMemoryMib, systemCpuMilli, kubeCpuMilli, reducedReservations, true);
+ }
+
+ private static int cpuReservationMilli(int vcpus) {
+ int first = Math.min(vcpus, 1) * 60;
+ int second = Math.min(Math.max(vcpus - 1, 0), 1) * 10;
+ int third = Math.min(Math.max(vcpus - 2, 0), 2) * 5;
+ int rest = Math.max(vcpus - 4, 0) * 25 / 10;
+ return first + second + third + rest;
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java
index 9fd6315082..412e55733c 100644
--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java
@@ -48,6 +48,7 @@ import org.jboss.logging.Logger;
import java.time.Instant;
import java.util.ArrayList;
+import java.util.Comparator;
import java.util.HashMap;
import java.util.HashSet;
import java.util.HexFormat;
@@ -102,6 +103,8 @@ public class EksService implements TagHandler, ResourceProvider {
private final EksAddonService addons;
private final ScheduledExecutorService poller = Executors.newSingleThreadScheduledExecutor();
private final Map<String, UserDataClaim> appliedClusterUserData = new ConcurrentHashMap<>();
+ private final Map<String, Object> nodeGroupCapacityLocks = new ConcurrentHashMap<>();
+ private final Map<String, CompletableFuture<Boolean>> pendingFirstNodeGroups = new ConcurrentHashMap<>();
@Inject
public EksService(StorageFactory storageFactory, EmulatorConfig config,
@@ -204,6 +207,10 @@ public class EksService implements TagHandler, ResourceProvider {
LOG.infov("Restoring k3s container for persisted EKS cluster {0}", cluster.getName());
cluster.setStatus(ClusterStatus.CREATING);
cluster.setPodCidr(EksClusterManager.DEFAULT_POD_CIDR);
+ if (cluster.getNodeInstanceType() == null) {
+ firstNodeGroup(cluster.getName(), entry.accountId()).ifPresent(group ->
+ cluster.setNodeInstanceType(selectedNodeInstanceType(group)));
+ }
clusterManager.restoreCluster(cluster);
} catch (Exception e) {
if (!clusterManager.isDockerReachable()) {
@@ -233,6 +240,22 @@ public class EksService implements TagHandler, ResourceProvider {
.toList();
}
+ private Optional<Nodegroup> firstNodeGroup(String clusterName, String accountId) {
+ String prefix = clusterName + "/";
+ List<Nodegroup> groups;
+ if (nodeGroupStorage instanceof AccountAwareStorageBackend<Nodegroup> aware) {
+ groups = aware.scanAllAccountEntries(key -> key.startsWith(prefix)).stream()
+ .filter(entry -> accountId.equals(entry.accountId()))
+ .map(AccountAwareStorageBackend.AccountEntry::value).toList();
+ } else {
+ groups = nodeGroupStorage.scan(key -> key.startsWith(prefix));
+ }
+ return groups.stream().filter(group -> group.getStatus() == NodegroupStatus.ACTIVE)
+ .min(Comparator.comparing(Nodegroup::getCreatedAt,
+ Comparator.nullsLast(Comparator.naturalOrder()))
+ .thenComparing(Nodegroup::getNodegroupName));
+ }
+
/**
* Gives clusters persisted before IRSA support an OIDC issuer and signing key. Without this,
* a cluster restored from {@code eks-clusters.json} would report no
@@ -440,10 +463,15 @@ public class EksService implements TagHandler, ResourceProvider {
if (!config.services().eks().mock()) {
boolean keepRunning = config.services().eks().keepRunningOnShutdown();
for (Cluster cluster : allClusters()) {
- if (keepRunning) {
- clusterManager.detachCluster(cluster);
- } else {
- clusterManager.stopCluster(cluster);
+ try {
+ if (keepRunning) {
+ clusterManager.detachCluster(cluster);
+ } else {
+ clusterManager.stopCluster(cluster);
+ }
+ } catch (RuntimeException e) {
+ LOG.warnv("Could not clean up EKS cluster {0} during shutdown: {1}",
+ cluster.getName(), e.getMessage());
}
}
}
@@ -713,15 +741,130 @@ public class EksService implements TagHandler, ResourceProvider {
nodeGroup.setLabels(request.getLabels() != null ? new HashMap<>(request.getLabels()) : null);
nodeGroup.setTags(request.getTags() != null ? new HashMap<>(request.getTags()) : new HashMap<>());
- if (launchTemplateData != null && launchTemplateData.getUserData() != null
- && !launchTemplateData.getUserData().isBlank()) {
- applyNodeGroupUserData(cluster, nodegroupName, launchTemplateData.getUserData(), nodeGroup);
- }
+ String capacityKey = accountId + "/" + clusterName;
+ Object capacityLock = nodeGroupCapacityLocks.computeIfAbsent(capacityKey,
+ ignored -> new Object());
+ boolean hasUserData = launchTemplateData != null && launchTemplateData.getUserData() != null
+ && !launchTemplateData.getUserData().isBlank();
+ Cluster currentCluster;
+ CompletableFuture<Boolean> pendingFirst;
+ boolean ownsPendingFirst = false;
+ synchronized (capacityLock) {
+ if (nodeGroupStorage.get(storageKey).isPresent()) {
+ throw new AwsException("ResourceInUseException",
+ "Nodegroup already exists: " + nodegroupName, 409);
+ }
+ currentCluster = describeCluster(clusterName);
+ pendingFirst = pendingFirstNodeGroups.get(capacityKey);
+ boolean firstGroup = firstNodeGroup(clusterName, accountId).isEmpty() && pendingFirst == null;
+ if (firstGroup && !config.services().eks().mock() && clusterManager != null) {
+ applyFirstNodeGroupCapacity(clusterName, nodegroupName, currentCluster, nodeGroup);
+ } else if (!firstGroup && pendingFirst == null
+ && !config.services().eks().mock() && clusterManager != null) {
+ if (currentCluster.getNodeInstanceType() == null) {
+ firstNodeGroup(clusterName, accountId).ifPresent(group ->
+ currentCluster.setNodeInstanceType(selectedNodeInstanceType(group)));
+ storage.put(clusterName, currentCluster);
+ }
+ if (!nodeGroup.getInstanceTypes().isEmpty()
+ && !nodeGroup.getInstanceTypes().contains(currentCluster.getNodeInstanceType())) {
+ LOG.warnv("EKS cluster {0} has one shared node; nodegroup {1} cannot change its capacity from {2}",
+ clusterName, nodegroupName, currentCluster.getNodeInstanceType());
+ }
+ }
- nodeGroupStorage.put(storageKey, nodeGroup);
+ if (nodeGroup.getStatus() == NodegroupStatus.ACTIVE && (hasUserData || pendingFirst != null)) {
+ nodeGroup.setStatus(NodegroupStatus.CREATING);
+ if (firstGroup && hasUserData) {
+ pendingFirst = new CompletableFuture<>();
+ ownsPendingFirst = true;
+ }
+ }
+ nodeGroupStorage.put(storageKey, nodeGroup);
+ if (ownsPendingFirst) {
+ pendingFirstNodeGroups.put(capacityKey, pendingFirst);
+ }
+ }
+ if (nodeGroup.getStatus() == NodegroupStatus.CREATING && hasUserData) {
+ try {
+ applyNodeGroupUserData(currentCluster, nodegroupName, launchTemplateData.getUserData(), nodeGroup);
+ } catch (Throwable e) {
+ synchronized (capacityLock) {
+ if (nodeGroupStorage.get(storageKey)
+ .filter(saved -> arn.equals(saved.getNodegroupArn())).isPresent()) {
+ nodeGroupStorage.delete(storageKey);
+ }
+ if (ownsPendingFirst) {
+ pendingFirstNodeGroups.remove(capacityKey, pendingFirst);
+ pendingFirst.complete(false);
+ }
+ }
+ throw e;
+ }
+ }
+ if (pendingFirst != null && !ownsPendingFirst) {
+ pendingFirst.join();
+ }
+ synchronized (capacityLock) {
+ boolean stored = false;
+ try {
+ if (nodeGroupStorage.get(storageKey)
+ .filter(saved -> arn.equals(saved.getNodegroupArn())).isPresent()) {
+ if (nodeGroup.getStatus() == NodegroupStatus.CREATING) {
+ if (!ownsPendingFirst && firstNodeGroup(clusterName, accountId).isEmpty()
+ && !config.services().eks().mock() && clusterManager != null) {
+ Cluster refreshedCluster = describeCluster(clusterName);
+ applyFirstNodeGroupCapacity(clusterName, nodegroupName, refreshedCluster, nodeGroup);
+ }
+ if (nodeGroup.getStatus() == NodegroupStatus.CREATING) {
+ nodeGroup.setStatus(NodegroupStatus.ACTIVE);
+ }
+ }
+ nodeGroupStorage.put(storageKey, nodeGroup);
+ stored = true;
+ }
+ } finally {
+ if (ownsPendingFirst) {
+ pendingFirstNodeGroups.remove(capacityKey, pendingFirst);
+ pendingFirst.complete(stored && nodeGroup.getStatus() == NodegroupStatus.ACTIVE);
+ }
+ }
+ }
return nodeGroup;
}
+ private void applyFirstNodeGroupCapacity(String clusterName, String nodegroupName,
+ Cluster currentCluster, Nodegroup nodeGroup) {
+ String selectedType = selectedNodeInstanceType(nodeGroup);
+ String previousType = EksClusterManager.nodeInstanceType(currentCluster);
+ currentCluster.setNodeInstanceType(selectedType);
+ if (currentCluster.getContainerId() != null && !selectedType.equals(previousType)) {
+ currentCluster.setStatus(ClusterStatus.CREATING);
+ try {
+ if (!clusterManager.restartForNodeCapacity(currentCluster, previousType)) {
+ nodeGroup.setStatus(NodegroupStatus.CREATE_FAILED);
+ nodeGroup.setHealth(failedNodeGroupHealth(nodegroupName,
+ "Could not apply node capacity; the previous node was restored"));
+ }
+ } catch (RuntimeException e) {
+ currentCluster.setNodeInstanceType(previousType);
+ currentCluster.setStatus(ClusterStatus.FAILED);
+ nodeGroup.setStatus(NodegroupStatus.CREATE_FAILED);
+ nodeGroup.setHealth(failedNodeGroupHealth(nodegroupName, e.getMessage()));
+ LOG.errorv("EKS cluster {0} could not recover after node capacity change: {1}",
+ clusterName, e.getMessage());
+ }
+ }
+ storage.put(clusterName, currentCluster);
+ }
+
+ private String selectedNodeInstanceType(Nodegroup group) {
+ String requestedType = group.getInstanceTypes() == null || group.getInstanceTypes().isEmpty()
+ ? null : group.getInstanceTypes().getFirst();
+ String selectedType = clusterManager.selectNodeInstanceType(requestedType);
+ return selectedType != null ? selectedType : EksClusterManager.DEFAULT_NODE_INSTANCE_TYPE;
+ }
+
private LaunchTemplateData validateLaunchTemplate(String region, Object launchTemplateObj) {
if (!(launchTemplateObj instanceof Map<?, ?> map)) {
return null;
@@ -775,11 +918,15 @@ public class EksService implements TagHandler, ResourceProvider {
}
public Nodegroup deleteNodeGroup(String clusterName, String nodegroupName) {
- Nodegroup nodeGroup = describeNodeGroup(clusterName, nodegroupName);
- nodeGroup.setStatus(NodegroupStatus.DELETING);
- nodeGroup.setModifiedAt(Instant.now());
- nodeGroupStorage.delete(nodeGroupKey(clusterName, nodegroupName));
- return nodeGroup;
+ Object capacityLock = nodeGroupCapacityLocks.computeIfAbsent(
+ regionResolver.getAccountId() + "/" + clusterName, ignored -> new Object());
+ synchronized (capacityLock) {
+ Nodegroup nodeGroup = describeNodeGroup(clusterName, nodegroupName);
+ nodeGroup.setStatus(NodegroupStatus.DELETING);
+ nodeGroup.setModifiedAt(Instant.now());
+ nodeGroupStorage.delete(nodeGroupKey(clusterName, nodegroupName));
+ return nodeGroup;
+ }
}
public FargateProfile createFargateProfile(String clusterName, CreateFargateProfileRequest request) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/model/Cluster.java b/src/main/java/io/github/hectorvent/floci/services/eks/model/Cluster.java
index 2843690687..38243c63fe 100644
--- a/src/main/java/io/github/hectorvent/floci/services/eks/model/Cluster.java
+++ b/src/main/java/io/github/hectorvent/floci/services/eks/model/Cluster.java
@@ -96,6 +96,11 @@ public class Cluster {
@JsonIgnore
private String dockerName;
+ /** Instance type selected for the single shared node container. Internal persisted metadata. */
+ @JsonProperty("nodeInstanceType")
+ @JsonInclude(JsonInclude.Include.NON_NULL)
+ private String nodeInstanceType;
+
private AccessConfig accessConfig;
public AccessConfig getAccessConfig() { return accessConfig; }
@@ -145,6 +150,9 @@ public class Cluster {
public String getContainerId() { return containerId; }
public void setContainerId(String containerId) { this.containerId = containerId; }
+ public String getNodeInstanceType() { return nodeInstanceType; }
+ public void setNodeInstanceType(String nodeInstanceType) { this.nodeInstanceType = nodeInstanceType; }
+
public String getAccountId() { return accountId; }
public void setAccountId(String accountId) { this.accountId = accountId; }
@@ -193,6 +201,7 @@ public class Cluster {
c.podCidr = this.podCidr;
c.explicitVersion = this.explicitVersion;
c.dockerName = this.dockerName;
+ c.nodeInstanceType = this.nodeInstanceType;
c.accessConfig = this.accessConfig;
return c;
}
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 60a78fb133..1ccadd0247 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -647,6 +647,8 @@ floci:
eks:
enabled: true
mock: false
+ max-memory-mib: 0 # optional k3s container memory ceiling; 0 follows the node type
+ max-vcpus: 0 # optional k3s container CPU ceiling; 0 follows the node type
provider: k3s
default-image: "rancher/k3s:latest"
# image-template: "myregistry.io/k3s:v%s" # optional custom image format template taking cluster version
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecTest.java
index 88c316dbb5..bfac749398 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerExecTest.java
@@ -12,6 +12,8 @@ import com.github.dockerjava.api.model.StreamType;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import java.io.Closeable;
+import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.function.Consumer;
@@ -106,8 +108,10 @@ class ContainerExecTest {
}
@Test
- void aCommandThatOutlivesItsTimeoutIsReportedWithWhatItPrintedSoFar() {
+ void aCommandThatOutlivesItsTimeoutIsReportedWithWhatItPrintedSoFar() throws IOException {
+ Closeable attach = mock(Closeable.class);
startWith(callback -> {
+ callback.onStart(attach);
callback.onNext(frame(StreamType.STDOUT, "partial"));
callback.onNext(frame(StreamType.STDERR, "warning"));
});
@@ -118,6 +122,7 @@ class ContainerExecTest {
assertEquals(-1L, result.exitCode());
assertEquals("partial", result.stdout());
assertEquals("warning\nTimed out after 0s", result.stderr());
+ verify(attach).close();
verify(dockerClient, never()).inspectExecCmd(EXEC_ID);
}
@@ -172,13 +177,14 @@ class ContainerExecTest {
}
@Test
- void anInterruptIsRaisedAndTheInterruptFlagKept() {
- startWith(callback -> {
- });
+ void anInterruptIsRaisedAndTheInterruptFlagKept() throws IOException {
+ Closeable attach = mock(Closeable.class);
+ startWith(callback -> callback.onStart(attach));
Thread.currentThread().interrupt();
try {
assertThrows(IllegalStateException.class, () -> ContainerExec.run(dockerClient, CONTAINER_ID, CMD, 5));
assertTrue(Thread.currentThread().isInterrupted());
+ verify(attach).close();
} finally {
Thread.interrupted();
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java
index fa3a9c2d69..5ad7b7cf3f 100644
--- a/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java
@@ -39,7 +39,9 @@ import com.github.dockerjava.api.command.InspectExecResponse;
import com.github.dockerjava.api.command.InspectVolumeCmd;
import com.github.dockerjava.api.model.Container;
import com.github.dockerjava.api.model.Frame;
+import com.github.dockerjava.api.model.Info;
import com.github.dockerjava.api.model.NetworkSettings;
+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog;
import io.github.hectorvent.floci.services.ec2.Ec2MetadataServer;
import io.github.hectorvent.floci.services.ec2.model.Instance;
import org.junit.jupiter.api.BeforeEach;
@@ -319,7 +321,8 @@ class EksClusterManagerTest {
"io.floci.service", "eks",
"io.floci.resource-id", "my-cluster",
"io.floci.account", "000000000000",
- "io.floci.region", "us-east-1"));
+ "io.floci.region", "us-east-1",
+ "io.floci.eks.node-capacity", "m5.large:unbounded"));
}
@ParameterizedTest
@@ -362,6 +365,7 @@ class EksClusterManagerTest {
private EmulatorConfig.StorageConfig storage;
private ContainerLifecycleManager lifecycleManager;
private PortAllocator portAllocator;
+ private DockerClient dockerClient;
private EksClusterManager manager;
@BeforeEach
@@ -388,6 +392,8 @@ class EksClusterManagerTest {
when(containerBuilder.newContainer(anyString())).thenReturn(builder);
when(builder.build()).thenReturn(Mockito.mock(ContainerSpec.class));
portAllocator = Mockito.mock(PortAllocator.class);
+ dockerClient = Mockito.mock(DockerClient.class, Mockito.RETURNS_DEEP_STUBS);
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
manager = new EksClusterManager(containerBuilder, lifecycleManager,
Mockito.mock(ContainerDetector.class), portAllocator,
@@ -397,12 +403,14 @@ class EksClusterManagerTest {
// Container's getters are final, so survivors are built from JSON instead of mocked.
private Container survivingContainer(String id) {
- return containerFromJson("{\"Id\":\"" + id + "\"}");
+ return containerFromJson("{\"Id\":\"" + id + "\","
+ + "\"Labels\":{\"io.floci.eks.node-capacity\":\"m5.large:unbounded\"}}");
}
private Container survivingContainerOwnedBy(String id, String accountId) {
return containerFromJson("{\"Id\":\"" + id + "\","
- + "\"Labels\":{\"io.floci.account\":\"" + accountId + "\"}}");
+ + "\"Labels\":{\"io.floci.account\":\"" + accountId + "\","
+ + "\"io.floci.eks.node-capacity\":\"m5.large:unbounded\"}}");
}
private Container containerFromJson(String json) {
@@ -446,6 +454,136 @@ class EksClusterManagerTest {
verify(lifecycleManager, never()).create(any());
}
+ @Test
+ void recreatesAnOldSurvivorWithoutCapacityLimits() {
+ when(lifecycleManager.findByName("floci-eks-demo"))
+ .thenReturn(Optional.of(containerFromJson("{\"Id\":\"cid-old\"}")));
+ when(lifecycleManager.adopt("cid-old", List.of(6443)))
+ .thenReturn(new ContainerInfo("cid-old", Map.of(), Map.of(6443, 6440)));
+ stubFreshStart("cid-new", 6440);
+
+ Cluster cluster = cluster();
+ manager.restoreCluster(cluster);
+
+ assertEquals("cid-new", cluster.getContainerId());
+ verify(lifecycleManager).create(any());
+ verify(portAllocator, never()).allocate(6440, 6499);
+ verify(dockerClient).stopContainerCmd("cid-old");
+ }
+
+ @Test
+ void restoresOldSurvivorWhenCapacityReplacementFails() {
+ when(lifecycleManager.findByName("floci-eks-demo"))
+ .thenReturn(Optional.of(containerFromJson("{\"Id\":\"cid-old\"}")));
+ when(lifecycleManager.adopt("cid-old", List.of(6443)))
+ .thenReturn(new ContainerInfo("cid-old", Map.of(), Map.of(6443, 6440)));
+ when(lifecycleManager.create(any())).thenReturn("cid-new");
+ when(lifecycleManager.startCreated(any(), any()))
+ .thenThrow(new RuntimeException("replacement failed"));
+
+ Cluster cluster = cluster();
+ manager.restoreCluster(cluster);
+
+ assertEquals("cid-old", cluster.getContainerId());
+ assertEquals(6440, cluster.getHostPort());
+ verify(lifecycleManager).removeIfExistsStrict("floci-eks-demo");
+ verify(dockerClient).stopContainerCmd("cid-old");
+ verify(portAllocator, never()).allocate(6440, 6499);
+ }
+
+ @Test
+ void firstNodeGroupReplacementKeepsTheClusterEndpoint() {
+ when(lifecycleManager.create(any())).thenReturn("cid-new");
+ when(lifecycleManager.startCreated(any(), any()))
+ .thenReturn(new ContainerInfo("cid-new", Map.of()));
+ Cluster cluster = cluster();
+ cluster.setDockerName("floci-eks-demo");
+ cluster.setContainerId("cid-old");
+ cluster.setHostPort(6441);
+ cluster.setNodeInstanceType("t3.medium");
+
+ assertTrue(manager.restartForNodeCapacity(cluster, "m5.large"));
+
+ assertEquals("cid-new", cluster.getContainerId());
+ assertEquals(6441, cluster.getHostPort());
+ assertEquals("https://localhost:6441", cluster.getEndpoint());
+ verify(portAllocator, never()).allocate(6440, 6499);
+ verify(portAllocator, never()).release(6441);
+ verify(dockerClient).stopContainerCmd("cid-old");
+ verify(lifecycleManager, Mockito.times(2))
+ .removeIfExistsStrict("floci-eks-demo.capacity-backup");
+ }
+
+ @Test
+ void capacityBackupDoesNotCollideWithAnotherClusterName() {
+ when(lifecycleManager.create(any())).thenReturn("cid-new");
+ when(lifecycleManager.startCreated(any(), any()))
+ .thenReturn(new ContainerInfo("cid-new", Map.of()));
+ Cluster cluster = cluster();
+ cluster.setName("foo");
+ cluster.setDockerName("floci-eks-foo");
+ cluster.setContainerId("cid-foo");
+ cluster.setHostPort(6441);
+ cluster.setNodeInstanceType("t3.medium");
+ Cluster other = cluster();
+ other.setName("foo-capacity-backup");
+ other.setDockerName("floci-eks-foo-capacity-backup");
+ other.setContainerId("cid-other");
+
+ assertTrue(manager.restartForNodeCapacity(cluster, "m5.large"));
+ manager.stopCluster(cluster);
+
+ assertTrue(other.getName().matches(EksService.CLUSTER_NAME_REGEX));
+ assertFalse("foo.capacity-backup".matches(EksService.CLUSTER_NAME_REGEX));
+ assertEquals("floci-eks-foo-capacity-backup", manager.clusterResourceName(other));
+ verify(lifecycleManager, Mockito.times(3))
+ .removeIfExistsStrict("floci-eks-foo.capacity-backup");
+ verify(lifecycleManager, never()).removeIfExistsStrict(other.getDockerName());
+ verify(lifecycleManager, never()).stopAndRemove("cid-other", null);
+ }
+
+ @Test
+ void failedFirstNodeGroupReplacementRestoresPreviousContainerAndType() {
+ when(lifecycleManager.create(any())).thenReturn("cid-new");
+ when(lifecycleManager.startCreated(any(), any()))
+ .thenThrow(new IllegalStateException("replacement failed"));
+ when(lifecycleManager.adopt("cid-old", List.of(6443)))
+ .thenReturn(new ContainerInfo("cid-old", Map.of(), Map.of(6443, 6441)));
+ Cluster cluster = cluster();
+ cluster.setDockerName("floci-eks-demo");
+ cluster.setContainerId("cid-old");
+ cluster.setHostPort(6441);
+ cluster.setNodeInstanceType("t3.medium");
+
+ assertFalse(manager.restartForNodeCapacity(cluster, "m5.large"));
+
+ assertEquals("m5.large", cluster.getNodeInstanceType());
+ assertEquals("cid-old", cluster.getContainerId());
+ assertEquals(6441, cluster.getHostPort());
+ assertEquals("https://localhost:6441", cluster.getEndpoint());
+ verify(portAllocator, never()).allocate(6440, 6499);
+ verify(portAllocator, never()).release(6441);
+ verify(lifecycleManager, never()).removeVolume(anyString());
+ }
+
+ @Test
+ void backupCleanupFailureKeepsTheNewNodeRunning() {
+ when(lifecycleManager.findByName("floci-eks-demo"))
+ .thenReturn(Optional.of(containerFromJson("{\"Id\":\"cid-old\"}")));
+ when(lifecycleManager.adopt("cid-old", List.of(6443)))
+ .thenReturn(new ContainerInfo("cid-old", Map.of(), Map.of(6443, 6440)));
+ stubFreshStart("cid-new", 6440);
+ Mockito.doNothing().doThrow(new IllegalStateException("Docker cleanup failed"))
+ .when(lifecycleManager).removeIfExistsStrict("floci-eks-demo.capacity-backup");
+
+ Cluster cluster = cluster();
+ manager.restoreCluster(cluster);
+
+ assertEquals("cid-new", cluster.getContainerId());
+ verify(lifecycleManager, Mockito.times(1)).adopt("cid-old", List.of(6443));
+ verify(lifecycleManager, never()).removeIfExistsStrict("floci-eks-demo");
+ }
+
@Test
void recreatesTheContainerWhenNoneSurvives() {
when(lifecycleManager.findByName("floci-eks-demo")).thenReturn(Optional.empty());
@@ -504,6 +642,20 @@ class EksClusterManagerTest {
verify(lifecycleManager, never()).removeVolume(anyString());
}
+ @Test
+ void backupCleanupFailureDoesNotLeaveTheLiveNodeRunning() {
+ Cluster cluster = cluster();
+ cluster.setDockerName("floci-eks-demo");
+ cluster.setContainerId("cid-new");
+ Mockito.doThrow(new IllegalStateException("Docker cleanup failed"))
+ .when(lifecycleManager).removeIfExistsStrict("floci-eks-demo.capacity-backup");
+
+ assertThrows(IllegalStateException.class, () -> manager.stopCluster(cluster));
+
+ verify(lifecycleManager).stopAndRemove("cid-new", null);
+ verify(lifecycleManager, never()).removeVolume(anyString());
+ }
+
@Test
void stopClusterRemovesTheDataVolumeInMemoryStorageMode() {
when(storage.mode()).thenReturn("memory");
@@ -1454,6 +1606,8 @@ class EksClusterManagerTest {
Container container = Mockito.mock(Container.class);
when(container.getId()).thenReturn("surviving-container-id");
+ when(container.getLabels()).thenReturn(Map.of(
+ "io.floci.eks.node-capacity", "m5.large:unbounded"));
when(lifecycleManager.findByName("floci-eks-my-cluster")).thenReturn(Optional.of(container));
when(lifecycleManager.adopt(anyString(), any())).thenReturn(
new ContainerInfo("surviving-container-id", Map.of(6443, new ContainerLifecycleManager.EndpointInfo("localhost", 6500)), Map.of(6443, 6500)));
@@ -1745,6 +1899,74 @@ class EksClusterManagerTest {
config, regionResolver, null, Mockito.mock(EksOidcService.class));
}
+ @Test
+ void nodeGroupTypeControlsContainerAndSynthesizedInstance() {
+ DockerClient dockerClient = Mockito.mock(DockerClient.class, Mockito.RETURNS_DEEP_STUBS);
+ Info host = Mockito.mock(Info.class);
+ when(host.getMemTotal()).thenReturn(16L * 1024 * 1024 * 1024);
+ when(host.getNCPU()).thenReturn(8);
+ when(dockerClient.infoCmd().exec()).thenReturn(host);
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
+
+ Cluster cluster = new Cluster();
+ cluster.setName("capacity-cluster");
+ cluster.setNodeInstanceType(manager.selectNodeInstanceType("t3.medium"));
+ manager.startCluster(cluster);
+
+ Ec2InstanceTypeCatalog catalog = new Ec2InstanceTypeCatalog();
+ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(
+ catalog.find("t3.medium").orElseThrow(), host.getMemTotal(), host.getNCPU(), 0, 0);
+ verify(builder).withMemoryBytes(limits.memoryBytes());
+ verify(builder).withCpuUnits(limits.vcpus() * 1024);
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> expectedArgs = new ArrayList<>();
+ limits.addKubeletArgs(expectedArgs);
+ assertTrue(cmdCaptor.getValue().containsAll(expectedArgs));
+ assertEquals("t3.medium", manager.synthesizeClusterNodeInstance(
+ cluster, "172.17.0.2", "us-east-1", "000000000000").getInstanceType());
+ }
+
+ @Test
+ void hostTooSmallLeavesClusterUnboundedAndStarting() {
+ DockerClient dockerClient = Mockito.mock(DockerClient.class, Mockito.RETURNS_DEEP_STUBS);
+ Info host = Mockito.mock(Info.class);
+ when(host.getMemTotal()).thenReturn(1024L * 1024 * 1024);
+ when(host.getNCPU()).thenReturn(2);
+ when(dockerClient.infoCmd().exec()).thenReturn(host);
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
+
+ Cluster cluster = new Cluster();
+ cluster.setName("small-host");
+ assertDoesNotThrow(() -> manager.startCluster(cluster));
+ verify(builder, never()).withMemoryBytes(Mockito.anyLong());
+ verify(builder, never()).withCpuUnits(Mockito.anyInt());
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ assertFalse(cmdCaptor.getValue().stream().anyMatch(arg -> arg.contains("reserved=")
+ || arg.contains("eviction-hard=")));
+ }
+
+ @Test
+ void explicitCeilingSurvivesMissingDockerHostInformation() {
+ when(eks.maxMemoryMib()).thenReturn(512);
+ when(eks.maxVcpus()).thenReturn(1);
+ Cluster cluster = new Cluster();
+ cluster.setName("capped-cluster");
+
+ manager.startCluster(cluster);
+
+ verify(builder).withMemoryBytes(512L * 1024 * 1024);
+ verify(builder).withCpuUnits(1024);
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ assertFalse(cmdCaptor.getValue().stream().anyMatch(arg -> arg.contains("reserved=")
+ || arg.contains("eviction-hard=")));
+ }
+
@Test
void resolveClusterImageMapsSupportedVersions() {
Cluster cluster = new Cluster();
@@ -2208,6 +2430,8 @@ class EksClusterManagerTest {
void restoreClusterAttachesLogsForAdoptedContainer() {
Container container = Mockito.mock(Container.class);
when(container.getId()).thenReturn("adopted-container-id-12345678901234567890");
+ when(container.getLabels()).thenReturn(Map.of(
+ "io.floci.eks.node-capacity", "m5.large:unbounded"));
when(lifecycleManager.findByName("floci-eks-prod-cluster")).thenReturn(Optional.of(container));
when(lifecycleManager.adopt(anyString(), any())).thenReturn(
new ContainerInfo("adopted-container-id-12345678901234567890",
diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksLaunchTemplateUserDataTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksLaunchTemplateUserDataTest.java
index 8684b1ccc8..e39ca3c573 100644
--- a/src/test/java/io/github/hectorvent/floci/services/eks/EksLaunchTemplateUserDataTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksLaunchTemplateUserDataTest.java
@@ -342,6 +342,52 @@ class EksLaunchTemplateUserDataTest {
}
}
+ @Test
+ void laterGroupWaitsForFirstBootstrapBeforeBecomingActive() throws InterruptedException {
+ String script = "#!/bin/bash\nexit 1\n";
+ String encoded = Base64.getEncoder().encodeToString(script.getBytes(StandardCharsets.UTF_8));
+ String ltId = createLaunchTemplate("lt-pending-first", encoded);
+ CountDownLatch enteredExecution = new CountDownLatch(1);
+ CountDownLatch releaseExecution = new CountDownLatch(1);
+ when(clusterManager.executeUserData(any(Cluster.class), eq("ng-1"), eq(encoded)))
+ .thenAnswer(invocation -> {
+ enteredExecution.countDown();
+ assertTrue(releaseExecution.await(5, TimeUnit.SECONDS));
+ return UserDataPipeline.ExecutionResult.failed(1L, 1, 1, "bootstrap failed");
+ });
+
+ CreateNodeGroupRequest firstRequest = nodeGroupRequest("ng-1");
+ firstRequest.setLaunchTemplate(Map.of("id", ltId));
+ AtomicReference<Nodegroup> firstResult = new AtomicReference<>();
+ Thread first = new Thread(() -> firstResult.set(eksService.createNodeGroup(CLUSTER_NAME, firstRequest)));
+ first.start();
+ try {
+ assertTrue(enteredExecution.await(5, TimeUnit.SECONDS));
+ assertEquals(NodegroupStatus.CREATING,
+ eksService.describeNodeGroup(CLUSTER_NAME, "ng-1").getStatus());
+
+ AtomicReference<Nodegroup> secondResult = new AtomicReference<>();
+ Thread second = new Thread(() -> secondResult.set(
+ eksService.createNodeGroup(CLUSTER_NAME, nodeGroupRequest("ng-2"))));
+ second.start();
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
+ while (!eksService.listNodeGroups(CLUSTER_NAME).contains("ng-2")
+ && System.nanoTime() < deadline) {
+ Thread.sleep(10);
+ }
+ assertEquals(NodegroupStatus.CREATING,
+ eksService.describeNodeGroup(CLUSTER_NAME, "ng-2").getStatus());
+
+ releaseExecution.countDown();
+ first.join(5000);
+ second.join(5000);
+ assertEquals(NodegroupStatus.CREATE_FAILED, firstResult.get().getStatus());
+ assertEquals(NodegroupStatus.ACTIVE, secondResult.get().getStatus());
+ } finally {
+ releaseExecution.countDown();
+ }
+ }
+
@Test
void unstubbedClusterManagerDoesNotThrowNpe() {
String script = "#!/bin/bash\necho 'unstubbed'\n";
diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksNodeCapacityTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksNodeCapacityTest.java
new file mode 100644
index 0000000000..508a763236
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksNodeCapacityTest.java
@@ -0,0 +1,84 @@
+package io.github.hectorvent.floci.services.eks;
+
+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog;
+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogInstanceType;
+import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogNetworkCard;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class EksNodeCapacityTest {
+ private static final long MIB = 1024L * 1024L;
+ private final Ec2InstanceTypeCatalog catalog = new Ec2InstanceTypeCatalog();
+
+ @Test
+ void reservationsFollowTheAppliedLimitAndTheInstancePodDensity() {
+ CatalogInstanceType type = catalog.find("m5.large").orElseThrow();
+ long hostMib = 16384;
+ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(type, hostMib * MIB, 8, 0, 0);
+ long limitMib = limits.memoryBytes() / MIB;
+ int maxPods = type.networkCards.getFirst().maximumNetworkInterfaces
+ * (type.ipv4AddressesPerInterface - 1) + 2;
+
+ assertEquals(type.memoryMib + Math.max(128, type.memoryMib / 10), limitMib);
+ assertEquals(type.vcpu, limits.vcpus());
+ assertEquals(255 + 11L * maxPods, limits.kubeMemoryMib());
+ assertEquals(Math.max(128, limitMib / 10), limits.systemMemoryMib());
+ assertEquals(hostMib - limitMib + 100, limits.evictionMemoryMib());
+ assertEquals((8 - limits.vcpus()) * 1000, limits.systemCpuMilli());
+ assertEquals(70, limits.kubeCpuMilli());
+
+ List<String> args = new ArrayList<>();
+ limits.addKubeletArgs(args);
+ assertTrue(args.contains("--kubelet-arg=system-reserved=cpu=" + limits.systemCpuMilli()
+ + "m,memory=" + limits.systemMemoryMib() + "Mi"));
+ assertTrue(args.contains("--kubelet-arg=kube-reserved=cpu=" + limits.kubeCpuMilli()
+ + "m,memory=" + limits.kubeMemoryMib() + "Mi"));
+ assertTrue(args.contains("--kubelet-arg=eviction-hard=memory.available<"
+ + limits.evictionMemoryMib() + "Mi,nodefs.available<10%,nodefs.inodesFree<5%"));
+ }
+
+ @Test
+ void hostAndUserCeilingsBoundTheContainerAndReservations() {
+ CatalogInstanceType type = catalog.find("m5.large").orElseThrow();
+ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(type, 8192 * MIB, 4, 4096, 1);
+ assertEquals(4096 * MIB, limits.memoryBytes());
+ assertEquals(1, limits.vcpus());
+ assertEquals(60, limits.kubeCpuMilli());
+ assertEquals(3000, limits.systemCpuMilli());
+ assertEquals(8192 - 4096 + 100, limits.evictionMemoryMib());
+ }
+
+ @Test
+ void largeManagedNodeUsesThe250PodCapForMemoryReservation() {
+ CatalogInstanceType type = new CatalogInstanceType();
+ type.vcpu = 32;
+ type.memoryMib = 131072;
+ type.ipv4AddressesPerInterface = 50;
+ CatalogNetworkCard card = new CatalogNetworkCard();
+ card.networkCardIndex = 0;
+ card.maximumNetworkInterfaces = 15;
+ type.networkCards = List.of(card);
+
+ EksNodeCapacity.Limits limits = EksNodeCapacity.calculate(type, 262144 * MIB, 64, 0, 0);
+ assertEquals(255 + 11 * 250, limits.kubeMemoryMib());
+ }
+
+ @Test
+ void tooSmallHostOrMissingHostInformationFallsBackToUnbounded() {
+ CatalogInstanceType type = catalog.find("m5.large").orElseThrow();
+ assertNull(EksNodeCapacity.calculate(type, 1024 * MIB, 2, 0, 0));
+ assertNull(EksNodeCapacity.calculate(type, 0, 0, 0, 0));
+ EksNodeCapacity.Limits capped = EksNodeCapacity.calculate(type, 8192 * MIB, 2, 512, 0);
+ assertNotNull(capped);
+ assertEquals(512 * MIB, capped.memoryBytes());
+ assertTrue(capped.kubeMemoryMib() + capped.systemMemoryMib()
+ + 100 < capped.memoryBytes() / MIB);
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java
index 090370b885..2ecca6c458 100644
--- a/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java
@@ -52,14 +52,21 @@ import java.nio.file.Path;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@@ -901,6 +908,69 @@ class EksServiceTest {
assertEquals("1.30-eks-1", nodeGroup.getReleaseVersion());
}
+ @Test
+ void firstNodeGroupSelectsAndRestartsTheSharedNodeOnce() {
+ EksClusterManager clusterManager = mock(EksClusterManager.class);
+ when(clusterManager.tryStartCluster(any())).thenReturn(true);
+ when(clusterManager.selectNodeInstanceType("t3.medium")).thenReturn("t3.medium");
+ when(clusterManager.restartForNodeCapacity(any(), anyString())).thenReturn(true);
+ EksService service = newService(clusterManager, false);
+ CreateClusterRequest request = new CreateClusterRequest();
+ request.setName("capacity-cluster");
+ request.setRoleArn("arn:aws:iam::000000000000:role/eks-role");
+ Cluster cluster = service.createCluster(request);
+ cluster.setContainerId("existing-container");
+
+ service.createNodeGroup("capacity-cluster", nodeGroupRequest("first"));
+ assertEquals("t3.medium", cluster.getNodeInstanceType());
+ verify(clusterManager).restartForNodeCapacity(cluster, "m5.large");
+
+ service.createNodeGroup("capacity-cluster", nodeGroupRequest("second"));
+ verify(clusterManager).restartForNodeCapacity(cluster, "m5.large");
+ }
+
+ @Test
+ void concurrentFirstNodeGroupsChooseOnlyOneCapacity() throws Exception {
+ EksClusterManager clusterManager = mock(EksClusterManager.class);
+ when(clusterManager.tryStartCluster(any())).thenReturn(true);
+ when(clusterManager.selectNodeInstanceType("t3.medium")).thenReturn("t3.medium");
+ CountDownLatch restartEntered = new CountDownLatch(1);
+ CountDownLatch releaseRestart = new CountDownLatch(1);
+ when(clusterManager.restartForNodeCapacity(any(), anyString())).thenAnswer(invocation -> {
+ restartEntered.countDown();
+ if (!releaseRestart.await(10, TimeUnit.SECONDS)) {
+ throw new IllegalStateException("Timed out waiting to finish the first restart");
+ }
+ return true;
+ });
+ EksService service = newService(clusterManager, false);
+ CreateClusterRequest request = new CreateClusterRequest();
+ request.setName("concurrent-capacity");
+ request.setRoleArn("arn:aws:iam::000000000000:role/eks-role");
+ Cluster cluster = service.createCluster(request);
+ cluster.setContainerId("existing-container");
+
+ ExecutorService executor = Executors.newFixedThreadPool(2);
+ try {
+ Future<Nodegroup> first = executor.submit(() -> service.createNodeGroup(
+ "concurrent-capacity", nodeGroupRequest("first")));
+ assertTrue(restartEntered.await(10, TimeUnit.SECONDS));
+ CreateNodeGroupRequest secondRequest = nodeGroupRequest("second");
+ secondRequest.setInstanceTypes(List.of("m5.large"));
+ Future<Nodegroup> second = executor.submit(() -> service.createNodeGroup(
+ "concurrent-capacity", secondRequest));
+ releaseRestart.countDown();
+ first.get(10, TimeUnit.SECONDS);
+ second.get(10, TimeUnit.SECONDS);
+
+ assertEquals("t3.medium", cluster.getNodeInstanceType());
+ verify(clusterManager, times(1)).restartForNodeCapacity(cluster, "m5.large");
+ } finally {
+ releaseRestart.countDown();
+ executor.shutdownNow();
+ }
+ }
+
@Test
void nodeGroupLifecycleDescribeListDelete() {
createTestCluster("my-eks-cluster");
diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml
index cf67c9d5d6..9ba51826f8 100644
--- a/src/test/resources/application.yml
+++ b/src/test/resources/application.yml
@@ -332,6 +332,8 @@ floci:
eks:
enabled: true
mock: true
+ max-memory-mib: 0
+ max-vcpus: 0
ecr-registry-mirror: true
imds: false
irsa-signing-key: true