Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(eks): apply nodegroup labels, taints and capacity type to the node (#5063)

Simone Primarosa383c32443d77
diff --git a/docs/services/eks.md b/docs/services/eks.md
index 94d35ee784..e96a558e76 100644
--- a/docs/services/eks.md
+++ b/docs/services/eks.md
@@ -209,13 +209,14 @@ The supported component prefixes correspond to k3s server argument flags:
Floci manages a specific set of flags required for container networking, IAM authentication, audit logging, and topology emulation. Passing a conflicting value or modifier (such as `+=` or `-=`) for any of these flags causes `CreateCluster` to reject the request with `InvalidParameterException` (HTTP 400).
-Floci manages the following 16 flags:
+Floci manages the following 17 flags:
- Kubelet:
- `provider-id`: manages EC2 worker node identity.
- - `node-labels`: manages topology zones and instance types.
+ - `node-labels`: manages topology zones, instance types, and nodegroup labels. Nodegroup labels merge directly into this managed argument rather than through the passthrough.
- `system-reserved`: manages memory and CPU reservations.
- `kube-reserved`: manages kubelet resource allocations.
- `eviction-hard`: manages memory pressure thresholds.
+ - `register-with-taints`: manages worker node taints configured on managed nodegroups.
- API server:
- `authentication-token-webhook-config-file`: manages worker IAM authentication.
- `authentication-token-webhook-version`: token webhook protocol version.
@@ -346,12 +347,37 @@ When a node group specifies a `launchTemplate`, Floci resolves the launch templa
- **Failure handling**: If any user data script exits with a non-zero status or times out (30-minute limit), the node group status is set to `CREATE_FAILED` and `health.issues` is populated with code `NodeCreationFailure`, the failure message, and the node group name in `resourceIds`.
- **Differences from EC2**: Scripts run inside the existing shared k3s container rather than an isolated VM instance. Other launch template settings (AMI, instance type, block device mappings, network interfaces) remain metadata-only and do not alter container provisioning.
+### Nodegroup labels, taints and capacity type
+
+When managed nodegroups are defined for a cluster, Floci applies their labels and taints to the Kubernetes worker node at startup via `--kubelet-arg` flags.
+
+#### Applied labels
+Floci configures `--kubelet-arg=node-labels=...` with the following entries:
+- Standard topology labels: `topology.kubernetes.io/region` and `topology.kubernetes.io/zone`.
+- Instance type: `node.kubernetes.io/instance-type` (sourced from the nodegroup's first configured instance type, or defaulting to `m5.large`).
+- EKS system labels:
+ - `eks.amazonaws.com/nodegroup=<nodegroupName>`
+ - `eks.amazonaws.com/capacityType=<ON_DEMAND|SPOT|CAPACITY_BLOCK>` (defaults to `ON_DEMAND` when omitted)
+ - `eks.amazonaws.com/nodegroup-image=<imageId>` (sourced from `releaseVersion`, launch template AMI, or defaulting to `ami-eks-k3s`)
+- User-defined labels: all key-value pairs specified in the nodegroup's `labels` map, excluding reserved system keys (`topology.kubernetes.io/*`, `eks.amazonaws.com/*`, and `node.kubernetes.io/instance-type`).
+
+#### Applied taints
+Floci configures `--kubelet-arg=register-with-taints=...` using the taints specified on the nodegroup. Each taint is formatted as `key=value:effect` (or `key:effect` if value is omitted or empty). AWS taint effects are mapped to standard Kubernetes casing:
+- `NO_SCHEDULE` maps to `NoSchedule`
+- `NO_EXECUTE` maps to `NoExecute`
+- `PREFER_NO_SCHEDULE` maps to `PreferNoSchedule`
+
+#### Single shared node mapping
+Floci runs one k3s container per cluster. The worker node reflects the metadata of the first active nodegroup (ordered by `createdAt` ascending, then `nodegroupName`). If additional nodegroups are created on the same cluster, Floci logs a warning that only the first nodegroup provides node metadata to the shared node.
+
+#### Timing and post-start nodegroups
+Kubelet flags are fixed when the cluster container starts. For nodegroups created while the cluster is already running, metadata is recorded in the EKS store and Floci logs an informational message that labels and taints are not applied to the running node. Surviving cluster containers are adopted as-is without rebuilding kubelet flags on restore.
+
### Metadata only inputs
The remaining structured inputs are recorded as metadata:
- `remoteAccess` does not open SSH access, and the referenced key pair and security groups are not wired up.
-- `taints` are not applied to Kubernetes nodes, so pods are not repelled from them.
- `nodeRepairConfig` starts no repair loop, and `warmPoolConfig` pre-initializes no instances.
The value of storing them is state fidelity: infrastructure tools read `DescribeNodegroup` back and compare it against their declared configuration. Dropping `launchTemplate` in particular made the OpenTofu AWS provider see a node group with no launch template where one was declared, which is a `ForceNew` difference and caused a destroy and recreate on every plan.
diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterArgs.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterArgs.java
index 1c5fd71fc3..36fd399776 100644
--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterArgs.java
+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterArgs.java
@@ -61,7 +61,8 @@ public final class EksClusterArgs {
"nodelabels",
"systemreserved",
"kubereserved",
- "evictionhard"
+ "evictionhard",
+ "registerwithtaints"
);
/**
diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java
index 4175ee3ea2..248757111f 100644
--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksClusterManager.java
@@ -1,5 +1,6 @@
package io.github.hectorvent.floci.services.eks;
+import com.fasterxml.jackson.core.type.TypeReference;
import com.github.dockerjava.api.DockerClient;
import com.github.dockerjava.api.command.ExecCreateCmdResponse;
import com.github.dockerjava.api.command.InspectContainerResponse;
@@ -28,6 +29,9 @@ import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
import io.github.hectorvent.floci.core.common.docker.PortAllocator;
import io.github.hectorvent.floci.core.common.docker.RetryingTarCopier;
import io.github.hectorvent.floci.core.common.docker.UserDataPipeline;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
import io.github.hectorvent.floci.services.ec2.ClusterNodeInstanceProvider;
import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog;
import io.github.hectorvent.floci.services.ec2.Ec2InstanceTypeCatalog.CatalogInstanceType;
@@ -37,6 +41,7 @@ import io.github.hectorvent.floci.services.ec2.Ec2Service;
import io.github.hectorvent.floci.services.ec2.VpcRouteTableListener;
import io.github.hectorvent.floci.services.ec2.model.Instance;
import io.github.hectorvent.floci.services.ec2.model.InstanceState;
+import io.github.hectorvent.floci.services.ec2.model.LaunchTemplateData;
import io.github.hectorvent.floci.services.ec2.model.Placement;
import io.github.hectorvent.floci.services.ec2.model.RouteTable;
import io.github.hectorvent.floci.services.ec2.model.Tag;
@@ -46,6 +51,8 @@ import io.github.hectorvent.floci.services.eks.model.Cluster;
import io.github.hectorvent.floci.services.eks.model.ClusterIdentity;
import io.github.hectorvent.floci.services.eks.model.ClusterOidcKey;
import io.github.hectorvent.floci.services.eks.model.LogSetup;
+import io.github.hectorvent.floci.services.eks.model.Nodegroup;
+import io.github.hectorvent.floci.services.eks.model.NodegroupStatus;
import io.github.hectorvent.floci.services.eks.model.OidcIdentity;
import io.github.hectorvent.floci.services.eks.model.ResourcesVpcConfig;
import jakarta.enterprise.context.ApplicationScoped;
@@ -73,6 +80,7 @@ import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Base64;
+import java.util.Comparator;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
@@ -83,6 +91,7 @@ import java.util.Set;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.function.BiFunction;
import java.util.function.Consumer;
import java.util.stream.Collectors;
@@ -167,6 +176,7 @@ public class EksClusterManager
@Inject
jakarta.enterprise.inject.Instance<Ec2Service> ec2ServiceInstance;
private Ec2Service ec2Service;
+ private final StorageBackend<String, Nodegroup> nodeGroupStorage;
private final Map<String, Set<String>> programmedClusterRoutes = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, Object> clusterRouteLocks = new ConcurrentHashMap<>();
@@ -365,7 +375,6 @@ public class EksClusterManager
null, logStreamer);
}
- @Inject
public EksClusterManager(ContainerBuilder containerBuilder,
ContainerLifecycleManager lifecycleManager,
ContainerDetector containerDetector,
@@ -378,6 +387,24 @@ public class EksClusterManager
EksOidcService oidcService,
FlociCertificateAuthority certificateAuthority,
ContainerLogStreamer logStreamer) {
+ this(containerBuilder, lifecycleManager, containerDetector, portAllocator,
+ dockerHostResolver, ecrRegistryManager, config, regionResolver, metadataServer,
+ oidcService, certificateAuthority, logStreamer, (StorageBackend<String, Nodegroup>) null);
+ }
+
+ public EksClusterManager(ContainerBuilder containerBuilder,
+ ContainerLifecycleManager lifecycleManager,
+ ContainerDetector containerDetector,
+ PortAllocator portAllocator,
+ DockerHostResolver dockerHostResolver,
+ EcrRegistryManager ecrRegistryManager,
+ EmulatorConfig config,
+ RegionResolver regionResolver,
+ Ec2MetadataServer metadataServer,
+ EksOidcService oidcService,
+ FlociCertificateAuthority certificateAuthority,
+ ContainerLogStreamer logStreamer,
+ StorageBackend<String, Nodegroup> nodeGroupStorage) {
this.containerBuilder = containerBuilder;
this.lifecycleManager = lifecycleManager;
this.containerDetector = containerDetector;
@@ -390,6 +417,30 @@ public class EksClusterManager
this.oidcService = oidcService;
this.certificateAuthority = certificateAuthority;
this.logStreamer = logStreamer;
+ this.nodeGroupStorage = nodeGroupStorage;
+ }
+
+ @Inject
+ public EksClusterManager(ContainerBuilder containerBuilder,
+ ContainerLifecycleManager lifecycleManager,
+ ContainerDetector containerDetector,
+ PortAllocator portAllocator,
+ DockerHostResolver dockerHostResolver,
+ EcrRegistryManager ecrRegistryManager,
+ EmulatorConfig config,
+ RegionResolver regionResolver,
+ Ec2MetadataServer metadataServer,
+ EksOidcService oidcService,
+ FlociCertificateAuthority certificateAuthority,
+ ContainerLogStreamer logStreamer,
+ StorageFactory storageFactory) {
+ this(containerBuilder, lifecycleManager, containerDetector, portAllocator,
+ dockerHostResolver, ecrRegistryManager, config, regionResolver, metadataServer,
+ oidcService, certificateAuthority, logStreamer,
+ storageFactory != null
+ ? storageFactory.create("eks", "eks-nodegroups.json",
+ new TypeReference<Map<String, Nodegroup>>() {})
+ : null);
}
/**
@@ -530,8 +581,15 @@ public class EksClusterManager
try {
String region = clusterRegion(cluster);
String az = deriveClusterNodeAvailabilityZone(cluster, region);
- serverArgs.add("--kubelet-arg=node-labels=topology.kubernetes.io/zone=" + az
- + ",topology.kubernetes.io/region=" + region);
+ Nodegroup nodegroup = selectFirstNodegroup(cluster);
+ String nodeLabels = buildNodeLabels(cluster, nodegroup, az, region);
+ if (nodeLabels != null && !nodeLabels.isBlank()) {
+ serverArgs.add("--kubelet-arg=node-labels=" + nodeLabels);
+ }
+ String taints = buildRegisterWithTaints(nodegroup);
+ if (taints != null && !taints.isBlank()) {
+ serverArgs.add("--kubelet-arg=register-with-taints=" + taints);
+ }
} catch (Exception e) {
String clusterName = cluster != null ? cluster.getName() : "unknown";
LOG.warnv("EKS node topology labels injection disabled for cluster {0}: could not derive topology labels: {1}",
@@ -2434,6 +2492,202 @@ public class EksClusterManager
return "us-east-1"; // partition-literal: unreachable under CDI, where the resolver or config answers
}
+ List<Nodegroup> listActiveNodegroups(Cluster cluster) {
+ if (nodeGroupStorage == null || cluster == null || cluster.getName() == null) {
+ return List.of();
+ }
+ String prefix = cluster.getName() + "/";
+ String accountId = resolveClusterAccountId(cluster);
+ List<Nodegroup> groups;
+ if (nodeGroupStorage instanceof AccountAwareStorageBackend<Nodegroup> aware) {
+ groups = aware.scanAllAccountEntries(key -> key.startsWith(prefix)).stream()
+ .filter(entry -> accountId == null || accountId.equals(entry.accountId()))
+ .map(AccountAwareStorageBackend.AccountEntry::value)
+ .toList();
+ } else {
+ groups = nodeGroupStorage.scan(key -> key.startsWith(prefix));
+ }
+ return groups.stream()
+ .filter(g -> g.getStatus() == null || g.getStatus() == NodegroupStatus.ACTIVE)
+ .sorted(Comparator.comparing(Nodegroup::getCreatedAt,
+ Comparator.nullsLast(Comparator.naturalOrder()))
+ .thenComparing(Nodegroup::getNodegroupName,
+ Comparator.nullsLast(Comparator.naturalOrder())))
+ .toList();
+ }
+
+ Nodegroup selectFirstNodegroup(Cluster cluster) {
+ List<Nodegroup> active = listActiveNodegroups(cluster);
+ if (active.isEmpty()) {
+ return null;
+ }
+ Nodegroup first = active.getFirst();
+ if (active.size() > 1) {
+ String clusterName = cluster != null ? cluster.getName() : "unknown";
+ for (int i = 1; i < active.size(); i++) {
+ Nodegroup later = active.get(i);
+ LOG.warnv(
+ "EKS cluster {0} has one shared node; nodegroup {1} metadata (labels/taints) is not applied to the node (already represented by nodegroup {2})",
+ clusterName, later.getNodegroupName(), first.getNodegroupName());
+ }
+ }
+ return first;
+ }
+
+ String buildNodeLabels(Cluster cluster, Nodegroup nodegroup, String az, String region) {
+ Map<String, String> labels = new LinkedHashMap<>();
+ if (az != null && !az.isBlank()) {
+ labels.put("topology.kubernetes.io/zone", az);
+ }
+ if (region != null && !region.isBlank()) {
+ labels.put("topology.kubernetes.io/region", region);
+ }
+
+ if (nodegroup != null) {
+ String capacityType = nodegroup.getCapacityType() != null && !nodegroup.getCapacityType().isBlank()
+ ? nodegroup.getCapacityType() : "ON_DEMAND";
+ labels.put("eks.amazonaws.com/capacityType", capacityType); // partition-literal: Kubernetes node label key is partition-invariant
+
+ if (nodegroup.getNodegroupName() != null && !nodegroup.getNodegroupName().isBlank()) {
+ labels.put("eks.amazonaws.com/nodegroup", nodegroup.getNodegroupName()); // partition-literal: Kubernetes node label key is partition-invariant
+ }
+
+ String imageId = resolveNodegroupImageId(nodegroup);
+ if (imageId != null && !imageId.isBlank()) {
+ labels.put("eks.amazonaws.com/nodegroup-image", imageId); // partition-literal: Kubernetes node label key is partition-invariant
+ }
+
+ String instanceType = resolveNodegroupInstanceType(cluster, nodegroup);
+ if (instanceType != null && !instanceType.isBlank()) {
+ labels.put("node.kubernetes.io/instance-type", instanceType);
+ }
+
+ if (nodegroup.getLabels() != null) {
+ for (Map.Entry<String, String> entry : nodegroup.getLabels().entrySet()) {
+ String k = entry.getKey();
+ String v = entry.getValue();
+ if (k != null && !k.isBlank() && v != null && !isReservedNodeLabelKey(k)) {
+ labels.put(k, v);
+ }
+ }
+ }
+ }
+
+ if (labels.isEmpty()) {
+ return null;
+ }
+ StringBuilder sb = new StringBuilder();
+ for (Map.Entry<String, String> entry : labels.entrySet()) {
+ if (!sb.isEmpty()) {
+ sb.append(",");
+ }
+ sb.append(entry.getKey()).append("=").append(entry.getValue());
+ }
+ return sb.toString();
+ }
+
+ private static boolean isReservedNodeLabelKey(String key) {
+ return key.startsWith("topology.kubernetes.io/")
+ || key.startsWith("eks.amazonaws.com/") // partition-literal: Kubernetes label prefix is partition-invariant
+ || "node.kubernetes.io/instance-type".equals(key);
+ }
+
+ private String resolveNodegroupInstanceType(Cluster cluster, Nodegroup nodegroup) {
+ if (nodegroup != null && nodegroup.getInstanceTypes() != null && !nodegroup.getInstanceTypes().isEmpty()) {
+ String first = nodegroup.getInstanceTypes().getFirst();
+ if (first != null && !first.isBlank()) {
+ return selectNodeInstanceType(first.trim());
+ }
+ }
+ return nodeInstanceType(cluster);
+ }
+
+ private String resolveNodegroupImageId(Nodegroup nodegroup) {
+ if (nodegroup != null) {
+ if (nodegroup.getReleaseVersion() != null && !nodegroup.getReleaseVersion().isBlank()) {
+ return nodegroup.getReleaseVersion().trim();
+ }
+ if (nodegroup.getLaunchTemplate() instanceof Map<?, ?> map) {
+ Object directImage = map.get("imageId");
+ if (directImage != null && !directImage.toString().isBlank()) {
+ return directImage.toString().trim();
+ }
+ Ec2Service ec2 = ec2Service();
+ if (ec2 != null) {
+ String id = map.get("id") != null ? map.get("id").toString().trim() : null;
+ String name = map.get("name") != null ? map.get("name").toString().trim() : null;
+ String version = map.get("version") != null ? map.get("version").toString().trim() : null;
+ if ((id != null && !id.isBlank()) || (name != null && !name.isBlank())) {
+ try {
+ LaunchTemplateData data = ec2.resolveLaunchTemplateData(null, id, name, version);
+ if (data != null && data.getImageId() != null && !data.getImageId().isBlank()) {
+ return data.getImageId().trim();
+ }
+ } catch (Exception ignored) {
+ // Swallowing is safe because falling back to default image id is standard
+ }
+ }
+ }
+ }
+ }
+ return "ami-eks-k3s";
+ }
+
+ String buildRegisterWithTaints(Nodegroup nodegroup) {
+ if (nodegroup == null || nodegroup.getTaints() == null || nodegroup.getTaints().isEmpty()) {
+ return null;
+ }
+ List<String> taints = new ArrayList<>();
+ for (Object taintObj : nodegroup.getTaints()) {
+ String formatted = formatTaint(taintObj);
+ if (formatted != null && !formatted.isBlank()) {
+ taints.add(formatted);
+ }
+ }
+ if (taints.isEmpty()) {
+ return null;
+ }
+ return String.join(",", taints);
+ }
+
+ static String formatTaint(Object taintObj) {
+ if (taintObj == null) {
+ return null;
+ }
+ if (taintObj instanceof Map<?, ?> map) {
+ Object k = map.get("key");
+ if (k == null || k.toString().isBlank()) {
+ return null;
+ }
+ String key = k.toString().trim();
+ Object v = map.get("value");
+ String val = (v != null) ? v.toString().trim() : null;
+ Object eff = map.get("effect");
+ String effect = mapTaintEffect(eff != null ? eff.toString() : null);
+ if (val != null && !val.isEmpty()) {
+ return key + "=" + val + ":" + effect;
+ } else {
+ return key + ":" + effect;
+ }
+ } else if (taintObj instanceof String str && !str.isBlank()) {
+ return str.trim();
+ }
+ return null;
+ }
+
+ static String mapTaintEffect(String awsEffect) {
+ if (awsEffect == null || awsEffect.isBlank()) {
+ return "NoSchedule";
+ }
+ String normalized = awsEffect.trim().toUpperCase(Locale.ROOT);
+ return switch (normalized) {
+ case "NO_SCHEDULE", "NOSCHEDULE" -> "NoSchedule";
+ case "NO_EXECUTE", "NOEXECUTE" -> "NoExecute";
+ case "PREFER_NO_SCHEDULE", "PREFERNOSCHEDULE" -> "PreferNoSchedule";
+ default -> awsEffect.trim();
+ };
+ }
+
String deriveClusterNodeAvailabilityZone(Cluster cluster, String region) {
String safeRegion = (region != null && !region.isBlank()) ? region : clusterRegion(cluster);
return safeRegion + "a";
diff --git a/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java b/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java
index 3afce2eabb..1485c5f16a 100644
--- a/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/eks/EksService.java
@@ -259,6 +259,16 @@ public class EksService implements TagHandler, ResourceProvider {
.thenComparing(Nodegroup::getNodegroupName));
}
+ List<Nodegroup> listNodeGroupsForCluster(String clusterName, String accountId) {
+ String prefix = clusterName + "/";
+ if (nodeGroupStorage instanceof AccountAwareStorageBackend<Nodegroup> aware) {
+ return aware.scanAllAccountEntries(key -> key.startsWith(prefix)).stream()
+ .filter(entry -> accountId == null || accountId.equals(entry.accountId()))
+ .map(AccountAwareStorageBackend.AccountEntry::value).toList();
+ }
+ return nodeGroupStorage.scan(key -> key.startsWith(prefix));
+ }
+
/**
* Gives clusters persisted before IRSA support an OIDC issuer and signing key. Without this,
* a cluster restored from {@code eks-clusters.json} would report no
@@ -790,6 +800,10 @@ public class EksService implements TagHandler, ResourceProvider {
boolean firstGroup = firstNodeGroup(clusterName, accountId).isEmpty() && pendingFirst == null;
if (firstGroup && !config.services().eks().mock() && clusterManager != null) {
applyFirstNodeGroupCapacity(clusterName, nodegroupName, currentCluster, nodeGroup);
+ if (currentCluster.getContainerId() != null && currentCluster.getStatus() == ClusterStatus.ACTIVE) {
+ LOG.infov("EKS cluster {0} is already running; nodegroup {1} node labels and taints are not applied to the running node",
+ clusterName, nodegroupName);
+ }
} else if (!firstGroup && pendingFirst == null
&& !config.services().eks().mock() && clusterManager != null) {
if (currentCluster.getNodeInstanceType() == null) {
@@ -802,6 +816,13 @@ public class EksService implements TagHandler, ResourceProvider {
LOG.warnv("EKS cluster {0} has one shared node; nodegroup {1} cannot change its capacity from {2}",
clusterName, nodegroupName, currentCluster.getNodeInstanceType());
}
+ firstNodeGroup(clusterName, accountId).ifPresent(first -> {
+ if (!nodegroupName.equals(first.getNodegroupName())) {
+ LOG.warnv(
+ "EKS cluster {0} has one shared node; nodegroup {1} metadata (labels/taints) is not applied to the node (already represented by nodegroup {2})",
+ clusterName, nodegroupName, first.getNodegroupName());
+ }
+ });
}
if (nodeGroup.getStatus() == NodegroupStatus.ACTIVE && (hasUserData || pendingFirst != null)) {
diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterArgsTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterArgsTest.java
index 3d83ccffba..2f038240e2 100644
--- a/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterArgsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterArgsTest.java
@@ -118,11 +118,15 @@ class EksClusterArgsTest {
"system-reserved",
"kube-reserved",
"eviction-hard",
+ "register-with-taints",
"--provider-id=custom-id",
"providerId=custom-id",
"node-labels=topology.kubernetes.io/zone=custom",
"provider-id+=aws:///custom-id",
- "node-labels-=zone=custom"
+ "node-labels-=zone=custom",
+ "--register-with-taints=dedicated=gpu:NoSchedule",
+ "registerWithTaints=dedicated=gpu:NoSchedule",
+ "register-with-taints+=dedicated=gpu:NoSchedule"
})
void rejectsCollidingKubeletArguments(String collidingFlag) {
Map<String, String> tags = Map.of("floci:kubelet-arg:" + collidingFlag, "true");
diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java
index b5eddbfd75..cf09fb0a20 100644
--- a/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksClusterManagerTest.java
@@ -14,6 +14,8 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.C
import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
import io.github.hectorvent.floci.core.common.docker.PortAllocator;
+import io.github.hectorvent.floci.core.storage.InMemoryStorage;
+import io.github.hectorvent.floci.core.storage.StorageBackend;
import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;
import io.github.hectorvent.floci.services.eks.model.CertificateAuthority;
import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;
@@ -22,6 +24,7 @@ import io.github.hectorvent.floci.services.eks.model.ClusterIdentity;
import io.github.hectorvent.floci.services.eks.model.ClusterOidcKey;
import io.github.hectorvent.floci.services.eks.model.LogSetup;
import io.github.hectorvent.floci.services.eks.model.Logging;
+import io.github.hectorvent.floci.services.eks.model.Nodegroup;
import io.github.hectorvent.floci.services.eks.model.OidcIdentity;
import io.github.hectorvent.floci.services.eks.model.ResourcesVpcConfig;
import io.github.hectorvent.floci.testutil.LogCapture;
@@ -68,6 +71,7 @@ import java.time.Instant;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
@@ -1901,6 +1905,7 @@ class EksClusterManagerTest {
private EksOidcService oidcService;
private RegionResolver regionResolver;
private EksClusterManager manager;
+ private StorageBackend<String, Nodegroup> nodeGroupStorage;
@BeforeEach
void setUp() {
@@ -1944,10 +1949,11 @@ class EksClusterManagerTest {
when(oidcService.newIssuerUrl(anyString())).thenReturn(
"https://oidc.eks.us-east-1.amazonaws.com/id/TESTISSUER12345678901234567890");
+ nodeGroupStorage = new InMemoryStorage<>();
manager = new EksClusterManager(containerBuilder, lifecycleManager,
Mockito.mock(ContainerDetector.class), Mockito.mock(PortAllocator.class),
Mockito.mock(DockerHostResolver.class), Mockito.mock(EcrRegistryManager.class),
- config, regionResolver, null, oidcService);
+ config, regionResolver, null, oidcService, null, null, nodeGroupStorage);
}
@Test
@@ -2296,6 +2302,333 @@ class EksClusterManagerTest {
assertFalse(cmd.stream().anyMatch(arg -> arg.startsWith("--kubelet-arg=node-labels=")));
}
+ @Test
+ void nodegroupLabelsAppearOnNode() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ Nodegroup nodegroup = new Nodegroup();
+ nodegroup.setNodegroupName("worker-group-1");
+ nodegroup.setCapacityType("ON_DEMAND");
+ nodegroup.setInstanceTypes(List.of("t3.medium"));
+ nodegroup.setLabels(Map.of("environment", "production", "team", "platform"));
+ nodeGroupStorage.put("prod-cluster/worker-group-1", nodegroup);
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ String nodeLabelsArg = cmd.stream()
+ .filter(arg -> arg.startsWith("--kubelet-arg=node-labels="))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("node-labels arg missing"));
+ Map<String, String> labels = parseNodeLabels(nodeLabelsArg);
+
+ assertEquals("us-west-2a", labels.get("topology.kubernetes.io/zone"));
+ assertEquals("us-west-2", labels.get("topology.kubernetes.io/region"));
+ assertEquals("worker-group-1", labels.get("eks.amazonaws.com/nodegroup"));
+ assertEquals("ON_DEMAND", labels.get("eks.amazonaws.com/capacityType"));
+ assertEquals("ami-eks-k3s", labels.get("eks.amazonaws.com/nodegroup-image"));
+ assertEquals("t3.medium", labels.get("node.kubernetes.io/instance-type"));
+ assertEquals("production", labels.get("environment"));
+ assertEquals("platform", labels.get("team"));
+ }
+
+ @Test
+ void nodegroupTaintsAppearWithKubernetesEffectSpelling() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ Nodegroup nodegroup = new Nodegroup();
+ nodegroup.setNodegroupName("gpu-group");
+ nodegroup.setTaints(List.of(
+ Map.of("key", "dedicated", "value", "gpu", "effect", "NO_SCHEDULE"),
+ Map.of("key", "evict", "value", "true", "effect", "NO_EXECUTE"),
+ Map.of("key", "spot", "value", "preemptible", "effect", "PREFER_NO_SCHEDULE")
+ ));
+ nodeGroupStorage.put("prod-cluster/gpu-group", nodegroup);
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ assertTrue(cmd.contains("--kubelet-arg=register-with-taints="
+ + "dedicated=gpu:NoSchedule,evict=true:NoExecute,spot=preemptible:PreferNoSchedule"));
+ }
+
+ @Test
+ void nodegroupCapacityTypeSpot() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ Nodegroup nodegroup = new Nodegroup();
+ nodegroup.setNodegroupName("spot-group");
+ nodegroup.setCapacityType("SPOT");
+ nodeGroupStorage.put("prod-cluster/spot-group", nodegroup);
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ String nodeLabelsArg = cmd.stream()
+ .filter(arg -> arg.startsWith("--kubelet-arg=node-labels="))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("node-labels arg missing"));
+ Map<String, String> labels = parseNodeLabels(nodeLabelsArg);
+ assertEquals("SPOT", labels.get("eks.amazonaws.com/capacityType"));
+ assertEquals("spot-group", labels.get("eks.amazonaws.com/nodegroup"));
+ }
+
+ @Test
+ void nodegroupCapacityTypeCapacityBlock() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ Nodegroup nodegroup = new Nodegroup();
+ nodegroup.setNodegroupName("block-group");
+ nodegroup.setCapacityType("CAPACITY_BLOCK");
+ nodeGroupStorage.put("prod-cluster/block-group", nodegroup);
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ String nodeLabelsArg = cmd.stream()
+ .filter(arg -> arg.startsWith("--kubelet-arg=node-labels="))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("node-labels arg missing"));
+ Map<String, String> labels = parseNodeLabels(nodeLabelsArg);
+ assertEquals("CAPACITY_BLOCK", labels.get("eks.amazonaws.com/capacityType"));
+ assertEquals("block-group", labels.get("eks.amazonaws.com/nodegroup"));
+ }
+
+ @Test
+ void clusterWithNoNodegroupsProducesSameArgumentsAsBefore() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ String expectedAz = manager.deriveClusterNodeAvailabilityZone(cluster, "us-west-2");
+ assertTrue(cmd.contains("--kubelet-arg=node-labels=topology.kubernetes.io/zone=" + expectedAz
+ + ",topology.kubernetes.io/region=us-west-2"));
+ assertFalse(cmd.stream().anyMatch(arg -> arg.startsWith("--kubelet-arg=register-with-taints=")));
+ assertFalse(cmd.stream().anyMatch(arg -> arg.contains("eks.amazonaws.com/")));
+ }
+
+ @Test
+ void multipleNodegroupsAppliesFirstAndIgnoresLater() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ Instant now = Instant.now();
+ Nodegroup first = new Nodegroup();
+ first.setNodegroupName("first-ng");
+ first.setCreatedAt(now.minusSeconds(60));
+ first.setLabels(Map.of("role", "frontend"));
+ first.setTaints(List.of(Map.of("key", "tier", "value", "frontend", "effect", "NO_SCHEDULE")));
+
+ Nodegroup second = new Nodegroup();
+ second.setNodegroupName("second-ng");
+ second.setCreatedAt(now);
+ second.setLabels(Map.of("role", "backend", "secondary", "true"));
+ second.setTaints(List.of(Map.of("key", "tier", "value", "backend", "effect", "NO_SCHEDULE")));
+
+ nodeGroupStorage.put("prod-cluster/second-ng", second);
+ nodeGroupStorage.put("prod-cluster/first-ng", first);
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ String nodeLabelsArg = cmd.stream()
+ .filter(arg -> arg.startsWith("--kubelet-arg=node-labels="))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("node-labels arg missing"));
+ Map<String, String> labels = parseNodeLabels(nodeLabelsArg);
+
+ assertEquals("first-ng", labels.get("eks.amazonaws.com/nodegroup"));
+ assertEquals("frontend", labels.get("role"));
+ assertNull(labels.get("secondary"));
+
+ String taintsArg = cmd.stream()
+ .filter(arg -> arg.startsWith("--kubelet-arg=register-with-taints="))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("register-with-taints arg missing"));
+ assertEquals("--kubelet-arg=register-with-taints=tier=frontend:NoSchedule", taintsArg);
+ }
+
+ @Test
+ void topologyLabelsAndProviderIdPreservedWithNodegroup() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ Nodegroup nodegroup = new Nodegroup();
+ nodegroup.setNodegroupName("app-workers");
+ nodegroup.setLabels(Map.of("app", "test"));
+ nodeGroupStorage.put("prod-cluster/app-workers", nodegroup);
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ String expectedProviderId = manager.deriveClusterNodeProviderId(cluster);
+ assertTrue(cmd.contains("--kubelet-arg=provider-id=" + expectedProviderId));
+
+ String nodeLabelsArg = cmd.stream()
+ .filter(arg -> arg.startsWith("--kubelet-arg=node-labels="))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("node-labels arg missing"));
+ Map<String, String> labels = parseNodeLabels(nodeLabelsArg);
+ assertEquals("us-west-2a", labels.get("topology.kubernetes.io/zone"));
+ assertEquals("us-west-2", labels.get("topology.kubernetes.io/region"));
+ }
+
+ @Test
+ void userLabelsCannotOverwriteReservedKeys() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ Nodegroup nodegroup = new Nodegroup();
+ nodegroup.setNodegroupName("worker-group");
+ nodegroup.setCapacityType("ON_DEMAND");
+ nodegroup.setInstanceTypes(List.of("m5.large"));
+ nodegroup.setLabels(Map.of(
+ "topology.kubernetes.io/zone", "fake-zone",
+ "topology.kubernetes.io/region", "fake-region",
+ "eks.amazonaws.com/capacityType", "FAKE_CAPACITY",
+ "eks.amazonaws.com/nodegroup", "fake-group",
+ "eks.amazonaws.com/nodegroup-image", "fake-image",
+ "node.kubernetes.io/instance-type", "fake-type",
+ "custom-label", "valid-value"
+ ));
+ nodeGroupStorage.put("prod-cluster/worker-group", nodegroup);
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ String nodeLabelsArg = cmd.stream()
+ .filter(arg -> arg.startsWith("--kubelet-arg=node-labels="))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("node-labels arg missing"));
+ Map<String, String> labels = parseNodeLabels(nodeLabelsArg);
+
+ assertEquals("us-west-2a", labels.get("topology.kubernetes.io/zone"));
+ assertEquals("us-west-2", labels.get("topology.kubernetes.io/region"));
+ assertEquals("ON_DEMAND", labels.get("eks.amazonaws.com/capacityType"));
+ assertEquals("worker-group", labels.get("eks.amazonaws.com/nodegroup"));
+ assertEquals("ami-eks-k3s", labels.get("eks.amazonaws.com/nodegroup-image"));
+ assertEquals("m5.large", labels.get("node.kubernetes.io/instance-type"));
+ assertEquals("valid-value", labels.get("custom-label"));
+ }
+
+ @Test
+ void unrecognizedInstanceTypeFallsBackToDefaultCatalogType() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ Nodegroup nodegroup = new Nodegroup();
+ nodegroup.setNodegroupName("custom-group");
+ nodegroup.setInstanceTypes(List.of("nonexistent.nano"));
+ nodeGroupStorage.put("prod-cluster/custom-group", nodegroup);
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ String nodeLabelsArg = cmd.stream()
+ .filter(arg -> arg.startsWith("--kubelet-arg=node-labels="))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("node-labels arg missing"));
+ Map<String, String> labels = parseNodeLabels(nodeLabelsArg);
+ assertEquals("m5.large", labels.get("node.kubernetes.io/instance-type"));
+ }
+
+ @Test
+ void nodegroupReleaseVersionUsedAsImageId() {
+ Cluster cluster = new Cluster();
+ cluster.setName("prod-cluster");
+ cluster.setAccountId("123456789012");
+ cluster.setArn("arn:aws:eks:us-west-2:123456789012:cluster/prod-cluster");
+
+ Nodegroup nodegroup = new Nodegroup();
+ nodegroup.setNodegroupName("versioned-group");
+ nodegroup.setReleaseVersion("1.30.2-20240703");
+ nodeGroupStorage.put("prod-cluster/versioned-group", nodegroup);
+
+ manager.startCluster(cluster);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> cmdCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withCmd(cmdCaptor.capture());
+ List<String> cmd = cmdCaptor.getValue();
+
+ String nodeLabelsArg = cmd.stream()
+ .filter(arg -> arg.startsWith("--kubelet-arg=node-labels="))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("node-labels arg missing"));
+ Map<String, String> labels = parseNodeLabels(nodeLabelsArg);
+ assertEquals("1.30.2-20240703", labels.get("eks.amazonaws.com/nodegroup-image"));
+ }
+
+ private static Map<String, String> parseNodeLabels(String nodeLabelsArg) {
+ String raw = nodeLabelsArg.substring("--kubelet-arg=node-labels=".length());
+ Map<String, String> map = new LinkedHashMap<>();
+ for (String pair : raw.split(",")) {
+ String[] kv = pair.split("=", 2);
+ map.put(kv[0], kv.length > 1 ? kv[1] : "");
+ }
+ return map;
+ }
+
@Test
void signingKeyFilesWrittenWithRestrictivePermissions(@TempDir Path tempDir) throws Exception {
when(eks.irsaSigningKey()).thenReturn(true);
diff --git a/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java
index 3dc73be3be..8367668774 100644
--- a/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/eks/EksServiceTest.java
@@ -1036,6 +1036,28 @@ class EksServiceTest {
assertEquals(List.of("nodegroup-b"), eksService.listNodeGroups("my-eks-cluster"));
}
+ @Test
+ void createNodeGroupRecordsLabelsAndTaints() {
+ createTestCluster("metadata-cluster");
+ CreateNodeGroupRequest request = nodeGroupRequest("labeled-ng");
+ request.setLabels(Map.of("role", "worker", "tier", "frontend"));
+ request.setTaints(List.of(Map.of("key", "dedicated", "value", "special", "effect", "NO_SCHEDULE")));
+ request.setCapacityType("SPOT");
+
+ Nodegroup nodegroup = eksService.createNodeGroup("metadata-cluster", request);
+ assertEquals("SPOT", nodegroup.getCapacityType());
+ assertEquals("worker", nodegroup.getLabels().get("role"));
+ assertEquals(1, nodegroup.getTaints().size());
+
+ Nodegroup retrieved = eksService.describeNodeGroup("metadata-cluster", "labeled-ng");
+ assertEquals("SPOT", retrieved.getCapacityType());
+ assertEquals("worker", retrieved.getLabels().get("role"));
+ assertEquals(1, retrieved.getTaints().size());
+
+ eksService.deleteNodeGroup("metadata-cluster", "labeled-ng");
+ assertEquals(List.of(), eksService.listNodeGroups("metadata-cluster"));
+ }
+
@Test
void createNodeGroupDuplicateFails() {
createTestCluster("my-eks-cluster");