Light, fluffy, and always free - The AWS Local Emulator alternative
feat(lambda): support kubernetes as the lambda runner (#1941)
TATSUNO “Taz” Yasuhirobc1380112168
diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.mdindex 331e5c129d..a08ca653bb 100644--- a/docs/configuration/environment-variables.md+++ b/docs/configuration/environment-variables.md@@ -202,6 +202,11 @@ See [Initialization Hooks](./initialization-hooks.md) for lifecycle phases and s| `FLOCI_SERVICES_LAMBDA_CONTAINER_NAME_PREFIX` | `floci` | Base name prefix for Lambda-spawned containers and code volumes (must match `[A-Za-z0-9][A-Za-z0-9_.-]*`) || `FLOCI_SERVICES_LAMBDA_DOCKER_HOST_OVERRIDE` | _(none)_ | Explicit host/IP Lambda containers use to reach the Runtime API, bypassing auto-detection (e.g. rootless Podman) || `FLOCI_SERVICES_LAMBDA_AWS_CONFIG_PATH` | _(none)_ | Host path bind-mounted read-only at `/opt/aws-config` inside Lambda containers for real credential discovery |+| `FLOCI_SERVICES_LAMBDA_EXECUTOR` | `docker` | Execution backend for Lambda environments: `docker` (containers) or `kubernetes` (pods) |+| `FLOCI_SERVICES_LAMBDA_KUBERNETES_NAMESPACE` | `default` | Namespace Lambda pods are created in |+| `FLOCI_SERVICES_LAMBDA_KUBERNETES_LABELS` | _(none)_ | Extra pod labels as comma-separated `key=value` entries |+| `FLOCI_SERVICES_LAMBDA_KUBERNETES_FLOCI_ADDRESS` | _(none)_ | Host/IP Lambda pods use to reach Floci; auto-detected when Floci runs in-cluster |+| `FLOCI_SERVICES_LAMBDA_KUBERNETES_INIT_IMAGE` | `busybox:1.36` | Init-container image that downloads function code into the pod (needs `sh`, `wget`, `unzip`) |### API Gatewaydiff --git a/docs/services/lambda.md b/docs/services/lambda.mdindex 0612bc7b27..045612cb8f 100644--- a/docs/services/lambda.md+++ b/docs/services/lambda.md@@ -206,6 +206,11 @@ These AWS Lambda operations have no handler in Floci. Calls will return `404` or| `FLOCI_SERVICES_LAMBDA_EXTRA_HOSTS` | *(unset)* | Comma-separated `hostname:ip` entries added to each Lambda container's `/etc/hosts`; `ip` may be `host-gateway`, mirroring `docker run --add-host` || `FLOCI_SERVICES_LAMBDA_DOCKER_HOST_OVERRIDE` | *(unset)* | Explicit host/IP that spawned Lambda containers use to reach Floci's Runtime API, bypassing auto-detection || `FLOCI_SERVICES_LAMBDA_CONTAINER_NAME_PREFIX` | `floci` | Base name prefix for spawned Lambda containers and code volumes (e.g. `acme` → `acme-<function>-<id>` containers, `acme-code-<function>-<hash>` volumes). Must be a valid Docker name segment (`[A-Za-z0-9][A-Za-z0-9_.-]*`); invalid values are ignored with a warning |+| `FLOCI_SERVICES_LAMBDA_EXECUTOR` | `docker` | Execution backend: `docker` (containers) or `kubernetes` (pods) |+| `FLOCI_SERVICES_LAMBDA_KUBERNETES_NAMESPACE` | `default` | Namespace Lambda pods are created in |+| `FLOCI_SERVICES_LAMBDA_KUBERNETES_LABELS` | *(unset)* | Extra pod labels as comma-separated `key=value` entries |+| `FLOCI_SERVICES_LAMBDA_KUBERNETES_FLOCI_ADDRESS` | *(unset)* | Host/IP pods use to reach Floci; auto-detected when Floci runs in-cluster |+| `FLOCI_SERVICES_LAMBDA_KUBERNETES_INIT_IMAGE` | `busybox:1.36` | Init-container image that downloads function code (needs `sh`, `wget`, `unzip`) |!!! note "Changing the container name prefix"Code volumes are resolved by name, and a Floci process only manages resources under its@@ -243,7 +248,8 @@ for a full rootless Podman walkthrough.### Docker socket requirement-Lambda requires the Docker socket. Mount it in your compose file:+With the default `docker` executor, Lambda requires the Docker socket. Mount it+in your compose file:```yamlservices:@@ -252,6 +258,169 @@ services:- /var/run/docker.sock:/var/run/docker.sock```+If mounting the Docker socket is not acceptable (for example on a hardened+Kubernetes cluster), use the [Kubernetes executor](#kubernetes-executor)+instead — it needs no privileged access and no socket.++### Kubernetes executor++Set `FLOCI_SERVICES_LAMBDA_EXECUTOR=kubernetes` to run each Lambda execution+environment as a Kubernetes pod instead of a Docker container. This is designed+for CI/CD clusters where privileged containers and `docker.sock` access are not+allowed. Floci talks to the cluster through the standard Kubernetes API: when+running inside the cluster it uses its ServiceAccount, and when running outside+it uses your local kubeconfig.++How an invocation works:++1. On a cold start Floci creates a pod from the function's runtime image+ (`public.ecr.aws/lambda/*`, same mapping as the Docker executor).+2. An init container (`busybox` by default) downloads the function's deployment+ package — and any layers — from Floci's S3 over HTTP and unpacks them into+ shared `emptyDir` volumes at `/var/task` and `/opt`.+3. The runtime container polls Floci's Lambda Runtime API+ (`AWS_LAMBDA_RUNTIME_API`) exactly like a Docker container would.+4. Warm pods are reused across invocations and reaped after+ `FLOCI_SERVICES_LAMBDA_CONTAINER_IDLE_TIMEOUT_SECONDS` of inactivity.+ Pods left behind by a crashed Floci are swept on startup via the+ `app.kubernetes.io/managed-by=floci` label.++Lambda pods connect **back** to Floci on the main port (4566) and the Runtime+API port range (9200–9299), so those ports must be reachable from pods in the+namespace. When Floci runs in-cluster this works out of the box (pod-to-pod+traffic); when Floci runs outside the cluster, set+`FLOCI_SERVICES_LAMBDA_KUBERNETES_FLOCI_ADDRESS` to an address the cluster's+pods can reach (for example your machine's LAN IP for a `kind` cluster).++#### Required RBAC++The ServiceAccount Floci runs under needs these permissions in the Lambda+namespace. The manifest below is complete, so applying it as-is (together+with the Deployment in the next section) yields a working setup:++```yaml+apiVersion: v1+kind: ServiceAccount+metadata:+ name: floci+ namespace: floci+---+apiVersion: rbac.authorization.k8s.io/v1+kind: Role+metadata:+ name: floci-lambda+ namespace: floci+rules:+ - apiGroups: [""]+ resources: ["pods"]+ verbs: ["create", "get", "list", "watch", "delete", "deletecollection"]+ - apiGroups: [""]+ resources: ["pods/log"]+ verbs: ["get", "watch"]+ # Only needed when FLOCI_TLS_ENABLED=true (CA cert is shared via a ConfigMap)+ - apiGroups: [""]+ resources: ["configmaps"]+ verbs: ["create", "get", "update", "patch"]+---+apiVersion: rbac.authorization.k8s.io/v1+kind: RoleBinding+metadata:+ name: floci-lambda+ namespace: floci+roleRef:+ apiGroup: rbac.authorization.k8s.io+ kind: Role+ name: floci-lambda+subjects:+ - kind: ServiceAccount+ name: floci+ namespace: floci+```++#### Running Floci in-cluster++A minimal Deployment (namespace `floci` assumed, RBAC from above bound to the+`floci` ServiceAccount):++```yaml+apiVersion: apps/v1+kind: Deployment+metadata:+ name: floci+spec:+ replicas: 1+ selector:+ matchLabels: { app: floci }+ template:+ metadata:+ labels: { app: floci }+ spec:+ serviceAccountName: floci+ # Required when a Service named `floci` exists in the namespace: service+ # links inject FLOCI_PORT=tcp://<ip>:4566, which collides with Floci's+ # FLOCI_* configuration convention and fails startup.+ enableServiceLinks: false+ containers:+ - name: floci+ image: floci/floci:latest+ env:+ - name: FLOCI_SERVICES_LAMBDA_EXECUTOR+ value: kubernetes+ - name: FLOCI_SERVICES_LAMBDA_KUBERNETES_NAMESPACE+ value: floci+ ports:+ - containerPort: 4566+---+apiVersion: v1+kind: Service+metadata:+ name: floci+spec:+ selector: { app: floci }+ ports:+ - port: 4566+ targetPort: 4566+```++Lambda pods reach Floci by pod IP, so the Runtime API ports need no Service+entries; only clients of the emulator itself use port 4566.++#### Limitations++- Hot reload (both bind-mount and Reactive S3 Sync bind variants) and+ `FLOCI_SERVICES_LAMBDA_AWS_CONFIG_PATH` rely on bind mounts and are not+ supported; hot-reload functions fail to launch with a clear error.+- `Image` package type URIs are passed to the kubelet unchanged. Images in+ Floci's emulated ECR registry are not pullable by cluster nodes — pre-load+ them onto the nodes (e.g. `kind load docker-image`) or use a real registry.+- Lambda pods carry no `imagePullSecrets`. If your runtime or init images come+ from an authenticated registry, attach the pull secret to the namespace's+ `default` ServiceAccount+ (`kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"<secret>"}]}'`)+ so the kubelet uses it for every pod in the namespace.+- A cold start waits up to 300 seconds for the pod to reach `Running` (broken+ images and failing init containers are detected and reported much earlier).+- The init container downloads code and layers over plain HTTP on the+ emulator port even when `FLOCI_TLS_ENABLED=true`. busybox `wget` cannot+ complete a TLS handshake with Floci, and the Runtime API traffic on the+ same pod network is plain HTTP anyway. Use a NetworkPolicy if the pod+ network is part of your threat model.+- With `FLOCI_TLS_ENABLED=true`, AWS SDK calls made from inside the function+ fail TLS verification when pods reach Floci by pod IP, because the+ self-signed certificate carries no SAN for dynamic pod IPs. Set+ `FLOCI_SERVICES_LAMBDA_KUBERNETES_FLOCI_ADDRESS` to a hostname covered by+ the certificate, or provide your own certificate via `floci.tls.cert-path`.+- Prefer an IP for `FLOCI_SERVICES_LAMBDA_KUBERNETES_FLOCI_ADDRESS`. With a+ bare hostname, S3 SDKs inside functions may use virtual-hosted-style+ addressing (`bucket.<hostname>`), and nothing in the cluster resolves those+ subdomains.+- IPv6-only clusters are not supported. Floci advertises an IPv4 address to+ pods and fails fast when none is available.+- Each Floci instance sweeps all `managed-by=floci` Lambda pods in its+ namespace at startup; run multiple Floci instances in separate namespaces.+- Layers published by Floci versions before this feature have no stored+ archive; re-publish them once to make them downloadable by pods.+### S3 virtual-hosted-style addressing inside Lambda containersAWS SDKs use **virtual-hosted-style** S3 addressing by default, forming URLs likediff --git a/pom.xml b/pom.xmlindex 37c8b62d69..a6f37a9bf1 100644--- a/pom.xml+++ b/pom.xml@@ -146,6 +146,12 @@<artifactId>snappy-java</artifactId></dependency>+ <!-- Kubernetes client for the Lambda kubernetes executor -->+ <dependency>+ <groupId>io.quarkus</groupId>+ <artifactId>quarkus-kubernetes-client</artifactId>+ </dependency>+<!-- Docker client for Lambda container lifecycle management --><dependency><groupId>com.github.docker-java</groupId>@@ -281,6 +287,17 @@<version>4.3.0</version><scope>test</scope></dependency>+ <dependency>+ <groupId>io.fabric8</groupId>+ <artifactId>kubernetes-server-mock</artifactId>+ <scope>test</scope>+ </dependency>+ <dependency>+ <groupId>org.assertj</groupId>+ <artifactId>assertj-core</artifactId>+ <version>3.25.3</version>+ <scope>test</scope>+ </dependency><dependency><groupId>org.eclipse.paho</groupId><artifactId>org.eclipse.paho.client.mqttv3</artifactId>@@ -350,6 +367,9 @@<artifactId>maven-surefire-plugin</artifactId><version>3.5.6</version><configuration>+ <!-- The suite's end-of-run live set exceeds the 4g default heap of+ 16g CI runners; a full GC near the end reclaims almost nothing. -->+ <argLine>-Xmx6g</argLine><systemPropertyVariables><java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager></systemPropertyVariables>diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex 320a9b16e7..ca13b50e79 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -1577,6 +1577,55 @@ public interface EmulatorConfig {*/Optional<String> awsConfigPath();+ /**+ * Execution backend for Lambda environments: {@code docker} (default) runs each+ * environment as a Docker container, {@code kubernetes} runs it as a pod in the+ * cluster Floci is configured against (in-cluster config or local kubeconfig).+ *+ * Env var: FLOCI_SERVICES_LAMBDA_EXECUTOR+ */+ @WithDefault("docker")+ String executor();++ KubernetesExecutor kubernetes();++ interface KubernetesExecutor {+ /**+ * Namespace Lambda pods are created in. Multiple Floci instances must use+ * separate namespaces — orphaned pods are swept by label on startup.+ *+ * Env var: FLOCI_SERVICES_LAMBDA_KUBERNETES_NAMESPACE+ */+ @WithDefault("default")+ String namespace();++ /**+ * Extra labels applied to Lambda pods, as {@code key=value} entries.+ *+ * Env var: FLOCI_SERVICES_LAMBDA_KUBERNETES_LABELS (comma-separated)+ */+ Optional<List<String>> labels();++ /**+ * Host or IP that Lambda pods use to reach Floci (the Runtime API port range+ * and the main port). When unset, Floci auto-detects its own pod address if+ * running in-cluster; when running outside the cluster this must be set to an+ * address the cluster's pods can reach.+ *+ * Env var: FLOCI_SERVICES_LAMBDA_KUBERNETES_FLOCI_ADDRESS+ */+ Optional<String> flociAddress();++ /**+ * Image for the init container that downloads and unpacks function code into+ * the pod. Must provide sh, wget and unzip.+ *+ * Env var: FLOCI_SERVICES_LAMBDA_KUBERNETES_INIT_IMAGE+ */+ @WithDefault("busybox:1.36")+ String initImage();+ }+HotReload hotReload();interface HotReload {diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/LaunchedContainerAwsEnv.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/LaunchedContainerAwsEnv.javaindex 5d63cc4f67..0e4fa60c71 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/LaunchedContainerAwsEnv.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/LaunchedContainerAwsEnv.java@@ -47,7 +47,15 @@ public class LaunchedContainerAwsEnv {* credentials from there. Empty = inject host or placeholder credentials.*/public List<String> sdkBaselineEnv(String region, Optional<String> awsConfigMountDir) {- return sdkBaselineEnv(region, awsConfigMountDir, Optional.empty());+ return sdkBaselineEnv(region, awsConfigMountDir, reachableEndpoint.baseUrl(), Optional.empty());+ }++ /**+ * Variant for launchers whose workloads reach Floci at an address other than the+ * Docker-reachable one (e.g. Kubernetes pods reaching Floci's pod IP).+ */+ public List<String> sdkBaselineEnv(String region, Optional<String> awsConfigMountDir, String flociEndpoint) {+ return sdkBaselineEnv(region, awsConfigMountDir, flociEndpoint, Optional.empty());}/**@@ -56,25 +64,34 @@ public class LaunchedContainerAwsEnv {*/public List<String> sdkBaselineEnv(String region, Optional<String> awsConfigMountDir,Optional<SessionCreds> injectedCredentials) {- List<String> env = new ArrayList<>();+ return sdkBaselineEnv(region, awsConfigMountDir, reachableEndpoint.baseUrl(), injectedCredentials);+ }++ /**+ * Full variant taking both the Floci endpoint the workload should target and optional+ * workload-specific session credentials.+ */+ public List<String> sdkBaselineEnv(String region, Optional<String> awsConfigMountDir,+ String flociEndpoint, Optional<SessionCreds> injectedCredentials) {+ var env = new ArrayList<String>();env.add("AWS_DEFAULT_REGION=" + region);env.add("AWS_REGION=" + region);if (awsConfigMountDir.isPresent() && !awsConfigMountDir.get().isBlank()) {// ~/.aws is mounted, so don't inject credentials. Let the SDK discover them.// Set explicit file paths so discovery works regardless of container HOME.- String dir = awsConfigMountDir.get();+ var dir = awsConfigMountDir.get();env.add("AWS_SHARED_CREDENTIALS_FILE=" + dir + "/credentials");env.add("AWS_CONFIG_FILE=" + dir + "/config");} else if (injectedCredentials.isPresent()) {- SessionCreds credentials = injectedCredentials.get();+ var credentials = injectedCredentials.get();env.add("AWS_ACCESS_KEY_ID=" + credentials.accessKeyId());env.add("AWS_SECRET_ACCESS_KEY=" + credentials.secretAccessKey());env.add("AWS_SESSION_TOKEN=" + credentials.sessionToken());} else {// Use Floci's own env vars, falling back to placeholder credentials.- String ak = System.getenv("AWS_ACCESS_KEY_ID");- String sk = System.getenv("AWS_SECRET_ACCESS_KEY");- String st = System.getenv("AWS_SESSION_TOKEN");+ var ak = System.getenv("AWS_ACCESS_KEY_ID");+ var sk = System.getenv("AWS_SECRET_ACCESS_KEY");+ var st = System.getenv("AWS_SESSION_TOKEN");if ((ak != null || sk != null || st != null) && hostCredentialsWarned.compareAndSet(false, true)) {// The container runs workload code, so surface that it is being handed the// credentials Floci itself was started with (an exported key pair, aws-vault, a@@ -89,7 +106,6 @@ public class LaunchedContainerAwsEnv {env.add("AWS_SECRET_ACCESS_KEY=" + (sk != null ? sk : "test"));env.add("AWS_SESSION_TOKEN=" + (st != null ? st : "test"));}- String flociEndpoint = reachableEndpoint.baseUrl();env.add("FLOCI_HOSTNAME=" + URI.create(flociEndpoint).getHost());env.add("FLOCI_ENDPOINT=" + flociEndpoint);env.add("AWS_ENDPOINT_URL=" + flociEndpoint);diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaLayerController.java b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaLayerController.javaindex d7609cbec0..8028942242 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaLayerController.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaLayerController.java@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.lambda;import com.fasterxml.jackson.databind.ObjectMapper;import com.fasterxml.jackson.databind.node.ArrayNode;import com.fasterxml.jackson.databind.node.ObjectNode;+import io.github.hectorvent.floci.core.common.AwsArnUtils;import io.github.hectorvent.floci.core.common.RegionResolver;import io.github.hectorvent.floci.services.lambda.model.LambdaLayerVersion;import jakarta.inject.Inject;@@ -17,6 +18,7 @@ import jakarta.ws.rs.core.Context;import jakarta.ws.rs.core.HttpHeaders;import jakarta.ws.rs.core.MediaType;import jakarta.ws.rs.core.Response;+import jakarta.ws.rs.core.UriInfo;import java.util.List;import java.util.Map;@@ -52,10 +54,11 @@ public class LambdaLayerController {@Path("/layers/{layerName}/versions")public Response publishLayerVersion(@PathParam("layerName") String layerName,@Context HttpHeaders headers,+ @Context UriInfo uriInfo,Map<String, Object> request) {String region = regionResolver.resolveRegion(headers);LambdaLayerVersion lv = layerService.publishLayerVersion(region, layerName, request);- return Response.status(201).entity(buildLayerVersionResponse(lv)).build();+ return Response.status(201).entity(buildLayerVersionResponse(lv, region, uriInfo)).build();}@GET@@ -76,10 +79,11 @@ public class LambdaLayerController {@Path("/layers/{layerName}/versions/{versionNumber}")public Response getLayerVersion(@PathParam("layerName") String layerName,@PathParam("versionNumber") long versionNumber,- @Context HttpHeaders headers) {+ @Context HttpHeaders headers,+ @Context UriInfo uriInfo) {String region = regionResolver.resolveRegion(headers);LambdaLayerVersion lv = layerService.getLayerVersion(region, layerName, versionNumber);- return Response.ok(buildLayerVersionResponse(lv)).build();+ return Response.ok(buildLayerVersionResponse(lv, region, uriInfo)).build();}@DELETE@@ -122,7 +126,7 @@ public class LambdaLayerController {return Response.ok(root).build();}- private ObjectNode buildLayerVersionResponse(LambdaLayerVersion lv) {+ private ObjectNode buildLayerVersionResponse(LambdaLayerVersion lv, String region, UriInfo uriInfo) {ObjectNode node = objectMapper.createObjectNode();node.put("LayerArn", lv.getLayerArn());node.put("LayerVersionArn", lv.getLayerVersionArn());@@ -135,8 +139,10 @@ public class LambdaLayerController {ObjectNode contentNode = node.putObject("Content");contentNode.put("CodeSha256", lv.getCodeSha256() != null ? lv.getCodeSha256() : "");contentNode.put("CodeSize", lv.getCodeSizeBytes());- // Location is a presigned URL in real AWS; we provide a placeholder- contentNode.put("Location", "");+ // Only advertise a fetchable Location when the archive was actually stored; AWS+ // clients treat Content.Location as a downloadable URL, so a placeholder empty+ // string is correct for pre-upgrade or failed-store layers.+ contentNode.put("Location", lv.isArchiveStored() ? tasksLocation(lv, region, uriInfo) : "");if (lv.getCompatibleRuntimes() != null && !lv.getCompatibleRuntimes().isEmpty()) {ArrayNode runtimes = node.putArray("CompatibleRuntimes");@@ -149,6 +155,28 @@ public class LambdaLayerController {return node;}+ /**+ * Path-style URL to the archive in Floci's own S3 (a presigned URL in real AWS),+ * built from the request so it targets the same endpoint the client is talking to.+ * Segments are percent-encoded directly rather than through {@code UriBuilder.path},+ * whose template syntax would throw on a layer name containing braces.+ * Clients fetch this URL unsigned, so an {@code X-Amz-Credential} query steers+ * Floci's account filter to the layer's owning account; without it a layer owned+ * by a non-default account resolves the default-account bucket and 404s.+ */+ private String tasksLocation(LambdaLayerVersion lv, String region, UriInfo uriInfo) {+ var base = uriInfo.getBaseUri().toString();+ if (base.endsWith("/")) {+ base = base.substring(0, base.length() - 1);+ }+ var account = AwsArnUtils.accountOrDefault(lv.getLayerVersionArn(), "000000000000");+ var bucket = LambdaService.tasksBucketName(region);+ var key = LambdaService.layerObjectKey(account, lv.getLayerName(), lv.getVersion());+ return base + "/" + LambdaService.encodeObjectPath(bucket)+ + "/" + LambdaService.encodeObjectPath(key)+ + "?X-Amz-Credential=" + account + "%2F00010101%2F" + region + "%2Fs3%2Faws4_request";+ }+private ObjectNode buildLayerVersionSummary(LambdaLayerVersion lv) {ObjectNode node = objectMapper.createObjectNode();node.put("LayerVersionArn", lv.getLayerVersionArn());diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaLayerService.java b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaLayerService.javaindex 37f0f74c59..c152833780 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaLayerService.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaLayerService.java@@ -112,11 +112,33 @@ public class LambdaLayerService {layerVersion.setCodeSha256(codeSha256);layerVersion.setCodeLocalPath(layerPath.toAbsolutePath().normalize().toString());+ // Store the archive before persisting the version so a load-bearing store failure+ // (kubernetes executor) fails the publish instead of leaving a version pods cannot use.+ boolean stored = storeLayerArchive(region, accountId, layerName, nextVersion, zipBytes);+ if (!stored && LambdaService.requiresStoredTasksObject(config)) {+ throw new AwsException("ServiceException",+ "Could not store the layer archive for '" + layerName + "' v" + nextVersion+ + " in Floci's S3, which the kubernetes Lambda executor needs. The "+ + "layer version was not published.", 500);+ }+ layerVersion.setArchiveStored(stored);layerStore.save(region, layerVersion);LOG.infov("Published layer version: {0} v{1} in region {2}", layerName, nextVersion, region);return layerVersion;}+ /**+ * Keeps the exact layer archive in Floci's S3 so GetLayerVersion can serve a real+ * Content.Location and the kubernetes executor's init container can download it.+ * Best-effort unless the active Lambda executor requires stored tasks-bucket objects.+ */+ private boolean storeLayerArchive(String region, String accountId, String layerName,+ long version, byte[] zipBytes) {+ return LambdaService.putTasksObjectQuietly(s3Service, region,+ LambdaService.layerObjectKey(accountId, layerName, version), zipBytes,+ "layer archive for " + layerName + " v" + version);+ }+/*** Returns information about a specific layer version.*/@@ -157,9 +179,24 @@ public class LambdaLayerService {}layerStore.delete(region, layerName, versionNumber);+ deleteLayerArchive(region, lv);LOG.infov("Deleted layer version: {0} v{1} in region {2}", layerName, versionNumber, region);}+ private void deleteLayerArchive(String region, LambdaLayerVersion lv) {+ if (s3Service == null) {+ return;+ }+ try {+ var account = AwsArnUtils.accountOrDefault(lv.getLayerVersionArn(), "000000000000");+ s3Service.deleteObject(LambdaService.tasksBucketName(region),+ LambdaService.layerObjectKey(account, lv.getLayerName(), lv.getVersion()));+ } catch (Exception e) {+ LOG.debugv("Could not delete stored layer archive for {0} v{1}: {2}",+ lv.getLayerName(), lv.getVersion(), e.getMessage());+ }+ }+/*** Resolves a layer version ARN to its local code path.* Used by the container launcher to copy layer content into /opt.diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.javaindex b6dfb74aec..b1dd4b13a6 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/LambdaService.java@@ -1524,6 +1524,25 @@ public class LambdaService {return "snapshots/" + account + "/" + fn.getFunctionName();}+ /** Stable, account-scoped S3 key for a published layer version's archive. */+ public static String layerObjectKey(String accountId, String layerName, long version) {+ var account = accountId != null ? accountId : "000000000000";+ return "layers/" + account + "/" + layerName + "/" + version;+ }++ /** Percent-encodes each path segment of a bucket path or object key for use in a URL. */+ public static String encodeObjectPath(String path) {+ var encoded = new StringBuilder();+ for (var segment : path.split("/", -1)) {+ if (!encoded.isEmpty()) {+ encoded.append('/');+ }+ encoded.append(java.net.URLEncoder.encode(segment, java.nio.charset.StandardCharsets.UTF_8)+ .replace("+", "%20"));+ }+ return encoded.toString();+ }+private void extractZipCode(LambdaFunction fn, String zipFileBase64, String region) {extractZipCodeBytes(fn, Base64.getDecoder().decode(zipFileBase64), region);}@@ -1575,8 +1594,25 @@ public class LambdaService {}private void storeDeploymentPackage(LambdaFunction fn, byte[] zipBytes, String region) {+ boolean stored = putTasksObjectQuietly(s3Service, region, codeObjectKey(fn), zipBytes,+ "deployment package for " + fn.getFunctionName());+ if (!stored && requiresStoredTasksObject(config)) {+ throw new AwsException("ServiceException",+ "Could not store the deployment package for '" + fn.getFunctionName()+ + "' in Floci's S3, which the kubernetes Lambda executor needs to "+ + "launch pods. The function was not deployed.", 500);+ }+ }++ /**+ * Best-effort put into the per-region tasks bucket, shared by function-code and+ * layer-archive storage. Returns whether the object was stored so a caller whose+ * executor depends on it can fail loudly instead of silently succeeding.+ */+ static boolean putTasksObjectQuietly(S3Service s3Service, String region, String key,+ byte[] zipBytes, String what) {if (s3Service == null) {- return;+ return false;}String bucket = tasksBucketName(region);try {@@ -1588,15 +1624,24 @@ public class LambdaService {throw e;}}- s3Service.putObject(bucket, codeObjectKey(fn), zipBytes,- "application/zip", java.util.Map.of());+ s3Service.putObject(bucket, key, zipBytes, "application/zip", java.util.Map.of());+ return true;} catch (Exception e) {- // Never fail a deploy because the convenience copy failed.- LOG.warnv("Could not store deployment package for {0}: {1}",- fn.getFunctionName(), e.getMessage());+ LOG.warnv("Could not store {0}: {1}", what, e.getMessage());+ return false;}}+ /**+ * Whether the tasks-bucket copy is load-bearing for the active executor. The+ * kubernetes executor's pods download code and layers from it, so a store failure+ * must fail the deploy rather than surface later as broken cold starts.+ */+ static boolean requiresStoredTasksObject(EmulatorConfig config) {+ return config != null+ && "kubernetes".equalsIgnoreCase(config.services().lambda().executor().trim());+ }+private void extractZipCodeFromS3(LambdaFunction fn, String s3Bucket, String s3Key, String region) {if (s3Service == null) {throw new AwsException("ServiceUnavailableException", "S3 service not available", 503);diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/WarmPool.java b/src/main/java/io/github/hectorvent/floci/services/lambda/WarmPool.javaindex 930fcc3190..03f1b4bdfe 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambda/WarmPool.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/WarmPool.java@@ -3,7 +3,7 @@ package io.github.hectorvent.floci.services.lambda;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.ContainerTeardown;import io.github.hectorvent.floci.services.lambda.launcher.ContainerHandle;-import io.github.hectorvent.floci.services.lambda.launcher.ContainerLauncher;+import io.github.hectorvent.floci.services.lambda.launcher.LambdaRuntimeLauncher;import io.github.hectorvent.floci.services.lambda.model.ContainerState;import io.github.hectorvent.floci.services.lambda.model.LambdaFunction;import jakarta.annotation.PostConstruct;@@ -40,7 +40,7 @@ public class WarmPool implements ContainerTeardown {private static final int DEFAULT_MAX_POOL_SIZE = Math.max(4, Runtime.getRuntime().availableProcessors());- private final ContainerLauncher containerLauncher;+ private final LambdaRuntimeLauncher lambdaRuntimeLauncher;private final EmulatorConfig config;private final int maxPoolSizePerFunction;private final ConcurrentHashMap<String, PoolState> poolStates = new ConcurrentHashMap<>();@@ -57,15 +57,15 @@ public class WarmPool implements ContainerTeardown {}@Inject- public WarmPool(ContainerLauncher containerLauncher, EmulatorConfig config) {- this.containerLauncher = containerLauncher;+ public WarmPool(LambdaRuntimeLauncher lambdaRuntimeLauncher, EmulatorConfig config) {+ this.lambdaRuntimeLauncher = lambdaRuntimeLauncher;this.config = config;this.maxPoolSizePerFunction = DEFAULT_MAX_POOL_SIZE;}/** Package-private constructor for testing (empty pool, no containers to drain). */WarmPool() {- this.containerLauncher = null;+ this.lambdaRuntimeLauncher = null;this.config = null;this.maxPoolSizePerFunction = DEFAULT_MAX_POOL_SIZE;}@@ -144,7 +144,7 @@ public class WarmPool implements ContainerTeardown {stopQuietly(candidate);continue;}- if (containerLauncher.isAlive(candidate)) {+ if (lambdaRuntimeLauncher.isAlive(candidate)) {handle = candidate;break;}@@ -157,7 +157,7 @@ public class WarmPool implements ContainerTeardown {if (handle == null) {LOG.debugv(ephemeral ? "Ephemeral start for function: {0}" : "Cold start for function: {0}",fn.getFunctionName());- handle = containerLauncher.launch(fn);+ handle = lambdaRuntimeLauncher.launch(fn);} else {LOG.debugv("Reusing warm container for function: {0}", fn.getFunctionName());}@@ -367,7 +367,7 @@ public class WarmPool implements ContainerTeardown {private void stopQuietly(ContainerHandle handle) {try {- containerLauncher.stop(handle);+ lambdaRuntimeLauncher.stop(handle);} catch (Exception e) {LOG.warnv("Error stopping container {0}: {1}", handle.getContainerId(), e.getMessage());}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncher.java b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncher.javaindex f4412e0834..97f20b4aa1 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncher.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/ContainerLauncher.java@@ -22,6 +22,7 @@ import com.github.dockerjava.api.exception.NotFoundException;import jakarta.annotation.PostConstruct;import jakarta.annotation.PreDestroy;import jakarta.enterprise.context.ApplicationScoped;+import jakarta.enterprise.inject.Typed;import jakarta.inject.Inject;import org.jboss.logging.Logger;@@ -48,6 +49,7 @@ import java.util.ArrayList;import java.util.HexFormat;import java.util.List;import java.util.Optional;+import java.util.concurrent.CancellationException;import java.util.concurrent.ExecutionException;import java.util.concurrent.TimeUnit;import java.util.concurrent.TimeoutException;@@ -62,7 +64,8 @@ import java.util.concurrent.atomic.AtomicReference;* rather than a bind mount, so it works when Floci itself runs inside Docker.*/@ApplicationScoped-public class ContainerLauncher {+@Typed(ContainerLauncher.class)+public class ContainerLauncher implements LambdaRuntimeLauncher {private static final Logger LOG = Logger.getLogger(ContainerLauncher.class);private static final String TASK_DIR = "/var/task";@@ -101,7 +104,8 @@ public class ContainerLauncher {*/private static final String FLOCI_CA_DIR = "/etc";private static final String FLOCI_CA_FILE_NAME = "floci-ca.crt";- private static final String FLOCI_CA_CONTAINER_PATH = FLOCI_CA_DIR + "/" + FLOCI_CA_FILE_NAME;+ /** Shared with the kubernetes executor, which mounts the CA ConfigMap at the same path. */+ public static final String FLOCI_CA_CONTAINER_PATH = FLOCI_CA_DIR + "/" + FLOCI_CA_FILE_NAME;/** Self-signed cert filename produced by {@code TlsConfigSource} under {persistent-path}/tls/. */private static final String SELF_SIGNED_CERT_NAME = "floci-selfsigned.crt";@@ -502,6 +506,16 @@ public class ContainerLauncher {// null by then); otherwise the failure happened before Docker ever saw the volume, so// its in-flight reference must be released here or cleanup would wait on it forever.releaseCodeVolumeReference(reservedCodeVolume);+ // Stop the server before releasing its port, or the still-listening Vert.x server+ // makes a later cold start that reuses the port serve this runtime too.+ try {+ runtimeApiServer.stop().get(5, TimeUnit.SECONDS);+ } catch (InterruptedException ie) {+ Thread.currentThread().interrupt();+ } catch (Exception stopFailure) {+ LOG.debugv("Runtime API server stop failed during launch cleanup: {0}",+ stopFailure.getMessage());+ }try {runtimeApiServerFactory.release(runtimeApiServer);} catch (Exception cleanupError) {@@ -547,7 +561,9 @@ public class ContainerLauncher {server.close().get(5, TimeUnit.SECONDS);} catch (InterruptedException e) {Thread.currentThread().interrupt();- } catch (ExecutionException | TimeoutException e) {+ } catch (ExecutionException | TimeoutException | CancellationException e) {+ // CancellationException is unchecked; catch it so a cancelled close future+ // is logged here instead of propagating out of stop().LOG.warnv(e, "RuntimeApiServer did not close cleanly for container {0}",handle.getContainerId());} finally {@@ -1307,8 +1323,8 @@ public class ContainerLauncher {* {@code floci.tls.cert-path} that points at a leaf/server certificate is accepted but only pins* that exact certificate (it cannot validate a chain it signs), so a warning is logged.*/- static Optional<Path> resolveFlociCaCertPath(boolean tlsEnabled, Optional<String> userCertPath,- String persistentPath) {+ public static Optional<Path> resolveFlociCaCertPath(boolean tlsEnabled, Optional<String> userCertPath,+ String persistentPath) {if (!tlsEnabled) {return Optional.empty();}@@ -1364,7 +1380,7 @@ public class ContainerLauncher {* which breaks every external HTTPS call (curl, openssl, requests/botocore) the Lambda makes.* Returns an empty list when no CA cert is available (TLS off).*/- static List<String> flociCaEnv(Optional<Path> caCert) {+ public static List<String> flociCaEnv(Optional<Path> caCert) {if (caCert.isEmpty()) {return List.of();}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncher.java b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncher.javanew file mode 100644index 0000000000..feaccc01a6--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncher.java@@ -0,0 +1,18 @@+package io.github.hectorvent.floci.services.lambda.launcher;++import io.github.hectorvent.floci.services.lambda.model.LambdaFunction;++/**+ * Starts and stops Lambda execution environments. Implementations back each+ * environment with a concrete runtime: a Docker container ({@link ContainerLauncher})+ * or a Kubernetes pod ({@code KubernetesPodLauncher}). Selected at startup by+ * {@code floci.services.lambda.executor}.+ */+public interface LambdaRuntimeLauncher {++ ContainerHandle launch(LambdaFunction fn);++ void stop(ContainerHandle handle);++ boolean isAlive(ContainerHandle handle);+}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncherProducer.java b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncherProducer.javanew file mode 100644index 0000000000..73929c88f4--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncherProducer.java@@ -0,0 +1,63 @@+package io.github.hectorvent.floci.services.lambda.launcher;++import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.services.lambda.launcher.kubernetes.KubernetesPodLauncher;+import io.quarkus.runtime.StartupEvent;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.enterprise.event.Observes;+import jakarta.enterprise.inject.Instance;+import jakarta.enterprise.inject.Produces;+import org.jboss.logging.Logger;++import java.util.Locale;++/**+ * Selects the Lambda execution backend at startup. {@code Instance<>} lookups keep+ * the unselected launcher (and its Docker/Kubernetes client) uninstantiated, so a+ * Docker-less cluster deployment never touches docker.sock and vice versa.+ */+@ApplicationScoped+public class LambdaRuntimeLauncherProducer {++ private static final Logger LOG = Logger.getLogger(LambdaRuntimeLauncherProducer.class);++ /**+ * The producer itself is lazy, so a bad value would otherwise surface only on the+ * first invocation. This makes a misconfigured executor fail startup instead.+ *+ * <p>For the kubernetes executor the client is also initialized here, on the+ * startup thread. Left to the first cold start, the Fabric8 client's Vert.x+ * HTTP client would be created on a request's Vert.x context and closed with+ * the HTTP server verticles at shutdown, so the WarmPool drain could no longer+ * delete pods ("Client is closed").+ */+ void validateExecutor(@Observes StartupEvent event, EmulatorConfig config,+ Instance<KubernetesPodLauncher> kubernetes) {+ if (requireValidExecutor(config).equals("kubernetes")) {+ kubernetes.get().initializeClient();+ }+ }++ @Produces+ @ApplicationScoped+ LambdaRuntimeLauncher launcher(EmulatorConfig config,+ Instance<ContainerLauncher> docker,+ Instance<KubernetesPodLauncher> kubernetes) {+ var executor = requireValidExecutor(config);+ if (executor.equals("kubernetes")) {+ LOG.info("Lambda executor: kubernetes (environments run as pods)");+ return kubernetes.get();+ }+ return docker.get();+ }++ private static String requireValidExecutor(EmulatorConfig config) {+ var executor = config.services().lambda().executor().trim().toLowerCase(Locale.ROOT);+ if (!executor.equals("docker") && !executor.equals("kubernetes")) {+ throw new IllegalArgumentException(+ "Unknown floci.services.lambda.executor '" + executor+ + "'. Valid values: docker, kubernetes");+ }+ return executor;+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesFlociAddressResolver.java b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesFlociAddressResolver.javanew file mode 100644index 0000000000..a20aaa55f0--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesFlociAddressResolver.java@@ -0,0 +1,148 @@+package io.github.hectorvent.floci.services.lambda.launcher.kubernetes;++import io.github.hectorvent.floci.config.EmulatorConfig;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.inject.Inject;+import org.jboss.logging.Logger;++import java.net.Inet4Address;+import java.net.InetAddress;+import java.net.NetworkInterface;+import java.nio.file.Files;+import java.nio.file.Path;++/**+ * Resolves the address Lambda pods use to reach the Floci process: the Runtime API+ * port range (runtime-api-base-port..max-port) and the main emulator port. Pods talk+ * straight to this address; there is no Docker-style host gateway or embedded DNS,+ * so a wrong value surfaces only as {@code Function.TimedOut}. That is why an+ * unresolvable address fails fast instead of guessing.+ */+@ApplicationScoped+public class KubernetesFlociAddressResolver {++ private static final Logger LOG = Logger.getLogger(KubernetesFlociAddressResolver.class);+ private static final Path SERVICE_ACCOUNT_TOKEN =+ Path.of("/var/run/secrets/kubernetes.io/serviceaccount/token");++ private final EmulatorConfig config;+ private String cachedAddress;++ @Inject+ public KubernetesFlociAddressResolver(EmulatorConfig config) {+ this.config = config;+ }++ /** Host or IP only, e.g. {@code 10.42.0.7}. */+ public synchronized String resolve() {+ if (cachedAddress == null) {+ cachedAddress = doResolve();+ // Ports go in as strings: MessageFormat would render 9200 as "9,200".+ LOG.infov("Lambda pods will reach Floci at {0} (runtime API ports {1}-{2}, main port {3})",+ cachedAddress,+ Integer.toString(config.services().lambda().runtimeApiBasePort()),+ Integer.toString(config.services().lambda().runtimeApiMaxPort()),+ Integer.toString(mainPort()));+ }+ return cachedAddress;+ }++ /**+ * Base URL for FLOCI_ENDPOINT / AWS_ENDPOINT_URL inside the pod. Built from+ * {@link #resolve()} rather than {@code ContainerReachableEndpoint}, which is+ * Docker/embedded-DNS specific.+ */+ public String flociBaseUrl() {+ var scheme = config.tls().enabled() ? "https" : "http";+ return scheme + "://" + resolve() + ":" + mainPort();+ }++ /**+ * Base URL for init-container code and layer downloads: always plain HTTP.+ * busybox wget's built-in TLS cannot complete a handshake with Floci's TLS+ * stack, and Floci's TLS proxy serves both protocols on the emulator port.+ * The runtime API traffic on this same pod network is plain HTTP too, so+ * TLS here would add no trust the executor doesn't already assume.+ */+ public String downloadBaseUrl() {+ return "http://" + resolve() + ":" + mainPort();+ }++ private int mainPort() {+ // The port Floci actually listens on. base-url may describe a proxied+ // client-facing endpoint; pods talk to this process directly.+ return config.port();+ }++ private String doResolve() {+ var override = config.services().lambda().kubernetes().flociAddress()+ .filter(s -> !s.isBlank());+ if (override.isPresent()) {+ var address = override.get().trim();+ // Reachability can only be judged from the pods' network, not from here,+ // but anything beyond hostname/IP characters is always a mistake worth+ // failing on: the value is embedded in URLs and in single-quoted+ // init-container shell commands.+ if (!address.matches("[A-Za-z0-9.-]+")) {+ throw new IllegalStateException(+ "floci.services.lambda.kubernetes.floci-address must be a bare host or IP "+ + "(no scheme, port or path), got '" + address + "'. Ports are "+ + "derived from floci.port and the runtime-api port range.");+ }+ return address;+ }+ if (!isRunningInCluster()) {+ throw new IllegalStateException(+ "The kubernetes Lambda executor needs an address that pods can use to reach "+ + "Floci, and Floci is not running inside the cluster. Set "+ + "FLOCI_SERVICES_LAMBDA_KUBERNETES_FLOCI_ADDRESS "+ + "(floci.services.lambda.kubernetes.floci-address) to a host/IP "+ + "reachable from the cluster's pods, e.g. this machine's LAN IP.");+ }+ return ownPodAddress();+ }++ private boolean isRunningInCluster() {+ return System.getenv("KUBERNETES_SERVICE_HOST") != null+ || Files.exists(SERVICE_ACCOUNT_TOKEN);+ }++ /**+ * Inside a pod the hostname resolves to the pod IP, so getLocalHost is authoritative.+ * The interface scan is a fallback for images with an unresolvable hostname.+ */+ private String ownPodAddress() {+ // IPv4 only: an IPv6 literal would need bracketing in every URL this address+ // feeds, and busybox wget cannot fetch from unbracketed IPv6 hosts.+ try {+ var local = InetAddress.getLocalHost();+ if (local instanceof Inet4Address && !local.isLoopbackAddress()) {+ return local.getHostAddress();+ }+ } catch (Exception e) {+ LOG.debugv("getLocalHost failed, scanning interfaces: {0}", e.getMessage());+ }+ try {+ var interfaces = NetworkInterface.getNetworkInterfaces();+ while (interfaces.hasMoreElements()) {+ var nic = interfaces.nextElement();+ if (!nic.isUp() || nic.isLoopback()) {+ continue;+ }+ var addresses = nic.getInetAddresses();+ while (addresses.hasMoreElements()) {+ var address = addresses.nextElement();+ if (address instanceof Inet4Address && !address.isLoopbackAddress()) {+ return address.getHostAddress();+ }+ }+ }+ } catch (Exception e) {+ LOG.warnv("Interface scan for pod IP failed: {0}", e.getMessage());+ }+ throw new IllegalStateException(+ "Could not determine an IPv4 address of this pod for Lambda pods to connect "+ + "back to (IPv6-only clusters are not supported yet). Set "+ + "FLOCI_SERVICES_LAMBDA_KUBERNETES_FLOCI_ADDRESS explicitly.");+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesPodLauncher.java b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesPodLauncher.javanew file mode 100644index 0000000000..79c593faf7--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesPodLauncher.java@@ -0,0 +1,578 @@+package io.github.hectorvent.floci.services.lambda.launcher.kubernetes;++import io.fabric8.kubernetes.api.model.ConfigMapBuilder;+import io.fabric8.kubernetes.api.model.ContainerStatus;+import io.fabric8.kubernetes.api.model.Pod;+import io.fabric8.kubernetes.client.KubernetesClient;+import io.fabric8.kubernetes.client.KubernetesClientException;+import io.fabric8.kubernetes.client.dsl.NonDeletingOperation;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsArnUtils;+import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;+import io.github.hectorvent.floci.services.lambda.LambdaLayerService;+import io.github.hectorvent.floci.services.lambda.LambdaService;+import io.github.hectorvent.floci.services.lambda.launcher.ContainerHandle;+import io.github.hectorvent.floci.services.lambda.launcher.ContainerLauncher;+import io.github.hectorvent.floci.services.lambda.launcher.ImageResolver;+import io.github.hectorvent.floci.services.lambda.launcher.LambdaRuntimeLauncher;+import io.github.hectorvent.floci.services.lambda.model.ContainerState;+import io.github.hectorvent.floci.services.lambda.model.LambdaFunction;+import io.github.hectorvent.floci.services.lambda.runtime.RuntimeApiServerFactory;+import io.github.hectorvent.floci.services.s3.S3Service;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.enterprise.inject.Typed;+import jakarta.inject.Inject;+import org.jboss.logging.Logger;++import java.nio.file.Files;+import java.nio.file.Path;+import java.util.ArrayList;+import java.util.List;+import java.util.Optional;+import java.util.Set;+import java.util.UUID;+import java.util.concurrent.CancellationException;+import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.ExecutionException;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.TimeoutException;+import java.util.concurrent.atomic.AtomicBoolean;++/**+ * Runs each Lambda execution environment as a Kubernetes pod. Mirrors+ * {@link ContainerLauncher}'s contract: the Runtime API server is started before the+ * pod so the runtime connects on boot, and any launch failure after the server is+ * allocated releases its port and reaps the half-built pod.+ *+ * <p>Pods reach Floci over the network only (no bind mounts, no host gateway):+ * an init container downloads the deployment package from Floci's S3, and the+ * runtime polls the Runtime API at the address {@link KubernetesFlociAddressResolver}+ * advertises. Hot reload is therefore unsupported here.+ */+@ApplicationScoped+@Typed(KubernetesPodLauncher.class)+public class KubernetesPodLauncher implements LambdaRuntimeLauncher {++ private static final Logger LOG = Logger.getLogger(KubernetesPodLauncher.class);++ static final String CA_CONFIG_MAP_NAME = "floci-lambda-ca";+ static final String CA_CONFIG_MAP_KEY = "ca.crt";++ /**+ * Generous enough for a node's first pull of a multi-hundred-MB runtime image;+ * a pod that will never run (bad image, failing init) is detected much earlier+ * via its terminal waiting/terminated states.+ */+ private static final int POD_STARTUP_TIMEOUT_SECONDS = 300;++ /**+ * How long a stop waits for a deleted pod to actually disappear before giving up+ * and keeping its Runtime API port reserved. A force delete normally clears within+ * a second; the cap only bites when the API server or kubelet is stuck.+ */+ private static final int POD_DELETE_TIMEOUT_SECONDS = 15;++ /**+ * Waiting-state reasons treated as fatal for the cold start. ErrImagePull is+ * deliberately absent: it appears on the first failed pull attempt, which the+ * kubelet retries; only the backoff state marks repeated failures.+ */+ private static final Set<String> TERMINAL_WAITING_REASONS = Set.of(+ "ImagePullBackOff", "InvalidImageName", "CrashLoopBackOff",+ "CreateContainerError", "CreateContainerConfigError", "RunContainerError");++ private final KubernetesClient client;+ private final EmulatorConfig config;+ private final RuntimeApiServerFactory runtimeApiServerFactory;+ private final ImageResolver imageResolver;+ private final KubernetesFlociAddressResolver addressResolver;+ private final LaunchedContainerAwsEnv awsEnv;+ private final LambdaLayerService layerService;+ private final LambdaPodSpecFactory podSpecFactory;+ private final KubernetesPodLogStreamer logStreamer;+ private final S3Service s3Service;++ private final Object orphanSweepLock = new Object();+ private volatile boolean orphansSwept = false;+ /** Names of pods created by this process, so a retried orphan sweep never deletes them. */+ private final Set<String> ownPodNames = ConcurrentHashMap.newKeySet();+ private final AtomicBoolean caConfigMapApplied = new AtomicBoolean(false);+ private final AtomicBoolean awsConfigPathWarned = new AtomicBoolean(false);++ @Inject+ public KubernetesPodLauncher(KubernetesClient client,+ EmulatorConfig config,+ RuntimeApiServerFactory runtimeApiServerFactory,+ ImageResolver imageResolver,+ KubernetesFlociAddressResolver addressResolver,+ LaunchedContainerAwsEnv awsEnv,+ LambdaLayerService layerService,+ LambdaPodSpecFactory podSpecFactory,+ KubernetesPodLogStreamer logStreamer,+ S3Service s3Service) {+ this.client = client;+ this.config = config;+ this.runtimeApiServerFactory = runtimeApiServerFactory;+ this.imageResolver = imageResolver;+ this.addressResolver = addressResolver;+ this.awsEnv = awsEnv;+ this.layerService = layerService;+ this.podSpecFactory = podSpecFactory;+ this.logStreamer = logStreamer;+ this.s3Service = s3Service;+ }++ /**+ * Forces the lazy client bean (and its underlying Vert.x HTTP client) to be+ * built on the caller's thread. Called from the startup observer so the HTTP+ * client is not born on a request's Vert.x context, whose close at shutdown+ * would leave every later Kubernetes API call failing with "Client is closed".+ */+ public void initializeClient() {+ client.getConfiguration();+ // Resolve the address pods use to reach Floci now, so the most common+ // misconfiguration (running outside the cluster without a floci-address)+ // fails startup instead of every later cold start.+ addressResolver.resolve();+ }++ @Override+ public ContainerHandle launch(LambdaFunction fn) {+ if (fn.isHotReload()) {+ throw new RuntimeException("Hot reload requires a bind mount and is not supported by the "+ + "kubernetes Lambda executor. Use the docker executor for hot-reload functions.");+ }+ if (config.services().lambda().awsConfigPath().filter(s -> !s.isBlank()).isPresent()+ && awsConfigPathWarned.compareAndSet(false, true)) {+ LOG.warn("floci.services.lambda.aws-config-path is a bind mount and is ignored by the "+ + "kubernetes executor; pods receive placeholder credentials instead");+ }++ var namespace = namespace();+ sweepOrphansOnce(namespace);+ LOG.infov("Launching pod for function: {0}", fn.getFunctionName());++ var runtimeApiServer = runtimeApiServerFactory.create();++ // Mirrors ContainerLauncher: any failure after the runtime-api server is allocated+ // must release its port and delete a half-created pod, or cold-start bursts leak+ // ports until the pool is exhausted.+ String podName = null;+ try {+ var imagePackage = "Image".equals(fn.getPackageType()) && fn.getImageUri() != null;+ // No emulated-ECR rewrite here: the kubelet pulls images, and Floci's loopback+ // registry is not reachable from cluster nodes. URIs pass through unchanged.+ var image = imagePackage ? fn.getImageUri() : imageResolver.resolve(fn.getRuntime());++ var shortId = UUID.randomUUID().toString().replace("-", "").substring(0, 8);+ podName = LambdaPodSpecFactory.podName(fn.getFunctionName(), shortId);+ var region = AwsArnUtils.regionOrDefault(fn.getFunctionArn(), config.defaultRegion());+ var cwLogGroup = "/aws/lambda/" + fn.getFunctionName();+ var cwLogStream = logStreamer.logStreamName(shortId);++ var codeDownloadUrl = imagePackage ? null : codeDownloadUrl(fn, region);+ var layerUrls = imagePackage ? List.<String>of() : layerDownloadUrls(fn, region);++ var caCert = config.tls().enabled()+ ? ContainerLauncher.resolveFlociCaCertPath(true, config.tls().certPath(),+ config.storage().persistentPath())+ : Optional.<Path>empty();+ var caConfigMap = caCert.map(cert -> ensureCaConfigMap(namespace, cert));++ var env = new ArrayList<String>();+ env.add("AWS_LAMBDA_RUNTIME_API=" + addressResolver.resolve() + ":" + runtimeApiServer.getPort());+ env.add("AWS_LAMBDA_FUNCTION_NAME=" + fn.getFunctionName());+ env.add("AWS_LAMBDA_FUNCTION_MEMORY_SIZE=" + fn.getMemorySize());+ env.add("AWS_LAMBDA_FUNCTION_TIMEOUT=" + fn.getTimeout());+ env.add("AWS_LAMBDA_FUNCTION_VERSION=$LATEST");+ env.add("AWS_LAMBDA_LOG_GROUP_NAME=" + cwLogGroup);+ env.add("AWS_LAMBDA_LOG_STREAM_NAME=" + cwLogStream);+ if (fn.getHandler() != null && !fn.getHandler().isBlank()) {+ env.add("_HANDLER=" + fn.getHandler());+ }+ env.addAll(awsEnv.sdkBaselineEnv(region, Optional.empty(), addressResolver.flociBaseUrl()));+ env.addAll(ContainerLauncher.flociCaEnv(caCert));+ if (fn.getEnvironment() != null) {+ fn.getEnvironment().forEach((k, v) -> env.add(k + "=" + v));+ }++ var imageConfig = imagePackage+ ? new LambdaPodSpecFactory.ImageConfig(fn.getImageConfigEntryPoint(),+ fn.getImageConfigCommand(), fn.getImageConfigWorkingDirectory())+ : null;++ var pod = podSpecFactory.buildPod(podName, fn.getFunctionName(), image, env,+ codeDownloadUrl, layerUrls, isProvidedRuntime(fn.getRuntime()),+ imagePackage ? null : fn.getHandler(), imageConfig, fn.getMemorySize(), caConfigMap);++ ownPodNames.add(podName);+ client.pods().inNamespace(namespace).resource(pod).create();+ LOG.infov("Created pod {0} for function {1}", podName, fn.getFunctionName());++ awaitRunning(namespace, podName, fn.getFunctionName());++ var handle = new ContainerHandle(podName, fn.getFunctionName(),+ runtimeApiServer, ContainerState.WARM, false);+ // Log streaming is non-essential and runs after the pod is already Running,+ // so a failure here (e.g. a missing pods/log RBAC verb) must not tear down a+ // healthy execution environment.+ try {+ var logHandle = logStreamer.attach(namespace, podName, cwLogGroup, cwLogStream,+ region, "lambda:" + fn.getFunctionName());+ handle.setLogStream(logHandle);+ } catch (Exception logFailure) {+ LOG.warnv("Log streaming for pod {0} could not start; the environment still "+ + "runs but its logs will not reach CloudWatch: {1}", podName,+ logFailure.getMessage());+ }+ return handle;+ } catch (RuntimeException e) {+ LOG.errorv(e, "Pod launch failed for function {0}; cleaning up", fn.getFunctionName());+ boolean podGone = podName == null || deletePod(namespace, podName);+ // Stop the server before releasing its port number, or the still-listening+ // Vert.x server makes the port unusable for every later cold start.+ try {+ runtimeApiServer.stop().get(5, TimeUnit.SECONDS);+ } catch (InterruptedException ie) {+ Thread.currentThread().interrupt();+ } catch (Exception stopFailure) {+ LOG.debugv("Runtime API server stop failed during launch cleanup: {0}",+ stopFailure.getMessage());+ }+ // Same rule as stop(): the port number goes back to the pool only once the pod+ // is confirmed gone. A half-created pod that later reaches Running polls+ // AWS_LAMBDA_RUNTIME_API on this port, and a released port could by then be+ // serving a different environment.+ if (podGone) {+ try {+ runtimeApiServerFactory.release(runtimeApiServer);+ } catch (Exception ignore) {+ LOG.debugv("Runtime API port release failed during launch cleanup: {0}", ignore.getMessage());+ }+ } else {+ LOG.warnv("Keeping the Runtime API port reserved for pod {0} because its delete "+ + "did not succeed.", podName);+ }+ throw e;+ }+ }++ @Override+ public void stop(ContainerHandle handle) {+ LOG.infov("Stopping pod {0}", handle.getContainerId());+ handle.setState(ContainerState.STOPPED);+ try {+ handle.getRuntimeApiServer().stop().get(5, TimeUnit.SECONDS);+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ } catch (ExecutionException | TimeoutException | CancellationException e) {+ // CancellationException is unchecked; catch it here so a cancelled stop future+ // cannot skip the log-stream close, pod deletion, and port release below.+ LOG.warnv(e, "RuntimeApiServer did not close cleanly for pod {0}", handle.getContainerId());+ }+ if (handle.getLogStream() != null) {+ try {+ handle.getLogStream().close();+ } catch (Exception e) {+ LOG.debugv("Closing log stream for pod {0} failed: {1}",+ handle.getContainerId(), e.getMessage());+ }+ }+ // Return the Runtime API port to the pool only once the pod is gone. If the delete+ // did not succeed, the pod's runtime may still be polling that port, and reusing it+ // for another cold start could hand that runtime a different function's invocation,+ // so keep the port reserved until a later sweep reaps the pod.+ if (deletePod(namespace(), handle.getContainerId())) {+ try {+ runtimeApiServerFactory.release(handle.getRuntimeApiServer());+ } catch (Exception e) {+ LOG.warnv("Runtime API port release failed for pod {0}: {1}",+ handle.getContainerId(), e.getMessage());+ }+ } else {+ LOG.warnv("Keeping the Runtime API port reserved for pod {0} because its delete "+ + "did not succeed.", handle.getContainerId());+ }+ }++ @Override+ public boolean isAlive(ContainerHandle handle) {+ try {+ var pod = client.pods().inNamespace(namespace()).withName(handle.getContainerId()).get();+ return pod != null+ && pod.getStatus() != null+ && "Running".equals(pod.getStatus().getPhase())+ && pod.getMetadata().getDeletionTimestamp() == null;+ } catch (Exception e) {+ // A missing pod comes back as null above; reaching here means the API server+ // itself was unreachable. Unlike docker's local socket the API server is remote,+ // so a transient blip must not read as "dead" — that would cull the whole warm+ // pool at once. Assume alive; a genuinely dead pod fails the next invocation.+ LOG.warnv("Liveness probe for pod {0} could not reach the API server, assuming "+ + "alive: {1}", handle.getContainerId(), e.getMessage());+ return true;+ }+ }++ private String namespace() {+ return config.services().lambda().kubernetes().namespace();+ }++ /**+ * Deletes pods left behind by a previous Floci process. They can never be adopted:+ * their runtimes poll Runtime API ports that died with that process.+ *+ * <p>Every launch blocks here until one sweep has succeeded; a failed sweep is+ * retried by the next launch. Launches proceed after a failed sweep and their+ * pods carry the same labels, so a retried sweep skips every name in+ * {@link #ownPodNames} and deletion targets the listed pod names rather than+ * the label selector. Both are required so a retried or concurrent sweep can+ * never kill a fresh pod of this process.+ */+ private void sweepOrphansOnce(String namespace) {+ if (orphansSwept) {+ return;+ }+ synchronized (orphanSweepLock) {+ if (orphansSwept) {+ return;+ }+ try {+ var orphans = client.pods().inNamespace(namespace)+ .withLabels(LambdaPodSpecFactory.managedPodSelector())+ .list().getItems().stream()+ .filter(pod -> !ownPodNames.contains(pod.getMetadata().getName()))+ .toList();+ if (!orphans.isEmpty()) {+ LOG.infov("Deleting {0} orphaned Lambda pod(s) from a previous run in namespace {1}",+ orphans.size(), namespace);+ for (var orphan : orphans) {+ client.pods().inNamespace(namespace)+ .withName(orphan.getMetadata().getName())+ .withGracePeriod(0)+ .delete();+ }+ }+ orphansSwept = true;+ } catch (KubernetesClientException e) {+ if (e.getCode() == 401 || e.getCode() == 403) {+ // A permission gap never heals within this process, so stop retrying+ // it on every cold start; the ServiceAccount simply cannot sweep.+ LOG.warnv("Orphaned Lambda pod sweep is not permitted in namespace {0} "+ + "(the ServiceAccount lacks pod list/delete); skipping it for this "+ + "process: {1}", namespace, e.getMessage());+ orphansSwept = true;+ } else {+ LOG.warnv("Orphaned Lambda pod sweep failed in namespace {0} and will be "+ + "retried on the next launch: {1}", namespace, e.getMessage());+ }+ } catch (Exception e) {+ LOG.warnv("Orphaned Lambda pod sweep failed in namespace {0} and will be retried "+ + "on the next launch: {1}", namespace, e.getMessage());+ }+ }+ }++ private String codeDownloadUrl(LambdaFunction fn, String region) {+ var bucket = LambdaService.tasksBucketName(region);+ var key = LambdaService.codeObjectKey(fn);+ try {+ s3Service.headObject(bucket, key);+ } catch (AwsException e) {+ throw new RuntimeException("Deployment package for function '" + fn.getFunctionName()+ + "' is not available at s3://" + bucket + "/" + key+ + " — the kubernetes executor downloads code from Floci's S3, and the copy "+ + "stored at deploy time is missing. Re-deploy the function code.", e);+ }+ var account = fn.getAccountId() != null ? fn.getAccountId() : config.defaultAccountId();+ return downloadUrl(bucket, key, account, region);+ }++ private List<String> layerDownloadUrls(LambdaFunction fn, String functionRegion) {+ if (fn.getLayers() == null || fn.getLayers().isEmpty()) {+ return List.of();+ }+ var urls = new ArrayList<String>();+ for (var layerArn : fn.getLayers()) {+ var layer = layerService.resolveLayerByArn(layerArn);+ if (layer == null) {+ LOG.warnv("Could not resolve layer ARN {0} for function {1}", layerArn, fn.getFunctionName());+ continue;+ }+ // The archive lives under the layer's own region and account (the canonical+ // ARN assigned at publish), which may differ from the function's.+ var arn = layer.getLayerVersionArn();+ var layerRegion = AwsArnUtils.regionOrDefault(arn, functionRegion);+ var account = AwsArnUtils.accountOrDefault(arn, config.defaultAccountId());+ var bucket = LambdaService.tasksBucketName(layerRegion);+ var key = LambdaService.layerObjectKey(account, layer.getLayerName(), layer.getVersion());+ try {+ s3Service.headObject(bucket, key);+ } catch (AwsException e) {+ throw new RuntimeException("Layer zip for '" + layerArn + "' is not available at s3://"+ + bucket + "/" + key + " — layers published before kubernetes executor support "+ + "have no stored archive. Re-publish the layer version.", e);+ }+ urls.add(downloadUrl(bucket, key, account, layerRegion));+ }+ return urls;+ }++ /**+ * Download URL for a tasks-bucket object. The init container fetches it over plain+ * HTTP with no SigV4 header, so an {@code X-Amz-Credential} query steers Floci's+ * account filter to the object's owning account. Without it a function or layer+ * owned by a non-default account resolves the default-account prefix and 404s. A+ * 12-digit account id doubles as its own access key id (LocalStack convention).+ */+ private String downloadUrl(String bucket, String key, String account, String region) {+ return addressResolver.downloadBaseUrl() + "/" + bucket + "/"+ + LambdaService.encodeObjectPath(key)+ + "?X-Amz-Credential=" + account + "%2F00010101%2F" + region + "%2Fs3%2Faws4_request";+ }++ private void awaitRunning(String namespace, String podName, String functionName) {+ var timeoutSeconds = POD_STARTUP_TIMEOUT_SECONDS;+ Pod pod;+ try {+ // A null pod (deleted out-of-band) is terminal too, otherwise the wait+ // would block the invocation for the full timeout.+ pod = client.pods().inNamespace(namespace).withName(podName)+ .waitUntilCondition(p -> p == null+ || (p.getStatus() != null && (isRunning(p) || hasTerminalFailure(p))),+ timeoutSeconds, TimeUnit.SECONDS);+ } catch (Exception e) {+ throw new RuntimeException("Pod " + podName + " for function '" + functionName+ + "' did not reach Running within " + timeoutSeconds + "s: " + e.getMessage(), e);+ }+ if (pod == null || !isRunning(pod)) {+ throw new RuntimeException("Pod " + podName + " for function '" + functionName+ + "' failed to start: " + describeFailure(pod));+ }+ }++ private static boolean isRunning(Pod pod) {+ return "Running".equals(pod.getStatus().getPhase());+ }++ private static boolean hasTerminalFailure(Pod pod) {+ var phase = pod.getStatus().getPhase();+ // Succeeded means the runtime exited before serving — terminal for a server pod.+ if ("Failed".equals(phase) || "Succeeded".equals(phase)) {+ return true;+ }+ return unschedulableReason(pod) != null || describeTerminalReason(pod) != null;+ }++ private static String unschedulableReason(Pod pod) {+ if (pod.getStatus().getConditions() == null) {+ return null;+ }+ for (var condition : pod.getStatus().getConditions()) {+ // A pod the scheduler cannot place — e.g. a memory request above every node's+ // capacity — stays Pending with no container statuses; fail the cold start+ // instead of blocking the invoker for the full startup timeout.+ if ("PodScheduled".equals(condition.getType()) && "False".equals(condition.getStatus())+ && "Unschedulable".equals(condition.getReason())) {+ return "Unschedulable: " + condition.getMessage();+ }+ }+ return null;+ }++ private static String describeTerminalReason(Pod pod) {+ var statuses = new ArrayList<ContainerStatus>();+ if (pod.getStatus().getInitContainerStatuses() != null) {+ statuses.addAll(pod.getStatus().getInitContainerStatuses());+ }+ if (pod.getStatus().getContainerStatuses() != null) {+ statuses.addAll(pod.getStatus().getContainerStatuses());+ }+ for (var status : statuses) {+ if (status.getState() == null) {+ continue;+ }+ var waiting = status.getState().getWaiting();+ // getReason() is null while a reason is absent; Set.of(...).contains(null)+ // throws, which would abort an otherwise healthy cold start.+ if (waiting != null && waiting.getReason() != null+ && TERMINAL_WAITING_REASONS.contains(waiting.getReason())) {+ return status.getName() + ": " + waiting.getReason()+ + " (" + waiting.getMessage() + ")";+ }+ if (status.getState().getTerminated() != null+ && status.getState().getTerminated().getExitCode() != null+ && status.getState().getTerminated().getExitCode() != 0) {+ return status.getName() + ": exited with code "+ + status.getState().getTerminated().getExitCode();+ }+ }+ return null;+ }++ private static String describeFailure(Pod pod) {+ if (pod == null || pod.getStatus() == null) {+ return "pod no longer exists";+ }+ var reason = describeTerminalReason(pod);+ if (reason == null) {+ reason = unschedulableReason(pod);+ }+ return reason != null ? reason : "phase=" + pod.getStatus().getPhase();+ }++ /**+ * Publishes Floci's CA cert as a ConfigMap so pods can trust Floci's HTTPS endpoint.+ * Applied once per process; the cert cannot rotate within a Floci lifetime.+ */+ private String ensureCaConfigMap(String namespace, Path caCertPath) {+ if (caConfigMapApplied.get()) {+ return CA_CONFIG_MAP_NAME;+ }+ try {+ var pem = Files.readString(caCertPath);+ var configMap = new ConfigMapBuilder()+ .withNewMetadata()+ .withName(CA_CONFIG_MAP_NAME)+ .withLabels(LambdaPodSpecFactory.managedPodSelector())+ .endMetadata()+ .addToData(CA_CONFIG_MAP_KEY, pem)+ .build();+ client.configMaps().inNamespace(namespace).resource(configMap)+ .createOr(NonDeletingOperation::update);+ caConfigMapApplied.set(true);+ return CA_CONFIG_MAP_NAME;+ } catch (Exception e) {+ throw new RuntimeException("Could not publish Floci CA cert ConfigMap '" + CA_CONFIG_MAP_NAME+ + "' in namespace " + namespace + ": " + e.getMessage(), e);+ }+ }++ /**+ * Force-deletes the pod and waits for it to actually disappear. Returns whether the+ * pod is confirmed gone: only then is it safe to reuse resources tied to it (its+ * Runtime API port), because until the object vanishes the runtime container may+ * still be polling that port. A rejected delete or a wait that times out returns+ * false so the caller keeps the port reserved.+ */+ private boolean deletePod(String namespace, String podName) {+ // Dropped from the own-pod set even when the delete fails: the pod is+ // abandoned either way, and a retried orphan sweep may still collect it.+ ownPodNames.remove(podName);+ try {+ client.pods().inNamespace(namespace).withName(podName).withGracePeriod(0).delete();+ client.pods().inNamespace(namespace).withName(podName)+ .waitUntilCondition(p -> p == null, POD_DELETE_TIMEOUT_SECONDS, TimeUnit.SECONDS);+ return true;+ } catch (Exception e) {+ LOG.warnv("Pod {0} was not confirmed deleted within {1}s: {2}",+ podName, POD_DELETE_TIMEOUT_SECONDS, e.getMessage());+ return false;+ }+ }++ private static boolean isProvidedRuntime(String runtime) {+ return runtime != null && runtime.startsWith("provided");+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesPodLogStreamer.java b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesPodLogStreamer.javanew file mode 100644index 0000000000..0539a61293--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesPodLogStreamer.java@@ -0,0 +1,78 @@+package io.github.hectorvent.floci.services.lambda.launcher.kubernetes;++import io.fabric8.kubernetes.client.KubernetesClient;+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.inject.Inject;+import org.jboss.logging.Logger;++import java.io.BufferedReader;+import java.io.Closeable;+import java.io.IOException;+import java.io.InputStreamReader;+import java.nio.charset.StandardCharsets;++/**+ * Streams a Lambda pod's runtime-container logs to CloudWatch Logs and the console,+ * matching what {@link ContainerLogStreamer} does for Docker containers. The CloudWatch+ * write path is delegated to {@code ContainerLogStreamer} so both executors produce+ * identical log groups, streams and events.+ */+@ApplicationScoped+public class KubernetesPodLogStreamer {++ private static final Logger LOG = Logger.getLogger(KubernetesPodLogStreamer.class);++ private final KubernetesClient client;+ private final ContainerLogStreamer cloudWatchWriter;++ @Inject+ public KubernetesPodLogStreamer(KubernetesClient client, ContainerLogStreamer cloudWatchWriter) {+ this.client = client;+ this.cloudWatchWriter = cloudWatchWriter;+ }++ /** CloudWatch log stream name for one execution environment, same shape as the docker path. */+ public String logStreamName(String shortId) {+ return cloudWatchWriter.generateLogStreamName("[$LATEST]" + shortId);+ }++ public Closeable attach(String namespace, String podName, String logGroup,+ String logStream, String region, String logPrefix) {+ cloudWatchWriter.ensureLogGroupAndStream(logGroup, logStream, region);++ var watch = client.pods().inNamespace(namespace).withName(podName)+ .inContainer("runtime")+ .watchLog();++ // Virtual thread: one blocking reader per warm pod is near-free this way.+ var reader = Thread.ofVirtual().name("lambda-pod-logs-" + podName).start(() -> {+ try (var lines = new BufferedReader(+ new InputStreamReader(watch.getOutput(), StandardCharsets.UTF_8))) {+ String line;+ while ((line = lines.readLine()) != null) {+ // Match the docker log path: trim trailing whitespace and drop blank+ // lines so both executors write identical CloudWatch events.+ var trimmed = line.stripTrailing();+ if (trimmed.isEmpty()) {+ continue;+ }+ LOG.infov("[{0}] {1}", logPrefix, trimmed);+ cloudWatchWriter.streamToCloudWatchLogs(logGroup, logStream, region, trimmed);+ }+ } catch (IOException e) {+ // Normal on pod deletion: the log stream just ends.+ LOG.debugv("Log stream for pod {0} ended: {1}", podName, e.getMessage());+ }+ });++ return () -> {+ try {+ watch.close();+ } catch (Exception e) {+ LOG.debugv("Closing log watch for pod {0} failed: {1}", podName, e.getMessage());+ }+ reader.interrupt();+ };+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/LambdaPodSpecFactory.java b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/LambdaPodSpecFactory.javanew file mode 100644index 0000000000..16f8a7d2cd--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/LambdaPodSpecFactory.java@@ -0,0 +1,296 @@+package io.github.hectorvent.floci.services.lambda.launcher.kubernetes;++import io.fabric8.kubernetes.api.model.Container;+import io.fabric8.kubernetes.api.model.ContainerBuilder;+import io.fabric8.kubernetes.api.model.EnvVar;+import io.fabric8.kubernetes.api.model.Pod;+import io.fabric8.kubernetes.api.model.PodBuilder;+import io.fabric8.kubernetes.api.model.Quantity;+import io.fabric8.kubernetes.api.model.Volume;+import io.fabric8.kubernetes.api.model.VolumeBuilder;+import io.fabric8.kubernetes.api.model.VolumeMount;+import io.fabric8.kubernetes.api.model.VolumeMountBuilder;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.services.lambda.launcher.ContainerLauncher;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.inject.Inject;+import org.jboss.logging.Logger;++import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Locale;+import java.util.Map;+import java.util.Optional;++/**+ * Builds the Pod manifest for one Lambda execution environment. Code cannot be+ * copied into a pod before it starts (no docker-cp equivalent), so an init+ * container downloads the deployment package (and layers) from Floci's S3 over+ * HTTP into an emptyDir mounted at /var/task before the runtime starts.+ */+@ApplicationScoped+public class LambdaPodSpecFactory {++ private static final Logger LOG = Logger.getLogger(LambdaPodSpecFactory.class);++ /** Label key: optional DNS-subdomain prefix (dot-separated valid segments), then a 63-char-max name. */+ private static final java.util.regex.Pattern LABEL_KEY =+ java.util.regex.Pattern.compile("([a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?[a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?");+ /** Label value: empty, or 63-char-max alphanumeric-bounded. */+ private static final java.util.regex.Pattern LABEL_VALUE =+ java.util.regex.Pattern.compile("([a-zA-Z0-9]([-a-zA-Z0-9_.]{0,61}[a-zA-Z0-9])?)?");++ static final String MANAGED_BY_LABEL = "app.kubernetes.io/managed-by";+ static final String MANAGED_BY_VALUE = "floci";+ static final String SERVICE_LABEL = "floci.io/service";+ static final String SERVICE_VALUE = "lambda";+ static final String FUNCTION_LABEL = "floci.io/function-name";++ private static final String TASK_DIR = "/var/task";+ private static final String OPT_DIR = "/opt";+ private static final String RUNTIME_DIR = "/var/runtime";+ private static final int MAX_NAME_LENGTH = 63;++ private final EmulatorConfig config;++ @Inject+ public LambdaPodSpecFactory(EmulatorConfig config) {+ this.config = config;+ }++ /**+ * @param podName DNS-safe pod name (see {@link #podName})+ * @param image resolved runtime image+ * @param env {@code KEY=value} entries, same shape the Docker launcher builds+ * @param codeDownloadUrl URL of the function's deployment zip; null for Image package type+ * @param layerDownloadUrls layer zips to unpack into /opt, in merge order+ * @param providedRuntime whether the runtime is provided.* (bootstrap needs an exec bit)+ * @param handlerOrNull function handler used as the runtime container arg (Zip functions)+ * @param imageConfig Image-package-type entrypoint/command/workingdir; empty lists/null when unset+ * @param caConfigMapName ConfigMap holding Floci's CA cert, mounted at /etc/floci-ca.crt when present+ */+ public Pod buildPod(String podName,+ String functionName,+ String image,+ List<String> env,+ String codeDownloadUrl,+ List<String> layerDownloadUrls,+ boolean providedRuntime,+ String handlerOrNull,+ ImageConfig imageConfig,+ int memoryMb,+ Optional<String> caConfigMapName) {+ var hasCode = codeDownloadUrl != null;+ var hasLayers = !layerDownloadUrls.isEmpty();++ var volumes = new ArrayList<Volume>();+ var runtimeMounts = new ArrayList<VolumeMount>();+ if (hasCode) {+ volumes.add(new VolumeBuilder().withName("task").withNewEmptyDir().endEmptyDir().build());+ runtimeMounts.add(mount("task", TASK_DIR));+ }+ if (hasLayers) {+ volumes.add(new VolumeBuilder().withName("opt").withNewEmptyDir().endEmptyDir().build());+ runtimeMounts.add(mount("opt", OPT_DIR));+ }+ // The provided.* base images exec /var/runtime/bootstrap with no /var/task+ // fallback, and their /var/runtime ships empty, so masking it with an emptyDir+ // that the init container copies bootstrap into is safe and required.+ var needsRuntimeDir = providedRuntime && hasCode;+ if (needsRuntimeDir) {+ volumes.add(new VolumeBuilder().withName("runtime").withNewEmptyDir().endEmptyDir().build());+ runtimeMounts.add(mount("runtime", RUNTIME_DIR));+ }+ caConfigMapName.ifPresent(cm -> {+ volumes.add(new VolumeBuilder().withName("floci-ca")+ .withNewConfigMap().withName(cm).endConfigMap().build());+ runtimeMounts.add(new VolumeMountBuilder()+ .withName("floci-ca")+ .withMountPath(ContainerLauncher.FLOCI_CA_CONTAINER_PATH)+ .withSubPath(KubernetesPodLauncher.CA_CONFIG_MAP_KEY)+ .withReadOnly(true)+ .build());+ });++ var runtime = new ContainerBuilder()+ .withName("runtime")+ .withImage(image)+ // Explicit IfNotPresent: the default for :latest/untagged is Always,+ // which breaks images pre-loaded onto nodes (kind load docker-image).+ .withImagePullPolicy("IfNotPresent")+ .withEnv(toEnvVars(env))+ .withVolumeMounts(runtimeMounts)+ .withNewResources()+ .addToRequests("memory", new Quantity(memoryMb + "Mi"))+ .addToLimits("memory", new Quantity(memoryMb + "Mi"))+ .endResources();++ if (imageConfig != null && !imageConfig.entryPoint().isEmpty()) {+ runtime.withCommand(escapeDollars(imageConfig.entryPoint()));+ }+ if (imageConfig != null && !imageConfig.command().isEmpty()) {+ runtime.withArgs(escapeDollars(imageConfig.command()));+ } else if (handlerOrNull != null && !handlerOrNull.isBlank()) {+ runtime.withArgs(escapeDollar(handlerOrNull));+ }+ if (imageConfig != null && imageConfig.workingDirectory() != null+ && !imageConfig.workingDirectory().isBlank()) {+ runtime.withWorkingDir(imageConfig.workingDirectory());+ }++ var initContainers = new ArrayList<Container>();+ if (hasCode) {+ var initMounts = new ArrayList<VolumeMount>();+ initMounts.add(mount("task", TASK_DIR));+ if (hasLayers) {+ initMounts.add(mount("opt", OPT_DIR));+ }+ if (needsRuntimeDir) {+ initMounts.add(mount("runtime", RUNTIME_DIR));+ }+ initContainers.add(new ContainerBuilder()+ .withName("code-download")+ .withImage(config.services().lambda().kubernetes().initImage())+ .withImagePullPolicy("IfNotPresent")+ .withCommand("sh", "-c", initScript(codeDownloadUrl, layerDownloadUrls, providedRuntime))+ .withVolumeMounts(initMounts)+ .build());+ }++ return new PodBuilder()+ .withNewMetadata()+ .withName(podName)+ .withLabels(podLabels(functionName))+ .endMetadata()+ .withNewSpec()+ .withRestartPolicy("Never")+ .withTerminationGracePeriodSeconds(5L)+ .withInitContainers(initContainers)+ .withContainers(runtime.build())+ .withVolumes(volumes)+ .endSpec()+ .build();+ }++ /** Entrypoint/command/workingdir of an Image-package-type function. */+ public record ImageConfig(List<String> entryPoint, List<String> command, String workingDirectory) {+ public ImageConfig {+ entryPoint = entryPoint == null ? List.of() : entryPoint;+ command = command == null ? List.of() : command;+ }+ }++ String initScript(String codeDownloadUrl, List<String> layerDownloadUrls, boolean providedRuntime) {+ // Download URLs are always plain HTTP (see KubernetesFlociAddressResolver+ // .downloadBaseUrl): busybox wget's built-in TLS cannot handshake with Floci.+ var wget = "wget -q";+ var script = new StringBuilder("set -e\n");+ script.append(wget).append(" -O /tmp/code.zip '").append(codeDownloadUrl).append("'\n");+ script.append("unzip -oq /tmp/code.zip -d ").append(TASK_DIR).append("\n");+ for (var i = 0; i < layerDownloadUrls.size(); i++) {+ script.append(wget).append(" -O /tmp/layer").append(i).append(".zip '")+ .append(layerDownloadUrls.get(i)).append("'\n");+ script.append("unzip -oq /tmp/layer").append(i).append(".zip -d ").append(OPT_DIR).append("\n");+ }+ script.append("rm -f /tmp/code.zip /tmp/layer*.zip\n");+ if (providedRuntime) {+ // The provided.* entrypoint execs /var/runtime/bootstrap, mirroring the+ // docker path's tar-copy into RUNTIME_DIR. busybox unzip drops unix mode+ // bits, so the exec bit must be restored explicitly.+ script.append("if [ -f ").append(TASK_DIR).append("/bootstrap ]; then cp ")+ .append(TASK_DIR).append("/bootstrap ").append(RUNTIME_DIR)+ .append("/bootstrap && chmod +x ").append(RUNTIME_DIR).append("/bootstrap; fi\n");+ }+ return script.toString();+ }++ Map<String, String> podLabels(String functionName) {+ var labels = new LinkedHashMap<String, String>();+ config.services().lambda().kubernetes().labels().orElse(List.of()).forEach(entry -> {+ var eq = entry.indexOf('=');+ var key = eq > 0 ? entry.substring(0, eq).trim() : "";+ var value = eq > 0 ? entry.substring(eq + 1).trim() : "";+ // Invalid entries are dropped, not sanitized: a silently rewritten label+ // would no longer match the NetworkPolicy/selector the user wrote it for,+ // and the API server would reject the pod at cold start otherwise.+ if (LABEL_KEY.matcher(key).matches() && LABEL_VALUE.matcher(value).matches()) {+ labels.put(key, value);+ } else {+ LOG.warnv("Ignoring invalid Lambda pod label entry ''{0}'' "+ + "(expected key=value with Kubernetes label syntax)", entry);+ }+ });+ // Managed labels go last so user entries can never overwrite them — the+ // orphan sweep selects on these, and an overwritten label leaks pods forever.+ labels.put(MANAGED_BY_LABEL, MANAGED_BY_VALUE);+ labels.put(SERVICE_LABEL, SERVICE_VALUE);+ labels.put(FUNCTION_LABEL, sanitizeLabelValue(functionName));+ return labels;+ }++ /** Selector matching every pod this Floci instance's Lambda executor manages. */+ static Map<String, String> managedPodSelector() {+ return Map.of(MANAGED_BY_LABEL, MANAGED_BY_VALUE, SERVICE_LABEL, SERVICE_VALUE);+ }++ /**+ * DNS-1123 pod name: lowercase alphanumerics and dashes, max 63 chars. The random+ * suffix stays intact; the function name is truncated to fit.+ */+ static String podName(String functionName, String shortId) {+ var prefix = "floci-lambda-";+ var sanitized = functionName.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9-]", "-");+ var budget = MAX_NAME_LENGTH - prefix.length() - shortId.length() - 1;+ if (sanitized.length() > budget) {+ sanitized = sanitized.substring(0, budget);+ }+ sanitized = sanitized.replaceAll("^-+|-+$", "");+ if (sanitized.isEmpty()) {+ sanitized = "fn";+ }+ return prefix + sanitized + "-" + shortId;+ }++ /** Label values allow [a-zA-Z0-9-_.], max 63 chars, alphanumeric at both ends. */+ static String sanitizeLabelValue(String value) {+ var sanitized = value.replaceAll("[^a-zA-Z0-9-_.]", "-");+ if (sanitized.length() > MAX_NAME_LENGTH) {+ sanitized = sanitized.substring(0, MAX_NAME_LENGTH);+ }+ sanitized = sanitized.replaceAll("^[^a-zA-Z0-9]+", "").replaceAll("[^a-zA-Z0-9]+$", "");+ return sanitized;+ }++ private static VolumeMount mount(String name, String path) {+ return new VolumeMountBuilder().withName(name).withMountPath(path).build();+ }++ private static List<String> escapeDollars(List<String> values) {+ return values.stream().map(LambdaPodSpecFactory::escapeDollar).toList();+ }++ private static String escapeDollar(String value) {+ // Kubernetes expands $(VAR) and collapses $$ in command/args exactly as in env+ // values, so escape $ to deliver them byte-for-byte like the docker executor.+ return value.replace("$", "$$");+ }++ private static List<EnvVar> toEnvVars(List<String> env) {+ var vars = new ArrayList<EnvVar>(env.size());+ for (var entry : env) {+ var eq = entry.indexOf('=');+ var key = eq >= 0 ? entry.substring(0, eq) : entry;+ var value = eq >= 0 ? entry.substring(eq + 1) : "";+ // The API server rejects a pod whose env var has an empty name.+ if (key.isEmpty()) {+ LOG.warnv("Skipping env entry with an empty name in the pod spec: ''{0}''", entry);+ continue;+ }+ // Kubernetes expands $(VAR) in env values and collapses $$ to $.+ // Escaping every $ delivers values byte-for-byte like the docker executor.+ vars.add(new EnvVar(key, value.replace("$", "$$"), null));+ }+ return vars;+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/lambda/model/LambdaLayerVersion.java b/src/main/java/io/github/hectorvent/floci/services/lambda/model/LambdaLayerVersion.javaindex c288c6de92..ddcca420cf 100644--- a/src/main/java/io/github/hectorvent/floci/services/lambda/model/LambdaLayerVersion.java+++ b/src/main/java/io/github/hectorvent/floci/services/lambda/model/LambdaLayerVersion.java@@ -24,10 +24,14 @@ public class LambdaLayerVersion {private long codeSizeBytes;private String codeSha256;private String codeLocalPath;+ private boolean archiveStored;public LambdaLayerVersion() {}+ public boolean isArchiveStored() { return archiveStored; }+ public void setArchiveStored(boolean archiveStored) { this.archiveStored = archiveStored; }+public String getLayerName() { return layerName; }public void setLayerName(String layerName) { this.layerName = layerName; }diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex b9218f7e65..b1ecd608c6 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -17,6 +17,11 @@ quarkus:max-body-size: ${floci.max-request-size}Mdatasource:active: false+ kubernetes-client:+ devservices:+ # Never auto-start a test API server; the kubernetes Lambda executor+ # connects to a real cluster (in-cluster config or kubeconfig) only.+ enabled: falsesecurity:security-providers: BCshutdown:@@ -245,6 +250,13 @@ floci:# aws-config-path: # Host path to bind-mount (read-only) at /opt/aws-config for real credential discovery# extra-hosts: # Extra /etc/hosts entries for Lambda containers, "hostname:ip" ("host-gateway" allowed as ip)# container-name-prefix: floci # FLOCI_SERVICES_LAMBDA_CONTAINER_NAME_PREFIX — base name prefix for spawned containers and code volumes ([A-Za-z0-9][A-Za-z0-9_.-]*; invalid values fall back to "floci")+ executor: docker # docker | kubernetes (run Lambda environments as pods)+ # kubernetes:+ # namespace: default # Namespace Lambda pods are created in+ # labels: # Extra pod labels, key=value entries+ # - team=platform+ # floci-address: # Host/IP pods use to reach Floci; auto-detected in-cluster+ # init-image: busybox:1.36 # Init container image (needs sh, wget, unzip)hot-reload:enabled: falseapigateway:diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaLayerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaLayerIntegrationTest.javaindex 7f4397b0d8..925cb1c5e4 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaLayerIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/LambdaLayerIntegrationTest.java@@ -77,6 +77,9 @@ class LambdaLayerIntegrationTest {.body("CompatibleArchitectures", hasItem("x86_64")).body("Content.CodeSize", greaterThan(0)).body("Content.CodeSha256", not(emptyString()))+ .body("Content.Location", containsString("awslambda-us-east-1-tasks/layers/"))+ .body("Content.Location", endsWith("/" + LAYER_NAME+ + "/1?X-Amz-Credential=000000000000%2F00010101%2Fus-east-1%2Fs3%2Faws4_request")).body("CreatedDate", not(emptyString()));}@@ -293,6 +296,66 @@ class LambdaLayerIntegrationTest {.statusCode(204);}+ // ── Stored layer archive in Floci's S3 ────────────────────────────────────++ @Test+ @Order(17)+ void layerArchive_version2_isDownloadableFromTasksBucket() {+ // Publish stores the archive in Floci's S3; the kubernetes executor's init+ // container downloads it from exactly this path. Version 2 is still present here.+ given()+ .when()+ .get("/awslambda-us-east-1-tasks/layers/000000000000/" + LAYER_NAME + "/2")+ .then()+ .statusCode(200);+ }++ @Test+ @Order(18)+ void layerArchive_version1_isDeletedWithTheVersion() {+ // Version 1 was deleted at @Order(13); its stored archive must be gone too.+ given()+ .when()+ .get("/awslambda-us-east-1-tasks/layers/000000000000/" + LAYER_NAME + "/1")+ .then()+ .statusCode(404);+ }++ @Test+ @Order(19)+ void contentLocation_ofNonDefaultAccountLayer_isFetchable() throws Exception {+ // The Location URL is fetched unsigned, so it must carry the owning account in+ // X-Amz-Credential. Without it the download resolves the default account's+ // bucket namespace and 404s for a layer published under another account.+ String location = given()+ .header("Authorization", "AWS4-HMAC-SHA256 Credential=000000000001/20260215/us-east-1/lambda/aws4_request, SignedHeaders=host, Signature=abc")+ .contentType("application/json")+ .body("""+ { "Content": { "ZipFile": "%s" } }+ """.formatted(layerZipBase64()))+ .when()+ .post("/2018-10-31/layers/" + LAYER_NAME + "-alt/versions")+ .then()+ .statusCode(201)+ .body("Content.Location", containsString("/layers/000000000001/"))+ .body("Content.Location", containsString("X-Amz-Credential=000000000001%2F"))+ .extract().path("Content.Location");++ given()+ .urlEncodingEnabled(false)+ .when()+ .get(location)+ .then()+ .statusCode(200);++ given()+ .header("Authorization", "AWS4-HMAC-SHA256 Credential=000000000001/20260215/us-east-1/lambda/aws4_request, SignedHeaders=host, Signature=abc")+ .when()+ .delete("/2018-10-31/layers/" + LAYER_NAME + "-alt/versions/1")+ .then()+ .statusCode(204);+ }+// ── PublishLayerVersion validation ────────────────────────────────────────@Testdiff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/WarmPoolTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/WarmPoolTest.javaindex bf40d92f8e..948913bb46 100644--- a/src/test/java/io/github/hectorvent/floci/services/lambda/WarmPoolTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/WarmPoolTest.java@@ -2,7 +2,7 @@ package io.github.hectorvent.floci.services.lambda;import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.services.lambda.launcher.ContainerHandle;-import io.github.hectorvent.floci.services.lambda.launcher.ContainerLauncher;+import io.github.hectorvent.floci.services.lambda.launcher.LambdaRuntimeLauncher;import io.github.hectorvent.floci.services.lambda.model.ContainerState;import io.github.hectorvent.floci.services.lambda.model.LambdaFunction;import io.github.hectorvent.floci.services.lambda.runtime.RuntimeApiServer;@@ -32,7 +32,7 @@ import static org.mockito.Mockito.when;@ExtendWith(MockitoExtension.class)class WarmPoolTest {- @Mock ContainerLauncher containerLauncher;+ @Mock LambdaRuntimeLauncher containerLauncher;@Mock EmulatorConfig config;private WarmPool buildPool() {diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncherProducerTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncherProducerTest.javanew file mode 100644index 0000000000..aa10c7380f--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/LambdaRuntimeLauncherProducerTest.java@@ -0,0 +1,95 @@+package io.github.hectorvent.floci.services.lambda.launcher;++import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.services.lambda.launcher.kubernetes.KubernetesPodLauncher;+import io.quarkus.runtime.StartupEvent;+import jakarta.enterprise.inject.Instance;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.extension.ExtendWith;+import org.mockito.Mock;+import org.mockito.junit.jupiter.MockitoExtension;+import org.mockito.junit.jupiter.MockitoSettings;+import org.mockito.quality.Strictness;++import static org.assertj.core.api.Assertions.assertThat;+import static org.assertj.core.api.Assertions.assertThatCode;+import static org.assertj.core.api.Assertions.assertThatThrownBy;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++@ExtendWith(MockitoExtension.class)+@MockitoSettings(strictness = Strictness.LENIENT)+class LambdaRuntimeLauncherProducerTest {++ @Mock EmulatorConfig config;+ @Mock Instance<ContainerLauncher> dockerInstance;+ @Mock Instance<KubernetesPodLauncher> kubernetesInstance;+ @Mock ContainerLauncher containerLauncher;+ @Mock KubernetesPodLauncher kubernetesPodLauncher;++ private EmulatorConfig.LambdaServiceConfig lambda;+ private final LambdaRuntimeLauncherProducer producer = new LambdaRuntimeLauncherProducer();++ @BeforeEach+ void setUp() {+ var services = mock(EmulatorConfig.ServicesConfig.class);+ lambda = mock(EmulatorConfig.LambdaServiceConfig.class);+ when(config.services()).thenReturn(services);+ when(services.lambda()).thenReturn(lambda);+ when(dockerInstance.get()).thenReturn(containerLauncher);+ when(kubernetesInstance.get()).thenReturn(kubernetesPodLauncher);+ }++ @Test+ void dockerExecutorSelectsContainerLauncher() {+ when(lambda.executor()).thenReturn("docker");+ assertThat(producer.launcher(config, dockerInstance, kubernetesInstance))+ .isSameAs(containerLauncher);+ verify(kubernetesInstance, never()).get();+ }++ @Test+ void kubernetesExecutorSelectsPodLauncher() {+ when(lambda.executor()).thenReturn("kubernetes");+ assertThat(producer.launcher(config, dockerInstance, kubernetesInstance))+ .isSameAs(kubernetesPodLauncher);+ verify(dockerInstance, never()).get();+ }++ @Test+ void executorValueIsCaseInsensitiveAndTrimmed() {+ when(lambda.executor()).thenReturn(" Kubernetes\n");+ assertThat(producer.launcher(config, dockerInstance, kubernetesInstance))+ .isSameAs(kubernetesPodLauncher);+ }++ @Test+ void unknownExecutorFailsWithValidValues() {+ when(lambda.executor()).thenReturn("podman");+ assertThatThrownBy(() -> producer.launcher(config, dockerInstance, kubernetesInstance))+ .isInstanceOf(IllegalArgumentException.class)+ .hasMessageContaining("docker, kubernetes");+ }++ @Test+ void startupValidationRejectsUnknownExecutorBeforeFirstInvoke() {+ when(lambda.executor()).thenReturn("podman");+ assertThatThrownBy(() -> producer.validateExecutor(new StartupEvent(), config, kubernetesInstance))+ .isInstanceOf(IllegalArgumentException.class);+ }++ @Test+ void startupValidationAcceptsValidExecutors() {+ when(lambda.executor()).thenReturn("docker");+ assertThatCode(() -> producer.validateExecutor(new StartupEvent(), config, kubernetesInstance))+ .doesNotThrowAnyException();+ verify(kubernetesPodLauncher, never()).initializeClient();+ when(lambda.executor()).thenReturn("kubernetes");+ assertThatCode(() -> producer.validateExecutor(new StartupEvent(), config, kubernetesInstance))+ .doesNotThrowAnyException();+ verify(kubernetesPodLauncher).initializeClient();+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesFlociAddressResolverTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesFlociAddressResolverTest.javanew file mode 100644index 0000000000..adf65916e1--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesFlociAddressResolverTest.java@@ -0,0 +1,91 @@+package io.github.hectorvent.floci.services.lambda.launcher.kubernetes;++import io.github.hectorvent.floci.config.EmulatorConfig;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.extension.ExtendWith;+import org.mockito.Mock;+import org.mockito.junit.jupiter.MockitoExtension;+import org.mockito.junit.jupiter.MockitoSettings;+import org.mockito.quality.Strictness;++import java.util.Optional;++import static org.assertj.core.api.Assertions.assertThat;+import static org.assertj.core.api.Assertions.assertThatThrownBy;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++@ExtendWith(MockitoExtension.class)+@MockitoSettings(strictness = Strictness.LENIENT)+class KubernetesFlociAddressResolverTest {++ @Mock EmulatorConfig config;+ private EmulatorConfig.LambdaServiceConfig.KubernetesExecutor kubernetes;+ private EmulatorConfig.TlsConfig tls;+ private KubernetesFlociAddressResolver resolver;++ @BeforeEach+ void setUp() {+ var services = mock(EmulatorConfig.ServicesConfig.class);+ var lambda = mock(EmulatorConfig.LambdaServiceConfig.class);+ kubernetes = mock(EmulatorConfig.LambdaServiceConfig.KubernetesExecutor.class);+ tls = mock(EmulatorConfig.TlsConfig.class);+ when(config.services()).thenReturn(services);+ when(services.lambda()).thenReturn(lambda);+ when(lambda.kubernetes()).thenReturn(kubernetes);+ when(lambda.runtimeApiBasePort()).thenReturn(9200);+ when(lambda.runtimeApiMaxPort()).thenReturn(9299);+ when(config.tls()).thenReturn(tls);+ when(tls.enabled()).thenReturn(false);+ when(config.port()).thenReturn(4566);+ resolver = new KubernetesFlociAddressResolver(config);+ }++ @Test+ void configuredAddressIsUsedVerbatim() {+ when(kubernetes.flociAddress()).thenReturn(Optional.of("192.168.1.20"));+ assertThat(resolver.resolve()).isEqualTo("192.168.1.20");+ assertThat(resolver.flociBaseUrl()).isEqualTo("http://192.168.1.20:4566");+ }++ @Test+ void addressWithSchemeIsRejectedEarly() {+ when(kubernetes.flociAddress()).thenReturn(Optional.of("http://192.168.1.20"));+ assertThatThrownBy(resolver::resolve)+ .isInstanceOf(IllegalStateException.class)+ .hasMessageContaining("bare host or IP");+ }++ @Test+ void addressWithPortIsRejectedEarly() {+ when(kubernetes.flociAddress()).thenReturn(Optional.of("192.168.1.20:4566"));+ assertThatThrownBy(resolver::resolve)+ .isInstanceOf(IllegalStateException.class)+ .hasMessageContaining("bare host or IP");+ }++ @Test+ void addressWithShellHostileCharactersIsRejectedEarly() {+ // resolve() output is embedded in single-quoted init-container commands,+ // so anything beyond hostname/IP characters must fail fast.+ when(kubernetes.flociAddress()).thenReturn(Optional.of("10.0.0.5'; reboot;'"));+ assertThatThrownBy(resolver::resolve)+ .isInstanceOf(IllegalStateException.class)+ .hasMessageContaining("bare host or IP");+ }++ @Test+ void tlsSwitchesBaseUrlScheme() {+ when(kubernetes.flociAddress()).thenReturn(Optional.of("floci.internal"));+ when(tls.enabled()).thenReturn(true);+ assertThat(resolver.flociBaseUrl()).isEqualTo("https://floci.internal:4566");+ }++ @Test+ void downloadBaseUrlStaysPlainHttpEvenWithTls() {+ when(kubernetes.flociAddress()).thenReturn(Optional.of("floci.internal"));+ when(tls.enabled()).thenReturn(true);+ assertThat(resolver.downloadBaseUrl()).isEqualTo("http://floci.internal:4566");+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesPodLauncherTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesPodLauncherTest.javanew file mode 100644index 0000000000..080225dfb0--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/KubernetesPodLauncherTest.java@@ -0,0 +1,376 @@+package io.github.hectorvent.floci.services.lambda.launcher.kubernetes;++import io.fabric8.kubernetes.api.model.EnvVar;+import io.fabric8.kubernetes.api.model.PodBuilder;+import io.fabric8.kubernetes.api.model.PodStatusBuilder;+import io.fabric8.kubernetes.client.KubernetesClient;+import io.fabric8.kubernetes.client.server.mock.EnableKubernetesMockClient;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;+import io.github.hectorvent.floci.services.lambda.LambdaLayerService;+import io.github.hectorvent.floci.services.lambda.launcher.ContainerHandle;+import io.github.hectorvent.floci.services.lambda.launcher.ImageResolver;+import io.github.hectorvent.floci.services.lambda.model.ContainerState;+import io.github.hectorvent.floci.services.lambda.model.LambdaFunction;+import io.github.hectorvent.floci.services.lambda.runtime.RuntimeApiServer;+import io.github.hectorvent.floci.services.lambda.runtime.RuntimeApiServerFactory;+import io.github.hectorvent.floci.services.s3.S3Service;+import org.awaitility.Awaitility;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.time.Duration;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.CompletableFuture;+import java.util.concurrent.atomic.AtomicBoolean;++import static org.assertj.core.api.Assertions.assertThat;+import static org.assertj.core.api.Assertions.assertThatThrownBy;+import static org.assertj.core.api.Assertions.tuple;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.Mockito.doAnswer;+import static org.mockito.Mockito.doThrow;+import static org.mockito.Mockito.lenient;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.spy;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++@EnableKubernetesMockClient(crud = true)+class KubernetesPodLauncherTest {+ KubernetesClient client;++ private EmulatorConfig config;+ private RuntimeApiServerFactory runtimeApiServerFactory;+ private RuntimeApiServer runtimeApiServer;+ private ImageResolver imageResolver;+ private KubernetesFlociAddressResolver addressResolver;+ private LaunchedContainerAwsEnv awsEnv;+ private LambdaLayerService layerService;+ private KubernetesPodLogStreamer logStreamer;+ private S3Service s3Service;+ private KubernetesPodLauncher launcher;++ @BeforeEach+ void setUp() {+ config = mock(EmulatorConfig.class);+ var services = mock(EmulatorConfig.ServicesConfig.class);+ var lambda = mock(EmulatorConfig.LambdaServiceConfig.class);+ var kubernetes = mock(EmulatorConfig.LambdaServiceConfig.KubernetesExecutor.class);+ var tls = mock(EmulatorConfig.TlsConfig.class);+ lenient().when(config.services()).thenReturn(services);+ lenient().when(services.lambda()).thenReturn(lambda);+ lenient().when(lambda.kubernetes()).thenReturn(kubernetes);+ lenient().when(lambda.awsConfigPath()).thenReturn(Optional.empty());+ lenient().when(config.tls()).thenReturn(tls);+ lenient().when(tls.enabled()).thenReturn(false);+ lenient().when(config.defaultRegion()).thenReturn("us-east-1");+ lenient().when(kubernetes.namespace()).thenReturn("default");+ lenient().when(kubernetes.initImage()).thenReturn("busybox:1.36");+ lenient().when(kubernetes.labels()).thenReturn(Optional.empty());++ runtimeApiServerFactory = mock(RuntimeApiServerFactory.class);+ runtimeApiServer = mock(RuntimeApiServer.class);+ lenient().when(runtimeApiServerFactory.create()).thenReturn(runtimeApiServer);+ lenient().when(runtimeApiServer.getPort()).thenReturn(9200);+ lenient().when(runtimeApiServer.stop()).thenReturn(CompletableFuture.completedFuture(null));++ imageResolver = mock(ImageResolver.class);+ lenient().when(imageResolver.resolve("python3.12")).thenReturn("public.ecr.aws/lambda/python:3.12");++ addressResolver = mock(KubernetesFlociAddressResolver.class);+ lenient().when(addressResolver.resolve()).thenReturn("10.0.0.5");+ lenient().when(addressResolver.flociBaseUrl()).thenReturn("http://10.0.0.5:4566");+ lenient().when(addressResolver.downloadBaseUrl()).thenReturn("http://10.0.0.5:4566");++ awsEnv = mock(LaunchedContainerAwsEnv.class);+ lenient().when(awsEnv.sdkBaselineEnv(anyString(), any(), anyString()))+ .thenReturn(List.of("AWS_REGION=us-east-1"));++ layerService = mock(LambdaLayerService.class);+ logStreamer = mock(KubernetesPodLogStreamer.class);+ lenient().when(logStreamer.attach(anyString(), anyString(), anyString(), anyString(),+ anyString(), anyString())).thenReturn(() -> { });+ lenient().when(logStreamer.logStreamName(anyString())).thenReturn("2026/07/25/[$LATEST]test");+ s3Service = mock(S3Service.class);++ launcher = new KubernetesPodLauncher(client, config, runtimeApiServerFactory, imageResolver,+ addressResolver, awsEnv, layerService, new LambdaPodSpecFactory(config),+ logStreamer, s3Service);+ }++ private LambdaFunction function() {+ var fn = new LambdaFunction();+ fn.setFunctionName("my-fn");+ fn.setAccountId("000000000000");+ fn.setFunctionArn("arn:aws:lambda:us-east-1:000000000000:function:my-fn");+ fn.setRuntime("python3.12");+ fn.setHandler("index.handler");+ fn.setMemorySize(128);+ fn.setTimeout(3);+ return fn;+ }++ /** Plays the kubelet: waits for a pod with the given name prefix, then sets its phase. */+ private void markPodPhaseInBackground(String namePrefix, String phase) {+ CompletableFuture.runAsync(() -> {+ Awaitility.await().atMost(Duration.ofSeconds(5)).until(+ () -> client.pods().inNamespace("default").list().getItems().stream()+ .anyMatch(p -> p.getMetadata().getName().startsWith(namePrefix)));+ var pod = client.pods().inNamespace("default").list().getItems().stream()+ .filter(p -> p.getMetadata().getName().startsWith(namePrefix))+ .findFirst().orElseThrow();+ pod.setStatus(new PodStatusBuilder().withPhase(phase).build());+ client.pods().inNamespace("default").resource(pod).updateStatus();+ });+ }++ @Test+ void launchCreatesPodAndReturnsHandle() {+ markPodPhaseInBackground("floci-lambda-my-fn-", "Running");++ var handle = launcher.launch(function());++ assertThat(handle).isNotNull();+ assertThat(handle.getFunctionName()).isEqualTo("my-fn");+ assertThat(handle.getContainerId()).startsWith("floci-lambda-my-fn-");++ var pod = client.pods().inNamespace("default").withName(handle.getContainerId()).get();+ assertThat(pod).isNotNull();+ assertThat(pod.getMetadata().getLabels())+ .containsEntry("app.kubernetes.io/managed-by", "floci");+ assertThat(pod.getSpec().getInitContainers()).hasSize(1);+ assertThat(pod.getSpec().getContainers().getFirst().getEnv())+ .extracting(EnvVar::getName, EnvVar::getValue)+ .contains(tuple("AWS_LAMBDA_RUNTIME_API", "10.0.0.5:9200"));+ assertThat(pod.getSpec().getInitContainers().getFirst().getCommand().get(2))+ .contains("http://10.0.0.5:4566/awslambda-us-east-1-tasks/snapshots/000000000000/my-fn");+ }++ @Test+ void downloadUrlCarriesTheOwningAccountForNonDefaultAccounts() {+ var fn = function();+ fn.setAccountId("111122223333");+ fn.setFunctionArn("arn:aws:lambda:us-east-1:111122223333:function:my-fn");+ markPodPhaseInBackground("floci-lambda-my-fn-", "Running");++ var handle = launcher.launch(fn);++ // The init container fetches unauthenticated, so the download URL must carry the+ // owning account or the object resolves under the default account and 404s.+ var pod = client.pods().inNamespace("default").withName(handle.getContainerId()).get();+ assertThat(pod.getSpec().getInitContainers().getFirst().getCommand().get(2))+ .contains("snapshots/111122223333/my-fn?X-Amz-Credential=111122223333%2F");+ }++ @Test+ void launchFailureReleasesPortAndDeletesPod() {+ markPodPhaseInBackground("floci-lambda-my-fn-", "Failed");++ assertThatThrownBy(() -> launcher.launch(function()))+ .isInstanceOf(RuntimeException.class)+ .hasMessageContaining("failed to start");++ verify(runtimeApiServer).stop();+ verify(runtimeApiServerFactory).release(runtimeApiServer);+ Awaitility.await().atMost(Duration.ofSeconds(5)).until(+ () -> client.pods().inNamespace("default").list().getItems().isEmpty());+ }++ @Test+ void launchFailureKeepsPortReservedWhenPodDeleteIsNotConfirmed() {+ // Once the pod is parked in Failed, every further Kubernetes API call fails, so+ // the cleanup delete can never confirm the pod is gone. The pod could still reach+ // Running later and poll AWS_LAMBDA_RUNTIME_API, so the port must stay reserved+ // instead of going back to the pool for another environment.+ var failingClient = spy(client);+ doAnswer(invocation -> {+ var failedPodExists = client.pods().inNamespace("default").list().getItems().stream()+ .anyMatch(p -> p.getStatus() != null && "Failed".equals(p.getStatus().getPhase()));+ if (failedPodExists) {+ throw new RuntimeException("kube api down");+ }+ return invocation.callRealMethod();+ }).when(failingClient).pods();+ var failingLauncher = new KubernetesPodLauncher(failingClient, config,+ runtimeApiServerFactory, imageResolver, addressResolver, awsEnv, layerService,+ new LambdaPodSpecFactory(config), logStreamer, s3Service);+ markPodPhaseInBackground("floci-lambda-my-fn-", "Failed");++ assertThatThrownBy(() -> failingLauncher.launch(function()))+ .isInstanceOf(RuntimeException.class);++ verify(runtimeApiServer).stop();+ verify(runtimeApiServerFactory, never()).release(runtimeApiServer);+ assertThat(client.pods().inNamespace("default").list().getItems())+ .as("the unconfirmed pod is left for a later sweep")+ .isNotEmpty();+ }++ @Test+ void podThatExitsCleanlyIsTerminalNotATimeout() {+ // A runtime that exits before serving parks the pod in Succeeded; the launch+ // must fail fast instead of blocking for the full startup timeout.+ markPodPhaseInBackground("floci-lambda-my-fn-", "Succeeded");++ assertThatThrownBy(() -> launcher.launch(function()))+ .isInstanceOf(RuntimeException.class)+ .hasMessageContaining("failed to start");+ verify(runtimeApiServerFactory).release(runtimeApiServer);+ }++ @Test+ void hotReloadIsRejectedBeforeAllocatingAnything() {+ var fn = function();+ fn.setHotReloadHostPath("/tmp/code");++ assertThatThrownBy(() -> launcher.launch(fn))+ .isInstanceOf(RuntimeException.class)+ .hasMessageContaining("Hot reload");+ verify(runtimeApiServerFactory, never()).create();+ }++ @Test+ void missingCodeSnapshotFailsWithActionableMessage() {+ when(s3Service.headObject(anyString(), anyString()))+ .thenThrow(new AwsException("NoSuchKey", "not found", 404));++ assertThatThrownBy(() -> launcher.launch(function()))+ .isInstanceOf(RuntimeException.class)+ .hasMessageContaining("Re-deploy the function code");+ verify(runtimeApiServerFactory).release(runtimeApiServer);+ }++ @Test+ void stopDeletesPodAndReleasesRuntimeApiPort() {+ var pod = new PodBuilder()+ .withNewMetadata().withName("floci-lambda-my-fn-abc12345").endMetadata()+ .build();+ client.pods().inNamespace("default").resource(pod).create();++ var handle = new ContainerHandle("floci-lambda-my-fn-abc12345", "my-fn",+ runtimeApiServer, ContainerState.WARM);+ launcher.stop(handle);++ assertThat(client.pods().inNamespace("default").withName("floci-lambda-my-fn-abc12345").get())+ .isNull();+ verify(runtimeApiServer).stop();+ verify(runtimeApiServerFactory).release(runtimeApiServer);+ assertThat(handle.getState()).isEqualTo(ContainerState.STOPPED);+ }++ @Test+ void isAliveReflectsPodPhase() {+ var running = new PodBuilder()+ .withNewMetadata().withName("pod-running").endMetadata()+ .withStatus(new PodStatusBuilder().withPhase("Running").build())+ .build();+ var pending = new PodBuilder()+ .withNewMetadata().withName("pod-pending").endMetadata()+ .withStatus(new PodStatusBuilder().withPhase("Pending").build())+ .build();+ client.pods().inNamespace("default").resource(running).create();+ client.pods().inNamespace("default").resource(pending).create();++ assertThat(launcher.isAlive(handleFor("pod-running"))).isTrue();+ assertThat(launcher.isAlive(handleFor("pod-pending"))).isFalse();+ assertThat(launcher.isAlive(handleFor("pod-absent"))).isFalse();+ }++ @Test+ void orphanSweepDeletesOnlyManagedPods() {+ var orphan = new PodBuilder()+ .withNewMetadata().withName("floci-lambda-old-fn-dead1234")+ .withLabels(Map.of(+ "app.kubernetes.io/managed-by", "floci",+ "floci.io/service", "lambda"))+ .endMetadata()+ .build();+ var unrelated = new PodBuilder()+ .withNewMetadata().withName("some-other-pod").endMetadata()+ .build();+ client.pods().inNamespace("default").resource(orphan).create();+ client.pods().inNamespace("default").resource(unrelated).create();++ // Fail the launch right after the sweep so the test doesn't need a running pod.+ when(imageResolver.resolve(anyString())).thenThrow(new RuntimeException("no image"));+ assertThatThrownBy(() -> launcher.launch(function()))+ .isInstanceOf(RuntimeException.class);++ Awaitility.await().atMost(Duration.ofSeconds(5)).until(+ () -> client.pods().inNamespace("default").withName("floci-lambda-old-fn-dead1234").get() == null);+ assertThat(client.pods().inNamespace("default").withName("some-other-pod").get()).isNotNull();+ }++ @Test+ void stopDeletesPodEvenWhenPortReleaseThrows() {+ var pod = new PodBuilder()+ .withNewMetadata().withName("floci-lambda-my-fn-abc12345").endMetadata()+ .build();+ client.pods().inNamespace("default").resource(pod).create();+ doThrow(new IllegalStateException("release failed"))+ .when(runtimeApiServerFactory).release(runtimeApiServer);+ var streamClosed = new AtomicBoolean();+ var handle = new ContainerHandle("floci-lambda-my-fn-abc12345", "my-fn",+ runtimeApiServer, ContainerState.WARM);+ handle.setLogStream(() -> streamClosed.set(true));++ launcher.stop(handle);++ assertThat(client.pods().inNamespace("default").withName("floci-lambda-my-fn-abc12345").get())+ .isNull();+ assertThat(streamClosed).isTrue();+ }++ @Test+ void failedOrphanSweepIsRetriedWithoutKillingOwnPods() {+ var orphan = new PodBuilder()+ .withNewMetadata().withName("floci-lambda-old-fn-dead1234")+ .withLabels(Map.of(+ "app.kubernetes.io/managed-by", "floci",+ "floci.io/service", "lambda"))+ .endMetadata()+ .build();+ client.pods().inNamespace("default").resource(orphan).create();++ // First Kubernetes API access (the sweep's pod list) fails once.+ var failedOnce = new AtomicBoolean();+ var flakyClient = spy(client);+ doAnswer(invocation -> {+ if (failedOnce.compareAndSet(false, true)) {+ throw new RuntimeException("kube api down");+ }+ return invocation.callRealMethod();+ }).when(flakyClient).pods();+ var retryingLauncher = new KubernetesPodLauncher(flakyClient, config,+ runtimeApiServerFactory, imageResolver, addressResolver, awsEnv, layerService,+ new LambdaPodSpecFactory(config), logStreamer, s3Service);++ markPodPhaseInBackground("floci-lambda-my-fn-", "Running");+ var handle = retryingLauncher.launch(function());+ assertThat(client.pods().inNamespace("default").withName("floci-lambda-old-fn-dead1234").get())+ .as("orphan survives the failed sweep")+ .isNotNull();++ // The next launch retries the sweep. Fail it right after the sweep so the+ // test does not need a second running pod.+ when(imageResolver.resolve(anyString())).thenThrow(new RuntimeException("no image"));+ assertThatThrownBy(() -> retryingLauncher.launch(function()))+ .isInstanceOf(RuntimeException.class);++ Awaitility.await().atMost(Duration.ofSeconds(5)).until(+ () -> client.pods().inNamespace("default").withName("floci-lambda-old-fn-dead1234").get() == null);+ assertThat(client.pods().inNamespace("default").withName(handle.getContainerId()).get())+ .as("the retried sweep must not delete this process's own pod")+ .isNotNull();+ }++ private ContainerHandle handleFor(String podName) {+ return new ContainerHandle(podName, "my-fn", runtimeApiServer, ContainerState.WARM);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/LambdaPodSpecFactoryTest.java b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/LambdaPodSpecFactoryTest.javanew file mode 100644index 0000000000..8e9d72ece2--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/lambda/launcher/kubernetes/LambdaPodSpecFactoryTest.java@@ -0,0 +1,310 @@+package io.github.hectorvent.floci.services.lambda.launcher.kubernetes;++import io.fabric8.kubernetes.api.model.EnvVar;+import io.fabric8.kubernetes.api.model.Pod;+import io.fabric8.kubernetes.api.model.Volume;+import io.fabric8.kubernetes.api.model.VolumeMount;+import io.github.hectorvent.floci.config.EmulatorConfig;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.extension.ExtendWith;+import org.mockito.Mock;+import org.mockito.junit.jupiter.MockitoExtension;+import org.mockito.junit.jupiter.MockitoSettings;+import org.mockito.quality.Strictness;++import java.util.List;+import java.util.Optional;++import static org.assertj.core.api.Assertions.assertThat;+import static org.assertj.core.api.Assertions.tuple;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++@ExtendWith(MockitoExtension.class)+@MockitoSettings(strictness = Strictness.LENIENT)+class LambdaPodSpecFactoryTest {++ @Mock+ EmulatorConfig config;+ private LambdaPodSpecFactory factory;++ private EmulatorConfig.LambdaServiceConfig.KubernetesExecutor kubernetes;+ private EmulatorConfig.TlsConfig tls;++ @BeforeEach+ void setUp() {+ var services = mock(EmulatorConfig.ServicesConfig.class);+ var lambda = mock(EmulatorConfig.LambdaServiceConfig.class);+ kubernetes = mock(EmulatorConfig.LambdaServiceConfig.KubernetesExecutor.class);+ tls = mock(EmulatorConfig.TlsConfig.class);+ when(config.services()).thenReturn(services);+ when(services.lambda()).thenReturn(lambda);+ when(lambda.kubernetes()).thenReturn(kubernetes);+ when(config.tls()).thenReturn(tls);+ when(tls.enabled()).thenReturn(false);+ when(kubernetes.initImage()).thenReturn("busybox:1.36");+ when(kubernetes.labels()).thenReturn(Optional.empty());+ factory = new LambdaPodSpecFactory(config);+ }++ private Pod buildZipPod() {+ return factory.buildPod("floci-lambda-my-fn-abc12345", "my-fn",+ "public.ecr.aws/lambda/python:3.12",+ List.of("AWS_LAMBDA_RUNTIME_API=10.0.0.5:9200", "_HANDLER=index.handler"),+ "http://10.0.0.5:4566/awslambda-us-east-1-tasks/snapshots/000000000000/my-fn",+ List.of(), false, "index.handler", null, 256, Optional.empty());+ }++ @Test+ void zipFunctionPodHasInitContainerTaskVolumeAndEnv() {+ var spec = buildZipPod().getSpec();++ assertThat(spec.getRestartPolicy()).isEqualTo("Never");+ assertThat(spec.getTerminationGracePeriodSeconds()).isEqualTo(5L);+ assertThat(spec.getInitContainers()).hasSize(1);++ var init = spec.getInitContainers().getFirst();+ assertThat(init.getImage()).isEqualTo("busybox:1.36");+ assertThat(init.getCommand().get(2))+ .contains("wget -q -O /tmp/code.zip")+ .contains("unzip -oq /tmp/code.zip -d /var/task")+ .doesNotContain("--no-check-certificate");+ assertThat(init.getVolumeMounts())+ .extracting(VolumeMount::getName, VolumeMount::getMountPath)+ .containsExactly(tuple("task", "/var/task"));++ var runtime = spec.getContainers().getFirst();+ assertThat(runtime.getName()).isEqualTo("runtime");+ assertThat(runtime.getArgs()).containsExactly("index.handler");+ assertThat(runtime.getResources().getLimits().get("memory")).hasToString("256Mi");+ assertThat(runtime.getResources().getRequests().get("memory")).hasToString("256Mi");+ assertThat(runtime.getVolumeMounts())+ .extracting(VolumeMount::getName, VolumeMount::getMountPath)+ .contains(tuple("task", "/var/task"));+ assertThat(runtime.getEnv())+ .extracting(EnvVar::getName, EnvVar::getValue)+ .contains(tuple("AWS_LAMBDA_RUNTIME_API", "10.0.0.5:9200"));++ assertThat(spec.getVolumes()).extracting(Volume::getName).containsExactly("task");+ }++ @Test+ void standardLabelsAreApplied() {+ var labels = buildZipPod().getMetadata().getLabels();+ assertThat(labels)+ .containsEntry("app.kubernetes.io/managed-by", "floci")+ .containsEntry("floci.io/service", "lambda")+ .containsEntry("floci.io/function-name", "my-fn");+ }++ @Test+ void userLabelsAreParsedAndApplied() {+ when(kubernetes.labels()).thenReturn(Optional.of(+ List.of("team=platform", "floci.io/env=ci", "empty-ok=")));+ var labels = buildZipPod().getMetadata().getLabels();+ assertThat(labels)+ .containsEntry("team", "platform")+ .containsEntry("floci.io/env", "ci")+ .containsEntry("empty-ok", "");+ }++ @Test+ void invalidUserLabelsAreDroppedNotSanitized() {+ // '=' in a value and spaces in a key are rejected by the API server, so+ // such entries must never reach the pod spec.+ when(kubernetes.labels()).thenReturn(Optional.of(+ List.of("malformed", "a=b=c", "bad key=x", "team=platform")));+ var labels = buildZipPod().getMetadata().getLabels();+ assertThat(labels)+ .containsEntry("team", "platform")+ .doesNotContainKeys("malformed", "a", "bad key");+ }++ @Test+ void layersAreDownloadedInOrderIntoOpt() {+ var pod = factory.buildPod("floci-lambda-my-fn-abc12345", "my-fn",+ "public.ecr.aws/lambda/nodejs:20",+ List.of(),+ "http://10.0.0.5:4566/awslambda-us-east-1-tasks/snapshots/000000000000/my-fn",+ List.of("http://10.0.0.5:4566/b/layers/1", "http://10.0.0.5:4566/b/layers/2"),+ false, "index.handler", null, 128, Optional.empty());++ var spec = pod.getSpec();+ assertThat(spec.getVolumes()).extracting(Volume::getName).contains("opt");+ assertThat(spec.getContainers().getFirst().getVolumeMounts())+ .extracting(VolumeMount::getName, VolumeMount::getMountPath)+ .contains(tuple("opt", "/opt"));+ assertThat(spec.getInitContainers().getFirst().getVolumeMounts())+ .extracting(VolumeMount::getName, VolumeMount::getMountPath)+ .contains(tuple("opt", "/opt"));++ assertThat(spec.getInitContainers().getFirst().getCommand().get(2))+ .contains("unzip -oq /tmp/layer0.zip -d /opt")+ .containsSubsequence(+ "wget -q -O /tmp/layer0.zip 'http://10.0.0.5:4566/b/layers/1'",+ "wget -q -O /tmp/layer1.zip 'http://10.0.0.5:4566/b/layers/2'");+ }++ @Test+ void providedRuntimeCopiesBootstrapIntoVarRuntime() {+ // The provided.* entrypoint execs /var/runtime/bootstrap with no /var/task+ // fallback, so the init container must copy it there and restore the exec bit.+ var pod = factory.buildPod("floci-lambda-custom-abc12345", "custom",+ "public.ecr.aws/lambda/provided:al2023",+ List.of(),+ "http://10.0.0.5:4566/awslambda-us-east-1-tasks/snapshots/000000000000/custom",+ List.of(), true, "bootstrap", null, 128, Optional.empty());++ var spec = pod.getSpec();+ assertThat(spec.getVolumes()).extracting(Volume::getName).contains("runtime");+ assertThat(spec.getContainers().getFirst().getVolumeMounts())+ .extracting(VolumeMount::getName, VolumeMount::getMountPath)+ .contains(tuple("runtime", "/var/runtime"));+ assertThat(spec.getInitContainers().getFirst().getVolumeMounts())+ .extracting(VolumeMount::getName)+ .contains("runtime");+ assertThat(spec.getInitContainers().getFirst().getCommand().get(2))+ .contains("cp /var/task/bootstrap /var/runtime/bootstrap")+ .contains("chmod +x /var/runtime/bootstrap");+ }++ @Test+ void nonProvidedRuntimeDoesNotMaskVarRuntime() {+ // Masking /var/runtime on a normal runtime would delete the runtime interface client.+ assertThat(buildZipPod().getSpec().getVolumes())+ .extracting(Volume::getName)+ .doesNotContain("runtime");+ }++ @Test+ void managedLabelsCannotBeOverriddenByUserLabels() {+ when(kubernetes.labels()).thenReturn(Optional.of(+ List.of("app.kubernetes.io/managed-by=evil", "floci.io/service=other")));+ var labels = buildZipPod().getMetadata().getLabels();+ assertThat(labels)+ .containsEntry("app.kubernetes.io/managed-by", "floci")+ .containsEntry("floci.io/service", "lambda");+ }++ @Test+ void dollarSignsInEnvValuesAreEscapedForKubernetesExpansion() {+ var pod = factory.buildPod("floci-lambda-my-fn-abc12345", "my-fn",+ "public.ecr.aws/lambda/python:3.12",+ List.of("SECRET=pa$$word", "TEMPLATE=$(HOME)/x"),+ "http://10.0.0.5:4566/b/k", List.of(), false, "h", null, 128, Optional.empty());+ assertThat(pod.getSpec().getContainers().getFirst().getEnv())+ .extracting(EnvVar::getName, EnvVar::getValue)+ .contains(+ tuple("SECRET", "pa$$$$word"),+ tuple("TEMPLATE", "$$(HOME)/x"));+ }++ @Test+ void tlsMountsCaCertAndStillDownloadsWithPlainWget() {+ // Downloads stay plain HTTP even in TLS mode (busybox wget cannot TLS-handshake+ // with Floci); the CA mount is for SDK calls made from inside the function.+ when(tls.enabled()).thenReturn(true);+ var pod = factory.buildPod("floci-lambda-my-fn-abc12345", "my-fn",+ "public.ecr.aws/lambda/python:3.12",+ List.of(),+ "http://10.0.0.5:4566/awslambda-us-east-1-tasks/snapshots/000000000000/my-fn",+ List.of(), false, "index.handler", null, 128,+ Optional.of(KubernetesPodLauncher.CA_CONFIG_MAP_NAME));++ var spec = pod.getSpec();+ assertThat(spec.getVolumes()).anySatisfy(volume -> {+ assertThat(volume.getName()).isEqualTo("floci-ca");+ assertThat(volume.getConfigMap().getName()).isEqualTo(KubernetesPodLauncher.CA_CONFIG_MAP_NAME);+ });+ assertThat(spec.getContainers().getFirst().getVolumeMounts()).anySatisfy(volumeMount -> {+ assertThat(volumeMount.getName()).isEqualTo("floci-ca");+ assertThat(volumeMount.getMountPath()).isEqualTo("/etc/floci-ca.crt");+ assertThat(volumeMount.getSubPath()).isEqualTo(KubernetesPodLauncher.CA_CONFIG_MAP_KEY);+ });+ assertThat(spec.getInitContainers().getFirst().getCommand().get(2))+ .contains("wget -q")+ .doesNotContain("--no-check-certificate");+ }++ @Test+ void dollarSignsInCommandAndArgsAreEscapedForKubernetesExpansion() {+ var pod = factory.buildPod("floci-lambda-img-abc12345", "img",+ "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",+ List.of(), null, List.of(), false, null,+ new LambdaPodSpecFactory.ImageConfig(+ List.of("/entry.sh"), List.of("run('$(AWS_REGION)')", "pa$$word"), "/work"),+ 512, Optional.empty());+ assertThat(pod.getSpec().getContainers().getFirst().getArgs())+ .containsExactly("run('$$(AWS_REGION)')", "pa$$$$word");+ }++ @Test+ void imagePackageTypeHasNoInitContainerAndMapsImageConfig() {+ var pod = factory.buildPod("floci-lambda-img-abc12345", "img",+ "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",+ List.of(), null, List.of(), false, null,+ new LambdaPodSpecFactory.ImageConfig(+ List.of("/entry.sh"), List.of("arg1", "arg2"), "/work"),+ 512, Optional.empty());++ var spec = pod.getSpec();+ assertThat(spec.getInitContainers()).isEmpty();+ assertThat(spec.getVolumes()).isEmpty();++ var runtime = spec.getContainers().getFirst();+ assertThat(runtime.getCommand()).containsExactly("/entry.sh");+ assertThat(runtime.getArgs()).containsExactly("arg1", "arg2");+ assertThat(runtime.getWorkingDir()).isEqualTo("/work");+ }++ @Test+ void envEntriesSplitOnFirstEquals() {+ var pod = factory.buildPod("floci-lambda-my-fn-abc12345", "my-fn",+ "public.ecr.aws/lambda/python:3.12",+ List.of("KEY=a=b", "EMPTY="),+ "http://10.0.0.5:4566/b/k", List.of(), false, "h", null, 128, Optional.empty());+ assertThat(pod.getSpec().getContainers().getFirst().getEnv())+ .extracting(EnvVar::getName, EnvVar::getValue)+ .contains(+ tuple("KEY", "a=b"),+ tuple("EMPTY", ""));+ }++ @Test+ void envEntriesWithEmptyNamesAreDropped() {+ // The API server rejects a pod whose env var has an empty name.+ var pod = factory.buildPod("floci-lambda-my-fn-abc12345", "my-fn",+ "public.ecr.aws/lambda/python:3.12",+ List.of("=oops", "KEY=1"),+ "http://10.0.0.5:4566/b/k", List.of(), false, "h", null, 128, Optional.empty());+ assertThat(pod.getSpec().getContainers().getFirst().getEnv())+ .extracting(EnvVar::getName)+ .containsExactly("KEY");+ }++ @Test+ void podNameIsDnsSafeAndClamped() {+ var name = LambdaPodSpecFactory.podName("My_Function.With.Dots", "abc12345");+ assertThat(name).isEqualTo("floci-lambda-my-function-with-dots-abc12345");+ assertThat(name).hasSizeLessThanOrEqualTo(63);++ var longName = LambdaPodSpecFactory.podName("a".repeat(100), "abc12345");+ assertThat(longName)+ .hasSizeLessThanOrEqualTo(63)+ .endsWith("-abc12345")+ .matches("[a-z0-9]([a-z0-9-]*[a-z0-9])?");++ assertThat(LambdaPodSpecFactory.podName("_foo", "abc12345"))+ .isEqualTo("floci-lambda-foo-abc12345");+ assertThat(LambdaPodSpecFactory.podName("___", "abc12345"))+ .isEqualTo("floci-lambda-fn-abc12345");+ }++ @Test+ void labelValueSanitization() {+ assertThat(LambdaPodSpecFactory.sanitizeLabelValue("fn:name")).isEqualTo("fn-name");+ assertThat(LambdaPodSpecFactory.sanitizeLabelValue("-fn-")).isEqualTo("fn");+ assertThat(LambdaPodSpecFactory.sanitizeLabelValue("x".repeat(100))).hasSizeLessThanOrEqualTo(63);+ }+}diff --git a/src/test/resources/application.yml b/src/test/resources/application.ymlindex a13b41277c..5457c7ed3d 100644--- a/src/test/resources/application.yml+++ b/src/test/resources/application.yml@@ -6,6 +6,9 @@ quarkus:max-body-size: ${floci.max-request-size}Mdatasource:active: false+ kubernetes-client:+ devservices:+ enabled: false# Groovy (a REST Assured dependency) starts DFA-cache-cleaner threads that never stop.# Loaded parent-first they live in the shared system classloader, so they cannot pin a# per-profile application classloader across the many app restarts of a test run.