Light, fluffy, and always free - The AWS Local Emulator alternative
feat(ecs): let the startup container sweep be turned off, and derive the credentials proxy owner from the shared helper (#4696)
Abhigya55d37cfc4117
diff --git a/docs/configuration/advanced/application-yml.md b/docs/configuration/advanced/application-yml.mdindex 03bd53a24a..d29b5b5418 100644--- a/docs/configuration/advanced/application-yml.md+++ b/docs/configuration/advanced/application-yml.md@@ -337,6 +337,7 @@ All keys in this table are declared on `EmulatorConfig` and accept environment v| `FLOCI_SERVICES_ECS_DEFAULT_CPU_UNITS` | `256` | Default CPU units when task definition omits it || `FLOCI_SERVICES_ECS_HOST_VOLUME_ROOTS` | *(unset)* | Comma-separated allowlist of parent directories for host volume bind mounts; by default (unset, and `ALLOW_UNSAFE_HOST_VOLUMES=false`) every host volume `sourcePath` is rejected || `FLOCI_SERVICES_ECS_ALLOW_UNSAFE_HOST_VOLUMES` | `false` | Allow any host path, bypassing the host-volume-roots allowlist (traversal, bare root, and the Docker socket or an ancestor directory of it are still always rejected) |+| `FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP` | `true` | Remove the ECS containers a previous run of this Floci left on the daemon before replacement tasks start || `FLOCI_SERVICES_IAM_ENFORCEMENT_ENABLED` | `false` | Enforce IAM identity-based policies on every request when `true` || `FLOCI_SERVICES_OPENSEARCH_MOCK` | `false` | Skip Docker; domains appear active immediately (useful for CI) || `FLOCI_SERVICES_OPENSEARCH_KEEP_RUNNING_ON_SHUTDOWN` | `false` | Leave OpenSearch containers running after Floci stops |diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.mdindex a1edd2ec9a..8e99083f61 100644--- a/docs/configuration/environment-variables.md+++ b/docs/configuration/environment-variables.md@@ -503,6 +503,7 @@ These services spawn Docker containers. They require access to the Docker socket| `FLOCI_SERVICES_ECS_DOCKER_NETWORK` | _(none)_ | Docker network for ECS task containers || `FLOCI_SERVICES_ECS_HOST_VOLUME_ROOTS` | _(none)_ | Comma-separated allowlist of parent directories host volume `sourcePath`s must resolve under. By default (unset, and `ALLOW_UNSAFE_HOST_VOLUMES=false`) every host volume `sourcePath` is rejected || `FLOCI_SERVICES_ECS_ALLOW_UNSAFE_HOST_VOLUMES` | `false` | Allow any host path, bypassing `HOST_VOLUME_ROOTS`; traversal, the bare root, and the Docker socket (or an ancestor directory of it, e.g. `/var/run`) are still always rejected |+| `FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP` | `true` | On startup, in Docker mode, remove the ECS containers a previous run of this Floci left on the daemon (matched by resource namespace and API port) before replacement tasks start || `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_ENABLED` | `false` | Vend real task IAM role credentials to task containers over the AWS container-credentials contract. Requires `FLOCI_SERVICES_ECS_DOCKER_NETWORK` to name a user-defined Docker network: the default bridge will not work || `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_TTL_SECONDS` | `21600` | Lifetime of the vended credentials. Matches the six hours AWS documents for task-role credentials || `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_PORT` | `51679` | Port on the Floci host serving the credentials endpoint. Not the address task containers use, which is always `169.254.170.2:80` |diff --git a/docs/services/ecs.md b/docs/services/ecs.mdindex c1c290f5c7..d4522a62ff 100644--- a/docs/services/ecs.md+++ b/docs/services/ecs.md@@ -524,7 +524,14 @@ nothing manages:ECS starts no task until it is gone, retrying the removal before each launch: a service'sreplacement on a reconciliation tick, or a `RunTask` or `StartTask` call, whose task then stopswith a `TaskFailedToStart` reason naming the leftovers. Each distinct failure is logged as a- warning once.+ warning once. Two Floci instances that share a daemon with the same API port and no+ `FLOCI_DOCKER_RESOURCE_NAMESPACE` carry the same owner label and would remove each other's task+ containers: give each its own namespace, or set+ `FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP=false` to skip this sweep, as EC2's+ `FLOCI_SERVICES_EC2_RECONCILE_CONTAINERS_ON_STARTUP` does for instance containers. It also+ skips the startup removal of task-role credentials proxies, which carry the same label.+ Registrations are still released either way, since each Floci releases only what its own+ persisted state recorded.- Every load balancer target and Cloud Map instance a previous run's tasks registered isderegistered. Floci registers task containers by address, and Docker hands a dead container'saddress to the next container it starts, so a stale target would route to an unrelated@@ -750,6 +757,7 @@ Every `awsvpc` task receives an ENI in its subnet. With `FLOCI_NETWORK_SECURITY_| `FLOCI_SERVICES_ECS_DEFAULT_CPU_UNITS` | `256` | Default CPU units when the task definition omits it || `FLOCI_SERVICES_ECS_HOST_VOLUME_ROOTS` | *(unset)* | Approved parent directories for host volume bind mounts (`volumes[].host.sourcePath`) || `FLOCI_SERVICES_ECS_ALLOW_UNSAFE_HOST_VOLUMES` | `false` | Allow any host path, bypassing the `HOST_VOLUME_ROOTS` allowlist; traversal, the bare root, and the Docker socket are still always rejected |+| `FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP` | `true` | On startup, in Docker mode, remove the ECS containers a previous run of this Floci left on the daemon before the scheduler starts replacement tasks (see [Restarts](#restarts)) || `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_ENABLED` | `false` | Vend real task IAM role credentials to task containers || `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_TTL_SECONDS` | `21600` | Lifetime of the vended credentials || `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_PORT` | `51679` | Floci-side port serving the credentials endpoint |diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.javaindex 3593ef73db..7f0664c298 100644--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java@@ -2098,6 +2098,18 @@ public interface EmulatorConfig {@WithDefault("false")boolean allowUnsafeHostVolumes();+ /**+ * When true, Floci removes on startup, in Docker mode, every ECS container a previous run+ * of <em>this same</em> Floci left on the daemon (matched by the {@code floci_owner_port}+ * label), task-role credentials proxies included, before the service scheduler starts+ * replacement tasks. Turn it off when two Floci instances share a daemon with the same+ * port and no resource namespace, so one does not remove the other's containers.+ *+ * Env var: FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP+ */+ @WithDefault("true")+ boolean reconcileContainersOnStartup();+EcsTaskRoleCredentialsConfig taskRoleCredentials();}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.javaindex 5de43acdbb..ab63043d68 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java@@ -114,6 +114,7 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab// what every such caller already expects.private final EcsServiceDiscoveryRegistrar discoveryRegistrar;private final boolean dockerMode;+ private final boolean sweepLeftoverContainers;private final String baseUrl;// Replaced by afterReset() after a state reset, whose container teardown shuts this scheduler down.private volatile ScheduledExecutorService reconciler = newReconciler();@@ -210,6 +211,7 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettabthis.regionResolver = regionResolver;this.containerManager = containerManager;this.dockerMode = !config.services().ecs().mock();+ this.sweepLeftoverContainers = dockerMode && config.services().ecs().reconcileContainersOnStartup();this.baseUrl = config.effectiveBaseUrl();this.lbRegistrar = lbRegistrar;this.storageFactory = storageFactory;@@ -242,10 +244,11 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab* target would follow its address to whichever container Docker hands it to next. Only what* the registrars recorded registering is released, never a target or instance registered by* hand. A container Docker would not remove is retried by {@link #leftoverContainersCleared()}- * before each task launch, and no task starts until it is gone.+ * before each task launch, and no task starts until it is gone. With+ * {@code reconcile-containers-on-startup} off, containers are left alone and tasks start at once.*/void releasePreviousRunLeftovers() {- if (dockerMode) {+ if (sweepLeftoverContainers) {leftoverContainersRemoved = containerManager.removeLeftoverContainers();}if (lbRegistrar != null) {diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxy.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxy.javaindex 3e6b8ed01a..3b861f9ce9 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxy.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxy.java@@ -74,14 +74,21 @@ public class EcsCredentialsProxy implements ContainerTeardown {* SecurityGroupFirewallManager} uses: several Floci processes can share one Docker daemon, and* a label match alone would let one instance's restart tear down another's still-running,* still-in-use proxy.+ *+ * <p>Skipped with {@code reconcile-containers-on-startup} off, like the task container sweep:+ * that is the setting for instances sharing one owner identity, where a proxy carrying this+ * owner label may still serve another instance's running tasks.*/@PostConstructvoid reapSurvivingProxies() {+ if (!config.services().ecs().reconcileContainersOnStartup()) {+ return;+ }try {dockerClient.listContainersCmd().withShowAll(true).withLabelFilter(Map.of(OWNS_NETWORK_LABEL, "true")).exec().stream()- .filter(container -> owner().equals(container.getLabels().get("floci_owner_port")))+ .filter(container -> owner().equals(container.getLabels().get(ContainerStorageHelper.OWNER_LABEL))).forEach(container -> lifecycleManager.removeIfExists(container.getId()));} catch (Exception e) {LOG.warnv("Could not reap surviving ECS credentials proxy containers: {0}", e.getMessage());@@ -90,8 +97,7 @@ public class EcsCredentialsProxy implements ContainerTeardown {/** Identifies this Floci process among others that may share the same Docker daemon. */String owner() {- String namespace = config.docker().resourceNamespace().orElse("");- return namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();+ return ContainerStorageHelper.ownerIdentity(config);}/**@@ -148,7 +154,7 @@ public class EcsCredentialsProxy implements ContainerTeardown {.withCmd(List.of("TCP-LISTEN:80,bind=" + CREDENTIALS_ADDRESS + ",fork,reuseaddr","TCP:" + flociHost + ":" + port))- .withLabels(Map.of(OWNS_NETWORK_LABEL, "true", "floci_owner_port", owner()))+ .withLabels(Map.of(OWNS_NETWORK_LABEL, "true", ContainerStorageHelper.OWNER_LABEL, owner())).build();String proxyId = null;@@ -198,7 +204,7 @@ public class EcsCredentialsProxy implements ContainerTeardown {}Map<String, String> labels = candidate.getLabels();boolean isOurProxy = labels != null && "true".equals(labels.get(OWNS_NETWORK_LABEL));- String candidateOwner = labels == null ? null : labels.get("floci_owner_port");+ String candidateOwner = labels == null ? null : labels.get(ContainerStorageHelper.OWNER_LABEL);if (isOurProxy && (candidateOwner == null || owner().equals(candidateOwner))) {lifecycleManager.removeIfExists(candidate.getId());}diff --git a/src/main/resources/application.yml b/src/main/resources/application.ymlindex 9a61471988..41ffa24eca 100644--- a/src/main/resources/application.yml+++ b/src/main/resources/application.yml@@ -600,6 +600,9 @@ floci:publish-awsvpc-ports-to-host: falsehost-volume-roots: []allow-unsafe-host-volumes: false+ # Remove the ECS containers a previous run of this deployment left on the daemon before+ # the service scheduler starts replacements. Scoped to this deployment's own containers.+ reconcile-containers-on-startup: truetask-role-credentials:enabled: falsettl-seconds: 21600diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.javaindex 7de7cadfe3..4327722b81 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java@@ -140,6 +140,29 @@ class EcsServiceRestartLeftoversTest {verify(containerManager, never()).removeLeftoverContainers();}+ @Test+ void optingOutLeavesThePreviousRunsContainersButReleasesItsRegistrations() {+ EcsContainerManager containerManager = mock(EcsContainerManager.class);+ EcsLoadBalancerRegistrar lbRegistrar = mock(EcsLoadBalancerRegistrar.class);+ EcsServiceDiscoveryRegistrar discoveryRegistrar = mock(EcsServiceDiscoveryRegistrar.class);+ EcsService restarted = service(new SharedStorageFactory(), false, false, containerManager,+ lbRegistrar, discoveryRegistrar);+ restarted.createCluster("app-cluster", Map.of(), REGION);+ ContainerDefinition container = new ContainerDefinition();+ container.setName("web");+ container.setImage("nginx:alpine");+ restarted.registerTaskDefinition("web", List.of(container), NetworkMode.bridge, null, null,+ null, null, List.of(), REGION);++ restarted.releasePreviousRunLeftovers();+ restarted.runTask("app-cluster", "web", 1, LaunchType.EC2, null, null, null, null, REGION);++ verify(containerManager, never()).removeLeftoverContainers();+ verify(containerManager).startTask(any(), any(), any(), anyString());+ verify(lbRegistrar).releaseRecordedTargets();+ verify(discoveryRegistrar).releaseRecordedInstances();+ }+@Testvoid startupReleasesLeftoversBeforeTheSchedulerRuns() {EcsContainerManager containerManager = mock(EcsContainerManager.class);@@ -223,8 +246,17 @@ class EcsServiceRestartLeftoversTest {EcsContainerManager containerManager,EcsLoadBalancerRegistrar lbRegistrar,EcsServiceDiscoveryRegistrar discoveryRegistrar) {+ return service(storage, mockMode, true, containerManager, lbRegistrar, discoveryRegistrar);+ }++ private static EcsService service(StorageFactory storage, boolean mockMode,+ boolean reconcileContainersOnStartup,+ EcsContainerManager containerManager,+ EcsLoadBalancerRegistrar lbRegistrar,+ EcsServiceDiscoveryRegistrar discoveryRegistrar) {EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);when(config.services().ecs().mock()).thenReturn(mockMode);+ when(config.services().ecs().reconcileContainersOnStartup()).thenReturn(reconcileContainersOnStartup);when(config.effectiveBaseUrl()).thenReturn("http://localhost:4566");EcsService service = new EcsService(new RegionResolver(REGION, "000000000000"), containerManager,config, lbRegistrar, storage, null, new EcsExecSessionRegistry(), discoveryRegistrar);diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxyTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxyTest.javaindex 712e12838c..41af460195 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxyTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxyTest.java@@ -224,6 +224,7 @@ class EcsCredentialsProxyTest {@Testvoid reapSurvivingProxiesRemovesOnlyThisInstancesOwnLeftovers() {+ when(config.services().ecs().reconcileContainersOnStartup()).thenReturn(true);when(config.docker().resourceNamespace()).thenReturn(Optional.empty());when(config.port()).thenReturn(4566);ListContainersCmd listCmd = mock(ListContainersCmd.class, RETURNS_SELF);@@ -244,8 +245,19 @@ class EcsCredentialsProxyTest {verify(lifecycleManager, never()).removeIfExists("other-instance-proxy");}+ @Test+ void reapSurvivingProxiesLeavesProxiesAloneWhenStartupReconciliationIsOff() {+ when(config.services().ecs().reconcileContainersOnStartup()).thenReturn(false);++ proxy.reapSurvivingProxies();++ verify(dockerClient, never()).listContainersCmd();+ verify(lifecycleManager, never()).removeIfExists(any());+ }+@Testvoid reapSurvivingProxiesToleratesADockerFailure() {+ when(config.services().ecs().reconcileContainersOnStartup()).thenReturn(true);when(dockerClient.listContainersCmd()).thenThrow(new RuntimeException("daemon unreachable"));proxy.reapSurvivingProxies();