Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(ecs): let the startup container sweep be turned off, and derive the credentials proxy owner from the shared helper (#4696)

Abhigya55d37cfc4117
diff --git a/docs/configuration/advanced/application-yml.md b/docs/configuration/advanced/application-yml.md
index 03bd53a24a..d29b5b5418 100644
--- a/docs/configuration/advanced/application-yml.md
+++ b/docs/configuration/advanced/application-yml.md
@@ -337,6 +337,7 @@ All keys in this table are declared on `EmulatorConfig` and accept environment v
| `FLOCI_SERVICES_ECS_DEFAULT_CPU_UNITS` | `256` | Default CPU units when task definition omits it |
| `FLOCI_SERVICES_ECS_HOST_VOLUME_ROOTS` | *(unset)* | Comma-separated allowlist of parent directories for host volume bind mounts; by default (unset, and `ALLOW_UNSAFE_HOST_VOLUMES=false`) every host volume `sourcePath` is rejected |
| `FLOCI_SERVICES_ECS_ALLOW_UNSAFE_HOST_VOLUMES` | `false` | Allow any host path, bypassing the host-volume-roots allowlist (traversal, bare root, and the Docker socket or an ancestor directory of it are still always rejected) |
+| `FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP` | `true` | Remove the ECS containers a previous run of this Floci left on the daemon before replacement tasks start |
| `FLOCI_SERVICES_IAM_ENFORCEMENT_ENABLED` | `false` | Enforce IAM identity-based policies on every request when `true` |
| `FLOCI_SERVICES_OPENSEARCH_MOCK` | `false` | Skip Docker; domains appear active immediately (useful for CI) |
| `FLOCI_SERVICES_OPENSEARCH_KEEP_RUNNING_ON_SHUTDOWN` | `false` | Leave OpenSearch containers running after Floci stops |
diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.md
index a1edd2ec9a..8e99083f61 100644
--- a/docs/configuration/environment-variables.md
+++ b/docs/configuration/environment-variables.md
@@ -503,6 +503,7 @@ These services spawn Docker containers. They require access to the Docker socket
| `FLOCI_SERVICES_ECS_DOCKER_NETWORK` | _(none)_ | Docker network for ECS task containers |
| `FLOCI_SERVICES_ECS_HOST_VOLUME_ROOTS` | _(none)_ | Comma-separated allowlist of parent directories host volume `sourcePath`s must resolve under. By default (unset, and `ALLOW_UNSAFE_HOST_VOLUMES=false`) every host volume `sourcePath` is rejected |
| `FLOCI_SERVICES_ECS_ALLOW_UNSAFE_HOST_VOLUMES` | `false` | Allow any host path, bypassing `HOST_VOLUME_ROOTS`; traversal, the bare root, and the Docker socket (or an ancestor directory of it, e.g. `/var/run`) are still always rejected |
+| `FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP` | `true` | On startup, in Docker mode, remove the ECS containers a previous run of this Floci left on the daemon (matched by resource namespace and API port) before replacement tasks start |
| `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_ENABLED` | `false` | Vend real task IAM role credentials to task containers over the AWS container-credentials contract. Requires `FLOCI_SERVICES_ECS_DOCKER_NETWORK` to name a user-defined Docker network: the default bridge will not work |
| `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_TTL_SECONDS` | `21600` | Lifetime of the vended credentials. Matches the six hours AWS documents for task-role credentials |
| `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_PORT` | `51679` | Port on the Floci host serving the credentials endpoint. Not the address task containers use, which is always `169.254.170.2:80` |
diff --git a/docs/services/ecs.md b/docs/services/ecs.md
index c1c290f5c7..d4522a62ff 100644
--- a/docs/services/ecs.md
+++ b/docs/services/ecs.md
@@ -524,7 +524,14 @@ nothing manages:
ECS starts no task until it is gone, retrying the removal before each launch: a service's
replacement on a reconciliation tick, or a `RunTask` or `StartTask` call, whose task then stops
with a `TaskFailedToStart` reason naming the leftovers. Each distinct failure is logged as a
- warning once.
+ warning once. Two Floci instances that share a daemon with the same API port and no
+ `FLOCI_DOCKER_RESOURCE_NAMESPACE` carry the same owner label and would remove each other's task
+ containers: give each its own namespace, or set
+ `FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP=false` to skip this sweep, as EC2's
+ `FLOCI_SERVICES_EC2_RECONCILE_CONTAINERS_ON_STARTUP` does for instance containers. It also
+ skips the startup removal of task-role credentials proxies, which carry the same label.
+ Registrations are still released either way, since each Floci releases only what its own
+ persisted state recorded.
- Every load balancer target and Cloud Map instance a previous run's tasks registered is
deregistered. Floci registers task containers by address, and Docker hands a dead container's
address to the next container it starts, so a stale target would route to an unrelated
@@ -750,6 +757,7 @@ Every `awsvpc` task receives an ENI in its subnet. With `FLOCI_NETWORK_SECURITY_
| `FLOCI_SERVICES_ECS_DEFAULT_CPU_UNITS` | `256` | Default CPU units when the task definition omits it |
| `FLOCI_SERVICES_ECS_HOST_VOLUME_ROOTS` | *(unset)* | Approved parent directories for host volume bind mounts (`volumes[].host.sourcePath`) |
| `FLOCI_SERVICES_ECS_ALLOW_UNSAFE_HOST_VOLUMES` | `false` | Allow any host path, bypassing the `HOST_VOLUME_ROOTS` allowlist; traversal, the bare root, and the Docker socket are still always rejected |
+| `FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP` | `true` | On startup, in Docker mode, remove the ECS containers a previous run of this Floci left on the daemon before the scheduler starts replacement tasks (see [Restarts](#restarts)) |
| `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_ENABLED` | `false` | Vend real task IAM role credentials to task containers |
| `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_TTL_SECONDS` | `21600` | Lifetime of the vended credentials |
| `FLOCI_SERVICES_ECS_TASK_ROLE_CREDENTIALS_PORT` | `51679` | Floci-side port serving the credentials endpoint |
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 3593ef73db..7f0664c298 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -2098,6 +2098,18 @@ public interface EmulatorConfig {
@WithDefault("false")
boolean allowUnsafeHostVolumes();
+ /**
+ * When true, Floci removes on startup, in Docker mode, every ECS container a previous run
+ * of <em>this same</em> Floci left on the daemon (matched by the {@code floci_owner_port}
+ * label), task-role credentials proxies included, before the service scheduler starts
+ * replacement tasks. Turn it off when two Floci instances share a daemon with the same
+ * port and no resource namespace, so one does not remove the other's containers.
+ *
+ * Env var: FLOCI_SERVICES_ECS_RECONCILE_CONTAINERS_ON_STARTUP
+ */
+ @WithDefault("true")
+ boolean reconcileContainersOnStartup();
+
EcsTaskRoleCredentialsConfig taskRoleCredentials();
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java
index 5de43acdbb..ab63043d68 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java
@@ -114,6 +114,7 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
// what every such caller already expects.
private final EcsServiceDiscoveryRegistrar discoveryRegistrar;
private final boolean dockerMode;
+ private final boolean sweepLeftoverContainers;
private final String baseUrl;
// Replaced by afterReset() after a state reset, whose container teardown shuts this scheduler down.
private volatile ScheduledExecutorService reconciler = newReconciler();
@@ -210,6 +211,7 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
this.regionResolver = regionResolver;
this.containerManager = containerManager;
this.dockerMode = !config.services().ecs().mock();
+ this.sweepLeftoverContainers = dockerMode && config.services().ecs().reconcileContainersOnStartup();
this.baseUrl = config.effectiveBaseUrl();
this.lbRegistrar = lbRegistrar;
this.storageFactory = storageFactory;
@@ -242,10 +244,11 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
* target would follow its address to whichever container Docker hands it to next. Only what
* the registrars recorded registering is released, never a target or instance registered by
* hand. A container Docker would not remove is retried by {@link #leftoverContainersCleared()}
- * before each task launch, and no task starts until it is gone.
+ * before each task launch, and no task starts until it is gone. With
+ * {@code reconcile-containers-on-startup} off, containers are left alone and tasks start at once.
*/
void releasePreviousRunLeftovers() {
- if (dockerMode) {
+ if (sweepLeftoverContainers) {
leftoverContainersRemoved = containerManager.removeLeftoverContainers();
}
if (lbRegistrar != null) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxy.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxy.java
index 3e6b8ed01a..3b861f9ce9 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxy.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxy.java
@@ -74,14 +74,21 @@ public class EcsCredentialsProxy implements ContainerTeardown {
* SecurityGroupFirewallManager} uses: several Floci processes can share one Docker daemon, and
* a label match alone would let one instance's restart tear down another's still-running,
* still-in-use proxy.
+ *
+ * <p>Skipped with {@code reconcile-containers-on-startup} off, like the task container sweep:
+ * that is the setting for instances sharing one owner identity, where a proxy carrying this
+ * owner label may still serve another instance's running tasks.
*/
@PostConstruct
void reapSurvivingProxies() {
+ if (!config.services().ecs().reconcileContainersOnStartup()) {
+ return;
+ }
try {
dockerClient.listContainersCmd().withShowAll(true)
.withLabelFilter(Map.of(OWNS_NETWORK_LABEL, "true"))
.exec().stream()
- .filter(container -> owner().equals(container.getLabels().get("floci_owner_port")))
+ .filter(container -> owner().equals(container.getLabels().get(ContainerStorageHelper.OWNER_LABEL)))
.forEach(container -> lifecycleManager.removeIfExists(container.getId()));
} catch (Exception e) {
LOG.warnv("Could not reap surviving ECS credentials proxy containers: {0}", e.getMessage());
@@ -90,8 +97,7 @@ public class EcsCredentialsProxy implements ContainerTeardown {
/** Identifies this Floci process among others that may share the same Docker daemon. */
String owner() {
- String namespace = config.docker().resourceNamespace().orElse("");
- return namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();
+ return ContainerStorageHelper.ownerIdentity(config);
}
/**
@@ -148,7 +154,7 @@ public class EcsCredentialsProxy implements ContainerTeardown {
.withCmd(List.of(
"TCP-LISTEN:80,bind=" + CREDENTIALS_ADDRESS + ",fork,reuseaddr",
"TCP:" + flociHost + ":" + port))
- .withLabels(Map.of(OWNS_NETWORK_LABEL, "true", "floci_owner_port", owner()))
+ .withLabels(Map.of(OWNS_NETWORK_LABEL, "true", ContainerStorageHelper.OWNER_LABEL, owner()))
.build();
String proxyId = null;
@@ -198,7 +204,7 @@ public class EcsCredentialsProxy implements ContainerTeardown {
}
Map<String, String> labels = candidate.getLabels();
boolean isOurProxy = labels != null && "true".equals(labels.get(OWNS_NETWORK_LABEL));
- String candidateOwner = labels == null ? null : labels.get("floci_owner_port");
+ String candidateOwner = labels == null ? null : labels.get(ContainerStorageHelper.OWNER_LABEL);
if (isOurProxy && (candidateOwner == null || owner().equals(candidateOwner))) {
lifecycleManager.removeIfExists(candidate.getId());
}
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 9a61471988..41ffa24eca 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -600,6 +600,9 @@ floci:
publish-awsvpc-ports-to-host: false
host-volume-roots: []
allow-unsafe-host-volumes: false
+ # Remove the ECS containers a previous run of this deployment left on the daemon before
+ # the service scheduler starts replacements. Scoped to this deployment's own containers.
+ reconcile-containers-on-startup: true
task-role-credentials:
enabled: false
ttl-seconds: 21600
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java
index 7de7cadfe3..4327722b81 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java
@@ -140,6 +140,29 @@ class EcsServiceRestartLeftoversTest {
verify(containerManager, never()).removeLeftoverContainers();
}
+ @Test
+ void optingOutLeavesThePreviousRunsContainersButReleasesItsRegistrations() {
+ EcsContainerManager containerManager = mock(EcsContainerManager.class);
+ EcsLoadBalancerRegistrar lbRegistrar = mock(EcsLoadBalancerRegistrar.class);
+ EcsServiceDiscoveryRegistrar discoveryRegistrar = mock(EcsServiceDiscoveryRegistrar.class);
+ EcsService restarted = service(new SharedStorageFactory(), false, false, containerManager,
+ lbRegistrar, discoveryRegistrar);
+ restarted.createCluster("app-cluster", Map.of(), REGION);
+ ContainerDefinition container = new ContainerDefinition();
+ container.setName("web");
+ container.setImage("nginx:alpine");
+ restarted.registerTaskDefinition("web", List.of(container), NetworkMode.bridge, null, null,
+ null, null, List.of(), REGION);
+
+ restarted.releasePreviousRunLeftovers();
+ restarted.runTask("app-cluster", "web", 1, LaunchType.EC2, null, null, null, null, REGION);
+
+ verify(containerManager, never()).removeLeftoverContainers();
+ verify(containerManager).startTask(any(), any(), any(), anyString());
+ verify(lbRegistrar).releaseRecordedTargets();
+ verify(discoveryRegistrar).releaseRecordedInstances();
+ }
+
@Test
void startupReleasesLeftoversBeforeTheSchedulerRuns() {
EcsContainerManager containerManager = mock(EcsContainerManager.class);
@@ -223,8 +246,17 @@ class EcsServiceRestartLeftoversTest {
EcsContainerManager containerManager,
EcsLoadBalancerRegistrar lbRegistrar,
EcsServiceDiscoveryRegistrar discoveryRegistrar) {
+ return service(storage, mockMode, true, containerManager, lbRegistrar, discoveryRegistrar);
+ }
+
+ private static EcsService service(StorageFactory storage, boolean mockMode,
+ boolean reconcileContainersOnStartup,
+ EcsContainerManager containerManager,
+ EcsLoadBalancerRegistrar lbRegistrar,
+ EcsServiceDiscoveryRegistrar discoveryRegistrar) {
EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
when(config.services().ecs().mock()).thenReturn(mockMode);
+ when(config.services().ecs().reconcileContainersOnStartup()).thenReturn(reconcileContainersOnStartup);
when(config.effectiveBaseUrl()).thenReturn("http://localhost:4566");
EcsService service = new EcsService(new RegionResolver(REGION, "000000000000"), containerManager,
config, lbRegistrar, storage, null, new EcsExecSessionRegistry(), discoveryRegistrar);
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxyTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxyTest.java
index 712e12838c..41af460195 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxyTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsCredentialsProxyTest.java
@@ -224,6 +224,7 @@ class EcsCredentialsProxyTest {
@Test
void reapSurvivingProxiesRemovesOnlyThisInstancesOwnLeftovers() {
+ when(config.services().ecs().reconcileContainersOnStartup()).thenReturn(true);
when(config.docker().resourceNamespace()).thenReturn(Optional.empty());
when(config.port()).thenReturn(4566);
ListContainersCmd listCmd = mock(ListContainersCmd.class, RETURNS_SELF);
@@ -244,8 +245,19 @@ class EcsCredentialsProxyTest {
verify(lifecycleManager, never()).removeIfExists("other-instance-proxy");
}
+ @Test
+ void reapSurvivingProxiesLeavesProxiesAloneWhenStartupReconciliationIsOff() {
+ when(config.services().ecs().reconcileContainersOnStartup()).thenReturn(false);
+
+ proxy.reapSurvivingProxies();
+
+ verify(dockerClient, never()).listContainersCmd();
+ verify(lifecycleManager, never()).removeIfExists(any());
+ }
+
@Test
void reapSurvivingProxiesToleratesADockerFailure() {
+ when(config.services().ecs().reconcileContainersOnStartup()).thenReturn(true);
when(dockerClient.listContainersCmd()).thenThrow(new RuntimeException("daemon unreachable"));
proxy.reapSurvivingProxies();