Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

fix(ecs): release stale task containers and registrations, and evict ones that would misroute a reused address (#4513)

Abhigya451a30f98e9b
diff --git a/docs/services/ecs.md b/docs/services/ecs.md
index 037e7b408b..92fa31d403 100644
--- a/docs/services/ecs.md
+++ b/docs/services/ecs.md
@@ -496,6 +496,45 @@ Known differences from AWS:
registers `HEALTHY`, which is Cloud Map's own default for a `RegisterInstance` that names no
`AWS_INIT_HEALTH_STATUS`, and stays there until it is deregistered.
+#### Restarts
+
+Task state is memory-only, so a restarted Floci knows no task and the service scheduler starts
+new ones to reach each service's desired count. Floci therefore clears what the previous run's
+tasks left behind before it starts their replacements, so a service is never served by a task
+nothing manages:
+
+- In Docker mode, every ECS container a previous run of this Floci left on the daemon is
+ removed. A graceful shutdown already stops them; this covers a run that ended without one
+ (SIGKILL, OOM, a stop timeout that expired mid-drain). Containers are recognised by the
+ `floci_owner_port` label (the resource namespace and API port), so the containers of another
+ Floci sharing the daemon stay, and are told apart from the current run's by a per-process
+ `floci.ecs-run` label rather than by creation time. Containers created by a Floci version without that label are
+ not recognised and must be removed by hand once. If Docker cannot list or remove one, the
+ ECS starts no task until it is gone, retrying the removal before each launch: a service's
+ replacement on a reconciliation tick, or a `RunTask` or `StartTask` call, whose task then stops
+ with a `TaskFailedToStart` reason naming the leftovers. Each distinct failure is logged as a
+ warning once.
+- Every load balancer target and Cloud Map instance a previous run's tasks registered is
+ deregistered. Floci registers task containers by address, and Docker hands a dead container's
+ address to the next container it starts, so a stale target would route to an unrelated
+ container. ECS records each registration it makes in its persisted state and releases exactly
+ those, so a target or instance registered through the ELBv2 or Cloud Map API, even into the
+ same target group or Cloud Map service, stays. A registration is recorded before it is made,
+ so one cut short by the process dying is still released. A target already in its group when a
+ task starts is recorded as the task's own, since its address and port reach only that task, so
+ it is deregistered when the task stops, as ECS does on AWS.
+- Registrations no ECS task recorded, as those a Floci version without the record made, are
+ removed once they would misroute: when a task starts, each registration that points at it and
+ that no running task recorded is deregistered from the target groups and Cloud Map services an
+ ECS service names, and a warning names it. Such an entry was left by a task that is gone, and
+ would otherwise send another service's traffic to the new task. A target points at the task
+ when it is at its container's address on the Docker network and on a port the container
+ publishes. A Cloud Map instance points at it when it is at that container address, on any port
+ since a DNS answer carries none, or at the task's ENI address in a namespace of the task's own
+ VPC, as ENI addresses are unique only within a VPC. A loopback address, which every task
+ published on the host shares, matches only on the task's host ports. Target groups and Cloud
+ Map services no ECS service names are never touched.
+
#### Unknown services
A service reference that does not resolve is returned in `failures` with
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.java
index 33f0eceeae..f9e3eba1c4 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.java
@@ -35,6 +35,8 @@ public final class ContainerStorageHelper {
/** {@link #NAME_PREFIX} with its separator, the literal prefix of every name produced here. */
static final String CONTAINER_PREFIX = NAME_PREFIX + "-";
+ /** The label carrying {@link #ownerIdentity(EmulatorConfig)}. */
+ public static final String OWNER_LABEL = "floci_owner_port";
/**
* The prefix this emulator used before it took the shared {@code floci-<cloud>-} convention.
@@ -184,6 +186,18 @@ public final class ContainerStorageHelper {
return labels;
}
+ /**
+ * Identity of the Floci deployment that owns a container, stamped as {@link #OWNER_LABEL} so a
+ * startup sweep collects only its own leftovers. The API port alone collides when two
+ * independently namespaced Flocis share a Docker daemon on the same internal port, so the
+ * resource namespace goes in front of it; an unnamespaced Floci keeps the bare port.
+ */
+ public static String ownerIdentity(EmulatorConfig config) {
+ String namespace = config.docker() == null || config.docker().resourceNamespace() == null
+ ? "" : config.docker().resourceNamespace().orElse("");
+ return namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();
+ }
+
private static void putIfNotBlank(Map<String, String> labels, String key, String value) {
if (value != null && !value.isBlank()) {
labels.put(key, value);
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java
index 6e12d7ffa1..8679095119 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java
@@ -116,7 +116,7 @@ public class Ec2ContainerManager {
* unless a resource namespace is configured, so it cannot scope the default configuration.
* Containers created before this label existed carry no owner and are therefore never swept.
*/
- static final String LABEL_OWNER_PORT = "floci_owner_port";
+ static final String LABEL_OWNER_PORT = ContainerStorageHelper.OWNER_LABEL;
/**
* Identity of the Floci deployment that owns a container, for scoping the startup sweep.
@@ -126,9 +126,7 @@ public class Ec2ContainerManager {
* unnamespaced single Floci keeps the bare port it already stamped.
*/
private String ownerIdentity() {
- String ns = config.docker() == null || config.docker().resourceNamespace() == null
- ? "" : config.docker().resourceNamespace().orElse("");
- return ns.isBlank() ? String.valueOf(config.port()) : ns + "/" + config.port();
+ return ContainerStorageHelper.ownerIdentity(config);
}
static final String LABEL_SERVICE = "io.floci.service";
static final String SERVICE_VALUE = "ec2";
@@ -192,7 +190,7 @@ public class Ec2ContainerManager {
if (!"true".equals(labels.get("floci.security-group-helper"))
|| !"ec2".equals(labels.get(LABEL_SERVICE))
|| !instance.getInstanceId().equals(labels.get(LABEL_RESOURCE_ID))
- || !ownerIdentity().equals(labels.get("floci_owner_port"))) {
+ || !ownerIdentity().equals(labels.get(LABEL_OWNER_PORT))) {
throw new IllegalStateException("EC2 workload network namespace is not Floci protected");
}
String address = helper.getNetworkSettings().getNetworks().values().stream()
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java
index e6590faad0..ed0ef2d1b7 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java
@@ -67,12 +67,11 @@ public class SecurityGroupFirewallManager {
return;
}
try {
- String namespace = config.docker().resourceNamespace().orElse("");
- String owner = namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();
+ String owner = ContainerStorageHelper.ownerIdentity(config);
dockerClient.listContainersCmd().withShowAll(true)
.withLabelFilter(Map.of("floci.security-group-helper", "true"))
.exec().stream()
- .filter(container -> owner.equals(container.getLabels().get("floci_owner_port")))
+ .filter(container -> owner.equals(container.getLabels().get(ContainerStorageHelper.OWNER_LABEL)))
.forEach(container -> {
if ("running".equals(container.getState())) {
quarantine(container.getId());
@@ -92,6 +91,13 @@ public class SecurityGroupFirewallManager {
/** The helper starts before any workload process and owns all published ports. */
public Namespace createNamespace(String service, String resourceId, String accountId, String region,
Optional<String> dockerNetwork, Map<Integer, Integer> portBindings) {
+ return createNamespace(service, resourceId, accountId, region, dockerNetwork, portBindings, Map.of());
+ }
+
+ /** {@link #createNamespace} with extra labels on the helper, as the owning service tracks it by. */
+ public Namespace createNamespace(String service, String resourceId, String accountId, String region,
+ Optional<String> dockerNetwork, Map<Integer, Integer> portBindings,
+ Map<String, String> extraLabels) {
if (!enabled()) {
throw new IllegalStateException("Security-group enforcement is disabled");
}
@@ -106,8 +112,6 @@ public class SecurityGroupFirewallManager {
ensureHelperImage();
String name = ContainerStorageHelper.resourceName(config, "sg", null,
resourceId.replaceAll("[^a-zA-Z0-9_.-]", "-"));
- String namespace = config.docker().resourceNamespace().orElse("");
- String owner = namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();
ContainerBuilder.Builder builder = containerBuilder.newContainer(config.network().securityGroupEnforcement().helperImage())
.withName(name)
.withDockerNetwork(dockerNetwork)
@@ -117,7 +121,9 @@ public class SecurityGroupFirewallManager {
.withEntrypoint(List.of("sh", "-c"))
.withCmd(List.of("exec sleep 2147483647"))
.withLabels(ContainerStorageHelper.resourceIdentityLabels(service, resourceId, accountId, region))
- .withLabels(Map.of("floci.security-group-helper", "true", "floci_owner_port", owner));
+ .withLabels(Map.of("floci.security-group-helper", "true",
+ ContainerStorageHelper.OWNER_LABEL, ContainerStorageHelper.ownerIdentity(config)))
+ .withLabels(extraLabels);
if (portBindings != null) {
portBindings.forEach(builder::withPortBinding);
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrar.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrar.java
index d8d4372001..14e8db8500 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrar.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrar.java
@@ -1,18 +1,31 @@
package io.github.hectorvent.floci.services.ecs;
+import com.fasterxml.jackson.core.type.TypeReference;
+import io.github.hectorvent.floci.core.common.RequestScopes;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;
import io.github.hectorvent.floci.services.ecs.model.Container;
import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;
+import io.github.hectorvent.floci.services.ecs.model.EcsRegisteredTargets;
import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;
import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;
import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;
import io.github.hectorvent.floci.services.elbv2.ElbV2Service;
import io.github.hectorvent.floci.services.elbv2.model.TargetDescription;
+import io.github.hectorvent.floci.services.elbv2.model.TargetHealth;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import org.jboss.logging.Logger;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
import java.util.function.BiConsumer;
/**
@@ -30,37 +43,155 @@ public class EcsLoadBalancerRegistrar {
private final ElbV2Service elbV2Service;
private final EcsContainerManager containerManager;
+ // taskArn → the targets registered for it, persisted so they can be released after a restart
+ private final AccountAwareStorageBackend<EcsRegisteredTargets> ledger;
@Inject
- public EcsLoadBalancerRegistrar(ElbV2Service elbV2Service, EcsContainerManager containerManager) {
+ public EcsLoadBalancerRegistrar(ElbV2Service elbV2Service, EcsContainerManager containerManager,
+ StorageFactory storageFactory) {
this.elbV2Service = elbV2Service;
this.containerManager = containerManager;
+ this.ledger = storageFactory.create("ecs", "ecs-registered-targets.json",
+ new TypeReference<Map<String, EcsRegisteredTargets>>() {});
}
- /** Registers the task's load-balanced containers as ELBv2 targets. */
+ /**
+ * Registers the task's load-balanced containers as ELBv2 targets. The targets are recorded
+ * before they are registered, so a process killed in between still leaves a record for the
+ * next run to release. A target already in its group is recorded too: its address and port
+ * are the task's own container, so it reaches nothing else, and once the task stops it would
+ * reach nothing or whichever container Docker hands the address to next. ECS on AWS likewise
+ * deregisters a stopping task's target whoever registered it.
+ */
public void registerTask(EcsTask task, EcsServiceModel svc, String region) {
- forEachTarget(task, svc, (tgArn, td) -> {
+ List<EcsRegisteredTargets.Target> planned = new ArrayList<>();
+ forEachTarget(task, svc, (tgArn, td) ->
+ planned.add(new EcsRegisteredTargets.Target(tgArn, td.getId(), td.getPort())));
+ if (planned.isEmpty()) {
+ return;
+ }
+ String taskArn = task.getTaskArn();
+ if (taskArn != null) {
+ ledger.put(taskArn, new EcsRegisteredTargets(region, List.copyOf(planned)));
+ }
+ List<EcsRegisteredTargets.Target> registered = new ArrayList<>();
+ for (EcsRegisteredTargets.Target target : planned) {
try {
- elbV2Service.registerTargets(region, tgArn, List.of(td));
+ elbV2Service.registerTargets(region, target.targetGroupArn(), List.of(target.toDescription()));
+ registered.add(target);
LOG.infov("Registered ECS task target {0}:{1} into target group {2}",
- td.getId(), td.getPort(), tgArn);
+ target.id(), target.port(), target.targetGroupArn());
} catch (Exception e) {
- LOG.warnv("Could not register ECS target into {0}: {1}", tgArn, e.getMessage());
+ LOG.warnv("Could not register ECS target into {0}: {1}", target.targetGroupArn(), e.getMessage());
}
- });
+ }
+ if (taskArn == null || registered.size() == planned.size()) {
+ return;
+ }
+ if (registered.isEmpty()) {
+ ledger.delete(taskArn);
+ } else {
+ ledger.put(taskArn, new EcsRegisteredTargets(region, List.copyOf(registered)));
+ }
}
- /** Deregisters the task's load-balanced containers from their ELBv2 target groups. */
+ /**
+ * Deregisters the targets {@link #registerTask} recorded for the task, and no other: a task
+ * with no record registered nothing, as when none of its containers had a reachable binding.
+ * Only a task without an ARN, which {@link #registerTask} cannot record, falls back to the
+ * targets its containers resolve to now.
+ */
public void deregisterTask(EcsTask task, EcsServiceModel svc, String region) {
- forEachTarget(task, svc, (tgArn, td) -> {
+ if (task.getTaskArn() == null) {
+ forEachTarget(task, svc, (tgArn, td) -> deregister(region, tgArn, td));
+ return;
+ }
+ Optional<EcsRegisteredTargets> recorded = ledger.get(task.getTaskArn());
+ if (recorded.isPresent()) {
+ deregister(recorded.get());
+ ledger.delete(task.getTaskArn());
+ }
+ }
+
+ /**
+ * Deregisters every target ECS recorded for a task, in every account. Only for startup, when
+ * ECS holds no task at all: task state is memory-only, so each recorded target belongs to a
+ * task of a previous run whose container is gone or is being removed, and Docker is free to
+ * hand its address to an unrelated container. Targets ECS did not register are never touched.
+ */
+ public void releaseRecordedTargets() {
+ for (AccountAwareStorageBackend.AccountEntry<EcsRegisteredTargets> entry
+ : ledger.scanAllAccountEntries(key -> true)) {
+ RequestScopes.runAs(entry.accountId(), () -> deregister(entry.value()));
+ ledger.deleteForAccount(entry.accountId(), entry.key());
+ LOG.infov("Released the load balancer targets of ECS task {0} left by a previous run", entry.key());
+ }
+ }
+
+ /**
+ * Deregisters, from the given target groups, each target at one of a starting task's addresses
+ * and ports that no ECS task recorded. The address was free until this task took it, so such a target
+ * was left by a task that is gone, as one registered by a Floci version that kept no record,
+ * and would now send the group's traffic to this task, which belongs to some other service.
+ * A target a live task recorded stays, whichever group it is in.
+ */
+ public void evictUnrecordedTargets(EcsTask task, Collection<String> targetGroupArns,
+ Collection<EcsTaskAddress> addresses, String region) {
+ if (targetGroupArns.isEmpty() || addresses.isEmpty()) {
+ return;
+ }
+ Set<EcsRegisteredTargets.Target> recorded = new HashSet<>();
+ for (EcsRegisteredTargets entry : ledger.scan(key -> true)) {
+ if (entry.targets() != null) {
+ recorded.addAll(entry.targets());
+ }
+ }
+ for (String tgArn : targetGroupArns) {
+ List<TargetHealth> health;
try {
- elbV2Service.deregisterTargets(region, tgArn, List.of(td));
- LOG.infov("Deregistered ECS task target {0}:{1} from target group {2}",
- td.getId(), td.getPort(), tgArn);
+ health = elbV2Service.describeTargetHealth(region, tgArn, null);
} catch (Exception e) {
- LOG.warnv("Could not deregister ECS target from {0}: {1}", tgArn, e.getMessage());
+ // A target group deleted while a service still names it: nothing in it to evict.
+ LOG.debugv("Could not list the targets of {0}: {1}", tgArn, e.getMessage());
+ continue;
+ }
+ for (TargetHealth th : health) {
+ TargetDescription td = th.getTarget();
+ boolean atTaskAddress = addresses.stream().anyMatch(a -> a.matches(td.getId(), td.getPort()));
+ if (!atTaskAddress
+ || recorded.contains(new EcsRegisteredTargets.Target(tgArn, td.getId(), td.getPort()))) {
+ continue;
+ }
+ try {
+ elbV2Service.deregisterTargets(region, tgArn, List.of(td));
+ LOG.warnv("Deregistered stale target {0}:{1} from target group {2}: no ECS task registered it,"
+ + " and its address now belongs to ECS task {3}", td.getId(), td.getPort(), tgArn,
+ task.getTaskArn());
+ } catch (Exception e) {
+ LOG.warnv("Could not deregister stale target {0}:{1} from {2}: {3}", td.getId(), td.getPort(),
+ tgArn, e.getMessage());
+ }
}
- });
+ }
+ }
+
+ private void deregister(EcsRegisteredTargets recorded) {
+ if (recorded.targets() == null) {
+ return;
+ }
+ for (EcsRegisteredTargets.Target target : recorded.targets()) {
+ deregister(recorded.region(), target.targetGroupArn(), target.toDescription());
+ }
+ }
+
+ private void deregister(String region, String tgArn, TargetDescription td) {
+ try {
+ elbV2Service.deregisterTargets(region, tgArn, List.of(td));
+ LOG.infov("Deregistered ECS task target {0}:{1} from target group {2}",
+ td.getId(), td.getPort(), tgArn);
+ } catch (Exception e) {
+ LOG.warnv("Could not deregister ECS target from {0}: {1}", tgArn, e.getMessage());
+ }
}
private void forEachTarget(EcsTask task, EcsServiceModel svc,
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java
index 5c8ee4e462..3610a2ac41 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java
@@ -37,11 +37,13 @@ import io.github.hectorvent.floci.services.ecs.model.EcsCluster;
import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;
import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;
import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;
import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;
import io.github.hectorvent.floci.services.ecs.model.KeyValuePair;
import io.github.hectorvent.floci.services.ecs.model.LaunchType;
import io.github.hectorvent.floci.services.ecs.model.ListTasksRequest;
import io.github.hectorvent.floci.services.ecs.model.ManagedAgent;
+import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;
import io.github.hectorvent.floci.services.ecs.model.NetworkConfiguration;
import io.github.hectorvent.floci.services.ecs.model.NetworkMode;
import io.github.hectorvent.floci.services.ecs.model.ProtectedTask;
@@ -116,6 +118,9 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
// Replaced by afterReset() after a state reset, whose container teardown shuts this scheduler down.
private volatile ScheduledExecutorService reconciler = newReconciler();
private final Object reconcilerLock = new Object();
+ // False until a previous run's task containers are confirmed gone; no task starts before.
+ private volatile boolean leftoverContainersRemoved = true;
+ private volatile boolean leftoverContainersBlockReported;
// region::clusterName → EcsCluster
private Map<String, EcsCluster> clusters = new ConcurrentHashMap<>();
@@ -218,9 +223,33 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
@PostConstruct
void init() {
initializeStorage();
+ releasePreviousRunLeftovers();
scheduleReconciliation(reconciler);
}
+ /**
+ * Clears what the tasks of a previous run left behind, before the scheduler starts their
+ * replacements. Task state is memory-only, so a restarted Floci knows no task, yet a run that
+ * ended without a graceful shutdown leaves its task containers serving, and even a graceful one
+ * leaves their load balancer targets and Cloud Map instances registered. Left alone, each
+ * service would serve from its replacements and a task nothing manages, and a dead task's
+ * target would follow its address to whichever container Docker hands it to next. Only what
+ * the registrars recorded registering is released, never a target or instance registered by
+ * hand. A container Docker would not remove is retried by {@link #leftoverContainersCleared()}
+ * before each task launch, and no task starts until it is gone.
+ */
+ void releasePreviousRunLeftovers() {
+ if (dockerMode) {
+ leftoverContainersRemoved = containerManager.removeLeftoverContainers();
+ }
+ if (lbRegistrar != null) {
+ lbRegistrar.releaseRecordedTargets();
+ }
+ if (discoveryRegistrar != null) {
+ discoveryRegistrar.releaseRecordedInstances();
+ }
+ }
+
private void scheduleReconciliation(ScheduledExecutorService scheduler) {
scheduler.scheduleAtFixedRate(this::reconcile, 5, 5, TimeUnit.SECONDS);
}
@@ -1308,6 +1337,10 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
if (dockerMode) {
try {
+ if (!leftoverContainersCleared()) {
+ throw new IllegalStateException(
+ "the containers a previous run of Floci left on the Docker daemon could not be removed yet");
+ }
task.setPullStartedAt(Instant.now());
EcsTaskHandle handle = containerManager.startTask(task, taskDef, containerOverrides, region);
task.setPullStoppedAt(Instant.now());
@@ -1321,6 +1354,7 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
if (!stopRequested) {
registerTaskWithLoadBalancers(task, cluster, region);
registerTaskForServiceDiscovery(task, cluster, region);
+ evictStaleRegistrationsAt(task, region);
}
if (eventPublisher != null) {
eventPublisher.emitTaskLadder(task, TaskStatus.PENDING, TaskStatus.RUNNING, region);
@@ -1792,6 +1826,102 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
}
}
+ /**
+ * Removes the load balancer targets and Cloud Map instances that point at a freshly-started
+ * task's addresses but that no ECS task recorded, from every target group and Cloud Map service
+ * an ECS service in the region names. Docker hands a dead container's address to the next
+ * container it starts, so a registration its task never released, as one made by a Floci
+ * version that kept no record, would otherwise route another service's traffic to this task.
+ * Runs for every task, not only a service's: any container can take the address.
+ */
+ private void evictStaleRegistrationsAt(EcsTask task, String region) {
+ Set<String> targetGroupArns = new LinkedHashSet<>();
+ Set<String> cloudMapServiceIds = new LinkedHashSet<>();
+ for (Map.Entry<String, EcsServiceModel> entry : services.entrySet()) {
+ EcsServiceModel svc = entry.getValue();
+ if (!region.equals(extractRegionFromServiceKey(entry.getKey())) || "INACTIVE".equals(svc.getStatus())) {
+ continue;
+ }
+ for (EcsLoadBalancer lb : svc.getLoadBalancers()) {
+ if (lb.getTargetGroupArn() != null && !lb.getTargetGroupArn().isBlank()) {
+ targetGroupArns.add(lb.getTargetGroupArn());
+ }
+ }
+ if (discoveryRegistrar != null) {
+ cloudMapServiceIds.addAll(discoveryRegistrar.cloudMapServiceIds(svc));
+ }
+ }
+ if (targetGroupArns.isEmpty() && cloudMapServiceIds.isEmpty()) {
+ return;
+ }
+ if (lbRegistrar != null && !targetGroupArns.isEmpty()) {
+ lbRegistrar.evictUnrecordedTargets(task, targetGroupArns, targetAddresses(task), region);
+ }
+ if (discoveryRegistrar != null && !cloudMapServiceIds.isEmpty()) {
+ discoveryRegistrar.evictUnrecordedInstances(task, cloudMapServiceIds, instanceAddresses(task, region),
+ region);
+ }
+ }
+
+ /**
+ * The address and port pairs at which a load balancer target reaches a running task: each
+ * container's address, on the ports its bindings declare. The ENI address is not among them,
+ * since ECS registers no target at it and it is unique only within its VPC, and a target on a
+ * port the task does not serve sends it nothing.
+ */
+ private Set<EcsTaskAddress> targetAddresses(EcsTask task) {
+ Set<EcsTaskAddress> addresses = new LinkedHashSet<>();
+ if (task.getContainers() == null) {
+ return addresses;
+ }
+ for (Container container : task.getContainers()) {
+ String host = containerManager.resolveContainerHost(container);
+ if (host == null || host.isBlank() || container.getNetworkBindings() == null) {
+ continue;
+ }
+ for (NetworkBinding binding : container.getNetworkBindings()) {
+ addresses.add(new EcsTaskAddress(host, binding.hostPort()));
+ if (!EcsTaskAddress.isLoopback(host)) {
+ addresses.add(new EcsTaskAddress(host, binding.containerPort()));
+ }
+ }
+ }
+ return addresses;
+ }
+
+ /**
+ * The addresses at which a Cloud Map instance resolves to a running task: its ENI address,
+ * within the task's VPC only, and each container's address on Docker, on any port since a
+ * DNS answer carries none. A loopback address counts only on the task's host ports.
+ */
+ private Set<EcsTaskAddress> instanceAddresses(EcsTask task, String region) {
+ Set<EcsTaskAddress> addresses = new LinkedHashSet<>();
+ String eniAddress = task.getPrivateIpAddress();
+ if (eniAddress != null && !eniAddress.isBlank() && !EcsTaskAddress.isLoopback(eniAddress)) {
+ String vpcId = containerManager.taskVpcId(task, region);
+ if (vpcId != null) {
+ addresses.add(new EcsTaskAddress(eniAddress, null, vpcId));
+ }
+ }
+ if (task.getContainers() == null) {
+ return addresses;
+ }
+ for (Container container : task.getContainers()) {
+ String host = containerManager.resolveContainerHost(container);
+ if (host == null || host.isBlank()) {
+ continue;
+ }
+ if (!EcsTaskAddress.isLoopback(host)) {
+ addresses.add(new EcsTaskAddress(host, null));
+ } else if (container.getNetworkBindings() != null) {
+ for (NetworkBinding binding : container.getNetworkBindings()) {
+ addresses.add(EcsTaskAddress.of(host, binding.hostPort()));
+ }
+ }
+ }
+ return addresses;
+ }
+
/**
* Resolves the service that actually launched {@code task}, or {@code null} for a
* caller-driven task. Keyed off the reconciler-stamped {@code owningServiceArn} rather than
@@ -3949,6 +4079,29 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
}
}
+ /**
+ * Whether a task may start: true once the containers a previous run left behind are gone,
+ * retrying the sweep otherwise. It runs before every task launch, RunTask and StartTask
+ * included, and the service scheduler asks it before planning one, so a service it blocks
+ * creates no failed task on every tick. Nothing retries it while no task is due, so an
+ * unreachable Docker daemon costs nothing then, and a blocked launch is reported once.
+ */
+ private boolean leftoverContainersCleared() {
+ if (leftoverContainersRemoved) {
+ return true;
+ }
+ leftoverContainersRemoved = containerManager.removeLeftoverContainers();
+ if (leftoverContainersRemoved) {
+ LOG.info("The containers a previous ECS run left behind are gone; starting service tasks");
+ } else {
+ LOG.log(leftoverContainersBlockReported ? Logger.Level.DEBUG : Logger.Level.WARN,
+ "Not starting ECS tasks while a previous run's containers remain;"
+ + " retrying before each task launch");
+ leftoverContainersBlockReported = true;
+ }
+ return leftoverContainersRemoved;
+ }
+
private Set<String> reconcilableAccountIds() {
Set<String> accountIds = new LinkedHashSet<>();
accountIds.add(regionResolver.getAccountId());
@@ -4236,6 +4389,9 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
if (covered.contains(ci.getContainerInstanceArn())) {
continue;
}
+ if (!leftoverContainersCleared()) {
+ break;
+ }
try {
EcsTask launched = launchServiceTask(cluster, svc, LaunchType.EC2,
ci.getContainerInstanceArn(), region);
@@ -4373,6 +4529,9 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
}
if (current < svc.getDesiredCount()) {
+ if (!leftoverContainersCleared()) {
+ return;
+ }
int toStart = svc.getDesiredCount() - (int) current;
for (int i = 0; i < toStart; i++) {
try {
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrar.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrar.java
index 50f13ff7d0..1c82e9b572 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrar.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrar.java
@@ -1,15 +1,25 @@
package io.github.hectorvent.floci.services.ecs;
+import com.fasterxml.jackson.core.type.TypeReference;
+import io.github.hectorvent.floci.core.common.RequestScopes;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
import io.github.hectorvent.floci.services.cloudmap.CloudMapService;
+import io.github.hectorvent.floci.services.cloudmap.model.Instance;
import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;
import io.github.hectorvent.floci.services.ecs.model.Container;
+import io.github.hectorvent.floci.services.ecs.model.EcsRegisteredInstances;
import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;
import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;
import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import org.jboss.logging.Logger;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
@@ -35,17 +45,26 @@ public class EcsServiceDiscoveryRegistrar {
private final CloudMapService cloudMapService;
private final EcsContainerManager containerManager;
+ // taskArn → the instance registered for it, persisted so it can be released after a restart
+ private final AccountAwareStorageBackend<EcsRegisteredInstances> ledger;
@Inject
- public EcsServiceDiscoveryRegistrar(CloudMapService cloudMapService, EcsContainerManager containerManager) {
+ public EcsServiceDiscoveryRegistrar(CloudMapService cloudMapService, EcsContainerManager containerManager,
+ StorageFactory storageFactory) {
this.cloudMapService = cloudMapService;
this.containerManager = containerManager;
+ this.ledger = storageFactory.create("ecs", "ecs-registered-instances.json",
+ new TypeReference<Map<String, EcsRegisteredInstances>>() {});
}
- /** Registers the task as a Cloud Map instance of every service registry the ECS service declares. */
+ /**
+ * Registers the task as a Cloud Map instance of every service registry the ECS service declares.
+ * The registrations are recorded before they are made, so a process killed in between still
+ * leaves a record for the next run to release; releasing one that never registered is harmless.
+ */
public void registerTask(EcsTask task, EcsServiceModel svc, String region) {
String instanceId = instanceId(task);
- Set<String> registered = new LinkedHashSet<>();
+ Map<String, Map<String, String>> planned = new LinkedHashMap<>();
for (Map<String, Object> registry : registries(svc)) {
String cloudMapServiceId = cloudMapServiceId(registry);
if (cloudMapServiceId == null) {
@@ -57,6 +76,16 @@ public class EcsServiceDiscoveryRegistrar {
task.getTaskArn(), cloudMapServiceId);
continue;
}
+ planned.putIfAbsent(cloudMapServiceId, attributes);
+ }
+ String taskArn = task.getTaskArn();
+ if (!planned.isEmpty() && taskArn != null) {
+ ledger.put(taskArn, new EcsRegisteredInstances(region, instanceId, List.copyOf(planned.keySet())));
+ }
+ Set<String> registered = new LinkedHashSet<>();
+ for (Map.Entry<String, Map<String, String>> entry : planned.entrySet()) {
+ String cloudMapServiceId = entry.getKey();
+ Map<String, String> attributes = entry.getValue();
try {
cloudMapService.registerInstance(cloudMapServiceId, instanceId, null, attributes, region);
registered.add(cloudMapServiceId);
@@ -68,6 +97,14 @@ public class EcsServiceDiscoveryRegistrar {
}
}
task.setServiceDiscoveryServiceIds(List.copyOf(registered));
+ if (taskArn == null || registered.size() == planned.size()) {
+ return;
+ }
+ if (registered.isEmpty()) {
+ ledger.delete(taskArn);
+ } else {
+ ledger.put(taskArn, new EcsRegisteredInstances(region, instanceId, List.copyOf(registered)));
+ }
}
/** Deregisters the task from the Cloud Map services it actually registered in. */
@@ -86,12 +123,123 @@ public class EcsServiceDiscoveryRegistrar {
}
}
task.setServiceDiscoveryServiceIds(List.of());
+ if (task.getTaskArn() != null) {
+ ledger.delete(task.getTaskArn());
+ }
+ }
+
+ /**
+ * Deregisters every Cloud Map instance ECS recorded for a task, in every account. Only for
+ * startup, when ECS holds no task at all: task state is memory-only, so none of them can still
+ * be a live task. Instances ECS did not register, including ones registered through the Cloud
+ * Map API, are never touched.
+ */
+ public void releaseRecordedInstances() {
+ for (AccountAwareStorageBackend.AccountEntry<EcsRegisteredInstances> entry
+ : ledger.scanAllAccountEntries(key -> true)) {
+ EcsRegisteredInstances recorded = entry.value();
+ List<String> cloudMapServiceIds = recorded.cloudMapServiceIds() != null
+ ? recorded.cloudMapServiceIds() : List.of();
+ RequestScopes.runAs(entry.accountId(), () -> {
+ for (String cloudMapServiceId : cloudMapServiceIds) {
+ try {
+ cloudMapService.deregisterInstance(cloudMapServiceId, recorded.instanceId(),
+ recorded.region());
+ LOG.infov("Deregistered Cloud Map instance {0} of {1} left by a previous ECS task",
+ recorded.instanceId(), cloudMapServiceId);
+ } catch (Exception e) {
+ // Already gone, as when its Cloud Map service was deleted: nothing left to release.
+ LOG.debugv("Could not deregister Cloud Map instance {0} of {1}: {2}",
+ recorded.instanceId(), cloudMapServiceId, e.getMessage());
+ }
+ }
+ });
+ ledger.deleteForAccount(entry.accountId(), entry.key());
+ }
+ }
+
+ /**
+ * Deregisters, from the given Cloud Map services, each instance at one of a starting task's
+ * addresses that no ECS task recorded. The counterpart of
+ * {@link EcsLoadBalancerRegistrar#evictUnrecordedTargets}: such an instance was left by a task
+ * that is gone, as one registered by a Floci version that kept no record, and would now resolve
+ * the service's name to this task. An instance a live task recorded stays, and an address
+ * scoped to a VPC counts only in a Cloud Map service whose namespace is in that VPC.
+ */
+ public void evictUnrecordedInstances(EcsTask task, Collection<String> cloudMapServiceIds,
+ Collection<EcsTaskAddress> addresses, String region) {
+ if (cloudMapServiceIds.isEmpty() || addresses.isEmpty()) {
+ return;
+ }
+ Set<String> recorded = new HashSet<>();
+ for (EcsRegisteredInstances entry : ledger.scan(key -> true)) {
+ if (entry.cloudMapServiceIds() != null) {
+ entry.cloudMapServiceIds().forEach(id -> recorded.add(id + "/" + entry.instanceId()));
+ }
+ }
+ for (String cloudMapServiceId : cloudMapServiceIds) {
+ List<Instance> instances;
+ List<EcsTaskAddress> scoped;
+ try {
+ String vpc = cloudMapService.getNamespace(
+ cloudMapService.getService(cloudMapServiceId).getNamespaceId()).getVpc();
+ scoped = addresses.stream().filter(a -> a.appliesTo(vpc)).toList();
+ instances = cloudMapService.listInstances(cloudMapServiceId);
+ } catch (Exception e) {
+ // A Cloud Map service deleted while an ECS service still names it: nothing to evict.
+ LOG.debugv("Could not list the instances of Cloud Map service {0}: {1}", cloudMapServiceId,
+ e.getMessage());
+ continue;
+ }
+ for (Instance instance : instances) {
+ Map<String, String> attributes = instance.getAttributes() != null ? instance.getAttributes() : Map.of();
+ String ip = attributes.get("AWS_INSTANCE_IPV4");
+ Integer port = parsePort(attributes.get("AWS_INSTANCE_PORT"));
+ if (ip == null || scoped.stream().noneMatch(a -> a.matches(ip, port))
+ || recorded.contains(cloudMapServiceId + "/" + instance.getInstanceId())) {
+ continue;
+ }
+ try {
+ cloudMapService.deregisterInstance(cloudMapServiceId, instance.getInstanceId(), region);
+ LOG.warnv("Deregistered stale Cloud Map instance {0} of {1} at {2}: no ECS task registered it,"
+ + " and its address now belongs to ECS task {3}", instance.getInstanceId(),
+ cloudMapServiceId, ip, task.getTaskArn());
+ } catch (Exception e) {
+ LOG.warnv("Could not deregister stale Cloud Map instance {0} of {1}: {2}",
+ instance.getInstanceId(), cloudMapServiceId, e.getMessage());
+ }
+ }
+ }
}
public boolean hasRegistries(EcsServiceModel svc) {
return !registries(svc).isEmpty();
}
+ /** The Cloud Map services the ECS service's registries name. */
+ public List<String> cloudMapServiceIds(EcsServiceModel svc) {
+ List<String> ids = new ArrayList<>();
+ for (Map<String, Object> registry : registries(svc)) {
+ String id = cloudMapServiceId(registry);
+ if (id != null) {
+ ids.add(id);
+ }
+ }
+ return ids;
+ }
+
+ private static Integer parsePort(String port) {
+ if (port == null) {
+ return null;
+ }
+ try {
+ return Integer.valueOf(port);
+ } catch (NumberFormatException e) {
+ LOG.debugv("Ignoring a Cloud Map instance port that is not a number: {0}", port);
+ return null;
+ }
+ }
+
private List<Map<String, Object>> registries(EcsServiceModel svc) {
List<Map<String, Object>> registries = svc.getServiceRegistries();
return registries != null ? registries : List.of();
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java
index 6de4d15219..0f99f1213f 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java
@@ -77,6 +77,7 @@ import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
@@ -93,6 +94,8 @@ public class EcsContainerManager {
private static final Logger LOG = Logger.getLogger(EcsContainerManager.class);
private static final String ATTACHMENT_DELETED = "DELETED";
+ /** The label naming the Floci process that created a container, set by {@link #ownerLabels()}. */
+ public static final String RUN_LABEL = "floci.ecs-run";
/** EC2 error codes the task ENI path can raise, none of which RunTask declares. */
private static final Set<String> EC2_NETWORK_LOOKUP_FAILURES =
@@ -133,6 +136,11 @@ public class EcsContainerManager {
private final EcsTaskRoleCredentials taskRoleCredentials;
private final EcsCredentialsProxy credentialsProxy;
private final EcsTaskLinkLocalAddresses linkLocalAddresses;
+ // Stamped as RUN_LABEL on every container this process creates, so a sweep can tell its own apart.
+ private final String runId = UUID.randomUUID().toString();
+ // A sweep failure is retried before each task launch; each distinct failure is a WARN once.
+ private volatile boolean leftoverListFailureReported;
+ private final Set<String> leftoverRemovalFailuresReported = ConcurrentHashMap.newKeySet();
@Inject
public EcsContainerManager(ContainerBuilder containerBuilder,
@@ -312,7 +320,8 @@ public class EcsContainerManager {
.withHostDockerInternalOnLinux()
.withEmbeddedDns()
.withLabels(ContainerStorageHelper.resourceIdentityLabels(
- "ecs", taskId, regionResolver.getAccountId(), region));
+ "ecs", taskId, regionResolver.getAccountId(), region))
+ .withLabels(ownerLabels());
if (protectedNetwork != null) {
specBuilder.withNetworkMode("container:" + protectedNetwork.namespace().helperId());
specBuilder.withLabels(Map.of("floci.security-group-workload", "true"));
@@ -991,6 +1000,30 @@ public class EcsContainerManager {
.ifPresent(subnet -> task.setAvailabilityZone(subnet.getAvailabilityZone()));
}
+ /**
+ * The VPC an awsvpc task's interface was created in, read from its subnet, or {@code null}
+ * for a task without one or whose subnet is gone.
+ */
+ public String taskVpcId(EcsTask task, String region) {
+ if (task.getNetworkInterfaceId() == null || ec2Service == null) {
+ return null;
+ }
+ AwsVpcConfiguration awsvpc = task.getNetworkConfiguration() == null ? null
+ : task.getNetworkConfiguration().getAwsvpcConfiguration();
+ if (awsvpc == null || awsvpc.getSubnets() == null || awsvpc.getSubnets().isEmpty()) {
+ return null;
+ }
+ try {
+ return ec2Service.describeSubnets(region, List.of(awsvpc.getSubnets().getFirst()), Map.of()).stream()
+ .findFirst()
+ .map(subnet -> subnet.getVpcId())
+ .orElse(null);
+ } catch (AwsException e) {
+ LOG.debugv("Could not look up the VPC of ECS task {0}: {1}", task.getTaskArn(), e.getMessage());
+ return null;
+ }
+ }
+
/**
* Restates an EC2 lookup failure as the error RunTask declares. A caller naming a subnet or a
* security group that is not there asked for something the task cannot have, and
@@ -1030,6 +1063,76 @@ public class EcsContainerManager {
}
}
+ /**
+ * Removes the ECS containers a previous run of this Floci left on the daemon: task containers
+ * and the security-group helpers whose network namespace they share. Task state is memory-only,
+ * so no container carrying this deployment's owner label and another run's {@link #RUN_LABEL}
+ * belongs to a task this process knows. A graceful shutdown already stops them; this
+ * covers a run that ended without one (SIGKILL, OOM, a stop timeout that expired mid-drain),
+ * whose containers would otherwise keep serving beside the replacements the service scheduler
+ * starts. Containers created before the owner label existed carry none and are left alone.
+ * Leaving out the ones this process created is what makes a retry safe once tasks run. The run
+ * label decides that rather than the creation time, which Docker reports to the second and from
+ * a clock that can drift from this process's.
+ *
+ * Each distinct failure, a failed listing or a container that would not go, is logged as a
+ * WARN the first time and at DEBUG on every retry that repeats it, so an unreachable Docker
+ * daemon does not log on every launch attempt while a new failure is still reported.
+ *
+ * @return whether every leftover is gone: false when Docker could not list or remove one
+ */
+ public boolean removeLeftoverContainers() {
+ return removeLeftoverContainers(runId);
+ }
+
+ /** The labels that tie a container to this deployment and to this process's run of it. */
+ private Map<String, String> ownerLabels() {
+ return Map.of(ContainerStorageHelper.OWNER_LABEL, ContainerStorageHelper.ownerIdentity(config),
+ RUN_LABEL, runId);
+ }
+
+ /** {@link #removeLeftoverContainers()} keeping only the containers of run {@code currentRunId}. */
+ public boolean removeLeftoverContainers(String currentRunId) {
+ String owner = ContainerStorageHelper.ownerIdentity(config);
+ List<com.github.dockerjava.api.model.Container> containers;
+ try {
+ // Docker's container summary, not the ECS model Container this class imports.
+ containers = lifecycleManager.getDockerClient()
+ .listContainersCmd()
+ .withShowAll(true)
+ .withLabelFilter(Map.of("io.floci.service", "ecs", ContainerStorageHelper.OWNER_LABEL, owner))
+ .exec();
+ } catch (Exception e) {
+ LOG.logv(leftoverListFailureReported ? Logger.Level.DEBUG : Logger.Level.WARN,
+ "Could not list the ECS containers a previous run left behind: {0}", e.getMessage());
+ leftoverListFailureReported = true;
+ return false;
+ }
+ leftoverListFailureReported = false;
+ boolean allRemoved = true;
+ for (com.github.dockerjava.api.model.Container container : containers) {
+ if (container.getLabels() != null && currentRunId.equals(container.getLabels().get(RUN_LABEL))) {
+ continue;
+ }
+ try {
+ lifecycleManager.removeIfExistsStrict(container.getId());
+ leftoverRemovalFailuresReported.remove(container.getId());
+ LOG.infov("Removed ECS container {0} ({1}) left by a previous run", container.getId(),
+ container.getNames() == null ? "" : String.join(",", container.getNames()));
+ } catch (Exception e) {
+ allRemoved = false;
+ Logger.Level level = leftoverRemovalFailuresReported.add(container.getId())
+ ? Logger.Level.WARN : Logger.Level.DEBUG;
+ LOG.logv(level, "Could not remove ECS container {0} left by a previous run: {1}",
+ container.getId(), e.getMessage());
+ }
+ }
+ if (allRemoved) {
+ leftoverRemovalFailuresReported.clear();
+ }
+ return allRemoved;
+ }
+
/** The ENI {@link #attachTaskNetwork} allocated for this task, or null if it is already gone. */
private NetworkInterface existingTaskEni(EcsTask task, String region) {
if (task.getNetworkInterfaceId() == null) {
@@ -1071,7 +1174,7 @@ public class EcsContainerManager {
containerDetector.isRunningInContainer(),
config.services().ecs().publishAwsvpcPortsToHost());
namespace = firewallManager.createNamespace("ecs", taskId, regionResolver.getAccountId(),
- region, config.services().ecs().dockerNetwork(), bindings);
+ region, config.services().ecs().dockerNetwork(), bindings, Map.of(RUN_LABEL, runId));
List<String> groupIds = eni.getGroups().stream().map(g -> g.getGroupId()).toList();
List<SecurityGroup> groups = ec2Service.describeSecurityGroups(region, groupIds, List.of(), Map.of());
if (groups.size() != groupIds.size()) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredInstances.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredInstances.java
new file mode 100644
index 0000000000..210bb2173d
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredInstances.java
@@ -0,0 +1,14 @@
+package io.github.hectorvent.floci.services.ecs.model;
+
+import io.quarkus.runtime.annotations.RegisterForReflection;
+
+import java.util.List;
+
+/**
+ * The Cloud Map instance ECS registered for one task and the Cloud Map services it joined,
+ * persisted so a restarted Floci, which knows no task, deregisters exactly these and never an
+ * instance registered through the Cloud Map API.
+ */
+@RegisterForReflection
+public record EcsRegisteredInstances(String region, String instanceId, List<String> cloudMapServiceIds) {
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredTargets.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredTargets.java
new file mode 100644
index 0000000000..fbd5807882
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredTargets.java
@@ -0,0 +1,25 @@
+package io.github.hectorvent.floci.services.ecs.model;
+
+import io.github.hectorvent.floci.services.elbv2.model.TargetDescription;
+import io.quarkus.runtime.annotations.RegisterForReflection;
+
+import java.util.List;
+
+/**
+ * The ELBv2 targets ECS registered for one task, persisted so a restarted Floci, which knows no
+ * task, deregisters exactly these and never a target someone else registered in the same group.
+ */
+@RegisterForReflection
+public record EcsRegisteredTargets(String region, List<Target> targets) {
+
+ @RegisterForReflection
+ public record Target(String targetGroupArn, String id, Integer port) {
+
+ public TargetDescription toDescription() {
+ TargetDescription td = new TargetDescription();
+ td.setId(id);
+ td.setPort(port);
+ return td;
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTaskAddress.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTaskAddress.java
new file mode 100644
index 0000000000..11b4a6f8d5
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTaskAddress.java
@@ -0,0 +1,34 @@
+package io.github.hectorvent.floci.services.ecs.model;
+
+/**
+ * An address a running task holds. A task's own address on a Docker network reaches only that
+ * task, so {@code port} is {@code null} when any port there counts as the task's. A loopback
+ * address is shared by every task Floci publishes on the host, so it is the task's only together
+ * with one of the host ports its containers were published on. An ENI address is unique only
+ * within its VPC, so {@code vpcId} names the VPC it belongs to; it is {@code null} for a Docker
+ * or loopback address, which no VPC scopes.
+ */
+public record EcsTaskAddress(String ip, Integer port, String vpcId) {
+
+ public EcsTaskAddress(String ip, Integer port) {
+ this(ip, port, null);
+ }
+
+ public static EcsTaskAddress of(String ip, int hostPort) {
+ return isLoopback(ip) ? new EcsTaskAddress(ip, hostPort) : new EcsTaskAddress(ip, null);
+ }
+
+ public static boolean isLoopback(String ip) {
+ return ip.startsWith("127.") || "localhost".equals(ip) || "::1".equals(ip);
+ }
+
+ /** Whether an endpoint registered at {@code ip} and {@code port} reaches this task. */
+ public boolean matches(String ip, Integer port) {
+ return this.ip.equals(ip) && (this.port == null || this.port.equals(port));
+ }
+
+ /** Whether this address means this task in {@code vpc}, the VPC an endpoint is scoped to. */
+ public boolean appliesTo(String vpc) {
+ return vpcId == null || vpcId.equals(vpc);
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLeftoverContainersDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLeftoverContainersDockerIntegrationTest.java
new file mode 100644
index 0000000000..ed47595650
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLeftoverContainersDockerIntegrationTest.java
@@ -0,0 +1,143 @@
+package io.github.hectorvent.floci.services.ecs;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.command.CreateContainerResponse;
+import com.github.dockerjava.api.exception.NotFoundException;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;
+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;
+import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.LaunchType;
+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;
+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;
+import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.TestProfile;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * A Floci that ends without a graceful shutdown leaves its task containers running, and the next
+ * run knows no task, since task state is memory-only. Those containers must go at startup, or the
+ * service scheduler starts a second task beside each one. The sweep recognises them by the owner
+ * label a task container is created with, so a container of another Floci sharing the daemon stays,
+ * and tells this run's containers from a previous run's by the run label rather than by time.
+ */
+@QuarkusTest
+@TestProfile(EcsServiceDiscoveryDockerIntegrationTest.DockerEcsProfile.class)
+class EcsLeftoverContainersDockerIntegrationTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String BUSYBOX_IMAGE = "public.ecr.aws/docker/library/busybox:latest";
+
+ @Inject
+ EcsService ecsService;
+
+ @Inject
+ EcsContainerManager containerManager;
+
+ @Inject
+ EmulatorConfig config;
+
+ @Inject
+ DockerClient dockerClient;
+
+ @BeforeEach
+ void requireDocker() {
+ Assumptions.assumeTrue(isDockerAvailable(),
+ "Docker daemon must be available for the ECS leftover container test");
+ }
+
+ @Test
+ void theStartupSweepRemovesTaskContainersAndKeepsAnotherFlocisContainers() {
+ String clusterName = unique("leftover-cluster");
+ ecsService.createCluster(clusterName, REGION);
+ ContainerDefinition app = new ContainerDefinition();
+ app.setName("app");
+ app.setImage(BUSYBOX_IMAGE);
+ app.setCommand(List.of("sleep", "120"));
+ TaskDefinition taskDef = ecsService.registerTaskDefinition(unique("leftover-td"), List.of(app),
+ NetworkMode.bridge, null, null, null, null, null, REGION);
+
+ EcsTask task = ecsService.runTask(clusterName, taskDef.getTaskDefinitionArn(), 1, LaunchType.EC2, null, null,
+ null, null, REGION).getFirst();
+ String foreignId = null;
+ String previousRunId = null;
+ try {
+ String taskContainerId = task.getContainers().getFirst().getDockerId();
+ assertNotNull(taskContainerId, "the task must have started a container: " + task.getStoppedReason());
+ assertEquals(ContainerStorageHelper.ownerIdentity(config), labels(taskContainerId)
+ .get(ContainerStorageHelper.OWNER_LABEL), "a task container must carry its owner");
+
+ foreignId = createEcsContainer(Map.of(ContainerStorageHelper.OWNER_LABEL, "another-floci/4566"));
+ // Created a moment ago, as by a run killed within the second this one started.
+ previousRunId = createEcsContainer(Map.of(ContainerStorageHelper.OWNER_LABEL,
+ ContainerStorageHelper.ownerIdentity(config), EcsContainerManager.RUN_LABEL, "a-previous-run"));
+
+ assertTrue(containerManager.removeLeftoverContainers(), "the sweep must succeed");
+ assertTrue(exists(taskContainerId), "a task container this run started must stay");
+ assertFalse(exists(previousRunId), "a previous run's container must be removed however recent");
+
+ assertTrue(containerManager.removeLeftoverContainers("the-next-run"), "the sweep must succeed");
+ assertFalse(exists(taskContainerId), "the leftover task container must be removed");
+ assertTrue(exists(foreignId), "another Floci's container must stay");
+ } finally {
+ ecsService.stopTask(clusterName, task.getTaskArn(), "test teardown", REGION);
+ for (String containerId : new String[] {foreignId, previousRunId}) {
+ if (containerId != null && exists(containerId)) {
+ dockerClient.removeContainerCmd(containerId).withForce(true).exec();
+ }
+ }
+ }
+ }
+
+ /** An ECS container carrying {@code ownership}, as another Floci or another run creates. */
+ private String createEcsContainer(Map<String, String> ownership) {
+ Map<String, String> labels = new HashMap<>(ownership);
+ labels.put("io.floci.service", "ecs");
+ CreateContainerResponse created = dockerClient.createContainerCmd(BUSYBOX_IMAGE)
+ .withName(unique("floci-other-ecs"))
+ .withCmd("sleep", "120")
+ .withLabels(labels)
+ .exec();
+ return created.getId();
+ }
+
+ private Map<String, String> labels(String containerId) {
+ return dockerClient.inspectContainerCmd(containerId).exec().getConfig().getLabels();
+ }
+
+ private boolean exists(String containerId) {
+ try {
+ dockerClient.inspectContainerCmd(containerId).exec();
+ return true;
+ } catch (NotFoundException e) {
+ return false;
+ }
+ }
+
+ private boolean isDockerAvailable() {
+ try {
+ dockerClient.pingCmd().exec();
+ return true;
+ } catch (RuntimeException e) {
+ return false;
+ }
+ }
+
+ private static String unique(String prefix) {
+ return prefix + "-" + UUID.randomUUID().toString().substring(0, 8);
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrarTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrarTest.java
index 258375ad3a..8660aac791 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrarTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrarTest.java
@@ -4,19 +4,27 @@ import io.github.hectorvent.floci.services.ecs.model.Container;
import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;
import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;
import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;
import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;
import io.github.hectorvent.floci.services.elbv2.ElbV2Service;
+import io.github.hectorvent.floci.services.elbv2.model.TargetDescription;
import io.github.hectorvent.floci.services.elbv2.model.TargetGroup;
import io.github.hectorvent.floci.services.elbv2.model.TargetHealth;
import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.mockito.InjectSpy;
import jakarta.inject.Inject;
import org.junit.jupiter.api.Test;
import java.util.List;
import java.util.Map;
+import java.util.UUID;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doAnswer;
/**
* Component test for {@link EcsLoadBalancerRegistrar}: drives the register/deregister
@@ -31,7 +39,7 @@ class EcsLoadBalancerRegistrarTest {
@Inject
EcsLoadBalancerRegistrar registrar;
- @Inject
+ @InjectSpy
ElbV2Service elbV2Service;
private String createTargetGroup(String name) {
@@ -47,7 +55,7 @@ class EcsLoadBalancerRegistrarTest {
container.setNetworkBindings(List.of(
new NetworkBinding("0.0.0.0", containerPort, hostPort, "tcp")));
EcsTask task = new EcsTask();
- task.setTaskArn("arn:aws:ecs:" + REGION + ":000000000000:task/c/regtest");
+ task.setTaskArn("arn:aws:ecs:" + REGION + ":000000000000:task/c/" + UUID.randomUUID());
task.setGroup("regtest-svc");
task.setContainers(List.of(container));
return task;
@@ -125,4 +133,153 @@ class EcsLoadBalancerRegistrarTest {
assertTrue(elbV2Service.describeTargetHealth(REGION, tgArn, null).isEmpty(),
"no network binding for the declared containerPort -> no target");
}
+
+ @Test
+ void releaseRecordedTargetsKeepsATargetRegisteredByHand() {
+ String tgArn = createTargetGroup("reg-tg-stale");
+ registrar.registerTask(taskWithContainer("web", 8080, 34567), serviceWithLb(tgArn, "web", 8080), REGION);
+ elbV2Service.registerTargets(REGION, tgArn, List.of(target("172.18.0.7", 8080)));
+
+ registrar.releaseRecordedTargets();
+
+ List<TargetHealth> health = elbV2Service.describeTargetHealth(REGION, tgArn, null);
+ assertEquals(1, health.size(), "only the target registered by hand should survive");
+ assertEquals("172.18.0.7", health.get(0).getTarget().getId());
+ }
+
+ @Test
+ void deregisterTaskRemovesTheTargetItRecordedEvenAfterTheBindingChanged() {
+ String tgArn = createTargetGroup("reg-tg-recorded");
+ EcsTask task = taskWithContainer("web", 8080, 35000);
+ EcsServiceModel svc = serviceWithLb(tgArn, "web", 8080);
+ registrar.registerTask(task, svc, REGION);
+ task.getContainers().getFirst().setNetworkBindings(List.of(
+ new NetworkBinding("0.0.0.0", 8080, 36000, "tcp")));
+
+ registrar.deregisterTask(task, svc, REGION);
+
+ assertTrue(elbV2Service.describeTargetHealth(REGION, tgArn, null).isEmpty(),
+ "the recorded target should be deregistered");
+ }
+
+ @Test
+ void releaseRecordedTargetsToleratesADeletedTargetGroup() {
+ String tgArn = createTargetGroup("reg-tg-gone");
+ EcsTask task = taskWithContainer("web", 8080, 37000);
+ registrar.registerTask(task, serviceWithLb(tgArn, "web", 8080), REGION);
+ elbV2Service.deleteTargetGroup(REGION, tgArn);
+
+ registrar.releaseRecordedTargets();
+ }
+
+ @Test
+ void releaseRecordedTargetsReleasesATargetWhoseRegistrationWasCutShort() {
+ String tgArn = createTargetGroup("reg-tg-killed");
+ doAnswer(invocation -> {
+ invocation.callRealMethod();
+ throw new ProcessKilled();
+ }).when(elbV2Service).registerTargets(eq(REGION), eq(tgArn), any());
+
+ assertThrows(ProcessKilled.class, () -> registrar.registerTask(
+ taskWithContainer("web", 8080, 38000), serviceWithLb(tgArn, "web", 8080), REGION));
+ registrar.releaseRecordedTargets();
+
+ assertTrue(elbV2Service.describeTargetHealth(REGION, tgArn, null).isEmpty(),
+ "a target registered just before the process died should be released");
+ }
+
+ @Test
+ void aTargetAlreadyAtTheTasksAddressIsAdoptedAndRemovedWhenTheTaskStops() {
+ // Left by a task of a run that kept no record: the next task to take the address adopts it.
+ String tgArn = createTargetGroup("reg-tg-preexisting");
+ elbV2Service.registerTargets(REGION, tgArn, List.of(target("127.0.0.1", 39000)));
+ EcsTask task = taskWithContainer("web", 8080, 39000);
+ EcsServiceModel svc = serviceWithLb(tgArn, "web", 8080);
+
+ registrar.registerTask(task, svc, REGION);
+ assertEquals(1, elbV2Service.describeTargetHealth(REGION, tgArn, null).size());
+ registrar.deregisterTask(task, svc, REGION);
+
+ assertTrue(elbV2Service.describeTargetHealth(REGION, tgArn, null).isEmpty(),
+ "the target at the stopped task's address should be deregistered");
+ }
+
+ @Test
+ void evictUnrecordedTargetsRemovesAStaleTargetAtTheTasksAddressFromEveryGroup() {
+ String alphaTg = createTargetGroup("reg-tg-evict-alpha");
+ String betaTg = createTargetGroup("reg-tg-evict-beta");
+ elbV2Service.registerTargets(REGION, alphaTg, List.of(target("172.19.0.4", 8080), target("172.19.0.3", 8080)));
+ elbV2Service.registerTargets(REGION, betaTg, List.of(target("172.19.0.4", 8080)));
+ EcsTask betaTask = taskWithContainer("web", 8080, 8080);
+
+ registrar.evictUnrecordedTargets(betaTask, List.of(alphaTg, betaTg),
+ List.of(new EcsTaskAddress("172.19.0.4", 8080)), REGION);
+
+ List<TargetHealth> alpha = elbV2Service.describeTargetHealth(REGION, alphaTg, null);
+ assertEquals(1, alpha.size(), "only the target at another address should stay in alpha's group");
+ assertEquals("172.19.0.3", alpha.get(0).getTarget().getId());
+ assertTrue(elbV2Service.describeTargetHealth(REGION, betaTg, null).isEmpty(),
+ "a target at the task's address and port should be evicted");
+ }
+
+ @Test
+ void evictUnrecordedTargetsKeepsATargetOnAPortTheTaskDoesNotServe() {
+ String tgArn = createTargetGroup("reg-tg-evict-port");
+ elbV2Service.registerTargets(REGION, tgArn, List.of(target("172.19.0.5", 8080), target("172.19.0.5", 9000)));
+
+ registrar.evictUnrecordedTargets(taskWithContainer("web", 8080, 8080), List.of(tgArn),
+ List.of(new EcsTaskAddress("172.19.0.5", 8080)), REGION);
+
+ List<TargetHealth> health = elbV2Service.describeTargetHealth(REGION, tgArn, null);
+ assertEquals(1, health.size(), "the target on a port the task does not serve should stay");
+ assertEquals(9000, health.get(0).getTarget().getPort());
+ }
+
+ @Test
+ void evictUnrecordedTargetsKeepsATargetALiveTaskRecorded() {
+ String tgArn = createTargetGroup("reg-tg-evict-recorded");
+ EcsTask owner = taskWithContainer("web", 8080, 40000);
+ registrar.registerTask(owner, serviceWithLb(tgArn, "web", 8080), REGION);
+ EcsTask other = taskWithContainer("web", 8080, 40000);
+
+ registrar.evictUnrecordedTargets(other, List.of(tgArn),
+ List.of(EcsTaskAddress.of("127.0.0.1", 40000)), REGION);
+
+ assertEquals(1, elbV2Service.describeTargetHealth(REGION, tgArn, null).size(),
+ "a target a live task recorded should stay");
+ registrar.deregisterTask(owner, serviceWithLb(tgArn, "web", 8080), REGION);
+ }
+
+ @Test
+ void evictUnrecordedTargetsMatchesALoopbackAddressOnlyByItsHostPort() {
+ String tgArn = createTargetGroup("reg-tg-evict-loopback");
+ elbV2Service.registerTargets(REGION, tgArn, List.of(target("127.0.0.1", 41000), target("127.0.0.1", 41001)));
+
+ registrar.evictUnrecordedTargets(taskWithContainer("web", 8080, 41000), List.of(tgArn),
+ List.of(EcsTaskAddress.of("127.0.0.1", 41000)), REGION);
+
+ List<TargetHealth> health = elbV2Service.describeTargetHealth(REGION, tgArn, null);
+ assertEquals(1, health.size(), "the loopback target on another host port should stay");
+ assertEquals(41001, health.get(0).getTarget().getPort());
+ }
+
+ @Test
+ void evictUnrecordedTargetsToleratesADeletedTargetGroup() {
+ String tgArn = createTargetGroup("reg-tg-evict-gone");
+ elbV2Service.deleteTargetGroup(REGION, tgArn);
+
+ registrar.evictUnrecordedTargets(taskWithContainer("web", 8080, 42000), List.of(tgArn),
+ List.of(new EcsTaskAddress("172.19.0.9", 8080)), REGION);
+ }
+
+ /** Stands in for the process dying between the registration and anything after it. */
+ private static final class ProcessKilled extends Error {
+ }
+
+ private static TargetDescription target(String id, int port) {
+ TargetDescription td = new TargetDescription();
+ td.setId(id);
+ td.setPort(port);
+ return td;
+ }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrarTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrarTest.java
index 9d6d391802..0c2899ca59 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrarTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrarTest.java
@@ -7,8 +7,10 @@ import io.github.hectorvent.floci.services.cloudmap.model.Service;
import io.github.hectorvent.floci.services.ecs.model.Container;
import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;
import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;
import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;
import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.mockito.InjectSpy;
import jakarta.inject.Inject;
import org.junit.jupiter.api.Test;
@@ -17,7 +19,11 @@ import java.util.Map;
import java.util.UUID;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.doAnswer;
/**
* Component test for {@link EcsServiceDiscoveryRegistrar}: drives register/deregister directly
@@ -33,7 +39,7 @@ class EcsServiceDiscoveryRegistrarTest {
@Inject
EcsServiceDiscoveryRegistrar registrar;
- @Inject
+ @InjectSpy
CloudMapService cloudMapService;
@Test
@@ -139,6 +145,105 @@ class EcsServiceDiscoveryRegistrarTest {
assertTrue(!registrar.hasRegistries(svc));
}
+ @Test
+ void releaseRecordedInstancesKeepsInstancesRegisteredByHand() {
+ String namespace = uniqueName("svcdisc") + ".internal";
+ Service cloudMapSvc = createDnsService(namespace, "web");
+ EcsServiceModel web = serviceWithRegistry(cloudMapSvc.getArn(), "web", 8080);
+ web.setServiceName("web-svc");
+ registrar.registerTask(taskInCluster("172.31.0.11", "payments"), web, REGION);
+ registrar.registerTask(taskInCluster("172.31.0.12", "payments"), web, REGION);
+ cloudMapService.registerInstance(cloudMapSvc.getId(), "registered-by-hand", null,
+ Map.of("AWS_INSTANCE_IPV4", "172.31.0.15",
+ "ECS_SERVICE_NAME", "web-svc", "ECS_CLUSTER_NAME", "payments"), REGION);
+
+ registrar.releaseRecordedInstances();
+
+ List<String> remaining = cloudMapService.listInstances(cloudMapSvc.getId()).stream()
+ .map(Instance::getInstanceId)
+ .toList();
+ assertEquals(List.of("registered-by-hand"), remaining);
+ }
+
+ @Test
+ void releaseRecordedInstancesReleasesAnInstanceWhoseRegistrationWasCutShort() {
+ String namespace = uniqueName("svcdisc") + ".internal";
+ Service cloudMapSvc = createDnsService(namespace, "killed");
+ doAnswer(invocation -> {
+ invocation.callRealMethod();
+ throw new ProcessKilled();
+ }).when(cloudMapService).registerInstance(eq(cloudMapSvc.getId()), any(), any(), any(), any());
+
+ assertThrows(ProcessKilled.class, () -> registrar.registerTask(task("172.31.0.13", "killed", 8080, 8080),
+ serviceWithRegistry(cloudMapSvc.getArn(), "killed", 8080), REGION));
+ registrar.releaseRecordedInstances();
+
+ assertTrue(cloudMapService.listInstances(cloudMapSvc.getId()).isEmpty(),
+ "an instance registered just before the process died should be released");
+ }
+
+ @Test
+ void evictUnrecordedInstancesRemovesAStaleInstanceAtTheTasksAddress() {
+ String namespace = uniqueName("svcdisc") + ".internal";
+ Service auth = createDnsService(namespace, "auth");
+ Service document = cloudMapService.createService("document", auth.getNamespaceId(),
+ null, null, null, null, null, null, Map.of(), REGION);
+ // Left by an auth task of a run that kept no record; the address now belongs to a document task.
+ cloudMapService.registerInstance(auth.getId(), "bdf778f0", null,
+ Map.of("AWS_INSTANCE_IPV4", "172.31.0.21"), REGION);
+ cloudMapService.registerInstance(auth.getId(), "elsewhere", null,
+ Map.of("AWS_INSTANCE_IPV4", "172.31.0.17"), REGION);
+ EcsTask documentTask = task("172.31.0.21", "document", 8080, 8080);
+ registrar.registerTask(documentTask, serviceWithRegistry(document.getArn(), "document", 8080), REGION);
+
+ registrar.evictUnrecordedInstances(documentTask, List.of(auth.getId(), document.getId()),
+ List.of(new EcsTaskAddress("172.31.0.21", null)), REGION);
+
+ assertEquals(List.of("elsewhere"), cloudMapService.listInstances(auth.getId()).stream()
+ .map(Instance::getInstanceId).toList());
+ assertEquals(1, cloudMapService.listInstances(document.getId()).size(),
+ "the task's own recorded instance should stay");
+ assertEquals(List.of("172.31.0.17"), cloudMapService.resolveDnsName("auth." + namespace));
+ }
+
+ @Test
+ void evictUnrecordedInstancesMatchesALoopbackAddressOnlyByItsPort() {
+ String namespace = uniqueName("svcdisc") + ".internal";
+ Service cloudMapSvc = createDnsService(namespace, "api");
+ cloudMapService.registerInstance(cloudMapSvc.getId(), "stale", null,
+ Map.of("AWS_INSTANCE_IPV4", "127.0.0.1", "AWS_INSTANCE_PORT", "32768"), REGION);
+ cloudMapService.registerInstance(cloudMapSvc.getId(), "other-port", null,
+ Map.of("AWS_INSTANCE_IPV4", "127.0.0.1", "AWS_INSTANCE_PORT", "32769"), REGION);
+
+ registrar.evictUnrecordedInstances(task(null, "api", 8080, 32768), List.of(cloudMapSvc.getId()),
+ List.of(EcsTaskAddress.of("127.0.0.1", 32768)), REGION);
+
+ assertEquals(List.of("other-port"), cloudMapService.listInstances(cloudMapSvc.getId()).stream()
+ .map(Instance::getInstanceId).toList());
+ }
+
+ @Test
+ void evictUnrecordedInstancesMatchesAnEniAddressOnlyInItsOwnVpc() {
+ String namespace = uniqueName("svcdisc") + ".internal";
+ Service cloudMapSvc = createDnsService(namespace, "api");
+ cloudMapService.registerInstance(cloudMapSvc.getId(), "by-hand", null,
+ Map.of("AWS_INSTANCE_IPV4", "10.0.1.5"), REGION);
+
+ registrar.evictUnrecordedInstances(task("10.0.1.5", "api", 8080, 8080), List.of(cloudMapSvc.getId()),
+ List.of(new EcsTaskAddress("10.0.1.5", null, "vpc-elsewhere")), REGION);
+ assertEquals(List.of("by-hand"), cloudMapService.listInstances(cloudMapSvc.getId()).stream()
+ .map(Instance::getInstanceId).toList(), "an instance in another VPC's namespace should stay");
+
+ registrar.evictUnrecordedInstances(task("10.0.1.5", "api", 8080, 8080), List.of(cloudMapSvc.getId()),
+ List.of(new EcsTaskAddress("10.0.1.5", null, "vpc-svcdisc")), REGION);
+ assertTrue(cloudMapService.listInstances(cloudMapSvc.getId()).isEmpty(),
+ "an instance at the task's ENI address in its own VPC should be evicted");
+ }
+
+ /** Stands in for the process dying between the registration and anything after it. */
+ private static final class ProcessKilled extends Error {
+ }
+
private Service createDnsService(String namespaceName, String serviceName) {
Operation operation = cloudMapService.createPrivateDnsNamespace(
namespaceName, "vpc-svcdisc", null, null, Map.of(), REGION);
@@ -168,6 +273,12 @@ class EcsServiceDiscoveryRegistrarTest {
return task;
}
+ private EcsTask taskInCluster(String privateIpAddress, String clusterName) {
+ EcsTask task = task(privateIpAddress, "web", 8080, 8080);
+ task.setClusterArn("arn:aws:ecs:" + REGION + ":000000000000:cluster/" + clusterName);
+ return task;
+ }
+
private static String uniqueName(String prefix) {
return prefix + UUID.randomUUID().toString().substring(0, 8);
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java
new file mode 100644
index 0000000000..cde3674653
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java
@@ -0,0 +1,248 @@
+package io.github.hectorvent.floci.services.ecs;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageBackend;
+import io.github.hectorvent.floci.core.storage.StorageFactory;
+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;
+import io.github.hectorvent.floci.services.ecs.container.EcsTaskHandle;
+import io.github.hectorvent.floci.services.ecs.exec.EcsExecSessionRegistry;
+import io.github.hectorvent.floci.services.ecs.model.Container;
+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;
+import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;
+import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;
+import io.github.hectorvent.floci.services.ecs.model.LaunchType;
+import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;
+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsString;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/**
+ * A restarted Floci knows no task, since task state is memory-only, so what the previous run's
+ * tasks left behind has to go before the scheduler starts their replacements: containers a run
+ * without a graceful shutdown left serving, and the load balancer targets and Cloud Map instances
+ * that even a graceful one leaves registered.
+ */
+class EcsServiceRestartLeftoversTest {
+
+ private static final String REGION = "us-east-1";
+ private static final String TARGET_GROUP_ARN =
+ "arn:aws:elasticloadbalancing:us-east-1:000000000000:targetgroup/web/0123456789abcdef";
+
+ @Test
+ void releasesTheRegistrationsThePreviousRunRecorded() {
+ EcsLoadBalancerRegistrar lbRegistrar = mock(EcsLoadBalancerRegistrar.class);
+ EcsServiceDiscoveryRegistrar discoveryRegistrar = mock(EcsServiceDiscoveryRegistrar.class);
+
+ service(new SharedStorageFactory(), true, mock(EcsContainerManager.class),
+ lbRegistrar, discoveryRegistrar).releasePreviousRunLeftovers();
+
+ verify(lbRegistrar).releaseRecordedTargets();
+ verify(discoveryRegistrar).releaseRecordedInstances();
+ }
+
+ @Test
+ void theSchedulerWaitsUntilThePreviousRunsContainersAreRemoved() {
+ SharedStorageFactory storage = new SharedStorageFactory();
+ persistServiceWithLoadBalancer(storage);
+ EcsContainerManager containerManager = mock(EcsContainerManager.class);
+ when(containerManager.removeLeftoverContainers()).thenReturn(false, false, true);
+ EcsService restarted = service(storage, false, containerManager,
+ mock(EcsLoadBalancerRegistrar.class), null);
+
+ restarted.releasePreviousRunLeftovers();
+ restarted.reconcile();
+ verify(containerManager, never()).startTask(any(), any(), any(), anyString());
+
+ restarted.reconcile();
+ restarted.reconcile();
+ verify(containerManager, times(3)).removeLeftoverContainers();
+ }
+
+ @Test
+ void theSweepIsNotRetriedWhileNoServiceNeedsATask() {
+ EcsContainerManager containerManager = mock(EcsContainerManager.class);
+ when(containerManager.removeLeftoverContainers()).thenReturn(false);
+ EcsService restarted = service(new SharedStorageFactory(), false, containerManager,
+ mock(EcsLoadBalancerRegistrar.class), null);
+
+ restarted.releasePreviousRunLeftovers();
+ restarted.reconcile();
+ restarted.reconcile();
+
+ verify(containerManager).removeLeftoverContainers();
+ }
+
+ @Test
+ void runTaskRetriesTheSweepAndStartsNothingBesideAPreviousRunsContainers() {
+ EcsContainerManager containerManager = mock(EcsContainerManager.class);
+ when(containerManager.removeLeftoverContainers()).thenReturn(false, false, true);
+ EcsService restarted = service(new SharedStorageFactory(), false, containerManager,
+ mock(EcsLoadBalancerRegistrar.class), null);
+ restarted.createCluster("app-cluster", Map.of(), REGION);
+ ContainerDefinition container = new ContainerDefinition();
+ container.setName("web");
+ container.setImage("nginx:alpine");
+ restarted.registerTaskDefinition("web", List.of(container), NetworkMode.bridge, null, null,
+ null, null, List.of(), REGION);
+
+ restarted.releasePreviousRunLeftovers();
+ EcsTask blocked = restarted.runTask("app-cluster", "web", 1, LaunchType.EC2, null, null,
+ null, null, REGION).getFirst();
+
+ assertEquals("STOPPED", blocked.getLastStatus());
+ assertThat(blocked.getStoppedReason(), containsString("previous run"));
+ verify(containerManager, never()).startTask(any(), any(), any(), anyString());
+
+ restarted.runTask("app-cluster", "web", 1, LaunchType.EC2, null, null, null, null, REGION);
+
+ verify(containerManager, times(3)).removeLeftoverContainers();
+ verify(containerManager).startTask(any(), any(), any(), anyString());
+ }
+
+ @Test
+ void dockerModeRemovesTheContainersAPreviousRunLeft() {
+ EcsContainerManager containerManager = mock(EcsContainerManager.class);
+
+ service(new SharedStorageFactory(), false, containerManager,
+ mock(EcsLoadBalancerRegistrar.class), null).releasePreviousRunLeftovers();
+
+ verify(containerManager).removeLeftoverContainers();
+ }
+
+ @Test
+ void mockModeHasNoContainersToRemove() {
+ EcsContainerManager containerManager = mock(EcsContainerManager.class);
+
+ service(new SharedStorageFactory(), true, containerManager,
+ mock(EcsLoadBalancerRegistrar.class), null).releasePreviousRunLeftovers();
+
+ verify(containerManager, never()).removeLeftoverContainers();
+ }
+
+ @Test
+ void startupReleasesLeftoversBeforeTheSchedulerRuns() {
+ EcsContainerManager containerManager = mock(EcsContainerManager.class);
+ EcsService service = service(new SharedStorageFactory(), false, containerManager,
+ mock(EcsLoadBalancerRegistrar.class), null);
+
+ service.init();
+ try {
+ verify(containerManager).removeLeftoverContainers();
+ verify(containerManager, never()).startTask(any(), any(), any(), anyString());
+ } finally {
+ service.stopManagedContainers();
+ }
+ }
+
+ @Test
+ void aStartingTaskEvictsTheUnrecordedRegistrationsAtItsAddress() {
+ EcsContainerManager containerManager = mock(EcsContainerManager.class);
+ when(containerManager.removeLeftoverContainers()).thenReturn(true);
+ when(containerManager.startTask(any(), any(), any(), anyString())).thenAnswer(invocation -> {
+ EcsTask task = invocation.getArgument(0);
+ Container container = new Container();
+ container.setName("web");
+ container.setNetworkBindings(List.of(new NetworkBinding("0.0.0.0", 80, 80, "tcp")));
+ task.setContainers(List.of(container));
+ task.setPrivateIpAddress("10.0.1.5");
+ return new EcsTaskHandle(task.getTaskArn(), Map.of("web", "docker-id"), Map.of());
+ });
+ when(containerManager.resolveContainerHost(any())).thenReturn("172.19.0.4");
+ when(containerManager.taskVpcId(any(), anyString())).thenReturn("vpc-app");
+ EcsLoadBalancerRegistrar lbRegistrar = mock(EcsLoadBalancerRegistrar.class);
+ EcsServiceDiscoveryRegistrar discoveryRegistrar = mock(EcsServiceDiscoveryRegistrar.class);
+ when(discoveryRegistrar.cloudMapServiceIds(any())).thenReturn(List.of("srv-alpha"));
+ EcsService ecs = service(new SharedStorageFactory(), false, containerManager, lbRegistrar,
+ discoveryRegistrar);
+ ecs.releasePreviousRunLeftovers();
+ ecs.createCluster("app-cluster", Map.of(), REGION);
+ ContainerDefinition definition = new ContainerDefinition();
+ definition.setName("web");
+ definition.setImage("nginx:alpine");
+ ecs.registerTaskDefinition("web", List.of(definition), NetworkMode.bridge, null, null,
+ null, null, List.of(), REGION);
+ EcsLoadBalancer lb = new EcsLoadBalancer();
+ lb.setTargetGroupArn(TARGET_GROUP_ARN);
+ lb.setContainerName("web");
+ lb.setContainerPort(80);
+ ecs.createService("app-cluster", "alpha", "web", 0, LaunchType.EC2, List.of(lb), null, REGION);
+
+ // A task of no service at all: any container can take the address a stale entry points at.
+ EcsTask task = ecs.runTask("app-cluster", "web", 1, LaunchType.EC2, null, null, null, null, REGION)
+ .getFirst();
+
+ assertEquals("RUNNING", task.getLastStatus());
+ // A target only on the ports the container serves, and never at the VPC-scoped ENI address.
+ verify(lbRegistrar).evictUnrecordedTargets(task, Set.of(TARGET_GROUP_ARN),
+ Set.of(new EcsTaskAddress("172.19.0.4", 80)), REGION);
+ verify(discoveryRegistrar).evictUnrecordedInstances(task, Set.of("srv-alpha"),
+ Set.of(new EcsTaskAddress("10.0.1.5", null, "vpc-app"), new EcsTaskAddress("172.19.0.4", null)),
+ REGION);
+ }
+
+ private static void persistServiceWithLoadBalancer(StorageFactory storage) {
+ EcsService first = service(storage, true, mock(EcsContainerManager.class),
+ mock(EcsLoadBalancerRegistrar.class), null);
+ first.createCluster("app-cluster", Map.of(), REGION);
+ ContainerDefinition container = new ContainerDefinition();
+ container.setName("web");
+ container.setImage("nginx:alpine");
+ first.registerTaskDefinition("web", List.of(container), NetworkMode.bridge, null, null,
+ null, null, List.of(), REGION);
+ EcsLoadBalancer lb = new EcsLoadBalancer();
+ lb.setTargetGroupArn(TARGET_GROUP_ARN);
+ lb.setContainerName("web");
+ lb.setContainerPort(80);
+ first.createService("app-cluster", "web-svc", "web", 1, LaunchType.EC2, List.of(lb), null, REGION);
+ }
+
+ private static EcsService service(StorageFactory storage, boolean mockMode,
+ EcsContainerManager containerManager,
+ EcsLoadBalancerRegistrar lbRegistrar,
+ EcsServiceDiscoveryRegistrar discoveryRegistrar) {
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.services().ecs().mock()).thenReturn(mockMode);
+ when(config.effectiveBaseUrl()).thenReturn("http://localhost:4566");
+ EcsService service = new EcsService(new RegionResolver(REGION, "000000000000"), containerManager,
+ config, lbRegistrar, storage, null, new EcsExecSessionRegistry(), discoveryRegistrar);
+ service.initializeStorage();
+ return service;
+ }
+
+ private static final class SharedStorageFactory extends StorageFactory {
+ private final Map<String, StorageBackend<String, ?>> stores = new HashMap<>();
+
+ private SharedStorageFactory() {
+ super(null, null);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <V> AccountAwareStorageBackend<V> create(String serviceName,
+ String fileName,
+ TypeReference<Map<String, V>> typeReference) {
+ return (AccountAwareStorageBackend<V>) stores.computeIfAbsent(fileName,
+ ignored -> AccountAwareStorageBackend.inMemory("000000000000"));
+ }
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java
index 024150fd83..0085ce351e 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java
@@ -98,7 +98,7 @@ class EcsContainerManagerSecurityGroupTest {
void awsvpcTaskWithFirewallJoinsHelperNamespaceAndSkipsHostPorts() {
when(containerDetector.isRunningInContainer()).thenReturn(false);
when(firewallManager.enabled()).thenReturn(true);
- when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any()))
+ when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any(), any()))
.thenReturn(new SecurityGroupFirewallManager.Namespace("helper-id", "10.0.0.5"));
NetworkInterface eni = new NetworkInterface();
@@ -149,7 +149,7 @@ class EcsContainerManagerSecurityGroupTest {
void stopTaskUnregistersEniAndReleasingTheTaskNetworkDeletesIt() {
when(containerDetector.isRunningInContainer()).thenReturn(false);
when(firewallManager.enabled()).thenReturn(true);
- when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any()))
+ when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any(), any()))
.thenReturn(new SecurityGroupFirewallManager.Namespace("helper-id", "10.0.0.5"));
NetworkInterface eni = new NetworkInterface();
@@ -185,7 +185,7 @@ class EcsContainerManagerSecurityGroupTest {
void firelensRouterAndAppBothJoinTheHelperNamespace() {
when(containerDetector.isRunningInContainer()).thenReturn(false);
when(firewallManager.enabled()).thenReturn(true);
- when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any()))
+ when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any(), any()))
.thenReturn(new SecurityGroupFirewallManager.Namespace("helper-id", "10.0.0.5"));
NetworkInterface eni = new NetworkInterface();
@@ -233,7 +233,7 @@ class EcsContainerManagerSecurityGroupTest {
verify(builder, times(2)).withNetworkMode("container:helper-id");
verify(builder, times(2)).withLabels(Map.of("floci.security-group-workload", "true"));
- verify(firewallManager).createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any());
+ verify(firewallManager).createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any(), any());
verify(lifecycleManager).create(any());
verify(lifecycleManager).startCreated(eq("router-id"), any());
verify(lifecycleManager).createAndStart(any());