Light, fluffy, and always free - The AWS Local Emulator alternative
fix(ecs): release stale task containers and registrations, and evict ones that would misroute a reused address (#4513)
Abhigya451a30f98e9b
diff --git a/docs/services/ecs.md b/docs/services/ecs.mdindex 037e7b408b..92fa31d403 100644--- a/docs/services/ecs.md+++ b/docs/services/ecs.md@@ -496,6 +496,45 @@ Known differences from AWS:registers `HEALTHY`, which is Cloud Map's own default for a `RegisterInstance` that names no`AWS_INIT_HEALTH_STATUS`, and stays there until it is deregistered.+#### Restarts++Task state is memory-only, so a restarted Floci knows no task and the service scheduler starts+new ones to reach each service's desired count. Floci therefore clears what the previous run's+tasks left behind before it starts their replacements, so a service is never served by a task+nothing manages:++- In Docker mode, every ECS container a previous run of this Floci left on the daemon is+ removed. A graceful shutdown already stops them; this covers a run that ended without one+ (SIGKILL, OOM, a stop timeout that expired mid-drain). Containers are recognised by the+ `floci_owner_port` label (the resource namespace and API port), so the containers of another+ Floci sharing the daemon stay, and are told apart from the current run's by a per-process+ `floci.ecs-run` label rather than by creation time. Containers created by a Floci version without that label are+ not recognised and must be removed by hand once. If Docker cannot list or remove one, the+ ECS starts no task until it is gone, retrying the removal before each launch: a service's+ replacement on a reconciliation tick, or a `RunTask` or `StartTask` call, whose task then stops+ with a `TaskFailedToStart` reason naming the leftovers. Each distinct failure is logged as a+ warning once.+- Every load balancer target and Cloud Map instance a previous run's tasks registered is+ deregistered. Floci registers task containers by address, and Docker hands a dead container's+ address to the next container it starts, so a stale target would route to an unrelated+ container. ECS records each registration it makes in its persisted state and releases exactly+ those, so a target or instance registered through the ELBv2 or Cloud Map API, even into the+ same target group or Cloud Map service, stays. A registration is recorded before it is made,+ so one cut short by the process dying is still released. A target already in its group when a+ task starts is recorded as the task's own, since its address and port reach only that task, so+ it is deregistered when the task stops, as ECS does on AWS.+- Registrations no ECS task recorded, as those a Floci version without the record made, are+ removed once they would misroute: when a task starts, each registration that points at it and+ that no running task recorded is deregistered from the target groups and Cloud Map services an+ ECS service names, and a warning names it. Such an entry was left by a task that is gone, and+ would otherwise send another service's traffic to the new task. A target points at the task+ when it is at its container's address on the Docker network and on a port the container+ publishes. A Cloud Map instance points at it when it is at that container address, on any port+ since a DNS answer carries none, or at the task's ENI address in a namespace of the task's own+ VPC, as ENI addresses are unique only within a VPC. A loopback address, which every task+ published on the host shares, matches only on the task's host ports. Target groups and Cloud+ Map services no ECS service names are never touched.+#### Unknown servicesA service reference that does not resolve is returned in `failures` withdiff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.javaindex 33f0eceeae..f9e3eba1c4 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerStorageHelper.java@@ -35,6 +35,8 @@ public final class ContainerStorageHelper {/** {@link #NAME_PREFIX} with its separator, the literal prefix of every name produced here. */static final String CONTAINER_PREFIX = NAME_PREFIX + "-";+ /** The label carrying {@link #ownerIdentity(EmulatorConfig)}. */+ public static final String OWNER_LABEL = "floci_owner_port";/*** The prefix this emulator used before it took the shared {@code floci-<cloud>-} convention.@@ -184,6 +186,18 @@ public final class ContainerStorageHelper {return labels;}+ /**+ * Identity of the Floci deployment that owns a container, stamped as {@link #OWNER_LABEL} so a+ * startup sweep collects only its own leftovers. The API port alone collides when two+ * independently namespaced Flocis share a Docker daemon on the same internal port, so the+ * resource namespace goes in front of it; an unnamespaced Floci keeps the bare port.+ */+ public static String ownerIdentity(EmulatorConfig config) {+ String namespace = config.docker() == null || config.docker().resourceNamespace() == null+ ? "" : config.docker().resourceNamespace().orElse("");+ return namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();+ }+private static void putIfNotBlank(Map<String, String> labels, String key, String value) {if (value != null && !value.isBlank()) {labels.put(key, value);diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.javaindex 6e12d7ffa1..8679095119 100644--- a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java@@ -116,7 +116,7 @@ public class Ec2ContainerManager {* unless a resource namespace is configured, so it cannot scope the default configuration.* Containers created before this label existed carry no owner and are therefore never swept.*/- static final String LABEL_OWNER_PORT = "floci_owner_port";+ static final String LABEL_OWNER_PORT = ContainerStorageHelper.OWNER_LABEL;/*** Identity of the Floci deployment that owns a container, for scoping the startup sweep.@@ -126,9 +126,7 @@ public class Ec2ContainerManager {* unnamespaced single Floci keeps the bare port it already stamped.*/private String ownerIdentity() {- String ns = config.docker() == null || config.docker().resourceNamespace() == null- ? "" : config.docker().resourceNamespace().orElse("");- return ns.isBlank() ? String.valueOf(config.port()) : ns + "/" + config.port();+ return ContainerStorageHelper.ownerIdentity(config);}static final String LABEL_SERVICE = "io.floci.service";static final String SERVICE_VALUE = "ec2";@@ -192,7 +190,7 @@ public class Ec2ContainerManager {if (!"true".equals(labels.get("floci.security-group-helper"))|| !"ec2".equals(labels.get(LABEL_SERVICE))|| !instance.getInstanceId().equals(labels.get(LABEL_RESOURCE_ID))- || !ownerIdentity().equals(labels.get("floci_owner_port"))) {+ || !ownerIdentity().equals(labels.get(LABEL_OWNER_PORT))) {throw new IllegalStateException("EC2 workload network namespace is not Floci protected");}String address = helper.getNetworkSettings().getNetworks().values().stream()diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.javaindex e6590faad0..ed0ef2d1b7 100644--- a/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java@@ -67,12 +67,11 @@ public class SecurityGroupFirewallManager {return;}try {- String namespace = config.docker().resourceNamespace().orElse("");- String owner = namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();+ String owner = ContainerStorageHelper.ownerIdentity(config);dockerClient.listContainersCmd().withShowAll(true).withLabelFilter(Map.of("floci.security-group-helper", "true")).exec().stream()- .filter(container -> owner.equals(container.getLabels().get("floci_owner_port")))+ .filter(container -> owner.equals(container.getLabels().get(ContainerStorageHelper.OWNER_LABEL))).forEach(container -> {if ("running".equals(container.getState())) {quarantine(container.getId());@@ -92,6 +91,13 @@ public class SecurityGroupFirewallManager {/** The helper starts before any workload process and owns all published ports. */public Namespace createNamespace(String service, String resourceId, String accountId, String region,Optional<String> dockerNetwork, Map<Integer, Integer> portBindings) {+ return createNamespace(service, resourceId, accountId, region, dockerNetwork, portBindings, Map.of());+ }++ /** {@link #createNamespace} with extra labels on the helper, as the owning service tracks it by. */+ public Namespace createNamespace(String service, String resourceId, String accountId, String region,+ Optional<String> dockerNetwork, Map<Integer, Integer> portBindings,+ Map<String, String> extraLabels) {if (!enabled()) {throw new IllegalStateException("Security-group enforcement is disabled");}@@ -106,8 +112,6 @@ public class SecurityGroupFirewallManager {ensureHelperImage();String name = ContainerStorageHelper.resourceName(config, "sg", null,resourceId.replaceAll("[^a-zA-Z0-9_.-]", "-"));- String namespace = config.docker().resourceNamespace().orElse("");- String owner = namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();ContainerBuilder.Builder builder = containerBuilder.newContainer(config.network().securityGroupEnforcement().helperImage()).withName(name).withDockerNetwork(dockerNetwork)@@ -117,7 +121,9 @@ public class SecurityGroupFirewallManager {.withEntrypoint(List.of("sh", "-c")).withCmd(List.of("exec sleep 2147483647")).withLabels(ContainerStorageHelper.resourceIdentityLabels(service, resourceId, accountId, region))- .withLabels(Map.of("floci.security-group-helper", "true", "floci_owner_port", owner));+ .withLabels(Map.of("floci.security-group-helper", "true",+ ContainerStorageHelper.OWNER_LABEL, ContainerStorageHelper.ownerIdentity(config)))+ .withLabels(extraLabels);if (portBindings != null) {portBindings.forEach(builder::withPortBinding);}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrar.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrar.javaindex d8d4372001..14e8db8500 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrar.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrar.java@@ -1,18 +1,31 @@package io.github.hectorvent.floci.services.ecs;+import com.fasterxml.jackson.core.type.TypeReference;+import io.github.hectorvent.floci.core.common.RequestScopes;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.StorageFactory;import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;import io.github.hectorvent.floci.services.ecs.model.Container;import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;+import io.github.hectorvent.floci.services.ecs.model.EcsRegisteredTargets;import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;import io.github.hectorvent.floci.services.elbv2.ElbV2Service;import io.github.hectorvent.floci.services.elbv2.model.TargetDescription;+import io.github.hectorvent.floci.services.elbv2.model.TargetHealth;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;+import java.util.ArrayList;+import java.util.Collection;+import java.util.HashSet;import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.Set;import java.util.function.BiConsumer;/**@@ -30,37 +43,155 @@ public class EcsLoadBalancerRegistrar {private final ElbV2Service elbV2Service;private final EcsContainerManager containerManager;+ // taskArn → the targets registered for it, persisted so they can be released after a restart+ private final AccountAwareStorageBackend<EcsRegisteredTargets> ledger;@Inject- public EcsLoadBalancerRegistrar(ElbV2Service elbV2Service, EcsContainerManager containerManager) {+ public EcsLoadBalancerRegistrar(ElbV2Service elbV2Service, EcsContainerManager containerManager,+ StorageFactory storageFactory) {this.elbV2Service = elbV2Service;this.containerManager = containerManager;+ this.ledger = storageFactory.create("ecs", "ecs-registered-targets.json",+ new TypeReference<Map<String, EcsRegisteredTargets>>() {});}- /** Registers the task's load-balanced containers as ELBv2 targets. */+ /**+ * Registers the task's load-balanced containers as ELBv2 targets. The targets are recorded+ * before they are registered, so a process killed in between still leaves a record for the+ * next run to release. A target already in its group is recorded too: its address and port+ * are the task's own container, so it reaches nothing else, and once the task stops it would+ * reach nothing or whichever container Docker hands the address to next. ECS on AWS likewise+ * deregisters a stopping task's target whoever registered it.+ */public void registerTask(EcsTask task, EcsServiceModel svc, String region) {- forEachTarget(task, svc, (tgArn, td) -> {+ List<EcsRegisteredTargets.Target> planned = new ArrayList<>();+ forEachTarget(task, svc, (tgArn, td) ->+ planned.add(new EcsRegisteredTargets.Target(tgArn, td.getId(), td.getPort())));+ if (planned.isEmpty()) {+ return;+ }+ String taskArn = task.getTaskArn();+ if (taskArn != null) {+ ledger.put(taskArn, new EcsRegisteredTargets(region, List.copyOf(planned)));+ }+ List<EcsRegisteredTargets.Target> registered = new ArrayList<>();+ for (EcsRegisteredTargets.Target target : planned) {try {- elbV2Service.registerTargets(region, tgArn, List.of(td));+ elbV2Service.registerTargets(region, target.targetGroupArn(), List.of(target.toDescription()));+ registered.add(target);LOG.infov("Registered ECS task target {0}:{1} into target group {2}",- td.getId(), td.getPort(), tgArn);+ target.id(), target.port(), target.targetGroupArn());} catch (Exception e) {- LOG.warnv("Could not register ECS target into {0}: {1}", tgArn, e.getMessage());+ LOG.warnv("Could not register ECS target into {0}: {1}", target.targetGroupArn(), e.getMessage());}- });+ }+ if (taskArn == null || registered.size() == planned.size()) {+ return;+ }+ if (registered.isEmpty()) {+ ledger.delete(taskArn);+ } else {+ ledger.put(taskArn, new EcsRegisteredTargets(region, List.copyOf(registered)));+ }}- /** Deregisters the task's load-balanced containers from their ELBv2 target groups. */+ /**+ * Deregisters the targets {@link #registerTask} recorded for the task, and no other: a task+ * with no record registered nothing, as when none of its containers had a reachable binding.+ * Only a task without an ARN, which {@link #registerTask} cannot record, falls back to the+ * targets its containers resolve to now.+ */public void deregisterTask(EcsTask task, EcsServiceModel svc, String region) {- forEachTarget(task, svc, (tgArn, td) -> {+ if (task.getTaskArn() == null) {+ forEachTarget(task, svc, (tgArn, td) -> deregister(region, tgArn, td));+ return;+ }+ Optional<EcsRegisteredTargets> recorded = ledger.get(task.getTaskArn());+ if (recorded.isPresent()) {+ deregister(recorded.get());+ ledger.delete(task.getTaskArn());+ }+ }++ /**+ * Deregisters every target ECS recorded for a task, in every account. Only for startup, when+ * ECS holds no task at all: task state is memory-only, so each recorded target belongs to a+ * task of a previous run whose container is gone or is being removed, and Docker is free to+ * hand its address to an unrelated container. Targets ECS did not register are never touched.+ */+ public void releaseRecordedTargets() {+ for (AccountAwareStorageBackend.AccountEntry<EcsRegisteredTargets> entry+ : ledger.scanAllAccountEntries(key -> true)) {+ RequestScopes.runAs(entry.accountId(), () -> deregister(entry.value()));+ ledger.deleteForAccount(entry.accountId(), entry.key());+ LOG.infov("Released the load balancer targets of ECS task {0} left by a previous run", entry.key());+ }+ }++ /**+ * Deregisters, from the given target groups, each target at one of a starting task's addresses+ * and ports that no ECS task recorded. The address was free until this task took it, so such a target+ * was left by a task that is gone, as one registered by a Floci version that kept no record,+ * and would now send the group's traffic to this task, which belongs to some other service.+ * A target a live task recorded stays, whichever group it is in.+ */+ public void evictUnrecordedTargets(EcsTask task, Collection<String> targetGroupArns,+ Collection<EcsTaskAddress> addresses, String region) {+ if (targetGroupArns.isEmpty() || addresses.isEmpty()) {+ return;+ }+ Set<EcsRegisteredTargets.Target> recorded = new HashSet<>();+ for (EcsRegisteredTargets entry : ledger.scan(key -> true)) {+ if (entry.targets() != null) {+ recorded.addAll(entry.targets());+ }+ }+ for (String tgArn : targetGroupArns) {+ List<TargetHealth> health;try {- elbV2Service.deregisterTargets(region, tgArn, List.of(td));- LOG.infov("Deregistered ECS task target {0}:{1} from target group {2}",- td.getId(), td.getPort(), tgArn);+ health = elbV2Service.describeTargetHealth(region, tgArn, null);} catch (Exception e) {- LOG.warnv("Could not deregister ECS target from {0}: {1}", tgArn, e.getMessage());+ // A target group deleted while a service still names it: nothing in it to evict.+ LOG.debugv("Could not list the targets of {0}: {1}", tgArn, e.getMessage());+ continue;+ }+ for (TargetHealth th : health) {+ TargetDescription td = th.getTarget();+ boolean atTaskAddress = addresses.stream().anyMatch(a -> a.matches(td.getId(), td.getPort()));+ if (!atTaskAddress+ || recorded.contains(new EcsRegisteredTargets.Target(tgArn, td.getId(), td.getPort()))) {+ continue;+ }+ try {+ elbV2Service.deregisterTargets(region, tgArn, List.of(td));+ LOG.warnv("Deregistered stale target {0}:{1} from target group {2}: no ECS task registered it,"+ + " and its address now belongs to ECS task {3}", td.getId(), td.getPort(), tgArn,+ task.getTaskArn());+ } catch (Exception e) {+ LOG.warnv("Could not deregister stale target {0}:{1} from {2}: {3}", td.getId(), td.getPort(),+ tgArn, e.getMessage());+ }}- });+ }+ }++ private void deregister(EcsRegisteredTargets recorded) {+ if (recorded.targets() == null) {+ return;+ }+ for (EcsRegisteredTargets.Target target : recorded.targets()) {+ deregister(recorded.region(), target.targetGroupArn(), target.toDescription());+ }+ }++ private void deregister(String region, String tgArn, TargetDescription td) {+ try {+ elbV2Service.deregisterTargets(region, tgArn, List.of(td));+ LOG.infov("Deregistered ECS task target {0}:{1} from target group {2}",+ td.getId(), td.getPort(), tgArn);+ } catch (Exception e) {+ LOG.warnv("Could not deregister ECS target from {0}: {1}", tgArn, e.getMessage());+ }}private void forEachTarget(EcsTask task, EcsServiceModel svc,diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.javaindex 5c8ee4e462..3610a2ac41 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java@@ -37,11 +37,13 @@ import io.github.hectorvent.floci.services.ecs.model.EcsCluster;import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;import io.github.hectorvent.floci.services.ecs.model.KeyValuePair;import io.github.hectorvent.floci.services.ecs.model.LaunchType;import io.github.hectorvent.floci.services.ecs.model.ListTasksRequest;import io.github.hectorvent.floci.services.ecs.model.ManagedAgent;+import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;import io.github.hectorvent.floci.services.ecs.model.NetworkConfiguration;import io.github.hectorvent.floci.services.ecs.model.NetworkMode;import io.github.hectorvent.floci.services.ecs.model.ProtectedTask;@@ -116,6 +118,9 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab// Replaced by afterReset() after a state reset, whose container teardown shuts this scheduler down.private volatile ScheduledExecutorService reconciler = newReconciler();private final Object reconcilerLock = new Object();+ // False until a previous run's task containers are confirmed gone; no task starts before.+ private volatile boolean leftoverContainersRemoved = true;+ private volatile boolean leftoverContainersBlockReported;// region::clusterName → EcsClusterprivate Map<String, EcsCluster> clusters = new ConcurrentHashMap<>();@@ -218,9 +223,33 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab@PostConstructvoid init() {initializeStorage();+ releasePreviousRunLeftovers();scheduleReconciliation(reconciler);}+ /**+ * Clears what the tasks of a previous run left behind, before the scheduler starts their+ * replacements. Task state is memory-only, so a restarted Floci knows no task, yet a run that+ * ended without a graceful shutdown leaves its task containers serving, and even a graceful one+ * leaves their load balancer targets and Cloud Map instances registered. Left alone, each+ * service would serve from its replacements and a task nothing manages, and a dead task's+ * target would follow its address to whichever container Docker hands it to next. Only what+ * the registrars recorded registering is released, never a target or instance registered by+ * hand. A container Docker would not remove is retried by {@link #leftoverContainersCleared()}+ * before each task launch, and no task starts until it is gone.+ */+ void releasePreviousRunLeftovers() {+ if (dockerMode) {+ leftoverContainersRemoved = containerManager.removeLeftoverContainers();+ }+ if (lbRegistrar != null) {+ lbRegistrar.releaseRecordedTargets();+ }+ if (discoveryRegistrar != null) {+ discoveryRegistrar.releaseRecordedInstances();+ }+ }+private void scheduleReconciliation(ScheduledExecutorService scheduler) {scheduler.scheduleAtFixedRate(this::reconcile, 5, 5, TimeUnit.SECONDS);}@@ -1308,6 +1337,10 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettabif (dockerMode) {try {+ if (!leftoverContainersCleared()) {+ throw new IllegalStateException(+ "the containers a previous run of Floci left on the Docker daemon could not be removed yet");+ }task.setPullStartedAt(Instant.now());EcsTaskHandle handle = containerManager.startTask(task, taskDef, containerOverrides, region);task.setPullStoppedAt(Instant.now());@@ -1321,6 +1354,7 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettabif (!stopRequested) {registerTaskWithLoadBalancers(task, cluster, region);registerTaskForServiceDiscovery(task, cluster, region);+ evictStaleRegistrationsAt(task, region);}if (eventPublisher != null) {eventPublisher.emitTaskLadder(task, TaskStatus.PENDING, TaskStatus.RUNNING, region);@@ -1792,6 +1826,102 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab}}+ /**+ * Removes the load balancer targets and Cloud Map instances that point at a freshly-started+ * task's addresses but that no ECS task recorded, from every target group and Cloud Map service+ * an ECS service in the region names. Docker hands a dead container's address to the next+ * container it starts, so a registration its task never released, as one made by a Floci+ * version that kept no record, would otherwise route another service's traffic to this task.+ * Runs for every task, not only a service's: any container can take the address.+ */+ private void evictStaleRegistrationsAt(EcsTask task, String region) {+ Set<String> targetGroupArns = new LinkedHashSet<>();+ Set<String> cloudMapServiceIds = new LinkedHashSet<>();+ for (Map.Entry<String, EcsServiceModel> entry : services.entrySet()) {+ EcsServiceModel svc = entry.getValue();+ if (!region.equals(extractRegionFromServiceKey(entry.getKey())) || "INACTIVE".equals(svc.getStatus())) {+ continue;+ }+ for (EcsLoadBalancer lb : svc.getLoadBalancers()) {+ if (lb.getTargetGroupArn() != null && !lb.getTargetGroupArn().isBlank()) {+ targetGroupArns.add(lb.getTargetGroupArn());+ }+ }+ if (discoveryRegistrar != null) {+ cloudMapServiceIds.addAll(discoveryRegistrar.cloudMapServiceIds(svc));+ }+ }+ if (targetGroupArns.isEmpty() && cloudMapServiceIds.isEmpty()) {+ return;+ }+ if (lbRegistrar != null && !targetGroupArns.isEmpty()) {+ lbRegistrar.evictUnrecordedTargets(task, targetGroupArns, targetAddresses(task), region);+ }+ if (discoveryRegistrar != null && !cloudMapServiceIds.isEmpty()) {+ discoveryRegistrar.evictUnrecordedInstances(task, cloudMapServiceIds, instanceAddresses(task, region),+ region);+ }+ }++ /**+ * The address and port pairs at which a load balancer target reaches a running task: each+ * container's address, on the ports its bindings declare. The ENI address is not among them,+ * since ECS registers no target at it and it is unique only within its VPC, and a target on a+ * port the task does not serve sends it nothing.+ */+ private Set<EcsTaskAddress> targetAddresses(EcsTask task) {+ Set<EcsTaskAddress> addresses = new LinkedHashSet<>();+ if (task.getContainers() == null) {+ return addresses;+ }+ for (Container container : task.getContainers()) {+ String host = containerManager.resolveContainerHost(container);+ if (host == null || host.isBlank() || container.getNetworkBindings() == null) {+ continue;+ }+ for (NetworkBinding binding : container.getNetworkBindings()) {+ addresses.add(new EcsTaskAddress(host, binding.hostPort()));+ if (!EcsTaskAddress.isLoopback(host)) {+ addresses.add(new EcsTaskAddress(host, binding.containerPort()));+ }+ }+ }+ return addresses;+ }++ /**+ * The addresses at which a Cloud Map instance resolves to a running task: its ENI address,+ * within the task's VPC only, and each container's address on Docker, on any port since a+ * DNS answer carries none. A loopback address counts only on the task's host ports.+ */+ private Set<EcsTaskAddress> instanceAddresses(EcsTask task, String region) {+ Set<EcsTaskAddress> addresses = new LinkedHashSet<>();+ String eniAddress = task.getPrivateIpAddress();+ if (eniAddress != null && !eniAddress.isBlank() && !EcsTaskAddress.isLoopback(eniAddress)) {+ String vpcId = containerManager.taskVpcId(task, region);+ if (vpcId != null) {+ addresses.add(new EcsTaskAddress(eniAddress, null, vpcId));+ }+ }+ if (task.getContainers() == null) {+ return addresses;+ }+ for (Container container : task.getContainers()) {+ String host = containerManager.resolveContainerHost(container);+ if (host == null || host.isBlank()) {+ continue;+ }+ if (!EcsTaskAddress.isLoopback(host)) {+ addresses.add(new EcsTaskAddress(host, null));+ } else if (container.getNetworkBindings() != null) {+ for (NetworkBinding binding : container.getNetworkBindings()) {+ addresses.add(EcsTaskAddress.of(host, binding.hostPort()));+ }+ }+ }+ return addresses;+ }+/*** Resolves the service that actually launched {@code task}, or {@code null} for a* caller-driven task. Keyed off the reconciler-stamped {@code owningServiceArn} rather than@@ -3949,6 +4079,29 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab}}+ /**+ * Whether a task may start: true once the containers a previous run left behind are gone,+ * retrying the sweep otherwise. It runs before every task launch, RunTask and StartTask+ * included, and the service scheduler asks it before planning one, so a service it blocks+ * creates no failed task on every tick. Nothing retries it while no task is due, so an+ * unreachable Docker daemon costs nothing then, and a blocked launch is reported once.+ */+ private boolean leftoverContainersCleared() {+ if (leftoverContainersRemoved) {+ return true;+ }+ leftoverContainersRemoved = containerManager.removeLeftoverContainers();+ if (leftoverContainersRemoved) {+ LOG.info("The containers a previous ECS run left behind are gone; starting service tasks");+ } else {+ LOG.log(leftoverContainersBlockReported ? Logger.Level.DEBUG : Logger.Level.WARN,+ "Not starting ECS tasks while a previous run's containers remain;"+ + " retrying before each task launch");+ leftoverContainersBlockReported = true;+ }+ return leftoverContainersRemoved;+ }+private Set<String> reconcilableAccountIds() {Set<String> accountIds = new LinkedHashSet<>();accountIds.add(regionResolver.getAccountId());@@ -4236,6 +4389,9 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettabif (covered.contains(ci.getContainerInstanceArn())) {continue;}+ if (!leftoverContainersCleared()) {+ break;+ }try {EcsTask launched = launchServiceTask(cluster, svc, LaunchType.EC2,ci.getContainerInstanceArn(), region);@@ -4373,6 +4529,9 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab}if (current < svc.getDesiredCount()) {+ if (!leftoverContainersCleared()) {+ return;+ }int toStart = svc.getDesiredCount() - (int) current;for (int i = 0; i < toStart; i++) {try {diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrar.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrar.javaindex 50f13ff7d0..1c82e9b572 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrar.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrar.java@@ -1,15 +1,25 @@package io.github.hectorvent.floci.services.ecs;+import com.fasterxml.jackson.core.type.TypeReference;+import io.github.hectorvent.floci.core.common.RequestScopes;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.StorageFactory;import io.github.hectorvent.floci.services.cloudmap.CloudMapService;+import io.github.hectorvent.floci.services.cloudmap.model.Instance;import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;import io.github.hectorvent.floci.services.ecs.model.Container;+import io.github.hectorvent.floci.services.ecs.model.EcsRegisteredInstances;import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.jboss.logging.Logger;+import java.util.ArrayList;+import java.util.Collection;+import java.util.HashSet;import java.util.LinkedHashMap;import java.util.LinkedHashSet;import java.util.List;@@ -35,17 +45,26 @@ public class EcsServiceDiscoveryRegistrar {private final CloudMapService cloudMapService;private final EcsContainerManager containerManager;+ // taskArn → the instance registered for it, persisted so it can be released after a restart+ private final AccountAwareStorageBackend<EcsRegisteredInstances> ledger;@Inject- public EcsServiceDiscoveryRegistrar(CloudMapService cloudMapService, EcsContainerManager containerManager) {+ public EcsServiceDiscoveryRegistrar(CloudMapService cloudMapService, EcsContainerManager containerManager,+ StorageFactory storageFactory) {this.cloudMapService = cloudMapService;this.containerManager = containerManager;+ this.ledger = storageFactory.create("ecs", "ecs-registered-instances.json",+ new TypeReference<Map<String, EcsRegisteredInstances>>() {});}- /** Registers the task as a Cloud Map instance of every service registry the ECS service declares. */+ /**+ * Registers the task as a Cloud Map instance of every service registry the ECS service declares.+ * The registrations are recorded before they are made, so a process killed in between still+ * leaves a record for the next run to release; releasing one that never registered is harmless.+ */public void registerTask(EcsTask task, EcsServiceModel svc, String region) {String instanceId = instanceId(task);- Set<String> registered = new LinkedHashSet<>();+ Map<String, Map<String, String>> planned = new LinkedHashMap<>();for (Map<String, Object> registry : registries(svc)) {String cloudMapServiceId = cloudMapServiceId(registry);if (cloudMapServiceId == null) {@@ -57,6 +76,16 @@ public class EcsServiceDiscoveryRegistrar {task.getTaskArn(), cloudMapServiceId);continue;}+ planned.putIfAbsent(cloudMapServiceId, attributes);+ }+ String taskArn = task.getTaskArn();+ if (!planned.isEmpty() && taskArn != null) {+ ledger.put(taskArn, new EcsRegisteredInstances(region, instanceId, List.copyOf(planned.keySet())));+ }+ Set<String> registered = new LinkedHashSet<>();+ for (Map.Entry<String, Map<String, String>> entry : planned.entrySet()) {+ String cloudMapServiceId = entry.getKey();+ Map<String, String> attributes = entry.getValue();try {cloudMapService.registerInstance(cloudMapServiceId, instanceId, null, attributes, region);registered.add(cloudMapServiceId);@@ -68,6 +97,14 @@ public class EcsServiceDiscoveryRegistrar {}}task.setServiceDiscoveryServiceIds(List.copyOf(registered));+ if (taskArn == null || registered.size() == planned.size()) {+ return;+ }+ if (registered.isEmpty()) {+ ledger.delete(taskArn);+ } else {+ ledger.put(taskArn, new EcsRegisteredInstances(region, instanceId, List.copyOf(registered)));+ }}/** Deregisters the task from the Cloud Map services it actually registered in. */@@ -86,12 +123,123 @@ public class EcsServiceDiscoveryRegistrar {}}task.setServiceDiscoveryServiceIds(List.of());+ if (task.getTaskArn() != null) {+ ledger.delete(task.getTaskArn());+ }+ }++ /**+ * Deregisters every Cloud Map instance ECS recorded for a task, in every account. Only for+ * startup, when ECS holds no task at all: task state is memory-only, so none of them can still+ * be a live task. Instances ECS did not register, including ones registered through the Cloud+ * Map API, are never touched.+ */+ public void releaseRecordedInstances() {+ for (AccountAwareStorageBackend.AccountEntry<EcsRegisteredInstances> entry+ : ledger.scanAllAccountEntries(key -> true)) {+ EcsRegisteredInstances recorded = entry.value();+ List<String> cloudMapServiceIds = recorded.cloudMapServiceIds() != null+ ? recorded.cloudMapServiceIds() : List.of();+ RequestScopes.runAs(entry.accountId(), () -> {+ for (String cloudMapServiceId : cloudMapServiceIds) {+ try {+ cloudMapService.deregisterInstance(cloudMapServiceId, recorded.instanceId(),+ recorded.region());+ LOG.infov("Deregistered Cloud Map instance {0} of {1} left by a previous ECS task",+ recorded.instanceId(), cloudMapServiceId);+ } catch (Exception e) {+ // Already gone, as when its Cloud Map service was deleted: nothing left to release.+ LOG.debugv("Could not deregister Cloud Map instance {0} of {1}: {2}",+ recorded.instanceId(), cloudMapServiceId, e.getMessage());+ }+ }+ });+ ledger.deleteForAccount(entry.accountId(), entry.key());+ }+ }++ /**+ * Deregisters, from the given Cloud Map services, each instance at one of a starting task's+ * addresses that no ECS task recorded. The counterpart of+ * {@link EcsLoadBalancerRegistrar#evictUnrecordedTargets}: such an instance was left by a task+ * that is gone, as one registered by a Floci version that kept no record, and would now resolve+ * the service's name to this task. An instance a live task recorded stays, and an address+ * scoped to a VPC counts only in a Cloud Map service whose namespace is in that VPC.+ */+ public void evictUnrecordedInstances(EcsTask task, Collection<String> cloudMapServiceIds,+ Collection<EcsTaskAddress> addresses, String region) {+ if (cloudMapServiceIds.isEmpty() || addresses.isEmpty()) {+ return;+ }+ Set<String> recorded = new HashSet<>();+ for (EcsRegisteredInstances entry : ledger.scan(key -> true)) {+ if (entry.cloudMapServiceIds() != null) {+ entry.cloudMapServiceIds().forEach(id -> recorded.add(id + "/" + entry.instanceId()));+ }+ }+ for (String cloudMapServiceId : cloudMapServiceIds) {+ List<Instance> instances;+ List<EcsTaskAddress> scoped;+ try {+ String vpc = cloudMapService.getNamespace(+ cloudMapService.getService(cloudMapServiceId).getNamespaceId()).getVpc();+ scoped = addresses.stream().filter(a -> a.appliesTo(vpc)).toList();+ instances = cloudMapService.listInstances(cloudMapServiceId);+ } catch (Exception e) {+ // A Cloud Map service deleted while an ECS service still names it: nothing to evict.+ LOG.debugv("Could not list the instances of Cloud Map service {0}: {1}", cloudMapServiceId,+ e.getMessage());+ continue;+ }+ for (Instance instance : instances) {+ Map<String, String> attributes = instance.getAttributes() != null ? instance.getAttributes() : Map.of();+ String ip = attributes.get("AWS_INSTANCE_IPV4");+ Integer port = parsePort(attributes.get("AWS_INSTANCE_PORT"));+ if (ip == null || scoped.stream().noneMatch(a -> a.matches(ip, port))+ || recorded.contains(cloudMapServiceId + "/" + instance.getInstanceId())) {+ continue;+ }+ try {+ cloudMapService.deregisterInstance(cloudMapServiceId, instance.getInstanceId(), region);+ LOG.warnv("Deregistered stale Cloud Map instance {0} of {1} at {2}: no ECS task registered it,"+ + " and its address now belongs to ECS task {3}", instance.getInstanceId(),+ cloudMapServiceId, ip, task.getTaskArn());+ } catch (Exception e) {+ LOG.warnv("Could not deregister stale Cloud Map instance {0} of {1}: {2}",+ instance.getInstanceId(), cloudMapServiceId, e.getMessage());+ }+ }+ }}public boolean hasRegistries(EcsServiceModel svc) {return !registries(svc).isEmpty();}+ /** The Cloud Map services the ECS service's registries name. */+ public List<String> cloudMapServiceIds(EcsServiceModel svc) {+ List<String> ids = new ArrayList<>();+ for (Map<String, Object> registry : registries(svc)) {+ String id = cloudMapServiceId(registry);+ if (id != null) {+ ids.add(id);+ }+ }+ return ids;+ }++ private static Integer parsePort(String port) {+ if (port == null) {+ return null;+ }+ try {+ return Integer.valueOf(port);+ } catch (NumberFormatException e) {+ LOG.debugv("Ignoring a Cloud Map instance port that is not a number: {0}", port);+ return null;+ }+ }+private List<Map<String, Object>> registries(EcsServiceModel svc) {List<Map<String, Object>> registries = svc.getServiceRegistries();return registries != null ? registries : List.of();diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.javaindex 6de4d15219..0f99f1213f 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java@@ -77,6 +77,7 @@ import java.util.Map;import java.util.Optional;import java.util.Set;import java.util.UUID;+import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.CopyOnWriteArrayList;import java.util.concurrent.CountDownLatch;import java.util.concurrent.TimeUnit;@@ -93,6 +94,8 @@ public class EcsContainerManager {private static final Logger LOG = Logger.getLogger(EcsContainerManager.class);private static final String ATTACHMENT_DELETED = "DELETED";+ /** The label naming the Floci process that created a container, set by {@link #ownerLabels()}. */+ public static final String RUN_LABEL = "floci.ecs-run";/** EC2 error codes the task ENI path can raise, none of which RunTask declares. */private static final Set<String> EC2_NETWORK_LOOKUP_FAILURES =@@ -133,6 +136,11 @@ public class EcsContainerManager {private final EcsTaskRoleCredentials taskRoleCredentials;private final EcsCredentialsProxy credentialsProxy;private final EcsTaskLinkLocalAddresses linkLocalAddresses;+ // Stamped as RUN_LABEL on every container this process creates, so a sweep can tell its own apart.+ private final String runId = UUID.randomUUID().toString();+ // A sweep failure is retried before each task launch; each distinct failure is a WARN once.+ private volatile boolean leftoverListFailureReported;+ private final Set<String> leftoverRemovalFailuresReported = ConcurrentHashMap.newKeySet();@Injectpublic EcsContainerManager(ContainerBuilder containerBuilder,@@ -312,7 +320,8 @@ public class EcsContainerManager {.withHostDockerInternalOnLinux().withEmbeddedDns().withLabels(ContainerStorageHelper.resourceIdentityLabels(- "ecs", taskId, regionResolver.getAccountId(), region));+ "ecs", taskId, regionResolver.getAccountId(), region))+ .withLabels(ownerLabels());if (protectedNetwork != null) {specBuilder.withNetworkMode("container:" + protectedNetwork.namespace().helperId());specBuilder.withLabels(Map.of("floci.security-group-workload", "true"));@@ -991,6 +1000,30 @@ public class EcsContainerManager {.ifPresent(subnet -> task.setAvailabilityZone(subnet.getAvailabilityZone()));}+ /**+ * The VPC an awsvpc task's interface was created in, read from its subnet, or {@code null}+ * for a task without one or whose subnet is gone.+ */+ public String taskVpcId(EcsTask task, String region) {+ if (task.getNetworkInterfaceId() == null || ec2Service == null) {+ return null;+ }+ AwsVpcConfiguration awsvpc = task.getNetworkConfiguration() == null ? null+ : task.getNetworkConfiguration().getAwsvpcConfiguration();+ if (awsvpc == null || awsvpc.getSubnets() == null || awsvpc.getSubnets().isEmpty()) {+ return null;+ }+ try {+ return ec2Service.describeSubnets(region, List.of(awsvpc.getSubnets().getFirst()), Map.of()).stream()+ .findFirst()+ .map(subnet -> subnet.getVpcId())+ .orElse(null);+ } catch (AwsException e) {+ LOG.debugv("Could not look up the VPC of ECS task {0}: {1}", task.getTaskArn(), e.getMessage());+ return null;+ }+ }+/*** Restates an EC2 lookup failure as the error RunTask declares. A caller naming a subnet or a* security group that is not there asked for something the task cannot have, and@@ -1030,6 +1063,76 @@ public class EcsContainerManager {}}+ /**+ * Removes the ECS containers a previous run of this Floci left on the daemon: task containers+ * and the security-group helpers whose network namespace they share. Task state is memory-only,+ * so no container carrying this deployment's owner label and another run's {@link #RUN_LABEL}+ * belongs to a task this process knows. A graceful shutdown already stops them; this+ * covers a run that ended without one (SIGKILL, OOM, a stop timeout that expired mid-drain),+ * whose containers would otherwise keep serving beside the replacements the service scheduler+ * starts. Containers created before the owner label existed carry none and are left alone.+ * Leaving out the ones this process created is what makes a retry safe once tasks run. The run+ * label decides that rather than the creation time, which Docker reports to the second and from+ * a clock that can drift from this process's.+ *+ * Each distinct failure, a failed listing or a container that would not go, is logged as a+ * WARN the first time and at DEBUG on every retry that repeats it, so an unreachable Docker+ * daemon does not log on every launch attempt while a new failure is still reported.+ *+ * @return whether every leftover is gone: false when Docker could not list or remove one+ */+ public boolean removeLeftoverContainers() {+ return removeLeftoverContainers(runId);+ }++ /** The labels that tie a container to this deployment and to this process's run of it. */+ private Map<String, String> ownerLabels() {+ return Map.of(ContainerStorageHelper.OWNER_LABEL, ContainerStorageHelper.ownerIdentity(config),+ RUN_LABEL, runId);+ }++ /** {@link #removeLeftoverContainers()} keeping only the containers of run {@code currentRunId}. */+ public boolean removeLeftoverContainers(String currentRunId) {+ String owner = ContainerStorageHelper.ownerIdentity(config);+ List<com.github.dockerjava.api.model.Container> containers;+ try {+ // Docker's container summary, not the ECS model Container this class imports.+ containers = lifecycleManager.getDockerClient()+ .listContainersCmd()+ .withShowAll(true)+ .withLabelFilter(Map.of("io.floci.service", "ecs", ContainerStorageHelper.OWNER_LABEL, owner))+ .exec();+ } catch (Exception e) {+ LOG.logv(leftoverListFailureReported ? Logger.Level.DEBUG : Logger.Level.WARN,+ "Could not list the ECS containers a previous run left behind: {0}", e.getMessage());+ leftoverListFailureReported = true;+ return false;+ }+ leftoverListFailureReported = false;+ boolean allRemoved = true;+ for (com.github.dockerjava.api.model.Container container : containers) {+ if (container.getLabels() != null && currentRunId.equals(container.getLabels().get(RUN_LABEL))) {+ continue;+ }+ try {+ lifecycleManager.removeIfExistsStrict(container.getId());+ leftoverRemovalFailuresReported.remove(container.getId());+ LOG.infov("Removed ECS container {0} ({1}) left by a previous run", container.getId(),+ container.getNames() == null ? "" : String.join(",", container.getNames()));+ } catch (Exception e) {+ allRemoved = false;+ Logger.Level level = leftoverRemovalFailuresReported.add(container.getId())+ ? Logger.Level.WARN : Logger.Level.DEBUG;+ LOG.logv(level, "Could not remove ECS container {0} left by a previous run: {1}",+ container.getId(), e.getMessage());+ }+ }+ if (allRemoved) {+ leftoverRemovalFailuresReported.clear();+ }+ return allRemoved;+ }+/** The ENI {@link #attachTaskNetwork} allocated for this task, or null if it is already gone. */private NetworkInterface existingTaskEni(EcsTask task, String region) {if (task.getNetworkInterfaceId() == null) {@@ -1071,7 +1174,7 @@ public class EcsContainerManager {containerDetector.isRunningInContainer(),config.services().ecs().publishAwsvpcPortsToHost());namespace = firewallManager.createNamespace("ecs", taskId, regionResolver.getAccountId(),- region, config.services().ecs().dockerNetwork(), bindings);+ region, config.services().ecs().dockerNetwork(), bindings, Map.of(RUN_LABEL, runId));List<String> groupIds = eni.getGroups().stream().map(g -> g.getGroupId()).toList();List<SecurityGroup> groups = ec2Service.describeSecurityGroups(region, groupIds, List.of(), Map.of());if (groups.size() != groupIds.size()) {diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredInstances.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredInstances.javanew file mode 100644index 0000000000..210bb2173d--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredInstances.java@@ -0,0 +1,14 @@+package io.github.hectorvent.floci.services.ecs.model;++import io.quarkus.runtime.annotations.RegisterForReflection;++import java.util.List;++/**+ * The Cloud Map instance ECS registered for one task and the Cloud Map services it joined,+ * persisted so a restarted Floci, which knows no task, deregisters exactly these and never an+ * instance registered through the Cloud Map API.+ */+@RegisterForReflection+public record EcsRegisteredInstances(String region, String instanceId, List<String> cloudMapServiceIds) {+}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredTargets.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredTargets.javanew file mode 100644index 0000000000..fbd5807882--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsRegisteredTargets.java@@ -0,0 +1,25 @@+package io.github.hectorvent.floci.services.ecs.model;++import io.github.hectorvent.floci.services.elbv2.model.TargetDescription;+import io.quarkus.runtime.annotations.RegisterForReflection;++import java.util.List;++/**+ * The ELBv2 targets ECS registered for one task, persisted so a restarted Floci, which knows no+ * task, deregisters exactly these and never a target someone else registered in the same group.+ */+@RegisterForReflection+public record EcsRegisteredTargets(String region, List<Target> targets) {++ @RegisterForReflection+ public record Target(String targetGroupArn, String id, Integer port) {++ public TargetDescription toDescription() {+ TargetDescription td = new TargetDescription();+ td.setId(id);+ td.setPort(port);+ return td;+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTaskAddress.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTaskAddress.javanew file mode 100644index 0000000000..11b4a6f8d5--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTaskAddress.java@@ -0,0 +1,34 @@+package io.github.hectorvent.floci.services.ecs.model;++/**+ * An address a running task holds. A task's own address on a Docker network reaches only that+ * task, so {@code port} is {@code null} when any port there counts as the task's. A loopback+ * address is shared by every task Floci publishes on the host, so it is the task's only together+ * with one of the host ports its containers were published on. An ENI address is unique only+ * within its VPC, so {@code vpcId} names the VPC it belongs to; it is {@code null} for a Docker+ * or loopback address, which no VPC scopes.+ */+public record EcsTaskAddress(String ip, Integer port, String vpcId) {++ public EcsTaskAddress(String ip, Integer port) {+ this(ip, port, null);+ }++ public static EcsTaskAddress of(String ip, int hostPort) {+ return isLoopback(ip) ? new EcsTaskAddress(ip, hostPort) : new EcsTaskAddress(ip, null);+ }++ public static boolean isLoopback(String ip) {+ return ip.startsWith("127.") || "localhost".equals(ip) || "::1".equals(ip);+ }++ /** Whether an endpoint registered at {@code ip} and {@code port} reaches this task. */+ public boolean matches(String ip, Integer port) {+ return this.ip.equals(ip) && (this.port == null || this.port.equals(port));+ }++ /** Whether this address means this task in {@code vpc}, the VPC an endpoint is scoped to. */+ public boolean appliesTo(String vpc) {+ return vpcId == null || vpcId.equals(vpc);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLeftoverContainersDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLeftoverContainersDockerIntegrationTest.javanew file mode 100644index 0000000000..ed47595650--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLeftoverContainersDockerIntegrationTest.java@@ -0,0 +1,143 @@+package io.github.hectorvent.floci.services.ecs;++import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.command.CreateContainerResponse;+import com.github.dockerjava.api.exception.NotFoundException;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;+import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.LaunchType;+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;+import io.quarkus.test.junit.QuarkusTest;+import io.quarkus.test.junit.TestProfile;+import jakarta.inject.Inject;+import org.junit.jupiter.api.Assumptions;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.UUID;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertNotNull;+import static org.junit.jupiter.api.Assertions.assertTrue;++/**+ * A Floci that ends without a graceful shutdown leaves its task containers running, and the next+ * run knows no task, since task state is memory-only. Those containers must go at startup, or the+ * service scheduler starts a second task beside each one. The sweep recognises them by the owner+ * label a task container is created with, so a container of another Floci sharing the daemon stays,+ * and tells this run's containers from a previous run's by the run label rather than by time.+ */+@QuarkusTest+@TestProfile(EcsServiceDiscoveryDockerIntegrationTest.DockerEcsProfile.class)+class EcsLeftoverContainersDockerIntegrationTest {++ private static final String REGION = "us-east-1";+ private static final String BUSYBOX_IMAGE = "public.ecr.aws/docker/library/busybox:latest";++ @Inject+ EcsService ecsService;++ @Inject+ EcsContainerManager containerManager;++ @Inject+ EmulatorConfig config;++ @Inject+ DockerClient dockerClient;++ @BeforeEach+ void requireDocker() {+ Assumptions.assumeTrue(isDockerAvailable(),+ "Docker daemon must be available for the ECS leftover container test");+ }++ @Test+ void theStartupSweepRemovesTaskContainersAndKeepsAnotherFlocisContainers() {+ String clusterName = unique("leftover-cluster");+ ecsService.createCluster(clusterName, REGION);+ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage(BUSYBOX_IMAGE);+ app.setCommand(List.of("sleep", "120"));+ TaskDefinition taskDef = ecsService.registerTaskDefinition(unique("leftover-td"), List.of(app),+ NetworkMode.bridge, null, null, null, null, null, REGION);++ EcsTask task = ecsService.runTask(clusterName, taskDef.getTaskDefinitionArn(), 1, LaunchType.EC2, null, null,+ null, null, REGION).getFirst();+ String foreignId = null;+ String previousRunId = null;+ try {+ String taskContainerId = task.getContainers().getFirst().getDockerId();+ assertNotNull(taskContainerId, "the task must have started a container: " + task.getStoppedReason());+ assertEquals(ContainerStorageHelper.ownerIdentity(config), labels(taskContainerId)+ .get(ContainerStorageHelper.OWNER_LABEL), "a task container must carry its owner");++ foreignId = createEcsContainer(Map.of(ContainerStorageHelper.OWNER_LABEL, "another-floci/4566"));+ // Created a moment ago, as by a run killed within the second this one started.+ previousRunId = createEcsContainer(Map.of(ContainerStorageHelper.OWNER_LABEL,+ ContainerStorageHelper.ownerIdentity(config), EcsContainerManager.RUN_LABEL, "a-previous-run"));++ assertTrue(containerManager.removeLeftoverContainers(), "the sweep must succeed");+ assertTrue(exists(taskContainerId), "a task container this run started must stay");+ assertFalse(exists(previousRunId), "a previous run's container must be removed however recent");++ assertTrue(containerManager.removeLeftoverContainers("the-next-run"), "the sweep must succeed");+ assertFalse(exists(taskContainerId), "the leftover task container must be removed");+ assertTrue(exists(foreignId), "another Floci's container must stay");+ } finally {+ ecsService.stopTask(clusterName, task.getTaskArn(), "test teardown", REGION);+ for (String containerId : new String[] {foreignId, previousRunId}) {+ if (containerId != null && exists(containerId)) {+ dockerClient.removeContainerCmd(containerId).withForce(true).exec();+ }+ }+ }+ }++ /** An ECS container carrying {@code ownership}, as another Floci or another run creates. */+ private String createEcsContainer(Map<String, String> ownership) {+ Map<String, String> labels = new HashMap<>(ownership);+ labels.put("io.floci.service", "ecs");+ CreateContainerResponse created = dockerClient.createContainerCmd(BUSYBOX_IMAGE)+ .withName(unique("floci-other-ecs"))+ .withCmd("sleep", "120")+ .withLabels(labels)+ .exec();+ return created.getId();+ }++ private Map<String, String> labels(String containerId) {+ return dockerClient.inspectContainerCmd(containerId).exec().getConfig().getLabels();+ }++ private boolean exists(String containerId) {+ try {+ dockerClient.inspectContainerCmd(containerId).exec();+ return true;+ } catch (NotFoundException e) {+ return false;+ }+ }++ private boolean isDockerAvailable() {+ try {+ dockerClient.pingCmd().exec();+ return true;+ } catch (RuntimeException e) {+ return false;+ }+ }++ private static String unique(String prefix) {+ return prefix + "-" + UUID.randomUUID().toString().substring(0, 8);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrarTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrarTest.javaindex 258375ad3a..8660aac791 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrarTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsLoadBalancerRegistrarTest.java@@ -4,19 +4,27 @@ import io.github.hectorvent.floci.services.ecs.model.Container;import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;import io.github.hectorvent.floci.services.elbv2.ElbV2Service;+import io.github.hectorvent.floci.services.elbv2.model.TargetDescription;import io.github.hectorvent.floci.services.elbv2.model.TargetGroup;import io.github.hectorvent.floci.services.elbv2.model.TargetHealth;import io.quarkus.test.junit.QuarkusTest;+import io.quarkus.test.junit.mockito.InjectSpy;import jakarta.inject.Inject;import org.junit.jupiter.api.Test;import java.util.List;import java.util.Map;+import java.util.UUID;import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertThrows;import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.doAnswer;/*** Component test for {@link EcsLoadBalancerRegistrar}: drives the register/deregister@@ -31,7 +39,7 @@ class EcsLoadBalancerRegistrarTest {@InjectEcsLoadBalancerRegistrar registrar;- @Inject+ @InjectSpyElbV2Service elbV2Service;private String createTargetGroup(String name) {@@ -47,7 +55,7 @@ class EcsLoadBalancerRegistrarTest {container.setNetworkBindings(List.of(new NetworkBinding("0.0.0.0", containerPort, hostPort, "tcp")));EcsTask task = new EcsTask();- task.setTaskArn("arn:aws:ecs:" + REGION + ":000000000000:task/c/regtest");+ task.setTaskArn("arn:aws:ecs:" + REGION + ":000000000000:task/c/" + UUID.randomUUID());task.setGroup("regtest-svc");task.setContainers(List.of(container));return task;@@ -125,4 +133,153 @@ class EcsLoadBalancerRegistrarTest {assertTrue(elbV2Service.describeTargetHealth(REGION, tgArn, null).isEmpty(),"no network binding for the declared containerPort -> no target");}++ @Test+ void releaseRecordedTargetsKeepsATargetRegisteredByHand() {+ String tgArn = createTargetGroup("reg-tg-stale");+ registrar.registerTask(taskWithContainer("web", 8080, 34567), serviceWithLb(tgArn, "web", 8080), REGION);+ elbV2Service.registerTargets(REGION, tgArn, List.of(target("172.18.0.7", 8080)));++ registrar.releaseRecordedTargets();++ List<TargetHealth> health = elbV2Service.describeTargetHealth(REGION, tgArn, null);+ assertEquals(1, health.size(), "only the target registered by hand should survive");+ assertEquals("172.18.0.7", health.get(0).getTarget().getId());+ }++ @Test+ void deregisterTaskRemovesTheTargetItRecordedEvenAfterTheBindingChanged() {+ String tgArn = createTargetGroup("reg-tg-recorded");+ EcsTask task = taskWithContainer("web", 8080, 35000);+ EcsServiceModel svc = serviceWithLb(tgArn, "web", 8080);+ registrar.registerTask(task, svc, REGION);+ task.getContainers().getFirst().setNetworkBindings(List.of(+ new NetworkBinding("0.0.0.0", 8080, 36000, "tcp")));++ registrar.deregisterTask(task, svc, REGION);++ assertTrue(elbV2Service.describeTargetHealth(REGION, tgArn, null).isEmpty(),+ "the recorded target should be deregistered");+ }++ @Test+ void releaseRecordedTargetsToleratesADeletedTargetGroup() {+ String tgArn = createTargetGroup("reg-tg-gone");+ EcsTask task = taskWithContainer("web", 8080, 37000);+ registrar.registerTask(task, serviceWithLb(tgArn, "web", 8080), REGION);+ elbV2Service.deleteTargetGroup(REGION, tgArn);++ registrar.releaseRecordedTargets();+ }++ @Test+ void releaseRecordedTargetsReleasesATargetWhoseRegistrationWasCutShort() {+ String tgArn = createTargetGroup("reg-tg-killed");+ doAnswer(invocation -> {+ invocation.callRealMethod();+ throw new ProcessKilled();+ }).when(elbV2Service).registerTargets(eq(REGION), eq(tgArn), any());++ assertThrows(ProcessKilled.class, () -> registrar.registerTask(+ taskWithContainer("web", 8080, 38000), serviceWithLb(tgArn, "web", 8080), REGION));+ registrar.releaseRecordedTargets();++ assertTrue(elbV2Service.describeTargetHealth(REGION, tgArn, null).isEmpty(),+ "a target registered just before the process died should be released");+ }++ @Test+ void aTargetAlreadyAtTheTasksAddressIsAdoptedAndRemovedWhenTheTaskStops() {+ // Left by a task of a run that kept no record: the next task to take the address adopts it.+ String tgArn = createTargetGroup("reg-tg-preexisting");+ elbV2Service.registerTargets(REGION, tgArn, List.of(target("127.0.0.1", 39000)));+ EcsTask task = taskWithContainer("web", 8080, 39000);+ EcsServiceModel svc = serviceWithLb(tgArn, "web", 8080);++ registrar.registerTask(task, svc, REGION);+ assertEquals(1, elbV2Service.describeTargetHealth(REGION, tgArn, null).size());+ registrar.deregisterTask(task, svc, REGION);++ assertTrue(elbV2Service.describeTargetHealth(REGION, tgArn, null).isEmpty(),+ "the target at the stopped task's address should be deregistered");+ }++ @Test+ void evictUnrecordedTargetsRemovesAStaleTargetAtTheTasksAddressFromEveryGroup() {+ String alphaTg = createTargetGroup("reg-tg-evict-alpha");+ String betaTg = createTargetGroup("reg-tg-evict-beta");+ elbV2Service.registerTargets(REGION, alphaTg, List.of(target("172.19.0.4", 8080), target("172.19.0.3", 8080)));+ elbV2Service.registerTargets(REGION, betaTg, List.of(target("172.19.0.4", 8080)));+ EcsTask betaTask = taskWithContainer("web", 8080, 8080);++ registrar.evictUnrecordedTargets(betaTask, List.of(alphaTg, betaTg),+ List.of(new EcsTaskAddress("172.19.0.4", 8080)), REGION);++ List<TargetHealth> alpha = elbV2Service.describeTargetHealth(REGION, alphaTg, null);+ assertEquals(1, alpha.size(), "only the target at another address should stay in alpha's group");+ assertEquals("172.19.0.3", alpha.get(0).getTarget().getId());+ assertTrue(elbV2Service.describeTargetHealth(REGION, betaTg, null).isEmpty(),+ "a target at the task's address and port should be evicted");+ }++ @Test+ void evictUnrecordedTargetsKeepsATargetOnAPortTheTaskDoesNotServe() {+ String tgArn = createTargetGroup("reg-tg-evict-port");+ elbV2Service.registerTargets(REGION, tgArn, List.of(target("172.19.0.5", 8080), target("172.19.0.5", 9000)));++ registrar.evictUnrecordedTargets(taskWithContainer("web", 8080, 8080), List.of(tgArn),+ List.of(new EcsTaskAddress("172.19.0.5", 8080)), REGION);++ List<TargetHealth> health = elbV2Service.describeTargetHealth(REGION, tgArn, null);+ assertEquals(1, health.size(), "the target on a port the task does not serve should stay");+ assertEquals(9000, health.get(0).getTarget().getPort());+ }++ @Test+ void evictUnrecordedTargetsKeepsATargetALiveTaskRecorded() {+ String tgArn = createTargetGroup("reg-tg-evict-recorded");+ EcsTask owner = taskWithContainer("web", 8080, 40000);+ registrar.registerTask(owner, serviceWithLb(tgArn, "web", 8080), REGION);+ EcsTask other = taskWithContainer("web", 8080, 40000);++ registrar.evictUnrecordedTargets(other, List.of(tgArn),+ List.of(EcsTaskAddress.of("127.0.0.1", 40000)), REGION);++ assertEquals(1, elbV2Service.describeTargetHealth(REGION, tgArn, null).size(),+ "a target a live task recorded should stay");+ registrar.deregisterTask(owner, serviceWithLb(tgArn, "web", 8080), REGION);+ }++ @Test+ void evictUnrecordedTargetsMatchesALoopbackAddressOnlyByItsHostPort() {+ String tgArn = createTargetGroup("reg-tg-evict-loopback");+ elbV2Service.registerTargets(REGION, tgArn, List.of(target("127.0.0.1", 41000), target("127.0.0.1", 41001)));++ registrar.evictUnrecordedTargets(taskWithContainer("web", 8080, 41000), List.of(tgArn),+ List.of(EcsTaskAddress.of("127.0.0.1", 41000)), REGION);++ List<TargetHealth> health = elbV2Service.describeTargetHealth(REGION, tgArn, null);+ assertEquals(1, health.size(), "the loopback target on another host port should stay");+ assertEquals(41001, health.get(0).getTarget().getPort());+ }++ @Test+ void evictUnrecordedTargetsToleratesADeletedTargetGroup() {+ String tgArn = createTargetGroup("reg-tg-evict-gone");+ elbV2Service.deleteTargetGroup(REGION, tgArn);++ registrar.evictUnrecordedTargets(taskWithContainer("web", 8080, 42000), List.of(tgArn),+ List.of(new EcsTaskAddress("172.19.0.9", 8080)), REGION);+ }++ /** Stands in for the process dying between the registration and anything after it. */+ private static final class ProcessKilled extends Error {+ }++ private static TargetDescription target(String id, int port) {+ TargetDescription td = new TargetDescription();+ td.setId(id);+ td.setPort(port);+ return td;+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrarTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrarTest.javaindex 9d6d391802..0c2899ca59 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrarTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceDiscoveryRegistrarTest.java@@ -7,8 +7,10 @@ import io.github.hectorvent.floci.services.cloudmap.model.Service;import io.github.hectorvent.floci.services.ecs.model.Container;import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;import io.quarkus.test.junit.QuarkusTest;+import io.quarkus.test.junit.mockito.InjectSpy;import jakarta.inject.Inject;import org.junit.jupiter.api.Test;@@ -17,7 +19,11 @@ import java.util.Map;import java.util.UUID;import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertThrows;import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.doAnswer;/*** Component test for {@link EcsServiceDiscoveryRegistrar}: drives register/deregister directly@@ -33,7 +39,7 @@ class EcsServiceDiscoveryRegistrarTest {@InjectEcsServiceDiscoveryRegistrar registrar;- @Inject+ @InjectSpyCloudMapService cloudMapService;@Test@@ -139,6 +145,105 @@ class EcsServiceDiscoveryRegistrarTest {assertTrue(!registrar.hasRegistries(svc));}+ @Test+ void releaseRecordedInstancesKeepsInstancesRegisteredByHand() {+ String namespace = uniqueName("svcdisc") + ".internal";+ Service cloudMapSvc = createDnsService(namespace, "web");+ EcsServiceModel web = serviceWithRegistry(cloudMapSvc.getArn(), "web", 8080);+ web.setServiceName("web-svc");+ registrar.registerTask(taskInCluster("172.31.0.11", "payments"), web, REGION);+ registrar.registerTask(taskInCluster("172.31.0.12", "payments"), web, REGION);+ cloudMapService.registerInstance(cloudMapSvc.getId(), "registered-by-hand", null,+ Map.of("AWS_INSTANCE_IPV4", "172.31.0.15",+ "ECS_SERVICE_NAME", "web-svc", "ECS_CLUSTER_NAME", "payments"), REGION);++ registrar.releaseRecordedInstances();++ List<String> remaining = cloudMapService.listInstances(cloudMapSvc.getId()).stream()+ .map(Instance::getInstanceId)+ .toList();+ assertEquals(List.of("registered-by-hand"), remaining);+ }++ @Test+ void releaseRecordedInstancesReleasesAnInstanceWhoseRegistrationWasCutShort() {+ String namespace = uniqueName("svcdisc") + ".internal";+ Service cloudMapSvc = createDnsService(namespace, "killed");+ doAnswer(invocation -> {+ invocation.callRealMethod();+ throw new ProcessKilled();+ }).when(cloudMapService).registerInstance(eq(cloudMapSvc.getId()), any(), any(), any(), any());++ assertThrows(ProcessKilled.class, () -> registrar.registerTask(task("172.31.0.13", "killed", 8080, 8080),+ serviceWithRegistry(cloudMapSvc.getArn(), "killed", 8080), REGION));+ registrar.releaseRecordedInstances();++ assertTrue(cloudMapService.listInstances(cloudMapSvc.getId()).isEmpty(),+ "an instance registered just before the process died should be released");+ }++ @Test+ void evictUnrecordedInstancesRemovesAStaleInstanceAtTheTasksAddress() {+ String namespace = uniqueName("svcdisc") + ".internal";+ Service auth = createDnsService(namespace, "auth");+ Service document = cloudMapService.createService("document", auth.getNamespaceId(),+ null, null, null, null, null, null, Map.of(), REGION);+ // Left by an auth task of a run that kept no record; the address now belongs to a document task.+ cloudMapService.registerInstance(auth.getId(), "bdf778f0", null,+ Map.of("AWS_INSTANCE_IPV4", "172.31.0.21"), REGION);+ cloudMapService.registerInstance(auth.getId(), "elsewhere", null,+ Map.of("AWS_INSTANCE_IPV4", "172.31.0.17"), REGION);+ EcsTask documentTask = task("172.31.0.21", "document", 8080, 8080);+ registrar.registerTask(documentTask, serviceWithRegistry(document.getArn(), "document", 8080), REGION);++ registrar.evictUnrecordedInstances(documentTask, List.of(auth.getId(), document.getId()),+ List.of(new EcsTaskAddress("172.31.0.21", null)), REGION);++ assertEquals(List.of("elsewhere"), cloudMapService.listInstances(auth.getId()).stream()+ .map(Instance::getInstanceId).toList());+ assertEquals(1, cloudMapService.listInstances(document.getId()).size(),+ "the task's own recorded instance should stay");+ assertEquals(List.of("172.31.0.17"), cloudMapService.resolveDnsName("auth." + namespace));+ }++ @Test+ void evictUnrecordedInstancesMatchesALoopbackAddressOnlyByItsPort() {+ String namespace = uniqueName("svcdisc") + ".internal";+ Service cloudMapSvc = createDnsService(namespace, "api");+ cloudMapService.registerInstance(cloudMapSvc.getId(), "stale", null,+ Map.of("AWS_INSTANCE_IPV4", "127.0.0.1", "AWS_INSTANCE_PORT", "32768"), REGION);+ cloudMapService.registerInstance(cloudMapSvc.getId(), "other-port", null,+ Map.of("AWS_INSTANCE_IPV4", "127.0.0.1", "AWS_INSTANCE_PORT", "32769"), REGION);++ registrar.evictUnrecordedInstances(task(null, "api", 8080, 32768), List.of(cloudMapSvc.getId()),+ List.of(EcsTaskAddress.of("127.0.0.1", 32768)), REGION);++ assertEquals(List.of("other-port"), cloudMapService.listInstances(cloudMapSvc.getId()).stream()+ .map(Instance::getInstanceId).toList());+ }++ @Test+ void evictUnrecordedInstancesMatchesAnEniAddressOnlyInItsOwnVpc() {+ String namespace = uniqueName("svcdisc") + ".internal";+ Service cloudMapSvc = createDnsService(namespace, "api");+ cloudMapService.registerInstance(cloudMapSvc.getId(), "by-hand", null,+ Map.of("AWS_INSTANCE_IPV4", "10.0.1.5"), REGION);++ registrar.evictUnrecordedInstances(task("10.0.1.5", "api", 8080, 8080), List.of(cloudMapSvc.getId()),+ List.of(new EcsTaskAddress("10.0.1.5", null, "vpc-elsewhere")), REGION);+ assertEquals(List.of("by-hand"), cloudMapService.listInstances(cloudMapSvc.getId()).stream()+ .map(Instance::getInstanceId).toList(), "an instance in another VPC's namespace should stay");++ registrar.evictUnrecordedInstances(task("10.0.1.5", "api", 8080, 8080), List.of(cloudMapSvc.getId()),+ List.of(new EcsTaskAddress("10.0.1.5", null, "vpc-svcdisc")), REGION);+ assertTrue(cloudMapService.listInstances(cloudMapSvc.getId()).isEmpty(),+ "an instance at the task's ENI address in its own VPC should be evicted");+ }++ /** Stands in for the process dying between the registration and anything after it. */+ private static final class ProcessKilled extends Error {+ }+private Service createDnsService(String namespaceName, String serviceName) {Operation operation = cloudMapService.createPrivateDnsNamespace(namespaceName, "vpc-svcdisc", null, null, Map.of(), REGION);@@ -168,6 +273,12 @@ class EcsServiceDiscoveryRegistrarTest {return task;}+ private EcsTask taskInCluster(String privateIpAddress, String clusterName) {+ EcsTask task = task(privateIpAddress, "web", 8080, 8080);+ task.setClusterArn("arn:aws:ecs:" + REGION + ":000000000000:cluster/" + clusterName);+ return task;+ }+private static String uniqueName(String prefix) {return prefix + UUID.randomUUID().toString().substring(0, 8);}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.javanew file mode 100644index 0000000000..cde3674653--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsServiceRestartLeftoversTest.java@@ -0,0 +1,248 @@+package io.github.hectorvent.floci.services.ecs;++import com.fasterxml.jackson.core.type.TypeReference;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.storage.AccountAwareStorageBackend;+import io.github.hectorvent.floci.core.storage.StorageBackend;+import io.github.hectorvent.floci.core.storage.StorageFactory;+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;+import io.github.hectorvent.floci.services.ecs.container.EcsTaskHandle;+import io.github.hectorvent.floci.services.ecs.exec.EcsExecSessionRegistry;+import io.github.hectorvent.floci.services.ecs.model.Container;+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;+import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;+import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.EcsTaskAddress;+import io.github.hectorvent.floci.services.ecs.model.LaunchType;+import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;+import org.junit.jupiter.api.Test;++import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Set;++import static org.hamcrest.MatcherAssert.assertThat;+import static org.hamcrest.Matchers.containsString;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.times;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++/**+ * A restarted Floci knows no task, since task state is memory-only, so what the previous run's+ * tasks left behind has to go before the scheduler starts their replacements: containers a run+ * without a graceful shutdown left serving, and the load balancer targets and Cloud Map instances+ * that even a graceful one leaves registered.+ */+class EcsServiceRestartLeftoversTest {++ private static final String REGION = "us-east-1";+ private static final String TARGET_GROUP_ARN =+ "arn:aws:elasticloadbalancing:us-east-1:000000000000:targetgroup/web/0123456789abcdef";++ @Test+ void releasesTheRegistrationsThePreviousRunRecorded() {+ EcsLoadBalancerRegistrar lbRegistrar = mock(EcsLoadBalancerRegistrar.class);+ EcsServiceDiscoveryRegistrar discoveryRegistrar = mock(EcsServiceDiscoveryRegistrar.class);++ service(new SharedStorageFactory(), true, mock(EcsContainerManager.class),+ lbRegistrar, discoveryRegistrar).releasePreviousRunLeftovers();++ verify(lbRegistrar).releaseRecordedTargets();+ verify(discoveryRegistrar).releaseRecordedInstances();+ }++ @Test+ void theSchedulerWaitsUntilThePreviousRunsContainersAreRemoved() {+ SharedStorageFactory storage = new SharedStorageFactory();+ persistServiceWithLoadBalancer(storage);+ EcsContainerManager containerManager = mock(EcsContainerManager.class);+ when(containerManager.removeLeftoverContainers()).thenReturn(false, false, true);+ EcsService restarted = service(storage, false, containerManager,+ mock(EcsLoadBalancerRegistrar.class), null);++ restarted.releasePreviousRunLeftovers();+ restarted.reconcile();+ verify(containerManager, never()).startTask(any(), any(), any(), anyString());++ restarted.reconcile();+ restarted.reconcile();+ verify(containerManager, times(3)).removeLeftoverContainers();+ }++ @Test+ void theSweepIsNotRetriedWhileNoServiceNeedsATask() {+ EcsContainerManager containerManager = mock(EcsContainerManager.class);+ when(containerManager.removeLeftoverContainers()).thenReturn(false);+ EcsService restarted = service(new SharedStorageFactory(), false, containerManager,+ mock(EcsLoadBalancerRegistrar.class), null);++ restarted.releasePreviousRunLeftovers();+ restarted.reconcile();+ restarted.reconcile();++ verify(containerManager).removeLeftoverContainers();+ }++ @Test+ void runTaskRetriesTheSweepAndStartsNothingBesideAPreviousRunsContainers() {+ EcsContainerManager containerManager = mock(EcsContainerManager.class);+ when(containerManager.removeLeftoverContainers()).thenReturn(false, false, true);+ EcsService restarted = service(new SharedStorageFactory(), false, containerManager,+ mock(EcsLoadBalancerRegistrar.class), null);+ restarted.createCluster("app-cluster", Map.of(), REGION);+ ContainerDefinition container = new ContainerDefinition();+ container.setName("web");+ container.setImage("nginx:alpine");+ restarted.registerTaskDefinition("web", List.of(container), NetworkMode.bridge, null, null,+ null, null, List.of(), REGION);++ restarted.releasePreviousRunLeftovers();+ EcsTask blocked = restarted.runTask("app-cluster", "web", 1, LaunchType.EC2, null, null,+ null, null, REGION).getFirst();++ assertEquals("STOPPED", blocked.getLastStatus());+ assertThat(blocked.getStoppedReason(), containsString("previous run"));+ verify(containerManager, never()).startTask(any(), any(), any(), anyString());++ restarted.runTask("app-cluster", "web", 1, LaunchType.EC2, null, null, null, null, REGION);++ verify(containerManager, times(3)).removeLeftoverContainers();+ verify(containerManager).startTask(any(), any(), any(), anyString());+ }++ @Test+ void dockerModeRemovesTheContainersAPreviousRunLeft() {+ EcsContainerManager containerManager = mock(EcsContainerManager.class);++ service(new SharedStorageFactory(), false, containerManager,+ mock(EcsLoadBalancerRegistrar.class), null).releasePreviousRunLeftovers();++ verify(containerManager).removeLeftoverContainers();+ }++ @Test+ void mockModeHasNoContainersToRemove() {+ EcsContainerManager containerManager = mock(EcsContainerManager.class);++ service(new SharedStorageFactory(), true, containerManager,+ mock(EcsLoadBalancerRegistrar.class), null).releasePreviousRunLeftovers();++ verify(containerManager, never()).removeLeftoverContainers();+ }++ @Test+ void startupReleasesLeftoversBeforeTheSchedulerRuns() {+ EcsContainerManager containerManager = mock(EcsContainerManager.class);+ EcsService service = service(new SharedStorageFactory(), false, containerManager,+ mock(EcsLoadBalancerRegistrar.class), null);++ service.init();+ try {+ verify(containerManager).removeLeftoverContainers();+ verify(containerManager, never()).startTask(any(), any(), any(), anyString());+ } finally {+ service.stopManagedContainers();+ }+ }++ @Test+ void aStartingTaskEvictsTheUnrecordedRegistrationsAtItsAddress() {+ EcsContainerManager containerManager = mock(EcsContainerManager.class);+ when(containerManager.removeLeftoverContainers()).thenReturn(true);+ when(containerManager.startTask(any(), any(), any(), anyString())).thenAnswer(invocation -> {+ EcsTask task = invocation.getArgument(0);+ Container container = new Container();+ container.setName("web");+ container.setNetworkBindings(List.of(new NetworkBinding("0.0.0.0", 80, 80, "tcp")));+ task.setContainers(List.of(container));+ task.setPrivateIpAddress("10.0.1.5");+ return new EcsTaskHandle(task.getTaskArn(), Map.of("web", "docker-id"), Map.of());+ });+ when(containerManager.resolveContainerHost(any())).thenReturn("172.19.0.4");+ when(containerManager.taskVpcId(any(), anyString())).thenReturn("vpc-app");+ EcsLoadBalancerRegistrar lbRegistrar = mock(EcsLoadBalancerRegistrar.class);+ EcsServiceDiscoveryRegistrar discoveryRegistrar = mock(EcsServiceDiscoveryRegistrar.class);+ when(discoveryRegistrar.cloudMapServiceIds(any())).thenReturn(List.of("srv-alpha"));+ EcsService ecs = service(new SharedStorageFactory(), false, containerManager, lbRegistrar,+ discoveryRegistrar);+ ecs.releasePreviousRunLeftovers();+ ecs.createCluster("app-cluster", Map.of(), REGION);+ ContainerDefinition definition = new ContainerDefinition();+ definition.setName("web");+ definition.setImage("nginx:alpine");+ ecs.registerTaskDefinition("web", List.of(definition), NetworkMode.bridge, null, null,+ null, null, List.of(), REGION);+ EcsLoadBalancer lb = new EcsLoadBalancer();+ lb.setTargetGroupArn(TARGET_GROUP_ARN);+ lb.setContainerName("web");+ lb.setContainerPort(80);+ ecs.createService("app-cluster", "alpha", "web", 0, LaunchType.EC2, List.of(lb), null, REGION);++ // A task of no service at all: any container can take the address a stale entry points at.+ EcsTask task = ecs.runTask("app-cluster", "web", 1, LaunchType.EC2, null, null, null, null, REGION)+ .getFirst();++ assertEquals("RUNNING", task.getLastStatus());+ // A target only on the ports the container serves, and never at the VPC-scoped ENI address.+ verify(lbRegistrar).evictUnrecordedTargets(task, Set.of(TARGET_GROUP_ARN),+ Set.of(new EcsTaskAddress("172.19.0.4", 80)), REGION);+ verify(discoveryRegistrar).evictUnrecordedInstances(task, Set.of("srv-alpha"),+ Set.of(new EcsTaskAddress("10.0.1.5", null, "vpc-app"), new EcsTaskAddress("172.19.0.4", null)),+ REGION);+ }++ private static void persistServiceWithLoadBalancer(StorageFactory storage) {+ EcsService first = service(storage, true, mock(EcsContainerManager.class),+ mock(EcsLoadBalancerRegistrar.class), null);+ first.createCluster("app-cluster", Map.of(), REGION);+ ContainerDefinition container = new ContainerDefinition();+ container.setName("web");+ container.setImage("nginx:alpine");+ first.registerTaskDefinition("web", List.of(container), NetworkMode.bridge, null, null,+ null, null, List.of(), REGION);+ EcsLoadBalancer lb = new EcsLoadBalancer();+ lb.setTargetGroupArn(TARGET_GROUP_ARN);+ lb.setContainerName("web");+ lb.setContainerPort(80);+ first.createService("app-cluster", "web-svc", "web", 1, LaunchType.EC2, List.of(lb), null, REGION);+ }++ private static EcsService service(StorageFactory storage, boolean mockMode,+ EcsContainerManager containerManager,+ EcsLoadBalancerRegistrar lbRegistrar,+ EcsServiceDiscoveryRegistrar discoveryRegistrar) {+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.services().ecs().mock()).thenReturn(mockMode);+ when(config.effectiveBaseUrl()).thenReturn("http://localhost:4566");+ EcsService service = new EcsService(new RegionResolver(REGION, "000000000000"), containerManager,+ config, lbRegistrar, storage, null, new EcsExecSessionRegistry(), discoveryRegistrar);+ service.initializeStorage();+ return service;+ }++ private static final class SharedStorageFactory extends StorageFactory {+ private final Map<String, StorageBackend<String, ?>> stores = new HashMap<>();++ private SharedStorageFactory() {+ super(null, null);+ }++ @Override+ @SuppressWarnings("unchecked")+ public <V> AccountAwareStorageBackend<V> create(String serviceName,+ String fileName,+ TypeReference<Map<String, V>> typeReference) {+ return (AccountAwareStorageBackend<V>) stores.computeIfAbsent(fileName,+ ignored -> AccountAwareStorageBackend.inMemory("000000000000"));+ }+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.javaindex 024150fd83..0085ce351e 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java@@ -98,7 +98,7 @@ class EcsContainerManagerSecurityGroupTest {void awsvpcTaskWithFirewallJoinsHelperNamespaceAndSkipsHostPorts() {when(containerDetector.isRunningInContainer()).thenReturn(false);when(firewallManager.enabled()).thenReturn(true);- when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any()))+ when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any(), any())).thenReturn(new SecurityGroupFirewallManager.Namespace("helper-id", "10.0.0.5"));NetworkInterface eni = new NetworkInterface();@@ -149,7 +149,7 @@ class EcsContainerManagerSecurityGroupTest {void stopTaskUnregistersEniAndReleasingTheTaskNetworkDeletesIt() {when(containerDetector.isRunningInContainer()).thenReturn(false);when(firewallManager.enabled()).thenReturn(true);- when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any()))+ when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any(), any())).thenReturn(new SecurityGroupFirewallManager.Namespace("helper-id", "10.0.0.5"));NetworkInterface eni = new NetworkInterface();@@ -185,7 +185,7 @@ class EcsContainerManagerSecurityGroupTest {void firelensRouterAndAppBothJoinTheHelperNamespace() {when(containerDetector.isRunningInContainer()).thenReturn(false);when(firewallManager.enabled()).thenReturn(true);- when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any()))+ when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any(), any())).thenReturn(new SecurityGroupFirewallManager.Namespace("helper-id", "10.0.0.5"));NetworkInterface eni = new NetworkInterface();@@ -233,7 +233,7 @@ class EcsContainerManagerSecurityGroupTest {verify(builder, times(2)).withNetworkMode("container:helper-id");verify(builder, times(2)).withLabels(Map.of("floci.security-group-workload", "true"));- verify(firewallManager).createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any());+ verify(firewallManager).createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any(), any());verify(lifecycleManager).create(any());verify(lifecycleManager).startCreated(eq("router-id"), any());verify(lifecycleManager).createAndStart(any());