Light, fluffy, and always free - The AWS Local Emulator alternative
feat(ecs): serve the task metadata endpoint v4 (#4120)
Abhigya7cd441231f54
diff --git a/docs/services/ecs.md b/docs/services/ecs.mdindex 23ef13b771..b15041d892 100644--- a/docs/services/ecs.md+++ b/docs/services/ecs.md@@ -166,7 +166,7 @@ object ARN with `Invalid arn syntax`. A Fargate task can still take its config fdocuments, by giving the aws-for-fluent-bit init process its `aws_fluent_bit_init_s3_*`environment variables. ECS never inspects those and Floci passes them through, so thatregistration is accepted here too; the init process reads the task metadata endpoint before-downloading.+downloading, and Floci serves one (see [Task metadata endpoint](#task-metadata-endpoint)).Floci does not validate a task definition's `compatibilities` /`requiresCompatibilities` against `RunTask` `launchType`; a Fargate-compatibledefinition can still be run with `launchType=EC2` (and the reverse).@@ -297,6 +297,54 @@ launch type, which is what AWS documents for the `Service` shape.With no launch type, no strategy and no cluster default, a task keeps Floci's `FARGATE` default: alocal cluster has no container instances, so an EC2 default would have nowhere to place it.+#### Task metadata endpoint++Floci serves the [task metadata endpoint version+4](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4-fargate.html)+and injects `ECS_CONTAINER_METADATA_URI_V4` into every container it launches. AWS serves it on the+link-local address `169.254.170.2`, which a local container cannot be given, so Floci serves the+same paths on its own port; applications and the AWS SDKs read the environment variable rather than+the address, so they work unchanged.++| Path | Returns |+|---|---|+| `/v4/{id}` | The container's metadata |+| `/v4/{id}/task` | The task's metadata, including every container |+| `/v4/{id}/taskWithTags` | The task's metadata with its tags and its container instance's (EC2 only) |+| `/v4/{id}/stats` | The container's Docker stats |+| `/v4/{id}/task/stats` | The Docker stats of every container in the task, keyed by Docker id |++The `{id}` is minted per container at launch, as the ECS agent mints it. The task document carries+`Cluster`, `TaskARN`, `Family`, `Revision`, the statuses, `Limits` (CPU in vCPUs), the pull+timestamps, `AvailabilityZone`, `LaunchType`, `ServiceName` for a service's task, `VPCID` for an+EC2 task, and, for Fargate, `ClockDrift` and `EphemeralStorageMetrics`. Floci has no clock drift to+report and does not meter the disk, so those two report a synchronized clock and zero usage.++An `awsvpc` container's `Networks` object describes the task ENI and the subnet it sits in:+`NetworkMode`, `IPv4Addresses`, `AttachmentIndex`, `MACAddress`, `PrivateDNSName`,+`IPv4SubnetCIDRBlock`, `SubnetGatewayIpv4Address`, `DomainNameServers` and `DomainNameSearchList`.+A local ENI has no DHCP option set behind it, so the gateway is derived as the first address of the+subnet's CIDR, the resolver as the third address of the VPC's, and the search domain from the+region, the way a real VPC assigns them.++The stats paths read the Docker daemon when the request arrives, so they return the same+[ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) document AWS+returns, plus the `network_rate_stats` the ECS agent adds. Docker reports cumulative counters only,+so the rates are taken across two consecutive samples: as on Fargate, that means a container has to+have run for about a second before its stats are there. A container Floci has no running Docker+container for reports an empty document instead, and one sampled only once reports its stats+without `network_rate_stats`: the path exists for every container in the task, whether or not the+daemon can measure it.++`/v4/{id}/task/stats` samples the task's containers in one pass, with every stream open at once,+rather than one after another. A sidecar polls that path for network metrics, so the response costs+about the single collection tick one container costs however many containers the task has.++`taskWithTags` is the container agent's path, so it answers for an EC2 task and 404s for a Fargate+one, as on AWS. A container's `CreatedAt`, `StartedAt` and `FinishedAt` are Docker's own for that+container, read when it starts and again when it stops; a task that never reached a daemon, which+means `mock: true` or one restored from storage, reports the task's timestamps instead.+### Services| Operation | Description |diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.javaindex 925821e029..3fa51b06fa 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java@@ -75,6 +75,7 @@ import java.util.List;import java.util.Locale;import java.util.Map;import java.util.Objects;+import java.util.Optional;import java.util.stream.Collectors;import java.util.Set;import java.util.Comparator;@@ -1657,6 +1658,7 @@ public class EcsService implements ContainerTeardown, ResourceProvider, ResettabderegisterTaskFromLoadBalancers(task, region);Map<String, Integer> exitCodes = Map.of();+ Map<String, Instant> finishedAt = Map.of();if (dockerMode) {EcsTaskHandle handle = taskHandles.remove(task.getTaskArn());try {@@ -1664,6 +1666,9 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab} finally {retainUnresolvedLogHandle(task.getTaskArn(), handle);}+ if (handle != null) {+ finishedAt = handle.getFinishedAt();+ }}task.setLastStatus(TaskStatus.STOPPED.name());@@ -1673,12 +1678,17 @@ public class EcsService implements ContainerTeardown, ResourceProvider, ResettabcontainerManager.releaseTaskNetwork(task, region);if (task.getContainers() != null) {final Map<String, Integer> codes = exitCodes;+ final Map<String, Instant> finishTimes = finishedAt;task.getContainers().forEach(c -> {c.setLastStatus("STOPPED");Integer code = codes.get(c.getName());if (code != null) {c.setExitCode(code);}+ Instant finished = finishTimes.get(c.getName());+ if (finished != null) {+ c.setFinishedAt(finished);+ }});}@@ -1736,6 +1746,42 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab}}+ /** A container reached through the task metadata endpoint, with the task it belongs to. */+ public record MetadataTarget(EcsTask task, Container container, TaskDefinition taskDefinition) {}++ /**+ * Resolves the {@code ECS_CONTAINER_METADATA_URI_V4} id a container was given at launch. The id+ * is minted per container and never reused, so it identifies both the container and its task.+ */+ public Optional<MetadataTarget> findByMetadataId(String metadataId) {+ if (metadataId == null || metadataId.isBlank()) {+ return Optional.empty();+ }+ for (EcsTask task : tasks.values()) {+ if (task.getContainers() == null) {+ continue;+ }+ for (Container container : task.getContainers()) {+ if (metadataId.equals(container.getMetadataId())) {+ return Optional.of(new MetadataTarget(task, container,+ taskDefinitionOf(task)));+ }+ }+ }+ return Optional.empty();+ }++ /** The task's definition, or null when it has since been deleted. */+ private TaskDefinition taskDefinitionOf(EcsTask task) {+ try {+ return resolveTaskDefinitionOrThrow(task.getTaskDefinitionArn(), taskRegion(task));+ } catch (AwsException e) {+ LOG.debugv("Task {0} references a task definition that is gone: {1}",+ task.getTaskArn(), e.getMessage());+ return null;+ }+ }+public List<EcsTask> describeTasks(String clusterRef, List<String> taskRefs, String region) {return describeTasksDetailed(clusterRef, taskRefs, region).tasks();}@@ -3772,14 +3818,20 @@ public class EcsService implements ContainerTeardown, ResourceProvider, ResettabupdateHealth(task, handle);- // Inspect every container; abort if any are still running.+ // Inspect every container; abort if any are still running. A container that exited on its+ // own has a finish time of its own, read here while the daemon still remembers it.Map<String, Integer> exitCodes = new LinkedHashMap<>();+ Map<String, Instant> finishedAt = new LinkedHashMap<>();for (Map.Entry<String, String> entry : handle.getContainerIds().entrySet()) {Integer code = containerManager.getExitCodeIfStopped(entry.getValue());if (code == null) {return;}exitCodes.put(entry.getKey(), code);+ Instant finished = containerManager.getFinishedAtIfStopped(entry.getValue());+ if (finished != null) {+ finishedAt.put(entry.getKey(), finished);+ }}// All containers have exited. Atomically claim the handle to avoid@@ -3799,6 +3851,10 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettabif (code != null) {c.setExitCode(code);}+ Instant finished = finishedAt.get(c.getName());+ if (finished != null) {+ c.setFinishedAt(finished);+ }});}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataController.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataController.javanew file mode 100644index 0000000000..d9a0f5e002--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataController.java@@ -0,0 +1,548 @@+package io.github.hectorvent.floci.services.ecs;++import io.github.hectorvent.floci.core.common.AwsArnUtils;+import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.services.ec2.Ec2Service;+import io.github.hectorvent.floci.services.ec2.model.Subnet;+import io.github.hectorvent.floci.services.ec2.model.Vpc;+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;+import io.github.hectorvent.floci.services.ecs.model.AwsVpcConfiguration;+import io.github.hectorvent.floci.services.ecs.model.Container;+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;+import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.LaunchType;+import io.github.hectorvent.floci.services.ecs.model.LogConfiguration;+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;+import com.fasterxml.jackson.annotation.JsonInclude;+import com.fasterxml.jackson.databind.ObjectMapper;+import com.fasterxml.jackson.databind.json.JsonMapper;+import com.fasterxml.jackson.databind.node.ArrayNode;+import com.fasterxml.jackson.databind.node.ObjectNode;+import jakarta.enterprise.context.ApplicationScoped;+import jakarta.inject.Inject;+import jakarta.ws.rs.GET;+import jakarta.ws.rs.Path;+import jakarta.ws.rs.PathParam;+import jakarta.ws.rs.Produces;+import jakarta.ws.rs.core.MediaType;+import jakarta.ws.rs.core.Response;++import org.jboss.logging.Logger;++import java.time.Instant;+import java.time.format.DateTimeFormatter;+import java.util.List;+import java.util.Map;+import java.util.Objects;+import java.util.Optional;++/**+ * The ECS task metadata endpoint, version 4.+ *+ * <p>On AWS this is served at {@code http://169.254.170.2/v4/<id>} and every container is told+ * where to find it through {@code ECS_CONTAINER_METADATA_URI_V4}. A local task cannot be given a+ * link-local address of its own, so Floci serves the same paths on its own port and injects the+ * matching URI into each container it launches; an application, the AWS SDKs' ECS credential and+ * metadata clients, and the aws-for-fluent-bit init process all read the variable rather than the+ * address, so they work unchanged.+ *+ * <p>The two {@code /stats} paths return the Docker stats AWS documents them as returning: one+ * sample per container, taken from the daemon when the request arrives. A container Floci has no+ * running Docker container for reports an empty document rather than an error, because the path+ * itself still exists for it.+ */+@Path("/v4")+@ApplicationScoped+public class EcsTaskMetadataController {++ private static final Logger LOG = Logger.getLogger(EcsTaskMetadataController.class);++ private static final DateTimeFormatter TIMESTAMP = DateTimeFormatter.ISO_INSTANT;+ private static final String SERVICE_GROUP_PREFIX = "service:";+ /** The search domain an instance gets, which us-east-1 spells differently from every other region. */+ private static final String LEGACY_SEARCH_DOMAIN_REGION = "us-east-1";++ /**+ * Docker's stats document, written back out the way the daemon sent it. docker-java's model+ * carries a field for every member of every API version, so the members this daemon left out+ * are dropped rather than written as nulls.+ */+ private static final ObjectMapper STATS_MAPPER = JsonMapper.builder()+ .serializationInclusion(JsonInclude.Include.NON_NULL)+ .build();++ private final EcsService service;+ private final EcsContainerManager containerManager;+ private final Ec2Service ec2Service;+ private final ObjectMapper objectMapper;++ @Inject+ public EcsTaskMetadataController(EcsService service, EcsContainerManager containerManager,+ Ec2Service ec2Service, ObjectMapper objectMapper) {+ this.service = service;+ this.containerManager = containerManager;+ this.ec2Service = ec2Service;+ this.objectMapper = objectMapper;+ }++ @GET+ @Path("/{id}")+ @Produces(MediaType.APPLICATION_JSON)+ public Response container(@PathParam("id") String id) {+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);+ if (target.isEmpty()) {+ return notFound();+ }+ EcsService.MetadataTarget found = target.get();+ return Response.ok(containerNode(found.task(), found.container(), found.taskDefinition())).build();+ }++ @GET+ @Path("/{id}/task")+ @Produces(MediaType.APPLICATION_JSON)+ public Response task(@PathParam("id") String id) {+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);+ if (target.isEmpty()) {+ return notFound();+ }+ EcsService.MetadataTarget found = target.get();+ return Response.ok(taskNode(found.task(), found.taskDefinition())).build();+ }++ /**+ * The task document with the task's own tags and, for a task placed on a container instance,+ * that instance's. The container agent serves this path, so it exists for the EC2 launch type+ * only; a Fargate task gets the 404 AWS gives it.+ */+ @GET+ @Path("/{id}/taskWithTags")+ @Produces(MediaType.APPLICATION_JSON)+ public Response taskWithTags(@PathParam("id") String id) {+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);+ if (target.isEmpty() || target.get().task().getLaunchType() != LaunchType.EC2) {+ return notFound();+ }+ EcsService.MetadataTarget found = target.get();+ ObjectNode node = taskNode(found.task(), found.taskDefinition());+ ArrayNode errors = objectMapper.createArrayNode();+ putTags(node, "TaskTags", found.task().getTaskArn(), errors);+ putTags(node, "ContainerInstanceTags", found.task().getContainerInstanceArn(), errors);+ if (!errors.isEmpty()) {+ node.set("Errors", errors);+ }+ return Response.ok(node).build();+ }++ /**+ * One tag collection, omitted when the resource carries none and reported under {@code Errors}+ * the way the agent reports a tag lookup it could not make.+ */+ private void putTags(ObjectNode target, String field, String resourceArn, ArrayNode errors) {+ if (resourceArn == null) {+ return;+ }+ try {+ Map<String, String> tags = service.listTagsForResource(resourceArn);+ if (!tags.isEmpty()) {+ ObjectNode node = target.putObject(field);+ tags.forEach(node::put);+ }+ } catch (AwsException e) {+ ObjectNode error = errors.addObject();+ error.put("ErrorField", field);+ error.put("ErrorCode", e.getErrorCode());+ error.put("ErrorMessage", e.getMessage());+ error.put("StatusCode", e.getHttpStatus());+ error.put("ResourceARN", resourceArn);+ }+ }++ @GET+ @Path("/{id}/stats")+ @Produces(MediaType.APPLICATION_JSON)+ public Response containerStats(@PathParam("id") String id) {+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);+ if (target.isEmpty()) {+ return notFound();+ }+ EcsContainerManager.ContainerStats sample = containerManager+ .sampleContainerStats(target.get().container().getDockerId()).orElse(null);+ return Response.ok(statsNode(sample)).build();+ }++ @GET+ @Path("/{id}/task/stats")+ @Produces(MediaType.APPLICATION_JSON)+ public Response taskStats(@PathParam("id") String id) {+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);+ if (target.isEmpty()) {+ return notFound();+ }+ List<Container> containers = target.get().task().getContainers();+ if (containers == null) {+ containers = List.of();+ }+ // The task's containers are sampled in one pass rather than one after another: Docker's+ // collection tick is a second, and this is the path a sidecar polls for network metrics.+ Map<String, EcsContainerManager.ContainerStats> sampled = containerManager.sampleContainerStats(+ containers.stream().map(Container::getDockerId).filter(Objects::nonNull).toList());++ ObjectNode stats = objectMapper.createObjectNode();+ for (Container container : containers) {+ stats.set(statsKey(container), statsNode(sampled.get(container.getDockerId())));+ }+ return Response.ok(stats).build();+ }++ /** Docker's stats for one container, or an empty document when the daemon had none to give. */+ private ObjectNode statsNode(EcsContainerManager.ContainerStats stats) {+ if (stats == null) {+ return objectMapper.createObjectNode();+ }+ ObjectNode node = STATS_MAPPER.valueToTree(stats.statistics());+ // Docker reports cumulative counters only. The per-second rates alongside them are the+ // agent's own, and are what a sidecar reads /task/stats for.+ if (stats.rxBytesPerSecond() != null && stats.txBytesPerSecond() != null) {+ ObjectNode rates = node.putObject("network_rate_stats");+ rates.put("rx_bytes_per_sec", stats.rxBytesPerSecond());+ rates.put("tx_bytes_per_sec", stats.txBytesPerSecond());+ }+ return node;+ }++ /** Task stats are keyed by Docker id, the way the daemon keys a container. */+ private static String statsKey(Container container) {+ return container.getDockerId() != null ? container.getDockerId() : container.getName();+ }++ private Response notFound() {+ ObjectNode error = objectMapper.createObjectNode();+ error.put("error", "Unable to get metadata for the requested id");+ return Response.status(Response.Status.NOT_FOUND).entity(error).build();+ }++ private ObjectNode containerNode(EcsTask task, Container container, TaskDefinition taskDef) {+ ContainerDefinition definition = definitionOf(taskDef, container.getName());+ ObjectNode n = objectMapper.createObjectNode();+ n.put("DockerId", container.getDockerId() != null ? container.getDockerId() : container.getMetadataId());+ n.put("Name", container.getName());+ n.put("DockerName", container.getName());+ n.put("Image", container.getImage());+ if (container.getImageDigest() != null && !container.getImageDigest().isBlank()) {+ n.put("ImageID", container.getImageDigest());+ }+ ArrayNode ports = portsNode(container);+ if (!ports.isEmpty()) {+ n.set("Ports", ports);+ }+ n.set("Labels", labelsNode(task, taskDef, container));+ n.put("DesiredStatus", task.getDesiredStatus());+ n.put("KnownStatus", container.getLastStatus());+ if (container.getExitCode() != null) {+ n.put("ExitCode", container.getExitCode());+ }+ ObjectNode limits = containerLimitsNode(definition);+ if (!limits.isEmpty()) {+ n.set("Limits", limits);+ }+ // The container's own clock when Docker gave Floci one, the task's otherwise: a mock-mode+ // task, or one restored from storage, has containers that never went through a daemon.+ putTimestamp(n, "CreatedAt", firstOf(container.getCreatedAt(), task.getCreatedAt()));+ putTimestamp(n, "StartedAt", firstOf(container.getStartedAt(), task.getStartedAt()));+ putTimestamp(n, "FinishedAt", firstOf(container.getFinishedAt(), task.getStoppedAt()));+ n.put("Type", "NORMAL");+ n.put("ContainerARN", container.getContainerArn());+ if (container.getHealthStatus() != null) {+ n.putObject("Health").put("status", container.getHealthStatus());+ }+ if (definition != null && definition.getLogConfiguration() != null) {+ LogConfiguration logConfiguration = definition.getLogConfiguration();+ n.put("LogDriver", logConfiguration.logDriver());+ if (logConfiguration.options() != null) {+ ObjectNode options = n.putObject("LogOptions");+ logConfiguration.options().forEach(options::put);+ }+ }+ n.set("Networks", networksNode(task, taskDef));+ if (task.getPlatformVersion() != null) {+ n.put("Snapshotter", "overlayfs");+ }+ return n;+ }++ private ArrayNode portsNode(Container container) {+ ArrayNode ports = objectMapper.createArrayNode();+ if (container.getNetworkBindings() != null) {+ container.getNetworkBindings().forEach(binding -> {+ ObjectNode port = ports.addObject();+ port.put("ContainerPort", binding.containerPort());+ port.put("Protocol", binding.protocol());+ if (binding.hostPort() > 0) {+ port.put("HostPort", binding.hostPort());+ }+ });+ }+ return ports;+ }++ private ObjectNode taskNode(EcsTask task, TaskDefinition taskDef) {+ ObjectNode n = objectMapper.createObjectNode();+ n.put("Cluster", task.getClusterArn());+ n.put("TaskARN", task.getTaskArn());+ if (taskDef != null) {+ n.put("Family", taskDef.getFamily());+ n.put("Revision", String.valueOf(taskDef.getRevision()));+ }+ n.put("DesiredStatus", task.getDesiredStatus());+ n.put("KnownStatus", task.getLastStatus());+ ObjectNode limits = taskLimitsNode(task);+ if (!limits.isEmpty()) {+ n.set("Limits", limits);+ }+ putTimestamp(n, "PullStartedAt", task.getPullStartedAt());+ putTimestamp(n, "PullStoppedAt", task.getPullStoppedAt());+ putTimestamp(n, "ExecutionStoppedAt", task.getExecutionStoppedAt());+ if (task.getAvailabilityZone() != null) {+ n.put("AvailabilityZone", task.getAvailabilityZone());+ }+ // A service's tasks are grouped as "service:<name>", which is where the endpoint's+ // ServiceName comes from; a standalone task has no service and reports none.+ if (task.getGroup() != null && task.getGroup().startsWith(SERVICE_GROUP_PREFIX)) {+ n.put("ServiceName", task.getGroup().substring(SERVICE_GROUP_PREFIX.length()));+ }+ if (task.getLaunchType() != null) {+ n.put("LaunchType", task.getLaunchType().name());+ }+ // VPCID is the container instance's, so AWS reports it for the EC2 launch type only.+ if (task.getLaunchType() == LaunchType.EC2) {+ Subnet subnet = taskSubnet(task);+ if (subnet != null && subnet.getVpcId() != null) {+ n.put("VPCID", subnet.getVpcId());+ }+ }+ ArrayNode containers = n.putArray("Containers");+ if (task.getContainers() != null) {+ task.getContainers().forEach(container ->+ containers.add(containerNode(task, container, taskDef)));+ }+ // Fargate reports the clock's accuracy and the task's ephemeral storage use. Floci has no+ // drift to report and does not meter the disk, so both are reported as a healthy baseline.+ if (task.getPlatformVersion() != null) {+ ObjectNode clockDrift = n.putObject("ClockDrift");+ clockDrift.put("ClockErrorBound", 0.0);+ clockDrift.put("ReferenceTimestamp", TIMESTAMP.format(Instant.now()));+ clockDrift.put("ClockSynchronizationStatus", "SYNCHRONIZED");+ if (task.getEphemeralStorage() != null) {+ ObjectNode storage = n.putObject("EphemeralStorageMetrics");+ storage.put("Utilized", 0);+ storage.put("Reserved", task.getEphemeralStorage().sizeInGiB() * 1024);+ }+ }+ return n;+ }++ private ObjectNode labelsNode(EcsTask task, TaskDefinition taskDef, Container container) {+ ObjectNode labels = objectMapper.createObjectNode();+ labels.put("com.amazonaws.ecs.cluster", task.getClusterArn());+ labels.put("com.amazonaws.ecs.container-name", container.getName());+ labels.put("com.amazonaws.ecs.task-arn", task.getTaskArn());+ if (taskDef != null) {+ labels.put("com.amazonaws.ecs.task-definition-family", taskDef.getFamily());+ labels.put("com.amazonaws.ecs.task-definition-version", String.valueOf(taskDef.getRevision()));+ }+ return labels;+ }++ /** The container's own limits, omitted member by member when the definition sets none. */+ private ObjectNode containerLimitsNode(ContainerDefinition definition) {+ ObjectNode limits = objectMapper.createObjectNode();+ if (definition == null) {+ return limits;+ }+ if (definition.getCpu() != null) {+ limits.put("CPU", definition.getCpu());+ }+ if (definition.getMemory() != null) {+ limits.put("Memory", definition.getMemory());+ }+ return limits;+ }++ /** The task's own limits: CPU as a vCPU count, memory in MiB, the way the endpoint reports them. */+ private ObjectNode taskLimitsNode(EcsTask task) {+ ObjectNode limits = objectMapper.createObjectNode();+ Integer cpuUnits = parseInteger(task.getCpu());+ if (cpuUnits != null) {+ limits.put("CPU", cpuUnits / 1024.0);+ }+ Integer memoryMb = parseInteger(task.getMemory());+ if (memoryMb != null) {+ limits.put("Memory", memoryMb);+ }+ return limits;+ }++ private ArrayNode networksNode(EcsTask task, TaskDefinition taskDef) {+ ArrayNode networks = objectMapper.createArrayNode();+ ObjectNode network = objectMapper.createObjectNode();+ network.put("NetworkMode", taskDef != null && taskDef.getNetworkMode() != null+ ? taskDef.getNetworkMode().name() : "bridge");+ ArrayNode addresses = network.putArray("IPv4Addresses");+ if (task.getPrivateIpAddress() != null) {+ addresses.add(task.getPrivateIpAddress());+ }+ if (taskSubnetId(task) != null) {+ network.put("AttachmentIndex", 0);+ }+ // The rest of the network object describes the task ENI and the subnet it sits in. Each+ // member is written only when the interface Floci allocated actually carries it.+ if (task.getMacAddress() != null) {+ network.put("MACAddress", task.getMacAddress());+ }+ putSubnetMembers(network, task);+ if (task.getPrivateDnsName() != null) {+ network.put("PrivateDNSName", task.getPrivateDnsName());+ }+ networks.add(network);+ return networks;+ }++ /**+ * The subnet's own view of the network: its CIDR, the gateway VPC reserves as the first+ * address in it, the resolver at the VPC's third address, and the search domain the region+ * hands out. All four are derived, since a local ENI has no DHCP option set behind it.+ */+ private void putSubnetMembers(ObjectNode network, EcsTask task) {+ Subnet subnet = taskSubnet(task);+ if (subnet == null || subnet.getCidrBlock() == null) {+ return;+ }+ network.put("IPv4SubnetCIDRBlock", subnet.getCidrBlock());+ String gateway = addressWithinCidr(subnet.getCidrBlock(), 1);+ if (gateway != null) {+ network.put("SubnetGatewayIpv4Address", gateway + "/" + prefixLengthOf(subnet.getCidrBlock()));+ }+ Vpc vpc = vpcOf(task, subnet);+ String resolver = vpc != null ? addressWithinCidr(vpc.getCidrBlock(), 2) : null;+ if (resolver != null) {+ network.putArray("DomainNameServers").add(resolver);+ }+ String region = taskRegion(task);+ if (region != null) {+ network.putArray("DomainNameSearchList").add(searchDomain(region));+ }+ }++ private Subnet taskSubnet(EcsTask task) {+ String subnetId = taskSubnetId(task);+ String region = taskRegion(task);+ if (subnetId == null || region == null) {+ return null;+ }+ try {+ return ec2Service.describeSubnets(region, List.of(subnetId), Map.of()).stream()+ .findFirst()+ .orElse(null);+ } catch (AwsException e) {+ LOG.debugv("Task {0} references a subnet that is gone: {1}", task.getTaskArn(), e.getMessage());+ return null;+ }+ }++ private Vpc vpcOf(EcsTask task, Subnet subnet) {+ String region = taskRegion(task);+ if (subnet.getVpcId() == null || region == null) {+ return null;+ }+ try {+ return ec2Service.describeVpcs(region, List.of(subnet.getVpcId()), Map.of()).stream()+ .findFirst()+ .orElse(null);+ } catch (AwsException e) {+ LOG.debugv("Subnet {0} references a VPC that is gone: {1}",+ subnet.getSubnetId(), e.getMessage());+ return null;+ }+ }++ private static String taskSubnetId(EcsTask task) {+ AwsVpcConfiguration awsvpc = task.getNetworkConfiguration() != null+ ? task.getNetworkConfiguration().getAwsvpcConfiguration() : null;+ if (awsvpc == null || awsvpc.getSubnets() == null || awsvpc.getSubnets().isEmpty()) {+ return null;+ }+ return awsvpc.getSubnets().getFirst();+ }++ private static String taskRegion(EcsTask task) {+ return AwsArnUtils.regionOrDefault(task.getTaskArn(), null);+ }++ /** The nth address of a CIDR block, or null when the block is not a readable IPv4 one. */+ private static String addressWithinCidr(String cidr, int offset) {+ if (cidr == null) {+ return null;+ }+ String[] parts = cidr.split("/");+ String[] octets = parts[0].split("\\.");+ if (octets.length != 4) {+ return null;+ }+ try {+ long address = 0;+ for (String octet : octets) {+ int value = Integer.parseInt(octet);+ if (value < 0 || value > 255) {+ return null;+ }+ address = (address << 8) | value;+ }+ address += offset;+ return "%d.%d.%d.%d".formatted((address >> 24) & 0xFF, (address >> 16) & 0xFF,+ (address >> 8) & 0xFF, address & 0xFF);+ } catch (NumberFormatException e) {+ LOG.debugv("CIDR block {0} is not a readable IPv4 block: {1}", cidr, e.getMessage());+ return null;+ }+ }++ private static String prefixLengthOf(String cidr) {+ int slash = cidr.indexOf('/');+ return slash < 0 ? "32" : cidr.substring(slash + 1);+ }++ private static String searchDomain(String region) {+ return LEGACY_SEARCH_DOMAIN_REGION.equals(region) ? "ec2.internal" : region + ".compute.internal";+ }++ private static ContainerDefinition definitionOf(TaskDefinition taskDef, String containerName) {+ if (taskDef == null || taskDef.getContainerDefinitions() == null) {+ return null;+ }+ return taskDef.getContainerDefinitions().stream()+ .filter(definition -> containerName.equals(definition.getName()))+ .findFirst()+ .orElse(null);+ }++ private static Integer parseInteger(String value) {+ if (value == null || value.isBlank()) {+ return null;+ }+ try {+ return Integer.parseInt(value.trim());+ } catch (NumberFormatException e) {+ return null;+ }+ }++ private static Instant firstOf(Instant preferred, Instant fallback) {+ return preferred != null ? preferred : fallback;+ }++ private static void putTimestamp(ObjectNode target, String field, Instant value) {+ if (value != null) {+ target.put(field, TIMESTAMP.format(value));+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.javaindex c880776c41..d6c923072b 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java@@ -43,11 +43,15 @@ import io.github.hectorvent.floci.services.s3.model.S3Object;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;import io.github.hectorvent.floci.services.ssm.SsmService;import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.async.ResultCallback;import com.github.dockerjava.api.command.InspectContainerResponse;+import com.github.dockerjava.api.command.StatsCmd;import com.github.dockerjava.api.exception.NotFoundException;import com.github.dockerjava.api.model.ContainerNetwork;import com.github.dockerjava.api.model.ExposedPort;import com.github.dockerjava.api.model.LogConfig;+import com.github.dockerjava.api.model.StatisticNetworksConfig;+import com.github.dockerjava.api.model.Statistics;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;import org.apache.commons.compress.archivers.tar.TarArchiveEntry;@@ -61,7 +65,9 @@ import java.io.IOException;import java.nio.charset.StandardCharsets;import java.security.MessageDigest;import java.security.NoSuchAlgorithmException;+import java.time.Duration;import java.time.Instant;+import java.time.format.DateTimeParseException;import java.util.ArrayList;import java.util.HashSet;import java.util.HexFormat;@@ -70,7 +76,11 @@ import java.util.List;import java.util.Map;import java.util.Optional;import java.util.Set;+import java.util.UUID;+import java.util.concurrent.CopyOnWriteArrayList;+import java.util.concurrent.CountDownLatch;import java.util.concurrent.TimeUnit;+import java.util.function.Function;import java.util.stream.Stream;/**@@ -92,6 +102,13 @@ public class EcsContainerManager {private static final long DEPENDENCY_POLL_MILLIS = 200;/** How long a killed container gets to register as exited before its code is read. */private static final int KILL_SETTLE_SECONDS = 5;+ /** How long a stats sample gets before the task metadata endpoint answers without one. */+ private static final int STATS_SAMPLE_SECONDS = 5;+ /**+ * Docker writes {@code 0001-01-01T00:00:00Z} for an event that has not happened. Anything at+ * or before the Unix epoch is that zero value rather than a time a container reached.+ */+ private static final Instant DOCKER_ZERO_TIMESTAMP = Instant.EPOCH;private final ContainerBuilder containerBuilder;private final ContainerLifecycleManager lifecycleManager;@@ -199,8 +216,14 @@ public class EcsContainerManager {Map<ContainerDefinition, List<String>> envVarsByContainer = new LinkedHashMap<>();// Resolved before any container is created, so a registry-startup failure can't leak one already started.Map<ContainerDefinition, String> imagesByContainer = new LinkedHashMap<>();+ // The task metadata id has to exist before the container does: its own environment carries+ // the URI, so it cannot be derived from the Docker id the daemon hands back afterwards.+ Map<String, String> metadataIdsByContainer = new LinkedHashMap<>();for (ContainerDefinition def : launchOrder) {- envVarsByContainer.put(def, buildEnvVars(def, overridesByName.get(def.getName()), region));+ String metadataId = UUID.randomUUID().toString().replace("-", "");+ metadataIdsByContainer.put(def.getName(), metadataId);+ envVarsByContainer.put(def, buildEnvVars(def, overridesByName.get(def.getName()), region,+ metadataId));imagesByContainer.put(def, ecrRegistryManager.rewriteImageUri(def.getImage()));}@@ -410,7 +433,8 @@ public class EcsContainerManager {protectedNetwork == null ? dockerId : protectedNetwork.namespace().helperId(), def);// Build ECS container model- Container container = buildContainer(task.getTaskArn(), def, dockerId, networkBindings, region);+ Container container = buildContainer(task.getTaskArn(), def, dockerId, networkBindings, region,+ metadataIdsByContainer.get(def.getName()));runtimeContainers.add(container);containerIds.put(def.getName(), dockerId);@@ -606,6 +630,211 @@ public class EcsContainerManager {}}+ /**+ * A container's Docker stats as the task metadata endpoint reports them: the latest sample,+ * plus the network byte rates. The rates are null when they cannot be derived, and Docker+ * never reports them itself: they are a delta the ECS agent takes across two samples.+ */+ public record ContainerStats(Statistics statistics, Double rxBytesPerSecond, Double txBytesPerSecond) {}++ /**+ * Reads a container's stats from the daemon. Empty when there is nothing to read: the+ * container has no Docker id, it has already gone, or the daemon is unreachable.+ *+ * <p>The stats stream is read rather than a single {@code stream=false} document, because the+ * network rates are a delta between consecutive samples. Docker sends the first sample at once+ * and the next on its collection tick a second later, which is the same second AWS documents a+ * container as having to run before its stats are available.+ */+ public Optional<ContainerStats> sampleContainerStats(String dockerId) {+ if (dockerId == null || dockerId.isBlank()) {+ return Optional.empty();+ }+ return Optional.ofNullable(sampleContainerStats(List.of(dockerId)).get(dockerId));+ }++ /**+ * Reads several containers' stats in one pass, keyed by Docker id. A container the daemon has+ * nothing to say about is absent from the map rather than present with an empty sample.+ *+ * <p>Every stream is opened before any of them is waited on, and the waits share one deadline.+ * Docker's stats stream is asynchronous, so the containers tick alongside each other and the+ * call costs about the single collection tick one container costs, however many containers the+ * task has. That matters on {@code /task/stats}, which is the path a sidecar polls: sampling+ * the containers one after another would cost a tick each.+ */+ public Map<String, ContainerStats> sampleContainerStats(List<String> dockerIds) {+ Map<String, ContainerStats> sampled = new LinkedHashMap<>();+ if (dockerIds == null || dockerIds.isEmpty()) {+ return sampled;+ }+ Map<String, OpenStatsStream> streams = openStatsStreams(dockerIds);+ try {+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(STATS_SAMPLE_SECONDS);+ for (Map.Entry<String, OpenStatsStream> stream : streams.entrySet()) {+ awaitPair(stream.getKey(), stream.getValue().samples(), deadline);+ containerStats(stream.getValue().samples().collected())+ .ifPresent(stats -> sampled.put(stream.getKey(), stats));+ }+ } finally {+ for (OpenStatsStream stream : streams.values()) {+ closeQuietly(stream.samples());+ closeQuietly(stream.command());+ }+ }+ return sampled;+ }++ /** A stats stream the daemon is already filling, and the command that has to be closed with it. */+ private record OpenStatsStream(StatsCmd command, ConsecutiveStatsSamples samples) {}++ /** Starts every container's stats stream, skipping the ones the daemon will not open. */+ private Map<String, OpenStatsStream> openStatsStreams(List<String> dockerIds) {+ Map<String, OpenStatsStream> streams = new LinkedHashMap<>();+ for (String dockerId : dockerIds) {+ if (dockerId == null || dockerId.isBlank() || streams.containsKey(dockerId)) {+ continue;+ }+ StatsCmd command = null;+ try {+ command = lifecycleManager.getDockerClient().statsCmd(dockerId);+ ConsecutiveStatsSamples samples = new ConsecutiveStatsSamples();+ command.exec(samples);+ streams.put(dockerId, new OpenStatsStream(command, samples));+ } catch (RuntimeException e) {+ closeQuietly(command);+ LOG.debugv("Could not sample stats for container {0}: {1}", dockerId, e.getMessage());+ }+ }+ return streams;+ }++ /** Waits for one stream's pair of samples, never past the deadline the whole pass shares. */+ private static void awaitPair(String dockerId, ConsecutiveStatsSamples samples, long deadlineNanos) {+ long remaining = deadlineNanos - System.nanoTime();+ try {+ if (remaining <= 0 || !samples.awaitPair(remaining, TimeUnit.NANOSECONDS)) {+ LOG.debugv("Docker sent fewer than two stats samples for container {0} within {1}s",+ dockerId, STATS_SAMPLE_SECONDS);+ }+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ LOG.debugv("Interrupted while sampling stats for container {0}", dockerId);+ }+ }++ private static void closeQuietly(Closeable closeable) {+ if (closeable == null) {+ return;+ }+ try {+ closeable.close();+ } catch (IOException | RuntimeException e) {+ LOG.debugv("Could not close a Docker stats stream: {0}", e.getMessage());+ }+ }++ /** The latest sample, with rates when two samples a measurable interval apart arrived. */+ private static Optional<ContainerStats> containerStats(List<Statistics> samples) {+ if (samples.isEmpty()) {+ return Optional.empty();+ }+ Statistics latest = samples.getLast();+ if (samples.size() < 2) {+ return Optional.of(new ContainerStats(latest, null, null));+ }+ Statistics earlier = samples.getFirst();+ double seconds = secondsBetween(earlier.getRead(), latest.getRead());+ Long rxDelta = byteDelta(earlier, latest, StatisticNetworksConfig::getRxBytes);+ Long txDelta = byteDelta(earlier, latest, StatisticNetworksConfig::getTxBytes);+ if (seconds <= 0 || rxDelta == null || txDelta == null) {+ return Optional.of(new ContainerStats(latest, null, null));+ }+ return Optional.of(new ContainerStats(latest, rxDelta / seconds, txDelta / seconds));+ }++ /** The counter's movement across the two samples, or null when neither reports an interface. */+ private static Long byteDelta(Statistics earlier, Statistics latest,+ Function<StatisticNetworksConfig, Long> counter) {+ Long earlierBytes = totalBytes(earlier, counter);+ Long latestBytes = totalBytes(latest, counter);+ if (earlierBytes == null || latestBytes == null) {+ return null;+ }+ return Math.max(0, latestBytes - earlierBytes);+ }++ private static Long totalBytes(Statistics statistics,+ Function<StatisticNetworksConfig, Long> counter) {+ Map<String, StatisticNetworksConfig> networks = statistics.getNetworks();+ if (networks == null || networks.isEmpty()) {+ return null;+ }+ long total = 0;+ for (StatisticNetworksConfig network : networks.values()) {+ Long bytes = counter.apply(network);+ total += bytes != null ? bytes : 0;+ }+ return total;+ }++ /** The interval between two Docker read timestamps, or 0 when either cannot be read. */+ private static double secondsBetween(String earlier, String latest) {+ if (earlier == null || latest == null) {+ return 0;+ }+ try {+ return Duration.between(Instant.parse(earlier), Instant.parse(latest)).toNanos() / 1_000_000_000.0;+ } catch (DateTimeParseException e) {+ LOG.debugv("Docker stats carried an unreadable read timestamp: {0}", e.getMessage());+ return 0;+ }+ }++ /** Docker's stats stream never ends on its own; this takes the first two samples and stops. */+ private static final class ConsecutiveStatsSamples extends ResultCallback.Adapter<Statistics> {++ private static final int WANTED = 2;++ private final CountDownLatch enough = new CountDownLatch(WANTED);+ private final List<Statistics> collected = new CopyOnWriteArrayList<>();++ @Override+ public void onNext(Statistics statistics) {+ if (collected.size() < WANTED) {+ collected.add(statistics);+ enough.countDown();+ }+ }++ @Override+ public void onError(Throwable throwable) {+ releaseWaiter();+ super.onError(throwable);+ }++ @Override+ public void onComplete() {+ releaseWaiter();+ super.onComplete();+ }++ /** A stream that ends or fails early must not hold the caller until the timeout. */+ private void releaseWaiter() {+ while (enough.getCount() > 0) {+ enough.countDown();+ }+ }++ boolean awaitPair(long timeout, TimeUnit unit) throws InterruptedException {+ return enough.await(timeout, unit);+ }++ List<Statistics> collected() {+ return List.copyOf(collected);+ }+ }+private List<ContainerDefinition> orderForDependencies(List<ContainerDefinition> definitions,ContainerDefinition firelensRouter) {Map<String, ContainerDefinition> definitionsByName = new LinkedHashMap<>();@@ -899,6 +1128,8 @@ public class EcsContainerManager {String name = entry.getKey();String dockerId = entry.getValue();exitCodes.put(name, getExitCodeIfStopped(dockerId));+ // Read before the removal below, which is the last moment the daemon still knows it.+ handle.recordFinishedAt(name, getFinishedAtIfStopped(dockerId));try {lifecycleManager.getDockerClient().removeContainerCmd(dockerId).withForce(true).exec();terminatedContainerIds.add(dockerId);@@ -1231,7 +1462,8 @@ public class EcsContainerManager {}}- private List<String> buildEnvVars(ContainerDefinition def, ContainerOverride override, String region) {+ private List<String> buildEnvVars(ContainerDefinition def, ContainerOverride override, String region,+ String metadataId) {// AWS SDK baseline (endpoint + region + credentials) first so the task can reach the// emulator, then the task-def environment, then task-def secrets, then the override// environment. Later entries win on key conflict, so an explicit task-def value or@@ -1243,6 +1475,12 @@ public class EcsContainerManager {envMap.put(kv.substring(0, eq), kv.substring(eq + 1));}}+ // The task metadata endpoint, which an application, the ECS SDK integrations and the+ // aws-for-fluent-bit init process all read from this variable.+ String flociEndpoint = awsEnv.flociEndpoint();+ if (metadataId != null && flociEndpoint != null) {+ envMap.put("ECS_CONTAINER_METADATA_URI_V4", flociEndpoint + "/v4/" + metadataId);+ }if (def.getEnvironment() != null) {for (var kv : def.getEnvironment()) {envMap.put(kv.name(), kv.value());@@ -1440,7 +1678,8 @@ public class EcsContainerManager {}private Container buildContainer(String taskArn, ContainerDefinition def, String dockerId,- List<NetworkBinding> networkBindings, String region) {+ List<NetworkBinding> networkBindings, String region,+ String metadataId) {Container container = new Container();container.setTaskArn(taskArn);container.setName(def.getName());@@ -1449,6 +1688,7 @@ public class EcsContainerManager {container.setNetworkBindings(networkBindings);container.setDockerId(dockerId);container.setRuntimeId(dockerId);+ container.setMetadataId(metadataId);container.setHealthStatus(def.getHealthCheck() != null ? "UNKNOWN" : null);if (def.getCpu() != null) {container.setCpu(String.valueOf(def.getCpu()));@@ -1461,9 +1701,62 @@ public class EcsContainerManager {}container.setContainerArn(regionResolver.buildArn("ecs", region,"container/" + extractTaskId(taskArn) + "/" + def.getName()));+ applyLaunchTimestamps(container, dockerId);return container;}+ /**+ * Stamps the container with Docker's own create and start times. A task's containers start one+ * at a time, holding for each other's {@code dependsOn} conditions, so these are the container's+ * and not the task's.+ */+ private void applyLaunchTimestamps(Container container, String dockerId) {+ InspectContainerResponse inspect = inspectOrNull(dockerId);+ if (inspect == null) {+ return;+ }+ container.setCreatedAt(dockerTimestamp(inspect.getCreated()));+ if (inspect.getState() != null) {+ container.setStartedAt(dockerTimestamp(inspect.getState().getStartedAt()));+ }+ }++ /**+ * When a container that has already stopped finished. Read before the container is removed,+ * since the daemon forgets it afterwards.+ */+ public Instant getFinishedAtIfStopped(String dockerId) {+ InspectContainerResponse inspect = inspectOrNull(dockerId);+ if (inspect == null || inspect.getState() == null+ || Boolean.TRUE.equals(inspect.getState().getRunning())) {+ return null;+ }+ return dockerTimestamp(inspect.getState().getFinishedAt());+ }++ private InspectContainerResponse inspectOrNull(String dockerId) {+ try {+ return lifecycleManager.getDockerClient().inspectContainerCmd(dockerId).exec();+ } catch (Exception e) {+ LOG.debugv("Could not inspect container {0}: {1}", dockerId, e.getMessage());+ return null;+ }+ }++ /** A Docker RFC 3339 timestamp, or null for its zero value, which means "never happened". */+ private static Instant dockerTimestamp(String value) {+ if (value == null || value.isBlank()) {+ return null;+ }+ try {+ Instant instant = Instant.parse(value);+ return instant.isAfter(DOCKER_ZERO_TIMESTAMP) ? instant : null;+ } catch (DateTimeParseException e) {+ LOG.debugv("Docker reported an unreadable timestamp {0}: {1}", value, e.getMessage());+ return null;+ }+ }+private static String extractTaskId(String arn) {if (arn == null) {return "";diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.javaindex deb19ccd0f..5cfee43984 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java@@ -1,6 +1,7 @@package io.github.hectorvent.floci.services.ecs.container;import java.io.Closeable;+import java.time.Instant;import java.util.LinkedHashMap;import java.util.Map;@@ -28,6 +29,11 @@ public class EcsTaskHandle {private final String region;/** Per-container {@code stopTimeout}, so teardown waits as long as the task definition asked. */private final Map<String, Integer> stopTimeouts;+ /**+ * When each container finished, read during teardown. Docker forgets a container once it is+ * removed, so the time has to be kept here for the caller that stamps the task's containers.+ */+ private final Map<String, Instant> finishedAt = new LinkedHashMap<>();public EcsTaskHandle(String taskArn, Map<String, String> containerIds,Map<String, Closeable> logStreamsByContainerId) {@@ -67,6 +73,18 @@ public class EcsTaskHandle {return configured != null ? configured : DEFAULT_STOP_TIMEOUT_SECONDS;}+ /** Notes when a container finished. A null is ignored, so a failed read leaves no entry. */+ public void recordFinishedAt(String containerName, Instant instant) {+ if (instant != null) {+ finishedAt.put(containerName, instant);+ }+ }++ /** When each container finished, for the containers teardown could read a time for. */+ public Map<String, Instant> getFinishedAt() {+ return finishedAt;+ }+/** Removes and returns the log stream that no longer needs task-level ownership. */public Closeable removeLogStream(String containerId) {return logStreamsByContainerId.remove(containerId);diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/Container.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/Container.javaindex 1a2e3ce270..fe95486fec 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/model/Container.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/Container.java@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.ecs.model;import io.quarkus.runtime.annotations.RegisterForReflection;+import java.time.Instant;import java.util.List;@RegisterForReflection@@ -19,10 +20,24 @@ public class Container {private List<ManagedAgent> managedAgents;private String healthStatus;private String runtimeId;+ /**+ * The opaque id in this container's {@code ECS_CONTAINER_METADATA_URI_V4}. Minted before the+ * container is created, the way the ECS agent mints one, so the URI can be injected into the+ * container's own environment.+ */+ private String metadataId;private String imageDigest;private String cpu;private String memory;private String memoryReservation;+ /**+ * When Docker created, started and finished this container. The ECS API's own container shape+ * carries none of these, so they never reach a DescribeTasks response; the task metadata+ * endpoint reports them, and its containers start and stop one at a time.+ */+ private Instant createdAt;+ private Instant startedAt;+ private Instant finishedAt;// transient — not persistedprivate transient String dockerId;@@ -65,6 +80,9 @@ public class Container {public String getRuntimeId() { return runtimeId; }public void setRuntimeId(String runtimeId) { this.runtimeId = runtimeId; }+ public String getMetadataId() { return metadataId; }+ public void setMetadataId(String metadataId) { this.metadataId = metadataId; }+public String getImageDigest() { return imageDigest; }public void setImageDigest(String imageDigest) { this.imageDigest = imageDigest; }@@ -77,6 +95,15 @@ public class Container {public String getMemoryReservation() { return memoryReservation; }public void setMemoryReservation(String memoryReservation) { this.memoryReservation = memoryReservation; }+ public Instant getCreatedAt() { return createdAt; }+ public void setCreatedAt(Instant createdAt) { this.createdAt = createdAt; }++ public Instant getStartedAt() { return startedAt; }+ public void setStartedAt(Instant startedAt) { this.startedAt = startedAt; }++ public Instant getFinishedAt() { return finishedAt; }+ public void setFinishedAt(Instant finishedAt) { this.finishedAt = finishedAt; }+public String getDockerId() { return dockerId; }public void setDockerId(String dockerId) { this.dockerId = dockerId; }}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataControllerTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataControllerTest.javanew file mode 100644index 0000000000..b0dd1ee62e--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataControllerTest.java@@ -0,0 +1,397 @@+package io.github.hectorvent.floci.services.ecs;++import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.services.ec2.Ec2Service;+import io.github.hectorvent.floci.services.ec2.model.Subnet;+import io.github.hectorvent.floci.services.ec2.model.Vpc;+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;+import io.github.hectorvent.floci.services.ecs.model.AwsVpcConfiguration;+import io.github.hectorvent.floci.services.ecs.model.Container;+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;+import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.EphemeralStorage;+import io.github.hectorvent.floci.services.ecs.model.LaunchType;+import io.github.hectorvent.floci.services.ecs.model.LogConfiguration;+import io.github.hectorvent.floci.services.ecs.model.NetworkConfiguration;+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;+import com.fasterxml.jackson.core.JsonProcessingException;+import com.fasterxml.jackson.databind.DeserializationFeature;+import com.fasterxml.jackson.databind.JsonNode;+import com.fasterxml.jackson.databind.ObjectMapper;+import com.fasterxml.jackson.databind.json.JsonMapper;+import com.github.dockerjava.api.model.Statistics;+import jakarta.ws.rs.core.Response;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.time.Instant;+import java.util.List;+import java.util.Map;+import java.util.Optional;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.anyList;+import static org.mockito.ArgumentMatchers.anyMap;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++/**+ * The task metadata endpoint's response shapes. A workload reads these through+ * {@code ECS_CONTAINER_METADATA_URI_V4} expecting AWS's exact PascalCase members, so the mapping+ * from Floci's task model onto them is what this pins down.+ */+class EcsTaskMetadataControllerTest {++ private static final String METADATA_ID = "9e2b1f0c4d5e4a1b8c7d6e5f4a3b2c1d";+ private static final String TASK_ARN =+ "arn:aws:ecs:us-east-1:000000000000:task/metadata-cluster/abc123";++ private ObjectMapper objectMapper;+ private EcsService service;+ private EcsContainerManager containerManager;+ private Ec2Service ec2Service;+ private EcsTask task;+ private EcsTaskMetadataController controller;++ @BeforeEach+ void setUp() {+ objectMapper = new ObjectMapper();+ service = mock(EcsService.class);+ containerManager = mock(EcsContainerManager.class);+ ec2Service = mock(Ec2Service.class);+ when(containerManager.sampleContainerStats(anyString())).thenReturn(Optional.empty());+ when(containerManager.sampleContainerStats(anyList())).thenReturn(Map.of());+ when(ec2Service.describeSubnets(anyString(), anyList(), anyMap())).thenReturn(List.of(subnet()));+ when(ec2Service.describeVpcs(anyString(), anyList(), anyMap())).thenReturn(List.of(vpc()));+ task = task();+ TaskDefinition taskDef = taskDefinition();+ when(service.findByMetadataId(anyString())).thenAnswer(inv ->+ METADATA_ID.equals(inv.getArgument(0))+ ? Optional.of(new EcsService.MetadataTarget(task,+ task.getContainers().getFirst(), taskDef))+ : Optional.empty());+ controller = new EcsTaskMetadataController(service, containerManager, ec2Service, objectMapper);+ }++ private static Subnet subnet() {+ Subnet subnet = new Subnet();+ subnet.setSubnetId("subnet-default-us-east-1-a");+ subnet.setVpcId("vpc-0a1b2c3d");+ subnet.setCidrBlock("172.31.32.0/20");+ subnet.setAvailabilityZone("us-east-1a");+ return subnet;+ }++ private static Vpc vpc() {+ Vpc vpc = new Vpc();+ vpc.setVpcId("vpc-0a1b2c3d");+ vpc.setCidrBlock("172.31.0.0/16");+ return vpc;+ }++ private static EcsTask task() {+ EcsTask task = new EcsTask();+ task.setTaskArn(TASK_ARN);+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/metadata-cluster");+ task.setTaskDefinitionArn("arn:aws:ecs:us-east-1:000000000000:task-definition/web:3");+ task.setLastStatus("RUNNING");+ task.setDesiredStatus("RUNNING");+ task.setLaunchType(LaunchType.FARGATE);+ task.setPlatformVersion("1.4.0");+ task.setPlatformFamily("Linux");+ task.setCpu("256");+ task.setMemory("512");+ task.setEphemeralStorage(new EphemeralStorage(20));+ task.setAvailabilityZone("us-east-1a");+ task.setCreatedAt(Instant.parse("2026-01-01T10:00:00Z"));+ task.setStartedAt(Instant.parse("2026-01-01T10:00:05Z"));+ task.setPullStartedAt(Instant.parse("2026-01-01T10:00:01Z"));+ task.setPullStoppedAt(Instant.parse("2026-01-01T10:00:04Z"));+ task.setPrivateIpAddress("172.31.0.42");+ task.setMacAddress("0e:98:9f:33:76:d3");+ task.setPrivateDnsName("ip-172-31-0-42.ec2.internal");++ AwsVpcConfiguration awsvpc = new AwsVpcConfiguration();+ awsvpc.setSubnets(List.of("subnet-default-us-east-1-a"));+ NetworkConfiguration networkConfiguration = new NetworkConfiguration();+ networkConfiguration.setAwsvpcConfiguration(awsvpc);+ task.setNetworkConfiguration(networkConfiguration);++ Container container = new Container();+ container.setName("app");+ container.setImage("nginx:latest");+ container.setLastStatus("RUNNING");+ container.setDockerId("docker-id-1");+ container.setMetadataId(METADATA_ID);+ container.setContainerArn("arn:aws:ecs:us-east-1:000000000000:container/abc123/app");+ container.setCreatedAt(Instant.parse("2026-01-01T10:00:03Z"));+ container.setStartedAt(Instant.parse("2026-01-01T10:00:07Z"));+ task.setContainers(List.of(container));+ return task;+ }++ private static TaskDefinition taskDefinition() {+ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage("nginx:latest");+ app.setCpu(128);+ app.setMemory(256);+ app.setLogConfiguration(new LogConfiguration("awslogs",+ Map.of("awslogs-group", "/ecs/web"), null));++ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("web");+ taskDef.setRevision(3);+ taskDef.setNetworkMode(NetworkMode.awsvpc);+ taskDef.setContainerDefinitions(List.of(app));+ return taskDef;+ }++ private JsonNode body(Response response) {+ return objectMapper.valueToTree(response.getEntity());+ }++ @Test+ void containerMetadataReportsTheAwsMembers() {+ Response response = controller.container(METADATA_ID);+ assertEquals(200, response.getStatus());++ JsonNode container = body(response);+ assertEquals("docker-id-1", container.path("DockerId").asText());+ assertEquals("app", container.path("Name").asText());+ assertEquals("nginx:latest", container.path("Image").asText());+ assertEquals("RUNNING", container.path("KnownStatus").asText());+ assertEquals("NORMAL", container.path("Type").asText());+ assertEquals(128, container.path("Limits").path("CPU").asInt());+ assertEquals(256, container.path("Limits").path("Memory").asInt());+ // SubnetId is not a member of the v4 network object; only the documented ones are written.+ assertTrue(container.path("Networks").get(0).path("SubnetId").isMissingNode(),+ "the network object must carry only the members AWS documents");+ assertEquals("awslogs", container.path("LogDriver").asText());+ assertEquals("/ecs/web", container.path("LogOptions").path("awslogs-group").asText());+ assertEquals("web", container.path("Labels").path("com.amazonaws.ecs.task-definition-family").asText());+ assertEquals("3", container.path("Labels").path("com.amazonaws.ecs.task-definition-version").asText());+ assertEquals(TASK_ARN, container.path("Labels").path("com.amazonaws.ecs.task-arn").asText());+ assertEquals("awsvpc", container.path("Networks").get(0).path("NetworkMode").asText());+ assertEquals("172.31.0.42", container.path("Networks").get(0).path("IPv4Addresses").get(0).asText());+ assertEquals(0, container.path("Networks").get(0).path("AttachmentIndex").asInt());+ assertEquals("0e:98:9f:33:76:d3", container.path("Networks").get(0).path("MACAddress").asText());+ assertEquals("ip-172-31-0-42.ec2.internal",+ container.path("Networks").get(0).path("PrivateDNSName").asText());+ // A container's own clock, not the task's: they start one at a time behind dependsOn.+ assertEquals("2026-01-01T10:00:03Z", container.path("CreatedAt").asText());+ assertEquals("2026-01-01T10:00:07Z", container.path("StartedAt").asText());+ }++ @Test+ void aContainerWithoutItsOwnClockFallsBackToTheTasks() {+ Container container = task.getContainers().getFirst();+ container.setCreatedAt(null);+ container.setStartedAt(null);++ JsonNode node = body(controller.container(METADATA_ID));++ assertEquals("2026-01-01T10:00:00Z", node.path("CreatedAt").asText());+ assertEquals("2026-01-01T10:00:05Z", node.path("StartedAt").asText());+ }++ @Test+ void aStoppedContainerReportsWhenItFinished() {+ task.getContainers().getFirst().setFinishedAt(Instant.parse("2026-01-01T10:30:00Z"));++ assertEquals("2026-01-01T10:30:00Z",+ body(controller.container(METADATA_ID)).path("FinishedAt").asText());+ }++ /**+ * The subnet half of the network object. AWS derives the gateway from the subnet's own CIDR,+ * the resolver from the VPC's, and names us-east-1's search domain {@code ec2.internal}.+ */+ @Test+ void theNetworkObjectDescribesTheSubnetTheTaskSitsIn() {+ JsonNode network = body(controller.container(METADATA_ID)).path("Networks").get(0);++ assertEquals("172.31.32.0/20", network.path("IPv4SubnetCIDRBlock").asText());+ assertEquals("172.31.32.1/20", network.path("SubnetGatewayIpv4Address").asText());+ assertEquals("172.31.0.2", network.path("DomainNameServers").get(0).asText());+ assertEquals("ec2.internal", network.path("DomainNameSearchList").get(0).asText());+ }++ @Test+ void aTaskWithNoNetworkConfigurationReportsNoSubnetMembers() {+ task.setNetworkConfiguration(null);++ JsonNode network = body(controller.container(METADATA_ID)).path("Networks").get(0);++ assertTrue(network.path("AttachmentIndex").isMissingNode());+ assertTrue(network.path("IPv4SubnetCIDRBlock").isMissingNode());+ assertTrue(network.path("DomainNameServers").isMissingNode());+ }++ @Test+ void taskMetadataReportsTheTaskAndItsContainers() {+ Response response = controller.task(METADATA_ID);+ assertEquals(200, response.getStatus());++ JsonNode task = body(response);+ assertEquals(TASK_ARN, task.path("TaskARN").asText());+ assertEquals("web", task.path("Family").asText());+ assertEquals("3", task.path("Revision").asText());+ assertEquals("FARGATE", task.path("LaunchType").asText());+ assertEquals("us-east-1a", task.path("AvailabilityZone").asText());+ // The task's CPU limit is a vCPU count, not the CPU units the API takes.+ assertEquals(0.25, task.path("Limits").path("CPU").asDouble(), 0.0001);+ assertEquals(512, task.path("Limits").path("Memory").asInt());+ assertEquals("2026-01-01T10:00:01Z", task.path("PullStartedAt").asText());+ assertEquals(1, task.path("Containers").size());+ assertEquals("app", task.path("Containers").get(0).path("Name").asText());+ // Fargate reports the clock's accuracy and the task's ephemeral storage alongside it.+ assertEquals("SYNCHRONIZED", task.path("ClockDrift").path("ClockSynchronizationStatus").asText());+ assertTrue(task.path("ClockDrift").has("ReferenceTimestamp"));+ assertEquals(20 * 1024, task.path("EphemeralStorageMetrics").path("Reserved").asInt());+ // VPCID is the container instance's, which a Fargate task does not have.+ assertTrue(task.path("VPCID").isMissingNode());+ }++ @Test+ void anEc2TaskReportsTheVpcItsSubnetBelongsTo() {+ task.setLaunchType(LaunchType.EC2);++ assertEquals("vpc-0a1b2c3d", body(controller.task(METADATA_ID)).path("VPCID").asText());+ }++ @Test+ void taskWithTagsAddsTheTaskAndContainerInstanceTags() {+ task.setLaunchType(LaunchType.EC2);+ task.setContainerInstanceArn(+ "arn:aws:ecs:us-east-1:000000000000:container-instance/metadata-cluster/ci-1");+ when(service.listTagsForResource(TASK_ARN)).thenReturn(Map.of("owner", "platform"));+ when(service.listTagsForResource(task.getContainerInstanceArn()))+ .thenReturn(Map.of("fleet", "spot"));++ JsonNode withTags = body(controller.taskWithTags(METADATA_ID));++ assertEquals(TASK_ARN, withTags.path("TaskARN").asText(), "the task document is carried whole");+ assertEquals("platform", withTags.path("TaskTags").path("owner").asText());+ assertEquals("spot", withTags.path("ContainerInstanceTags").path("fleet").asText());+ assertTrue(withTags.path("Errors").isMissingNode());+ }++ /** The container agent serves taskWithTags, and a Fargate task has no agent behind it. */+ @Test+ void taskWithTagsIsNotFoundForAFargateTask() {+ assertEquals(404, controller.taskWithTags(METADATA_ID).getStatus());+ }++ @Test+ void taskWithTagsReportsAFailedTagLookupUnderErrors() {+ task.setLaunchType(LaunchType.EC2);+ when(service.listTagsForResource(TASK_ARN))+ .thenThrow(new AwsException("InvalidParameterException", "Resource not found", 400));++ JsonNode withTags = body(controller.taskWithTags(METADATA_ID));++ assertTrue(withTags.path("TaskTags").isMissingNode());+ assertEquals("TaskTags", withTags.path("Errors").get(0).path("ErrorField").asText());+ assertEquals("InvalidParameterException",+ withTags.path("Errors").get(0).path("ErrorCode").asText());+ assertEquals(400, withTags.path("Errors").get(0).path("StatusCode").asInt());+ }++ @Test+ void aContainerWithoutLimitsReportsNoneRatherThanZeroes() {+ EcsService limitlessService = mock(EcsService.class);+ EcsTask limitlessTask = task();+ TaskDefinition taskDef = taskDefinition();+ taskDef.getContainerDefinitions().getFirst().setCpu(null);+ taskDef.getContainerDefinitions().getFirst().setMemory(null);+ when(limitlessService.findByMetadataId(anyString())).thenReturn(Optional.of(+ new EcsService.MetadataTarget(limitlessTask, limitlessTask.getContainers().getFirst(), taskDef)));++ JsonNode container = body(new EcsTaskMetadataController(limitlessService, containerManager,+ ec2Service, objectMapper).container(METADATA_ID));++ assertTrue(container.path("Limits").isMissingNode(),+ "Limits is omitted when the container definition sets none");+ assertTrue(container.path("ImageID").isMissingNode(),+ "ImageID is omitted until the image digest is known");+ }++ @Test+ void statsReportTheSampleDockerGaveUnderItsOwnMemberNames() throws JsonProcessingException {+ EcsContainerManager.ContainerStats stats1 =+ new EcsContainerManager.ContainerStats(sample(), 43.5, 215.25);+ when(containerManager.sampleContainerStats("docker-id-1")).thenReturn(Optional.of(stats1));+ when(containerManager.sampleContainerStats(List.of("docker-id-1")))+ .thenReturn(Map.of("docker-id-1", stats1));++ JsonNode stats = body(controller.containerStats(METADATA_ID));+ assertEquals("2026-01-01T10:05:00.000000000Z", stats.path("read").asText());+ assertEquals(42000000, stats.path("cpu_stats").path("cpu_usage").path("total_usage").asLong());+ assertEquals(1048576, stats.path("memory_stats").path("usage").asLong());+ assertEquals(84, stats.path("networks").path("eth0").path("rx_bytes").asLong());+ // The rates are the agent's, derived from consecutive samples; Docker reports only totals.+ assertEquals(43.5, stats.path("network_rate_stats").path("rx_bytes_per_sec").asDouble(), 0.0001);+ assertEquals(215.25, stats.path("network_rate_stats").path("tx_bytes_per_sec").asDouble(), 0.0001);+ // docker-java's model carries a field per API version; the ones this daemon left out are+ // dropped rather than written as nulls onto a document a client parses as Docker's.+ assertFalse(stats.has("network"), "a member the daemon did not send must not appear as null");++ JsonNode taskStats = body(controller.taskStats(METADATA_ID));+ assertEquals(1048576, taskStats.path("docker-id-1").path("memory_stats").path("usage").asLong(),+ "task stats are keyed by Docker id, each carrying that container's sample");+ }++ @Test+ void statsOmitTheRatesWhenOnlyOneSampleArrived() throws JsonProcessingException {+ when(containerManager.sampleContainerStats("docker-id-1")).thenReturn(Optional.of(+ new EcsContainerManager.ContainerStats(sample(), null, null)));++ JsonNode stats = body(controller.containerStats(METADATA_ID));++ assertEquals(1048576, stats.path("memory_stats").path("usage").asLong());+ assertTrue(stats.path("network_rate_stats").isMissingNode(),+ "a rate that could not be measured is left out rather than reported as zero");+ }++ @Test+ void statsAnswerWithAnEmptyDocumentWhenTheDaemonHasNoSample() {+ JsonNode stats = body(controller.containerStats(METADATA_ID));+ assertTrue(stats.isObject() && stats.isEmpty(), "an unsampled container reports {}");++ JsonNode taskStats = body(controller.taskStats(METADATA_ID));+ assertTrue(taskStats.path("docker-id-1").isObject() && taskStats.path("docker-id-1").isEmpty(),+ "every container in the task keeps an entry, empty rather than null");+ }++ /** A trimmed Docker stats response, read through docker-java's model the way the daemon's is. */+ private static Statistics sample() throws JsonProcessingException {+ String json = """+ {+ "read": "2026-01-01T10:05:00.000000000Z",+ "preread": "2026-01-01T10:04:59.000000000Z",+ "num_procs": 3,+ "cpu_stats": {"cpu_usage": {"total_usage": 42000000}, "system_cpu_usage": 90000000},+ "precpu_stats": {"cpu_usage": {"total_usage": 41000000}, "system_cpu_usage": 89000000},+ "memory_stats": {"usage": 1048576, "limit": 268435456},+ "pids_stats": {"current": 3},+ "networks": {"eth0": {"rx_bytes": 84, "tx_bytes": 120}}+ }+ """;+ return JsonMapper.builder()+ .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)+ .build()+ .readValue(json, Statistics.class);+ }++ @Test+ void anUnknownIdIsNotFound() {+ Response response = controller.container("not-a-metadata-id");+ assertEquals(404, response.getStatus());+ assertTrue(body(response).path("error").asText().contains("Unable to get metadata"));+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataIntegrationTest.javanew file mode 100644index 0000000000..a58b7ba1cc--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataIntegrationTest.java@@ -0,0 +1,172 @@+package io.github.hectorvent.floci.services.ecs;++import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;+import io.github.hectorvent.floci.services.ecs.model.Container;+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;+import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.LaunchType;+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;+import io.quarkus.test.InjectMock;+import io.quarkus.test.junit.QuarkusTest;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.util.List;+import java.util.Map;+import java.util.Optional;++import static io.restassured.RestAssured.given;+import static org.hamcrest.Matchers.anEmptyMap;+import static org.hamcrest.Matchers.containsString;+import static org.hamcrest.Matchers.equalTo;+import static org.mockito.ArgumentMatchers.anyList;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.Mockito.when;++/**+ * The task metadata endpoint over the wire. Floci serves it at the root, next to S3's+ * {@code /{bucket}/{key}} catch-all, so route registration and precedence are what these cover:+ * a miss would hand a workload S3's XML instead of the document it read the endpoint for. The+ * response shapes themselves are pinned down in {@link EcsTaskMetadataControllerTest}.+ */+@QuarkusTest+class EcsTaskMetadataIntegrationTest {++ private static final String METADATA_ID = "4c1b0a9e8d7f4a3b9c2d1e0f5a4b3c2d";+ private static final String DOCKER_ID = "0f9a8b7c6d5e";+ private static final String TASK_ARN =+ "arn:aws:ecs:us-east-1:000000000000:task/metadata-cluster/route123";++ @InjectMock+ EcsService ecsService;++ @InjectMock+ EcsContainerManager containerManager;++ private EcsTask task;++ @BeforeEach+ void resolveOneMetadataId() {+ task = task();+ when(ecsService.findByMetadataId(anyString())).thenAnswer(invocation ->+ METADATA_ID.equals(invocation.getArgument(0))+ ? Optional.of(new EcsService.MetadataTarget(task,+ task.getContainers().getFirst(), taskDefinition()))+ : Optional.empty());+ when(containerManager.sampleContainerStats(anyString())).thenReturn(Optional.empty());+ when(containerManager.sampleContainerStats(anyList())).thenReturn(Map.of());+ }++ @Test+ void containerDocumentIsServedAtTheRoot() {+ given()+ .when()+ .get("/v4/" + METADATA_ID)+ .then()+ .statusCode(200)+ .contentType(containsString("json"))+ .body("DockerId", equalTo(DOCKER_ID))+ .body("Name", equalTo("app"))+ .body("Labels.'com.amazonaws.ecs.task-arn'", equalTo(TASK_ARN));+ }++ @Test+ void taskDocumentIsServedUnderTheContainerPath() {+ given()+ .when()+ .get("/v4/" + METADATA_ID + "/task")+ .then()+ .statusCode(200)+ .contentType(containsString("json"))+ .body("TaskARN", equalTo(TASK_ARN))+ .body("Family", equalTo("web"))+ .body("Revision", equalTo("3"))+ .body("Containers[0].Name", equalTo("app"));+ }++ @Test+ void taskWithTagsIsRoutedAlongsideTheTaskPath() {+ when(ecsService.listTagsForResource(TASK_ARN)).thenReturn(Map.of("owner", "platform"));+ // taskWithTags is the container agent's path, so only an EC2 task has it.+ task.setLaunchType(LaunchType.EC2);++ given()+ .when()+ .get("/v4/" + METADATA_ID + "/taskWithTags")+ .then()+ .statusCode(200)+ .contentType(containsString("json"))+ .body("TaskARN", equalTo(TASK_ARN))+ .body("TaskTags.owner", equalTo("platform"));+ }++ @Test+ void bothStatsPathsAreRoutedAndAnswerWithJson() {+ given()+ .when()+ .get("/v4/" + METADATA_ID + "/stats")+ .then()+ .statusCode(200)+ .contentType(containsString("json"))+ .body("$", anEmptyMap());++ given()+ .when()+ .get("/v4/" + METADATA_ID + "/task/stats")+ .then()+ .statusCode(200)+ .contentType(containsString("json"))+ .body(DOCKER_ID, anEmptyMap());+ }++ /**+ * An unknown id must reach the endpoint and get its JSON error, not fall through to S3 and+ * come back as {@code NoSuchBucket} XML for a bucket named {@code v4}.+ */+ @Test+ void anUnknownIdAnswersWithTheEndpointsOwnNotFound() {+ given()+ .when()+ .get("/v4/not-a-metadata-id")+ .then()+ .statusCode(404)+ .contentType(containsString("json"))+ .body("error", containsString("Unable to get metadata"));+ }++ private static EcsTask task() {+ EcsTask task = new EcsTask();+ task.setTaskArn(TASK_ARN);+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/metadata-cluster");+ task.setTaskDefinitionArn("arn:aws:ecs:us-east-1:000000000000:task-definition/web:3");+ task.setLastStatus("RUNNING");+ task.setDesiredStatus("RUNNING");+ task.setLaunchType(LaunchType.FARGATE);+ task.setCpu("256");+ task.setMemory("512");++ Container container = new Container();+ container.setName("app");+ container.setImage("nginx:latest");+ container.setLastStatus("RUNNING");+ container.setDockerId(DOCKER_ID);+ container.setMetadataId(METADATA_ID);+ container.setContainerArn("arn:aws:ecs:us-east-1:000000000000:container/route123/app");+ task.setContainers(List.of(container));+ return task;+ }++ private static TaskDefinition taskDefinition() {+ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage("nginx:latest");++ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("web");+ taskDef.setRevision(3);+ taskDef.setNetworkMode(NetworkMode.awsvpc);+ taskDef.setContainerDefinitions(List.of(app));+ return taskDef;+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.javaindex 78692d89ac..b2d4610f6a 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java@@ -24,7 +24,9 @@ import org.mockito.ArgumentCaptor;import java.util.List;import java.util.Map;+import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertNotNull;import static org.junit.jupiter.api.Assertions.assertTrue;import static org.mockito.ArgumentMatchers.any;import static org.mockito.ArgumentMatchers.anyString;@@ -149,6 +151,36 @@ class EcsContainerManagerAwsBaselineTest {"overridden baseline endpoint should not remain");}+ @Test+ void injectsTheTaskMetadataEndpointEachContainerCanReach() {+ when(awsEnv.flociEndpoint()).thenReturn("http://host.docker.internal:4566");++ ContainerDefinition app = containerDef("app", "app:latest", List.of());+ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("test-family");+ taskDef.setContainerDefinitions(List.of(app));++ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");++ manager.startTask(task, taskDef, null, "us-east-1");++ @SuppressWarnings("unchecked")+ ArgumentCaptor<List<String>> envCaptor = ArgumentCaptor.forClass(List.class);+ verify(builder).withEnv(envCaptor.capture());+ String metadataUri = envCaptor.getValue().stream()+ .filter(entry -> entry.startsWith("ECS_CONTAINER_METADATA_URI_V4="))+ .findFirst()+ .orElse(null);++ assertNotNull(metadataUri, "every launched container is told where its metadata lives");+ assertTrue(metadataUri.startsWith("ECS_CONTAINER_METADATA_URI_V4=http://host.docker.internal:4566/v4/"),+ "the metadata URI must point at Floci's own v4 path: " + metadataUri);+ // The id in the URI is the one the container is then found by.+ String id = metadataUri.substring(metadataUri.lastIndexOf('/') + 1);+ assertEquals(id, task.getContainers().getFirst().getMetadataId());+ }+private static ContainerDefinition containerDef(String name, String image, List<KeyValuePair> env) {ContainerDefinition def = new ContainerDefinition();def.setName(name);diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.javaindex 1f8fcbe9f0..c5a74b3e65 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java@@ -120,8 +120,10 @@ class EcsContainerManagerSecurityGroupTest {verify(builder, never()).withPortBinding(anyInt(), anyInt());verify(builder, never()).withDynamicPort(anyInt());verify(builder, never()).withExposedPort(anyInt());+ // Port bindings come off the helper, whose namespace holds them; the workload container+ // shares that namespace and publishes none, and is inspected only for its launch clock.verify(dockerClient).inspectContainerCmd("helper-id");- verify(dockerClient, never()).inspectContainerCmd("docker-id");+ verify(dockerClient, times(1)).inspectContainerCmd("docker-id");verify(firewallManager).register(any(), eq("helper-id"), any());assertEquals("eni-1", handle.getNetworkInterfaceId());assertEquals("us-east-1", handle.getRegion());diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsDockerIntegrationTest.javanew file mode 100644index 0000000000..7816330294--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsDockerIntegrationTest.java@@ -0,0 +1,133 @@+package io.github.hectorvent.floci.services.ecs.container;++import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.model.Statistics;+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;+import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;+import io.quarkus.test.junit.QuarkusTest;+import jakarta.inject.Inject;+import org.junit.jupiter.api.Assumptions;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.UUID;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertNotNull;+import static org.junit.jupiter.api.Assertions.assertTrue;++/**+ * The sample behind the task metadata endpoint's {@code /stats} paths, taken from a real daemon.+ * AWS returns Docker's own stats document there, so what matters is that a sample arrives at all+ * and carries the counters a client reads; the serialized member names are pinned down in+ * {@code EcsTaskMetadataControllerTest}.+ */+@QuarkusTest+class EcsContainerManagerStatsDockerIntegrationTest {++ private static final String BUSYBOX_IMAGE = "public.ecr.aws/docker/library/busybox:latest";++ @Inject+ EcsContainerManager containerManager;++ @Inject+ DockerClient dockerClient;++ @BeforeEach+ void requireDocker() {+ Assumptions.assumeTrue(isDockerAvailable(),+ "Docker daemon must be available for ECS stats integration tests");+ }++ @Test+ void aRunningContainerReportsTheDaemonsStats() {+ String suffix = UUID.randomUUID().toString().substring(0, 8);+ TaskDefinition taskDefinition = new TaskDefinition();+ taskDefinition.setFamily("stats-docker-" + suffix);+ taskDefinition.setContainerDefinitions(List.of(sleeper("app")));++ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/stats/" + suffix);++ EcsTaskHandle handle = containerManager.startTask(task, taskDefinition, List.of(), "us-east-1");+ try {+ Optional<EcsContainerManager.ContainerStats> sample =+ containerManager.sampleContainerStats(handle.getContainerIds().get("app"));++ assertTrue(sample.isPresent(), "a running container must have stats to sample");+ Statistics statistics = sample.get().statistics();+ assertNotNull(statistics.getRead(), "the sample carries the instant Docker read it");+ assertNotNull(statistics.getCpuStats(), "the sample carries the container's CPU stats");+ assertNotNull(statistics.getMemoryStats(), "the sample carries the container's memory stats");+ assertNotNull(statistics.getPreCpuStats(),+ "the second sample carries the previous CPU reading a client computes against");+ // A container on a Docker network reports interfaces, so the rate delta is measurable.+ assertNotNull(sample.get().rxBytesPerSecond(), "two samples make the receive rate measurable");+ assertNotNull(sample.get().txBytesPerSecond(), "two samples make the transmit rate measurable");+ } finally {+ containerManager.stopTask(handle);+ }+ }++ /**+ * The daemon holds a stats stream open per container, so a task's containers are sampled in one+ * pass with every stream open at once. That the real daemon serves them alongside each other is+ * what this covers; the cost of the pass is pinned down in {@code EcsContainerManagerStatsSamplingTest}.+ */+ @Test+ void aTasksContainersAreAllSampledInOnePass() {+ String suffix = UUID.randomUUID().toString().substring(0, 8);+ TaskDefinition taskDefinition = new TaskDefinition();+ taskDefinition.setFamily("stats-pass-" + suffix);+ taskDefinition.setContainerDefinitions(List.of(+ sleeper("app"), sleeper("sidecar"), sleeper("logger")));++ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/stats/" + suffix);++ EcsTaskHandle handle = containerManager.startTask(task, taskDefinition, List.of(), "us-east-1");+ try {+ List<String> dockerIds = List.of(handle.getContainerIds().get("app"),+ handle.getContainerIds().get("sidecar"),+ handle.getContainerIds().get("logger"));++ Map<String, EcsContainerManager.ContainerStats> sampled =+ containerManager.sampleContainerStats(dockerIds);++ assertEquals(dockerIds.size(), sampled.size(), "every container in the task is sampled");+ for (String dockerId : dockerIds) {+ assertNotNull(sampled.get(dockerId).statistics().getRead(),+ "each sample carries the instant Docker read it");+ }+ } finally {+ containerManager.stopTask(handle);+ }+ }++ @Test+ void aContainerThatIsGoneReportsNoSampleRatherThanFailing() {+ assertTrue(containerManager.sampleContainerStats("floci-no-such-container").isEmpty());+ assertTrue(containerManager.sampleContainerStats((String) null).isEmpty());+ }++ private static ContainerDefinition sleeper(String name) {+ ContainerDefinition definition = new ContainerDefinition();+ definition.setName(name);+ definition.setImage(BUSYBOX_IMAGE);+ definition.setCommand(List.of("sleep", "60"));+ return definition;+ }++ private boolean isDockerAvailable() {+ try {+ dockerClient.pingCmd().exec();+ return true;+ } catch (Exception e) {+ return false;+ }+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsSamplingTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsSamplingTest.javanew file mode 100644index 0000000000..7ed83f9237--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsSamplingTest.java@@ -0,0 +1,185 @@+package io.github.hectorvent.floci.services.ecs.container;++import com.fasterxml.jackson.core.JsonProcessingException;+import com.fasterxml.jackson.databind.DeserializationFeature;+import com.fasterxml.jackson.databind.json.JsonMapper;+import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.async.ResultCallback;+import com.github.dockerjava.api.command.StatsCmd;+import com.github.dockerjava.api.model.Statistics;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import io.github.hectorvent.floci.core.common.docker.ContainerDetector;+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;+import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;+import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.ssm.SsmService;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.Timeout;++import java.io.IOException;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.CountDownLatch;+import java.util.concurrent.TimeUnit;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertNotNull;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++/**+ * How the task metadata endpoint's {@code /stats} paths sample the daemon. Docker only sends its+ * second stats sample on the next collection tick, so a task's containers have to be sampled+ * alongside each other: {@code /task/stats} is the path a sidecar polls, and sampling one+ * container after another would cost it a tick per container.+ *+ * <p>The daemon is mocked. A stream sends its first sample as soon as it is opened and holds its+ * second one back until every stream the pass opens is open, which is what a serial implementation+ * could never satisfy.+ */+class EcsContainerManagerStatsSamplingTest {++ private static final String FIRST_READ = "2026-01-01T10:05:00.000000000Z";+ private static final String SECOND_READ = "2026-01-01T10:05:01.000000000Z";++ private final Map<String, StatsCmd> commands = new ConcurrentHashMap<>();+ private EcsContainerManager manager;++ /**+ * Trips once every stream a pass is expected to open has been opened. Only the pass that is+ * about the ordering raises it: the other tests leave it open so their samples flow at once.+ */+ private CountDownLatch allStreamsOpen;++ @BeforeEach+ void setUp() {+ allStreamsOpen = new CountDownLatch(0);++ DockerClient dockerClient = mock(DockerClient.class);+ when(dockerClient.statsCmd(anyString())).thenAnswer(invocation ->+ commands.computeIfAbsent(invocation.getArgument(0), id -> tickingStatsCmd()));++ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);++ manager = new EcsContainerManager(mock(ContainerBuilder.class), lifecycleManager,+ mock(ContainerLogStreamer.class), mock(ContainerDetector.class),+ mock(EmulatorConfig.class, RETURNS_DEEP_STUBS), mock(RegionResolver.class),+ mock(LaunchedContainerAwsEnv.class), mock(SsmService.class),+ mock(SecretsManagerService.class), mock(S3Service.class),+ mock(EcrRegistryManager.class), mock(HostVolumePolicy.class));+ }++ @Test+ @Timeout(10)+ void aTasksContainersAreSampledAlongsideEachOtherRatherThanOneAfterAnother() {+ List<String> dockerIds = List.of("docker-1", "docker-2", "docker-3", "docker-4");+ // No stream sends its second sample until all four are open, so a pass that waits on one+ // stream before opening the next cannot collect a pair for any of them.+ allStreamsOpen = new CountDownLatch(dockerIds.size());++ Map<String, EcsContainerManager.ContainerStats> sampled = manager.sampleContainerStats(dockerIds);++ assertEquals(dockerIds.size(), sampled.size(), "every container in the task reports a sample");+ for (String dockerId : dockerIds) {+ EcsContainerManager.ContainerStats stats = sampled.get(dockerId);+ assertEquals(SECOND_READ, stats.statistics().getRead(), "the later sample is the one reported");+ assertNotNull(stats.rxBytesPerSecond(), "a pair arrived for " + dockerId + ", so it has rates");+ assertEquals(100.0, stats.rxBytesPerSecond(), 0.0001, "the rate is the delta across the pair");+ assertEquals(200.0, stats.txBytesPerSecond(), 0.0001);+ }+ }++ @Test+ void everyStreamIsClosedOnceThePassIsDone() throws IOException {+ manager.sampleContainerStats(List.of("docker-1", "docker-2"));++ for (StatsCmd command : commands.values()) {+ verify(command).close();+ }+ }++ @Test+ void aContainerTheDaemonRefusesIsLeftOutWithoutFailingTheRest() {+ commands.put("docker-gone", failingStatsCmd());++ Map<String, EcsContainerManager.ContainerStats> sampled =+ manager.sampleContainerStats(List.of("docker-gone", "docker-1"));++ assertFalse(sampled.containsKey("docker-gone"), "a container with no stream reports no sample");+ assertTrue(sampled.containsKey("docker-1"), "the containers alongside it are still sampled");+ }++ @Test+ void oneContainerIsSampledThroughTheSamePass() {+ Optional<EcsContainerManager.ContainerStats> stats = manager.sampleContainerStats("docker-1");++ assertTrue(stats.isPresent());+ assertEquals(SECOND_READ, stats.get().statistics().getRead());+ assertTrue(manager.sampleContainerStats((String) null).isEmpty(), "no id means no sample");+ }++ /**+ * A stats stream that sends its first sample as soon as it is opened and its second one only+ * once every stream of the pass is open, standing in for Docker's collection tick.+ */+ private StatsCmd tickingStatsCmd() {+ StatsCmd command = mock(StatsCmd.class);+ when(command.exec(any())).thenAnswer(invocation -> {+ ResultCallback<Statistics> callback = invocation.getArgument(0);+ allStreamsOpen.countDown();+ Thread emitter = new Thread(() -> {+ try {+ callback.onNext(sample(FIRST_READ, 100));+ if (allStreamsOpen.await(5, TimeUnit.SECONDS)) {+ callback.onNext(sample(SECOND_READ, 200));+ }+ } catch (InterruptedException e) {+ Thread.currentThread().interrupt();+ } catch (JsonProcessingException e) {+ throw new IllegalStateException("the fixed stats sample must parse", e);+ }+ });+ emitter.setDaemon(true);+ emitter.start();+ return callback;+ });+ return command;+ }++ /** A container the daemon has already forgotten: opening its stream throws. */+ private static StatsCmd failingStatsCmd() {+ StatsCmd command = mock(StatsCmd.class);+ when(command.exec(any())).thenThrow(new IllegalStateException("no such container"));+ return command;+ }++ private static Statistics sample(String read, long rxBytes) throws JsonProcessingException {+ String json = """+ {+ "read": "%s",+ "cpu_stats": {"cpu_usage": {"total_usage": 42000000}},+ "memory_stats": {"usage": 1048576},+ "networks": {"eth0": {"rx_bytes": %d, "tx_bytes": %d}}+ }+ """.formatted(read, rxBytes, rxBytes * 2);+ return JsonMapper.builder()+ .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)+ .build()+ .readValue(json, Statistics.class);+ }+}