Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(ecs): serve the task metadata endpoint v4 (#4120)

Abhigya7cd441231f54
diff --git a/docs/services/ecs.md b/docs/services/ecs.md
index 23ef13b771..b15041d892 100644
--- a/docs/services/ecs.md
+++ b/docs/services/ecs.md
@@ -166,7 +166,7 @@ object ARN with `Invalid arn syntax`. A Fargate task can still take its config f
documents, by giving the aws-for-fluent-bit init process its `aws_fluent_bit_init_s3_*`
environment variables. ECS never inspects those and Floci passes them through, so that
registration is accepted here too; the init process reads the task metadata endpoint before
-downloading.
+downloading, and Floci serves one (see [Task metadata endpoint](#task-metadata-endpoint)).
Floci does not validate a task definition's `compatibilities` /
`requiresCompatibilities` against `RunTask` `launchType`; a Fargate-compatible
definition can still be run with `launchType=EC2` (and the reverse).
@@ -297,6 +297,54 @@ launch type, which is what AWS documents for the `Service` shape.
With no launch type, no strategy and no cluster default, a task keeps Floci's `FARGATE` default: a
local cluster has no container instances, so an EC2 default would have nowhere to place it.
+#### Task metadata endpoint
+
+Floci serves the [task metadata endpoint version
+4](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4-fargate.html)
+and injects `ECS_CONTAINER_METADATA_URI_V4` into every container it launches. AWS serves it on the
+link-local address `169.254.170.2`, which a local container cannot be given, so Floci serves the
+same paths on its own port; applications and the AWS SDKs read the environment variable rather than
+the address, so they work unchanged.
+
+| Path | Returns |
+|---|---|
+| `/v4/{id}` | The container's metadata |
+| `/v4/{id}/task` | The task's metadata, including every container |
+| `/v4/{id}/taskWithTags` | The task's metadata with its tags and its container instance's (EC2 only) |
+| `/v4/{id}/stats` | The container's Docker stats |
+| `/v4/{id}/task/stats` | The Docker stats of every container in the task, keyed by Docker id |
+
+The `{id}` is minted per container at launch, as the ECS agent mints it. The task document carries
+`Cluster`, `TaskARN`, `Family`, `Revision`, the statuses, `Limits` (CPU in vCPUs), the pull
+timestamps, `AvailabilityZone`, `LaunchType`, `ServiceName` for a service's task, `VPCID` for an
+EC2 task, and, for Fargate, `ClockDrift` and `EphemeralStorageMetrics`. Floci has no clock drift to
+report and does not meter the disk, so those two report a synchronized clock and zero usage.
+
+An `awsvpc` container's `Networks` object describes the task ENI and the subnet it sits in:
+`NetworkMode`, `IPv4Addresses`, `AttachmentIndex`, `MACAddress`, `PrivateDNSName`,
+`IPv4SubnetCIDRBlock`, `SubnetGatewayIpv4Address`, `DomainNameServers` and `DomainNameSearchList`.
+A local ENI has no DHCP option set behind it, so the gateway is derived as the first address of the
+subnet's CIDR, the resolver as the third address of the VPC's, and the search domain from the
+region, the way a real VPC assigns them.
+
+The stats paths read the Docker daemon when the request arrives, so they return the same
+[ContainerStats](https://docs.docker.com/engine/api/v1.30/#operation/ContainerStats) document AWS
+returns, plus the `network_rate_stats` the ECS agent adds. Docker reports cumulative counters only,
+so the rates are taken across two consecutive samples: as on Fargate, that means a container has to
+have run for about a second before its stats are there. A container Floci has no running Docker
+container for reports an empty document instead, and one sampled only once reports its stats
+without `network_rate_stats`: the path exists for every container in the task, whether or not the
+daemon can measure it.
+
+`/v4/{id}/task/stats` samples the task's containers in one pass, with every stream open at once,
+rather than one after another. A sidecar polls that path for network metrics, so the response costs
+about the single collection tick one container costs however many containers the task has.
+
+`taskWithTags` is the container agent's path, so it answers for an EC2 task and 404s for a Fargate
+one, as on AWS. A container's `CreatedAt`, `StartedAt` and `FinishedAt` are Docker's own for that
+container, read when it starts and again when it stops; a task that never reached a daemon, which
+means `mock: true` or one restored from storage, reports the task's timestamps instead.
+
### Services
| Operation | Description |
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java
index 925821e029..3fa51b06fa 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java
@@ -75,6 +75,7 @@ import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
+import java.util.Optional;
import java.util.stream.Collectors;
import java.util.Set;
import java.util.Comparator;
@@ -1657,6 +1658,7 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
deregisterTaskFromLoadBalancers(task, region);
Map<String, Integer> exitCodes = Map.of();
+ Map<String, Instant> finishedAt = Map.of();
if (dockerMode) {
EcsTaskHandle handle = taskHandles.remove(task.getTaskArn());
try {
@@ -1664,6 +1666,9 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
} finally {
retainUnresolvedLogHandle(task.getTaskArn(), handle);
}
+ if (handle != null) {
+ finishedAt = handle.getFinishedAt();
+ }
}
task.setLastStatus(TaskStatus.STOPPED.name());
@@ -1673,12 +1678,17 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
containerManager.releaseTaskNetwork(task, region);
if (task.getContainers() != null) {
final Map<String, Integer> codes = exitCodes;
+ final Map<String, Instant> finishTimes = finishedAt;
task.getContainers().forEach(c -> {
c.setLastStatus("STOPPED");
Integer code = codes.get(c.getName());
if (code != null) {
c.setExitCode(code);
}
+ Instant finished = finishTimes.get(c.getName());
+ if (finished != null) {
+ c.setFinishedAt(finished);
+ }
});
}
@@ -1736,6 +1746,42 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
}
}
+ /** A container reached through the task metadata endpoint, with the task it belongs to. */
+ public record MetadataTarget(EcsTask task, Container container, TaskDefinition taskDefinition) {}
+
+ /**
+ * Resolves the {@code ECS_CONTAINER_METADATA_URI_V4} id a container was given at launch. The id
+ * is minted per container and never reused, so it identifies both the container and its task.
+ */
+ public Optional<MetadataTarget> findByMetadataId(String metadataId) {
+ if (metadataId == null || metadataId.isBlank()) {
+ return Optional.empty();
+ }
+ for (EcsTask task : tasks.values()) {
+ if (task.getContainers() == null) {
+ continue;
+ }
+ for (Container container : task.getContainers()) {
+ if (metadataId.equals(container.getMetadataId())) {
+ return Optional.of(new MetadataTarget(task, container,
+ taskDefinitionOf(task)));
+ }
+ }
+ }
+ return Optional.empty();
+ }
+
+ /** The task's definition, or null when it has since been deleted. */
+ private TaskDefinition taskDefinitionOf(EcsTask task) {
+ try {
+ return resolveTaskDefinitionOrThrow(task.getTaskDefinitionArn(), taskRegion(task));
+ } catch (AwsException e) {
+ LOG.debugv("Task {0} references a task definition that is gone: {1}",
+ task.getTaskArn(), e.getMessage());
+ return null;
+ }
+ }
+
public List<EcsTask> describeTasks(String clusterRef, List<String> taskRefs, String region) {
return describeTasksDetailed(clusterRef, taskRefs, region).tasks();
}
@@ -3772,14 +3818,20 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
updateHealth(task, handle);
- // Inspect every container; abort if any are still running.
+ // Inspect every container; abort if any are still running. A container that exited on its
+ // own has a finish time of its own, read here while the daemon still remembers it.
Map<String, Integer> exitCodes = new LinkedHashMap<>();
+ Map<String, Instant> finishedAt = new LinkedHashMap<>();
for (Map.Entry<String, String> entry : handle.getContainerIds().entrySet()) {
Integer code = containerManager.getExitCodeIfStopped(entry.getValue());
if (code == null) {
return;
}
exitCodes.put(entry.getKey(), code);
+ Instant finished = containerManager.getFinishedAtIfStopped(entry.getValue());
+ if (finished != null) {
+ finishedAt.put(entry.getKey(), finished);
+ }
}
// All containers have exited. Atomically claim the handle to avoid
@@ -3799,6 +3851,10 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettab
if (code != null) {
c.setExitCode(code);
}
+ Instant finished = finishedAt.get(c.getName());
+ if (finished != null) {
+ c.setFinishedAt(finished);
+ }
});
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataController.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataController.java
new file mode 100644
index 0000000000..d9a0f5e002
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataController.java
@@ -0,0 +1,548 @@
+package io.github.hectorvent.floci.services.ecs;
+
+import io.github.hectorvent.floci.core.common.AwsArnUtils;
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.ec2.Ec2Service;
+import io.github.hectorvent.floci.services.ec2.model.Subnet;
+import io.github.hectorvent.floci.services.ec2.model.Vpc;
+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;
+import io.github.hectorvent.floci.services.ecs.model.AwsVpcConfiguration;
+import io.github.hectorvent.floci.services.ecs.model.Container;
+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;
+import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.LaunchType;
+import io.github.hectorvent.floci.services.ecs.model.LogConfiguration;
+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.json.JsonMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+import jakarta.ws.rs.GET;
+import jakarta.ws.rs.Path;
+import jakarta.ws.rs.PathParam;
+import jakarta.ws.rs.Produces;
+import jakarta.ws.rs.core.MediaType;
+import jakarta.ws.rs.core.Response;
+
+import org.jboss.logging.Logger;
+
+import java.time.Instant;
+import java.time.format.DateTimeFormatter;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * The ECS task metadata endpoint, version 4.
+ *
+ * <p>On AWS this is served at {@code http://169.254.170.2/v4/<id>} and every container is told
+ * where to find it through {@code ECS_CONTAINER_METADATA_URI_V4}. A local task cannot be given a
+ * link-local address of its own, so Floci serves the same paths on its own port and injects the
+ * matching URI into each container it launches; an application, the AWS SDKs' ECS credential and
+ * metadata clients, and the aws-for-fluent-bit init process all read the variable rather than the
+ * address, so they work unchanged.
+ *
+ * <p>The two {@code /stats} paths return the Docker stats AWS documents them as returning: one
+ * sample per container, taken from the daemon when the request arrives. A container Floci has no
+ * running Docker container for reports an empty document rather than an error, because the path
+ * itself still exists for it.
+ */
+@Path("/v4")
+@ApplicationScoped
+public class EcsTaskMetadataController {
+
+ private static final Logger LOG = Logger.getLogger(EcsTaskMetadataController.class);
+
+ private static final DateTimeFormatter TIMESTAMP = DateTimeFormatter.ISO_INSTANT;
+ private static final String SERVICE_GROUP_PREFIX = "service:";
+ /** The search domain an instance gets, which us-east-1 spells differently from every other region. */
+ private static final String LEGACY_SEARCH_DOMAIN_REGION = "us-east-1";
+
+ /**
+ * Docker's stats document, written back out the way the daemon sent it. docker-java's model
+ * carries a field for every member of every API version, so the members this daemon left out
+ * are dropped rather than written as nulls.
+ */
+ private static final ObjectMapper STATS_MAPPER = JsonMapper.builder()
+ .serializationInclusion(JsonInclude.Include.NON_NULL)
+ .build();
+
+ private final EcsService service;
+ private final EcsContainerManager containerManager;
+ private final Ec2Service ec2Service;
+ private final ObjectMapper objectMapper;
+
+ @Inject
+ public EcsTaskMetadataController(EcsService service, EcsContainerManager containerManager,
+ Ec2Service ec2Service, ObjectMapper objectMapper) {
+ this.service = service;
+ this.containerManager = containerManager;
+ this.ec2Service = ec2Service;
+ this.objectMapper = objectMapper;
+ }
+
+ @GET
+ @Path("/{id}")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response container(@PathParam("id") String id) {
+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);
+ if (target.isEmpty()) {
+ return notFound();
+ }
+ EcsService.MetadataTarget found = target.get();
+ return Response.ok(containerNode(found.task(), found.container(), found.taskDefinition())).build();
+ }
+
+ @GET
+ @Path("/{id}/task")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response task(@PathParam("id") String id) {
+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);
+ if (target.isEmpty()) {
+ return notFound();
+ }
+ EcsService.MetadataTarget found = target.get();
+ return Response.ok(taskNode(found.task(), found.taskDefinition())).build();
+ }
+
+ /**
+ * The task document with the task's own tags and, for a task placed on a container instance,
+ * that instance's. The container agent serves this path, so it exists for the EC2 launch type
+ * only; a Fargate task gets the 404 AWS gives it.
+ */
+ @GET
+ @Path("/{id}/taskWithTags")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response taskWithTags(@PathParam("id") String id) {
+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);
+ if (target.isEmpty() || target.get().task().getLaunchType() != LaunchType.EC2) {
+ return notFound();
+ }
+ EcsService.MetadataTarget found = target.get();
+ ObjectNode node = taskNode(found.task(), found.taskDefinition());
+ ArrayNode errors = objectMapper.createArrayNode();
+ putTags(node, "TaskTags", found.task().getTaskArn(), errors);
+ putTags(node, "ContainerInstanceTags", found.task().getContainerInstanceArn(), errors);
+ if (!errors.isEmpty()) {
+ node.set("Errors", errors);
+ }
+ return Response.ok(node).build();
+ }
+
+ /**
+ * One tag collection, omitted when the resource carries none and reported under {@code Errors}
+ * the way the agent reports a tag lookup it could not make.
+ */
+ private void putTags(ObjectNode target, String field, String resourceArn, ArrayNode errors) {
+ if (resourceArn == null) {
+ return;
+ }
+ try {
+ Map<String, String> tags = service.listTagsForResource(resourceArn);
+ if (!tags.isEmpty()) {
+ ObjectNode node = target.putObject(field);
+ tags.forEach(node::put);
+ }
+ } catch (AwsException e) {
+ ObjectNode error = errors.addObject();
+ error.put("ErrorField", field);
+ error.put("ErrorCode", e.getErrorCode());
+ error.put("ErrorMessage", e.getMessage());
+ error.put("StatusCode", e.getHttpStatus());
+ error.put("ResourceARN", resourceArn);
+ }
+ }
+
+ @GET
+ @Path("/{id}/stats")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response containerStats(@PathParam("id") String id) {
+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);
+ if (target.isEmpty()) {
+ return notFound();
+ }
+ EcsContainerManager.ContainerStats sample = containerManager
+ .sampleContainerStats(target.get().container().getDockerId()).orElse(null);
+ return Response.ok(statsNode(sample)).build();
+ }
+
+ @GET
+ @Path("/{id}/task/stats")
+ @Produces(MediaType.APPLICATION_JSON)
+ public Response taskStats(@PathParam("id") String id) {
+ Optional<EcsService.MetadataTarget> target = service.findByMetadataId(id);
+ if (target.isEmpty()) {
+ return notFound();
+ }
+ List<Container> containers = target.get().task().getContainers();
+ if (containers == null) {
+ containers = List.of();
+ }
+ // The task's containers are sampled in one pass rather than one after another: Docker's
+ // collection tick is a second, and this is the path a sidecar polls for network metrics.
+ Map<String, EcsContainerManager.ContainerStats> sampled = containerManager.sampleContainerStats(
+ containers.stream().map(Container::getDockerId).filter(Objects::nonNull).toList());
+
+ ObjectNode stats = objectMapper.createObjectNode();
+ for (Container container : containers) {
+ stats.set(statsKey(container), statsNode(sampled.get(container.getDockerId())));
+ }
+ return Response.ok(stats).build();
+ }
+
+ /** Docker's stats for one container, or an empty document when the daemon had none to give. */
+ private ObjectNode statsNode(EcsContainerManager.ContainerStats stats) {
+ if (stats == null) {
+ return objectMapper.createObjectNode();
+ }
+ ObjectNode node = STATS_MAPPER.valueToTree(stats.statistics());
+ // Docker reports cumulative counters only. The per-second rates alongside them are the
+ // agent's own, and are what a sidecar reads /task/stats for.
+ if (stats.rxBytesPerSecond() != null && stats.txBytesPerSecond() != null) {
+ ObjectNode rates = node.putObject("network_rate_stats");
+ rates.put("rx_bytes_per_sec", stats.rxBytesPerSecond());
+ rates.put("tx_bytes_per_sec", stats.txBytesPerSecond());
+ }
+ return node;
+ }
+
+ /** Task stats are keyed by Docker id, the way the daemon keys a container. */
+ private static String statsKey(Container container) {
+ return container.getDockerId() != null ? container.getDockerId() : container.getName();
+ }
+
+ private Response notFound() {
+ ObjectNode error = objectMapper.createObjectNode();
+ error.put("error", "Unable to get metadata for the requested id");
+ return Response.status(Response.Status.NOT_FOUND).entity(error).build();
+ }
+
+ private ObjectNode containerNode(EcsTask task, Container container, TaskDefinition taskDef) {
+ ContainerDefinition definition = definitionOf(taskDef, container.getName());
+ ObjectNode n = objectMapper.createObjectNode();
+ n.put("DockerId", container.getDockerId() != null ? container.getDockerId() : container.getMetadataId());
+ n.put("Name", container.getName());
+ n.put("DockerName", container.getName());
+ n.put("Image", container.getImage());
+ if (container.getImageDigest() != null && !container.getImageDigest().isBlank()) {
+ n.put("ImageID", container.getImageDigest());
+ }
+ ArrayNode ports = portsNode(container);
+ if (!ports.isEmpty()) {
+ n.set("Ports", ports);
+ }
+ n.set("Labels", labelsNode(task, taskDef, container));
+ n.put("DesiredStatus", task.getDesiredStatus());
+ n.put("KnownStatus", container.getLastStatus());
+ if (container.getExitCode() != null) {
+ n.put("ExitCode", container.getExitCode());
+ }
+ ObjectNode limits = containerLimitsNode(definition);
+ if (!limits.isEmpty()) {
+ n.set("Limits", limits);
+ }
+ // The container's own clock when Docker gave Floci one, the task's otherwise: a mock-mode
+ // task, or one restored from storage, has containers that never went through a daemon.
+ putTimestamp(n, "CreatedAt", firstOf(container.getCreatedAt(), task.getCreatedAt()));
+ putTimestamp(n, "StartedAt", firstOf(container.getStartedAt(), task.getStartedAt()));
+ putTimestamp(n, "FinishedAt", firstOf(container.getFinishedAt(), task.getStoppedAt()));
+ n.put("Type", "NORMAL");
+ n.put("ContainerARN", container.getContainerArn());
+ if (container.getHealthStatus() != null) {
+ n.putObject("Health").put("status", container.getHealthStatus());
+ }
+ if (definition != null && definition.getLogConfiguration() != null) {
+ LogConfiguration logConfiguration = definition.getLogConfiguration();
+ n.put("LogDriver", logConfiguration.logDriver());
+ if (logConfiguration.options() != null) {
+ ObjectNode options = n.putObject("LogOptions");
+ logConfiguration.options().forEach(options::put);
+ }
+ }
+ n.set("Networks", networksNode(task, taskDef));
+ if (task.getPlatformVersion() != null) {
+ n.put("Snapshotter", "overlayfs");
+ }
+ return n;
+ }
+
+ private ArrayNode portsNode(Container container) {
+ ArrayNode ports = objectMapper.createArrayNode();
+ if (container.getNetworkBindings() != null) {
+ container.getNetworkBindings().forEach(binding -> {
+ ObjectNode port = ports.addObject();
+ port.put("ContainerPort", binding.containerPort());
+ port.put("Protocol", binding.protocol());
+ if (binding.hostPort() > 0) {
+ port.put("HostPort", binding.hostPort());
+ }
+ });
+ }
+ return ports;
+ }
+
+ private ObjectNode taskNode(EcsTask task, TaskDefinition taskDef) {
+ ObjectNode n = objectMapper.createObjectNode();
+ n.put("Cluster", task.getClusterArn());
+ n.put("TaskARN", task.getTaskArn());
+ if (taskDef != null) {
+ n.put("Family", taskDef.getFamily());
+ n.put("Revision", String.valueOf(taskDef.getRevision()));
+ }
+ n.put("DesiredStatus", task.getDesiredStatus());
+ n.put("KnownStatus", task.getLastStatus());
+ ObjectNode limits = taskLimitsNode(task);
+ if (!limits.isEmpty()) {
+ n.set("Limits", limits);
+ }
+ putTimestamp(n, "PullStartedAt", task.getPullStartedAt());
+ putTimestamp(n, "PullStoppedAt", task.getPullStoppedAt());
+ putTimestamp(n, "ExecutionStoppedAt", task.getExecutionStoppedAt());
+ if (task.getAvailabilityZone() != null) {
+ n.put("AvailabilityZone", task.getAvailabilityZone());
+ }
+ // A service's tasks are grouped as "service:<name>", which is where the endpoint's
+ // ServiceName comes from; a standalone task has no service and reports none.
+ if (task.getGroup() != null && task.getGroup().startsWith(SERVICE_GROUP_PREFIX)) {
+ n.put("ServiceName", task.getGroup().substring(SERVICE_GROUP_PREFIX.length()));
+ }
+ if (task.getLaunchType() != null) {
+ n.put("LaunchType", task.getLaunchType().name());
+ }
+ // VPCID is the container instance's, so AWS reports it for the EC2 launch type only.
+ if (task.getLaunchType() == LaunchType.EC2) {
+ Subnet subnet = taskSubnet(task);
+ if (subnet != null && subnet.getVpcId() != null) {
+ n.put("VPCID", subnet.getVpcId());
+ }
+ }
+ ArrayNode containers = n.putArray("Containers");
+ if (task.getContainers() != null) {
+ task.getContainers().forEach(container ->
+ containers.add(containerNode(task, container, taskDef)));
+ }
+ // Fargate reports the clock's accuracy and the task's ephemeral storage use. Floci has no
+ // drift to report and does not meter the disk, so both are reported as a healthy baseline.
+ if (task.getPlatformVersion() != null) {
+ ObjectNode clockDrift = n.putObject("ClockDrift");
+ clockDrift.put("ClockErrorBound", 0.0);
+ clockDrift.put("ReferenceTimestamp", TIMESTAMP.format(Instant.now()));
+ clockDrift.put("ClockSynchronizationStatus", "SYNCHRONIZED");
+ if (task.getEphemeralStorage() != null) {
+ ObjectNode storage = n.putObject("EphemeralStorageMetrics");
+ storage.put("Utilized", 0);
+ storage.put("Reserved", task.getEphemeralStorage().sizeInGiB() * 1024);
+ }
+ }
+ return n;
+ }
+
+ private ObjectNode labelsNode(EcsTask task, TaskDefinition taskDef, Container container) {
+ ObjectNode labels = objectMapper.createObjectNode();
+ labels.put("com.amazonaws.ecs.cluster", task.getClusterArn());
+ labels.put("com.amazonaws.ecs.container-name", container.getName());
+ labels.put("com.amazonaws.ecs.task-arn", task.getTaskArn());
+ if (taskDef != null) {
+ labels.put("com.amazonaws.ecs.task-definition-family", taskDef.getFamily());
+ labels.put("com.amazonaws.ecs.task-definition-version", String.valueOf(taskDef.getRevision()));
+ }
+ return labels;
+ }
+
+ /** The container's own limits, omitted member by member when the definition sets none. */
+ private ObjectNode containerLimitsNode(ContainerDefinition definition) {
+ ObjectNode limits = objectMapper.createObjectNode();
+ if (definition == null) {
+ return limits;
+ }
+ if (definition.getCpu() != null) {
+ limits.put("CPU", definition.getCpu());
+ }
+ if (definition.getMemory() != null) {
+ limits.put("Memory", definition.getMemory());
+ }
+ return limits;
+ }
+
+ /** The task's own limits: CPU as a vCPU count, memory in MiB, the way the endpoint reports them. */
+ private ObjectNode taskLimitsNode(EcsTask task) {
+ ObjectNode limits = objectMapper.createObjectNode();
+ Integer cpuUnits = parseInteger(task.getCpu());
+ if (cpuUnits != null) {
+ limits.put("CPU", cpuUnits / 1024.0);
+ }
+ Integer memoryMb = parseInteger(task.getMemory());
+ if (memoryMb != null) {
+ limits.put("Memory", memoryMb);
+ }
+ return limits;
+ }
+
+ private ArrayNode networksNode(EcsTask task, TaskDefinition taskDef) {
+ ArrayNode networks = objectMapper.createArrayNode();
+ ObjectNode network = objectMapper.createObjectNode();
+ network.put("NetworkMode", taskDef != null && taskDef.getNetworkMode() != null
+ ? taskDef.getNetworkMode().name() : "bridge");
+ ArrayNode addresses = network.putArray("IPv4Addresses");
+ if (task.getPrivateIpAddress() != null) {
+ addresses.add(task.getPrivateIpAddress());
+ }
+ if (taskSubnetId(task) != null) {
+ network.put("AttachmentIndex", 0);
+ }
+ // The rest of the network object describes the task ENI and the subnet it sits in. Each
+ // member is written only when the interface Floci allocated actually carries it.
+ if (task.getMacAddress() != null) {
+ network.put("MACAddress", task.getMacAddress());
+ }
+ putSubnetMembers(network, task);
+ if (task.getPrivateDnsName() != null) {
+ network.put("PrivateDNSName", task.getPrivateDnsName());
+ }
+ networks.add(network);
+ return networks;
+ }
+
+ /**
+ * The subnet's own view of the network: its CIDR, the gateway VPC reserves as the first
+ * address in it, the resolver at the VPC's third address, and the search domain the region
+ * hands out. All four are derived, since a local ENI has no DHCP option set behind it.
+ */
+ private void putSubnetMembers(ObjectNode network, EcsTask task) {
+ Subnet subnet = taskSubnet(task);
+ if (subnet == null || subnet.getCidrBlock() == null) {
+ return;
+ }
+ network.put("IPv4SubnetCIDRBlock", subnet.getCidrBlock());
+ String gateway = addressWithinCidr(subnet.getCidrBlock(), 1);
+ if (gateway != null) {
+ network.put("SubnetGatewayIpv4Address", gateway + "/" + prefixLengthOf(subnet.getCidrBlock()));
+ }
+ Vpc vpc = vpcOf(task, subnet);
+ String resolver = vpc != null ? addressWithinCidr(vpc.getCidrBlock(), 2) : null;
+ if (resolver != null) {
+ network.putArray("DomainNameServers").add(resolver);
+ }
+ String region = taskRegion(task);
+ if (region != null) {
+ network.putArray("DomainNameSearchList").add(searchDomain(region));
+ }
+ }
+
+ private Subnet taskSubnet(EcsTask task) {
+ String subnetId = taskSubnetId(task);
+ String region = taskRegion(task);
+ if (subnetId == null || region == null) {
+ return null;
+ }
+ try {
+ return ec2Service.describeSubnets(region, List.of(subnetId), Map.of()).stream()
+ .findFirst()
+ .orElse(null);
+ } catch (AwsException e) {
+ LOG.debugv("Task {0} references a subnet that is gone: {1}", task.getTaskArn(), e.getMessage());
+ return null;
+ }
+ }
+
+ private Vpc vpcOf(EcsTask task, Subnet subnet) {
+ String region = taskRegion(task);
+ if (subnet.getVpcId() == null || region == null) {
+ return null;
+ }
+ try {
+ return ec2Service.describeVpcs(region, List.of(subnet.getVpcId()), Map.of()).stream()
+ .findFirst()
+ .orElse(null);
+ } catch (AwsException e) {
+ LOG.debugv("Subnet {0} references a VPC that is gone: {1}",
+ subnet.getSubnetId(), e.getMessage());
+ return null;
+ }
+ }
+
+ private static String taskSubnetId(EcsTask task) {
+ AwsVpcConfiguration awsvpc = task.getNetworkConfiguration() != null
+ ? task.getNetworkConfiguration().getAwsvpcConfiguration() : null;
+ if (awsvpc == null || awsvpc.getSubnets() == null || awsvpc.getSubnets().isEmpty()) {
+ return null;
+ }
+ return awsvpc.getSubnets().getFirst();
+ }
+
+ private static String taskRegion(EcsTask task) {
+ return AwsArnUtils.regionOrDefault(task.getTaskArn(), null);
+ }
+
+ /** The nth address of a CIDR block, or null when the block is not a readable IPv4 one. */
+ private static String addressWithinCidr(String cidr, int offset) {
+ if (cidr == null) {
+ return null;
+ }
+ String[] parts = cidr.split("/");
+ String[] octets = parts[0].split("\\.");
+ if (octets.length != 4) {
+ return null;
+ }
+ try {
+ long address = 0;
+ for (String octet : octets) {
+ int value = Integer.parseInt(octet);
+ if (value < 0 || value > 255) {
+ return null;
+ }
+ address = (address << 8) | value;
+ }
+ address += offset;
+ return "%d.%d.%d.%d".formatted((address >> 24) & 0xFF, (address >> 16) & 0xFF,
+ (address >> 8) & 0xFF, address & 0xFF);
+ } catch (NumberFormatException e) {
+ LOG.debugv("CIDR block {0} is not a readable IPv4 block: {1}", cidr, e.getMessage());
+ return null;
+ }
+ }
+
+ private static String prefixLengthOf(String cidr) {
+ int slash = cidr.indexOf('/');
+ return slash < 0 ? "32" : cidr.substring(slash + 1);
+ }
+
+ private static String searchDomain(String region) {
+ return LEGACY_SEARCH_DOMAIN_REGION.equals(region) ? "ec2.internal" : region + ".compute.internal";
+ }
+
+ private static ContainerDefinition definitionOf(TaskDefinition taskDef, String containerName) {
+ if (taskDef == null || taskDef.getContainerDefinitions() == null) {
+ return null;
+ }
+ return taskDef.getContainerDefinitions().stream()
+ .filter(definition -> containerName.equals(definition.getName()))
+ .findFirst()
+ .orElse(null);
+ }
+
+ private static Integer parseInteger(String value) {
+ if (value == null || value.isBlank()) {
+ return null;
+ }
+ try {
+ return Integer.parseInt(value.trim());
+ } catch (NumberFormatException e) {
+ return null;
+ }
+ }
+
+ private static Instant firstOf(Instant preferred, Instant fallback) {
+ return preferred != null ? preferred : fallback;
+ }
+
+ private static void putTimestamp(ObjectNode target, String field, Instant value) {
+ if (value != null) {
+ target.put(field, TIMESTAMP.format(value));
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java
index c880776c41..d6c923072b 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java
@@ -43,11 +43,15 @@ import io.github.hectorvent.floci.services.s3.model.S3Object;
import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;
import io.github.hectorvent.floci.services.ssm.SsmService;
import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.async.ResultCallback;
import com.github.dockerjava.api.command.InspectContainerResponse;
+import com.github.dockerjava.api.command.StatsCmd;
import com.github.dockerjava.api.exception.NotFoundException;
import com.github.dockerjava.api.model.ContainerNetwork;
import com.github.dockerjava.api.model.ExposedPort;
import com.github.dockerjava.api.model.LogConfig;
+import com.github.dockerjava.api.model.StatisticNetworksConfig;
+import com.github.dockerjava.api.model.Statistics;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
@@ -61,7 +65,9 @@ import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
+import java.time.Duration;
import java.time.Instant;
+import java.time.format.DateTimeParseException;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.HexFormat;
@@ -70,7 +76,11 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
import java.util.stream.Stream;
/**
@@ -92,6 +102,13 @@ public class EcsContainerManager {
private static final long DEPENDENCY_POLL_MILLIS = 200;
/** How long a killed container gets to register as exited before its code is read. */
private static final int KILL_SETTLE_SECONDS = 5;
+ /** How long a stats sample gets before the task metadata endpoint answers without one. */
+ private static final int STATS_SAMPLE_SECONDS = 5;
+ /**
+ * Docker writes {@code 0001-01-01T00:00:00Z} for an event that has not happened. Anything at
+ * or before the Unix epoch is that zero value rather than a time a container reached.
+ */
+ private static final Instant DOCKER_ZERO_TIMESTAMP = Instant.EPOCH;
private final ContainerBuilder containerBuilder;
private final ContainerLifecycleManager lifecycleManager;
@@ -199,8 +216,14 @@ public class EcsContainerManager {
Map<ContainerDefinition, List<String>> envVarsByContainer = new LinkedHashMap<>();
// Resolved before any container is created, so a registry-startup failure can't leak one already started.
Map<ContainerDefinition, String> imagesByContainer = new LinkedHashMap<>();
+ // The task metadata id has to exist before the container does: its own environment carries
+ // the URI, so it cannot be derived from the Docker id the daemon hands back afterwards.
+ Map<String, String> metadataIdsByContainer = new LinkedHashMap<>();
for (ContainerDefinition def : launchOrder) {
- envVarsByContainer.put(def, buildEnvVars(def, overridesByName.get(def.getName()), region));
+ String metadataId = UUID.randomUUID().toString().replace("-", "");
+ metadataIdsByContainer.put(def.getName(), metadataId);
+ envVarsByContainer.put(def, buildEnvVars(def, overridesByName.get(def.getName()), region,
+ metadataId));
imagesByContainer.put(def, ecrRegistryManager.rewriteImageUri(def.getImage()));
}
@@ -410,7 +433,8 @@ public class EcsContainerManager {
protectedNetwork == null ? dockerId : protectedNetwork.namespace().helperId(), def);
// Build ECS container model
- Container container = buildContainer(task.getTaskArn(), def, dockerId, networkBindings, region);
+ Container container = buildContainer(task.getTaskArn(), def, dockerId, networkBindings, region,
+ metadataIdsByContainer.get(def.getName()));
runtimeContainers.add(container);
containerIds.put(def.getName(), dockerId);
@@ -606,6 +630,211 @@ public class EcsContainerManager {
}
}
+ /**
+ * A container's Docker stats as the task metadata endpoint reports them: the latest sample,
+ * plus the network byte rates. The rates are null when they cannot be derived, and Docker
+ * never reports them itself: they are a delta the ECS agent takes across two samples.
+ */
+ public record ContainerStats(Statistics statistics, Double rxBytesPerSecond, Double txBytesPerSecond) {}
+
+ /**
+ * Reads a container's stats from the daemon. Empty when there is nothing to read: the
+ * container has no Docker id, it has already gone, or the daemon is unreachable.
+ *
+ * <p>The stats stream is read rather than a single {@code stream=false} document, because the
+ * network rates are a delta between consecutive samples. Docker sends the first sample at once
+ * and the next on its collection tick a second later, which is the same second AWS documents a
+ * container as having to run before its stats are available.
+ */
+ public Optional<ContainerStats> sampleContainerStats(String dockerId) {
+ if (dockerId == null || dockerId.isBlank()) {
+ return Optional.empty();
+ }
+ return Optional.ofNullable(sampleContainerStats(List.of(dockerId)).get(dockerId));
+ }
+
+ /**
+ * Reads several containers' stats in one pass, keyed by Docker id. A container the daemon has
+ * nothing to say about is absent from the map rather than present with an empty sample.
+ *
+ * <p>Every stream is opened before any of them is waited on, and the waits share one deadline.
+ * Docker's stats stream is asynchronous, so the containers tick alongside each other and the
+ * call costs about the single collection tick one container costs, however many containers the
+ * task has. That matters on {@code /task/stats}, which is the path a sidecar polls: sampling
+ * the containers one after another would cost a tick each.
+ */
+ public Map<String, ContainerStats> sampleContainerStats(List<String> dockerIds) {
+ Map<String, ContainerStats> sampled = new LinkedHashMap<>();
+ if (dockerIds == null || dockerIds.isEmpty()) {
+ return sampled;
+ }
+ Map<String, OpenStatsStream> streams = openStatsStreams(dockerIds);
+ try {
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(STATS_SAMPLE_SECONDS);
+ for (Map.Entry<String, OpenStatsStream> stream : streams.entrySet()) {
+ awaitPair(stream.getKey(), stream.getValue().samples(), deadline);
+ containerStats(stream.getValue().samples().collected())
+ .ifPresent(stats -> sampled.put(stream.getKey(), stats));
+ }
+ } finally {
+ for (OpenStatsStream stream : streams.values()) {
+ closeQuietly(stream.samples());
+ closeQuietly(stream.command());
+ }
+ }
+ return sampled;
+ }
+
+ /** A stats stream the daemon is already filling, and the command that has to be closed with it. */
+ private record OpenStatsStream(StatsCmd command, ConsecutiveStatsSamples samples) {}
+
+ /** Starts every container's stats stream, skipping the ones the daemon will not open. */
+ private Map<String, OpenStatsStream> openStatsStreams(List<String> dockerIds) {
+ Map<String, OpenStatsStream> streams = new LinkedHashMap<>();
+ for (String dockerId : dockerIds) {
+ if (dockerId == null || dockerId.isBlank() || streams.containsKey(dockerId)) {
+ continue;
+ }
+ StatsCmd command = null;
+ try {
+ command = lifecycleManager.getDockerClient().statsCmd(dockerId);
+ ConsecutiveStatsSamples samples = new ConsecutiveStatsSamples();
+ command.exec(samples);
+ streams.put(dockerId, new OpenStatsStream(command, samples));
+ } catch (RuntimeException e) {
+ closeQuietly(command);
+ LOG.debugv("Could not sample stats for container {0}: {1}", dockerId, e.getMessage());
+ }
+ }
+ return streams;
+ }
+
+ /** Waits for one stream's pair of samples, never past the deadline the whole pass shares. */
+ private static void awaitPair(String dockerId, ConsecutiveStatsSamples samples, long deadlineNanos) {
+ long remaining = deadlineNanos - System.nanoTime();
+ try {
+ if (remaining <= 0 || !samples.awaitPair(remaining, TimeUnit.NANOSECONDS)) {
+ LOG.debugv("Docker sent fewer than two stats samples for container {0} within {1}s",
+ dockerId, STATS_SAMPLE_SECONDS);
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.debugv("Interrupted while sampling stats for container {0}", dockerId);
+ }
+ }
+
+ private static void closeQuietly(Closeable closeable) {
+ if (closeable == null) {
+ return;
+ }
+ try {
+ closeable.close();
+ } catch (IOException | RuntimeException e) {
+ LOG.debugv("Could not close a Docker stats stream: {0}", e.getMessage());
+ }
+ }
+
+ /** The latest sample, with rates when two samples a measurable interval apart arrived. */
+ private static Optional<ContainerStats> containerStats(List<Statistics> samples) {
+ if (samples.isEmpty()) {
+ return Optional.empty();
+ }
+ Statistics latest = samples.getLast();
+ if (samples.size() < 2) {
+ return Optional.of(new ContainerStats(latest, null, null));
+ }
+ Statistics earlier = samples.getFirst();
+ double seconds = secondsBetween(earlier.getRead(), latest.getRead());
+ Long rxDelta = byteDelta(earlier, latest, StatisticNetworksConfig::getRxBytes);
+ Long txDelta = byteDelta(earlier, latest, StatisticNetworksConfig::getTxBytes);
+ if (seconds <= 0 || rxDelta == null || txDelta == null) {
+ return Optional.of(new ContainerStats(latest, null, null));
+ }
+ return Optional.of(new ContainerStats(latest, rxDelta / seconds, txDelta / seconds));
+ }
+
+ /** The counter's movement across the two samples, or null when neither reports an interface. */
+ private static Long byteDelta(Statistics earlier, Statistics latest,
+ Function<StatisticNetworksConfig, Long> counter) {
+ Long earlierBytes = totalBytes(earlier, counter);
+ Long latestBytes = totalBytes(latest, counter);
+ if (earlierBytes == null || latestBytes == null) {
+ return null;
+ }
+ return Math.max(0, latestBytes - earlierBytes);
+ }
+
+ private static Long totalBytes(Statistics statistics,
+ Function<StatisticNetworksConfig, Long> counter) {
+ Map<String, StatisticNetworksConfig> networks = statistics.getNetworks();
+ if (networks == null || networks.isEmpty()) {
+ return null;
+ }
+ long total = 0;
+ for (StatisticNetworksConfig network : networks.values()) {
+ Long bytes = counter.apply(network);
+ total += bytes != null ? bytes : 0;
+ }
+ return total;
+ }
+
+ /** The interval between two Docker read timestamps, or 0 when either cannot be read. */
+ private static double secondsBetween(String earlier, String latest) {
+ if (earlier == null || latest == null) {
+ return 0;
+ }
+ try {
+ return Duration.between(Instant.parse(earlier), Instant.parse(latest)).toNanos() / 1_000_000_000.0;
+ } catch (DateTimeParseException e) {
+ LOG.debugv("Docker stats carried an unreadable read timestamp: {0}", e.getMessage());
+ return 0;
+ }
+ }
+
+ /** Docker's stats stream never ends on its own; this takes the first two samples and stops. */
+ private static final class ConsecutiveStatsSamples extends ResultCallback.Adapter<Statistics> {
+
+ private static final int WANTED = 2;
+
+ private final CountDownLatch enough = new CountDownLatch(WANTED);
+ private final List<Statistics> collected = new CopyOnWriteArrayList<>();
+
+ @Override
+ public void onNext(Statistics statistics) {
+ if (collected.size() < WANTED) {
+ collected.add(statistics);
+ enough.countDown();
+ }
+ }
+
+ @Override
+ public void onError(Throwable throwable) {
+ releaseWaiter();
+ super.onError(throwable);
+ }
+
+ @Override
+ public void onComplete() {
+ releaseWaiter();
+ super.onComplete();
+ }
+
+ /** A stream that ends or fails early must not hold the caller until the timeout. */
+ private void releaseWaiter() {
+ while (enough.getCount() > 0) {
+ enough.countDown();
+ }
+ }
+
+ boolean awaitPair(long timeout, TimeUnit unit) throws InterruptedException {
+ return enough.await(timeout, unit);
+ }
+
+ List<Statistics> collected() {
+ return List.copyOf(collected);
+ }
+ }
+
private List<ContainerDefinition> orderForDependencies(List<ContainerDefinition> definitions,
ContainerDefinition firelensRouter) {
Map<String, ContainerDefinition> definitionsByName = new LinkedHashMap<>();
@@ -899,6 +1128,8 @@ public class EcsContainerManager {
String name = entry.getKey();
String dockerId = entry.getValue();
exitCodes.put(name, getExitCodeIfStopped(dockerId));
+ // Read before the removal below, which is the last moment the daemon still knows it.
+ handle.recordFinishedAt(name, getFinishedAtIfStopped(dockerId));
try {
lifecycleManager.getDockerClient().removeContainerCmd(dockerId).withForce(true).exec();
terminatedContainerIds.add(dockerId);
@@ -1231,7 +1462,8 @@ public class EcsContainerManager {
}
}
- private List<String> buildEnvVars(ContainerDefinition def, ContainerOverride override, String region) {
+ private List<String> buildEnvVars(ContainerDefinition def, ContainerOverride override, String region,
+ String metadataId) {
// AWS SDK baseline (endpoint + region + credentials) first so the task can reach the
// emulator, then the task-def environment, then task-def secrets, then the override
// environment. Later entries win on key conflict, so an explicit task-def value or
@@ -1243,6 +1475,12 @@ public class EcsContainerManager {
envMap.put(kv.substring(0, eq), kv.substring(eq + 1));
}
}
+ // The task metadata endpoint, which an application, the ECS SDK integrations and the
+ // aws-for-fluent-bit init process all read from this variable.
+ String flociEndpoint = awsEnv.flociEndpoint();
+ if (metadataId != null && flociEndpoint != null) {
+ envMap.put("ECS_CONTAINER_METADATA_URI_V4", flociEndpoint + "/v4/" + metadataId);
+ }
if (def.getEnvironment() != null) {
for (var kv : def.getEnvironment()) {
envMap.put(kv.name(), kv.value());
@@ -1440,7 +1678,8 @@ public class EcsContainerManager {
}
private Container buildContainer(String taskArn, ContainerDefinition def, String dockerId,
- List<NetworkBinding> networkBindings, String region) {
+ List<NetworkBinding> networkBindings, String region,
+ String metadataId) {
Container container = new Container();
container.setTaskArn(taskArn);
container.setName(def.getName());
@@ -1449,6 +1688,7 @@ public class EcsContainerManager {
container.setNetworkBindings(networkBindings);
container.setDockerId(dockerId);
container.setRuntimeId(dockerId);
+ container.setMetadataId(metadataId);
container.setHealthStatus(def.getHealthCheck() != null ? "UNKNOWN" : null);
if (def.getCpu() != null) {
container.setCpu(String.valueOf(def.getCpu()));
@@ -1461,9 +1701,62 @@ public class EcsContainerManager {
}
container.setContainerArn(regionResolver.buildArn("ecs", region,
"container/" + extractTaskId(taskArn) + "/" + def.getName()));
+ applyLaunchTimestamps(container, dockerId);
return container;
}
+ /**
+ * Stamps the container with Docker's own create and start times. A task's containers start one
+ * at a time, holding for each other's {@code dependsOn} conditions, so these are the container's
+ * and not the task's.
+ */
+ private void applyLaunchTimestamps(Container container, String dockerId) {
+ InspectContainerResponse inspect = inspectOrNull(dockerId);
+ if (inspect == null) {
+ return;
+ }
+ container.setCreatedAt(dockerTimestamp(inspect.getCreated()));
+ if (inspect.getState() != null) {
+ container.setStartedAt(dockerTimestamp(inspect.getState().getStartedAt()));
+ }
+ }
+
+ /**
+ * When a container that has already stopped finished. Read before the container is removed,
+ * since the daemon forgets it afterwards.
+ */
+ public Instant getFinishedAtIfStopped(String dockerId) {
+ InspectContainerResponse inspect = inspectOrNull(dockerId);
+ if (inspect == null || inspect.getState() == null
+ || Boolean.TRUE.equals(inspect.getState().getRunning())) {
+ return null;
+ }
+ return dockerTimestamp(inspect.getState().getFinishedAt());
+ }
+
+ private InspectContainerResponse inspectOrNull(String dockerId) {
+ try {
+ return lifecycleManager.getDockerClient().inspectContainerCmd(dockerId).exec();
+ } catch (Exception e) {
+ LOG.debugv("Could not inspect container {0}: {1}", dockerId, e.getMessage());
+ return null;
+ }
+ }
+
+ /** A Docker RFC 3339 timestamp, or null for its zero value, which means "never happened". */
+ private static Instant dockerTimestamp(String value) {
+ if (value == null || value.isBlank()) {
+ return null;
+ }
+ try {
+ Instant instant = Instant.parse(value);
+ return instant.isAfter(DOCKER_ZERO_TIMESTAMP) ? instant : null;
+ } catch (DateTimeParseException e) {
+ LOG.debugv("Docker reported an unreadable timestamp {0}: {1}", value, e.getMessage());
+ return null;
+ }
+ }
+
private static String extractTaskId(String arn) {
if (arn == null) {
return "";
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java
index deb19ccd0f..5cfee43984 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java
@@ -1,6 +1,7 @@
package io.github.hectorvent.floci.services.ecs.container;
import java.io.Closeable;
+import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.Map;
@@ -28,6 +29,11 @@ public class EcsTaskHandle {
private final String region;
/** Per-container {@code stopTimeout}, so teardown waits as long as the task definition asked. */
private final Map<String, Integer> stopTimeouts;
+ /**
+ * When each container finished, read during teardown. Docker forgets a container once it is
+ * removed, so the time has to be kept here for the caller that stamps the task's containers.
+ */
+ private final Map<String, Instant> finishedAt = new LinkedHashMap<>();
public EcsTaskHandle(String taskArn, Map<String, String> containerIds,
Map<String, Closeable> logStreamsByContainerId) {
@@ -67,6 +73,18 @@ public class EcsTaskHandle {
return configured != null ? configured : DEFAULT_STOP_TIMEOUT_SECONDS;
}
+ /** Notes when a container finished. A null is ignored, so a failed read leaves no entry. */
+ public void recordFinishedAt(String containerName, Instant instant) {
+ if (instant != null) {
+ finishedAt.put(containerName, instant);
+ }
+ }
+
+ /** When each container finished, for the containers teardown could read a time for. */
+ public Map<String, Instant> getFinishedAt() {
+ return finishedAt;
+ }
+
/** Removes and returns the log stream that no longer needs task-level ownership. */
public Closeable removeLogStream(String containerId) {
return logStreamsByContainerId.remove(containerId);
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/Container.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/Container.java
index 1a2e3ce270..fe95486fec 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/model/Container.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/Container.java
@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.ecs.model;
import io.quarkus.runtime.annotations.RegisterForReflection;
+import java.time.Instant;
import java.util.List;
@RegisterForReflection
@@ -19,10 +20,24 @@ public class Container {
private List<ManagedAgent> managedAgents;
private String healthStatus;
private String runtimeId;
+ /**
+ * The opaque id in this container's {@code ECS_CONTAINER_METADATA_URI_V4}. Minted before the
+ * container is created, the way the ECS agent mints one, so the URI can be injected into the
+ * container's own environment.
+ */
+ private String metadataId;
private String imageDigest;
private String cpu;
private String memory;
private String memoryReservation;
+ /**
+ * When Docker created, started and finished this container. The ECS API's own container shape
+ * carries none of these, so they never reach a DescribeTasks response; the task metadata
+ * endpoint reports them, and its containers start and stop one at a time.
+ */
+ private Instant createdAt;
+ private Instant startedAt;
+ private Instant finishedAt;
// transient — not persisted
private transient String dockerId;
@@ -65,6 +80,9 @@ public class Container {
public String getRuntimeId() { return runtimeId; }
public void setRuntimeId(String runtimeId) { this.runtimeId = runtimeId; }
+ public String getMetadataId() { return metadataId; }
+ public void setMetadataId(String metadataId) { this.metadataId = metadataId; }
+
public String getImageDigest() { return imageDigest; }
public void setImageDigest(String imageDigest) { this.imageDigest = imageDigest; }
@@ -77,6 +95,15 @@ public class Container {
public String getMemoryReservation() { return memoryReservation; }
public void setMemoryReservation(String memoryReservation) { this.memoryReservation = memoryReservation; }
+ public Instant getCreatedAt() { return createdAt; }
+ public void setCreatedAt(Instant createdAt) { this.createdAt = createdAt; }
+
+ public Instant getStartedAt() { return startedAt; }
+ public void setStartedAt(Instant startedAt) { this.startedAt = startedAt; }
+
+ public Instant getFinishedAt() { return finishedAt; }
+ public void setFinishedAt(Instant finishedAt) { this.finishedAt = finishedAt; }
+
public String getDockerId() { return dockerId; }
public void setDockerId(String dockerId) { this.dockerId = dockerId; }
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataControllerTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataControllerTest.java
new file mode 100644
index 0000000000..b0dd1ee62e
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataControllerTest.java
@@ -0,0 +1,397 @@
+package io.github.hectorvent.floci.services.ecs;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.ec2.Ec2Service;
+import io.github.hectorvent.floci.services.ec2.model.Subnet;
+import io.github.hectorvent.floci.services.ec2.model.Vpc;
+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;
+import io.github.hectorvent.floci.services.ecs.model.AwsVpcConfiguration;
+import io.github.hectorvent.floci.services.ecs.model.Container;
+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;
+import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.EphemeralStorage;
+import io.github.hectorvent.floci.services.ecs.model.LaunchType;
+import io.github.hectorvent.floci.services.ecs.model.LogConfiguration;
+import io.github.hectorvent.floci.services.ecs.model.NetworkConfiguration;
+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;
+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.json.JsonMapper;
+import com.github.dockerjava.api.model.Statistics;
+import jakarta.ws.rs.core.Response;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.time.Instant;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.anyList;
+import static org.mockito.ArgumentMatchers.anyMap;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * The task metadata endpoint's response shapes. A workload reads these through
+ * {@code ECS_CONTAINER_METADATA_URI_V4} expecting AWS's exact PascalCase members, so the mapping
+ * from Floci's task model onto them is what this pins down.
+ */
+class EcsTaskMetadataControllerTest {
+
+ private static final String METADATA_ID = "9e2b1f0c4d5e4a1b8c7d6e5f4a3b2c1d";
+ private static final String TASK_ARN =
+ "arn:aws:ecs:us-east-1:000000000000:task/metadata-cluster/abc123";
+
+ private ObjectMapper objectMapper;
+ private EcsService service;
+ private EcsContainerManager containerManager;
+ private Ec2Service ec2Service;
+ private EcsTask task;
+ private EcsTaskMetadataController controller;
+
+ @BeforeEach
+ void setUp() {
+ objectMapper = new ObjectMapper();
+ service = mock(EcsService.class);
+ containerManager = mock(EcsContainerManager.class);
+ ec2Service = mock(Ec2Service.class);
+ when(containerManager.sampleContainerStats(anyString())).thenReturn(Optional.empty());
+ when(containerManager.sampleContainerStats(anyList())).thenReturn(Map.of());
+ when(ec2Service.describeSubnets(anyString(), anyList(), anyMap())).thenReturn(List.of(subnet()));
+ when(ec2Service.describeVpcs(anyString(), anyList(), anyMap())).thenReturn(List.of(vpc()));
+ task = task();
+ TaskDefinition taskDef = taskDefinition();
+ when(service.findByMetadataId(anyString())).thenAnswer(inv ->
+ METADATA_ID.equals(inv.getArgument(0))
+ ? Optional.of(new EcsService.MetadataTarget(task,
+ task.getContainers().getFirst(), taskDef))
+ : Optional.empty());
+ controller = new EcsTaskMetadataController(service, containerManager, ec2Service, objectMapper);
+ }
+
+ private static Subnet subnet() {
+ Subnet subnet = new Subnet();
+ subnet.setSubnetId("subnet-default-us-east-1-a");
+ subnet.setVpcId("vpc-0a1b2c3d");
+ subnet.setCidrBlock("172.31.32.0/20");
+ subnet.setAvailabilityZone("us-east-1a");
+ return subnet;
+ }
+
+ private static Vpc vpc() {
+ Vpc vpc = new Vpc();
+ vpc.setVpcId("vpc-0a1b2c3d");
+ vpc.setCidrBlock("172.31.0.0/16");
+ return vpc;
+ }
+
+ private static EcsTask task() {
+ EcsTask task = new EcsTask();
+ task.setTaskArn(TASK_ARN);
+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/metadata-cluster");
+ task.setTaskDefinitionArn("arn:aws:ecs:us-east-1:000000000000:task-definition/web:3");
+ task.setLastStatus("RUNNING");
+ task.setDesiredStatus("RUNNING");
+ task.setLaunchType(LaunchType.FARGATE);
+ task.setPlatformVersion("1.4.0");
+ task.setPlatformFamily("Linux");
+ task.setCpu("256");
+ task.setMemory("512");
+ task.setEphemeralStorage(new EphemeralStorage(20));
+ task.setAvailabilityZone("us-east-1a");
+ task.setCreatedAt(Instant.parse("2026-01-01T10:00:00Z"));
+ task.setStartedAt(Instant.parse("2026-01-01T10:00:05Z"));
+ task.setPullStartedAt(Instant.parse("2026-01-01T10:00:01Z"));
+ task.setPullStoppedAt(Instant.parse("2026-01-01T10:00:04Z"));
+ task.setPrivateIpAddress("172.31.0.42");
+ task.setMacAddress("0e:98:9f:33:76:d3");
+ task.setPrivateDnsName("ip-172-31-0-42.ec2.internal");
+
+ AwsVpcConfiguration awsvpc = new AwsVpcConfiguration();
+ awsvpc.setSubnets(List.of("subnet-default-us-east-1-a"));
+ NetworkConfiguration networkConfiguration = new NetworkConfiguration();
+ networkConfiguration.setAwsvpcConfiguration(awsvpc);
+ task.setNetworkConfiguration(networkConfiguration);
+
+ Container container = new Container();
+ container.setName("app");
+ container.setImage("nginx:latest");
+ container.setLastStatus("RUNNING");
+ container.setDockerId("docker-id-1");
+ container.setMetadataId(METADATA_ID);
+ container.setContainerArn("arn:aws:ecs:us-east-1:000000000000:container/abc123/app");
+ container.setCreatedAt(Instant.parse("2026-01-01T10:00:03Z"));
+ container.setStartedAt(Instant.parse("2026-01-01T10:00:07Z"));
+ task.setContainers(List.of(container));
+ return task;
+ }
+
+ private static TaskDefinition taskDefinition() {
+ ContainerDefinition app = new ContainerDefinition();
+ app.setName("app");
+ app.setImage("nginx:latest");
+ app.setCpu(128);
+ app.setMemory(256);
+ app.setLogConfiguration(new LogConfiguration("awslogs",
+ Map.of("awslogs-group", "/ecs/web"), null));
+
+ TaskDefinition taskDef = new TaskDefinition();
+ taskDef.setFamily("web");
+ taskDef.setRevision(3);
+ taskDef.setNetworkMode(NetworkMode.awsvpc);
+ taskDef.setContainerDefinitions(List.of(app));
+ return taskDef;
+ }
+
+ private JsonNode body(Response response) {
+ return objectMapper.valueToTree(response.getEntity());
+ }
+
+ @Test
+ void containerMetadataReportsTheAwsMembers() {
+ Response response = controller.container(METADATA_ID);
+ assertEquals(200, response.getStatus());
+
+ JsonNode container = body(response);
+ assertEquals("docker-id-1", container.path("DockerId").asText());
+ assertEquals("app", container.path("Name").asText());
+ assertEquals("nginx:latest", container.path("Image").asText());
+ assertEquals("RUNNING", container.path("KnownStatus").asText());
+ assertEquals("NORMAL", container.path("Type").asText());
+ assertEquals(128, container.path("Limits").path("CPU").asInt());
+ assertEquals(256, container.path("Limits").path("Memory").asInt());
+ // SubnetId is not a member of the v4 network object; only the documented ones are written.
+ assertTrue(container.path("Networks").get(0).path("SubnetId").isMissingNode(),
+ "the network object must carry only the members AWS documents");
+ assertEquals("awslogs", container.path("LogDriver").asText());
+ assertEquals("/ecs/web", container.path("LogOptions").path("awslogs-group").asText());
+ assertEquals("web", container.path("Labels").path("com.amazonaws.ecs.task-definition-family").asText());
+ assertEquals("3", container.path("Labels").path("com.amazonaws.ecs.task-definition-version").asText());
+ assertEquals(TASK_ARN, container.path("Labels").path("com.amazonaws.ecs.task-arn").asText());
+ assertEquals("awsvpc", container.path("Networks").get(0).path("NetworkMode").asText());
+ assertEquals("172.31.0.42", container.path("Networks").get(0).path("IPv4Addresses").get(0).asText());
+ assertEquals(0, container.path("Networks").get(0).path("AttachmentIndex").asInt());
+ assertEquals("0e:98:9f:33:76:d3", container.path("Networks").get(0).path("MACAddress").asText());
+ assertEquals("ip-172-31-0-42.ec2.internal",
+ container.path("Networks").get(0).path("PrivateDNSName").asText());
+ // A container's own clock, not the task's: they start one at a time behind dependsOn.
+ assertEquals("2026-01-01T10:00:03Z", container.path("CreatedAt").asText());
+ assertEquals("2026-01-01T10:00:07Z", container.path("StartedAt").asText());
+ }
+
+ @Test
+ void aContainerWithoutItsOwnClockFallsBackToTheTasks() {
+ Container container = task.getContainers().getFirst();
+ container.setCreatedAt(null);
+ container.setStartedAt(null);
+
+ JsonNode node = body(controller.container(METADATA_ID));
+
+ assertEquals("2026-01-01T10:00:00Z", node.path("CreatedAt").asText());
+ assertEquals("2026-01-01T10:00:05Z", node.path("StartedAt").asText());
+ }
+
+ @Test
+ void aStoppedContainerReportsWhenItFinished() {
+ task.getContainers().getFirst().setFinishedAt(Instant.parse("2026-01-01T10:30:00Z"));
+
+ assertEquals("2026-01-01T10:30:00Z",
+ body(controller.container(METADATA_ID)).path("FinishedAt").asText());
+ }
+
+ /**
+ * The subnet half of the network object. AWS derives the gateway from the subnet's own CIDR,
+ * the resolver from the VPC's, and names us-east-1's search domain {@code ec2.internal}.
+ */
+ @Test
+ void theNetworkObjectDescribesTheSubnetTheTaskSitsIn() {
+ JsonNode network = body(controller.container(METADATA_ID)).path("Networks").get(0);
+
+ assertEquals("172.31.32.0/20", network.path("IPv4SubnetCIDRBlock").asText());
+ assertEquals("172.31.32.1/20", network.path("SubnetGatewayIpv4Address").asText());
+ assertEquals("172.31.0.2", network.path("DomainNameServers").get(0).asText());
+ assertEquals("ec2.internal", network.path("DomainNameSearchList").get(0).asText());
+ }
+
+ @Test
+ void aTaskWithNoNetworkConfigurationReportsNoSubnetMembers() {
+ task.setNetworkConfiguration(null);
+
+ JsonNode network = body(controller.container(METADATA_ID)).path("Networks").get(0);
+
+ assertTrue(network.path("AttachmentIndex").isMissingNode());
+ assertTrue(network.path("IPv4SubnetCIDRBlock").isMissingNode());
+ assertTrue(network.path("DomainNameServers").isMissingNode());
+ }
+
+ @Test
+ void taskMetadataReportsTheTaskAndItsContainers() {
+ Response response = controller.task(METADATA_ID);
+ assertEquals(200, response.getStatus());
+
+ JsonNode task = body(response);
+ assertEquals(TASK_ARN, task.path("TaskARN").asText());
+ assertEquals("web", task.path("Family").asText());
+ assertEquals("3", task.path("Revision").asText());
+ assertEquals("FARGATE", task.path("LaunchType").asText());
+ assertEquals("us-east-1a", task.path("AvailabilityZone").asText());
+ // The task's CPU limit is a vCPU count, not the CPU units the API takes.
+ assertEquals(0.25, task.path("Limits").path("CPU").asDouble(), 0.0001);
+ assertEquals(512, task.path("Limits").path("Memory").asInt());
+ assertEquals("2026-01-01T10:00:01Z", task.path("PullStartedAt").asText());
+ assertEquals(1, task.path("Containers").size());
+ assertEquals("app", task.path("Containers").get(0).path("Name").asText());
+ // Fargate reports the clock's accuracy and the task's ephemeral storage alongside it.
+ assertEquals("SYNCHRONIZED", task.path("ClockDrift").path("ClockSynchronizationStatus").asText());
+ assertTrue(task.path("ClockDrift").has("ReferenceTimestamp"));
+ assertEquals(20 * 1024, task.path("EphemeralStorageMetrics").path("Reserved").asInt());
+ // VPCID is the container instance's, which a Fargate task does not have.
+ assertTrue(task.path("VPCID").isMissingNode());
+ }
+
+ @Test
+ void anEc2TaskReportsTheVpcItsSubnetBelongsTo() {
+ task.setLaunchType(LaunchType.EC2);
+
+ assertEquals("vpc-0a1b2c3d", body(controller.task(METADATA_ID)).path("VPCID").asText());
+ }
+
+ @Test
+ void taskWithTagsAddsTheTaskAndContainerInstanceTags() {
+ task.setLaunchType(LaunchType.EC2);
+ task.setContainerInstanceArn(
+ "arn:aws:ecs:us-east-1:000000000000:container-instance/metadata-cluster/ci-1");
+ when(service.listTagsForResource(TASK_ARN)).thenReturn(Map.of("owner", "platform"));
+ when(service.listTagsForResource(task.getContainerInstanceArn()))
+ .thenReturn(Map.of("fleet", "spot"));
+
+ JsonNode withTags = body(controller.taskWithTags(METADATA_ID));
+
+ assertEquals(TASK_ARN, withTags.path("TaskARN").asText(), "the task document is carried whole");
+ assertEquals("platform", withTags.path("TaskTags").path("owner").asText());
+ assertEquals("spot", withTags.path("ContainerInstanceTags").path("fleet").asText());
+ assertTrue(withTags.path("Errors").isMissingNode());
+ }
+
+ /** The container agent serves taskWithTags, and a Fargate task has no agent behind it. */
+ @Test
+ void taskWithTagsIsNotFoundForAFargateTask() {
+ assertEquals(404, controller.taskWithTags(METADATA_ID).getStatus());
+ }
+
+ @Test
+ void taskWithTagsReportsAFailedTagLookupUnderErrors() {
+ task.setLaunchType(LaunchType.EC2);
+ when(service.listTagsForResource(TASK_ARN))
+ .thenThrow(new AwsException("InvalidParameterException", "Resource not found", 400));
+
+ JsonNode withTags = body(controller.taskWithTags(METADATA_ID));
+
+ assertTrue(withTags.path("TaskTags").isMissingNode());
+ assertEquals("TaskTags", withTags.path("Errors").get(0).path("ErrorField").asText());
+ assertEquals("InvalidParameterException",
+ withTags.path("Errors").get(0).path("ErrorCode").asText());
+ assertEquals(400, withTags.path("Errors").get(0).path("StatusCode").asInt());
+ }
+
+ @Test
+ void aContainerWithoutLimitsReportsNoneRatherThanZeroes() {
+ EcsService limitlessService = mock(EcsService.class);
+ EcsTask limitlessTask = task();
+ TaskDefinition taskDef = taskDefinition();
+ taskDef.getContainerDefinitions().getFirst().setCpu(null);
+ taskDef.getContainerDefinitions().getFirst().setMemory(null);
+ when(limitlessService.findByMetadataId(anyString())).thenReturn(Optional.of(
+ new EcsService.MetadataTarget(limitlessTask, limitlessTask.getContainers().getFirst(), taskDef)));
+
+ JsonNode container = body(new EcsTaskMetadataController(limitlessService, containerManager,
+ ec2Service, objectMapper).container(METADATA_ID));
+
+ assertTrue(container.path("Limits").isMissingNode(),
+ "Limits is omitted when the container definition sets none");
+ assertTrue(container.path("ImageID").isMissingNode(),
+ "ImageID is omitted until the image digest is known");
+ }
+
+ @Test
+ void statsReportTheSampleDockerGaveUnderItsOwnMemberNames() throws JsonProcessingException {
+ EcsContainerManager.ContainerStats stats1 =
+ new EcsContainerManager.ContainerStats(sample(), 43.5, 215.25);
+ when(containerManager.sampleContainerStats("docker-id-1")).thenReturn(Optional.of(stats1));
+ when(containerManager.sampleContainerStats(List.of("docker-id-1")))
+ .thenReturn(Map.of("docker-id-1", stats1));
+
+ JsonNode stats = body(controller.containerStats(METADATA_ID));
+ assertEquals("2026-01-01T10:05:00.000000000Z", stats.path("read").asText());
+ assertEquals(42000000, stats.path("cpu_stats").path("cpu_usage").path("total_usage").asLong());
+ assertEquals(1048576, stats.path("memory_stats").path("usage").asLong());
+ assertEquals(84, stats.path("networks").path("eth0").path("rx_bytes").asLong());
+ // The rates are the agent's, derived from consecutive samples; Docker reports only totals.
+ assertEquals(43.5, stats.path("network_rate_stats").path("rx_bytes_per_sec").asDouble(), 0.0001);
+ assertEquals(215.25, stats.path("network_rate_stats").path("tx_bytes_per_sec").asDouble(), 0.0001);
+ // docker-java's model carries a field per API version; the ones this daemon left out are
+ // dropped rather than written as nulls onto a document a client parses as Docker's.
+ assertFalse(stats.has("network"), "a member the daemon did not send must not appear as null");
+
+ JsonNode taskStats = body(controller.taskStats(METADATA_ID));
+ assertEquals(1048576, taskStats.path("docker-id-1").path("memory_stats").path("usage").asLong(),
+ "task stats are keyed by Docker id, each carrying that container's sample");
+ }
+
+ @Test
+ void statsOmitTheRatesWhenOnlyOneSampleArrived() throws JsonProcessingException {
+ when(containerManager.sampleContainerStats("docker-id-1")).thenReturn(Optional.of(
+ new EcsContainerManager.ContainerStats(sample(), null, null)));
+
+ JsonNode stats = body(controller.containerStats(METADATA_ID));
+
+ assertEquals(1048576, stats.path("memory_stats").path("usage").asLong());
+ assertTrue(stats.path("network_rate_stats").isMissingNode(),
+ "a rate that could not be measured is left out rather than reported as zero");
+ }
+
+ @Test
+ void statsAnswerWithAnEmptyDocumentWhenTheDaemonHasNoSample() {
+ JsonNode stats = body(controller.containerStats(METADATA_ID));
+ assertTrue(stats.isObject() && stats.isEmpty(), "an unsampled container reports {}");
+
+ JsonNode taskStats = body(controller.taskStats(METADATA_ID));
+ assertTrue(taskStats.path("docker-id-1").isObject() && taskStats.path("docker-id-1").isEmpty(),
+ "every container in the task keeps an entry, empty rather than null");
+ }
+
+ /** A trimmed Docker stats response, read through docker-java's model the way the daemon's is. */
+ private static Statistics sample() throws JsonProcessingException {
+ String json = """
+ {
+ "read": "2026-01-01T10:05:00.000000000Z",
+ "preread": "2026-01-01T10:04:59.000000000Z",
+ "num_procs": 3,
+ "cpu_stats": {"cpu_usage": {"total_usage": 42000000}, "system_cpu_usage": 90000000},
+ "precpu_stats": {"cpu_usage": {"total_usage": 41000000}, "system_cpu_usage": 89000000},
+ "memory_stats": {"usage": 1048576, "limit": 268435456},
+ "pids_stats": {"current": 3},
+ "networks": {"eth0": {"rx_bytes": 84, "tx_bytes": 120}}
+ }
+ """;
+ return JsonMapper.builder()
+ .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
+ .build()
+ .readValue(json, Statistics.class);
+ }
+
+ @Test
+ void anUnknownIdIsNotFound() {
+ Response response = controller.container("not-a-metadata-id");
+ assertEquals(404, response.getStatus());
+ assertTrue(body(response).path("error").asText().contains("Unable to get metadata"));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataIntegrationTest.java
new file mode 100644
index 0000000000..a58b7ba1cc
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsTaskMetadataIntegrationTest.java
@@ -0,0 +1,172 @@
+package io.github.hectorvent.floci.services.ecs;
+
+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;
+import io.github.hectorvent.floci.services.ecs.model.Container;
+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;
+import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.LaunchType;
+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;
+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;
+import io.quarkus.test.InjectMock;
+import io.quarkus.test.junit.QuarkusTest;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import static io.restassured.RestAssured.given;
+import static org.hamcrest.Matchers.anEmptyMap;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.equalTo;
+import static org.mockito.ArgumentMatchers.anyList;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.when;
+
+/**
+ * The task metadata endpoint over the wire. Floci serves it at the root, next to S3's
+ * {@code /{bucket}/{key}} catch-all, so route registration and precedence are what these cover:
+ * a miss would hand a workload S3's XML instead of the document it read the endpoint for. The
+ * response shapes themselves are pinned down in {@link EcsTaskMetadataControllerTest}.
+ */
+@QuarkusTest
+class EcsTaskMetadataIntegrationTest {
+
+ private static final String METADATA_ID = "4c1b0a9e8d7f4a3b9c2d1e0f5a4b3c2d";
+ private static final String DOCKER_ID = "0f9a8b7c6d5e";
+ private static final String TASK_ARN =
+ "arn:aws:ecs:us-east-1:000000000000:task/metadata-cluster/route123";
+
+ @InjectMock
+ EcsService ecsService;
+
+ @InjectMock
+ EcsContainerManager containerManager;
+
+ private EcsTask task;
+
+ @BeforeEach
+ void resolveOneMetadataId() {
+ task = task();
+ when(ecsService.findByMetadataId(anyString())).thenAnswer(invocation ->
+ METADATA_ID.equals(invocation.getArgument(0))
+ ? Optional.of(new EcsService.MetadataTarget(task,
+ task.getContainers().getFirst(), taskDefinition()))
+ : Optional.empty());
+ when(containerManager.sampleContainerStats(anyString())).thenReturn(Optional.empty());
+ when(containerManager.sampleContainerStats(anyList())).thenReturn(Map.of());
+ }
+
+ @Test
+ void containerDocumentIsServedAtTheRoot() {
+ given()
+ .when()
+ .get("/v4/" + METADATA_ID)
+ .then()
+ .statusCode(200)
+ .contentType(containsString("json"))
+ .body("DockerId", equalTo(DOCKER_ID))
+ .body("Name", equalTo("app"))
+ .body("Labels.'com.amazonaws.ecs.task-arn'", equalTo(TASK_ARN));
+ }
+
+ @Test
+ void taskDocumentIsServedUnderTheContainerPath() {
+ given()
+ .when()
+ .get("/v4/" + METADATA_ID + "/task")
+ .then()
+ .statusCode(200)
+ .contentType(containsString("json"))
+ .body("TaskARN", equalTo(TASK_ARN))
+ .body("Family", equalTo("web"))
+ .body("Revision", equalTo("3"))
+ .body("Containers[0].Name", equalTo("app"));
+ }
+
+ @Test
+ void taskWithTagsIsRoutedAlongsideTheTaskPath() {
+ when(ecsService.listTagsForResource(TASK_ARN)).thenReturn(Map.of("owner", "platform"));
+ // taskWithTags is the container agent's path, so only an EC2 task has it.
+ task.setLaunchType(LaunchType.EC2);
+
+ given()
+ .when()
+ .get("/v4/" + METADATA_ID + "/taskWithTags")
+ .then()
+ .statusCode(200)
+ .contentType(containsString("json"))
+ .body("TaskARN", equalTo(TASK_ARN))
+ .body("TaskTags.owner", equalTo("platform"));
+ }
+
+ @Test
+ void bothStatsPathsAreRoutedAndAnswerWithJson() {
+ given()
+ .when()
+ .get("/v4/" + METADATA_ID + "/stats")
+ .then()
+ .statusCode(200)
+ .contentType(containsString("json"))
+ .body("$", anEmptyMap());
+
+ given()
+ .when()
+ .get("/v4/" + METADATA_ID + "/task/stats")
+ .then()
+ .statusCode(200)
+ .contentType(containsString("json"))
+ .body(DOCKER_ID, anEmptyMap());
+ }
+
+ /**
+ * An unknown id must reach the endpoint and get its JSON error, not fall through to S3 and
+ * come back as {@code NoSuchBucket} XML for a bucket named {@code v4}.
+ */
+ @Test
+ void anUnknownIdAnswersWithTheEndpointsOwnNotFound() {
+ given()
+ .when()
+ .get("/v4/not-a-metadata-id")
+ .then()
+ .statusCode(404)
+ .contentType(containsString("json"))
+ .body("error", containsString("Unable to get metadata"));
+ }
+
+ private static EcsTask task() {
+ EcsTask task = new EcsTask();
+ task.setTaskArn(TASK_ARN);
+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/metadata-cluster");
+ task.setTaskDefinitionArn("arn:aws:ecs:us-east-1:000000000000:task-definition/web:3");
+ task.setLastStatus("RUNNING");
+ task.setDesiredStatus("RUNNING");
+ task.setLaunchType(LaunchType.FARGATE);
+ task.setCpu("256");
+ task.setMemory("512");
+
+ Container container = new Container();
+ container.setName("app");
+ container.setImage("nginx:latest");
+ container.setLastStatus("RUNNING");
+ container.setDockerId(DOCKER_ID);
+ container.setMetadataId(METADATA_ID);
+ container.setContainerArn("arn:aws:ecs:us-east-1:000000000000:container/route123/app");
+ task.setContainers(List.of(container));
+ return task;
+ }
+
+ private static TaskDefinition taskDefinition() {
+ ContainerDefinition app = new ContainerDefinition();
+ app.setName("app");
+ app.setImage("nginx:latest");
+
+ TaskDefinition taskDef = new TaskDefinition();
+ taskDef.setFamily("web");
+ taskDef.setRevision(3);
+ taskDef.setNetworkMode(NetworkMode.awsvpc);
+ taskDef.setContainerDefinitions(List.of(app));
+ return taskDef;
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java
index 78692d89ac..b2d4610f6a 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java
@@ -24,7 +24,9 @@ import org.mockito.ArgumentCaptor;
import java.util.List;
import java.util.Map;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
@@ -149,6 +151,36 @@ class EcsContainerManagerAwsBaselineTest {
"overridden baseline endpoint should not remain");
}
+ @Test
+ void injectsTheTaskMetadataEndpointEachContainerCanReach() {
+ when(awsEnv.flociEndpoint()).thenReturn("http://host.docker.internal:4566");
+
+ ContainerDefinition app = containerDef("app", "app:latest", List.of());
+ TaskDefinition taskDef = new TaskDefinition();
+ taskDef.setFamily("test-family");
+ taskDef.setContainerDefinitions(List.of(app));
+
+ EcsTask task = new EcsTask();
+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");
+
+ manager.startTask(task, taskDef, null, "us-east-1");
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<String>> envCaptor = ArgumentCaptor.forClass(List.class);
+ verify(builder).withEnv(envCaptor.capture());
+ String metadataUri = envCaptor.getValue().stream()
+ .filter(entry -> entry.startsWith("ECS_CONTAINER_METADATA_URI_V4="))
+ .findFirst()
+ .orElse(null);
+
+ assertNotNull(metadataUri, "every launched container is told where its metadata lives");
+ assertTrue(metadataUri.startsWith("ECS_CONTAINER_METADATA_URI_V4=http://host.docker.internal:4566/v4/"),
+ "the metadata URI must point at Floci's own v4 path: " + metadataUri);
+ // The id in the URI is the one the container is then found by.
+ String id = metadataUri.substring(metadataUri.lastIndexOf('/') + 1);
+ assertEquals(id, task.getContainers().getFirst().getMetadataId());
+ }
+
private static ContainerDefinition containerDef(String name, String image, List<KeyValuePair> env) {
ContainerDefinition def = new ContainerDefinition();
def.setName(name);
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java
index 1f8fcbe9f0..c5a74b3e65 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java
@@ -120,8 +120,10 @@ class EcsContainerManagerSecurityGroupTest {
verify(builder, never()).withPortBinding(anyInt(), anyInt());
verify(builder, never()).withDynamicPort(anyInt());
verify(builder, never()).withExposedPort(anyInt());
+ // Port bindings come off the helper, whose namespace holds them; the workload container
+ // shares that namespace and publishes none, and is inspected only for its launch clock.
verify(dockerClient).inspectContainerCmd("helper-id");
- verify(dockerClient, never()).inspectContainerCmd("docker-id");
+ verify(dockerClient, times(1)).inspectContainerCmd("docker-id");
verify(firewallManager).register(any(), eq("helper-id"), any());
assertEquals("eni-1", handle.getNetworkInterfaceId());
assertEquals("us-east-1", handle.getRegion());
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsDockerIntegrationTest.java
new file mode 100644
index 0000000000..7816330294
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsDockerIntegrationTest.java
@@ -0,0 +1,133 @@
+package io.github.hectorvent.floci.services.ecs.container;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.model.Statistics;
+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;
+import io.github.hectorvent.floci.services.ecs.model.EcsTask;
+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;
+import io.quarkus.test.junit.QuarkusTest;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The sample behind the task metadata endpoint's {@code /stats} paths, taken from a real daemon.
+ * AWS returns Docker's own stats document there, so what matters is that a sample arrives at all
+ * and carries the counters a client reads; the serialized member names are pinned down in
+ * {@code EcsTaskMetadataControllerTest}.
+ */
+@QuarkusTest
+class EcsContainerManagerStatsDockerIntegrationTest {
+
+ private static final String BUSYBOX_IMAGE = "public.ecr.aws/docker/library/busybox:latest";
+
+ @Inject
+ EcsContainerManager containerManager;
+
+ @Inject
+ DockerClient dockerClient;
+
+ @BeforeEach
+ void requireDocker() {
+ Assumptions.assumeTrue(isDockerAvailable(),
+ "Docker daemon must be available for ECS stats integration tests");
+ }
+
+ @Test
+ void aRunningContainerReportsTheDaemonsStats() {
+ String suffix = UUID.randomUUID().toString().substring(0, 8);
+ TaskDefinition taskDefinition = new TaskDefinition();
+ taskDefinition.setFamily("stats-docker-" + suffix);
+ taskDefinition.setContainerDefinitions(List.of(sleeper("app")));
+
+ EcsTask task = new EcsTask();
+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/stats/" + suffix);
+
+ EcsTaskHandle handle = containerManager.startTask(task, taskDefinition, List.of(), "us-east-1");
+ try {
+ Optional<EcsContainerManager.ContainerStats> sample =
+ containerManager.sampleContainerStats(handle.getContainerIds().get("app"));
+
+ assertTrue(sample.isPresent(), "a running container must have stats to sample");
+ Statistics statistics = sample.get().statistics();
+ assertNotNull(statistics.getRead(), "the sample carries the instant Docker read it");
+ assertNotNull(statistics.getCpuStats(), "the sample carries the container's CPU stats");
+ assertNotNull(statistics.getMemoryStats(), "the sample carries the container's memory stats");
+ assertNotNull(statistics.getPreCpuStats(),
+ "the second sample carries the previous CPU reading a client computes against");
+ // A container on a Docker network reports interfaces, so the rate delta is measurable.
+ assertNotNull(sample.get().rxBytesPerSecond(), "two samples make the receive rate measurable");
+ assertNotNull(sample.get().txBytesPerSecond(), "two samples make the transmit rate measurable");
+ } finally {
+ containerManager.stopTask(handle);
+ }
+ }
+
+ /**
+ * The daemon holds a stats stream open per container, so a task's containers are sampled in one
+ * pass with every stream open at once. That the real daemon serves them alongside each other is
+ * what this covers; the cost of the pass is pinned down in {@code EcsContainerManagerStatsSamplingTest}.
+ */
+ @Test
+ void aTasksContainersAreAllSampledInOnePass() {
+ String suffix = UUID.randomUUID().toString().substring(0, 8);
+ TaskDefinition taskDefinition = new TaskDefinition();
+ taskDefinition.setFamily("stats-pass-" + suffix);
+ taskDefinition.setContainerDefinitions(List.of(
+ sleeper("app"), sleeper("sidecar"), sleeper("logger")));
+
+ EcsTask task = new EcsTask();
+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/stats/" + suffix);
+
+ EcsTaskHandle handle = containerManager.startTask(task, taskDefinition, List.of(), "us-east-1");
+ try {
+ List<String> dockerIds = List.of(handle.getContainerIds().get("app"),
+ handle.getContainerIds().get("sidecar"),
+ handle.getContainerIds().get("logger"));
+
+ Map<String, EcsContainerManager.ContainerStats> sampled =
+ containerManager.sampleContainerStats(dockerIds);
+
+ assertEquals(dockerIds.size(), sampled.size(), "every container in the task is sampled");
+ for (String dockerId : dockerIds) {
+ assertNotNull(sampled.get(dockerId).statistics().getRead(),
+ "each sample carries the instant Docker read it");
+ }
+ } finally {
+ containerManager.stopTask(handle);
+ }
+ }
+
+ @Test
+ void aContainerThatIsGoneReportsNoSampleRatherThanFailing() {
+ assertTrue(containerManager.sampleContainerStats("floci-no-such-container").isEmpty());
+ assertTrue(containerManager.sampleContainerStats((String) null).isEmpty());
+ }
+
+ private static ContainerDefinition sleeper(String name) {
+ ContainerDefinition definition = new ContainerDefinition();
+ definition.setName(name);
+ definition.setImage(BUSYBOX_IMAGE);
+ definition.setCommand(List.of("sleep", "60"));
+ return definition;
+ }
+
+ private boolean isDockerAvailable() {
+ try {
+ dockerClient.pingCmd().exec();
+ return true;
+ } catch (Exception e) {
+ return false;
+ }
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsSamplingTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsSamplingTest.java
new file mode 100644
index 0000000000..7ed83f9237
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerStatsSamplingTest.java
@@ -0,0 +1,185 @@
+package io.github.hectorvent.floci.services.ecs.container;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.json.JsonMapper;
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.async.ResultCallback;
+import com.github.dockerjava.api.command.StatsCmd;
+import com.github.dockerjava.api.model.Statistics;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.RegionResolver;
+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
+import io.github.hectorvent.floci.core.common.docker.ContainerDetector;
+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;
+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;
+import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;
+import io.github.hectorvent.floci.services.s3.S3Service;
+import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;
+import io.github.hectorvent.floci.services.ssm.SsmService;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/**
+ * How the task metadata endpoint's {@code /stats} paths sample the daemon. Docker only sends its
+ * second stats sample on the next collection tick, so a task's containers have to be sampled
+ * alongside each other: {@code /task/stats} is the path a sidecar polls, and sampling one
+ * container after another would cost it a tick per container.
+ *
+ * <p>The daemon is mocked. A stream sends its first sample as soon as it is opened and holds its
+ * second one back until every stream the pass opens is open, which is what a serial implementation
+ * could never satisfy.
+ */
+class EcsContainerManagerStatsSamplingTest {
+
+ private static final String FIRST_READ = "2026-01-01T10:05:00.000000000Z";
+ private static final String SECOND_READ = "2026-01-01T10:05:01.000000000Z";
+
+ private final Map<String, StatsCmd> commands = new ConcurrentHashMap<>();
+ private EcsContainerManager manager;
+
+ /**
+ * Trips once every stream a pass is expected to open has been opened. Only the pass that is
+ * about the ordering raises it: the other tests leave it open so their samples flow at once.
+ */
+ private CountDownLatch allStreamsOpen;
+
+ @BeforeEach
+ void setUp() {
+ allStreamsOpen = new CountDownLatch(0);
+
+ DockerClient dockerClient = mock(DockerClient.class);
+ when(dockerClient.statsCmd(anyString())).thenAnswer(invocation ->
+ commands.computeIfAbsent(invocation.getArgument(0), id -> tickingStatsCmd()));
+
+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);
+
+ manager = new EcsContainerManager(mock(ContainerBuilder.class), lifecycleManager,
+ mock(ContainerLogStreamer.class), mock(ContainerDetector.class),
+ mock(EmulatorConfig.class, RETURNS_DEEP_STUBS), mock(RegionResolver.class),
+ mock(LaunchedContainerAwsEnv.class), mock(SsmService.class),
+ mock(SecretsManagerService.class), mock(S3Service.class),
+ mock(EcrRegistryManager.class), mock(HostVolumePolicy.class));
+ }
+
+ @Test
+ @Timeout(10)
+ void aTasksContainersAreSampledAlongsideEachOtherRatherThanOneAfterAnother() {
+ List<String> dockerIds = List.of("docker-1", "docker-2", "docker-3", "docker-4");
+ // No stream sends its second sample until all four are open, so a pass that waits on one
+ // stream before opening the next cannot collect a pair for any of them.
+ allStreamsOpen = new CountDownLatch(dockerIds.size());
+
+ Map<String, EcsContainerManager.ContainerStats> sampled = manager.sampleContainerStats(dockerIds);
+
+ assertEquals(dockerIds.size(), sampled.size(), "every container in the task reports a sample");
+ for (String dockerId : dockerIds) {
+ EcsContainerManager.ContainerStats stats = sampled.get(dockerId);
+ assertEquals(SECOND_READ, stats.statistics().getRead(), "the later sample is the one reported");
+ assertNotNull(stats.rxBytesPerSecond(), "a pair arrived for " + dockerId + ", so it has rates");
+ assertEquals(100.0, stats.rxBytesPerSecond(), 0.0001, "the rate is the delta across the pair");
+ assertEquals(200.0, stats.txBytesPerSecond(), 0.0001);
+ }
+ }
+
+ @Test
+ void everyStreamIsClosedOnceThePassIsDone() throws IOException {
+ manager.sampleContainerStats(List.of("docker-1", "docker-2"));
+
+ for (StatsCmd command : commands.values()) {
+ verify(command).close();
+ }
+ }
+
+ @Test
+ void aContainerTheDaemonRefusesIsLeftOutWithoutFailingTheRest() {
+ commands.put("docker-gone", failingStatsCmd());
+
+ Map<String, EcsContainerManager.ContainerStats> sampled =
+ manager.sampleContainerStats(List.of("docker-gone", "docker-1"));
+
+ assertFalse(sampled.containsKey("docker-gone"), "a container with no stream reports no sample");
+ assertTrue(sampled.containsKey("docker-1"), "the containers alongside it are still sampled");
+ }
+
+ @Test
+ void oneContainerIsSampledThroughTheSamePass() {
+ Optional<EcsContainerManager.ContainerStats> stats = manager.sampleContainerStats("docker-1");
+
+ assertTrue(stats.isPresent());
+ assertEquals(SECOND_READ, stats.get().statistics().getRead());
+ assertTrue(manager.sampleContainerStats((String) null).isEmpty(), "no id means no sample");
+ }
+
+ /**
+ * A stats stream that sends its first sample as soon as it is opened and its second one only
+ * once every stream of the pass is open, standing in for Docker's collection tick.
+ */
+ private StatsCmd tickingStatsCmd() {
+ StatsCmd command = mock(StatsCmd.class);
+ when(command.exec(any())).thenAnswer(invocation -> {
+ ResultCallback<Statistics> callback = invocation.getArgument(0);
+ allStreamsOpen.countDown();
+ Thread emitter = new Thread(() -> {
+ try {
+ callback.onNext(sample(FIRST_READ, 100));
+ if (allStreamsOpen.await(5, TimeUnit.SECONDS)) {
+ callback.onNext(sample(SECOND_READ, 200));
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ } catch (JsonProcessingException e) {
+ throw new IllegalStateException("the fixed stats sample must parse", e);
+ }
+ });
+ emitter.setDaemon(true);
+ emitter.start();
+ return callback;
+ });
+ return command;
+ }
+
+ /** A container the daemon has already forgotten: opening its stream throws. */
+ private static StatsCmd failingStatsCmd() {
+ StatsCmd command = mock(StatsCmd.class);
+ when(command.exec(any())).thenThrow(new IllegalStateException("no such container"));
+ return command;
+ }
+
+ private static Statistics sample(String read, long rxBytes) throws JsonProcessingException {
+ String json = """
+ {
+ "read": "%s",
+ "cpu_stats": {"cpu_usage": {"total_usage": 42000000}},
+ "memory_stats": {"usage": 1048576},
+ "networks": {"eth0": {"rx_bytes": %d, "tx_bytes": %d}}
+ }
+ """.formatted(read, rxBytes, rxBytes * 2);
+ return JsonMapper.builder()
+ .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
+ .build()
+ .readValue(json, Statistics.class);
+ }
+}