Light, fluffy, and always free - The AWS Local Emulator alternative
feat(ecs): support fluentd FireLens routers and s3-hosted FireLens config (#3790)
z9rtme32266537e2a
diff --git a/docs/services/ecs.md b/docs/services/ecs.mdindex 93bcc1fbd2..ad9d563c46 100644--- a/docs/services/ecs.md+++ b/docs/services/ecs.md@@ -32,20 +32,63 @@ ECS emulates clusters, task definitions, tasks, and services. In the default con`runtimePlatform` and a container's `logConfiguration` are stored and returned exactly asregistered, so a client that reads back what it wrote (Terraform, or a deploy tool verifying its-own `RegisterTaskDefinition`) sees no drift. FireLens task definitions are also used at runtime:-Floci starts the configured `fluentbit` or `fluentd` router first, places routed containers in its-network namespace, and sends their output to the task-local forward port `24224` through Docker's-Fluentd log driver. The router container's output continues to be forwarded by Floci's existing-CloudWatch Logs integration. Custom S3 configuration files are not downloaded by Floci.-The `awsfirelens` output options are preserved in the task definition, but Floci does not generate-or inject a Fluent Bit or Fluentd output configuration from those options.--#### FireLens--A task using `awsfirelens` must contain exactly one container with `firelensConfiguration` and a-supported type of `fluentbit` or `fluentd`. The router must not publish port `24224`. Docker-backed-tasks require a router image configured to accept the Fluent Forward protocol on-`127.0.0.1:24224`; mock mode preserves the JSON fields without starting containers.+own `RegisterTaskDefinition`) sees no drift. `runtimePlatform` does not change where a local task+runs: Floci launches every task on the host's own architecture.++`firelensConfiguration` is stored and returned the same way. `RegisterTaskDefinition` rejects a+missing or unsupported `type` (`fluentd` and `fluentbit` only), and a task using `awsfirelens`+must name exactly one router: a task definition with two FireLens routers, or a router publishing+port `24224`, is rejected at launch. A `fluentbit` or `fluentd` FireLens+container is acted on at launch: Floci generates the router config (unix socket input, TCP forward+on bridge/awsvpc, ECS metadata, optional include of a `config-file-type=file` or `s3` extra+config, and one output per `awsfirelens` container), starts that router first, and points application+containers with `logDriver: awsfirelens` at the generated unix socket. Other log drivers,+including `awslogs`, still stream to CloudWatch via Floci rather than the configured driver.+An `[OUTPUT]` for an AWS destination whose plugin reads a URL from `endpoint` (`s3`,+`cloudwatch`, `firehose`) also gets `Endpoint` set to Floci's container-reachable base URL. The+Fluent Bit AWS plugins take a custom endpoint only from their own configuration and ignore the+`AWS_ENDPOINT_URL` injected into the container, so without it the router would ship logs to the+real service. An `endpoint` set in the task definition's log options is never overwritten, so+aiming one output at real AWS still works, and outputs declared in an `@INCLUDE`d or+`config-file-type=s3` config are not visible to Floci and keep whatever endpoint they were+written with.+The upstream C plugins (`cloudwatch_logs`, `kinesis_firehose`, `kinesis_streams`) are left+alone instead. They hand `endpoint` to `getaddrinfo` as a bare host name rather than parsing it as+a URL, and they always dial TLS, so Floci's `http://host:port` base URL fails there as+`Misformatted domain name` and a bare host fails certificate verification; no output-level switch+disables either. On the `aws-for-fluent-bit` 3.x line these plugins also honour a separate `port`+(undocumented for `kinesis_firehose` and `cloudwatch_logs`, but it works), so an output can be+aimed at Floci's port, but it still cannot complete the TLS handshake. Those outputs go wherever+the task definition points them.+An injected `http://` endpoint also gets `tls Off`. Fluent Bit 1.9 (the `aws-for-fluent-bit` 2.x+and `:latest` line) still calls `flb_tls_session_create` on HTTP S3 and SIGSEGVs on a NULL+TLS context; the scheme alone is not enough. A `tls` the task definition already set is left+alone.+The TCP forward listens on `0.0.0.0` rather than AWS's awsvpc `127.0.0.1` because Floci only+shares a network namespace when security-group enforcement is enabled for an awsvpc task. In every+other case, the injected `FLUENT_HOST` (the router's container IP) must be reachable. Fluent Bit+config is written to `/fluent-bit/etc/fluent-bit.conf`. Fluentd config is+written to `/fluentd/etc/fluent.conf` and uses `@type` (not `Name`) for output plugins; Floci+does not inject an `endpoint` into Fluentd outputs.+`config-file-type=s3` follows where ECS itself draws the line. `RegisterTaskDefinition` rejects+it for a Fargate-compatible task definition, with `Fargate launch type does not support+FirelensConfiguration config file from 's3'`, and rejects a `config-file-value` that is not an S3+object ARN with `Invalid arn syntax`. A Fargate task can still take its config from S3 the way AWS+documents, by giving the aws-for-fluent-bit init process its `aws_fluent_bit_init_s3_*`+environment variables. ECS never inspects those and Floci passes them through, so that+registration is accepted here too; it fetches nothing locally either, because Floci serves no ECS+task metadata endpoint, which the init process reads before downloading.+Floci also does not validate a task definition's `compatibilities` /+`requiresCompatibilities` against `RunTask` `launchType`; a Fargate-compatible+definition can still be run with `launchType=EC2` (and the reverse) the same+way a missing metadata endpoint is accepted at registration.+On an EC2-compatible task definition Floci reads the object from its own S3, writes it to the+fixed `external.conf` path next to the generated config (`/fluent-bit/etc/external.conf` or+`/fluentd/etc/external.conf`), and includes it from there, matching the paths the ECS agent uses.+The object is read before any container is created, so a missing bucket or key stops the task with+the agent's reason, `Unable to download firelens s3 config file: unable to download s3 config+<key> from bucket <bucket>: <detail>`, instead of leaking a started router. Shared network+namespaces (AppConfig agent on `127.0.0.1:2772`) are not implemented.### Tasksdiff --git a/src/main/java/io/github/hectorvent/floci/core/common/AwsArnUtils.java b/src/main/java/io/github/hectorvent/floci/core/common/AwsArnUtils.javaindex b02ecfd84e..4e33a9fd62 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/AwsArnUtils.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/AwsArnUtils.java@@ -159,6 +159,35 @@ public final class AwsArnUtils {return account != null && !account.isEmpty() && !account.equals(localAccountId);}+ /**+ * The bucket and key named by an S3 object ARN ({@code arn:aws:s3:::bucket/key}, or the+ * region/account-bearing form {@code arn:aws:s3:region:account:bucket/key}).+ *+ * <p>Returns {@code null} rather than throwing when the value is not one: not an ARN, not+ * the {@code s3} service, or a resource without both a bucket and a key. Callers decide+ * whether that is a registration error (ECS rejecting a FireLens {@code config-file-value})+ * or a launch error.+ */+ public static S3ObjectRef parseS3ObjectArn(String value) {+ if (!isArn(value)) {+ return null;+ }+ Arn arn = parse(value);+ if (!"s3".equals(arn.service())) {+ return null;+ }+ String resource = arn.resource();+ int slash = resource.indexOf('/');+ if (slash <= 0 || slash == resource.length() - 1) {+ return null;+ }+ return new S3ObjectRef(resource.substring(0, slash), resource.substring(slash + 1));+ }++ /** The bucket and key of an S3 object ARN, as returned by {@link #parseS3ObjectArn}. */+ public record S3ObjectRef(String bucket, String key) {+ }+/*** Converts an SQS ARN to a queue URL using the given base URL.* Example: arn:aws:sqs:us-east-1:000000000000:my-queue → http://localhost:4566/000000000000/my-queuediff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/LaunchedContainerAwsEnv.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/LaunchedContainerAwsEnv.javaindex be979b23c1..cdc2a6bdea 100644--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/LaunchedContainerAwsEnv.java+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/LaunchedContainerAwsEnv.java@@ -62,6 +62,16 @@ public class LaunchedContainerAwsEnv {return sdkBaselineEnv(region, awsConfigMountDir, reachableEndpoint.baseUrl(), Optional.empty());}+ /**+ * The Floci base URL this env points a launched container at, i.e. the value of the+ * {@code AWS_ENDPOINT_URL} entry above. Callers that configure something other than an+ * AWS SDK (a Fluent Bit output in a FireLens router, say) reuse it so both agree on how+ * the container reaches Floci.+ */+ public String flociEndpoint() {+ return reachableEndpoint.baseUrl();+ }+/*** Variant for launchers whose workloads reach Floci at an address other than the* Docker-reachable one (e.g. Kubernetes pods reaching Floci's pod IP).diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisioner.java b/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisioner.javaindex b655f33c28..104272da0c 100644--- a/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisioner.java+++ b/src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisioner.java@@ -10,8 +10,10 @@ import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;import io.github.hectorvent.floci.services.ecs.model.EcsCluster;import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;+import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;import io.github.hectorvent.floci.services.ecs.model.KeyValuePair;import io.github.hectorvent.floci.services.ecs.model.LaunchType;+import io.github.hectorvent.floci.services.ecs.model.LogConfiguration;import io.github.hectorvent.floci.services.ecs.model.NetworkConfiguration;import io.github.hectorvent.floci.services.ecs.model.NetworkMode;import io.github.hectorvent.floci.services.ecs.model.PortMapping;@@ -22,6 +24,7 @@ import jakarta.inject.Inject;import org.jboss.logging.Logger;import java.util.ArrayList;+import java.util.LinkedHashMap;import java.util.List;import java.util.Map;import java.util.Set;@@ -270,6 +273,8 @@ public class EcsCfnProvisioner implements CfnResourceProvisioner {def.setPortMappings(parsePortMappings(item.path("PortMappings")));def.setEnvironment(parseEnvironment(item.path("Environment")));def.setSecrets(parseSecrets(item.path("Secrets")));+ def.setLogConfiguration(parseLogConfiguration(item.path("LogConfiguration")));+ def.setFirelensConfiguration(parseFirelensConfiguration(item.path("FirelensConfiguration")));if (item.path("Command").isArray()) {def.setCommand(toStringList(item.path("Command")));}@@ -281,6 +286,31 @@ public class EcsCfnProvisioner implements CfnResourceProvisioner {return result;}+ private static LogConfiguration parseLogConfiguration(JsonNode node) {+ if (node == null || !node.isObject() || !node.hasNonNull("LogDriver")) {+ return null;+ }+ return new LogConfiguration(node.path("LogDriver").asText(),+ parseOptions(node.path("Options")),+ node.has("SecretOptions") ? parseSecrets(node.path("SecretOptions")) : null);+ }++ private static FirelensConfiguration parseFirelensConfiguration(JsonNode node) {+ if (node == null || !node.isObject() || !node.hasNonNull("Type")) {+ return null;+ }+ return new FirelensConfiguration(node.path("Type").asText(), parseOptions(node.path("Options")));+ }++ private static LinkedHashMap<String, String> parseOptions(JsonNode node) {+ if (node == null || !node.isObject()) {+ return null;+ }+ LinkedHashMap<String, String> options = new LinkedHashMap<>();+ node.fields().forEachRemaining(entry -> options.put(entry.getKey(), entry.getValue().asText()));+ return options;+ }+private static List<PortMapping> parsePortMappings(JsonNode node) {List<PortMapping> result = new ArrayList<>();if (node == null || !node.isArray()) {diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandler.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandler.javaindex 47ab287962..b036f56703 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandler.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandler.java@@ -18,6 +18,7 @@ import io.github.hectorvent.floci.services.ecs.model.EcsCluster;import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;import io.github.hectorvent.floci.services.ecs.model.KeyValuePair;import io.github.hectorvent.floci.services.ecs.model.LaunchType;import io.github.hectorvent.floci.services.ecs.model.LogConfiguration;@@ -1138,7 +1139,6 @@ public class EcsJsonHandler {}n.set("firelensConfiguration", firelensNode);}-if (def.getHealthCheck() != null) {HealthCheck hc = def.getHealthCheck();ObjectNode hcNode = objectMapper.createObjectNode();@@ -1497,7 +1497,8 @@ public class EcsJsonHandler {}def.setMountPoints(parseMountPoints(item.path("mountPoints")));def.setLogConfiguration(parseLogConfiguration(item.path("logConfiguration")));- def.setFirelensConfiguration(parseFirelensConfiguration(item.path("firelensConfiguration")));+ def.setFirelensConfiguration(parseFirelensConfiguration(+ item.path("firelensConfiguration"), result.size() + 1));if (item.has("healthCheck")) {def.setHealthCheck(parseHealthCheck(item.path("healthCheck")));}@@ -1585,20 +1586,33 @@ public class EcsJsonHandler {return new LogConfiguration(logDriver, options, secretOptions);}- private FirelensConfiguration parseFirelensConfiguration(JsonNode node) {- if (node == null || !node.isObject() || !node.has("type")) {+ private FirelensConfiguration parseFirelensConfiguration(JsonNode node, int containerIndex) {+ if (node == null || !node.isObject()) {return null;}+ if (!node.hasNonNull("type")) {+ throw new AwsException("ClientException",+ "1 validation error detected: Value null at 'containerDefinitions." + containerIndex+ + ".member.firelensConfiguration.type' failed to satisfy constraint: Member must not be null",+ 400);+ }+ String type = node.path("type").asText();+ if (!"fluentd".equals(type) && !"fluentbit".equals(type)) {+ throw new AwsException("ClientException",+ "1 validation error detected: Value '" + type + "' at 'containerDefinitions." + containerIndex+ + ".member.firelensConfiguration.type' failed to satisfy constraint: "+ + "Member must satisfy enum value set: [fluentd, fluentbit]",+ 400);+ }Map<String, String> options = null;if (node.path("options").isObject()) {- Map<String, String> parsed = new LinkedHashMap<>();+ LinkedHashMap<String, String> parsed = new LinkedHashMap<>();node.path("options").fields().forEachRemaining(entry -> parsed.put(entry.getKey(), entry.getValue().asText()));options = parsed;}- return new FirelensConfiguration(node.path("type").asText(), options);+ return new FirelensConfiguration(type, options);}-private HealthCheck parseHealthCheck(JsonNode node) {if (node == null || !node.isObject()) {return null;diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.javaindex ac37ef3712..d00166cd8d 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsService.java@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.ecs;import com.fasterxml.jackson.core.type.TypeReference;import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsArnUtils;import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.core.common.ContainerTeardown;import io.github.hectorvent.floci.core.common.RegionResolver;@@ -24,6 +25,7 @@ import io.github.hectorvent.floci.services.ecs.model.EcsCluster;import io.github.hectorvent.floci.services.ecs.model.EcsLoadBalancer;import io.github.hectorvent.floci.services.ecs.model.EcsServiceModel;import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;import io.github.hectorvent.floci.services.ecs.model.LaunchType;import io.github.hectorvent.floci.services.ecs.model.NetworkConfiguration;import io.github.hectorvent.floci.services.ecs.model.NetworkMode;@@ -61,7 +63,6 @@ import java.util.stream.Stream;import io.github.hectorvent.floci.core.resource.ExplorerResource;import io.github.hectorvent.floci.core.resource.ResourceProvider;import io.github.hectorvent.floci.core.resource.SupportedResourceType;-import io.github.hectorvent.floci.core.common.AwsArnUtils;@ApplicationScopedpublic class EcsService implements ContainerTeardown, ResourceProvider, Resettable {@@ -370,7 +371,8 @@ public class EcsService implements ContainerTeardown, ResourceProvider, ResettabString taskRoleArn, String executionRoleArn,List<String> requiresCompatibilities,Map<String, String> tags, String region) {- if (requiresCompatibilities != null && requiresCompatibilities.contains("FARGATE")) {+ boolean fargate = requiresCompatibilities != null && requiresCompatibilities.contains("FARGATE");+ if (fargate) {if (networkMode != NetworkMode.awsvpc) {throw new AwsException("ClientException", "Fargate only supports network mode 'awsvpc'.", 400);}@@ -384,6 +386,7 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettabthrow new AwsException("ClientException", "No Fargate configuration exists for given values.", 400);}}+ validateFirelensS3Config(containerDefs, fargate);int revision = latestRevisions.merge(family, 1, Integer::sum);TaskDefinition td = new TaskDefinition();@@ -415,6 +418,40 @@ public class EcsService implements ContainerTeardown, ResourceProvider, Resettabreturn td;}+ /**+ * RegisterTaskDefinition-time validation of a FireLens config that comes from S3, in the+ * order ECS applies it.+ *+ * <p>The Fargate platform cannot pull the object itself, so ECS rejects the combination+ * outright with this exact wording. A Fargate task can still take its config from S3 the way+ * AWS documents, by giving the aws-for-fluent-bit init process its+ * {@code aws_fluent_bit_init_s3_*} environment variables, which ECS never inspects. An+ * EC2-compatible task definition keeps {@code s3}: the agent pulls it, and Floci reads the+ * object from its own S3 at launch.+ *+ * <p>A {@code config-file-value} that does not name an S3 object is rejected as an ARN syntax+ * error, which is what the RegisterTaskDefinition API returns for this field.+ */+ private static void validateFirelensS3Config(List<ContainerDefinition> containerDefs, boolean fargate) {+ if (containerDefs == null) {+ return;+ }+ for (ContainerDefinition def : containerDefs) {+ FirelensConfiguration firelens = def.getFirelensConfiguration();+ if (firelens == null || firelens.options() == null+ || !"s3".equals(firelens.options().get("config-file-type"))) {+ continue;+ }+ if (fargate) {+ throw new AwsException("ClientException",+ "Fargate launch type does not support FirelensConfiguration config file from 's3'", 400);+ }+ if (AwsArnUtils.parseS3ObjectArn(firelens.options().get("config-file-value")) == null) {+ throw new AwsException("ClientException", "Invalid arn syntax", 400);+ }+ }+ }+private boolean isValidFargateCpuMemory(String cpuStr, String memStr) {if (cpuStr == null || memStr == null) return false;int cpu = parseCpu(cpuStr);diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.javaindex 7d9cc73fcb..2a9c0e25c9 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java@@ -25,6 +25,8 @@ import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;import io.github.hectorvent.floci.services.ecs.model.ContainerOverride;import io.github.hectorvent.floci.services.ecs.model.EcsTask;import io.github.hectorvent.floci.services.ecs.model.EfsVolumeConfiguration;+import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;+import io.github.hectorvent.floci.services.ecs.model.LogConfiguration;import io.github.hectorvent.floci.services.ecs.model.MountPoint;import io.github.hectorvent.floci.services.ecs.model.NetworkBinding;import io.github.hectorvent.floci.services.ecs.model.NetworkMode;@@ -32,17 +34,26 @@ import io.github.hectorvent.floci.services.ecs.model.PortMapping;import io.github.hectorvent.floci.services.ecs.model.Secret;import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;import io.github.hectorvent.floci.services.ecs.model.Volume;+import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.s3.model.S3Object;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;import io.github.hectorvent.floci.services.ssm.SsmService;import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.command.InspectContainerResponse;import com.github.dockerjava.api.exception.NotFoundException;+import com.github.dockerjava.api.model.ContainerNetwork;import com.github.dockerjava.api.model.ExposedPort;import com.github.dockerjava.api.model.LogConfig;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;+import org.apache.commons.compress.archivers.tar.TarArchiveEntry;+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;import org.jboss.logging.Logger;import java.io.Closeable;+import java.io.ByteArrayInputStream;+import java.io.ByteArrayOutputStream;+import java.io.IOException;import java.nio.charset.StandardCharsets;import java.security.MessageDigest;import java.security.NoSuchAlgorithmException;@@ -54,7 +65,6 @@ import java.util.LinkedHashMap;import java.util.List;import java.util.Map;import java.util.Optional;-import java.util.OptionalInt;import java.util.Set;import java.util.stream.Stream;@@ -76,60 +86,12 @@ public class EcsContainerManager {private final LaunchedContainerAwsEnv awsEnv;private final SsmService ssmService;private final SecretsManagerService secretsManagerService;+ private final S3Service s3Service;private final EcrRegistryManager ecrRegistryManager;private final HostVolumePolicy hostVolumePolicy;private Ec2Service ec2Service;private SecurityGroupFirewallManager firewallManager;- static LogConfig firelensLogConfig(String taskId, String containerName, int fluentdPort) {- return new LogConfig(LogConfig.LoggingType.FLUENTD, Map.of(- "fluentd-address", "127.0.0.1:" + fluentdPort,- "tag", taskId + "." + containerName,- "fluentd-async", "true"));- }-- static List<ContainerDefinition> orderFirelensContainers(List<ContainerDefinition> definitions) {- ContainerDefinition router = firelensRouter(definitions);- boolean hasFirelensApplication = definitions.stream().anyMatch(EcsContainerManager::usesFirelens);- if (router == null) {- if (hasFirelensApplication) {- throw new AwsException("ClientException",- "An awsfirelens log configuration requires a FireLens log router.", 400);- }- return definitions;- }- String type = router.getFirelensConfiguration().type();- if (!"fluentbit".equalsIgnoreCase(type) && !"fluentd".equalsIgnoreCase(type)) {- throw new AwsException("ClientException",- "FireLens configuration type must be fluentbit or fluentd.", 400);- }- if (router.getPortMappings() != null && router.getPortMappings().stream()- .anyMatch(port -> port.containerPort() == 24224)) {- throw new AwsException("ClientException",- "FireLens port 24224 must not be exposed.", 400);- }- List<ContainerDefinition> ordered = new ArrayList<>(definitions.size());- ordered.add(router);- definitions.stream().filter(definition -> definition != router).forEach(ordered::add);- return ordered;- }-- private static ContainerDefinition firelensRouter(List<ContainerDefinition> definitions) {- List<ContainerDefinition> routers = definitions.stream()- .filter(definition -> definition.getFirelensConfiguration() != null)- .toList();- if (routers.size() > 1) {- throw new AwsException("ClientException",- "A task definition can have only one FireLens log router.", 400);- }- return routers.isEmpty() ? null : routers.getFirst();- }-- private static boolean usesFirelens(ContainerDefinition definition) {- return definition.getLogConfiguration() != null- && "awsfirelens".equalsIgnoreCase(definition.getLogConfiguration().logDriver());- }-@Injectpublic EcsContainerManager(ContainerBuilder containerBuilder,ContainerLifecycleManager lifecycleManager,@@ -140,12 +102,13 @@ public class EcsContainerManager {LaunchedContainerAwsEnv awsEnv,SsmService ssmService,SecretsManagerService secretsManagerService,+ S3Service s3Service,EcrRegistryManager ecrRegistryManager,HostVolumePolicy hostVolumePolicy,Ec2Service ec2Service,SecurityGroupFirewallManager firewallManager) {this(containerBuilder, lifecycleManager, logStreamer, containerDetector, config, regionResolver,- awsEnv, ssmService, secretsManagerService, ecrRegistryManager, hostVolumePolicy);+ awsEnv, ssmService, secretsManagerService, s3Service, ecrRegistryManager, hostVolumePolicy);this.ec2Service = ec2Service;this.firewallManager = firewallManager;}@@ -159,6 +122,7 @@ public class EcsContainerManager {LaunchedContainerAwsEnv awsEnv,SsmService ssmService,SecretsManagerService secretsManagerService,+ S3Service s3Service,EcrRegistryManager ecrRegistryManager,HostVolumePolicy hostVolumePolicy) {this.containerBuilder = containerBuilder;@@ -170,6 +134,7 @@ public class EcsContainerManager {this.awsEnv = awsEnv;this.ssmService = ssmService;this.secretsManagerService = secretsManagerService;+ this.s3Service = s3Service;this.hostVolumePolicy = hostVolumePolicy;this.ecrRegistryManager = ecrRegistryManager;}@@ -205,180 +170,208 @@ public class EcsContainerManager {}Map<String, ContainerOverride> overridesByName = overridesByName(containerOverrides);- List<ContainerDefinition> orderedDefinitions = orderFirelensContainers(- taskDef.getContainerDefinitions());- ContainerDefinition firelensRouter = firelensRouter(orderedDefinitions);- if (firelensRouter != null && taskDef.getNetworkMode() == NetworkMode.awsvpc- && firewallManager != null && firewallManager.enabled()) {- throw new AwsException("ClientException",- "FireLens is not supported with security-group enforcement.", 400);- }- OptionalInt firelensHostPort = OptionalInt.empty();Map<ContainerDefinition, List<String>> envVarsByContainer = new LinkedHashMap<>();// Resolved before any container is created, so a registry-startup failure can't leak one already started.Map<ContainerDefinition, String> imagesByContainer = new LinkedHashMap<>();- for (ContainerDefinition def : orderedDefinitions) {+ for (ContainerDefinition def : taskDef.getContainerDefinitions()) {envVarsByContainer.put(def, buildEnvVars(def, overridesByName.get(def.getName()), region));imagesByContainer.put(def, ecrRegistryManager.rewriteImageUri(def.getImage()));}+ ContainerDefinition firelensRouter = findFirelensRouter(taskDef.getContainerDefinitions());+ Map<String, Map<String, String>> firelensLogOptions =+ awsFirelensLogOptions(taskDef.getContainerDefinitions());+ if (!firelensLogOptions.isEmpty() && firelensRouter == null) {+ throw new AwsException("ClientException",+ "awsfirelens log driver requires a firelensConfiguration container", 400);+ }+PreparedNetwork protectedNetwork = prepareNetwork(task, taskDef, region, taskId);+ String firelensVolumeName = null;+ String firelensSocketAddress = null;+ String firelensConfig = null;+ String firelensExternalConfig = null;+ if (firelensRouter != null) {+ firelensConfig = firelensConfig(task, taskDef, firelensRouter, firelensLogOptions);+ firelensExternalConfig = s3ExternalConfig(firelensRouter);+ firelensVolumeName = ContainerStorageHelper.dockerName(config, "floci-ecs-firelens-" + taskId);+ lifecycleManager.ensureVolume(firelensVolumeName);+ firelensSocketAddress = unixSocketAddress(firelensVolumeName);+ }++ List<ContainerDefinition> launchOrder =+ launchOrder(taskDef.getContainerDefinitions(), firelensRouter);+ String fluentHost = null;+ String networkModeName = taskDef.getNetworkMode() != null+ ? taskDef.getNetworkMode().name()+ : NetworkMode.bridge.name();+try {- for (ContainerDefinition def : orderedDefinitions) {- String containerName = ContainerStorageHelper.dockerName(config, "floci-ecs-" + taskId + "-" + def.getName());-- // RunTask containerOverrides matched by container name: command replaces- // the task-def command; environment is merged over the task-def environment.- ContainerOverride override = overridesByName.get(def.getName());-- // Build container spec- ContainerBuilder.Builder specBuilder = containerBuilder.newContainer(imagesByContainer.get(def))- .withName(containerName)- .withEnv(envVarsByContainer.get(def))- .withDockerNetwork(config.services().ecs().dockerNetwork())- // Resolve Floci's endpoint from inside the task container the same way Lambda- // containers do: host.docker.internal on Linux, plus Floci's embedded DNS so the- // reachable AWS_ENDPOINT_URL hostname resolves to Floci instead of the container's- // own loopback.- .withHostDockerInternalOnLinux()- .withEmbeddedDns()- .withLabels(ContainerStorageHelper.resourceIdentityLabels(- "ecs", taskId, regionResolver.getAccountId(), region));- boolean firelensApplication = usesFirelens(def);- if (!firelensApplication) {- specBuilder.withLogRotation();- }- if (protectedNetwork != null) {- specBuilder.withNetworkMode("container:" + protectedNetwork.namespace().helperId());- specBuilder.withLabels(Map.of("floci.security-group-workload", "true"));- } else if (firelensApplication) {- String routerId = containerIds.get(firelensRouter.getName());- specBuilder.withNetworkMode("container:" + routerId);- }- if (def == firelensRouter) {- specBuilder.withLoopbackPortBinding(24224, 0);- }-- // Add memory limit if specified- if (def.getMemory() != null) {- specBuilder.withMemoryMb(def.getMemory());- }-- // Add port mappings. In bridge/host mode an explicit hostPort is- // published to the Docker host literally, matching AWS bridge mode- // (mirrors the ECR registry's fixed-port publishing). In awsvpc mode- // every AWS task gets its own ENI, so a literal hostPort carries no- // host-binding semantics and would collide across tasks on the single- // local Docker host (#1778) — awsvpc mappings always get a dynamic- // host port in native mode, or expose-only in Docker mode where ECS- // consumers reach containers via the docker network IP.- if (protectedNetwork == null && !firelensApplication && def.getPortMappings() != null) {- boolean awsvpc = taskDef.getNetworkMode() == NetworkMode.awsvpc;- boolean publishToHost = !containerDetector.isRunningInContainer();- for (PortMapping pm : def.getPortMappings()) {- if (!awsvpc && pm.hostPort() > 0) {- specBuilder.withPortBinding(pm.containerPort(), pm.hostPort());- } else if (publishToHost) {- specBuilder.withDynamicPort(pm.containerPort());- } else {- specBuilder.withExposedPort(pm.containerPort());- }+ for (ContainerDefinition def : launchOrder) {+ String containerName = ContainerStorageHelper.dockerName(config, "floci-ecs-" + taskId + "-" + def.getName());++ // RunTask containerOverrides matched by container name: command replaces+ // the task-def command; environment is merged over the task-def environment.+ ContainerOverride override = overridesByName.get(def.getName());++ List<String> env = new ArrayList<>(envVarsByContainer.get(def));+ if (fluentHost != null && def != firelensRouter+ && FirelensConfigGenerator.addsTcpForward(networkModeName)) {+ env.add("FLUENT_HOST=" + fluentHost);+ env.add("FLUENT_PORT=" + FirelensConfigGenerator.FORWARD_PORT);}- }- // Add command and entrypoint if specified. An override command (from- // RunTask containerOverrides) takes precedence over the task-def command.- List<String> effectiveCommand =- (override != null && override.getCommand() != null && !override.getCommand().isEmpty())- ? override.getCommand()- : def.getCommand();- if (effectiveCommand != null && !effectiveCommand.isEmpty()) {- specBuilder.withCmd(effectiveCommand);- }- if (def.getEntryPoint() != null && !def.getEntryPoint().isEmpty()) {- specBuilder.withEntrypoint(def.getEntryPoint());- }+ // Build container spec+ ContainerBuilder.Builder specBuilder = containerBuilder.newContainer(imagesByContainer.get(def))+ .withName(containerName)+ .withEnv(env)+ .withDockerNetwork(config.services().ecs().dockerNetwork())+ // Resolve Floci's endpoint from inside the task container the same way Lambda+ // containers do: host.docker.internal on Linux, plus Floci's embedded DNS so the+ // reachable AWS_ENDPOINT_URL hostname resolves to Floci instead of the container's+ // own loopback.+ .withHostDockerInternalOnLinux()+ .withEmbeddedDns()+ .withLabels(ContainerStorageHelper.resourceIdentityLabels(+ "ecs", taskId, regionResolver.getAccountId(), region));+ if (protectedNetwork != null) {+ specBuilder.withNetworkMode("container:" + protectedNetwork.namespace().helperId());+ specBuilder.withLabels(Map.of("floci.security-group-workload", "true"));+ }- if (firelensApplication) {- specBuilder.withLogConfig(firelensLogConfig(- taskId, def.getName(), firelensHostPort.orElseThrow()));- }+ boolean awsFirelens = isAwsFirelens(def);+ if (awsFirelens) {+ specBuilder.withLogConfig(awsFirelensLogConfig(def, taskId, firelensSocketAddress));+ } else {+ specBuilder.withLogRotation();+ }- // Bind-mount task-level volumes referenced by this container's mountPoints.- // The host source path resolves on the Docker daemon (sibling-container launch),- // so it must be an absolute host path. Unresolved volume references are skipped.- if (def.getMountPoints() != null) {- for (MountPoint mp : def.getMountPoints()) {- if (mp.containerPath() == null) {- continue;- }- String sourcePath = volumeSourcePaths.get(mp.sourceVolume());- EfsVolumeConfiguration efs = efsVolumes.get(mp.sourceVolume());- if (sourcePath != null) {- // Host volume: bind-mount an absolute path on the Docker host. Re-validate- // here (not just at RegisterTaskDefinition time): this narrows the window- // between validation and mount for a symlink swapped in afterward, and- // also covers task definitions persisted before this policy existed. A- // rejection is not swallowed, it propagates out of startTask and is- // surfaced by EcsService as the task's stoppedReason.- hostVolumePolicy.validate(sourcePath);- if (mp.readOnly()) {- specBuilder.withReadOnlyBind(sourcePath, mp.containerPath());+ if (def == firelensRouter && firelensVolumeName != null) {+ specBuilder.withNamedVolume(firelensVolumeName, "/var/run");+ }++ // Add memory limit if specified+ if (def.getMemory() != null) {+ specBuilder.withMemoryMb(def.getMemory());+ }++ // Add port mappings. In bridge/host mode an explicit hostPort is+ // published to the Docker host literally, matching AWS bridge mode+ // (mirrors the ECR registry's fixed-port publishing). In awsvpc mode+ // every AWS task gets its own ENI, so a literal hostPort carries no+ // host-binding semantics and would collide across tasks on the single+ // local Docker host (#1778) — awsvpc mappings always get a dynamic+ // host port in native mode, or expose-only in Docker mode where ECS+ // consumers reach containers via the docker network IP.+ if (protectedNetwork == null && def.getPortMappings() != null) {+ boolean awsvpc = taskDef.getNetworkMode() == NetworkMode.awsvpc;+ boolean publishToHost = !containerDetector.isRunningInContainer();+ for (PortMapping pm : def.getPortMappings()) {+ if (!awsvpc && pm.hostPort() > 0) {+ specBuilder.withPortBinding(pm.containerPort(), pm.hostPort());+ } else if (publishToHost) {+ specBuilder.withDynamicPort(pm.containerPort());} else {- specBuilder.withBind(sourcePath, mp.containerPath());+ specBuilder.withExposedPort(pm.containerPort());}- } else if (efs != null) {- mountEfsVolume(specBuilder, efs, mp);- } else {- LOG.warnv("Skipping mountPoint with unresolved volume {0} on container {1}",- mp.sourceVolume(), def.getName());}}- }-- ContainerSpec spec = specBuilder.build();- // Create and start container- ContainerInfo info = lifecycleManager.createAndStart(spec);- String dockerId = info.containerId();-- if (def == firelensRouter) {- firelensHostPort = info.publishedHostPort(24224);- if (firelensHostPort.isEmpty()) {- throw new AwsException("ClientException",- "The FireLens log router did not publish port 24224.", 400);+ // Add command and entrypoint if specified. An override command (from+ // RunTask containerOverrides) takes precedence over the task-def command.+ List<String> effectiveCommand =+ (override != null && override.getCommand() != null && !override.getCommand().isEmpty())+ ? override.getCommand()+ : def.getCommand();+ if (effectiveCommand != null && !effectiveCommand.isEmpty()) {+ specBuilder.withCmd(effectiveCommand);+ }+ if (def.getEntryPoint() != null && !def.getEntryPoint().isEmpty()) {+ specBuilder.withEntrypoint(def.getEntryPoint());}- }-- LOG.infov("Created ECS container {0} for task {1} container {2}", dockerId, taskId, def.getName());-- // Resolve network bindings for ECS-specific model- List<NetworkBinding> networkBindings = resolveNetworkBindings(- protectedNetwork == null ? dockerId : protectedNetwork.namespace().helperId(), def);- // Build ECS container model- Container container = buildContainer(task.getTaskArn(), def, dockerId, networkBindings, region);- runtimeContainers.add(container);- containerIds.put(def.getName(), dockerId);+ // Bind-mount task-level volumes referenced by this container's mountPoints.+ // The host source path resolves on the Docker daemon (sibling-container launch),+ // so it must be an absolute host path. Unresolved volume references are skipped.+ if (def.getMountPoints() != null) {+ for (MountPoint mp : def.getMountPoints()) {+ if (mp.containerPath() == null) {+ continue;+ }+ String sourcePath = volumeSourcePaths.get(mp.sourceVolume());+ EfsVolumeConfiguration efs = efsVolumes.get(mp.sourceVolume());+ if (sourcePath != null) {+ // Host volume: bind-mount an absolute path on the Docker host. Re-validate+ // here (not just at RegisterTaskDefinition time): this narrows the window+ // between validation and mount for a symlink swapped in afterward, and+ // also covers task definitions persisted before this policy existed. A+ // rejection is not swallowed, it propagates out of startTask and is+ // surfaced by EcsService as the task's stoppedReason.+ hostVolumePolicy.validate(sourcePath);+ if (mp.readOnly()) {+ specBuilder.withReadOnlyBind(sourcePath, mp.containerPath());+ } else {+ specBuilder.withBind(sourcePath, mp.containerPath());+ }+ } else if (efs != null) {+ mountEfsVolume(specBuilder, efs, mp);+ } else {+ LOG.warnv("Skipping mountPoint with unresolved volume {0} on container {1}",+ mp.sourceVolume(), def.getName());+ }+ }+ }- // Attach log streaming- String logGroup = "/ecs/" + taskDef.getFamily();- String logStream = logStreamer.generateLogStreamName(def.getName() + "/" + taskId);+ ContainerSpec spec = specBuilder.build();++ String dockerId;+ if (def == firelensRouter) {+ dockerId = lifecycleManager.create(spec);+ try {+ copyFirelensConfig(dockerId, firelensConfig, isFluentdRouter(firelensRouter),+ firelensExternalConfig);+ lifecycleManager.startCreated(dockerId, spec);+ } catch (RuntimeException e) {+ lifecycleManager.removeIfExists(dockerId);+ throw e;+ }+ fluentHost = inspectContainerIp(dockerId);+ } else {+ ContainerInfo info = lifecycleManager.createAndStart(spec);+ dockerId = info.containerId();+ }- if (!firelensApplication) {- Closeable logHandle = logStreamer.attach(- dockerId, logGroup, logStream, region,- "ecs:" + taskDef.getFamily() + ":" + def.getName());- if (logHandle != null) {- logStreamsByContainerId.put(dockerId, logHandle);+ LOG.infov("Created ECS container {0} for task {1} container {2}", dockerId, taskId, def.getName());++ // Resolve network bindings for ECS-specific model+ List<NetworkBinding> networkBindings = resolveNetworkBindings(+ protectedNetwork == null ? dockerId : protectedNetwork.namespace().helperId(), def);++ // Build ECS container model+ Container container = buildContainer(task.getTaskArn(), def, dockerId, networkBindings, region);+ runtimeContainers.add(container);+ containerIds.put(def.getName(), dockerId);++ // awsfirelens containers are shipped to Fluent Bit by Docker; don't also scrape json-file.+ if (!awsFirelens) {+ String logGroup = "/ecs/" + taskDef.getFamily();+ String logStream = logStreamer.generateLogStreamName(def.getName() + "/" + taskId);+ Closeable logHandle = logStreamer.attach(+ dockerId, logGroup, logStream, region,+ "ecs:" + taskDef.getFamily() + ":" + def.getName());+ if (logHandle != null) {+ logStreamsByContainerId.put(dockerId, logHandle);+ }}}- }} catch (Exception e) {for (String dockerId : containerIds.values()) {lifecycleManager.stopAndRemove(dockerId, null);}+ if (firelensVolumeName != null) {+ lifecycleManager.removeVolume(firelensVolumeName);+ }if (protectedNetwork != null) {firewallManager.unregister(protectedNetwork.eni().getNetworkInterfaceId());ec2Service.deleteNetworkInterface(region, protectedNetwork.eni().getNetworkInterfaceId());@@ -392,6 +385,7 @@ public class EcsContainerManager {task.setStartedAt(Instant.now());return new EcsTaskHandle(task.getTaskArn(), containerIds, logStreamsByContainerId,+ firelensVolumeName,protectedNetwork == null ? null : protectedNetwork.eni().getNetworkInterfaceId(), region);}@@ -473,6 +467,7 @@ public class EcsContainerManager {cleanupProtectedNetwork(handle);new ArrayList<>(handle.getLogStreamsByContainerId().keySet()).forEach(dockerId -> finalizeLogStream(handle, dockerId));+ removeFirelensVolume(handle);}/**@@ -517,9 +512,255 @@ public class EcsContainerManager {// Preserve handles for any container that still may be running after both operations failed.terminatedContainerIds.forEach(dockerId -> finalizeLogStream(handle, dockerId));cleanupProtectedNetwork(handle);+ removeFirelensVolume(handle);return exitCodes;}+ private void removeFirelensVolume(EcsTaskHandle handle) {+ if (handle == null || handle.getFirelensVolumeName() == null) {+ return;+ }+ lifecycleManager.removeVolume(handle.getFirelensVolumeName());+ }++ private static ContainerDefinition findFirelensRouter(List<ContainerDefinition> defs) {+ if (defs == null) {+ return null;+ }+ ContainerDefinition router = null;+ for (ContainerDefinition def : defs) {+ FirelensConfiguration firelens = def.getFirelensConfiguration();+ if (firelens == null) {+ continue;+ }+ if (!"fluentbit".equalsIgnoreCase(firelens.type()) && !"fluentd".equalsIgnoreCase(firelens.type())) {+ throw new AwsException("ClientException",+ "FireLens configuration type must be fluentbit or fluentd.", 400);+ }+ if (router != null) {+ throw new AwsException("ClientException",+ "A task definition can have only one FireLens log router.", 400);+ }+ if (def.getPortMappings() != null && def.getPortMappings().stream()+ .anyMatch(port -> port.containerPort() == 24224)) {+ throw new AwsException("ClientException",+ "FireLens port 24224 must not be exposed.", 400);+ }+ router = def;+ }+ return router;+ }++ private static boolean isFluentdRouter(ContainerDefinition router) {+ return router != null+ && router.getFirelensConfiguration() != null+ && "fluentd".equalsIgnoreCase(router.getFirelensConfiguration().type());+ }++ private static Map<String, Map<String, String>> awsFirelensLogOptions(List<ContainerDefinition> defs) {+ LinkedHashMap<String, Map<String, String>> options = new LinkedHashMap<>();+ if (defs == null) {+ return options;+ }+ for (ContainerDefinition def : defs) {+ if (!isAwsFirelens(def)) {+ continue;+ }+ LogConfiguration log = def.getLogConfiguration();+ options.put(def.getName(), log.options() != null ? log.options() : Map.of());+ }+ return options;+ }++ private static boolean isAwsFirelens(ContainerDefinition def) {+ LogConfiguration log = def.getLogConfiguration();+ return log != null && "awsfirelens".equals(log.logDriver());+ }++ private static List<ContainerDefinition> launchOrder(+ List<ContainerDefinition> defs, ContainerDefinition firelensRouter) {+ if (firelensRouter == null) {+ return defs;+ }+ List<ContainerDefinition> ordered = new ArrayList<>();+ ordered.add(firelensRouter);+ for (ContainerDefinition def : defs) {+ if (def != firelensRouter) {+ ordered.add(def);+ }+ }+ return ordered;+ }++ private FirelensConfigGenerator.Context firelensContext(+ EcsTask task, TaskDefinition taskDef, ContainerDefinition router,+ Map<String, Map<String, String>> logOptions) {+ FirelensConfiguration firelens = router.getFirelensConfiguration();+ Map<String, String> options = firelens.options() != null ? firelens.options() : Map.of();+ boolean metadata = !"false".equalsIgnoreCase(options.get("enable-ecs-log-metadata"));+ String external = null;+ String externalType = options.get("config-file-type");+ if ("file".equals(externalType)) {+ external = options.get("config-file-value");+ } else if ("s3".equals(externalType)) {+ // The agent downloads the object to this fixed path and @INCLUDEs it; Floci+ // writes it there directly with the generated config.+ external = isFluentdRouter(router)+ ? "/fluentd/etc/external.conf"+ : "/fluent-bit/etc/external.conf";+ }+ int memoryMb = 0;+ if (router.getMemoryReservation() != null) {+ memoryMb = router.getMemoryReservation();+ } else if (router.getMemory() != null) {+ memoryMb = router.getMemory();+ }+ String cluster = extractTaskId(task.getClusterArn());+ String familyRevision = taskDef.getFamily() + ":" + taskDef.getRevision();+ String networkMode = taskDef.getNetworkMode() != null+ ? taskDef.getNetworkMode().name()+ : NetworkMode.bridge.name();+ return new FirelensConfigGenerator.Context(+ networkMode, metadata, cluster, task.getTaskArn(), familyRevision,+ memoryMb, external, awsEnv.flociEndpoint(), logOptions);+ }++ private String firelensConfig(+ EcsTask task, TaskDefinition taskDef, ContainerDefinition router,+ Map<String, Map<String, String>> logOptions) {+ FirelensConfigGenerator.Context ctx = firelensContext(task, taskDef, router, logOptions);+ return isFluentdRouter(router)+ ? FirelensConfigGenerator.fluentdConfig(ctx)+ : FirelensConfigGenerator.fluentBitConfig(ctx);+ }++ private String unixSocketAddress(String volumeName) {+ try {+ String mountpoint = lifecycleManager.getDockerClient()+ .inspectVolumeCmd(volumeName).exec().getMountpoint();+ if (mountpoint == null || mountpoint.isBlank()) {+ throw new AwsException("ClientException",+ "FireLens socket volume has no mountpoint: " + volumeName, 500);+ }+ String path = mountpoint.endsWith("/")+ ? mountpoint + "fluent.sock"+ : mountpoint + "/fluent.sock";+ return "unix://" + path;+ } catch (AwsException e) {+ throw e;+ } catch (Exception e) {+ throw new AwsException("ClientException",+ "unable to resolve FireLens socket path: " + e.getMessage(), 500);+ }+ }++ private LogConfig awsFirelensLogConfig(ContainerDefinition def, String taskId, String socketAddress) {+ LinkedHashMap<String, String> opts = new LinkedHashMap<>();+ opts.put("fluentd-address", socketAddress);+ opts.put("fluentd-async", "true");+ opts.put("fluentd-sub-second-precision", "true");+ opts.put("tag", def.getName() + "-firelens-" + taskId);+ Map<String, String> logOptions = def.getLogConfiguration().options();+ if (logOptions != null && logOptions.get("log-driver-buffer-limit") != null) {+ opts.put("fluentd-buffer-limit", logOptions.get("log-driver-buffer-limit"));+ }+ return new LogConfig(LogConfig.LoggingType.FLUENTD, opts);+ }++ /**+ * Reads a {@code config-file-type=s3} external config from Floci's S3, before any container is+ * created, so a missing object stops the task without leaking a started router.+ *+ * <p>Wording is the ECS agent's, and the error code is ResourceInitializationError so that+ * EcsService passes the message through verbatim as the task's stoppedReason rather than+ * wrapping it in its generic start-failure prefix.+ */+ private String s3ExternalConfig(ContainerDefinition router) {+ FirelensConfiguration firelens = router.getFirelensConfiguration();+ Map<String, String> options = firelens.options() != null ? firelens.options() : Map.of();+ if (!"s3".equals(options.get("config-file-type"))) {+ return null;+ }+ String value = options.get("config-file-value");+ // Registration already rejected a value that is not an S3 object ARN; this covers task+ // definitions stored before that check existed.+ AwsArnUtils.S3ObjectRef ref = AwsArnUtils.parseS3ObjectArn(value);+ if (ref == null) {+ throw firelensS3Failure("unable to parse s3 arn: " + value, 400);+ }+ try {+ S3Object object = s3Service.getObject(ref.bucket(), ref.key());+ return new String(object.getData(), StandardCharsets.UTF_8);+ } catch (AwsException e) {+ throw firelensS3Failure("unable to download s3 config " + ref.key()+ + " from bucket " + ref.bucket() + ": " + e.getMessage(), e.getHttpStatus());+ }+ }++ /**+ * A task-resource failure that never reaches a client: EcsService catches it and uses the+ * message as the task's stoppedReason, keying off the ResourceInitializationError code to+ * skip its generic "Failed to start:" prefix. The prefix on the message itself is the agent's.+ */+ private static AwsException firelensS3Failure(String detail, int httpStatus) {+ return new AwsException("ResourceInitializationError",+ "Unable to download firelens s3 config file: " + detail, httpStatus);+ }++ private void copyFirelensConfig(+ String containerId, String configText, boolean fluentd, String externalConfig) {+ byte[] content = configText.getBytes(StandardCharsets.UTF_8);+ String fileName = fluentd ? "fluent.conf" : "fluent-bit.conf";+ String remotePath = fluentd ? "/fluentd/etc" : "/fluent-bit/etc";+ ByteArrayOutputStream archive = new ByteArrayOutputStream(content.length + 1024);+ try (TarArchiveOutputStream tar = new TarArchiveOutputStream(archive)) {+ putTarEntry(tar, fileName, content);+ if (externalConfig != null) {+ putTarEntry(tar, "external.conf", externalConfig.getBytes(StandardCharsets.UTF_8));+ }+ } catch (IOException e) {+ throw new AwsException("ClientException",+ "unable to write FireLens config: " + e.getMessage(), 500);+ }+ lifecycleManager.getDockerClient().copyArchiveToContainerCmd(containerId)+ .withRemotePath(remotePath)+ .withTarInputStream(new ByteArrayInputStream(archive.toByteArray()))+ .exec();+ }++ private static void putTarEntry(TarArchiveOutputStream tar, String name, byte[] content) throws IOException {+ TarArchiveEntry entry = new TarArchiveEntry(name);+ entry.setSize(content.length);+ tar.putArchiveEntry(entry);+ tar.write(content);+ tar.closeArchiveEntry();+ }++ private String inspectContainerIp(String dockerId) {+ try {+ InspectContainerResponse inspect = lifecycleManager.getDockerClient().inspectContainerCmd(dockerId).exec();+ Map<String, ContainerNetwork> networks = inspect.getNetworkSettings().getNetworks();+ String configured = config.services().ecs().dockerNetwork().orElse(null);+ if (configured != null && networks.get(configured) != null+ && isUsableIp(networks.get(configured).getIpAddress())) {+ return networks.get(configured).getIpAddress();+ }+ for (Map.Entry<String, ContainerNetwork> entry : networks.entrySet()) {+ if (!isDefaultDockerNetwork(entry.getKey())+ && isUsableIp(entry.getValue().getIpAddress())) {+ return entry.getValue().getIpAddress();+ }+ }+ for (ContainerNetwork net : networks.values()) {+ if (isUsableIp(net.getIpAddress())) {+ return net.getIpAddress();+ }+ }+ } catch (Exception e) {+ LOG.warnv("Could not resolve FireLens container IP for {0}: {1}", dockerId, e.getMessage());+ }+ return "127.0.0.1";+ }private void cleanupProtectedNetwork(EcsTaskHandle handle) {if (firewallManager != null && handle.getNetworkInterfaceId() != null) {firewallManager.unregister(handle.getNetworkInterfaceId());@@ -777,9 +1018,12 @@ public class EcsContainerManager {return container;}- private static String extractTaskId(String taskArn) {- int slash = taskArn.lastIndexOf('/');- return slash >= 0 ? taskArn.substring(slash + 1) : taskArn;+ private static String extractTaskId(String arn) {+ if (arn == null) {+ return "";+ }+ int slash = arn.lastIndexOf('/');+ return slash >= 0 ? arn.substring(slash + 1) : arn;}/**diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.javaindex 2447408b23..e1db6786c8 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java@@ -13,20 +13,22 @@ public class EcsTaskHandle {private final String taskArn;private final Map<String, String> containerIds; // containerName → dockerIdprivate final Map<String, Closeable> logStreamsByContainerId;+ private final String firelensVolumeName;private final String networkInterfaceId;private final String region;public EcsTaskHandle(String taskArn, Map<String, String> containerIds,Map<String, Closeable> logStreamsByContainerId) {- this(taskArn, containerIds, logStreamsByContainerId, null, null);+ this(taskArn, containerIds, logStreamsByContainerId, null, null, null);}public EcsTaskHandle(String taskArn, Map<String, String> containerIds,Map<String, Closeable> logStreamsByContainerId,- String networkInterfaceId, String region) {+ String firelensVolumeName, String networkInterfaceId, String region) {this.taskArn = taskArn;this.containerIds = new LinkedHashMap<>(containerIds);this.logStreamsByContainerId = new LinkedHashMap<>(logStreamsByContainerId);+ this.firelensVolumeName = firelensVolumeName;this.networkInterfaceId = networkInterfaceId;this.region = region;}@@ -34,6 +36,7 @@ public class EcsTaskHandle {public String getTaskArn() { return taskArn; }public Map<String, String> getContainerIds() { return containerIds; }public Map<String, Closeable> getLogStreamsByContainerId() { return logStreamsByContainerId; }+ public String getFirelensVolumeName() { return firelensVolumeName; }public String getNetworkInterfaceId() { return networkInterfaceId; }public String getRegion() { return region; }diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/FirelensConfigGenerator.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/FirelensConfigGenerator.javanew file mode 100644index 0000000000..97eec13589--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/FirelensConfigGenerator.java@@ -0,0 +1,342 @@+package io.github.hectorvent.floci.services.ecs.container;++import java.util.LinkedHashMap;+import java.util.Map;+import java.util.Set;++/**+ * Generates the FireLens config ECS would write for a task.+ *+ * <p>Fluent Bit shape is taken from amazon-ecs-agent {@code firelensconfig_unix.go}: unix-socket+ * input, optional TCP forward + healthcheck, ECS metadata {@code record_modifier}, optional+ * {@code @INCLUDE}, then one {@code [OUTPUT]} per {@code awsfirelens} container.+ * {@link #fluentdConfig} is the Fluentd equivalent ({@code unix}/{@code forward} sources,+ * {@code record_transformer}, {@code @include}, {@code <match>}; no healthcheck).+ */+final class FirelensConfigGenerator {++ static final String SOCKET_PATH = "/var/run/fluent.sock";+ static final int FORWARD_PORT = 24224;+ static final int HEALTHCHECK_PORT = 8877;++ /**+ * Output plugins that read a URL from their {@code endpoint} property.+ *+ * <p>Everything else is left alone. Fluent Bit fails at startup on a property the plugin+ * does not know, so an {@code Endpoint} line would break outputs such as {@code stdout},+ * {@code null}, {@code es} or {@code http} that name their destination with a different+ * key.+ *+ * <p>These three are the AWS-written Go plugins, which parse the value as a URL, so+ * Floci's {@code http://host:port} base URL reaches them. The upstream C plugins+ * ({@code cloudwatch_logs}, {@code kinesis_firehose}, {@code kinesis_streams}) hand the+ * value to {@code getaddrinfo} as a bare host name and always dial TLS: a URL fails as+ * "Misformatted domain name", and a bare host fails certificate verification against+ * Floci's plain HTTP port, with no output-level switch for either. Injecting an endpoint+ * there cannot work, so those outputs keep whatever the task definition gave them.+ */+ private static final Set<String> ENDPOINT_PLUGINS = Set.of(+ "s3", "cloudwatch", "firehose");++ private FirelensConfigGenerator() {+ }++ record Context(+ String networkMode,+ boolean ecsMetadataEnabled,+ String cluster,+ String taskArn,+ String taskDefinition,+ int routerMemoryMb,+ String externalConfigPath,+ String pluginEndpoint,+ Map<String, Map<String, String>> containerLogOptions+ ) {+ }++ static String fluentBitConfig(Context ctx) {+ StringBuilder out = new StringBuilder();+ appendInput(out, "forward", null, options(+ "unix_path", SOCKET_PATH,+ "Mem_Buf_Limit", memBufLimit(ctx.routerMemoryMb())));++ if (addsTcpForward(ctx.networkMode())) {+ appendInput(out, "forward", null, options(+ "Listen", tcpListen(ctx.networkMode()),+ "Port", String.valueOf(FORWARD_PORT)));+ appendInput(out, "tcp", "firelens-healthcheck", options(+ "Listen", "127.0.0.1",+ "Port", String.valueOf(HEALTHCHECK_PORT)));+ }++ if (ctx.containerLogOptions() != null) {+ for (Map.Entry<String, Map<String, String>> entry : ctx.containerLogOptions().entrySet()) {+ String tag = entry.getKey() + "-firelens*";+ Map<String, String> options = entry.getValue();+ if (options == null) {+ continue;+ }+ String include = options.get("include-pattern");+ if (include != null) {+ appendGrep(out, tag, "Regex", include);+ }+ String exclude = options.get("exclude-pattern");+ if (exclude != null) {+ appendGrep(out, tag, "Exclude", exclude);+ }+ }+ }++ if (ctx.ecsMetadataEnabled()) {+ out.append("[FILTER]\n");+ out.append(" Name record_modifier\n");+ out.append(" Match *\n");+ appendRecord(out, "ecs_cluster", ctx.cluster());+ appendRecord(out, "ecs_task_arn", ctx.taskArn());+ appendRecord(out, "ecs_task_definition", ctx.taskDefinition());+ out.append('\n');+ }++ if (ctx.externalConfigPath() != null && !ctx.externalConfigPath().isBlank()) {+ out.append("@INCLUDE ").append(ctx.externalConfigPath()).append("\n\n");+ }++ if (addsTcpForward(ctx.networkMode())) {+ appendOutput(out, "null", "firelens-healthcheck", Map.of());+ }++ if (ctx.containerLogOptions() != null) {+ for (Map.Entry<String, Map<String, String>> entry : ctx.containerLogOptions().entrySet()) {+ Map<String, String> options = entry.getValue() == null ? Map.of() : entry.getValue();+ String plugin = options.get("Name");+ Map<String, String> pluginOptions = pluginOptions(options);+ if (plugin == null) {+ if (!pluginOptions.isEmpty()) {+ throw new IllegalArgumentException(+ "missing output key Name which is required for firelens configuration of type fluentbit");+ }+ continue;+ }+ pluginOptions = withPluginEndpoint(plugin, pluginOptions, ctx.pluginEndpoint());+ appendOutput(out, plugin, entry.getKey() + "-firelens*", pluginOptions);+ }+ }+ return out.toString();+ }++ /**+ * Fluentd FireLens config. Output plugin type comes from log option {@code @type}.+ * AWS plugins here are Ruby gems, not the Fluent Bit Go/C plugins, so Floci does+ * not inject an {@code Endpoint} line.+ */+ static String fluentdConfig(Context ctx) {+ StringBuilder out = new StringBuilder();+ appendFluentdBlock(out, "source", null, "unix", options("path", SOCKET_PATH));++ if (addsTcpForward(ctx.networkMode())) {+ appendFluentdBlock(out, "source", null, "forward", options(+ "bind", tcpListen(ctx.networkMode()),+ "port", String.valueOf(FORWARD_PORT)));+ }++ if (ctx.containerLogOptions() != null) {+ for (Map.Entry<String, Map<String, String>> entry : ctx.containerLogOptions().entrySet()) {+ String tag = entry.getKey() + "-firelens**";+ Map<String, String> options = entry.getValue();+ if (options == null) {+ continue;+ }+ String include = options.get("include-pattern");+ if (include != null) {+ appendFluentdGrep(out, tag, "regexp", include);+ }+ String exclude = options.get("exclude-pattern");+ if (exclude != null) {+ appendFluentdGrep(out, tag, "exclude", exclude);+ }+ }+ }++ if (ctx.ecsMetadataEnabled()) {+ out.append("<filter **>\n");+ out.append(" @type record_transformer\n");+ out.append(" <record>\n");+ appendFluentdRecord(out, "ecs_cluster", ctx.cluster());+ appendFluentdRecord(out, "ecs_task_arn", ctx.taskArn());+ appendFluentdRecord(out, "ecs_task_definition", ctx.taskDefinition());+ out.append(" </record>\n");+ out.append("</filter>\n\n");+ }++ if (ctx.externalConfigPath() != null && !ctx.externalConfigPath().isBlank()) {+ out.append("@include ").append(ctx.externalConfigPath()).append("\n\n");+ }++ if (ctx.containerLogOptions() != null) {+ for (Map.Entry<String, Map<String, String>> entry : ctx.containerLogOptions().entrySet()) {+ Map<String, String> options = entry.getValue() == null ? Map.of() : entry.getValue();+ String plugin = options.get("@type");+ Map<String, String> pluginOptions = pluginOptions(options);+ if (plugin == null) {+ if (!pluginOptions.isEmpty()) {+ throw new IllegalArgumentException(+ "missing output key @type which is required for firelens configuration of type fluentd");+ }+ continue;+ }+ appendFluentdBlock(out, "match", entry.getKey() + "-firelens**", plugin, pluginOptions);+ }+ }+ return out.toString();+ }++ /**+ * Points an AWS output at Floci. The Fluent Bit AWS plugins read a custom endpoint only+ * from their own configuration, so the {@code AWS_ENDPOINT_URL} Floci injects into the+ * container does not reach them and the output would otherwise go to the real service.+ * An endpoint the task definition set explicitly is never overwritten.+ */+ private static Map<String, String> withPluginEndpoint(+ String plugin, Map<String, String> pluginOptions, String pluginEndpoint) {+ if (pluginEndpoint == null || !ENDPOINT_PLUGINS.contains(plugin)) {+ return pluginOptions;+ }+ String existingEndpoint = null;+ for (String key : pluginOptions.keySet()) {+ if ("endpoint".equalsIgnoreCase(key)) {+ existingEndpoint = pluginOptions.get(key);+ break;+ }+ }+ if (existingEndpoint == null) {+ pluginOptions.put("Endpoint", pluginEndpoint);+ existingEndpoint = pluginEndpoint;+ }+ disableTlsForHttpEndpoint(pluginOptions, existingEndpoint);+ return pluginOptions;+ }++ /**+ * Fluent Bit 1.9 (the {@code aws-for-fluent-bit} 2.x / {@code :latest} line) still+ * calls {@code flb_tls_session_create} on an HTTP S3 endpoint and SIGSEGVs on a NULL+ * TLS context. {@code tls Off} is what that version actually honours; the {@code http://}+ * scheme alone is not enough.+ */+ private static void disableTlsForHttpEndpoint(Map<String, String> pluginOptions, String endpoint) {+ if (endpoint == null || !endpoint.regionMatches(true, 0, "http://", 0, 7)) {+ return;+ }+ for (String key : pluginOptions.keySet()) {+ if ("tls".equalsIgnoreCase(key)) {+ return;+ }+ }+ pluginOptions.put("tls", "Off");+ }++ static boolean addsTcpForward(String networkMode) {+ return "bridge".equals(networkMode) || "awsvpc".equals(networkMode);+ }++ static String tcpListen(String networkMode) {+ // AWS binds 127.0.0.1 in awsvpc because every container shares the task's network+ // namespace. Floci runs each container in its own namespace, so the router must listen+ // on all interfaces for the injected FLUENT_HOST to be reachable from sibling containers.+ return "0.0.0.0";+ }++ static String memBufLimit(int routerMemoryMb) {+ int limit = routerMemoryMb / 2;+ if (limit <= 0) {+ limit = 25;+ }+ return limit + "MB";+ }++ static Map<String, String> pluginOptions(Map<String, String> options) {+ LinkedHashMap<String, String> pluginOptions = new LinkedHashMap<>();+ for (Map.Entry<String, String> option : options.entrySet()) {+ String key = option.getKey();+ if ("Name".equals(key) || "@type".equals(key) || "include-pattern".equals(key)+ || "exclude-pattern".equals(key) || "log-driver-buffer-limit".equals(key)) {+ continue;+ }+ pluginOptions.put(key, option.getValue());+ }+ return pluginOptions;+ }++ private static Map<String, String> options(String... keyValues) {+ LinkedHashMap<String, String> options = new LinkedHashMap<>();+ for (int i = 0; i < keyValues.length; i += 2) {+ options.put(keyValues[i], keyValues[i + 1]);+ }+ return options;+ }++ private static void appendInput(StringBuilder out, String name, String tag, Map<String, String> options) {+ out.append("[INPUT]\n");+ out.append(" Name ").append(name).append('\n');+ if (tag != null) {+ out.append(" Tag ").append(tag).append('\n');+ }+ for (Map.Entry<String, String> option : options.entrySet()) {+ out.append(" ").append(option.getKey()).append(' ').append(option.getValue()).append('\n');+ }+ out.append('\n');+ }++ private static void appendGrep(StringBuilder out, String tag, String kind, String pattern) {+ out.append("[FILTER]\n");+ out.append(" Name grep\n");+ out.append(" Match ").append(tag).append('\n');+ out.append(" ").append(kind).append(" log ").append(pattern).append('\n');+ out.append('\n');+ }++ private static void appendRecord(StringBuilder out, String key, String value) {+ if (value != null && !value.isBlank()) {+ out.append(" Record ").append(key).append(' ').append(value).append('\n');+ }+ }++ private static void appendOutput(StringBuilder out, String name, String tag, Map<String, String> options) {+ out.append("[OUTPUT]\n");+ out.append(" Name ").append(name).append('\n');+ out.append(" Match ").append(tag).append('\n');+ for (Map.Entry<String, String> option : options.entrySet()) {+ out.append(" ").append(option.getKey()).append(' ').append(option.getValue()).append('\n');+ }+ out.append('\n');+ }++ private static void appendFluentdBlock(+ StringBuilder out, String kind, String tag, String type, Map<String, String> options) {+ out.append('<').append(kind);+ if (tag != null) {+ out.append(' ').append(tag);+ }+ out.append(">\n");+ out.append(" @type ").append(type).append('\n');+ for (Map.Entry<String, String> option : options.entrySet()) {+ out.append(" ").append(option.getKey()).append(' ').append(option.getValue()).append('\n');+ }+ out.append("</").append(kind).append(">\n\n");+ }++ private static void appendFluentdGrep(StringBuilder out, String tag, String kind, String pattern) {+ out.append("<filter ").append(tag).append(">\n");+ out.append(" @type grep\n");+ out.append(" <").append(kind).append(">\n");+ out.append(" key log\n");+ out.append(" pattern ").append(pattern).append('\n');+ out.append(" </").append(kind).append(">\n");+ out.append("</filter>\n\n");+ }++ private static void appendFluentdRecord(StringBuilder out, String key, String value) {+ if (value != null && !value.isBlank()) {+ out.append(" ").append(key).append(' ').append(value).append('\n');+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/ContainerDefinition.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/ContainerDefinition.javaindex 4051ddffa4..d5c7a66525 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/model/ContainerDefinition.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/ContainerDefinition.java@@ -66,7 +66,6 @@ public class ContainerDefinition {public void setFirelensConfiguration(FirelensConfiguration firelensConfiguration) {this.firelensConfiguration = firelensConfiguration;}-public HealthCheck getHealthCheck() { return healthCheck; }public void setHealthCheck(HealthCheck healthCheck) { this.healthCheck = healthCheck; }}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/FirelensConfiguration.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/FirelensConfiguration.javaindex daaf2a010f..04e41e9d28 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/model/FirelensConfiguration.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/FirelensConfiguration.java@@ -4,6 +4,14 @@ import io.quarkus.runtime.annotations.RegisterForReflection;import java.util.Map;+/**+ * The {@code firelensConfiguration} of an ECS container definition:+ * {@code {"type": "fluentbit"|"fluentd", "options": {...}}}.+ *+ * <p>Marks a container as the task's log router and, when {@code enable-ecs-log-metadata} is set,+ * has Floci append ECS metadata fields to routed records. {@code fluentbit} and {@code fluentd}+ * routers are both acted on at launch.+ */@RegisterForReflectionpublic record FirelensConfiguration(String type, Map<String, String> options) {}diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/LogConfiguration.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/LogConfiguration.javaindex 7a9c6c251f..0a36fdf2d8 100644--- a/src/main/java/io/github/hectorvent/floci/services/ecs/model/LogConfiguration.java+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/LogConfiguration.java@@ -9,9 +9,9 @@ import java.util.Map;* The {@code logConfiguration} of an ECS container definition:* {@code {"logDriver": "awslogs", "options": {...}, "secretOptions": [{"name": ..., "valueFrom": ...}]}}.*- * <p>Modelled for RegisterTaskDefinition/DescribeTaskDefinition round-trip fidelity. Floci does not- * route container output to the configured driver — a local task's logs stay with its Docker- * container — so the {@code awslogs} options are stored and returned rather than acted upon.+ * <p>Stored and returned for RegisterTaskDefinition/DescribeTaskDefinition round-trip. At launch,+ * {@code awsfirelens} is routed through the task's FireLens container. Other drivers+ * (including {@code awslogs}) stay with the Docker container and Floci's CloudWatch log streamer.*/@RegisterForReflectionpublic record LogConfiguration(String logDriver, Map<String, String> options, List<Secret> secretOptions) {diff --git a/src/test/java/io/github/hectorvent/floci/core/common/AwsArnUtilsTest.java b/src/test/java/io/github/hectorvent/floci/core/common/AwsArnUtilsTest.javaindex 80ddb4f34e..51e2c53079 100644--- a/src/test/java/io/github/hectorvent/floci/core/common/AwsArnUtilsTest.java+++ b/src/test/java/io/github/hectorvent/floci/core/common/AwsArnUtilsTest.java@@ -7,6 +7,7 @@ import org.junit.jupiter.params.provider.ValueSource;import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertNull;import static org.junit.jupiter.api.Assertions.assertThrows;import static org.junit.jupiter.api.Assertions.assertTrue;@@ -200,6 +201,29 @@ class AwsArnUtilsTest {assertFalse(AwsArnUtils.isArnFor(null, "dynamodb"));}+ @Test+ void s3ObjectArnSplitsBucketAndKey() {+ assertEquals(new AwsArnUtils.S3ObjectRef("logs", "firelens/extra.conf"),+ AwsArnUtils.parseS3ObjectArn("arn:aws:s3:::logs/firelens/extra.conf"));+ assertEquals(new AwsArnUtils.S3ObjectRef("logs", "extra.conf"),+ AwsArnUtils.parseS3ObjectArn("arn:aws:s3:us-east-1:000000000000:logs/extra.conf"));+ }++ /**+ * Everything ECS rejects as "Invalid arn syntax" for a FireLens {@code config-file-value}:+ * not an ARN at all (the {@code s3://bucket/key} form users try), another service's ARN, and+ * an ARN with no key or no bucket. Null rather than an exception, because the same helper+ * backs the defensive path at task launch.+ */+ @Test+ void s3ObjectArnIsNullForAnythingThatIsNotAnS3Object() {+ assertNull(AwsArnUtils.parseS3ObjectArn("s3://logs/extra.conf"));+ assertNull(AwsArnUtils.parseS3ObjectArn("arn:aws:s3:::logs"));+ assertNull(AwsArnUtils.parseS3ObjectArn("arn:aws:s3:::logs/"));+ assertNull(AwsArnUtils.parseS3ObjectArn("arn:aws:sqs:us-east-1:000000000000:my-queue"));+ assertNull(AwsArnUtils.parseS3ObjectArn(null));+ }+@Testvoid queueUrlIsAccountThenQueueName() {assertEquals("http://localhost:4566/000000000000/my-queue",diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.javaindex 055ca7493b..8d18640018 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java@@ -153,6 +153,10 @@ class EcsCfnProvisionerTest {container.putArray("Environment").addObject().put("Name", "MODE").put("Value", "prod");container.putArray("Secrets").addObject().put("Name", "DB").put("ValueFrom", "arn:aws:secretsmanager:x");container.putArray("Command").add("run").add("--fast");+ ObjectNode log = container.putObject("LogConfiguration");+ log.put("LogDriver", "awsfirelens");+ log.putObject("Options").put("Name", "s3").put("bucket", "logs");+ container.putObject("FirelensConfiguration").put("Type", "fluentbit");StackResource r = resource("AWS::ECS::TaskDefinition", "TaskDef");provisioner.provision(r, props, ctx());@@ -174,6 +178,10 @@ class EcsCfnProvisionerTest {assertEquals("prod", def.getEnvironment().get(0).value());assertEquals("arn:aws:secretsmanager:x", def.getSecrets().get(0).valueFrom());assertEquals(List.of("run", "--fast"), def.getCommand());+ assertEquals("awsfirelens", def.getLogConfiguration().logDriver());+ assertEquals("s3", def.getLogConfiguration().options().get("Name"));+ assertEquals("logs", def.getLogConfiguration().options().get("bucket"));+ assertEquals("fluentbit", def.getFirelensConfiguration().type());}@Testdiff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsFirelensS3ConfigTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsFirelensS3ConfigTest.javanew file mode 100644index 0000000000..84136f1e7d--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsFirelensS3ConfigTest.java@@ -0,0 +1,150 @@+package io.github.hectorvent.floci.services.ecs;++import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;+import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;+import io.github.hectorvent.floci.services.ecs.model.KeyValuePair;+import io.github.hectorvent.floci.services.ecs.model.LaunchType;+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;+import io.github.hectorvent.floci.services.ecs.model.TaskStatus;+import org.junit.jupiter.api.Test;++import java.util.List;+import java.util.Map;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.when;++/**+ * FireLens config that comes from S3, on both sides of the split ECS draws:+ *+ * <p>{@code RegisterTaskDefinition} rejects the combination for Fargate (the platform cannot pull+ * the object) and rejects a {@code config-file-value} that is not an S3 object ARN, while an+ * EC2-compatible task definition keeps it and the agent pulls it at launch. A failed pull becomes+ * the task's stopped reason in the agent's own wording.+ *+ * <p>A Fargate task can still take its config from S3 the way AWS documents, by giving the+ * aws-for-fluent-bit init process its {@code aws_fluent_bit_init_s3_*} environment variables,+ * which ECS never inspects; the last test pins that Floci does not over-reach and reject it.+ */+class EcsFirelensS3ConfigTest {++ private static final String S3_CONFIG_ARN = "arn:aws:s3:::firelens-configs/extra.conf";++ @Test+ void registerTaskDefinitionRejectsS3FirelensConfigForFargate() {+ AwsException failure = assertThrows(AwsException.class, () -> service("us-east-1")+ .registerTaskDefinition("fluentbit-s3", List.of(firelensRouter("fluentbit", S3_CONFIG_ARN)),+ NetworkMode.awsvpc, "256", "512", null, null, List.of("FARGATE"), "us-east-1"));++ assertEquals("ClientException", failure.getErrorCode());+ assertEquals(400, failure.getHttpStatus());+ assertEquals("Fargate launch type does not support FirelensConfiguration config file from 's3'",+ failure.getMessage());+ }++ @Test+ void registerTaskDefinitionRejectsS3FirelensConfigForFargateWithAFluentdRouter() {+ AwsException failure = assertThrows(AwsException.class, () -> service("us-east-1")+ .registerTaskDefinition("fluentd-s3", List.of(firelensRouter("fluentd", S3_CONFIG_ARN)),+ NetworkMode.awsvpc, "256", "512", null, null, List.of("FARGATE"), "us-east-1"));++ assertEquals("Fargate launch type does not support FirelensConfiguration config file from 's3'",+ failure.getMessage());+ }++ @Test+ void registerTaskDefinitionRejectsAConfigFileValueThatIsNotAnS3ObjectArn() {+ AwsException failure = assertThrows(AwsException.class, () -> service("us-east-1")+ .registerTaskDefinition("bad-arn", List.of(firelensRouter("fluentbit", "s3://logs/extra.conf")),+ NetworkMode.bridge, null, null, null, null, List.of("EC2"), "us-east-1"));++ assertEquals("ClientException", failure.getErrorCode());+ assertEquals("Invalid arn syntax", failure.getMessage());+ }++ @Test+ void ec2TaskDefinitionKeepsItsS3FirelensConfig() {+ TaskDefinition td = service("us-east-1").registerTaskDefinition("ec2-s3",+ List.of(firelensRouter("fluentbit", S3_CONFIG_ARN)),+ NetworkMode.bridge, null, null, null, null, List.of("EC2"), "us-east-1");++ FirelensConfiguration firelens = td.getContainerDefinitions().getFirst().getFirelensConfiguration();+ assertEquals("s3", firelens.options().get("config-file-type"));+ assertEquals(S3_CONFIG_ARN, firelens.options().get("config-file-value"));+ }++ @Test+ void fargateTaskDefinitionAcceptsS3ConfigThroughTheInitProcessEnvironment() {+ ContainerDefinition router = new ContainerDefinition();+ router.setName("log_router");+ router.setImage("public.ecr.aws/aws-observability/aws-for-fluent-bit:init-latest");+ router.setFirelensConfiguration(new FirelensConfiguration("fluentbit", Map.of()));+ router.setEnvironment(List.of(new KeyValuePair("aws_fluent_bit_init_s3_1", S3_CONFIG_ARN)));++ TaskDefinition td = service("us-east-1").registerTaskDefinition("init-s3", List.of(router),+ NetworkMode.awsvpc, "256", "512", null, null, List.of("FARGATE"), "us-east-1");++ assertEquals("init-s3", td.getFamily());+ }++ @Test+ void runTaskStopsWithTheAgentReasonWhenTheS3ConfigCannotBeDownloaded() {+ EcsContainerManager containerManager = mock(EcsContainerManager.class);+ // EcsContainerManager reads the object before creating any container and reports the+ // failure with the agent's wording under a ResourceInitializationError code; EcsService+ // keys off that code to use the message as the stopped reason verbatim.+ when(containerManager.startTask(any(), any(), any(), eq("us-east-1")))+ .thenThrow(new AwsException("ResourceInitializationError",+ "Unable to download firelens s3 config file: unable to download s3 config "+ + "extra.conf from bucket firelens-configs: The specified key does not exist.",+ 404));++ EcsService service = service(containerManager, false);+ service.createCluster("test-cluster", "us-east-1");+ service.registerTaskDefinition("ec2-s3", List.of(firelensRouter("fluentbit", S3_CONFIG_ARN)),+ NetworkMode.bridge, null, null, null, null, List.of("EC2"), "us-east-1");++ List<EcsTask> tasks = service.runTask("test-cluster", "ec2-s3", 1,+ LaunchType.EC2, null, null, List.of(), null, "us-east-1");++ EcsTask task = tasks.getFirst();+ assertEquals(TaskStatus.STOPPED.name(), task.getLastStatus());+ assertEquals("Unable to download firelens s3 config file: unable to download s3 config "+ + "extra.conf from bucket firelens-configs: The specified key does not exist.",+ task.getStoppedReason());+ }++ private static ContainerDefinition firelensRouter(String type, String configFileValue) {+ ContainerDefinition router = new ContainerDefinition();+ router.setName("log_router");+ router.setImage("public.ecr.aws/aws-observability/aws-for-fluent-bit:stable");+ router.setFirelensConfiguration(new FirelensConfiguration(type, Map.of(+ "config-file-type", "s3",+ "config-file-value", configFileValue)));+ return router;+ }++ private static EcsService service(String region) {+ return service(mock(EcsContainerManager.class), true);+ }++ private static EcsService service(EcsContainerManager containerManager, boolean mockMode) {+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ when(config.services().ecs().mock()).thenReturn(mockMode);+ when(config.effectiveBaseUrl()).thenReturn("http://localhost:4566");+ return new EcsService(new RegionResolver("us-east-1", "000000000000"),+ containerManager, config, mock(EcsLoadBalancerRegistrar.class), null, null);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandlerRuntimePlatformLogConfigurationTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandlerRuntimePlatformLogConfigurationTest.javaindex ed573d0d8d..6792d63b68 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandlerRuntimePlatformLogConfigurationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandlerRuntimePlatformLogConfigurationTest.java@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.ecs;import com.fasterxml.jackson.databind.JsonNode;import com.fasterxml.jackson.databind.ObjectMapper;import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.services.ecs.container.HostVolumePolicy;import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;import jakarta.ws.rs.core.Response;@@ -12,6 +13,7 @@ import org.junit.jupiter.api.Test;import java.util.List;import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertThrows;import static org.junit.jupiter.api.Assertions.assertTrue;import static org.mockito.ArgumentMatchers.any;import static org.mockito.ArgumentMatchers.anyString;@@ -149,4 +151,89 @@ class EcsJsonHandlerRuntimePlatformLogConfigurationTest {assertTrue(td.path("containerDefinitions").get(0).path("logConfiguration").isMissingNode(),"logConfiguration must stay absent for a container that never set it");}++ @Test+ void registerTaskDefinitionRoundTripsFirelensConfiguration() throws Exception {+ String requestJson = """+ {+ "family": "firelens-family",+ "containerDefinitions": [+ {+ "name": "log_router",+ "image": "amazon/aws-for-fluent-bit:stable",+ "firelensConfiguration": {+ "type": "fluentbit",+ "options": {+ "enable-ecs-log-metadata": "true",+ "config-file-type": "file",+ "config-file-value": "/extra.conf"+ }+ }+ }+ ]+ }+ """;+ JsonNode request = objectMapper.readTree(requestJson);++ Response response = handler.handle("RegisterTaskDefinition", request, "us-east-1");+ JsonNode firelens = objectMapper.valueToTree(response.getEntity())+ .path("taskDefinition").path("containerDefinitions").get(0).path("firelensConfiguration");++ assertEquals("fluentbit", firelens.path("type").asText());+ assertEquals("true", firelens.path("options").path("enable-ecs-log-metadata").asText());+ assertEquals("file", firelens.path("options").path("config-file-type").asText());+ assertEquals("/extra.conf", firelens.path("options").path("config-file-value").asText());+ }++ @Test+ void registerTaskDefinitionRejectsFirelensConfigurationWithMissingType() throws Exception {+ JsonNode request = objectMapper.readTree("""+ {+ "family": "missing-type",+ "containerDefinitions": [+ {+ "name": "log_router",+ "image": "amazon/aws-for-fluent-bit:stable",+ "firelensConfiguration": {+ "options": {"enable-ecs-log-metadata": "true"}+ }+ }+ ]+ }+ """);++ AwsException ex = assertThrows(AwsException.class, () ->+ handler.handle("RegisterTaskDefinition", request, "us-east-1"));+ assertEquals("ClientException", ex.getErrorCode());+ assertEquals(400, ex.getHttpStatus());+ assertEquals("1 validation error detected: Value null at "+ + "'containerDefinitions.1.member.firelensConfiguration.type' failed to satisfy "+ + "constraint: Member must not be null",+ ex.getMessage());+ }++ @Test+ void registerTaskDefinitionRejectsFirelensConfigurationWithUnknownType() throws Exception {+ JsonNode request = objectMapper.readTree("""+ {+ "family": "unknown-type",+ "containerDefinitions": [+ {+ "name": "log_router",+ "image": "amazon/aws-for-fluent-bit:stable",+ "firelensConfiguration": {"type": "foo"}+ }+ ]+ }+ """);++ AwsException ex = assertThrows(AwsException.class, () ->+ handler.handle("RegisterTaskDefinition", request, "us-east-1"));+ assertEquals("ClientException", ex.getErrorCode());+ assertEquals(400, ex.getHttpStatus());+ assertEquals("1 validation error detected: Value 'foo' at "+ + "'containerDefinitions.1.member.firelensConfiguration.type' failed to satisfy "+ + "constraint: Member must satisfy enum value set: [fluentd, fluentbit]",+ ex.getMessage());+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandlerTaskDefinitionPersistenceTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandlerTaskDefinitionPersistenceTest.javaindex 97e4548ed0..057b65e6a1 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandlerTaskDefinitionPersistenceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandlerTaskDefinitionPersistenceTest.java@@ -11,6 +11,7 @@ import io.github.hectorvent.floci.core.storage.PersistentStorage;import io.github.hectorvent.floci.core.storage.StorageFactory;import io.github.hectorvent.floci.services.ecs.container.EcsContainerManager;import io.github.hectorvent.floci.services.ecs.container.HostVolumePolicy;+import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;import io.github.hectorvent.floci.services.ecs.model.LaunchType;import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;import jakarta.ws.rs.core.Response;@@ -67,6 +68,10 @@ class EcsJsonHandlerTaskDefinitionPersistenceTest {"logConfiguration": {"logDriver": "awslogs","options": {"awslogs-group": "/ecs/restart-family"}+ },+ "firelensConfiguration": {+ "type": "fluentbit",+ "options": {"enable-ecs-log-metadata": "true"}}}],@@ -89,6 +94,11 @@ class EcsJsonHandlerTaskDefinitionPersistenceTest {assertEquals("awslogs", logConfiguration.logDriver());assertEquals("/ecs/restart-family", logConfiguration.options().get("awslogs-group"));+ FirelensConfiguration firelens = td.getContainerDefinitions().getFirst().getFirelensConfiguration();+ assertNotNull(firelens, "firelensConfiguration must survive a restart");+ assertEquals("fluentbit", firelens.type());+ assertEquals("true", firelens.options().get("enable-ecs-log-metadata"));+assertEquals(1, td.getVolumes().size(), "task-level volumes must survive a restart");assertEquals("/host/data", td.getVolumes().getFirst().hostSourcePath());}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.javaindex 3c7e0393c3..78692d89ac 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerAwsBaselineTest.java@@ -15,6 +15,7 @@ import io.github.hectorvent.floci.services.ecs.model.EcsTask;import io.github.hectorvent.floci.services.ecs.model.KeyValuePair;import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.s3.S3Service;import io.github.hectorvent.floci.services.ssm.SsmService;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;@@ -79,7 +80,7 @@ class EcsContainerManagerAwsBaselineTest {manager = new EcsContainerManager(containerBuilder, lifecycleManager, logStreamer,containerDetector, config, regionResolver, awsEnv, mock(SsmService.class),- mock(SecretsManagerService.class), ecrRegistryManager, mock(HostVolumePolicy.class));+ mock(SecretsManagerService.class), mock(S3Service.class), ecrRegistryManager, mock(HostVolumePolicy.class));}@Testdiff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerEcrRewriteTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerEcrRewriteTest.javaindex df84c986e0..d362436bba 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerEcrRewriteTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerEcrRewriteTest.java@@ -14,6 +14,7 @@ import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;import io.github.hectorvent.floci.services.ecs.model.EcsTask;import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.s3.S3Service;import io.github.hectorvent.floci.services.ssm.SsmService;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;@@ -79,7 +80,7 @@ class EcsContainerManagerEcrRewriteTest {when(ecrRegistryManager.rewriteImageUri("sidecar:latest")).thenReturn("sidecar:latest");manager = new EcsContainerManager(containerBuilder, lifecycleManager, logStreamer,- containerDetector, config, regionResolver, awsEnv, ssmService, secretsManagerService,+ containerDetector, config, regionResolver, awsEnv, ssmService, secretsManagerService, mock(S3Service.class),ecrRegistryManager, mock(HostVolumePolicy.class));}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerFirelensDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerFirelensDockerIntegrationTest.javaindex cbfbfbda5b..d1e5226a38 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerFirelensDockerIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerFirelensDockerIntegrationTest.java@@ -4,7 +4,7 @@ import com.github.dockerjava.api.DockerClient;import com.github.dockerjava.api.model.Frame;import com.github.dockerjava.api.model.StreamType;import com.github.dockerjava.core.command.ExecStartResultCallback;-import com.github.dockerjava.core.command.WaitContainerResultCallback;+import com.github.dockerjava.core.command.LogContainerResultCallback;import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;import io.github.hectorvent.floci.services.ecs.model.EcsTask;import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;@@ -34,6 +34,9 @@ import static org.junit.jupiter.api.Assertions.assertTrue;class EcsContainerManagerFirelensDockerIntegrationTest {private static final String IMAGE = "public.ecr.aws/docker/library/busybox:latest";+ /** A real FireLens router image: Floci writes the generated config to its own+ * {@code /fluent-bit/etc}, which no minimal image carries. */+ private static final String ROUTER_IMAGE = "public.ecr.aws/aws-observability/aws-for-fluent-bit:3";@InjectEcsContainerManager containerManager;@@ -58,19 +61,18 @@ class EcsContainerManagerFirelensDockerIntegrationTest {}@Test- void applicationStdoutReachesFirelensRouterThroughPublishedHostPort() throws Exception {+ void applicationRecordReachesTheRouterThroughTheGeneratedConfig() throws Exception {String marker = "firelens-marker-" + UUID.randomUUID();ContainerDefinition app = new ContainerDefinition();app.setName("app");app.setImage(IMAGE);- app.setCommand(List.of("sh", "-c", "echo " + marker + "; sleep 2"));- app.setLogConfiguration(new LogConfiguration("awsfirelens", Map.of(), null));+ app.setCommand(List.of("sh", "-c", "while true; do echo " + marker + "; sleep 1; done"));+ app.setLogConfiguration(new LogConfiguration("awsfirelens", Map.of("Name", "stdout"), null));ContainerDefinition router = new ContainerDefinition();router.setName("router");- router.setImage(IMAGE);- router.setCommand(List.of("sh", "-c", "while true; do nc -l -p 24224 >> /tmp/received; done"));+ router.setImage(ROUTER_IMAGE);router.setFirelensConfiguration(new FirelensConfiguration("fluentbit", Map.of()));TaskDefinition taskDefinition = new TaskDefinition();@@ -78,67 +80,86 @@ class EcsContainerManagerFirelensDockerIntegrationTest {taskDefinition.setContainerDefinitions(List.of(app, router));EcsTask task = new EcsTask();- task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/firelens/" + marker);+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/firelens/" + UUID.randomUUID());taskHandle = containerManager.startTask(task, taskDefinition, List.of(), "us-east-1");assertEquals(List.of("router", "app"), taskHandle.getContainerIds().keySet().stream().toList());String appId = taskHandle.getContainerIds().get("app");String routerId = taskHandle.getContainerIds().get("router");- dockerClient.waitContainerCmd(appId)- .exec(new WaitContainerResultCallback())- .awaitStatusCode(30, TimeUnit.SECONDS);- String received = awaitRouterBuffer(routerId, marker);- assertTrue(received.contains(marker), "FireLens router did not receive the app stdout record");+ // The Docker daemon, not the application container, makes this connection, so the+ // address has to be a path on the host: the socket the router creates inside the+ // task's own volume, which the daemon reaches at the volume's mountpoint.+ assertEquals("unix://" + firelensVolumeMountpoint() + "/fluent.sock", appLogDriverAddress(appId));++ String routerLogs = awaitRouterLogs(routerId, marker);+ assertTrue(routerLogs.contains(marker),+ "FireLens router did not print the application record routed by the generated config: "+ + routerLogs);+ assertTrue(routerLogs.contains("ecs_task_arn"),+ "The generated config's ECS metadata filter did not run: " + routerLogs);}- private String awaitRouterBuffer(String routerId, String marker) throws Exception {+ private String firelensVolumeMountpoint() throws Exception {+ return dockerClient.inspectVolumeCmd(taskHandle.getFirelensVolumeName())+ .exec()+ .getMountpoint();+ }++ private String appLogDriverAddress(String appId) {+ return dockerClient.inspectContainerCmd(appId)+ .exec()+ .getHostConfig()+ .getLogConfig()+ .getConfig()+ .get("fluentd-address");+ }++ private String awaitRouterLogs(String routerId, String marker) throws Exception {+ String logs = "";for (int attempt = 0; attempt < 30; attempt++) {- ExecResult result = readRouterBuffer(routerId, marker);- if (result.exitCode() == 0 && result.output().contains(marker)) {- return result.output();+ logs = readContainerLogs(routerId);+ if (logs.contains(marker)) {+ return logs;}Thread.sleep(1000);}- return readRouterBuffer(routerId, marker).output();+ return logs;}- private ExecResult readRouterBuffer(String routerId, String marker) throws Exception {- String execId = dockerClient.execCreateCmd(routerId)- .withCmd("sh", "-c", "grep -a " + marker + " /tmp/received")- .withAttachStdout(true)- .withAttachStderr(true)- .exec()- .getId();+ private String readContainerLogs(String containerId) throws Exception {ByteArrayOutputStream output = new ByteArrayOutputStream();CountDownLatch complete = new CountDownLatch(1);- Closeable callback = dockerClient.execStartCmd(execId).exec(new ExecStartResultCallback() {- @Override- public void onNext(Frame frame) {- try {- if (frame.getStreamType() == StreamType.STDOUT) {- output.write(frame.getPayload());+ Closeable callback = dockerClient.logContainerCmd(containerId)+ .withStdOut(true)+ .withStdErr(true)+ .exec(new LogContainerResultCallback() {+ @Override+ public void onNext(Frame frame) {+ try {+ if (frame.getStreamType() == StreamType.STDOUT+ || frame.getStreamType() == StreamType.STDERR) {+ output.write(frame.getPayload());+ }+ } catch (IOException e) {+ throw new IllegalStateException("Failed to capture router logs", e);+ }}- } catch (IOException e) {- throw new IllegalStateException("Failed to capture FireLens output", e);- }- }- @Override- public void onComplete() {- complete.countDown();- }+ @Override+ public void onComplete() {+ complete.countDown();+ }- @Override- public void onError(Throwable throwable) {- complete.countDown();- }- });+ @Override+ public void onError(Throwable throwable) {+ complete.countDown();+ }+ });try {- assertTrue(complete.await(10, TimeUnit.SECONDS), "Timed out reading FireLens router buffer");- long exitCode = dockerClient.inspectExecCmd(execId).exec().getExitCodeLong();- return new ExecResult(exitCode, output.toString(StandardCharsets.UTF_8));+ complete.await(10, TimeUnit.SECONDS);+ return output.toString(StandardCharsets.UTF_8);} finally {callback.close();}@@ -153,6 +174,4 @@ class EcsContainerManagerFirelensDockerIntegrationTest {}}- private record ExecResult(long exitCode, String output) {- }}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerFirelensTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerFirelensTest.javaindex e724974ec3..894c945330 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerFirelensTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerFirelensTest.java@@ -1,59 +1,421 @@package io.github.hectorvent.floci.services.ecs.container;+import com.github.dockerjava.api.DockerClient;+import com.github.dockerjava.api.command.CopyArchiveToContainerCmd;+import com.github.dockerjava.api.command.InspectVolumeCmd;+import com.github.dockerjava.api.command.InspectVolumeResponse;import com.github.dockerjava.api.model.LogConfig;+import io.github.hectorvent.floci.config.EmulatorConfig;import io.github.hectorvent.floci.core.common.AwsException;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import io.github.hectorvent.floci.core.common.docker.ContainerDetector;+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.ContainerInfo;+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;+import io.github.hectorvent.floci.core.common.docker.ContainerSpec;+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;+import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;+import io.github.hectorvent.floci.services.ecs.model.EcsTask;import io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration;import io.github.hectorvent.floci.services.ecs.model.LogConfiguration;+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;+import io.github.hectorvent.floci.services.ecs.model.PortMapping;+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;+import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.s3.model.S3Object;+import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.ssm.SsmService;+import org.apache.commons.compress.archivers.tar.TarArchiveEntry;+import org.apache.commons.compress.archivers.tar.TarArchiveInputStream;+import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;+import org.mockito.ArgumentCaptor;+import org.mockito.InOrder;+import java.io.ByteArrayInputStream;+import java.io.InputStream;+import java.nio.charset.StandardCharsets;+import java.util.LinkedHashMap;import java.util.List;import java.util.Map;import static org.junit.jupiter.api.Assertions.assertEquals;import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.RETURNS_SELF;+import static org.mockito.Mockito.inOrder;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;class EcsContainerManagerFirelensTest {+ private ContainerBuilder.Builder builder;+ private ContainerLifecycleManager lifecycleManager;+ private S3Service s3Service;+ private ContainerLogStreamer logStreamer;+ private DockerClient dockerClient;+ private CopyArchiveToContainerCmd copyCmd;+ private EcsContainerManager manager;++ @BeforeEach+ void setUp() {+ builder = mock(ContainerBuilder.Builder.class, RETURNS_SELF);+ when(builder.build()).thenReturn(new ContainerSpec("image"));+ ContainerBuilder containerBuilder = mock(ContainerBuilder.class);+ when(containerBuilder.newContainer(anyString())).thenReturn(builder);++ lifecycleManager = mock(ContainerLifecycleManager.class);+ when(lifecycleManager.createAndStart(any())).thenReturn(new ContainerInfo("app-id", Map.of()));+ when(lifecycleManager.create(any())).thenReturn("router-id");+ when(lifecycleManager.startCreated(anyString(), any())).thenReturn(new ContainerInfo("router-id", Map.of()));++ dockerClient = mock(DockerClient.class, RETURNS_DEEP_STUBS);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);+ InspectVolumeCmd inspectVolumeCmd = mock(InspectVolumeCmd.class);+ InspectVolumeResponse volume = mock(InspectVolumeResponse.class);+ when(dockerClient.inspectVolumeCmd(anyString())).thenReturn(inspectVolumeCmd);+ when(inspectVolumeCmd.exec()).thenReturn(volume);+ when(volume.getMountpoint()).thenReturn("/var/lib/docker/volumes/floci-ecs-firelens-abc123/_data");+ copyCmd = mock(CopyArchiveToContainerCmd.class, RETURNS_SELF);+ when(dockerClient.copyArchiveToContainerCmd("router-id")).thenReturn(copyCmd);++ logStreamer = mock(ContainerLogStreamer.class);+ when(logStreamer.generateLogStreamName(anyString())).thenReturn("stream");+ ContainerDetector containerDetector = mock(ContainerDetector.class);+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ RegionResolver regionResolver = mock(RegionResolver.class);+ LaunchedContainerAwsEnv awsEnv = mock(LaunchedContainerAwsEnv.class);+ when(awsEnv.sdkBaselineEnv(any(), any())).thenReturn(List.of());+ when(awsEnv.flociEndpoint()).thenReturn("http://host.docker.internal:4566");+ EcrRegistryManager ecrRegistryManager = mock(EcrRegistryManager.class);+ when(ecrRegistryManager.rewriteImageUri(anyString())).thenAnswer(inv -> inv.getArgument(0));++ s3Service = mock(S3Service.class);+ manager = new EcsContainerManager(containerBuilder, lifecycleManager, logStreamer,+ containerDetector, config, regionResolver, awsEnv, mock(SsmService.class),+ mock(SecretsManagerService.class), s3Service, ecrRegistryManager, new HostVolumePolicy(config));+ }++ @Test+ void startsRouterFirstAndWiresAwsfirelensToUnixSocket() {+ ContainerDefinition router = new ContainerDefinition();+ router.setName("log_router");+ router.setImage("amazon/aws-for-fluent-bit:stable");+ router.setFirelensConfiguration(new FirelensConfiguration("fluentbit", Map.of()));++ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage("app:latest");+ app.setLogConfiguration(new LogConfiguration("awsfirelens",+ Map.of("Name", "cloudwatch", "region", "us-east-1"), null));++ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("firelens-family");+ taskDef.setRevision(1);+ taskDef.setNetworkMode(NetworkMode.awsvpc);+ taskDef.setContainerDefinitions(List.of(app, router));++ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/test-cluster");++ EcsTaskHandle handle = manager.startTask(task, taskDef, null, "us-east-1");++ InOrder order = inOrder(lifecycleManager);+ order.verify(lifecycleManager).create(any(ContainerSpec.class));+ order.verify(lifecycleManager).startCreated(anyString(), any());+ order.verify(lifecycleManager).createAndStart(any(ContainerSpec.class));++ ArgumentCaptor<LogConfig> logConfig = ArgumentCaptor.forClass(LogConfig.class);+ verify(builder).withLogConfig(logConfig.capture());+ assertEquals(LogConfig.LoggingType.FLUENTD, logConfig.getValue().getType());+ assertEquals("unix:///var/lib/docker/volumes/floci-ecs-firelens-abc123/_data/fluent.sock",+ logConfig.getValue().getConfig().get("fluentd-address"));+ assertEquals("app-firelens-abc123", logConfig.getValue().getConfig().get("tag"));+ assertEquals("true", logConfig.getValue().getConfig().get("fluentd-async"));+ assertTrue(logConfig.getValue().getConfig().get("fluentd-async-connect") == null,+ "fluentd-async-connect was removed by Docker 28");++ verify(logStreamer).attach(eq("router-id"), anyString(), anyString(), anyString(), anyString());+ verify(logStreamer, never()).attach(eq("app-id"), anyString(), anyString(), anyString(), anyString());+ assertTrue(handle.getFirelensVolumeName().contains("firelens"));++ verify(copyCmd).withRemotePath("/fluent-bit/etc");+ ArgumentCaptor<InputStream> configArchive = ArgumentCaptor.forClass(InputStream.class);+ verify(copyCmd).withTarInputStream(configArchive.capture());+ assertTrue(readFluentBitConf(configArchive.getValue()).contains(+ " Endpoint http://host.docker.internal:4566"));+ }+@Test- void createsFluentdLogConfigForAwsFirelensApplication() {- LogConfig logConfig = EcsContainerManager.firelensLogConfig("task-123", "app", 32768);+ void startsFluentdRouterAndWritesFluentConf() {+ ContainerDefinition router = new ContainerDefinition();+ router.setName("log_router");+ router.setImage("fluent/fluentd:v1.16");+ router.setFirelensConfiguration(new FirelensConfiguration("fluentd", Map.of()));++ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage("app:latest");+ app.setLogConfiguration(new LogConfiguration("awsfirelens",+ Map.of("@type", "stdout"), null));++ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("firelens-fluentd");+ taskDef.setRevision(1);+ taskDef.setNetworkMode(NetworkMode.bridge);+ taskDef.setContainerDefinitions(List.of(app, router));++ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/test-cluster");++ manager.startTask(task, taskDef, null, "us-east-1");- assertEquals(LogConfig.LoggingType.FLUENTD, logConfig.getType());- assertEquals("127.0.0.1:32768", logConfig.getConfig().get("fluentd-address"));- assertEquals("task-123.app", logConfig.getConfig().get("tag"));- assertEquals("true", logConfig.getConfig().get("fluentd-async"));+ InOrder order = inOrder(lifecycleManager);+ order.verify(lifecycleManager).create(any(ContainerSpec.class));+ order.verify(lifecycleManager).startCreated(anyString(), any());+ order.verify(lifecycleManager).createAndStart(any(ContainerSpec.class));++ ArgumentCaptor<LogConfig> logConfig = ArgumentCaptor.forClass(LogConfig.class);+ verify(builder).withLogConfig(logConfig.capture());+ assertEquals(LogConfig.LoggingType.FLUENTD, logConfig.getValue().getType());+ assertEquals("unix:///var/lib/docker/volumes/floci-ecs-firelens-abc123/_data/fluent.sock",+ logConfig.getValue().getConfig().get("fluentd-address"));++ verify(copyCmd).withRemotePath("/fluentd/etc");+ ArgumentCaptor<InputStream> configArchive = ArgumentCaptor.forClass(InputStream.class);+ verify(copyCmd).withTarInputStream(configArchive.capture());+ String conf = readFluentBitConf(configArchive.getValue());+ assertTrue(conf.contains("@type unix"));+ assertTrue(conf.contains("<match app-firelens**>"));+ assertTrue(conf.contains("@type stdout"));+ assertTrue(!conf.contains("Endpoint"));}+@Test- void ordersRouterBeforeAwsFirelensApplications() {- ContainerDefinition app = container("app", "awsfirelens", null);- ContainerDefinition router = container("router", null,- new FirelensConfiguration("fluentbit", Map.of()));+ void writesS3ExternalConfigBesideTheGeneratedFluentBitConfig() {+ ContainerDefinition router = new ContainerDefinition();+ router.setName("log_router");+ router.setImage("amazon/aws-for-fluent-bit:stable");+ router.setFirelensConfiguration(new FirelensConfiguration("fluentbit", Map.of(+ "config-file-type", "s3",+ "config-file-value", "arn:aws:s3:::firelens-configs/extra.conf")));++ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage("app:latest");+ app.setLogConfiguration(new LogConfiguration("awsfirelens",+ Map.of("Name", "cloudwatch", "region", "us-east-1"), null));- List<ContainerDefinition> ordered = EcsContainerManager.orderFirelensContainers(List.of(app, router));+ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("firelens-s3");+ taskDef.setRevision(1);+ taskDef.setNetworkMode(NetworkMode.awsvpc);+ taskDef.setContainerDefinitions(List.of(app, router));- assertEquals("router", ordered.get(0).getName());- assertEquals("app", ordered.get(1).getName());+ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/test-cluster");++ String extra = "[OUTPUT]\n Name stdout\n Match app-firelens*\n";+ S3Object object = new S3Object();+ object.setData(extra.getBytes(StandardCharsets.UTF_8));+ when(s3Service.getObject("firelens-configs", "extra.conf")).thenReturn(object);++ manager.startTask(task, taskDef, null, "us-east-1");++ verify(s3Service).getObject("firelens-configs", "extra.conf");+ verify(copyCmd).withRemotePath("/fluent-bit/etc");+ ArgumentCaptor<InputStream> configArchive = ArgumentCaptor.forClass(InputStream.class);+ verify(copyCmd).withTarInputStream(configArchive.capture());+ Map<String, String> files = readFirelensArchive(configArchive.getValue());+ assertEquals(2, files.size(), files::toString);+ assertTrue(files.get("fluent-bit.conf").contains(+ "@INCLUDE /fluent-bit/etc/external.conf"), files.get("fluent-bit.conf"));+ assertEquals(extra, files.get("external.conf"));}@Test- void rejectsAwsFirelensApplicationWithoutRouter() {- ContainerDefinition app = container("app", "awsfirelens", null);+ void fluentdS3ExternalConfigUsesTheFluentdPath() {+ ContainerDefinition router = new ContainerDefinition();+ router.setName("log_router");+ router.setImage("fluent/fluentd:v1.16");+ router.setFirelensConfiguration(new FirelensConfiguration("fluentd", Map.of(+ "config-file-type", "s3",+ "config-file-value", "arn:aws:s3:::firelens-configs/extra.conf")));++ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage("app:latest");+ app.setLogConfiguration(new LogConfiguration("awsfirelens", Map.of("@type", "stdout"), null));++ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("firelens-s3-fluentd");+ taskDef.setRevision(1);+ taskDef.setNetworkMode(NetworkMode.bridge);+ taskDef.setContainerDefinitions(List.of(app, router));++ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/test-cluster");++ S3Object object = new S3Object();+ object.setData("<match **>\n @type stdout\n</match>\n".getBytes(StandardCharsets.UTF_8));+ when(s3Service.getObject("firelens-configs", "extra.conf")).thenReturn(object);++ manager.startTask(task, taskDef, null, "us-east-1");- assertThrows(AwsException.class,- () -> EcsContainerManager.orderFirelensContainers(List.of(app)));+ verify(copyCmd).withRemotePath("/fluentd/etc");+ ArgumentCaptor<InputStream> configArchive = ArgumentCaptor.forClass(InputStream.class);+ verify(copyCmd).withTarInputStream(configArchive.capture());+ Map<String, String> files = readFirelensArchive(configArchive.getValue());+ assertTrue(files.get("fluent.conf").contains(+ "@include /fluentd/etc/external.conf"), files.get("fluent.conf"));+ assertTrue(files.containsKey("external.conf"), files::toString);+ }++ @Test+ void missingS3ConfigFailsTheTaskBeforeAnyContainerIsCreated() {+ ContainerDefinition router = new ContainerDefinition();+ router.setName("log_router");+ router.setImage("amazon/aws-for-fluent-bit:stable");+ router.setFirelensConfiguration(new FirelensConfiguration("fluentbit", Map.of(+ "config-file-type", "s3",+ "config-file-value", "arn:aws:s3:::firelens-configs/missing.conf")));++ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("firelens-s3-missing");+ taskDef.setRevision(1);+ taskDef.setNetworkMode(NetworkMode.awsvpc);+ taskDef.setContainerDefinitions(List.of(router));++ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/test-cluster");++ when(s3Service.getObject("firelens-configs", "missing.conf"))+ .thenThrow(new AwsException("NoSuchKey", "The specified key does not exist.", 404));++ AwsException failure = assertThrows(AwsException.class,+ () -> manager.startTask(task, taskDef, null, "us-east-1"));+ assertEquals("ResourceInitializationError", failure.getErrorCode());+ assertEquals("Unable to download firelens s3 config file: unable to download s3 config "+ + "missing.conf from bucket firelens-configs: The specified key does not exist.",+ failure.getMessage());+ verify(lifecycleManager, never()).create(any(ContainerSpec.class));+ verify(lifecycleManager, never()).createAndStart(any());+ }++ @Test+ void rejectsAwsfirelensApplicationWithoutRouter() {+ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage("app:latest");+ app.setLogConfiguration(new LogConfiguration("awsfirelens", Map.of(), null));++ AwsException failure = assertThrows(AwsException.class,+ () -> manager.startTask(task(), taskDefinition(List.of(app)), null, "us-east-1"));++ assertEquals("ClientException", failure.getErrorCode());+ assertEquals("awsfirelens log driver requires a firelensConfiguration container",+ failure.getMessage());+ verify(lifecycleManager, never()).createAndStart(any());+ }++ @Test+ void rejectsTwoFirelensRouters() {+ AwsException failure = assertThrows(AwsException.class, () -> manager.startTask(+ task(),+ taskDefinition(List.of(+ router("fluentbit", List.of()),+ router("fluentd", List.of()))),+ null, "us-east-1"));++ assertEquals("ClientException", failure.getErrorCode());+ assertEquals("A task definition can have only one FireLens log router.", failure.getMessage());+ verify(lifecycleManager, never()).create(any(ContainerSpec.class));+ verify(lifecycleManager, never()).createAndStart(any());+ }++ @Test+ void rejectsFirelensRouterExposingPort24224() {+ AwsException failure = assertThrows(AwsException.class, () -> manager.startTask(+ task(),+ taskDefinition(List.of(router("fluentbit", List.of(new PortMapping(24224, 0, "tcp"))))),+ null, "us-east-1"));++ assertEquals("ClientException", failure.getErrorCode());+ assertEquals("FireLens port 24224 must not be exposed.", failure.getMessage());+ verify(lifecycleManager, never()).create(any(ContainerSpec.class));+ }++ @Test+ void rejectsUnsupportedFirelensRouterType() {+ AwsException failure = assertThrows(AwsException.class, () -> manager.startTask(+ task(),+ taskDefinition(List.of(router("fluent-plugin", List.of()))),+ null, "us-east-1"));++ assertEquals("ClientException", failure.getErrorCode());+ assertEquals("FireLens configuration type must be fluentbit or fluentd.", failure.getMessage());+ verify(lifecycleManager, never()).create(any(ContainerSpec.class));+ }++ private static ContainerDefinition router(String type, List<PortMapping> portMappings) {+ ContainerDefinition router = new ContainerDefinition();+ router.setName("log_router");+ router.setImage("amazon/aws-for-fluent-bit:stable");+ router.setFirelensConfiguration(new FirelensConfiguration(type, Map.of()));+ router.setPortMappings(portMappings);+ return router;+ }++ private static TaskDefinition taskDefinition(List<ContainerDefinition> definitions) {+ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("firelens-family");+ taskDef.setRevision(1);+ taskDef.setNetworkMode(NetworkMode.awsvpc);+ taskDef.setContainerDefinitions(definitions);+ return taskDef;+ }++ private static EcsTask task() {+ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");+ task.setClusterArn("arn:aws:ecs:us-east-1:000000000000:cluster/test-cluster");+ return task;+ }++ private static Map<String, String> readFirelensArchive(InputStream archive) {+ LinkedHashMap<String, String> files = new LinkedHashMap<>();+ try (TarArchiveInputStream tar = new TarArchiveInputStream(archive)) {+ TarArchiveEntry entry;+ while ((entry = tar.getNextEntry()) != null) {+ files.put(entry.getName(), new String(tar.readAllBytes(), StandardCharsets.UTF_8));+ }+ } catch (Exception e) {+ throw new IllegalStateException(e);+ }+ return files;}- private static ContainerDefinition container(String name, String driver,- FirelensConfiguration firelensConfiguration) {- ContainerDefinition definition = new ContainerDefinition();- definition.setName(name);- definition.setImage("alpine:latest");- definition.setFirelensConfiguration(firelensConfiguration);- if (driver != null) {- definition.setLogConfiguration(new LogConfiguration(driver, Map.of(), null));+ private static String readFluentBitConf(InputStream archive) {+ try (TarArchiveInputStream tar = new TarArchiveInputStream(archive)) {+ tar.getNextEntry();+ // TarArchiveInputStream.read stops at the end of the current entry.+ return new String(tar.readAllBytes(), StandardCharsets.UTF_8);+ } catch (Exception e) {+ throw new IllegalStateException(e);}- return definition;}}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerLifecycleTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerLifecycleTest.javaindex 2818da51ab..47b3f97013 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerLifecycleTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerLifecycleTest.java@@ -12,6 +12,7 @@ import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.s3.S3Service;import io.github.hectorvent.floci.services.ssm.SsmService;import org.junit.jupiter.api.Test;@@ -117,7 +118,7 @@ class EcsContainerManagerLifecycleTest {return new EcsContainerManager(mock(ContainerBuilder.class), lifecycleManager, mock(ContainerLogStreamer.class),mock(ContainerDetector.class), mock(EmulatorConfig.class), mock(RegionResolver.class),- mock(LaunchedContainerAwsEnv.class), mock(SsmService.class), mock(SecretsManagerService.class),+ mock(LaunchedContainerAwsEnv.class), mock(SsmService.class), mock(SecretsManagerService.class), mock(S3Service.class),mock(EcrRegistryManager.class), mock(HostVolumePolicy.class));}}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerOverridesTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerOverridesTest.javaindex b6297b55ad..7fa5c040d5 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerOverridesTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerOverridesTest.java@@ -15,6 +15,7 @@ import io.github.hectorvent.floci.services.ecs.model.EcsTask;import io.github.hectorvent.floci.services.ecs.model.KeyValuePair;import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.s3.S3Service;import io.github.hectorvent.floci.services.ssm.SsmService;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;@@ -75,7 +76,7 @@ class EcsContainerManagerOverridesTest {when(ecrRegistryManager.rewriteImageUri(anyString())).thenAnswer(inv -> inv.getArgument(0));manager = new EcsContainerManager(containerBuilder, lifecycleManager, logStreamer,- containerDetector, config, regionResolver, awsEnv, ssmService, secretsManagerService,+ containerDetector, config, regionResolver, awsEnv, ssmService, secretsManagerService, mock(S3Service.class),ecrRegistryManager, mock(HostVolumePolicy.class));}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerPortMappingsTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerPortMappingsTest.javaindex c5325c9df5..8bb3f73e8f 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerPortMappingsTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerPortMappingsTest.java@@ -2,6 +2,7 @@ package io.github.hectorvent.floci.services.ecs.container;import com.github.dockerjava.api.DockerClient;import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.AwsException;import io.github.hectorvent.floci.core.common.RegionResolver;import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;import io.github.hectorvent.floci.core.common.docker.ContainerDetector;@@ -18,6 +19,7 @@ import io.github.hectorvent.floci.services.ecs.model.NetworkMode;import io.github.hectorvent.floci.services.ecs.model.PortMapping;import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.s3.S3Service;import io.github.hectorvent.floci.services.ssm.SsmService;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;@@ -25,6 +27,8 @@ import org.junit.jupiter.api.Test;import java.util.List;import java.util.Map;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertThrows;import static org.mockito.ArgumentMatchers.any;import static org.mockito.ArgumentMatchers.anyInt;import static org.mockito.ArgumentMatchers.anyString;@@ -92,7 +96,7 @@ class EcsContainerManagerPortMappingsTest {when(ecrRegistryManager.rewriteImageUri(anyString())).thenAnswer(inv -> inv.getArgument(0));manager = new EcsContainerManager(containerBuilder, lifecycleManager, logStreamer,- containerDetector, config, regionResolver, awsEnv, ssmService, secretsManagerService,+ containerDetector, config, regionResolver, awsEnv, ssmService, secretsManagerService, mock(S3Service.class),ecrRegistryManager, mock(HostVolumePolicy.class));}@@ -225,12 +229,13 @@ class EcsContainerManagerPortMappingsTest {}@Test- void firelensRouterUsesLoopbackOnlyDynamicPortBinding() {+ void firelensRouterDeclaringTheForwardPortIsRejected() {when(containerDetector.isRunningInContainer()).thenReturn(false);ContainerDefinition router = new ContainerDefinition();router.setName("router");router.setImage("fluent/fluent-bit:latest");+ router.setPortMappings(List.of(new PortMapping(24224)));router.setFirelensConfiguration(new FirelensConfiguration("fluentbit", Map.of()));ContainerDefinition app = new ContainerDefinition();@@ -245,13 +250,10 @@ class EcsContainerManagerPortMappingsTest {EcsTask task = new EcsTask();task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");- when(lifecycleManager.createAndStart(any()))- .thenReturn(new ContainerInfo("router-id", Map.of(), Map.of(24224, 32768)))- .thenReturn(new ContainerInfo("app-id", Map.of()));-- manager.startTask(task, taskDef, List.of(), "us-east-1");+ AwsException failure = assertThrows(AwsException.class,+ () -> manager.startTask(task, taskDef, List.of(), "us-east-1"));- verify(builder).withLoopbackPortBinding(24224, 0);+ assertEquals("FireLens port 24224 must not be exposed.", failure.getMessage());verify(builder, never()).withDynamicPort(24224);}}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecretsTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecretsTest.javaindex da37574d8d..902c1a18f0 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecretsTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecretsTest.java@@ -17,6 +17,7 @@ import io.github.hectorvent.floci.services.ecs.model.KeyValuePair;import io.github.hectorvent.floci.services.ecs.model.Secret;import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.s3.S3Service;import io.github.hectorvent.floci.services.secretsmanager.model.SecretVersion;import io.github.hectorvent.floci.services.ssm.SsmService;import io.github.hectorvent.floci.services.ssm.model.Parameter;@@ -69,7 +70,7 @@ class EcsContainerManagerSecretsTest {when(ecrRegistryManager.rewriteImageUri(anyString())).thenAnswer(inv -> inv.getArgument(0));manager = new EcsContainerManager(containerBuilder, lifecycleManager, logStreamer,- containerDetector, config, regionResolver, awsEnv, ssmService, secretsManagerService,+ containerDetector, config, regionResolver, awsEnv, ssmService, secretsManagerService, mock(S3Service.class),ecrRegistryManager, mock(HostVolumePolicy.class));}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.javanew file mode 100644index 0000000000..b307de3174--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerSecurityGroupTest.java@@ -0,0 +1,254 @@+package io.github.hectorvent.floci.services.ecs.container;++import com.github.dockerjava.api.DockerClient;+import io.github.hectorvent.floci.config.EmulatorConfig;+import io.github.hectorvent.floci.core.common.RegionResolver;+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;+import io.github.hectorvent.floci.core.common.docker.ContainerDetector;+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.ContainerInfo;+import io.github.hectorvent.floci.core.common.docker.ContainerLogStreamer;+import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;+import io.github.hectorvent.floci.services.ec2.Ec2Service;+import io.github.hectorvent.floci.services.ec2.SecurityGroupFirewallManager;+import io.github.hectorvent.floci.services.ec2.model.GroupIdentifier;+import io.github.hectorvent.floci.services.ec2.model.NetworkInterface;+import io.github.hectorvent.floci.services.ec2.model.SecurityGroup;+import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;+import io.github.hectorvent.floci.services.ecs.model.AwsVpcConfiguration;+import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;+import io.github.hectorvent.floci.services.ecs.model.EcsTask;+import io.github.hectorvent.floci.services.ecs.model.NetworkConfiguration;+import io.github.hectorvent.floci.services.ecs.model.NetworkMode;+import io.github.hectorvent.floci.services.ecs.model.PortMapping;+import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;+import io.github.hectorvent.floci.services.s3.S3Service;+import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.ssm.SsmService;+import org.junit.jupiter.api.BeforeEach;+import org.junit.jupiter.api.Test;++import java.util.List;+import java.util.Map;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.mockito.ArgumentMatchers.any;+import static org.mockito.ArgumentMatchers.anyInt;+import static org.mockito.ArgumentMatchers.anyString;+import static org.mockito.ArgumentMatchers.eq;+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;+import static org.mockito.Mockito.RETURNS_SELF;+import static org.mockito.Mockito.mock;+import static org.mockito.Mockito.never;+import static org.mockito.Mockito.times;+import static org.mockito.Mockito.verify;+import static org.mockito.Mockito.when;++/**+ * Regression for the FireLens launch-loop rewrite dropping the awsvpc+ * security-group namespace join. prepareNetwork still creates the ENI;+ * this asserts the per-container loop actually joins it.+ */+class EcsContainerManagerSecurityGroupTest {++ private ContainerBuilder.Builder builder;+ private ContainerLifecycleManager lifecycleManager;+ private DockerClient dockerClient;+ private ContainerDetector containerDetector;+ private RegionResolver regionResolver;+ private Ec2Service ec2Service;+ private SecurityGroupFirewallManager firewallManager;+ private EcsContainerManager manager;++ @BeforeEach+ void setUp() {+ builder = mock(ContainerBuilder.Builder.class, RETURNS_SELF);+ ContainerBuilder containerBuilder = mock(ContainerBuilder.class);+ when(containerBuilder.newContainer(anyString())).thenReturn(builder);++ lifecycleManager = mock(ContainerLifecycleManager.class);+ when(lifecycleManager.createAndStart(any())).thenReturn(new ContainerInfo("docker-id", Map.of()));+ dockerClient = mock(DockerClient.class, RETURNS_DEEP_STUBS);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient);++ containerDetector = mock(ContainerDetector.class);+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);+ regionResolver = mock(RegionResolver.class);+ when(regionResolver.getAccountId()).thenReturn("000000000000");+ LaunchedContainerAwsEnv awsEnv = mock(LaunchedContainerAwsEnv.class);+ when(awsEnv.sdkBaselineEnv(any(), any())).thenReturn(List.of());+ EcrRegistryManager ecrRegistryManager = mock(EcrRegistryManager.class);+ when(ecrRegistryManager.rewriteImageUri(anyString())).thenAnswer(inv -> inv.getArgument(0));++ ec2Service = mock(Ec2Service.class);+ firewallManager = mock(SecurityGroupFirewallManager.class);++ manager = new EcsContainerManager(+ containerBuilder, lifecycleManager, mock(ContainerLogStreamer.class),+ containerDetector, config, regionResolver, awsEnv,+ mock(SsmService.class), mock(SecretsManagerService.class), mock(S3Service.class),+ ecrRegistryManager, mock(HostVolumePolicy.class),+ ec2Service, firewallManager);+ }++ @Test+ void awsvpcTaskWithFirewallJoinsHelperNamespaceAndSkipsHostPorts() {+ when(containerDetector.isRunningInContainer()).thenReturn(false);+ when(firewallManager.enabled()).thenReturn(true);+ when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any()))+ .thenReturn(new SecurityGroupFirewallManager.Namespace("helper-id", "10.0.0.5"));++ NetworkInterface eni = new NetworkInterface();+ eni.setNetworkInterfaceId("eni-1");+ eni.setVpcId("vpc-1");+ eni.setPrivateIpAddress("10.0.0.10");+ eni.setGroups(List.of(new GroupIdentifier("sg-1", "default")));+ when(ec2Service.createNetworkInterface(any(), eq("subnet-1"), any(), any(), any(), any(), any()))+ .thenReturn(eni);++ SecurityGroup sg = new SecurityGroup();+ sg.setGroupId("sg-1");+ when(ec2Service.describeSecurityGroups(any(), eq(List.of("sg-1")), any(), any()))+ .thenReturn(List.of(sg));++ EcsTaskHandle handle = manager.startTask(awsvpcTask(), awsvpcTaskDef(List.of(new PortMapping(80, 80, "tcp"))),+ List.of(), "us-east-1");++ verify(builder).withNetworkMode("container:helper-id");+ verify(builder).withLabels(Map.of("floci.security-group-workload", "true"));+ verify(builder, never()).withPortBinding(anyInt(), anyInt());+ verify(builder, never()).withDynamicPort(anyInt());+ verify(builder, never()).withExposedPort(anyInt());+ verify(dockerClient).inspectContainerCmd("helper-id");+ verify(dockerClient, never()).inspectContainerCmd("docker-id");+ verify(firewallManager).register(any(), eq("helper-id"), any());+ assertEquals("eni-1", handle.getNetworkInterfaceId());+ assertEquals("us-east-1", handle.getRegion());+ }++ @Test+ void disabledFirewallDoesNotJoinNamespace() {+ when(containerDetector.isRunningInContainer()).thenReturn(false);+ when(firewallManager.enabled()).thenReturn(false);++ manager.startTask(awsvpcTask(), awsvpcTaskDef(List.of(new PortMapping(80, 80, "tcp"))),+ List.of(), "us-east-1");++ verify(builder, never()).withNetworkMode(anyString());+ verify(builder, never()).withLabels(Map.of("floci.security-group-workload", "true"));+ verify(ec2Service, never()).createNetworkInterface(any(), any(), any(), any(), any(), any(), any());+ verify(builder).withDynamicPort(80);+ }++ @Test+ void stopTaskUnregistersEni() {+ when(containerDetector.isRunningInContainer()).thenReturn(false);+ when(firewallManager.enabled()).thenReturn(true);+ when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any()))+ .thenReturn(new SecurityGroupFirewallManager.Namespace("helper-id", "10.0.0.5"));++ NetworkInterface eni = new NetworkInterface();+ eni.setNetworkInterfaceId("eni-1");+ eni.setVpcId("vpc-1");+ eni.setPrivateIpAddress("10.0.0.10");+ eni.setGroups(List.of(new GroupIdentifier("sg-1", "default")));+ when(ec2Service.createNetworkInterface(any(), eq("subnet-1"), any(), any(), any(), any(), any()))+ .thenReturn(eni);+ SecurityGroup sg = new SecurityGroup();+ sg.setGroupId("sg-1");+ when(ec2Service.describeSecurityGroups(any(), eq(List.of("sg-1")), any(), any()))+ .thenReturn(List.of(sg));++ DockerClient stopClient = mock(DockerClient.class, RETURNS_DEEP_STUBS);+ when(lifecycleManager.getDockerClient()).thenReturn(dockerClient, dockerClient, stopClient);++ EcsTaskHandle handle = manager.startTask(awsvpcTask(), awsvpcTaskDef(List.of()),+ List.of(), "us-east-1");+ manager.stopTaskAndCollectExitCodes(handle);++ verify(firewallManager).unregister("eni-1");+ verify(ec2Service).deleteNetworkInterface("us-east-1", "eni-1");+ }++ @Test+ void firelensRouterAndAppBothJoinTheHelperNamespace() {+ when(containerDetector.isRunningInContainer()).thenReturn(false);+ when(firewallManager.enabled()).thenReturn(true);+ when(firewallManager.createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any()))+ .thenReturn(new SecurityGroupFirewallManager.Namespace("helper-id", "10.0.0.5"));++ NetworkInterface eni = new NetworkInterface();+ eni.setNetworkInterfaceId("eni-1");+ eni.setVpcId("vpc-1");+ eni.setPrivateIpAddress("10.0.0.10");+ eni.setGroups(List.of(new GroupIdentifier("sg-1", "default")));+ when(ec2Service.createNetworkInterface(any(), eq("subnet-1"), any(), any(), any(), any(), any()))+ .thenReturn(eni);+ SecurityGroup sg = new SecurityGroup();+ sg.setGroupId("sg-1");+ when(ec2Service.describeSecurityGroups(any(), eq(List.of("sg-1")), any(), any()))+ .thenReturn(List.of(sg));++ when(lifecycleManager.create(any())).thenReturn("router-id");+ when(lifecycleManager.startCreated(anyString(), any()))+ .thenReturn(new ContainerInfo("router-id", Map.of()));+ var inspectVolumeCmd = mock(com.github.dockerjava.api.command.InspectVolumeCmd.class);+ var volume = mock(com.github.dockerjava.api.command.InspectVolumeResponse.class);+ when(dockerClient.inspectVolumeCmd(anyString())).thenReturn(inspectVolumeCmd);+ when(inspectVolumeCmd.exec()).thenReturn(volume);+ when(volume.getMountpoint()).thenReturn("/var/lib/docker/volumes/floci-ecs-firelens-abc123/_data");+ var copyCmd = mock(com.github.dockerjava.api.command.CopyArchiveToContainerCmd.class, RETURNS_SELF);+ when(dockerClient.copyArchiveToContainerCmd("router-id")).thenReturn(copyCmd);++ ContainerDefinition router = new ContainerDefinition();+ router.setName("log_router");+ router.setImage("amazon/aws-for-fluent-bit:stable");+ router.setFirelensConfiguration(new io.github.hectorvent.floci.services.ecs.model.FirelensConfiguration(+ "fluentbit", Map.of()));+ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage("app:latest");+ app.setLogConfiguration(new io.github.hectorvent.floci.services.ecs.model.LogConfiguration(+ "awsfirelens", Map.of("Name", "cloudwatch"), null));++ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("firelens-family");+ taskDef.setRevision(1);+ taskDef.setNetworkMode(NetworkMode.awsvpc);+ taskDef.setContainerDefinitions(List.of(app, router));++ manager.startTask(awsvpcTask(), taskDef, List.of(), "us-east-1");++ verify(builder, times(2)).withNetworkMode("container:helper-id");+ verify(builder, times(2)).withLabels(Map.of("floci.security-group-workload", "true"));++ verify(firewallManager).createNamespace(eq("ecs"), eq("abc123"), any(), any(), any(), any());+ verify(lifecycleManager).create(any());+ verify(lifecycleManager).startCreated(eq("router-id"), any());+ verify(lifecycleManager).createAndStart(any());+ }++ private static EcsTask awsvpcTask() {+ AwsVpcConfiguration awsvpc = new AwsVpcConfiguration();+ awsvpc.setSubnets(List.of("subnet-1"));+ awsvpc.setSecurityGroups(List.of("sg-1"));+ NetworkConfiguration network = new NetworkConfiguration();+ network.setAwsvpcConfiguration(awsvpc);+ EcsTask task = new EcsTask();+ task.setTaskArn("arn:aws:ecs:us-east-1:000000000000:task/test-cluster/abc123");+ task.setNetworkConfiguration(network);+ return task;+ }++ private static TaskDefinition awsvpcTaskDef(List<PortMapping> ports) {+ ContainerDefinition app = new ContainerDefinition();+ app.setName("app");+ app.setImage("app:latest");+ app.setPortMappings(ports);+ TaskDefinition taskDef = new TaskDefinition();+ taskDef.setFamily("test-family");+ taskDef.setNetworkMode(NetworkMode.awsvpc);+ taskDef.setContainerDefinitions(List.of(app));+ return taskDef;+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerVolumesTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerVolumesTest.javaindex 8fb31968ae..94b944f5d1 100644--- a/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerVolumesTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManagerVolumesTest.java@@ -17,6 +17,7 @@ import io.github.hectorvent.floci.services.ecs.model.MountPoint;import io.github.hectorvent.floci.services.ecs.model.TaskDefinition;import io.github.hectorvent.floci.services.ecs.model.Volume;import io.github.hectorvent.floci.services.secretsmanager.SecretsManagerService;+import io.github.hectorvent.floci.services.s3.S3Service;import io.github.hectorvent.floci.services.ssm.SsmService;import org.junit.jupiter.api.BeforeEach;import org.junit.jupiter.api.Test;@@ -87,7 +88,7 @@ class EcsContainerManagerVolumesTest {manager = new EcsContainerManager(containerBuilder, lifecycleManager, logStreamer,containerDetector, config, regionResolver, awsEnv, ssmService, secretsManagerService,- ecrRegistryManager, new HostVolumePolicy(config));+ mock(S3Service.class), ecrRegistryManager, new HostVolumePolicy(config));}@Test@@ -182,7 +183,7 @@ class EcsContainerManagerVolumesTest {EcsContainerManager configured = new EcsContainerManager(containerBuilder, lifecycleManager,mock(ContainerLogStreamer.class), mock(ContainerDetector.class), cfg,mock(RegionResolver.class), awsEnv, mock(SsmService.class),- mock(SecretsManagerService.class), ecrRegistryManager, new HostVolumePolicy(cfg));+ mock(SecretsManagerService.class), mock(S3Service.class), ecrRegistryManager, new HostVolumePolicy(cfg));ContainerDefinition app = new ContainerDefinition();app.setName("app");@@ -217,7 +218,8 @@ class EcsContainerManagerVolumesTest {EcsContainerManager configured = new EcsContainerManager(containerBuilder, lifecycleManager,mock(ContainerLogStreamer.class), mock(ContainerDetector.class), cfg,mock(RegionResolver.class), awsEnv, mock(SsmService.class),- mock(SecretsManagerService.class), ecrRegistryManager, new HostVolumePolicy(cfg));+ mock(SecretsManagerService.class), mock(S3Service.class),+ ecrRegistryManager, new HostVolumePolicy(cfg));ContainerDefinition app = new ContainerDefinition();app.setName("app");diff --git a/src/test/java/io/github/hectorvent/floci/services/ecs/container/FirelensConfigGeneratorTest.java b/src/test/java/io/github/hectorvent/floci/services/ecs/container/FirelensConfigGeneratorTest.javanew file mode 100644index 0000000000..bd244afea8--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/ecs/container/FirelensConfigGeneratorTest.java@@ -0,0 +1,331 @@+package io.github.hectorvent.floci.services.ecs.container;++import org.junit.jupiter.api.Test;++import java.util.LinkedHashMap;+import java.util.Map;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;++class FirelensConfigGeneratorTest {++ @Test+ void fluentBitConfigMatchesEcsAgentShape() {+ LinkedHashMap<String, String> options = new LinkedHashMap<>();+ options.put("Name", "cloudwatch");+ options.put("region", "us-east-1");+ options.put("log_group_name", "app");+ options.put("include-pattern", "*failure*");+ options.put("exclude-pattern", "*success*");+ options.put("log-driver-buffer-limit", "123");++ LinkedHashMap<String, Map<String, String>> byContainer = new LinkedHashMap<>();+ byContainer.put("app", options);++ String config = FirelensConfigGenerator.fluentBitConfig(new FirelensConfigGenerator.Context(+ "awsvpc", true, "mycluster",+ "arn:aws:ecs:us-east-1:000000000000:task/mycluster/abc",+ "taskdefinition:1", 100, "/extra.conf", null, byContainer));++ assertEquals("""+ [INPUT]+ Name forward+ unix_path /var/run/fluent.sock+ Mem_Buf_Limit 50MB++ [INPUT]+ Name forward+ Listen 0.0.0.0+ Port 24224++ [INPUT]+ Name tcp+ Tag firelens-healthcheck+ Listen 127.0.0.1+ Port 8877++ [FILTER]+ Name grep+ Match app-firelens*+ Regex log *failure*++ [FILTER]+ Name grep+ Match app-firelens*+ Exclude log *success*++ [FILTER]+ Name record_modifier+ Match *+ Record ecs_cluster mycluster+ Record ecs_task_arn arn:aws:ecs:us-east-1:000000000000:task/mycluster/abc+ Record ecs_task_definition taskdefinition:1++ @INCLUDE /extra.conf++ [OUTPUT]+ Name null+ Match firelens-healthcheck++ [OUTPUT]+ Name cloudwatch+ Match app-firelens*+ region us-east-1+ log_group_name app++ """, config);+ }++ @Test+ void skipsGeneratedOutputWhenOnlyCustomFileIsUsed() {+ LinkedHashMap<String, Map<String, String>> byContainer = new LinkedHashMap<>();+ byContainer.put("app", Map.of());++ String config = FirelensConfigGenerator.fluentBitConfig(new FirelensConfigGenerator.Context(+ "bridge", false, "c", "arn", "fam:1", 0, null, null, byContainer));++ assertTrue(config.contains("Listen 0.0.0.0"));+ assertTrue(config.contains("Mem_Buf_Limit 25MB"));+ assertTrue(!config.contains("record_modifier"));+ assertTrue(!config.contains("Match app-firelens*"));+ }++ @Test+ void rejectsPluginOptionsWithoutName() {+ LinkedHashMap<String, Map<String, String>> byContainer = new LinkedHashMap<>();+ byContainer.put("app", Map.of("region", "us-east-1"));++ assertThrows(IllegalArgumentException.class, () -> FirelensConfigGenerator.fluentBitConfig(+ new FirelensConfigGenerator.Context(+ "bridge", false, "c", "arn", "fam:1", 0, null, null, byContainer)));+ }++ @Test+ void pointsAwsOutputsAtFlociUnlessTheTaskDefinitionSetAnEndpoint() {+ LinkedHashMap<String, String> app = new LinkedHashMap<>();+ app.put("Name", "s3");+ app.put("region", "us-east-1");+ app.put("bucket", "logs");++ LinkedHashMap<String, String> explicit = new LinkedHashMap<>();+ explicit.put("Name", "s3");+ explicit.put("endpoint", "https://real.example.com");++ LinkedHashMap<String, String> http = new LinkedHashMap<>();+ http.put("Name", "s3");+ http.put("endpoint", "http://minio:9000");++ LinkedHashMap<String, String> metrics = new LinkedHashMap<>();+ metrics.put("Name", "stdout");++ LinkedHashMap<String, Map<String, String>> byContainer = new LinkedHashMap<>();+ byContainer.put("app", app);+ byContainer.put("explicit", explicit);+ byContainer.put("http", http);+ byContainer.put("metrics", metrics);++ String config = FirelensConfigGenerator.fluentBitConfig(new FirelensConfigGenerator.Context(+ "bridge", false, "c", "arn", "fam:1", 0, null, "http://host.docker.internal:4566",+ byContainer));++ assertEquals("""+ [INPUT]+ Name forward+ unix_path /var/run/fluent.sock+ Mem_Buf_Limit 25MB++ [INPUT]+ Name forward+ Listen 0.0.0.0+ Port 24224++ [INPUT]+ Name tcp+ Tag firelens-healthcheck+ Listen 127.0.0.1+ Port 8877++ [OUTPUT]+ Name null+ Match firelens-healthcheck++ [OUTPUT]+ Name s3+ Match app-firelens*+ region us-east-1+ bucket logs+ Endpoint http://host.docker.internal:4566+ tls Off++ [OUTPUT]+ Name s3+ Match explicit-firelens*+ endpoint https://real.example.com++ [OUTPUT]+ Name s3+ Match http-firelens*+ endpoint http://minio:9000+ tls Off++ [OUTPUT]+ Name stdout+ Match metrics-firelens*++ """, config);+ }++ /**+ * The upstream C plugins read {@code endpoint} as a bare host name and always dial TLS, so a+ * Floci URL is a guaranteed DNS failure and a bare Floci host is a certificate failure. The+ * Go plugins parse a URL, so those are still pointed at Floci.+ */+ @Test+ void injectsEndpointOnlyIntoPluginsThatReadAUrl() {+ LinkedHashMap<String, String> kinesis = new LinkedHashMap<>();+ kinesis.put("Name", "kinesis_streams");+ kinesis.put("region", "us-east-1");+ kinesis.put("stream", "logs");++ LinkedHashMap<String, String> cwLogs = new LinkedHashMap<>();+ cwLogs.put("Name", "cloudwatch_logs");+ cwLogs.put("region", "us-east-1");+ cwLogs.put("log_group_name", "app");++ LinkedHashMap<String, String> goCloudwatch = new LinkedHashMap<>();+ goCloudwatch.put("Name", "cloudwatch");+ goCloudwatch.put("region", "us-east-1");+ goCloudwatch.put("log_group_name", "app");++ LinkedHashMap<String, Map<String, String>> byContainer = new LinkedHashMap<>();+ byContainer.put("kinesis", kinesis);+ byContainer.put("cwlogs", cwLogs);+ byContainer.put("gocw", goCloudwatch);++ String config = FirelensConfigGenerator.fluentBitConfig(new FirelensConfigGenerator.Context(+ "bridge", false, "c", "arn", "fam:1", 0, null, "http://host.docker.internal:4566",+ byContainer));++ assertTrue(config.contains("""+ [OUTPUT]+ Name kinesis_streams+ Match kinesis-firelens*+ region us-east-1+ stream logs++ """), config);+ assertTrue(config.contains("""+ [OUTPUT]+ Name cloudwatch_logs+ Match cwlogs-firelens*+ region us-east-1+ log_group_name app++ """), config);+ assertTrue(config.contains("""+ [OUTPUT]+ Name cloudwatch+ Match gocw-firelens*+ region us-east-1+ log_group_name app+ Endpoint http://host.docker.internal:4566+ tls Off++ """), config);+ }+ @Test+ void fluentdConfigMatchesEcsAgentShape() {+ LinkedHashMap<String, String> options = new LinkedHashMap<>();+ options.put("@type", "kinesis_firehose");+ options.put("region", "us-east-1");+ options.put("delivery_stream_name", "demo-stream");+ options.put("include-pattern", "*failure*");+ options.put("exclude-pattern", "*success*");+ options.put("log-driver-buffer-limit", "123");++ LinkedHashMap<String, Map<String, String>> byContainer = new LinkedHashMap<>();+ byContainer.put("app", options);++ String config = FirelensConfigGenerator.fluentdConfig(new FirelensConfigGenerator.Context(+ "awsvpc", true, "mycluster",+ "arn:aws:ecs:us-east-1:000000000000:task/mycluster/abc",+ "taskdefinition:1", 100, "/fluentd/etc/extra.conf",+ "http://host.docker.internal:4566", byContainer));++ assertEquals("""+ <source>+ @type unix+ path /var/run/fluent.sock+ </source>++ <source>+ @type forward+ bind 0.0.0.0+ port 24224+ </source>++ <filter app-firelens**>+ @type grep+ <regexp>+ key log+ pattern *failure*+ </regexp>+ </filter>++ <filter app-firelens**>+ @type grep+ <exclude>+ key log+ pattern *success*+ </exclude>+ </filter>++ <filter **>+ @type record_transformer+ <record>+ ecs_cluster mycluster+ ecs_task_arn arn:aws:ecs:us-east-1:000000000000:task/mycluster/abc+ ecs_task_definition taskdefinition:1+ </record>+ </filter>++ @include /fluentd/etc/extra.conf++ <match app-firelens**>+ @type kinesis_firehose+ region us-east-1+ delivery_stream_name demo-stream+ </match>++ """, config);+ assertTrue(!config.contains("Endpoint"));+ assertTrue(!config.contains("firelens-healthcheck"));+ }++ @Test+ void fluentdSkipsGeneratedOutputWhenOnlyCustomFileIsUsed() {+ LinkedHashMap<String, Map<String, String>> byContainer = new LinkedHashMap<>();+ byContainer.put("app", Map.of());++ String config = FirelensConfigGenerator.fluentdConfig(new FirelensConfigGenerator.Context(+ "bridge", false, "c", "arn", "fam:1", 0, null, null, byContainer));++ assertTrue(config.contains("@type unix"));+ assertTrue(config.contains("@type forward"));+ assertTrue(!config.contains("record_transformer"));+ assertTrue(!config.contains("<match app-firelens**>"));+ }++ @Test+ void fluentdRejectsPluginOptionsWithoutType() {+ LinkedHashMap<String, Map<String, String>> byContainer = new LinkedHashMap<>();+ byContainer.put("app", Map.of("region", "us-east-1"));++ assertThrows(IllegalArgumentException.class, () -> FirelensConfigGenerator.fluentdConfig(+ new FirelensConfigGenerator.Context(+ "bridge", false, "c", "arn", "fam:1", 0, null, null, byContainer)));+ }++}