Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(network): add opt-in security group enforcement (#3681)

Alexy Grabov7813d14ee4bb
diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.md
index b2a81f2e44..fc80e0f894 100644
--- a/docs/configuration/environment-variables.md
+++ b/docs/configuration/environment-variables.md
@@ -85,8 +85,12 @@ See [Storage Modes](./storage.md) for a full explanation of each mode.
## Docker Daemon
+This foundation release provides opt-in security-group filtering for EC2 Docker instances and ECS `awsvpc` tasks. A rootful Linux Docker daemon with nftables support is required. Before enabling it, terminate existing EC2 instances and ECS tasks, then launch replacements so their namespaces are prepared before application code starts. Restart Floci after changing this setting. Mock mode remains a control-plane simulation and does not filter packets.
+
| Variable | Default | Description |
|---|---|---|
+| `FLOCI_NETWORK_SECURITY_GROUP_ENFORCEMENT_ENABLED` | `false` | Opt in to filtering Docker-backed EC2 and ECS `awsvpc` traffic by attached security groups |
+| `FLOCI_NETWORK_SECURITY_GROUP_ENFORCEMENT_HELPER_IMAGE` | `floci/network-helper:local` | Linux helper image containing nftables; Floci builds the default image locally when missing |
| `FLOCI_DOCKER_DOCKER_HOST` | `unix:///var/run/docker.sock` | Docker daemon socket path or TCP address |
| `FLOCI_DOCKER_DOCKER_CONFIG_PATH` | _(none)_ | Path to a directory containing Docker's `config.json` for registry auth |
| `FLOCI_DOCKER_IMAGE_REGISTRY_BASE` | _(none)_ | Optional registry/repository base for every Docker image Floci launches. When set, `postgres:16-alpine` resolves as `<base>/postgres:16-alpine` and `public.ecr.aws/docker/library/ubuntu:24.04` resolves as `<base>/public.ecr.aws/docker/library/ubuntu:24.04` |
diff --git a/docs/services/ec2.md b/docs/services/ec2.md
index d0eaa60a2d..13bdedca92 100644
--- a/docs/services/ec2.md
+++ b/docs/services/ec2.md
@@ -78,7 +78,11 @@ Key pairs created with `CreateKeyPair` contain dummy private key material. Impor
## Security Group Port Publishing
-When an instance's security groups open a TCP port to a CIDR source, Floci publishes that port on the host so you can reach the app from `localhost`. For each opened port Floci starts a small `alpine/socat` sidecar container that binds an allocated host port (default range 30000–30999) and forwards it to the instance container's IP. This works both for rules present at launch and for rules added later with `authorize-security-group-ingress`; revoking the rule removes the forward. The mapping (`app port -> host port`) is written to the logs:
+With `FLOCI_NETWORK_SECURITY_GROUP_ENFORCEMENT_ENABLED=true`, Floci prepares a separate Linux network namespace and an nftables default-deny policy before starting each Docker-backed instance. It checks new managed connections against both the sender's egress rules and the receiver's ingress rules, using emulated ENI addresses and security-group membership before translating to Docker addresses. Established replies use connection tracking. This filters packets but does not emulate VPC routing, NACLs, NAT gateways, or peering. Rootless Docker and Windows containers are unsupported.
+
+Security-group permissions on a host-published port use the source visible inside Docker. Docker Desktop may replace the original external-client address. The legacy socat application publisher is disabled while enforcement is enabled because it obscures that source. SSH is published directly by the protected namespace. Application host ports remain available through the legacy publisher only when enforcement is explicitly disabled. Direct managed container traffic uses the logical ENI identity.
+
+When enforcement is disabled, an instance's security groups can open a TCP port to a CIDR source and Floci publishes that port on the host so you can reach the app from `localhost`. For each opened port Floci starts a small `alpine/socat` sidecar container that binds an allocated host port (default range 30000–30999) and forwards it to the instance container's IP. This works both for rules present at launch and for rules added later with `authorize-security-group-ingress`; revoking the rule removes the forward. The mapping (`app port -> host port`) is written to the logs:
```
Published EC2 instance i-0abc... app port 80 on host port 30000 (socat -> 172.17.0.3:80)
@@ -90,7 +94,7 @@ Notes and limitations:
- Only CIDR-sourced TCP rules are published. A port opened only to a referenced security group (or via a prefix list) is not published, matching AWS: those grant reachability from the referenced group's private IPs, not from the host. The source CIDR value itself is not enforced, so a CIDR-sourced port is reachable whether the rule is `0.0.0.0/0` or narrower.
- Ports are aggregated across all of the instance's security groups, SSH (22) is never re-forwarded, and any single rule whose port span exceeds `max-published-ports-per-instance` (default 20) is skipped so an allow-all range cannot spawn thousands of sidecars. The total published per instance is capped at the same limit.
- Stopping an instance tears down its forwards; starting it again does not automatically restore them (re-run `authorize-security-group-ingress`, or recreate the instance).
-- Set `publish-security-group-ports: false` (`FLOCI_SERVICES_EC2_PUBLISH_SECURITY_GROUP_PORTS=false`) to keep security groups as metadata only.
+- With enforcement disabled, set `publish-security-group-ports: false` (`FLOCI_SERVICES_EC2_PUBLISH_SECURITY_GROUP_PORTS=false`) to keep security groups as metadata only.
## UserData
diff --git a/docs/services/ecs.md b/docs/services/ecs.md
index eac56df605..7a1b4c1d39 100644
--- a/docs/services/ecs.md
+++ b/docs/services/ecs.md
@@ -199,6 +199,8 @@ unchanged.
## Configuration
+Docker-backed `awsvpc` tasks receive an emulated ENI and share one protected network namespace across their containers when `FLOCI_NETWORK_SECURITY_GROUP_ENFORCEMENT_ENABLED=true`. A task without explicit security groups uses its subnet VPC's default group. Containers in the same task can communicate over localhost. Bridge and host task networking do not attach task-level `awsvpc` security groups. Mock mode reports control-plane state and does not enforce packet filtering.
+
| Variable | Default | Description |
|---|---|---|
| `FLOCI_SERVICES_ECS_ENABLED` | `true` | Enable or disable the ECS service |
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index e7890cb421..0911a51c7d 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -78,6 +78,8 @@ public interface EmulatorConfig {
DnsConfig dns();
+ NetworkConfig network();
+
AuthConfig auth();
SecurityConfig security();
@@ -92,6 +94,18 @@ public interface EmulatorConfig {
ProtocolsConfig protocols();
+ interface NetworkConfig {
+ SecurityGroupEnforcementConfig securityGroupEnforcement();
+ }
+
+ interface SecurityGroupEnforcementConfig {
+ @WithDefault("false")
+ boolean enabled();
+
+ @WithDefault("floci/network-helper:local")
+ String helperImage();
+ }
+
interface ProtocolsConfig {
/**
* Maximum HTTP request body size, in megabytes. Feeds
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java
index 1e6feef48f..79a99b5baf 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManager.java
@@ -12,6 +12,7 @@ import com.github.dockerjava.api.command.ListVolumesResponse;
import com.github.dockerjava.api.exception.DockerException;
import com.github.dockerjava.api.exception.NotFoundException;
import com.github.dockerjava.api.model.Bind;
+import com.github.dockerjava.api.model.Capability;
import com.github.dockerjava.api.model.Container;
import com.github.dockerjava.api.model.ContainerNetwork;
import com.github.dockerjava.api.model.ExposedPort;
@@ -868,6 +869,14 @@ public class ContainerLifecycleManager {
if (spec.privileged()) {
hostConfig.withPrivileged(true);
}
+ if (spec.labels() != null && "true".equals(spec.labels().get("floci.security-group-workload"))) {
+ hostConfig.withCapDrop(Capability.NET_ADMIN, Capability.NET_RAW);
+ }
+ // The firewall helper only has to program nftables in the namespace it already owns,
+ // which needs CAP_NET_ADMIN and nothing else that privileged mode would also grant.
+ if (spec.labels() != null && "true".equals(spec.labels().get("floci.security-group-helper"))) {
+ hostConfig.withCapAdd(Capability.NET_ADMIN);
+ }
if (spec.cgroupnsMode() != null && !spec.cgroupnsMode().isBlank()) {
hostConfig.withCgroupnsMode(spec.cgroupnsMode());
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java
index a96159925d..ea0a2562e0 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManager.java
@@ -12,7 +12,9 @@ import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
import io.github.hectorvent.floci.core.common.docker.DockerHostResolver;
import io.github.hectorvent.floci.core.common.docker.PortAllocator;
import io.github.hectorvent.floci.services.ec2.model.Instance;
+import io.github.hectorvent.floci.services.ec2.model.InstanceNetworkInterface;
import io.github.hectorvent.floci.services.ec2.model.InstanceState;
+import io.github.hectorvent.floci.services.ec2.model.GroupIdentifier;
import io.github.hectorvent.floci.services.ec2.net.VpcNetworkManager;
import io.github.hectorvent.floci.services.ec2.portforward.Ec2PortForwardManager;
import com.github.dockerjava.api.DockerClient;
@@ -47,6 +49,7 @@ import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
+import io.github.hectorvent.floci.services.ec2.model.SecurityGroup;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ConcurrentHashMap;
@@ -60,6 +63,7 @@ import java.util.function.BiPredicate;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import java.util.stream.Collectors;
import java.util.zip.GZIPInputStream;
/**
@@ -171,13 +175,98 @@ public class Ec2ContainerManager {
private final ContainerNetworkReachability containerNetworkReachability;
private final VpcNetworkManager vpcNetworkManager;
private final ContainerReachableEndpoint reachableEndpoint;
+ private SecurityGroupFirewallManager firewallManager;
private final ExecutorService executor;
private final Duration userDataExecutionTimeout;
private final Set<ResultCallback<Frame>> activeUserDataCallbacks = ConcurrentHashMap.newKeySet();
private volatile boolean dockerUnavailableLogged;
+ public void refreshSecurityGroups(String region, Map<String, SecurityGroup> groups,
+ Map<String, List<String>> prefixLists) {
+ if (firewallManager != null && firewallManager.enabled()) {
+ firewallManager.refreshPolicies(region, groups, prefixLists);
+ }
+ }
+
+ public void updateSecurityGroups(String eniId, Set<String> groupIds,
+ Map<String, SecurityGroup> groups,
+ Map<String, List<String>> prefixLists) {
+ if (firewallManager != null && firewallManager.enabled()) {
+ firewallManager.updateGroups(eniId, groupIds, groups, prefixLists);
+ }
+ }
+
+ /** A workload's own Floci-owned namespace, or null when the instance is not protected. */
+ private ProtectedNamespace protectedNamespace(Instance instance) {
+ String containerId = instance.getDockerContainerId();
+ if (containerId == null) {
+ return null;
+ }
+ InspectContainerResponse worker = dockerClient.inspectContainerCmd(containerId).exec();
+ String mode = worker.getHostConfig() == null ? null : worker.getHostConfig().getNetworkMode();
+ if (mode == null || !mode.startsWith("container:")) {
+ return null;
+ }
+ String helperId = mode.substring("container:".length());
+ InspectContainerResponse helper = dockerClient.inspectContainerCmd(helperId).exec();
+ Map<String, String> labels = helper.getConfig().getLabels();
+ if (!"true".equals(labels.get("floci.security-group-helper"))
+ || !"ec2".equals(labels.get(LABEL_SERVICE))
+ || !instance.getInstanceId().equals(labels.get(LABEL_RESOURCE_ID))
+ || !ownerIdentity().equals(labels.get("floci_owner_port"))) {
+ throw new IllegalStateException("EC2 workload network namespace is not Floci protected");
+ }
+ String address = helper.getNetworkSettings().getNetworks().values().stream()
+ .map(network -> network.getIpAddress()).filter(ip -> ip != null && !ip.isBlank())
+ .findFirst().orElseThrow(() -> new IllegalStateException("EC2 firewall helper has no Docker IP"));
+ return new ProtectedNamespace(helperId, address);
+ }
+
+ private record ProtectedNamespace(String helperId, String transportAddress) {}
+
+ public void restoreSecurityGroups(Instance instance, String region, List<SecurityGroup> groups,
+ Map<String, List<String>> prefixLists) {
+ if (firewallManager == null || !firewallManager.enabled()) {
+ return;
+ }
+ ProtectedNamespace namespace = protectedNamespace(instance);
+ if (namespace == null) {
+ throw new IllegalStateException("EC2 workload has no protected network namespace");
+ }
+ String helperId = namespace.helperId();
+ String address = namespace.transportAddress();
+ InstanceNetworkInterface eni = instance.getNetworkInterfaces().getFirst();
+ String logical = instance.getLogicalPrivateIpAddress() != null
+ ? instance.getLogicalPrivateIpAddress() : eni.getPrivateIpAddress();
+ firewallManager.register(new SecurityGroupNftCompiler.Endpoint(regionResolver.getAccountId(), region,
+ instance.getVpcId(), eni.getNetworkInterfaceId(), logical, address,
+ instance.getSecurityGroups().stream().map(GroupIdentifier::getGroupId)
+ .collect(Collectors.toSet()), groups), helperId, prefixLists);
+ }
+
@Inject
+ public Ec2ContainerManager(ContainerBuilder containerBuilder,
+ ContainerLifecycleManager lifecycleManager,
+ ContainerLogStreamer logStreamer,
+ ContainerDetector containerDetector,
+ DockerHostResolver dockerHostResolver,
+ DockerClient dockerClient,
+ PortAllocator portAllocator,
+ EmulatorConfig config,
+ Ec2MetadataServer metadataServer,
+ Ec2PortForwardManager portForwardManager,
+ RegionResolver regionResolver,
+ ContainerNetworkReachability containerNetworkReachability,
+ VpcNetworkManager vpcNetworkManager,
+ ContainerReachableEndpoint reachableEndpoint,
+ SecurityGroupFirewallManager firewallManager) {
+ this(containerBuilder, lifecycleManager, logStreamer, containerDetector, dockerHostResolver,
+ dockerClient, portAllocator, config, metadataServer, portForwardManager, regionResolver,
+ containerNetworkReachability, vpcNetworkManager, reachableEndpoint);
+ this.firewallManager = firewallManager;
+ }
+
public Ec2ContainerManager(ContainerBuilder containerBuilder,
ContainerLifecycleManager lifecycleManager,
ContainerLogStreamer logStreamer,
@@ -278,6 +367,12 @@ public class Ec2ContainerManager {
* via socat sidecars once the container is running (empty for none)
*/
public void launch(Instance instance, ResolvedAmiImage image, String publicKey, String region, Set<Integer> appPorts) {
+ launch(instance, image, publicKey, region, appPorts, List.of(), Map.of());
+ }
+
+ public void launch(Instance instance, ResolvedAmiImage image, String publicKey, String region,
+ Set<Integer> appPorts, List<SecurityGroup> groups,
+ Map<String, List<String>> prefixLists) {
instance.setState(InstanceState.pending());
// An instance record is metadata: id, addresses, tags and lifecycle state are all
@@ -286,7 +381,11 @@ public class Ec2ContainerManager {
// instead of dying: Floci in Docker without a mounted socket, or a stopped daemon
// on the host, would otherwise terminate every instance the moment it launched.
if (!isDockerAvailable()) {
- markContainerlessRunning(instance);
+ if (firewallManager != null && firewallManager.enabled()) {
+ failLaunch(instance);
+ } else {
+ markContainerlessRunning(instance);
+ }
return;
}
@@ -303,14 +402,14 @@ public class Ec2ContainerManager {
String flociHost = dockerHostResolver.resolve();
int imdsPort = config.services().ec2().imdsPort();
StartedContainer started = createAndStartContainer(instance, image, region, flociHost, imdsPort,
- leasedPrivateIp);
+ leasedPrivateIp, groups, prefixLists);
if (started == null) {
return;
}
int sshHostPort = started.sshHostPort();
String containerId = started.containerId();
String vpcAddress = started.vpcAddress();
- if (vpcAddress == null) {
+ if (vpcAddress == null && started.namespace() == null) {
// Nothing holds the address, and moments from now this instance will be
// reporting its bridge address instead, so the lease would no longer be
// findable from the instance at terminate time. Give it back here.
@@ -350,8 +449,10 @@ public class Ec2ContainerManager {
// Docker can report the container as running before network
// settings are populated; wait here so IMDS is registered
// before link-local metadata validation and UserData run.
- String containerIp = waitForContainerBridgeIp(containerId, instanceId, instance);
- if (vpcAddress != null) {
+ String containerIp = started.namespace() == null
+ ? waitForContainerBridgeIp(containerId, instanceId, instance)
+ : started.namespace().transportAddress();
+ if (vpcAddress != null && started.namespace() == null) {
// The VPC address is the one Floci reports and the one peers in the same VPC
// dial. The bridge address still identifies this container to IMDS, because
// the default route, and so the source address of its metadata requests, is
@@ -364,7 +465,10 @@ public class Ec2ContainerManager {
}
if (containerIp != null && !containerIp.isBlank()) {
instance.setContainerBridgeIp(containerIp);
- exposeReachablePrivateAddress(instance, containerIp, config.services().ec2().awsFaithfulPrivateIp());
+ if (started.namespace() == null) {
+ exposeReachablePrivateAddress(instance, containerIp,
+ config.services().ec2().awsFaithfulPrivateIp());
+ }
metadataServer.registerContainer(containerIp, instanceId, instance);
}
else {
@@ -393,7 +497,8 @@ public class Ec2ContainerManager {
// IMDS proxy setup is best effort and has its own bounded commands. The instance
// is already running once Docker has assigned its reachable network address.
- configureLinkLocalMetadataEndpoint(containerId, instanceId, flociHost, imdsPort);
+ configureLinkLocalMetadataEndpoint(started.namespace() == null
+ ? containerId : started.namespace().helperId(), instanceId, flociHost, imdsPort);
// Publish security-group TCP ingress ports on the host via socat sidecars.
if (appPorts != null && !appPorts.isEmpty()) {
@@ -425,7 +530,7 @@ public class Ec2ContainerManager {
// The daemon can disappear between the probe above and any of the calls in
// this block. Losing Docker is not the instance's fault, so degrade to a
// metadata-only instance; a genuine container failure still fails the launch.
- if (isDockerAvailable()) {
+ if ((firewallManager != null && firewallManager.enabled()) || isDockerAvailable()) {
failLaunch(instance, leasedPrivateIp);
} else {
markContainerlessRunning(instance);
@@ -440,7 +545,9 @@ public class Ec2ContainerManager {
}
private StartedContainer createAndStartContainer(Instance instance, ResolvedAmiImage image, String region,
- String flociHost, int imdsPort, String leasedPrivateIp) {
+ String flociHost, int imdsPort, String leasedPrivateIp,
+ List<SecurityGroup> groups,
+ Map<String, List<String>> prefixLists) {
String instanceId = instance.getInstanceId();
String containerName = ContainerStorageHelper.resourceName(config, "ec2", null, instanceId);
String imdsEndpoint = "http://" + flociHost + ":" + imdsPort;
@@ -453,16 +560,38 @@ public class Ec2ContainerManager {
int sshHostPort = portAllocator.allocate(
config.services().ec2().sshPortRangeStart(),
config.services().ec2().sshPortRangeEnd());
- ContainerSpec spec = buildContainerSpec(containerName, image, region, serviceEndpoint, imdsEndpoint,
- instanceId, sshHostPort);
+ SecurityGroupFirewallManager.Namespace namespace = null;
+ String eniId = null;
String containerId = null;
boolean recorded = false;
try {
+ if (firewallManager != null && firewallManager.enabled()) {
+ if (instance.getNetworkInterfaces() == null || instance.getNetworkInterfaces().isEmpty()) {
+ throw new IllegalStateException("EC2 instance has no network interface to protect");
+ }
+ InstanceNetworkInterface eni = instance.getNetworkInterfaces().getFirst();
+ eniId = eni.getNetworkInterfaceId();
+ instance.setLogicalPrivateIpAddress(eni.getPrivateIpAddress());
+ namespace = firewallManager.createNamespace("ec2", instanceId,
+ regionResolver.getAccountId(), region, Optional.empty(), Map.of(22, sshHostPort));
+ firewallManager.register(new SecurityGroupNftCompiler.Endpoint(regionResolver.getAccountId(),
+ region, instance.getVpcId(), eniId, eni.getPrivateIpAddress(),
+ namespace.transportAddress(),
+ instance.getSecurityGroups().stream().map(g -> g.getGroupId())
+ .collect(Collectors.toSet()), groups),
+ namespace.helperId(), prefixLists);
+ }
+ ContainerSpec spec = buildContainerSpec(containerName, image, region, serviceEndpoint, imdsEndpoint,
+ instanceId, sshHostPort, namespace);
containerId = image.dockerPlatform() == null
? lifecycleManager.create(spec)
: lifecycleManager.create(spec, image.dockerPlatform());
if (!recordCreatedContainer(instance, containerId, sshHostPort)) {
lifecycleManager.removeIfExists(containerId);
+ if (namespace != null) {
+ firewallManager.unregister(eniId);
+ lifecycleManager.removeIfExists(namespace.helperId());
+ }
portAllocator.release(sshHostPort);
return null;
}
@@ -471,12 +600,12 @@ public class Ec2ContainerManager {
// container starts, so the guest comes up already holding its private IP. The
// default bridge attachment stays: it is what carries the published SSH host
// port, which a network mode set at creation time would suppress.
- String vpcAddress = vpcNetworkManager.attach(region, instance.getVpcId(), instance.getSubnetId(),
- containerId, leasedPrivateIp)
- .map(network -> leasedPrivateIp)
- .orElse(null);
+ String vpcAddress = namespace == null
+ ? vpcNetworkManager.attach(region, instance.getVpcId(), instance.getSubnetId(),
+ containerId, leasedPrivateIp).map(network -> leasedPrivateIp).orElse(null)
+ : null;
lifecycleManager.startCreated(containerId, spec);
- return new StartedContainer(containerId, sshHostPort, vpcAddress);
+ return new StartedContainer(containerId, sshHostPort, vpcAddress, namespace, eniId);
} catch (Exception e) {
boolean ownsCleanup = !recorded || clearRecordedContainer(instance, containerId, sshHostPort);
if (!ownsCleanup) {
@@ -485,6 +614,12 @@ public class Ec2ContainerManager {
if (containerId != null) {
lifecycleManager.removeIfExists(containerId);
}
+ if (namespace != null) {
+ if (eniId != null) {
+ firewallManager.unregister(eniId);
+ }
+ lifecycleManager.removeIfExists(namespace.helperId());
+ }
if (isHostPortCollision(e)) {
// Docker Desktop can own a published port without exposing it to a host-side
// ServerSocket probe. Keep it unavailable for this process and try the next port.
@@ -501,7 +636,7 @@ public class Ec2ContainerManager {
private ContainerSpec buildContainerSpec(String containerName, ResolvedAmiImage image, String region,
String serviceEndpoint, String imdsEndpoint, String instanceId,
- int sshHostPort) {
+ int sshHostPort, SecurityGroupFirewallManager.Namespace namespace) {
// Minimal images keep the historic tail command, while cloud-image AMI guests can boot their init.
ContainerBuilder.Builder specBuilder = containerBuilder.newContainer(image.dockerImage())
.withName(containerName)
@@ -509,7 +644,6 @@ public class Ec2ContainerManager {
.withDockerNetwork(Optional.empty())
.withEnv(localAwsEnvironment(region, serviceEndpoint, imdsEndpoint))
.withEnv("AWS_EC2_INSTANCE_ID", instanceId)
- .withPortBinding(22, sshHostPort)
.withHostDockerInternalOnLinux()
.withLogRotation()
.withLabels(ContainerStorageHelper.resourceIdentityLabels(
@@ -519,8 +653,14 @@ public class Ec2ContainerManager {
.withLabels(Map.of(LABEL_OWNER_PORT, ownerIdentity()))
// EC2 instances expose IMDS on 169.254.169.254. Floci needs network administration
// privileges in the local container to attach that link-local address.
- .withPrivileged(true)
+ .withPrivileged(namespace == null)
.withCmd(image.systemd() ? List.of("/sbin/init") : List.of("tail", "-f", "/dev/null"));
+ if (namespace == null) {
+ specBuilder.withPortBinding(22, sshHostPort);
+ } else {
+ specBuilder.withNetworkMode("container:" + namespace.helperId());
+ specBuilder.withLabels(Map.of("floci.security-group-workload", "true"));
+ }
if (image.systemd()) {
specBuilder
.withCgroupnsMode("host")
@@ -585,6 +725,10 @@ public class Ec2ContainerManager {
LOG.warnv("Error removing failed EC2 container {0}: {1}", containerId, e.getMessage());
}
}
+ if (firewallManager != null && firewallManager.enabled()
+ && instance.getNetworkInterfaces() != null && !instance.getNetworkInterfaces().isEmpty()) {
+ firewallManager.unregister(instance.getNetworkInterfaces().getFirst().getNetworkInterfaceId());
+ }
if (sshHostPort > 0) {
portAllocator.release(sshHostPort);
}
@@ -675,7 +819,8 @@ public class Ec2ContainerManager {
return false;
}
- private record StartedContainer(String containerId, int sshHostPort, String vpcAddress) {
+ private record StartedContainer(String containerId, int sshHostPort, String vpcAddress,
+ SecurityGroupFirewallManager.Namespace namespace, String eniId) {
}
/**
@@ -781,10 +926,19 @@ public class Ec2ContainerManager {
}
}
String instanceId = instance.getInstanceId();
- String containerIp = waitForContainerBridgeIp(containerId, instanceId);
+ // A protected workload shares the helper's namespace, so it has no Docker network
+ // attachment of its own to rediscover: the helper keeps the transport address
+ // across the workload's stop/start, exactly as launch used it.
+ ProtectedNamespace namespace = protectedNamespace(instance);
+ String containerIp = namespace == null
+ ? waitForContainerBridgeIp(containerId, instanceId)
+ : namespace.transportAddress();
if (containerIp != null && !containerIp.isBlank()) {
instance.setContainerBridgeIp(containerIp);
- exposeReachablePrivateAddress(instance, containerIp, config.services().ec2().awsFaithfulPrivateIp());
+ if (namespace == null) {
+ exposeReachablePrivateAddress(instance, containerIp,
+ config.services().ec2().awsFaithfulPrivateIp());
+ }
// Docker hands out a new bridge IP on restart, so the previously reported
// public address can now point at another container entirely.
if (instance.isAssociatePublicIp() || instance.getPublicIpAddress() != null) {
@@ -796,8 +950,12 @@ public class Ec2ContainerManager {
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
+ instance.setState(InstanceState.stopped());
+ return;
} catch (Exception e) {
LOG.warnv("Error starting EC2 container {0}: {1}", containerId, e.getMessage());
+ instance.setState(InstanceState.stopped());
+ return;
}
instance.setState(InstanceState.running());
});
@@ -1095,6 +1253,10 @@ public class Ec2ContainerManager {
Thread.currentThread().interrupt();
}
}
+ if (firewallManager != null && firewallManager.enabled()
+ && instance.getNetworkInterfaces() != null && !instance.getNetworkInterfaces().isEmpty()) {
+ firewallManager.unregister(instance.getNetworkInterfaces().getFirst().getNetworkInterfaceId());
+ }
if (sshHostPort > 0) {
portAllocator.release(sshHostPort);
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2Service.java b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2Service.java
index b738be134f..1e842cfb38 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2Service.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2Service.java
@@ -23,6 +23,7 @@ import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Function;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
import org.jboss.logging.Logger;
@@ -439,6 +440,15 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
if (!needsMetadataRegistration(instance)) {
continue;
}
+ try {
+ restoreInstanceFirewall(instance);
+ } catch (Exception e) {
+ LOG.warnv("Could not restore EC2 firewall for {0}: {1}", instance.getInstanceId(), e.getMessage());
+ containerManager.stopForShutdown(instance);
+ instance.setState(InstanceState.stopped());
+ instances.put(key, instance);
+ continue;
+ }
if (containerManager.restoreMetadataRegistration(instance)) {
instances.put(key, instance);
restored++;
@@ -1053,6 +1063,7 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
}
list.setState("modify-complete");
managedPrefixLists.put(key(region, prefixListId), list);
+ reconcileFirewallPolicies(region);
return list;
}
}
@@ -1063,6 +1074,7 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
requireCustomerManaged(list, "deleted");
managedPrefixLists.delete(key(region, prefixListId));
tags.delete(prefixListId);
+ reconcileFirewallPolicies(region);
// AWS reports delete-complete on the returned object even though it is now gone.
list.setState("delete-complete");
return list;
@@ -2731,7 +2743,12 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
}
}
for (Instance inst : launched) {
- containerManager.launch(inst, dockerImage, publicKey, region, desiredPublishedPorts(region, inst));
+ List<SecurityGroup> policyGroups = inst.getSecurityGroups().stream()
+ .map(group -> securityGroups.get(key(region, group.getGroupId()))
+ .orElseThrow(() -> new IllegalStateException("Missing security group " + group.getGroupId())))
+ .toList();
+ containerManager.launch(inst, dockerImage, publicKey, region, desiredPublishedPorts(region, inst),
+ policyGroups, policyPrefixLists(region, policyGroups));
}
}
@@ -2804,12 +2821,52 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
sgs, config.services().ec2().maxPublishedPortsPerInstance());
}
+ private Map<String, List<String>> policyPrefixLists(String region, List<SecurityGroup> groups) {
+ Map<String, List<String>> resolved = new LinkedHashMap<>();
+ for (SecurityGroup group : groups) {
+ Stream.concat(group.getIpPermissions().stream(),
+ group.getIpPermissionsEgress().stream())
+ .flatMap(permission -> permission.getPrefixListIds().stream())
+ .map(PrefixListId::getPrefixListId).distinct()
+ .forEach(id -> resolved.put(id, managedPrefixLists.get(key(region, id)).isEmpty()
+ ? List.of() : getManagedPrefixListEntries(region, id, null).stream()
+ .map(PrefixListEntry::getCidr).toList()));
+ }
+ return resolved;
+ }
+
+ private boolean securityGroupEnforcementEnabled() {
+ return config.network() != null && config.network().securityGroupEnforcement() != null
+ && config.network().securityGroupEnforcement().enabled();
+ }
+
+ private void reconcileFirewallPolicies(String region) {
+ if (!securityGroupEnforcementEnabled() || config.services().ec2().mock()) {
+ return;
+ }
+ List<SecurityGroup> current = securityGroups.scan(k -> k.startsWith(region + "::"));
+ Map<String, SecurityGroup> byId = current.stream()
+ .collect(Collectors.toMap(SecurityGroup::getGroupId, Function.identity()));
+ containerManager.refreshSecurityGroups(region, byId, policyPrefixLists(region, current));
+ }
+
+ private void restoreInstanceFirewall(Instance instance) {
+ if (!securityGroupEnforcementEnabled()) {
+ return;
+ }
+ String region = instance.getRegion();
+ List<SecurityGroup> attached = instance.getSecurityGroups().stream()
+ .map(group -> getRequiredSecurityGroup(region, group.getGroupId())).toList();
+ containerManager.restoreSecurityGroups(instance, region, attached, policyPrefixLists(region, attached));
+ }
+
/**
* Re-publishes host forwards for every running instance attached to the given security group,
* so ports opened or closed via authorize/revoke ingress take effect on already-running
* instances. No-op in mock mode or when publishing is disabled.
*/
private void reconcilePublishedPortsForGroup(String region, String groupId) {
+ reconcileFirewallPolicies(region);
if (!config.services().ec2().publishSecurityGroupPorts() || config.services().ec2().mock()) {
return;
}
@@ -3084,6 +3141,7 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
if (config.services().ec2().mock()) {
inst.setState(InstanceState.running());
} else {
+ restoreInstanceFirewall(inst);
containerManager.start(inst);
}
instances.put(key(region, id), inst);
@@ -3297,9 +3355,22 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
List<GroupIdentifier> identifiers = new ArrayList<>();
for (String groupId : groupIds) {
SecurityGroup sg = getRequiredSecurityGroup(region, groupId);
+ if (!Objects.equals(inst.getVpcId(), sg.getVpcId())) {
+ throw new AwsException("InvalidGroup.NotFound", "Security group is not in the instance VPC", 400);
+ }
identifiers.add(new GroupIdentifier(sg.getGroupId(), sg.getGroupName()));
}
+ if (securityGroupEnforcementEnabled() && !config.services().ec2().mock()
+ && inst.getState() != null && "running".equals(inst.getState().getName())
+ && inst.getNetworkInterfaces() != null && !inst.getNetworkInterfaces().isEmpty()) {
+ List<SecurityGroup> current = securityGroups.scan(k -> k.startsWith(region + "::"));
+ Map<String, SecurityGroup> byId = current.stream().collect(Collectors.toMap(
+ SecurityGroup::getGroupId, Function.identity()));
+ containerManager.updateSecurityGroups(inst.getNetworkInterfaces().getFirst().getNetworkInterfaceId(),
+ new HashSet<>(groupIds), byId, policyPrefixLists(region, current));
+ }
+
inst.setSecurityGroups(new ArrayList<>(identifiers));
if (inst.getNetworkInterfaces() != null) {
inst.getNetworkInterfaces().forEach(eni -> eni.setGroups(new ArrayList<>(identifiers)));
@@ -4122,6 +4193,7 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
sg.setIpPermissionsEgress(next);
securityGroups.put(key(region, groupId), sg);
}
+ reconcileFirewallPolicies(region);
return rules;
}
@@ -4266,6 +4338,7 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
sg.setIpPermissionsEgress(next);
securityGroups.put(key(region, groupId), sg);
}
+ reconcileFirewallPolicies(region);
}
/**
@@ -4302,6 +4375,8 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
}
if (!rule.isEgress()) {
reconcilePublishedPortsForGroup(region, groupId);
+ } else {
+ reconcileFirewallPolicies(region);
}
return true;
}
@@ -7790,10 +7865,17 @@ public class Ec2Service implements ContainerTeardown, ResourceProvider {
if (securityGroupIds != null && !securityGroupIds.isEmpty()) {
for (String sgId : securityGroupIds) {
SecurityGroup sg = getRequiredSecurityGroup(region, sgId);
+ if (!subnet.getVpcId().equals(sg.getVpcId())) {
+ throw new AwsException("InvalidGroup.NotFound",
+ "Security group " + sgId + " does not belong to the subnet VPC", 400);
+ }
sgIdentifiers.add(new GroupIdentifier(sg.getGroupId(), sg.getGroupName()));
}
} else {
- SecurityGroup defaultSg = securityGroups.get(key(region, resolveDefaultSecurityGroupId(region))).orElse(null);
+ SecurityGroup defaultSg = securityGroups.scan(k -> k.startsWith(region + "::")).stream()
+ .filter(group -> subnet.getVpcId().equals(group.getVpcId())
+ && "default".equals(group.getGroupName()))
+ .findFirst().orElse(null);
if (defaultSg != null) {
sgIdentifiers.add(new GroupIdentifier(defaultSg.getGroupId(), defaultSg.getGroupName()));
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java
new file mode 100644
index 0000000000..3640121257
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallManager.java
@@ -0,0 +1,363 @@
+package io.github.hectorvent.floci.services.ec2;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.async.ResultCallback;
+import com.github.dockerjava.api.command.BuildImageResultCallback;
+import com.github.dockerjava.api.command.InspectContainerResponse;
+import com.github.dockerjava.api.exception.NotFoundException;
+import com.github.dockerjava.api.model.Container;
+import com.github.dockerjava.api.model.Frame;
+import com.github.dockerjava.api.model.Info;
+import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
+import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
+import io.github.hectorvent.floci.services.ec2.model.SecurityGroup;
+import jakarta.annotation.PostConstruct;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+import org.jboss.logging.Logger;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+/** Owns the protected namespace and the Floci-only nftables table for each managed ENI. */
+@ApplicationScoped
+public class SecurityGroupFirewallManager {
+
+ private static final Logger LOG = Logger.getLogger(SecurityGroupFirewallManager.class);
+ private final DockerClient dockerClient;
+ private final ContainerBuilder containerBuilder;
+ private final ContainerLifecycleManager lifecycleManager;
+ private final EmulatorConfig config;
+ private final Map<String, ProtectedEndpoint> endpoints = new HashMap<>();
+
+ @Inject
+ public SecurityGroupFirewallManager(DockerClient dockerClient, ContainerBuilder containerBuilder,
+ ContainerLifecycleManager lifecycleManager, EmulatorConfig config) {
+ this.dockerClient = dockerClient;
+ this.containerBuilder = containerBuilder;
+ this.lifecycleManager = lifecycleManager;
+ this.config = config;
+ }
+
+ public boolean enabled() {
+ return config.network().securityGroupEnforcement().enabled();
+ }
+
+ @PostConstruct
+ void quarantineSurvivingNamespaces() {
+ if (!enabled()) {
+ return;
+ }
+ try {
+ String namespace = config.docker().resourceNamespace().orElse("");
+ String owner = namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();
+ dockerClient.listContainersCmd().withShowAll(true)
+ .withLabelFilter(Map.of("floci.security-group-helper", "true"))
+ .exec().stream()
+ .filter(container -> owner.equals(container.getLabels().get("floci_owner_port")))
+ .forEach(container -> {
+ if ("running".equals(container.getState())) {
+ quarantine(container.getId());
+ } else {
+ stopNamespaceWorkloads(container.getId());
+ }
+ if ("ecs".equals(container.getLabels().get("io.floci.service"))) {
+ removeNamespaceWorkloads(container.getId());
+ lifecycleManager.removeIfExists(container.getId());
+ }
+ });
+ } catch (Exception e) {
+ LOG.warnv("Could not inspect surviving security-group namespaces: {0}", e.getMessage());
+ }
+ }
+
+ /** The helper starts before any workload process and owns all published ports. */
+ public Namespace createNamespace(String service, String resourceId, String accountId, String region,
+ Optional<String> dockerNetwork, Map<Integer, Integer> portBindings) {
+ if (!enabled()) {
+ throw new IllegalStateException("Security-group enforcement is disabled");
+ }
+ Info daemon = dockerClient.infoCmd().exec();
+ if (!"linux".equalsIgnoreCase(daemon.getOsType())) {
+ throw new IllegalStateException("Security-group enforcement requires a Linux Docker daemon");
+ }
+ if (daemon.getSecurityOptions() != null && daemon.getSecurityOptions().stream()
+ .anyMatch(option -> option.toLowerCase(Locale.ROOT).contains("rootless"))) {
+ throw new IllegalStateException("Security-group enforcement requires rootful Docker");
+ }
+ ensureHelperImage();
+ String name = ContainerStorageHelper.resourceName(config, "sg", null,
+ resourceId.replaceAll("[^a-zA-Z0-9_.-]", "-"));
+ String namespace = config.docker().resourceNamespace().orElse("");
+ String owner = namespace.isBlank() ? String.valueOf(config.port()) : namespace + "/" + config.port();
+ ContainerBuilder.Builder builder = containerBuilder.newContainer(config.network().securityGroupEnforcement().helperImage())
+ .withName(name)
+ .withDockerNetwork(dockerNetwork)
+ .withEntrypoint(List.of("sh", "-c"))
+ .withCmd(List.of("exec sleep 2147483647"))
+ .withLabels(ContainerStorageHelper.resourceIdentityLabels(service, resourceId, accountId, region))
+ .withLabels(Map.of("floci.security-group-helper", "true", "floci_owner_port", owner));
+ if (portBindings != null) {
+ portBindings.forEach(builder::withPortBinding);
+ }
+ ContainerSpec spec = builder.build();
+ String helperId = null;
+ try {
+ helperId = lifecycleManager.createAndStart(spec).containerId();
+ String ip = dockerClient.inspectContainerCmd(helperId).exec().getNetworkSettings()
+ .getNetworks().values().stream().map(n -> n.getIpAddress())
+ .filter(address -> address != null && !address.isBlank())
+ .findFirst().orElseThrow(() -> new IllegalStateException("Firewall helper has no Docker IP"));
+ apply(helperId, SecurityGroupNftCompiler.initialRuleset());
+ return new Namespace(helperId, ip);
+ } catch (Exception e) {
+ if (helperId != null) {
+ lifecycleManager.removeIfExists(helperId);
+ }
+ throw new IllegalStateException("Cannot prepare protected network namespace for " + resourceId, e);
+ }
+ }
+
+ private synchronized void ensureHelperImage() {
+ String configured = config.network().securityGroupEnforcement().helperImage();
+ if (!"floci/network-helper:local".equals(configured)) {
+ return;
+ }
+ String image = containerBuilder.resolveImage(configured);
+ try {
+ dockerClient.inspectImageCmd(image).exec();
+ return;
+ } catch (NotFoundException missing) {
+ // Build the versioned Floci recipe in the daemon used for workloads.
+ }
+ try (InputStream dockerfile = getClass().getResourceAsStream("/docker/network-helper.Dockerfile")) {
+ if (dockerfile == null) {
+ throw new IllegalStateException("Floci network helper Dockerfile is missing");
+ }
+ byte[] content = dockerfile.readAllBytes();
+ ByteArrayOutputStream archive = new ByteArrayOutputStream(content.length + 1024);
+ try (TarArchiveOutputStream tar = new TarArchiveOutputStream(archive)) {
+ TarArchiveEntry entry = new TarArchiveEntry("Dockerfile");
+ entry.setSize(content.length);
+ tar.putArchiveEntry(entry);
+ tar.write(content);
+ tar.closeArchiveEntry();
+ }
+ try (BuildImageResultCallback callback = new BuildImageResultCallback()) {
+ String built = dockerClient.buildImageCmd(new ByteArrayInputStream(archive.toByteArray()))
+ .withTags(Set.of(image)).exec(callback).awaitImageId();
+ if (built == null || built.isBlank()) {
+ throw new IllegalStateException("Docker did not build the Floci network helper image");
+ }
+ }
+ } catch (IOException e) {
+ throw new IllegalStateException("Cannot build the Floci network helper image", e);
+ }
+ }
+
+ /** Registers only after the default-deny table exists, before workload startup. */
+ public synchronized void register(SecurityGroupNftCompiler.Endpoint endpoint, String helperId,
+ Map<String, List<String>> prefixLists) {
+ if (endpoints.values().stream().map(ProtectedEndpoint::endpoint)
+ .anyMatch(existing -> !existing.eniId().equals(endpoint.eniId())
+ && existing.transportAddress().equals(endpoint.transportAddress()))) {
+ quarantine(helperId);
+ throw new IllegalStateException("Two managed ENIs share a Docker transport address");
+ }
+ ProtectedEndpoint previous = endpoints.put(endpoint.eniId(),
+ new ProtectedEndpoint(endpoint, helperId, Map.copyOf(prefixLists)));
+ try {
+ reconcileAll();
+ } catch (RuntimeException e) {
+ if (previous == null) {
+ endpoints.remove(endpoint.eniId());
+ } else {
+ endpoints.put(endpoint.eniId(), previous);
+ }
+ quarantine(helperId);
+ throw e;
+ }
+ }
+
+ public synchronized void unregister(String eniId) {
+ ProtectedEndpoint removed = endpoints.remove(eniId);
+ if (removed != null) {
+ lifecycleManager.removeIfExists(removed.helperId());
+ reconcileAll();
+ }
+ }
+
+ public synchronized void updateGroups(String eniId, Set<String> groupIds,
+ Map<String, SecurityGroup> groups,
+ Map<String, List<String>> prefixLists) {
+ ProtectedEndpoint current = endpoints.get(eniId);
+ if (current == null) {
+ return;
+ }
+ SecurityGroupNftCompiler.Endpoint identity = current.endpoint();
+ List<SecurityGroup> attached = groupIds.stream().map(groups::get).toList();
+ if (attached.isEmpty() || attached.stream().anyMatch(group -> group == null)) {
+ quarantine(current.helperId());
+ throw new IllegalArgumentException("Protected endpoint needs valid security groups");
+ }
+ SecurityGroupNftCompiler.Endpoint updated = new SecurityGroupNftCompiler.Endpoint(
+ identity.accountId(), identity.region(), identity.vpcId(), identity.eniId(),
+ identity.logicalAddress(), identity.transportAddress(), Set.copyOf(groupIds), attached);
+ endpoints.put(eniId, new ProtectedEndpoint(updated, current.helperId(), Map.copyOf(prefixLists)));
+ reconcileAll();
+ }
+
+ public synchronized void reconcileAll() {
+ for (Map.Entry<String, ProtectedEndpoint> entry : new ArrayList<>(endpoints.entrySet())) {
+ boolean running;
+ try {
+ running = Boolean.TRUE.equals(dockerClient.inspectContainerCmd(entry.getValue().helperId())
+ .exec().getState().getRunning());
+ } catch (NotFoundException e) {
+ running = false;
+ }
+ if (!running) {
+ quarantine(entry.getValue().helperId());
+ endpoints.remove(entry.getKey());
+ }
+ }
+ List<SecurityGroupNftCompiler.Endpoint> peers = endpoints.values().stream()
+ .map(ProtectedEndpoint::endpoint).toList();
+ for (ProtectedEndpoint protectedEndpoint : new ArrayList<>(endpoints.values())) {
+ try {
+ String rules = SecurityGroupNftCompiler.compile(protectedEndpoint.endpoint(), peers,
+ protectedEndpoint.prefixLists());
+ apply(protectedEndpoint.helperId(), rules);
+ } catch (RuntimeException e) {
+ endpoints.values().forEach(endpoint -> quarantine(endpoint.helperId()));
+ throw e;
+ }
+ }
+ }
+
+ /** Refreshes every affected EC2 instance and ECS task after a control-plane rule change. */
+ public synchronized void refreshPolicies(String region, Map<String, SecurityGroup> groups,
+ Map<String, List<String>> prefixLists) {
+ for (Map.Entry<String, ProtectedEndpoint> entry : new ArrayList<>(endpoints.entrySet())) {
+ ProtectedEndpoint current = entry.getValue();
+ SecurityGroupNftCompiler.Endpoint identity = current.endpoint();
+ if (!region.equals(identity.region())) {
+ continue;
+ }
+ List<SecurityGroup> attached = identity.groupIds().stream().map(groups::get).toList();
+ if (attached.stream().anyMatch(group -> group == null)) {
+ endpoints.values().forEach(endpoint -> quarantine(endpoint.helperId()));
+ throw new IllegalStateException("A protected endpoint's security group disappeared");
+ }
+ SecurityGroupNftCompiler.Endpoint updated = new SecurityGroupNftCompiler.Endpoint(
+ identity.accountId(), identity.region(), identity.vpcId(), identity.eniId(),
+ identity.logicalAddress(), identity.transportAddress(), identity.groupIds(), attached);
+ endpoints.put(entry.getKey(), new ProtectedEndpoint(updated, current.helperId(),
+ Map.copyOf(prefixLists)));
+ }
+ reconcileAll();
+ }
+
+ private void quarantine(String helperId) {
+ try {
+ apply(helperId, "flush chain inet floci_sg ingress\nflush chain inet floci_sg egress\n");
+ } catch (Exception e) {
+ LOG.errorv(e, "Failed to quarantine security-group helper {0}", helperId);
+ try {
+ stopNamespaceWorkloads(helperId);
+ dockerClient.stopContainerCmd(helperId).withTimeout(0).exec();
+ } catch (Exception stopFailure) {
+ LOG.errorv(stopFailure, "Failed to stop workloads after quarantine failure for {0}", helperId);
+ }
+ }
+ }
+
+ private void stopNamespaceWorkloads(String helperId) {
+ for (Container container : dockerClient.listContainersCmd().exec()) {
+ InspectContainerResponse inspect = dockerClient.inspectContainerCmd(container.getId()).exec();
+ if (inspect.getHostConfig() != null && ("container:" + helperId)
+ .equals(inspect.getHostConfig().getNetworkMode())) {
+ dockerClient.stopContainerCmd(container.getId()).withTimeout(0).exec();
+ }
+ }
+ }
+
+ private void removeNamespaceWorkloads(String helperId) {
+ for (Container container : dockerClient.listContainersCmd().withShowAll(true).exec()) {
+ InspectContainerResponse inspect = dockerClient.inspectContainerCmd(container.getId()).exec();
+ if (inspect.getHostConfig() != null && ("container:" + helperId)
+ .equals(inspect.getHostConfig().getNetworkMode())) {
+ lifecycleManager.removeIfExists(container.getId());
+ }
+ }
+ }
+
+ private void apply(String helperId, String script) {
+ for (int attempt = 0; attempt < 120; attempt++) {
+ try {
+ byte[] content = script.getBytes(StandardCharsets.UTF_8);
+ ByteArrayOutputStream archive = new ByteArrayOutputStream(content.length + 1024);
+ try (TarArchiveOutputStream tar = new TarArchiveOutputStream(archive)) {
+ TarArchiveEntry entry = new TarArchiveEntry("floci-sg.nft");
+ entry.setSize(content.length);
+ tar.putArchiveEntry(entry);
+ tar.write(content);
+ tar.closeArchiveEntry();
+ }
+ dockerClient.copyArchiveToContainerCmd(helperId)
+ .withRemotePath("/tmp")
+ .withTarInputStream(new ByteArrayInputStream(archive.toByteArray()))
+ .exec();
+ String execId = dockerClient.execCreateCmd(helperId)
+ .withAttachStdout(true).withAttachStderr(true)
+ .withCmd("nft", "-f", "/tmp/floci-sg.nft").exec().getId();
+ StringBuilder output = new StringBuilder();
+ try (ResultCallback.Adapter<Frame> callback = new ResultCallback.Adapter<Frame>() {
+ @Override public void onNext(Frame frame) {
+ output.append(new String(frame.getPayload(), StandardCharsets.UTF_8));
+ }
+ }) {
+ boolean completed = dockerClient.execStartCmd(execId)
+ .exec(callback).awaitCompletion(15, TimeUnit.SECONDS);
+ if (!completed) {
+ throw new IllegalStateException("nft command timed out");
+ }
+ }
+ Long exit = dockerClient.inspectExecCmd(execId).exec().getExitCodeLong();
+ if (exit != null && exit == 0) {
+ return;
+ }
+ if (output.toString().contains("not found") && attempt < 119) {
+ Thread.sleep(500);
+ continue;
+ }
+ throw new IllegalStateException("nft rejected security-group policy: " + output);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IllegalStateException("Interrupted installing security-group policy", e);
+ } catch (Exception e) {
+ throw new IllegalStateException("Failed to install security-group policy", e);
+ }
+ }
+ }
+
+ public record Namespace(String helperId, String transportAddress) {}
+ private record ProtectedEndpoint(SecurityGroupNftCompiler.Endpoint endpoint, String helperId,
+ Map<String, List<String>> prefixLists) {}
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupNftCompiler.java b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupNftCompiler.java
new file mode 100644
index 0000000000..5c30d26fb8
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupNftCompiler.java
@@ -0,0 +1,189 @@
+package io.github.hectorvent.floci.services.ec2;
+
+import io.github.hectorvent.floci.services.ec2.model.IpPermission;
+import io.github.hectorvent.floci.services.ec2.model.SecurityGroup;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.Collection;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+
+/** Builds only rules for an endpoint's private nftables table. Docker's tables are untouched. */
+public final class SecurityGroupNftCompiler {
+
+ private SecurityGroupNftCompiler() {}
+
+ public record Endpoint(String accountId, String region, String vpcId, String eniId,
+ String logicalAddress, String transportAddress, Set<String> groupIds,
+ List<SecurityGroup> groups) {
+ SecurityGroupPolicy.Peer peer() {
+ return new SecurityGroupPolicy.Peer(logicalAddress, accountId, vpcId, groupIds);
+ }
+ }
+
+ public static String initialRuleset() {
+ return "add table inet floci_sg\n"
+ + "add chain inet floci_sg ingress { type filter hook input priority -10; policy drop; }\n"
+ + "add chain inet floci_sg egress { type filter hook output priority -10; policy drop; }\n";
+ }
+
+ public static String compile(Endpoint target, Collection<Endpoint> peers,
+ Map<String, List<String>> prefixLists) {
+ if (target == null || target.groups() == null || target.groups().isEmpty()) {
+ throw new IllegalArgumentException("Protected endpoint needs at least one security group");
+ }
+ StringBuilder rules = new StringBuilder("flush chain inet floci_sg ingress\n")
+ .append("flush chain inet floci_sg egress\n")
+ .append("add rule inet floci_sg ingress iifname lo accept\n")
+ .append("add rule inet floci_sg egress oifname lo accept\n")
+ .append("add rule inet floci_sg ingress ct state established,related accept\n")
+ .append("add rule inet floci_sg egress ct state established,related accept\n");
+
+ // Docker's embedded DNS is loopback and covered above. DHCP and the
+ // link-local metadata endpoint are the other local infrastructure exceptions.
+ rules.append("add rule inet floci_sg egress udp sport 68 udp dport 67 accept\n")
+ .append("add rule inet floci_sg ingress udp sport 67 udp dport 68 accept\n")
+ .append("add rule inet floci_sg egress ip daddr 169.254.169.254 accept\n")
+ .append("add rule inet floci_sg ingress ip saddr 169.254.169.254 accept\n");
+
+ List<Endpoint> managed = peers == null ? List.of() : peers.stream()
+ .filter(peer -> !target.eniId().equals(peer.eniId()))
+ .filter(peer -> peer.transportAddress() != null && literal(peer.transportAddress()))
+ .toList();
+
+ for (boolean egress : new boolean[]{false, true}) {
+ String chain = egress ? "egress" : "ingress";
+ String addressField = egress ? "daddr" : "saddr";
+ for (SecurityGroup group : target.groups()) {
+ List<IpPermission> permissions = egress
+ ? group.getIpPermissionsEgress() : group.getIpPermissions();
+ if (permissions == null) {
+ continue;
+ }
+ for (IpPermission permission : permissions) {
+ String protocol = protocolExpression(permission);
+ if (protocol == null) {
+ continue;
+ }
+ for (Endpoint peer : managed) {
+ if (target.accountId().equals(peer.accountId())
+ && target.region().equals(peer.region())
+ && target.vpcId().equals(peer.vpcId())
+ && SecurityGroupPolicy.matchesPeer(group, permission, peer.peer(), prefixLists)) {
+ appendRule(rules, chain, addressField, peer.transportAddress(), protocol);
+ }
+ }
+ }
+ }
+ // A managed peer must never fall through to a broad external CIDR rule
+ // evaluated against its Docker bridge IP instead of its logical ENI address.
+ for (Endpoint peer : managed) {
+ appendRule(rules, chain, addressField, peer.transportAddress(), "drop");
+ }
+ for (SecurityGroup group : target.groups()) {
+ List<IpPermission> permissions = egress
+ ? group.getIpPermissionsEgress() : group.getIpPermissions();
+ if (permissions == null) {
+ continue;
+ }
+ for (IpPermission permission : permissions) {
+ String protocol = protocolExpression(permission);
+ if (protocol == null) {
+ continue;
+ }
+ Set<String> cidrs = new LinkedHashSet<>();
+ if (permission.getIpRanges() != null) {
+ permission.getIpRanges().forEach(range -> cidrs.add(range.getCidrIp()));
+ }
+ if (permission.getIpv6Ranges() != null) {
+ permission.getIpv6Ranges().forEach(range -> cidrs.add(range.getCidrIpv6()));
+ }
+ if (permission.getPrefixListIds() != null && prefixLists != null) {
+ permission.getPrefixListIds().forEach(ref ->
+ cidrs.addAll(prefixLists.getOrDefault(ref.getPrefixListId(), List.of())));
+ }
+ for (String cidr : cidrs) {
+ if (SecurityGroupPolicy.validCidr(cidr)) {
+ appendRule(rules, chain, addressField, cidr, protocol);
+ }
+ }
+ }
+ }
+ }
+ return rules.toString();
+ }
+
+ private static void appendRule(StringBuilder rules, String chain, String addressField,
+ String address, String expression) {
+ rules.append("add rule inet floci_sg ").append(chain).append(' ')
+ .append(address.indexOf(':') >= 0 ? "ip6 " : "ip ")
+ .append(addressField).append(' ').append(address).append(' ')
+ .append(expression).append('\n');
+ }
+
+ private static String protocolExpression(IpPermission permission) {
+ String protocol = permission.getIpProtocol();
+ if (protocol == null) {
+ return null;
+ }
+ if ("-1".equals(protocol)) {
+ return "accept";
+ }
+ String normalized = switch (protocol.toLowerCase(Locale.ROOT)) {
+ case "6" -> "tcp";
+ case "17" -> "udp";
+ case "1" -> "icmp";
+ case "58" -> "icmpv6";
+ default -> protocol.toLowerCase(Locale.ROOT);
+ };
+ if ("tcp".equals(normalized) || "udp".equals(normalized)) {
+ Integer from = permission.getFromPort();
+ Integer to = permission.getToPort();
+ if (from == null || to == null || from < 0 || to > 65535 || to < from) {
+ return null;
+ }
+ return normalized + " dport " + (from.equals(to) ? from : from + "-" + to) + " accept";
+ }
+ if ("icmp".equals(normalized) || "icmpv6".equals(normalized)) {
+ // nftables header fields are separate expressions, so type and code each repeat
+ // the protocol keyword: "icmp type 8 icmp code 0", never "icmp type 8 code 0".
+ StringBuilder expression = new StringBuilder();
+ Integer type = permission.getFromPort();
+ Integer code = permission.getToPort();
+ if (type != null && type >= 0) {
+ expression.append(normalized).append(" type ").append(type).append(' ');
+ }
+ if (code != null && code >= 0) {
+ expression.append(normalized).append(" code ").append(code).append(' ');
+ }
+ if (expression.isEmpty()) {
+ expression.append("meta l4proto ").append(normalized).append(' ');
+ }
+ return expression.append("accept").toString();
+ }
+ try {
+ int number = Integer.parseInt(normalized);
+ return number >= 0 && number <= 255 ? "meta l4proto " + number + " accept" : null;
+ } catch (NumberFormatException e) {
+ return null;
+ }
+ }
+
+ private static boolean literal(String address) {
+ try {
+ if (address == null || address.contains("%")
+ || !(address.contains(":") ? address.matches("[0-9a-fA-F:.]+")
+ : address.matches("(?:[0-9]{1,3}\\.){3}[0-9]{1,3}"))) {
+ return false;
+ }
+ InetAddress.getByName(address);
+ return true;
+ } catch (UnknownHostException e) {
+ return false;
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupPolicy.java b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupPolicy.java
new file mode 100644
index 0000000000..7af14b313d
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/SecurityGroupPolicy.java
@@ -0,0 +1,149 @@
+package io.github.hectorvent.floci.services.ec2;
+
+import io.github.hectorvent.floci.services.ec2.model.IpPermission;
+import io.github.hectorvent.floci.services.ec2.model.PrefixListId;
+import io.github.hectorvent.floci.services.ec2.model.SecurityGroup;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+/** Evaluates the live permissions on security groups, rather than the separately stored rule records. */
+public final class SecurityGroupPolicy {
+
+ private SecurityGroupPolicy() {}
+
+ public record Peer(String logicalAddress, String accountId, String vpcId, Set<String> groupIds) {}
+
+ public static boolean allows(List<SecurityGroup> groups, boolean egress, String protocol,
+ int port, int icmpCode, Peer peer,
+ Map<String, List<String>> prefixLists) {
+ if (groups == null || peer == null || protocol == null) {
+ return false;
+ }
+ for (SecurityGroup group : groups) {
+ List<IpPermission> permissions = egress
+ ? group.getIpPermissionsEgress() : group.getIpPermissions();
+ if (permissions == null) {
+ continue;
+ }
+ for (IpPermission permission : permissions) {
+ if (matchesProtocol(permission, protocol, port, icmpCode)
+ && matchesPeer(group, permission, peer, prefixLists)) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ private static boolean matchesProtocol(IpPermission permission, String protocol, int port, int icmpCode) {
+ String ruleProtocol = permission.getIpProtocol();
+ if ("-1".equals(ruleProtocol)) {
+ return true;
+ }
+ if (!Objects.equals(normalizeProtocol(ruleProtocol), normalizeProtocol(protocol))) {
+ return false;
+ }
+ if ("tcp".equals(normalizeProtocol(protocol)) || "udp".equals(normalizeProtocol(protocol))) {
+ Integer from = permission.getFromPort();
+ Integer to = permission.getToPort();
+ return from != null && to != null && port >= from && port <= to;
+ }
+ if ("icmp".equals(normalizeProtocol(protocol)) || "icmpv6".equals(normalizeProtocol(protocol))) {
+ Integer type = permission.getFromPort();
+ Integer code = permission.getToPort();
+ return (type == null || type == -1 || type == port)
+ && (code == null || code == -1 || code == icmpCode);
+ }
+ return true;
+ }
+
+ private static String normalizeProtocol(String protocol) {
+ if (protocol == null) {
+ return null;
+ }
+ return switch (protocol.toLowerCase(Locale.ROOT)) {
+ case "6" -> "tcp";
+ case "17" -> "udp";
+ case "1" -> "icmp";
+ case "58" -> "icmpv6";
+ default -> protocol.toLowerCase(Locale.ROOT);
+ };
+ }
+
+ static boolean matchesPeer(SecurityGroup group, IpPermission permission, Peer peer,
+ Map<String, List<String>> prefixLists) {
+ if (permission.getIpRanges() != null && permission.getIpRanges().stream()
+ .anyMatch(range -> inCidr(peer.logicalAddress(), range.getCidrIp()))) {
+ return true;
+ }
+ if (permission.getIpv6Ranges() != null && permission.getIpv6Ranges().stream()
+ .anyMatch(range -> inCidr(peer.logicalAddress(), range.getCidrIpv6()))) {
+ return true;
+ }
+ if (permission.getPrefixListIds() != null && prefixLists != null) {
+ for (PrefixListId reference : permission.getPrefixListIds()) {
+ if (prefixLists.getOrDefault(reference.getPrefixListId(), List.of()).stream()
+ .anyMatch(cidr -> inCidr(peer.logicalAddress(), cidr))) {
+ return true;
+ }
+ }
+ }
+ if (permission.getUserIdGroupPairs() != null && peer.groupIds() != null
+ && Objects.equals(group.getVpcId(), peer.vpcId())) {
+ return permission.getUserIdGroupPairs().stream().anyMatch(reference ->
+ peer.groupIds().contains(reference.getGroupId())
+ && Objects.equals(reference.getUserId() == null
+ ? group.getOwnerId() : reference.getUserId(), peer.accountId()));
+ }
+ return false;
+ }
+
+ static boolean inCidr(String address, String cidr) {
+ if (address == null || cidr == null || address.contains("%") || cidr.contains("%")) {
+ return false;
+ }
+ String[] parts = cidr.split("/", -1);
+ if (parts.length != 2 || !literalAddress(address) || !literalAddress(parts[0])) {
+ return false;
+ }
+ try {
+ byte[] value = InetAddress.getByName(address).getAddress();
+ byte[] network = InetAddress.getByName(parts[0]).getAddress();
+ int bits = Integer.parseInt(parts[1]);
+ if (value.length != network.length || bits < 0 || bits > value.length * 8) {
+ return false;
+ }
+ int whole = bits / 8;
+ for (int i = 0; i < whole; i++) {
+ if (value[i] != network[i]) {
+ return false;
+ }
+ }
+ int remainder = bits % 8;
+ int mask = (0xff << (8 - remainder)) & 0xff;
+ return remainder == 0 || ((value[whole] & mask) == (network[whole] & mask));
+ } catch (UnknownHostException | NumberFormatException e) {
+ return false;
+ }
+ }
+
+ static boolean validCidr(String cidr) {
+ if (cidr == null) {
+ return false;
+ }
+ int slash = cidr.indexOf('/');
+ return slash > 0 && inCidr(cidr.substring(0, slash), cidr);
+ }
+
+ private static boolean literalAddress(String address) {
+ return address.indexOf(':') >= 0
+ ? address.matches("[0-9a-fA-F:.]+")
+ : address.matches("(?:[0-9]{1,3}\\.){3}[0-9]{1,3}");
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/model/Instance.java b/src/main/java/io/github/hectorvent/floci/services/ec2/model/Instance.java
index ac8f94427a..e0efab17e3 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ec2/model/Instance.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/model/Instance.java
@@ -22,6 +22,7 @@ public class Instance {
private String subnetId;
private String vpcId;
private String privateIpAddress;
+ private String logicalPrivateIpAddress;
private String publicIpAddress;
private String privateDnsName;
private String publicDnsName;
@@ -118,6 +119,10 @@ public class Instance {
public String getPrivateIpAddress() { return privateIpAddress; }
public void setPrivateIpAddress(String privateIpAddress) { this.privateIpAddress = privateIpAddress; }
+ public String getLogicalPrivateIpAddress() { return logicalPrivateIpAddress; }
+ public void setLogicalPrivateIpAddress(String logicalPrivateIpAddress) {
+ this.logicalPrivateIpAddress = logicalPrivateIpAddress;
+ }
public String getPublicIpAddress() { return publicIpAddress; }
public void setPublicIpAddress(String publicIpAddress) { this.publicIpAddress = publicIpAddress; }
diff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/portforward/Ec2PortForwardManager.java b/src/main/java/io/github/hectorvent/floci/services/ec2/portforward/Ec2PortForwardManager.java
index cb5b8db465..2716993df1 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ec2/portforward/Ec2PortForwardManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/portforward/Ec2PortForwardManager.java
@@ -132,7 +132,9 @@ public class Ec2PortForwardManager {
}
boolean enabled() {
- return config.services().ec2().publishSecurityGroupPorts() && !config.services().ec2().mock();
+ return config.services().ec2().publishSecurityGroupPorts() && !config.services().ec2().mock()
+ && (config.network() == null || config.network().securityGroupEnforcement() == null
+ || !config.network().securityGroupEnforcement().enabled());
}
/** Sets the callback used to persist an instance after its forwards change. */
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandler.java b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandler.java
index f462a61780..de694cbc68 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandler.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/EcsJsonHandler.java
@@ -1163,6 +1163,23 @@ public class EcsJsonHandler {
if (t.getStartedAt() != null) { n.put("startedAt", t.getStartedAt().toEpochMilli() / 1000.0); }
if (t.getStoppedAt() != null) { n.put("stoppedAt", t.getStoppedAt().toEpochMilli() / 1000.0); }
if (t.getStoppedReason() != null) { n.put("stoppedReason", t.getStoppedReason()); }
+ if (t.getNetworkInterfaceId() != null) {
+ ObjectNode attachment = objectMapper.createObjectNode();
+ attachment.put("id", "eni-attach-" + t.getNetworkInterfaceId());
+ attachment.put("type", "ElasticNetworkInterface");
+ attachment.put("status", "ATTACHED");
+ ArrayNode details = objectMapper.createArrayNode();
+ if (t.getNetworkConfiguration() != null
+ && t.getNetworkConfiguration().getAwsvpcConfiguration() != null
+ && !t.getNetworkConfiguration().getAwsvpcConfiguration().getSubnets().isEmpty()) {
+ details.addObject().put("name", "subnetId").put("value",
+ t.getNetworkConfiguration().getAwsvpcConfiguration().getSubnets().getFirst());
+ }
+ details.addObject().put("name", "networkInterfaceId").put("value", t.getNetworkInterfaceId());
+ details.addObject().put("name", "privateIPv4Address").put("value", t.getPrivateIpAddress());
+ attachment.set("details", details);
+ n.putArray("attachments").add(attachment);
+ }
ArrayNode containers = objectMapper.createArrayNode();
if (t.getContainers() != null) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java
index e3505c2aa1..63b0e292b0 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsContainerManager.java
@@ -13,6 +13,13 @@ import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
import io.github.hectorvent.floci.core.common.docker.LaunchedContainerAwsEnv;
import io.github.hectorvent.floci.services.ecr.registry.EcrRegistryManager;
+import io.github.hectorvent.floci.services.ec2.Ec2Service;
+import io.github.hectorvent.floci.services.ec2.SecurityGroupFirewallManager;
+import io.github.hectorvent.floci.services.ec2.SecurityGroupNftCompiler;
+import io.github.hectorvent.floci.services.ec2.model.NetworkInterface;
+import io.github.hectorvent.floci.services.ec2.model.PrefixListEntry;
+import io.github.hectorvent.floci.services.ec2.model.SecurityGroup;
+import io.github.hectorvent.floci.services.ecs.model.AwsVpcConfiguration;
import io.github.hectorvent.floci.services.ecs.model.Container;
import io.github.hectorvent.floci.services.ecs.model.ContainerDefinition;
import io.github.hectorvent.floci.services.ecs.model.ContainerOverride;
@@ -47,6 +54,7 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.stream.Stream;
/**
* Manages Docker container lifecycle for ECS tasks.
@@ -68,8 +76,29 @@ public class EcsContainerManager {
private final SecretsManagerService secretsManagerService;
private final EcrRegistryManager ecrRegistryManager;
private final HostVolumePolicy hostVolumePolicy;
+ private Ec2Service ec2Service;
+ private SecurityGroupFirewallManager firewallManager;
@Inject
+ public EcsContainerManager(ContainerBuilder containerBuilder,
+ ContainerLifecycleManager lifecycleManager,
+ ContainerLogStreamer logStreamer,
+ ContainerDetector containerDetector,
+ EmulatorConfig config,
+ RegionResolver regionResolver,
+ LaunchedContainerAwsEnv awsEnv,
+ SsmService ssmService,
+ SecretsManagerService secretsManagerService,
+ EcrRegistryManager ecrRegistryManager,
+ HostVolumePolicy hostVolumePolicy,
+ Ec2Service ec2Service,
+ SecurityGroupFirewallManager firewallManager) {
+ this(containerBuilder, lifecycleManager, logStreamer, containerDetector, config, regionResolver,
+ awsEnv, ssmService, secretsManagerService, ecrRegistryManager, hostVolumePolicy);
+ this.ec2Service = ec2Service;
+ this.firewallManager = firewallManager;
+ }
+
public EcsContainerManager(ContainerBuilder containerBuilder,
ContainerLifecycleManager lifecycleManager,
ContainerLogStreamer logStreamer,
@@ -133,6 +162,9 @@ public class EcsContainerManager {
imagesByContainer.put(def, ecrRegistryManager.rewriteImageUri(def.getImage()));
}
+ PreparedNetwork protectedNetwork = prepareNetwork(task, taskDef, region, taskId);
+
+ try {
for (ContainerDefinition def : taskDef.getContainerDefinitions()) {
String containerName = ContainerStorageHelper.dockerName(config, "floci-ecs-" + taskId + "-" + def.getName());
@@ -154,6 +186,10 @@ public class EcsContainerManager {
.withLogRotation()
.withLabels(ContainerStorageHelper.resourceIdentityLabels(
"ecs", taskId, regionResolver.getAccountId(), region));
+ if (protectedNetwork != null) {
+ specBuilder.withNetworkMode("container:" + protectedNetwork.namespace().helperId());
+ specBuilder.withLabels(Map.of("floci.security-group-workload", "true"));
+ }
// Add memory limit if specified
if (def.getMemory() != null) {
@@ -168,7 +204,7 @@ public class EcsContainerManager {
// local Docker host (#1778) — awsvpc mappings always get a dynamic
// host port in native mode, or expose-only in Docker mode where ECS
// consumers reach containers via the docker network IP.
- if (def.getPortMappings() != null) {
+ if (protectedNetwork == null && def.getPortMappings() != null) {
boolean awsvpc = taskDef.getNetworkMode() == NetworkMode.awsvpc;
boolean publishToHost = !containerDetector.isRunningInContainer();
for (PortMapping pm : def.getPortMappings()) {
@@ -236,7 +272,8 @@ public class EcsContainerManager {
LOG.infov("Created ECS container {0} for task {1} container {2}", dockerId, taskId, def.getName());
// Resolve network bindings for ECS-specific model
- List<NetworkBinding> networkBindings = resolveNetworkBindings(dockerId, def);
+ List<NetworkBinding> networkBindings = resolveNetworkBindings(
+ protectedNetwork == null ? dockerId : protectedNetwork.namespace().helperId(), def);
// Build ECS container model
Container container = buildContainer(task.getTaskArn(), def, dockerId, networkBindings, region);
@@ -254,15 +291,83 @@ public class EcsContainerManager {
logStreamsByContainerId.put(dockerId, logHandle);
}
}
+ } catch (Exception e) {
+ for (String dockerId : containerIds.values()) {
+ lifecycleManager.stopAndRemove(dockerId, null);
+ }
+ if (protectedNetwork != null) {
+ firewallManager.unregister(protectedNetwork.eni().getNetworkInterfaceId());
+ ec2Service.deleteNetworkInterface(region, protectedNetwork.eni().getNetworkInterfaceId());
+ }
+ throw e;
+ }
task.setContainers(runtimeContainers);
task.setLastStatus(TaskStatus.RUNNING.name());
task.setDesiredStatus(TaskStatus.RUNNING.name());
task.setStartedAt(Instant.now());
- return new EcsTaskHandle(task.getTaskArn(), containerIds, logStreamsByContainerId);
+ return new EcsTaskHandle(task.getTaskArn(), containerIds, logStreamsByContainerId,
+ protectedNetwork == null ? null : protectedNetwork.eni().getNetworkInterfaceId(), region);
}
+ private PreparedNetwork prepareNetwork(EcsTask task, TaskDefinition definition, String region, String taskId) {
+ if (definition.getNetworkMode() != NetworkMode.awsvpc
+ || firewallManager == null || !firewallManager.enabled()) {
+ return null;
+ }
+ AwsVpcConfiguration awsvpc = task.getNetworkConfiguration() == null ? null
+ : task.getNetworkConfiguration().getAwsvpcConfiguration();
+ if (awsvpc == null || awsvpc.getSubnets() == null || awsvpc.getSubnets().isEmpty()) {
+ throw new AwsException("ClientException", "awsvpc tasks require a subnet", 400);
+ }
+ NetworkInterface eni = ec2Service.createNetworkInterface(region, awsvpc.getSubnets().getFirst(),
+ "ECS task " + task.getTaskArn(), null, List.of(), awsvpc.getSecurityGroups(), List.of());
+ String eniId = eni.getNetworkInterfaceId();
+ SecurityGroupFirewallManager.Namespace namespace = null;
+ try {
+ Map<Integer, Integer> bindings = new LinkedHashMap<>();
+ if (!containerDetector.isRunningInContainer()) {
+ for (ContainerDefinition container : definition.getContainerDefinitions()) {
+ if (container.getPortMappings() != null) {
+ container.getPortMappings().forEach(port -> bindings.put(port.containerPort(), 0));
+ }
+ }
+ }
+ namespace = firewallManager.createNamespace("ecs", taskId, regionResolver.getAccountId(),
+ region, config.services().ecs().dockerNetwork(), bindings);
+ List<String> groupIds = eni.getGroups().stream().map(g -> g.getGroupId()).toList();
+ List<SecurityGroup> groups = ec2Service.describeSecurityGroups(region, groupIds, List.of(), Map.of());
+ if (groups.size() != groupIds.size()) {
+ throw new IllegalStateException("An ECS task security group could not be resolved");
+ }
+ Map<String, List<String>> prefixLists = new LinkedHashMap<>();
+ for (SecurityGroup group : groups) {
+ Stream.concat(group.getIpPermissions().stream(),
+ group.getIpPermissionsEgress().stream())
+ .flatMap(permission -> permission.getPrefixListIds().stream())
+ .map(reference -> reference.getPrefixListId())
+ .distinct()
+ .forEach(id -> prefixLists.put(id, ec2Service.getManagedPrefixListEntries(region, id, null)
+ .stream().map(PrefixListEntry::getCidr).toList()));
+ }
+ firewallManager.register(new SecurityGroupNftCompiler.Endpoint(regionResolver.getAccountId(),
+ region, eni.getVpcId(), eniId, eni.getPrivateIpAddress(),
+ namespace.transportAddress(), Set.copyOf(groupIds), groups), namespace.helperId(), prefixLists);
+ task.setNetworkInterfaceId(eniId);
+ task.setPrivateIpAddress(eni.getPrivateIpAddress());
+ return new PreparedNetwork(eni, namespace);
+ } catch (Exception e) {
+ if (namespace != null) {
+ lifecycleManager.removeIfExists(namespace.helperId());
+ }
+ ec2Service.deleteNetworkInterface(region, eniId);
+ throw e;
+ }
+ }
+
+ private record PreparedNetwork(NetworkInterface eni, SecurityGroupFirewallManager.Namespace namespace) {}
+
/**
* Stops and removes all Docker containers for a task.
*/
@@ -281,6 +386,7 @@ public class EcsContainerManager {
for (String dockerId : handle.getContainerIds().values()) {
lifecycleManager.stopAndRemove(dockerId, null);
}
+ cleanupProtectedNetwork(handle);
new ArrayList<>(handle.getLogStreamsByContainerId().keySet())
.forEach(dockerId -> finalizeLogStream(handle, dockerId));
}
@@ -326,9 +432,17 @@ public class EcsContainerManager {
// A force removal terminates Docker's follow-log transport even when the preceding stop failed.
// Preserve handles for any container that still may be running after both operations failed.
terminatedContainerIds.forEach(dockerId -> finalizeLogStream(handle, dockerId));
+ cleanupProtectedNetwork(handle);
return exitCodes;
}
+ private void cleanupProtectedNetwork(EcsTaskHandle handle) {
+ if (firewallManager != null && handle.getNetworkInterfaceId() != null) {
+ firewallManager.unregister(handle.getNetworkInterfaceId());
+ ec2Service.deleteNetworkInterface(handle.getRegion(), handle.getNetworkInterfaceId());
+ }
+ }
+
private void finalizeLogStream(EcsTaskHandle handle, String dockerId) {
Closeable logStream = handle.removeLogStream(dockerId);
if (logStream != null) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java
index 3f14cd33e4..2447408b23 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/container/EcsTaskHandle.java
@@ -13,17 +13,29 @@ public class EcsTaskHandle {
private final String taskArn;
private final Map<String, String> containerIds; // containerName → dockerId
private final Map<String, Closeable> logStreamsByContainerId;
+ private final String networkInterfaceId;
+ private final String region;
public EcsTaskHandle(String taskArn, Map<String, String> containerIds,
Map<String, Closeable> logStreamsByContainerId) {
+ this(taskArn, containerIds, logStreamsByContainerId, null, null);
+ }
+
+ public EcsTaskHandle(String taskArn, Map<String, String> containerIds,
+ Map<String, Closeable> logStreamsByContainerId,
+ String networkInterfaceId, String region) {
this.taskArn = taskArn;
this.containerIds = new LinkedHashMap<>(containerIds);
this.logStreamsByContainerId = new LinkedHashMap<>(logStreamsByContainerId);
+ this.networkInterfaceId = networkInterfaceId;
+ this.region = region;
}
public String getTaskArn() { return taskArn; }
public Map<String, String> getContainerIds() { return containerIds; }
public Map<String, Closeable> getLogStreamsByContainerId() { return logStreamsByContainerId; }
+ public String getNetworkInterfaceId() { return networkInterfaceId; }
+ public String getRegion() { return region; }
/** Removes and returns the log stream that no longer needs task-level ownership. */
public Closeable removeLogStream(String containerId) {
diff --git a/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTask.java b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTask.java
index b38c9213dc..ffdd541925 100644
--- a/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTask.java
+++ b/src/main/java/io/github/hectorvent/floci/services/ecs/model/EcsTask.java
@@ -39,6 +39,8 @@ public class EcsTask {
private Instant protectedUntil;
private Map<String, String> tags = new HashMap<>();
private NetworkConfiguration networkConfiguration;
+ private String networkInterfaceId;
+ private String privateIpAddress;
public EcsTask() {
}
@@ -72,6 +74,8 @@ public class EcsTask {
this.protectedUntil = other.protectedUntil;
this.tags = other.tags;
this.networkConfiguration = other.networkConfiguration;
+ this.networkInterfaceId = other.networkInterfaceId;
+ this.privateIpAddress = other.privateIpAddress;
}
public String getTaskArn() { return taskArn; }
@@ -127,6 +131,11 @@ public class EcsTask {
this.networkConfiguration = networkConfiguration;
}
+ public String getNetworkInterfaceId() { return networkInterfaceId; }
+ public void setNetworkInterfaceId(String networkInterfaceId) { this.networkInterfaceId = networkInterfaceId; }
+ public String getPrivateIpAddress() { return privateIpAddress; }
+ public void setPrivateIpAddress(String privateIpAddress) { this.privateIpAddress = privateIpAddress; }
+
public String getStoppedReason() { return stoppedReason; }
public void setStoppedReason(String stoppedReason) { this.stoppedReason = stoppedReason; }
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index c2a8e788b0..d738af19bf 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -116,6 +116,10 @@ quarkus:
floci:
port: 4566
base-url: "http://localhost:4566"
+ network:
+ security-group-enforcement:
+ enabled: false
+ helper-image: floci/network-helper:local
# hostname: "" # When set, overrides the hostname in response URLs (e.g. SQS QueueUrl).
# Needed for multi-container Docker setups. Example: FLOCI_HOSTNAME=floci
default-region: us-east-1
diff --git a/src/main/resources/docker/network-helper.Dockerfile b/src/main/resources/docker/network-helper.Dockerfile
new file mode 100644
index 0000000000..239b5ca125
--- /dev/null
+++ b/src/main/resources/docker/network-helper.Dockerfile
@@ -0,0 +1,3 @@
+FROM alpine:3.21
+RUN apk add --no-cache nftables iproute2 socat curl
+CMD ["sleep", "2147483647"]
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerLabelsTest.java b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerLabelsTest.java
index a229ef2121..6043e17cac 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerLabelsTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/docker/ContainerLifecycleManagerLabelsTest.java
@@ -9,6 +9,8 @@ import com.github.dockerjava.api.command.RemoveContainerCmd;
import com.github.dockerjava.api.command.StartContainerCmd;
import com.github.dockerjava.api.command.WaitContainerCmd;
import com.github.dockerjava.api.exception.NotFoundException;
+import com.github.dockerjava.api.model.Capability;
+import com.github.dockerjava.api.model.HostConfig;
import com.github.dockerjava.core.command.WaitContainerResultCallback;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.services.lambda.launcher.ImageCacheService;
@@ -26,6 +28,8 @@ import java.util.Optional;
import java.util.OptionalInt;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.Mockito.RETURNS_SELF;
@@ -99,6 +103,31 @@ class ContainerLifecycleManagerLabelsTest {
capturedLabels(createCmd));
}
+ @Test
+ void protectedWorkloadCannotAdministerOrSpoofItsNetwork() {
+ CreateContainerCmd createCmd = stubCreateContainer();
+
+ manager().create(specWithLabels(Map.of("floci.security-group-workload", "true")));
+
+ ArgumentCaptor<HostConfig> hostConfig = ArgumentCaptor.forClass(HostConfig.class);
+ verify(createCmd).withHostConfig(hostConfig.capture());
+ List<Capability> dropped = List.of(hostConfig.getValue().getCapDrop());
+ assertTrue(dropped.contains(Capability.NET_ADMIN));
+ assertTrue(dropped.contains(Capability.NET_RAW));
+ }
+
+ @Test
+ void firewallHelperGetsNetAdminInsteadOfFullPrivilege() {
+ CreateContainerCmd createCmd = stubCreateContainer();
+
+ manager().create(specWithLabels(Map.of("floci.security-group-helper", "true")));
+
+ ArgumentCaptor<HostConfig> hostConfig = ArgumentCaptor.forClass(HostConfig.class);
+ verify(createCmd).withHostConfig(hostConfig.capture());
+ assertEquals(List.of(Capability.NET_ADMIN), List.of(hostConfig.getValue().getCapAdd()));
+ assertNotEquals(Boolean.TRUE, hostConfig.getValue().getPrivileged());
+ }
+
@Test
void specLabelWinsOverDefaultOnKeyConflict() {
CreateContainerCmd createCmd = stubCreateContainer();
diff --git a/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2CapturedImageReclaimIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2CapturedImageReclaimIntegrationTest.java
index 2b4f22fc27..e700eed32b 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2CapturedImageReclaimIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2CapturedImageReclaimIntegrationTest.java
@@ -113,7 +113,7 @@ class Ec2CapturedImageReclaimIntegrationTest {
/** The image reference the container manager was asked to launch for the given instance. */
private ResolvedAmiImage launchedImageOf(Instance instance) {
ArgumentCaptor<ResolvedAmiImage> resolved = ArgumentCaptor.forClass(ResolvedAmiImage.class);
- verify(containerManager).launch(eq(instance), resolved.capture(), any(), anyString(), any());
+ verify(containerManager).launch(eq(instance), resolved.capture(), any(), anyString(), any(), any(), any());
return resolved.getValue();
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManagerTest.java
index 8a6fa051df..83b0e83cca 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManagerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2ContainerManagerTest.java
@@ -14,8 +14,10 @@ import com.github.dockerjava.api.command.ListContainersCmd;
import com.github.dockerjava.api.command.StartContainerCmd;
import com.github.dockerjava.api.exception.DockerException;
import com.github.dockerjava.api.model.Container;
+import com.github.dockerjava.api.model.ContainerConfig;
import com.github.dockerjava.api.model.ContainerNetwork;
import com.github.dockerjava.api.model.Frame;
+import com.github.dockerjava.api.model.HostConfig;
import com.github.dockerjava.api.model.StreamType;
import java.nio.charset.StandardCharsets;
import com.github.dockerjava.api.model.NetworkSettings;
@@ -333,6 +335,74 @@ class Ec2ContainerManagerTest {
+ "from an address nothing knows about");
}
+ @Test
+ void startOfAProtectedInstanceReusesTheHelperTransportAddress() throws Exception {
+ ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
+ when(lifecycleManager.isContainerRunning(TEST_CONTAINER_ID)).thenReturn(true);
+
+ DockerClient dockerClient = mock(DockerClient.class);
+ when(dockerClient.startContainerCmd(TEST_CONTAINER_ID))
+ .thenReturn(mock(StartContainerCmd.class, RETURNS_SELF));
+ // A workload in the helper's namespace has no Docker attachment of its own, so there is
+ // no bridge address on it to rediscover after the restart.
+ InspectContainerResponse workload = mock(InspectContainerResponse.class);
+ HostConfig hostConfig = mock(HostConfig.class);
+ when(hostConfig.getNetworkMode()).thenReturn("container:helper-1");
+ when(workload.getHostConfig()).thenReturn(hostConfig);
+ InspectContainerCmd inspectWorkload = mock(InspectContainerCmd.class);
+ when(inspectWorkload.exec()).thenReturn(workload);
+ when(dockerClient.inspectContainerCmd(TEST_CONTAINER_ID)).thenReturn(inspectWorkload);
+
+ InspectContainerResponse helper = mock(InspectContainerResponse.class);
+ ContainerConfig helperConfig = mock(ContainerConfig.class);
+ when(helperConfig.getLabels()).thenReturn(Map.of(
+ "floci.security-group-helper", "true",
+ "io.floci.service", "ec2",
+ "io.floci.resource-id", "i-protected",
+ "floci_owner_port", "4566"));
+ when(helper.getConfig()).thenReturn(helperConfig);
+ NetworkSettings helperNetworks = mock(NetworkSettings.class);
+ when(helperNetworks.getNetworks())
+ .thenReturn(Map.of("bridge", new ContainerNetwork().withIpv4Address("172.17.0.7")));
+ when(helper.getNetworkSettings()).thenReturn(helperNetworks);
+ InspectContainerCmd inspectHelper = mock(InspectContainerCmd.class);
+ when(inspectHelper.exec()).thenReturn(helper);
+ when(dockerClient.inspectContainerCmd("helper-1")).thenReturn(inspectHelper);
+
+ EmulatorConfig config = mock(EmulatorConfig.class, RETURNS_DEEP_STUBS);
+ when(config.port()).thenReturn(4566);
+ when(config.docker().resourceNamespace()).thenReturn(Optional.empty());
+
+ Ec2MetadataServer metadataServer = mock(Ec2MetadataServer.class);
+ Ec2ContainerManager manager = new Ec2ContainerManager(
+ mock(ContainerBuilder.class),
+ lifecycleManager,
+ mock(ContainerLogStreamer.class),
+ mock(ContainerDetector.class),
+ mock(DockerHostResolver.class),
+ dockerClient,
+ mock(PortAllocator.class),
+ config,
+ metadataServer,
+ mock(Ec2PortForwardManager.class),
+ mock(RegionResolver.class),
+ mock(ContainerNetworkReachability.class),
+ mock(VpcNetworkManager.class),
+ mock(ContainerReachableEndpoint.class),
+ mock(SecurityGroupFirewallManager.class));
+
+ Instance instance = new Instance();
+ instance.setInstanceId("i-protected");
+ instance.setDockerContainerId(TEST_CONTAINER_ID);
+
+ manager.start(instance);
+ awaitUntil(() -> "running".equals(instance.getState().getName()), Duration.ofSeconds(5));
+
+ verify(metadataServer, timeout(2000)).registerContainer("172.17.0.7", "i-protected", instance);
+ assertEquals("172.17.0.7", instance.getContainerBridgeIp(),
+ "StartInstances must keep addressing the instance through its protected namespace");
+ }
+
private static Ec2ContainerManager managerWith(ContainerLifecycleManager lifecycleManager,
DockerClient dockerClient,
Ec2MetadataServer metadataServer) {
diff --git a/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2NetworkInterfaceIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2NetworkInterfaceIntegrationTest.java
index 15ef32aec0..9b57d9388c 100644
--- a/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2NetworkInterfaceIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2NetworkInterfaceIntegrationTest.java
@@ -42,10 +42,23 @@ class Ec2NetworkInterfaceIntegrationTest {
@Test
@Order(1)
void discoverDefaultSubnetAndSecurityGroup() {
+ String vpcId = given()
+ .formParam("Action", "DescribeVpcs")
+ .formParam("Filter.1.Name", "is-default")
+ .formParam("Filter.1.Value.1", "true")
+ .header("Authorization", AUTH_HEADER)
+ .when()
+ .post("/")
+ .then()
+ .statusCode(200)
+ .extract().path("DescribeVpcsResponse.vpcSet.item[0].vpcId");
+
subnetId = given()
.formParam("Action", "DescribeSubnets")
.formParam("Filter.1.Name", "default-for-az")
.formParam("Filter.1.Value.1", "true")
+ .formParam("Filter.2.Name", "vpc-id")
+ .formParam("Filter.2.Value.1", vpcId)
.header("Authorization", AUTH_HEADER)
.when()
.post("/")
@@ -57,6 +70,8 @@ class Ec2NetworkInterfaceIntegrationTest {
.formParam("Action", "DescribeSecurityGroups")
.formParam("Filter.1.Name", "group-name")
.formParam("Filter.1.Value.1", "default")
+ .formParam("Filter.2.Name", "vpc-id")
+ .formParam("Filter.2.Value.1", vpcId)
.header("Authorization", AUTH_HEADER)
.when()
.post("/")
diff --git a/src/test/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallDockerIntegrationTest.java
new file mode 100644
index 0000000000..dd00d355be
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/ec2/SecurityGroupFirewallDockerIntegrationTest.java
@@ -0,0 +1,155 @@
+package io.github.hectorvent.floci.services.ec2;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.async.ResultCallback;
+import com.github.dockerjava.api.model.Frame;
+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
+import io.github.hectorvent.floci.services.ec2.model.IpPermission;
+import io.github.hectorvent.floci.services.ec2.model.IpRange;
+import io.github.hectorvent.floci.services.ec2.model.SecurityGroup;
+import io.quarkus.test.junit.QuarkusTest;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+
+@QuarkusTest
+class SecurityGroupFirewallDockerIntegrationTest {
+
+ @Inject SecurityGroupFirewallManager firewall;
+ @Inject ContainerLifecycleManager lifecycle;
+ @Inject ContainerBuilder builder;
+ @Inject DockerClient docker;
+
+ @Test
+ void deniedLogicalCidrBlocksManagedPeerThenRuleUpdatePermitsIt() throws Exception {
+ Assumptions.assumeTrue(firewall.enabled());
+ try {
+ Assumptions.assumeTrue("linux".equalsIgnoreCase(docker.infoCmd().exec().getOsType()));
+ } catch (Exception e) {
+ Assumptions.abort("Docker daemon is unavailable");
+ }
+
+ String suffix = UUID.randomUUID().toString().substring(0, 8);
+ String targetEni = "eni-target-" + suffix;
+ String sourceEni = "eni-source-" + suffix;
+ List<SecurityGroupFirewallManager.Namespace> namespaces = new ArrayList<>();
+ List<String> workers = new ArrayList<>();
+ try {
+ SecurityGroupFirewallManager.Namespace target = firewall.createNamespace("ec2", "sg-target-" + suffix, "000000000000",
+ "us-east-1", Optional.empty(), Map.of());
+ namespaces.add(target);
+ SecurityGroupFirewallManager.Namespace source = firewall.createNamespace("ec2", "sg-source-" + suffix, "000000000000",
+ "us-east-1", Optional.empty(), Map.of());
+ namespaces.add(source);
+
+ SecurityGroup targetGroup = group("sg-target", false, "10.0.0.0/24");
+ SecurityGroup sourceGroup = group("sg-source", true, "0.0.0.0/0");
+ firewall.register(endpoint(targetEni, "10.0.0.2", target, targetGroup),
+ target.helperId(), Map.of());
+ firewall.register(endpoint(sourceEni, "10.1.0.2", source, sourceGroup),
+ source.helperId(), Map.of());
+
+ workers.add(lifecycle.createAndStart(builder.newContainer("alpine:3.21")
+ .withName("floci-sg-test-target-" + suffix)
+ .withNetworkMode("container:" + target.helperId())
+ .withEntrypoint(List.of("sh", "-c"))
+ .withCmd(List.of("exec nc -lk -p 8080 -e /bin/cat"))
+ .build()).containerId());
+ workers.add(lifecycle.createAndStart(builder.newContainer("alpine:3.21")
+ .withName("floci-sg-test-source-" + suffix)
+ .withNetworkMode("container:" + source.helperId())
+ .withEntrypoint(List.of("sleep"))
+ .withCmd(List.of("600"))
+ .build()).containerId());
+
+ for (int attempt = 0; attempt < 20 && connect(workers.getFirst(), "127.0.0.1") != 0; attempt++) {
+ Thread.sleep(100);
+ }
+ assertEquals(0, connect(workers.getFirst(), "127.0.0.1"));
+ assertNotEquals(0, connect(workers.get(1), target.transportAddress()));
+
+ IpRange corrected = new IpRange();
+ corrected.setCidrIp("10.1.0.0/24");
+ targetGroup.getIpPermissions().getFirst().getIpRanges().add(corrected);
+ firewall.refreshPolicies("us-east-1", Map.of("sg-target", targetGroup,
+ "sg-source", sourceGroup), Map.of());
+ assertEquals(0, connect(workers.get(1), target.transportAddress()));
+
+ firewall.quarantineSurvivingNamespaces();
+ assertNotEquals(0, connect(workers.get(1), target.transportAddress()));
+ firewall.refreshPolicies("us-east-1", Map.of("sg-target", targetGroup,
+ "sg-source", sourceGroup), Map.of());
+ assertEquals(0, connect(workers.get(1), target.transportAddress()));
+
+ SecurityGroup replacement = group("sg-replacement", false, "192.0.2.0/24");
+ Map<String, SecurityGroup> groups = Map.of("sg-target", targetGroup, "sg-source", sourceGroup,
+ "sg-replacement", replacement);
+ firewall.updateGroups(targetEni, Set.of("sg-replacement"), groups, Map.of());
+ assertNotEquals(0, connect(workers.get(1), target.transportAddress()));
+ firewall.updateGroups(targetEni, Set.of("sg-target"), groups, Map.of());
+ assertEquals(0, connect(workers.get(1), target.transportAddress()));
+
+ docker.stopContainerCmd(target.helperId()).withTimeout(0).exec();
+ firewall.refreshPolicies("us-east-1", groups, Map.of());
+ assertFalse(docker.inspectContainerCmd(workers.getFirst()).exec().getState().getRunning());
+ } finally {
+ for (String worker : workers) {
+ lifecycle.removeIfExists(worker);
+ }
+ firewall.unregister(sourceEni);
+ firewall.unregister(targetEni);
+ for (SecurityGroupFirewallManager.Namespace namespace : namespaces) {
+ lifecycle.removeIfExists(namespace.helperId());
+ }
+ }
+ }
+
+ private int connect(String worker, String address) throws Exception {
+ String id = docker.execCreateCmd(worker).withAttachStdout(true).withAttachStderr(true)
+ .withCmd("nc", "-z", "-w", "2", address, "8080")
+ .exec().getId();
+ try (ResultCallback.Adapter<Frame> callback = new ResultCallback.Adapter<>()) {
+ docker.execStartCmd(id).exec(callback).awaitCompletion(5, TimeUnit.SECONDS);
+ }
+ return docker.inspectExecCmd(id).exec().getExitCodeLong().intValue();
+ }
+
+ private static SecurityGroup group(String id, boolean egress, String cidr) {
+ SecurityGroup group = new SecurityGroup();
+ group.setGroupId(id);
+ group.setVpcId("vpc-1");
+ group.setOwnerId("000000000000");
+ IpPermission permission = new IpPermission();
+ permission.setIpProtocol("tcp");
+ permission.setFromPort(8080);
+ permission.setToPort(8080);
+ IpRange range = new IpRange();
+ range.setCidrIp(cidr);
+ permission.getIpRanges().add(range);
+ if (egress) {
+ group.getIpPermissionsEgress().add(permission);
+ } else {
+ group.getIpPermissions().add(permission);
+ }
+ return group;
+ }
+
+ private static SecurityGroupNftCompiler.Endpoint endpoint(String eni, String logical,
+ SecurityGroupFirewallManager.Namespace namespace, SecurityGroup group) {
+ return new SecurityGroupNftCompiler.Endpoint("000000000000", "us-east-1", "vpc-1",
+ eni, logical, namespace.transportAddress(), Set.of(group.getGroupId()), List.of(group));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ec2/SecurityGroupNftCompilerTest.java b/src/test/java/io/github/hectorvent/floci/services/ec2/SecurityGroupNftCompilerTest.java
new file mode 100644
index 0000000000..18963de583
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/ec2/SecurityGroupNftCompilerTest.java
@@ -0,0 +1,80 @@
+package io.github.hectorvent.floci.services.ec2;
+
+import io.github.hectorvent.floci.services.ec2.model.IpPermission;
+import io.github.hectorvent.floci.services.ec2.model.IpRange;
+import io.github.hectorvent.floci.services.ec2.model.SecurityGroup;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class SecurityGroupNftCompilerTest {
+
+ @Test
+ void managedPeersAreCheckedBeforeDockerAddressCidrs() {
+ SecurityGroup group = new SecurityGroup();
+ group.setVpcId("vpc-1");
+ group.setOwnerId("000000000000");
+ IpPermission permission = new IpPermission();
+ permission.setIpProtocol("tcp");
+ permission.setFromPort(443);
+ permission.setToPort(443);
+ IpRange cidr = new IpRange();
+ cidr.setCidrIp("10.0.0.0/16");
+ permission.getIpRanges().add(cidr);
+ IpRange broadTransport = new IpRange();
+ broadTransport.setCidrIp("172.17.0.0/16");
+ permission.getIpRanges().add(broadTransport);
+ group.getIpPermissions().add(permission);
+
+ SecurityGroupNftCompiler.Endpoint target = endpoint("eni-a", "10.0.0.2", "172.17.0.2", group);
+ SecurityGroupNftCompiler.Endpoint allowed = endpoint("eni-b", "10.0.0.3", "172.17.0.3", group);
+ SecurityGroupNftCompiler.Endpoint denied = endpoint("eni-c", "10.1.0.3", "172.17.0.4", group);
+ String nft = SecurityGroupNftCompiler.compile(target, List.of(allowed, denied), Map.of());
+
+ assertTrue(nft.contains("ip saddr 172.17.0.3 tcp dport 443 accept"));
+ assertTrue(nft.indexOf("ip saddr 172.17.0.4 drop")
+ < nft.indexOf("ip saddr 172.17.0.0/16 tcp dport 443 accept"));
+ }
+
+ @Test
+ void icmpTypeAndCodeRepeatTheProtocolKeyword() {
+ SecurityGroup group = new SecurityGroup();
+ group.setVpcId("vpc-1");
+ group.setOwnerId("000000000000");
+ // AWS maps IpPermission.FromPort to the ICMP type and ToPort to the ICMP code.
+ IpPermission echo = new IpPermission();
+ echo.setIpProtocol("icmp");
+ echo.setFromPort(8);
+ echo.setToPort(0);
+ IpRange cidr = new IpRange();
+ cidr.setCidrIp("10.0.0.0/16");
+ echo.getIpRanges().add(cidr);
+ group.getIpPermissions().add(echo);
+ IpPermission anyIcmpv6 = new IpPermission();
+ anyIcmpv6.setIpProtocol("icmpv6");
+ anyIcmpv6.setFromPort(-1);
+ anyIcmpv6.setToPort(-1);
+ IpRange allV4 = new IpRange();
+ allV4.setCidrIp("10.1.0.0/16");
+ anyIcmpv6.getIpRanges().add(allV4);
+ group.getIpPermissions().add(anyIcmpv6);
+
+ String nft = SecurityGroupNftCompiler.compile(
+ endpoint("eni-a", "10.0.0.2", "172.17.0.2", group), List.of(), Map.of());
+
+ assertTrue(nft.contains("ip saddr 10.0.0.0/16 icmp type 8 icmp code 0 accept"));
+ assertFalse(nft.contains("icmp type 8 code 0"));
+ assertTrue(nft.contains("ip saddr 10.1.0.0/16 meta l4proto icmpv6 accept"));
+ }
+
+ private static SecurityGroupNftCompiler.Endpoint endpoint(String eni, String logical,
+ String transport, SecurityGroup group) {
+ return new SecurityGroupNftCompiler.Endpoint("000000000000", "us-east-1", "vpc-1",
+ eni, logical, transport, Set.of("sg-1"), List.of(group));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/ec2/SecurityGroupPolicyTest.java b/src/test/java/io/github/hectorvent/floci/services/ec2/SecurityGroupPolicyTest.java
new file mode 100644
index 0000000000..5707fd2f59
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/ec2/SecurityGroupPolicyTest.java
@@ -0,0 +1,91 @@
+package io.github.hectorvent.floci.services.ec2;
+
+import io.github.hectorvent.floci.services.ec2.model.IpPermission;
+import io.github.hectorvent.floci.services.ec2.model.IpRange;
+import io.github.hectorvent.floci.services.ec2.model.Ipv6Range;
+import io.github.hectorvent.floci.services.ec2.model.PrefixListId;
+import io.github.hectorvent.floci.services.ec2.model.SecurityGroup;
+import io.github.hectorvent.floci.services.ec2.model.UserIdGroupPair;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class SecurityGroupPolicyTest {
+
+ @Test
+ void evaluatesLogicalPeerIdentityAndPorts() {
+ SecurityGroup group = group("sg-target");
+ IpPermission rule = tcp(443);
+ IpRange cidr = new IpRange();
+ cidr.setCidrIp("10.42.0.0/16");
+ rule.getIpRanges().add(cidr);
+ group.getIpPermissions().add(rule);
+
+ assertTrue(SecurityGroupPolicy.allows(List.of(group), false, "6", 443, -1,
+ peer("10.42.7.9"), Map.of()));
+ assertFalse(SecurityGroupPolicy.allows(List.of(group), false, "tcp", 443, -1,
+ peer("10.43.7.9"), Map.of()));
+ assertFalse(SecurityGroupPolicy.allows(List.of(group), false, "tcp", 80, -1,
+ peer("10.42.7.9"), Map.of()));
+ assertFalse(SecurityGroupPolicy.allows(List.of(group), true, "tcp", 443, -1,
+ peer("10.42.7.9"), Map.of()));
+ }
+
+ @Test
+ void evaluatesReferencesPrefixListsAndIpv6() {
+ SecurityGroup group = group("sg-target");
+ IpPermission byGroup = tcp(80);
+ UserIdGroupPair reference = new UserIdGroupPair();
+ reference.setGroupId("sg-peer");
+ reference.setUserId("000000000000");
+ byGroup.getUserIdGroupPairs().add(reference);
+ group.getIpPermissions().add(byGroup);
+
+ assertTrue(SecurityGroupPolicy.allows(List.of(group), false, "tcp", 80, -1,
+ peer("172.17.0.4"), Map.of()));
+ assertFalse(SecurityGroupPolicy.allows(List.of(group), false, "tcp", 80, -1,
+ new SecurityGroupPolicy.Peer("172.17.0.4", "000000000000", "vpc-other", Set.of("sg-peer")), Map.of()));
+
+ IpPermission prefix = tcp(8080);
+ PrefixListId id = new PrefixListId();
+ id.setPrefixListId("pl-1");
+ prefix.getPrefixListIds().add(id);
+ group.getIpPermissions().add(prefix);
+ assertTrue(SecurityGroupPolicy.allows(List.of(group), false, "tcp", 8080, -1,
+ peer("10.42.7.9"), Map.of("pl-1", List.of("10.42.0.0/16"))));
+
+ IpPermission ipv6 = tcp(8443);
+ Ipv6Range range = new Ipv6Range();
+ range.setCidrIpv6("2001:db8::/32");
+ ipv6.getIpv6Ranges().add(range);
+ group.getIpPermissions().add(ipv6);
+ assertTrue(SecurityGroupPolicy.allows(List.of(group), false, "tcp", 8443, -1,
+ peer("2001:db8::1"), Map.of()));
+ assertFalse(SecurityGroupPolicy.inCidr("10.42.7.9", "invalid/16"));
+ }
+
+ private static SecurityGroup group(String id) {
+ SecurityGroup group = new SecurityGroup();
+ group.setGroupId(id);
+ group.setOwnerId("000000000000");
+ group.setVpcId("vpc-1");
+ return group;
+ }
+
+ private static IpPermission tcp(int port) {
+ IpPermission rule = new IpPermission();
+ rule.setIpProtocol("tcp");
+ rule.setFromPort(port);
+ rule.setToPort(port);
+ return rule;
+ }
+
+ private static SecurityGroupPolicy.Peer peer(String ip) {
+ return new SecurityGroupPolicy.Peer(ip, "000000000000", "vpc-1", Set.of("sg-peer"));
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/eventbridge/EventBridgeSchedulerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/eventbridge/EventBridgeSchedulerIntegrationTest.java
index 60cf54350d..ac7ccd7f8f 100644
--- a/src/test/java/io/github/hectorvent/floci/services/eventbridge/EventBridgeSchedulerIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/eventbridge/EventBridgeSchedulerIntegrationTest.java
@@ -300,6 +300,8 @@ class EventBridgeSchedulerIntegrationTest {
@Override
public StorageConfig storage() { return null; }
@Override
+ public NetworkConfig network() { return null; }
+ @Override
public DnsConfig dns() {
return new DnsConfig() {
@Override
diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml
index b0557a6316..010a662f05 100644
--- a/src/test/resources/application.yml
+++ b/src/test/resources/application.yml
@@ -12,6 +12,10 @@ quarkus:
floci:
base-url: "http://localhost:4566"
+ network:
+ security-group-enforcement:
+ enabled: true
+ helper-image: floci/network-helper:local
default-region: us-east-1
default-account-id: "000000000000"
ai-mock-config-file: ${AI_MOCK_CONFIG:src/test/resources/fixtures/ai-mock-config.json}