Light, fluffy, and always free - The AWS Local Emulator alternative
fix(ec2,ssm): explain missing IMDS registration for SSM-only managed instances (#3283)
Matheus Andréf77cf4da77fd
diff --git a/docs/services/ec2.md b/docs/services/ec2.mdindex fe7a417f5a..fa692e9bad 100644--- a/docs/services/ec2.md+++ b/docs/services/ec2.md@@ -138,6 +138,14 @@ curl -s -H "x-aws-ec2-metadata-token: $TOKEN" \IAM credentials are served when the instance has an `IamInstanceProfile.Arn` set at launch. The container can then call other Floci services with full SigV4 validation using the standard AWS SDK credential chain.+### IMDS and SSM managed instances++IMDS only knows about containers that Floci itself launched through `RunInstances`. That launch is what installs the link-local `169.254.169.254` proxy inside the container and maps the container's IP to its instance record.++Registering a container with SSM is independent of this: an SSM agent that calls `UpdateInstanceInformation` becomes a managed instance, but that does not create an EC2 instance record, does not install the IMDS proxy, and does not register the container with IMDS. From such a container, `curl http://169.254.169.254/...` fails outright (no proxy), and a request sent straight to the host IMDS port returns `404` with a message explaining that no EC2 instance is registered for the source IP. Floci also logs a warning when an SSM agent registers from a container that is not backed by a Floci EC2 instance.++To get a container that both answers IMDS (including instance profile credentials) and executes `SendCommand` directly, launch it with `RunInstances` first, then register that same container's SSM agent as a managed instance. Do not register an arbitrary or pre-existing container directly with SSM and expect IMDS to work. See [SSM](ssm.md#run-command-execution) for the SendCommand side of this.+## Default ResourcesFloci seeds the following resources on first use in each region so Terraform, the AWS CLI, and SDK clients work out of the box without any setup:diff --git a/docs/services/ssm.md b/docs/services/ssm.mdindex 2a097f168b..c2305e183c 100644--- a/docs/services/ssm.md+++ b/docs/services/ssm.md@@ -28,7 +28,7 @@| Action | Description ||---|---|-| `UpdateInstanceInformation` | Register or update an SSM agent record for an instance |+| `UpdateInstanceInformation` | Register or update an SSM agent record for an instance. Does not create an EC2 instance or register the container with IMDS (see [Run Command Execution](#run-command-execution)) || `DescribeInstanceInformation` | List registered SSM managed instances || `SendCommand` | Create command invocations for target instances || `GetCommandInvocation` | Return a command invocation result |@@ -50,6 +50,11 @@If the target is not a Floci EC2 container, or if the document is not supported for direct execution, Floci falls back to the SSM agent polling flow. In that mode, `SendCommand` queues an ec2messages payload and the invocation completes after an agent calls `SendReply`.+!!! note "Managed instance registration is independent of EC2 and IMDS"+ Registering a container's SSM agent with `UpdateInstanceInformation` only creates the managed instance record. It does not create an EC2 instance, does not install the link-local `169.254.169.254` proxy in the container, and does not register the container with the [EC2 IMDS server](ec2.md#imds-and-ssm-managed-instances). A container that was only registered with SSM therefore has no working IMDS endpoint and no instance profile credentials, and `SendCommand` against it uses the agent polling flow rather than direct execution.++ To get a container that both answers IMDS and runs `SendCommand` directly, launch it with EC2 `RunInstances` first and then register that same container's SSM agent. Floci logs a warning when an agent registers from a container that is not backed by a Floci EC2 instance.+Direct command output follows the AWS inline output limits: first 24,000 characters of stdout and first 8,000 characters of stderr. Commands that exceed `TimeoutSeconds` are constrained inside the target container when the container has the `timeout` command available, and terminal timeout results are marked `TimedOut` with `StatusDetails` set to `Execution Timed Out`; commands with nonzero exit codes are marked `Failed`.## Configurationdiff --git a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2MetadataServer.java b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2MetadataServer.javaindex 3cb696c290..450ffcd78c 100644--- a/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2MetadataServer.java+++ b/src/main/java/io/github/hectorvent/floci/services/ec2/Ec2MetadataServer.java@@ -143,6 +143,10 @@ public class Ec2MetadataServer {if (inst != null) {tokenToInstance.put(token, inst);}+ else {+ LOG.debugv("IMDS: token requested from {0}, which is not a registered EC2 container; "+ + "metadata requests with this token will fail", ctx.request().remoteAddress().host());+ }ctx.response().setStatusCode(200)@@ -354,12 +358,30 @@ public class Ec2MetadataServer {String remoteIp = ctx.request().remoteAddress().host();Instance inst = containerIpToInstance.get(remoteIp);if (inst == null) {- LOG.warnv("IMDS: could not identify instance for request from {0}", remoteIp);- ctx.response().setStatusCode(404).end("Instance not found");+ String message = unregisteredContainerMessage(remoteIp);+ LOG.warnv("IMDS: {0}", message);+ ctx.response().setStatusCode(404)+ .putHeader("content-type", "text/plain")+ .end(message);}return inst;}+ /**+ * Explains why IMDS has nothing to serve for a request coming from {@code remoteIp}.+ *+ * <p>IMDS only knows about containers that {@link Ec2ContainerManager} launched through EC2+ * {@code RunInstances}. Registering a container's SSM agent as a managed instance+ * ({@code UpdateInstanceInformation}) does not create an EC2 instance record, so a container+ * that was only registered with SSM ends up here.+ */+ static String unregisteredContainerMessage(String remoteIp) {+ return "Instance not found: no EC2 instance is registered for source IP " + remoteIp + ". "+ + "IMDS only serves containers launched through EC2 RunInstances; "+ + "registering a container as an SSM managed instance does not register it with IMDS. "+ + "Launch the container with RunInstances first, then register its SSM agent.";+ }+// ── Utilities ─────────────────────────────────────────────────────────────String resolveRoleName(String profileArn) {diff --git a/src/main/java/io/github/hectorvent/floci/services/ssm/SsmCommandService.java b/src/main/java/io/github/hectorvent/floci/services/ssm/SsmCommandService.javaindex d7a3dde254..2075500eb6 100644--- a/src/main/java/io/github/hectorvent/floci/services/ssm/SsmCommandService.java+++ b/src/main/java/io/github/hectorvent/floci/services/ssm/SsmCommandService.java@@ -114,12 +114,23 @@ public class SsmCommandService implements Resettable {info.setComputerName(request.path("Hostname").asText(instanceId));info.setRegion(region);- if (info.getRegistrationDate() == null) {+ boolean firstRegistration = info.getRegistrationDate() == null;+ if (firstRegistration) {info.setRegistrationDate(Instant.now());}instanceStore.put(instanceKey(region, instanceId), info);LOG.infov("SSM agent registered: instanceId={0} platform={1}/{2}", instanceId, info.getPlatformType(), info.getPlatformName());++ // Registering with SSM does not create an EC2 instance record. Make it obvious when the+ // agent lives in a container Floci did not launch through RunInstances: SendCommand falls+ // back to agent polling and IMDS (169.254.169.254) has nothing to serve for that container.+ if (firstRegistration && !directCommandExecutor.isContainerBacked(instanceId)) {+ LOG.warnv("SSM managed instance {0} is not backed by a running Floci EC2 container. "+ + "SendCommand will use the agent polling flow and IMDS will not answer for it. "+ + "To get both, launch the container with EC2 RunInstances before registering its SSM agent.",+ instanceId);+ }}public List<InstanceInformation> describeInstanceInformation(String region) {diff --git a/src/main/java/io/github/hectorvent/floci/services/ssm/SsmDirectCommandExecutor.java b/src/main/java/io/github/hectorvent/floci/services/ssm/SsmDirectCommandExecutor.javaindex b11b51f001..b3db5b5d57 100644--- a/src/main/java/io/github/hectorvent/floci/services/ssm/SsmDirectCommandExecutor.java+++ b/src/main/java/io/github/hectorvent/floci/services/ssm/SsmDirectCommandExecutor.java@@ -69,7 +69,15 @@ public class SsmDirectCommandExecutor {if (!"AWS-RunShellScript".equals(documentName)) {return false;}+ return isContainerBacked(instanceId);+ }+ /**+ * Whether {@code instanceId} is a Floci EC2 instance whose Docker container is running.+ * Only such instances get direct command execution and an IMDS registration; any other+ * managed instance is served through the SSM agent polling flow.+ */+ public boolean isContainerBacked(String instanceId) {Instance instance = ec2Service.findInstanceById(instanceId);if (instance == null || instance.getDockerContainerId() == null || instance.getDockerContainerId().isBlank()) {return false;diff --git a/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2MetadataServerTest.java b/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2MetadataServerTest.javaindex 6981f67c5b..5c1bdb8672 100644--- a/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2MetadataServerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ec2/Ec2MetadataServerTest.java@@ -89,6 +89,16 @@ class Ec2MetadataServerTest {server.registeredContainer("192.168.215.7").orElseThrow());}+ @Test+ void unregisteredContainerMessageExplainsMissingEc2Record() {+ String message = Ec2MetadataServer.unregisteredContainerMessage("172.17.0.9");++ assertTrue(message.startsWith("Instance not found"));+ assertTrue(message.contains("172.17.0.9"));+ assertTrue(message.contains("RunInstances"));+ assertTrue(message.contains("SSM managed instance"));+ }+@Testvoid iamCredentialRoleNameComesFromInstanceProfileRole() {IamService iamService = mock(IamService.class);diff --git a/src/test/java/io/github/hectorvent/floci/services/ssm/SsmCommandServiceDirectExecutionTest.java b/src/test/java/io/github/hectorvent/floci/services/ssm/SsmCommandServiceDirectExecutionTest.javaindex b2da0267f9..d2f5d5255f 100644--- a/src/test/java/io/github/hectorvent/floci/services/ssm/SsmCommandServiceDirectExecutionTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ssm/SsmCommandServiceDirectExecutionTest.java@@ -32,6 +32,8 @@ import static org.mockito.ArgumentMatchers.eq;import static org.mockito.Mockito.doAnswer;import static org.mockito.Mockito.mock;import static org.mockito.Mockito.spy;+import static org.mockito.Mockito.times;+import static org.mockito.Mockito.verify;import static org.mockito.Mockito.when;class SsmCommandServiceDirectExecutionTest {@@ -39,6 +41,32 @@ class SsmCommandServiceDirectExecutionTest {private final ObjectMapper objectMapper = new ObjectMapper();private final RegionResolver regionResolver = mock(RegionResolver.class);+ @Test+ void agentRegistrationChecksEc2BackingOnFirstRegistrationOnly() throws Exception {+ SsmDirectCommandExecutor executor = mock(SsmDirectCommandExecutor.class);+ when(executor.isContainerBacked("mi-agent-only")).thenReturn(false);++ SsmCommandService service = new SsmCommandService(+ new InMemoryStorageFactory(),+ objectMapper,+ regionResolver,+ executor);++ String registration = """+ {+ "InstanceId": "mi-agent-only",+ "AgentName": "amazon-ssm-agent",+ "IPAddress": "172.17.0.9"+ }+ """;+ service.updateInstanceInformation(objectMapper.readTree(registration), "us-west-2");+ service.updateInstanceInformation(objectMapper.readTree(registration), "us-west-2");++ verify(executor, times(1)).isContainerBacked("mi-agent-only");+ assertEquals(1, service.describeInstanceInformation("us-west-2").size());+ assertEquals("mi-agent-only", service.describeInstanceInformation("us-west-2").getFirst().getInstanceId());+ }+@Testvoid directExecutionCompletesCommandWithoutQueuedAgentMessage() throws Exception {SsmDirectCommandExecutor executor = mock(SsmDirectCommandExecutor.class);diff --git a/src/test/java/io/github/hectorvent/floci/services/ssm/SsmDirectCommandExecutorTest.java b/src/test/java/io/github/hectorvent/floci/services/ssm/SsmDirectCommandExecutorTest.javaindex 8aa1c40e5a..ee0c26c0ca 100644--- a/src/test/java/io/github/hectorvent/floci/services/ssm/SsmDirectCommandExecutorTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/ssm/SsmDirectCommandExecutorTest.java@@ -133,6 +133,29 @@ class SsmDirectCommandExecutorTest {assertTrue(result.isEmpty());}+ @Test+ void containerBackedRequiresRunningEc2Container() {+ DockerClient dockerClient = mock(DockerClient.class);+ Ec2Service ec2Service = mock(Ec2Service.class);+ Instance running = new Instance();+ running.setInstanceId("i-running");+ running.setDockerContainerId("container-1");+ Instance stopped = new Instance();+ stopped.setInstanceId("i-stopped");+ stopped.setDockerContainerId("container-2");+ when(ec2Service.findInstanceById("i-running")).thenReturn(running);+ when(ec2Service.findInstanceById("i-stopped")).thenReturn(stopped);+ when(ec2Service.findInstanceById("mi-agent-only")).thenReturn(null);+ when(ec2Service.isInstanceContainerRunning("i-running")).thenReturn(true);+ when(ec2Service.isInstanceContainerRunning("i-stopped")).thenReturn(false);++ SsmDirectCommandExecutor executor = new SsmDirectCommandExecutor(dockerClient, ec2Service);++ assertTrue(executor.isContainerBacked("i-running"));+ assertFalse(executor.isContainerBacked("i-stopped"));+ assertFalse(executor.isContainerBacked("mi-agent-only"));+ }+@Testvoid returnsEmptyForUnsupportedDocument() {DockerClient dockerClient = mock(DockerClient.class);