Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
feat(ec2): BYO-VM register image allows an image to run as EC2 (#1473)
Nahuel Nucerafa4b1f14cd9c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c44d3ed9..877a992d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,11 @@ Versioning follows [Semantic Versioning](https://semver.org/).## [Unreleased]+### Added+- **EC2 — instances can have a real box behind them (`RegisterImage`)** — `RunInstances` returned a `running` record and booted nothing. `RegisterImage` now takes a container reference in `ImageLocation` and returns an `ami-` id; launching that id boots the image as a container whose IP becomes the instance's address, so `ssm:SendCommand` can report a real exit code. Such an AMI is `instance-store` backed: no EBS root volume, `StopInstances` / `StartInstances` answer `UnsupportedOperation`, and a container that dies reports `terminated`. Snapshot-backed registration is accepted as AWS accepts it and fails at `RunInstances` with `InvalidAMIID.Unavailable`, as does a reference that cannot be pulled. Registering is the only opt-in — with nothing registered EC2 never reaches for Docker. Reported by @iot-rocket.+- **SSM — Run Command** — `SendCommand`, `GetCommandInvocation`, `ListCommands` and `DescribeInstanceInformation` returned `InvalidAction`, so the standard instance health probe could not run. Invocations are asynchronous as on AWS: `SendCommand` answers `Pending` and the caller polls to a terminal state. `AWS-RunShellScript` runs in the instance's container, so `Status` is `Success` or `Failed` on the real exit code, with the script's output. An instance with no agent answering is refused with `InvalidInstanceId`. Contributed by @bandle.+- **EC2 — `RebootInstances` rejects unknown instance ids** — it returned `true` for any id. It now answers `InvalidInstanceID.NotFound`, matching AWS.+## [1.4.21] — 2026-08-20### Addeddiff --git a/README.md b/README.mdindex 65495749..3707482f 100644--- a/README.md+++ b/README.md@@ -571,7 +571,7 @@ MiniStack expands AWS SAM transforms (`Transform: AWS::Serverless-2016-10-31`) i| **Athena** | StartQueryExecution, GetQueryExecution, GetQueryResults, StopQueryExecution, ListQueryExecutions, BatchGetQueryExecution, CreateWorkGroup, DeleteWorkGroup, GetWorkGroup, ListWorkGroups, UpdateWorkGroup, CreateNamedQuery, DeleteNamedQuery, GetNamedQuery, ListNamedQueries, BatchGetNamedQuery, CreateDataCatalog, GetDataCatalog, ListDataCatalogs, DeleteDataCatalog, UpdateDataCatalog, CreatePreparedStatement, GetPreparedStatement, DeletePreparedStatement, ListPreparedStatements, GetTableMetadata, ListTableMetadata, TagResource, UntagResource, ListTagsForResource | Real SQL via **DuckDB** when installed (`pip install duckdb`), otherwise returns mock results; result pagination; column type metadata || **Firehose** | CreateDeliveryStream, DeleteDeliveryStream, DescribeDeliveryStream, ListDeliveryStreams, PutRecord, PutRecordBatch, UpdateDestination, TagDeliveryStream, UntagDeliveryStream, ListTagsForDeliveryStream, StartDeliveryStreamEncryption, StopDeliveryStreamEncryption | S3 destinations write records to the local S3 emulator; all other destination types buffer in-memory; concurrency-safe `UpdateDestination` via `VersionId`; `DeliveryStreamType=KinesisStreamAsSource` consumes records from the source Kinesis stream on `PutRecord` / `PutRecords` and forwards them to the configured S3 destination, honoring `Prefix` and `DeliveryStartTimestamp`; `ProcessingConfiguration.Processors[].Type=Lambda` invoked per-batch with the AWS-shape event (`{invocationId, deliveryStreamArn, region, records:[{recordId, approximateArrivalTimestamp, data}]}`) — `Ok` records ship the transformed data downstream, `Dropped`/`ProcessingFailed` are omitted, Lambda failures pass records through unchanged (best-effort per AWS) || **Route53** | CreateHostedZone, GetHostedZone, DeleteHostedZone, ListHostedZones, ListHostedZonesByName, UpdateHostedZoneComment, ChangeResourceRecordSets (CREATE/UPSERT/DELETE), ListResourceRecordSets, GetChange, CreateHealthCheck, GetHealthCheck, DeleteHealthCheck, ListHealthChecks, UpdateHealthCheck, ChangeTagsForResource, ListTagsForResource | REST/XML protocol; SOA + NS records auto-created; CallerReference idempotency; alias records, weighted/failover/latency routing; marker-based pagination |-| **EC2** | RunInstances, DescribeInstances, DescribeInstanceAttribute, DescribeInstanceTypes, DescribeVpcAttribute, TerminateInstances, StopInstances, StartInstances, RebootInstances, AssociateIamInstanceProfile, DescribeIamInstanceProfileAssociations, DisassociateIamInstanceProfile, ReplaceIamInstanceProfileAssociation, DescribeImages, CreateSecurityGroup, DeleteSecurityGroup, DescribeSecurityGroups, AuthorizeSecurityGroupIngress, RevokeSecurityGroupIngress, AuthorizeSecurityGroupEgress, RevokeSecurityGroupEgress, DescribeSecurityGroupRules, CreateKeyPair, DeleteKeyPair, DescribeKeyPairs, ImportKeyPair, CreatePlacementGroup, DeletePlacementGroup, DescribePlacementGroups, CreateVpc, DeleteVpc, DescribeVpcs, ModifyVpcAttribute, CreateSubnet, DeleteSubnet, DescribeSubnets, ModifySubnetAttribute, CreateInternetGateway, DeleteInternetGateway, DescribeInternetGateways, AttachInternetGateway, DetachInternetGateway, CreateRouteTable, DeleteRouteTable, DescribeRouteTables, AssociateRouteTable, DisassociateRouteTable, ReplaceRouteTableAssociation, CreateRoute, ReplaceRoute, DeleteRoute, CreateNetworkInterface, DeleteNetworkInterface, DescribeNetworkInterfaces, AttachNetworkInterface, DetachNetworkInterface, CreateVpcEndpoint, DeleteVpcEndpoints, DescribeVpcEndpoints, ModifyVpcEndpoint, DescribePrefixLists, DescribeAvailabilityZones, AllocateAddress, ReleaseAddress, AssociateAddress, DisassociateAddress, DescribeAddresses, DescribeAddressesAttribute, CreateTags, DeleteTags, DescribeTags, CreateNatGateway, DescribeNatGateways, DeleteNatGateway, CreateNetworkAcl, DescribeNetworkAcls, DeleteNetworkAcl, CreateNetworkAclEntry, DeleteNetworkAclEntry, ReplaceNetworkAclEntry, ReplaceNetworkAclAssociation, CreateFlowLogs, DescribeFlowLogs, DeleteFlowLogs, CreateVpcPeeringConnection, AcceptVpcPeeringConnection, DescribeVpcPeeringConnections, DeleteVpcPeeringConnection, CreateDhcpOptions, AssociateDhcpOptions, DescribeDhcpOptions, DeleteDhcpOptions, CreateEgressOnlyInternetGateway, DescribeEgressOnlyInternetGateways, DeleteEgressOnlyInternetGateway, CreateManagedPrefixList, DescribeManagedPrefixLists, GetManagedPrefixListEntries, ModifyManagedPrefixList, DeleteManagedPrefixList, CreateVpnGateway, DescribeVpnGateways, AttachVpnGateway, DetachVpnGateway, DeleteVpnGateway, EnableVgwRoutePropagation, DisableVgwRoutePropagation, CreateCustomerGateway, DescribeCustomerGateways, DeleteCustomerGateway, DescribeInstanceCreditSpecifications, DescribeInstanceMaintenanceOptions, DescribeInstanceAutoRecoveryAttribute, ModifyInstanceMaintenanceOptions, DescribeInstanceTopology, DescribeSpotInstanceRequests, DescribeCapacityReservations, DescribeInstanceStatus, DescribeVpcClassicLink, DescribeVpcClassicLinkDnsSupport, CreateLaunchTemplate, CreateLaunchTemplateVersion, DescribeLaunchTemplates, DescribeLaunchTemplateVersions, ModifyLaunchTemplate, DeleteLaunchTemplate, CreateFleet, DescribeFleets | 145 actions; EC2 Fleet (`CreateFleet` / `DescribeFleets`) with `instant`-type synchronous launch and `maintain` / `request` async fulfillment, multi-config × multi-override round-robin capacity distribution, and `DefaultTargetCapacityType`-driven spot/on-demand selection — unblocks Karpenter / Cluster Autoscaler local validation; IAM instance profile association lifecycle is persisted (`Associate*`, `Describe*`, `Replace*`, `Disassociate*`) so Terraform `aws_instance.iam_instance_profile` round-trips without drift; `AuthorizeSecurityGroupIngress` is idempotent on duplicate rules (same behavior as egress; avoids Terraform re-apply failures); in-memory state only — no real VMs; CreateVpc provisions per-VPC default route table, network ACL, and security group; full Terraform VPC module v6.6.0 compatible; VPN/Customer gateways, managed prefix lists, VPC endpoints with modify support; launch templates with versioning ($Latest/$Default) |+| **EC2** | RunInstances, DescribeInstances, DescribeInstanceAttribute, DescribeInstanceTypes, DescribeVpcAttribute, TerminateInstances, StopInstances, StartInstances, RebootInstances, AssociateIamInstanceProfile, DescribeIamInstanceProfileAssociations, DisassociateIamInstanceProfile, ReplaceIamInstanceProfileAssociation, DescribeImages, RegisterImage, DeregisterImage, CreateSecurityGroup, DeleteSecurityGroup, DescribeSecurityGroups, AuthorizeSecurityGroupIngress, RevokeSecurityGroupIngress, AuthorizeSecurityGroupEgress, RevokeSecurityGroupEgress, DescribeSecurityGroupRules, CreateKeyPair, DeleteKeyPair, DescribeKeyPairs, ImportKeyPair, CreatePlacementGroup, DeletePlacementGroup, DescribePlacementGroups, CreateVpc, DeleteVpc, DescribeVpcs, ModifyVpcAttribute, CreateSubnet, DeleteSubnet, DescribeSubnets, ModifySubnetAttribute, CreateInternetGateway, DeleteInternetGateway, DescribeInternetGateways, AttachInternetGateway, DetachInternetGateway, CreateRouteTable, DeleteRouteTable, DescribeRouteTables, AssociateRouteTable, DisassociateRouteTable, ReplaceRouteTableAssociation, CreateRoute, ReplaceRoute, DeleteRoute, CreateNetworkInterface, DeleteNetworkInterface, DescribeNetworkInterfaces, AttachNetworkInterface, DetachNetworkInterface, CreateVpcEndpoint, DeleteVpcEndpoints, DescribeVpcEndpoints, ModifyVpcEndpoint, DescribePrefixLists, DescribeAvailabilityZones, AllocateAddress, ReleaseAddress, AssociateAddress, DisassociateAddress, DescribeAddresses, DescribeAddressesAttribute, CreateTags, DeleteTags, DescribeTags, CreateNatGateway, DescribeNatGateways, DeleteNatGateway, CreateNetworkAcl, DescribeNetworkAcls, DeleteNetworkAcl, CreateNetworkAclEntry, DeleteNetworkAclEntry, ReplaceNetworkAclEntry, ReplaceNetworkAclAssociation, CreateFlowLogs, DescribeFlowLogs, DeleteFlowLogs, CreateVpcPeeringConnection, AcceptVpcPeeringConnection, DescribeVpcPeeringConnections, DeleteVpcPeeringConnection, CreateDhcpOptions, AssociateDhcpOptions, DescribeDhcpOptions, DeleteDhcpOptions, CreateEgressOnlyInternetGateway, DescribeEgressOnlyInternetGateways, DeleteEgressOnlyInternetGateway, CreateManagedPrefixList, DescribeManagedPrefixLists, GetManagedPrefixListEntries, ModifyManagedPrefixList, DeleteManagedPrefixList, CreateVpnGateway, DescribeVpnGateways, AttachVpnGateway, DetachVpnGateway, DeleteVpnGateway, EnableVgwRoutePropagation, DisableVgwRoutePropagation, CreateCustomerGateway, DescribeCustomerGateways, DeleteCustomerGateway, DescribeInstanceCreditSpecifications, DescribeInstanceMaintenanceOptions, DescribeInstanceAutoRecoveryAttribute, ModifyInstanceMaintenanceOptions, DescribeInstanceTopology, DescribeSpotInstanceRequests, DescribeCapacityReservations, DescribeInstanceStatus, DescribeVpcClassicLink, DescribeVpcClassicLinkDnsSupport, CreateLaunchTemplate, CreateLaunchTemplateVersion, DescribeLaunchTemplates, DescribeLaunchTemplateVersions, ModifyLaunchTemplate, DeleteLaunchTemplate, CreateFleet, DescribeFleets | 147 actions; EC2 Fleet (`CreateFleet` / `DescribeFleets`) with `instant`-type synchronous launch and `maintain` / `request` async fulfillment, multi-config × multi-override round-robin capacity distribution, and `DefaultTargetCapacityType`-driven spot/on-demand selection — unblocks Karpenter / Cluster Autoscaler local validation; IAM instance profile association lifecycle is persisted (`Associate*`, `Describe*`, `Replace*`, `Disassociate*`) so Terraform `aws_instance.iam_instance_profile` round-trips without drift; `AuthorizeSecurityGroupIngress` is idempotent on duplicate rules (same behavior as egress; avoids Terraform re-apply failures); metadata-only instance records by default — `RegisterImage` with a container reference makes an AMI whose launches get a real box, so SSM Run Command returns a real exit code (see *Instances with a real box*); CreateVpc provisions per-VPC default route table, network ACL, and security group; full Terraform VPC module v6.6.0 compatible; VPN/Customer gateways, managed prefix lists, VPC endpoints with modify support; launch templates with versioning ($Latest/$Default) || **EBS** | CreateVolume, DeleteVolume, DescribeVolumes, DescribeVolumeStatus, AttachVolume, DetachVolume, ModifyVolume, DescribeVolumesModifications, EnableVolumeIO, ModifyVolumeAttribute, DescribeVolumeAttribute, CreateSnapshot, DeleteSnapshot, DescribeSnapshots, CopySnapshot, ModifySnapshotAttribute, DescribeSnapshotAttribute | Part of EC2 Query/XML service; attach/detach updates volume state; snapshots stored as completed immediately; Pro-only on LocalStack — free here || **EFS** | CreateFileSystem, DescribeFileSystems, DeleteFileSystem, UpdateFileSystem, CreateMountTarget, DescribeMountTargets, DeleteMountTarget, DescribeMountTargetSecurityGroups, ModifyMountTargetSecurityGroups, CreateAccessPoint, DescribeAccessPoints, DeleteAccessPoint, TagResource, UntagResource, ListTagsForResource, PutLifecycleConfiguration, DescribeLifecycleConfiguration, PutBackupPolicy, DescribeBackupPolicy, DescribeAccountPreferences, PutAccountPreferences | REST/JSON `/2015-02-01/*`; CreationToken idempotency; FileSystem deletion blocked when mount targets exist; Pro-only on LocalStack — free here || **EMR** | RunJobFlow, DescribeCluster, ListClusters, TerminateJobFlows, ModifyCluster, SetTerminationProtection, SetVisibleToAllUsers, AddJobFlowSteps, DescribeStep, ListSteps, CancelSteps, AddInstanceFleet, ListInstanceFleets, ModifyInstanceFleet, AddInstanceGroups, ListInstanceGroups, ModifyInstanceGroups, ListBootstrapActions, AddTags, RemoveTags, GetBlockPublicAccessConfiguration, PutBlockPublicAccessConfiguration | Control plane only — no real Spark/Hadoop; clusters start in WAITING (KeepAlive=true) or TERMINATED (KeepAlive=false); steps stored as COMPLETED immediately; all three instance modes (simple, InstanceGroups, InstanceFleets); TerminationProtected enforced; Pro-only on LocalStack — free here |@@ -979,6 +979,65 @@ boto3.client("s3", endpoint_url="http://infra_ministack:4566", ...)If `/var/run/docker.sock` is root-owned on the host, add `privileged: true` to the `ministack` service so it can talk to the daemon.+### Instances with a Real Box++`RunInstances` returns metadata by default — the right behaviour for CloudFormation and Terraform+stacks full of AMI ids that never existed. Register an AMI backed by a container reference and+launches against that id get a real box, so `ssm:SendCommand` can return a real exit code instead+of always reporting success.++```bash+# 1. Register any image your Docker or Podman daemon can pull as an AMI+aws --endpoint-url=http://localhost:4566 ec2 register-image \+ --name my-box --image-location ghcr.io/you/my-box:1.0+# { "ImageId": "ami-0a1b2c3d4e5f60718" }++# 2. Launch it+aws --endpoint-url=http://localhost:4566 ec2 run-instances \+ --image-id ami-0a1b2c3d4e5f60718 --count 1 --instance-type t3.micro++# 3. DescribeInstances reports the container's address, and the box answers on it+aws --endpoint-url=http://localhost:4566 ec2 describe-instances+```++No configuration: registering an image is the opt-in. With nothing registered, EC2 never reaches+for Docker at all, and an unregistered `ami-` id keeps launching the metadata-only record it always+did. Both kinds of instance coexist in one account and region.++- **Root device type:** a container has no persistent root disk, so an AMI registered from a+ container reference is `instance-store` backed — no EBS root volume, and `StopInstances` /+ `StartInstances` are refused with `UnsupportedOperation`, exactly as AWS refuses them for an+ instance store-backed instance. `RebootInstances` restarts the container, `TerminateInstances`+ removes it, and a container that dies out of band reports `terminated` on the next+ `DescribeInstances` — a lost root disk is a lost instance. `/_ministack/reset` and shutdown sweep+ this MiniStack's instance containers, leaving any other MiniStack sharing the daemon alone.+- **Snapshot-backed registration** works too, because on AWS `RegisterImage` is metadata-only and+ never checks that the image can boot: `import-snapshot` then `register-image` registers as normal+ and reports `ebs`. There is simply nothing here to run, so the failure lands on `RunInstances`+ with `InvalidAMIID.Unavailable` rather than at registration.+- **Addresses:** the container's IP replaces the synthetic private and public addresses and both DNS+ names, so the guest is reachable from the shared Docker network.+- **Failure:** a container that cannot be created or started fails the `RunInstances` call and backs+ the instance and its IAM profile association out. A running record with nothing behind it is+ exactly what this exists to prevent. An image reference that cannot be pulled answers+ `InvalidAMIID.Unavailable` rather than a server error, since retrying it would fail identically.+- **Guest contract:** the image is launched with init as PID 1 and a `sleep infinity` keepalive+ appended, so an image that would otherwise exit immediately stays up. An image with its own+ `ENTRYPOINT` receives that as argv and is free to ignore it. An image with no shell is treated the+ way AWS treats an instance with no SSM agent: unmanaged, and Run Command refuses it.+- **Registry rewrite:** registered references route through `MINISTACK_IMAGE_PREFIX` like every+ other pulled image, so mirrored and airgapped setups work unchanged.++**Divergences from AWS, stated plainly.** `ImageLocation` carries a container reference rather than+an S3 manifest path. A container is not a VM: there is no IMDS, no console output, no+security-group enforcement or port publishing, no `CreateImage` from a running instance, and an+attached EBS volume stays a record rather than appearing as a block device inside the guest.+Container filesystems do not survive an emulator restart — restored instances report `terminated`,+which is what AWS does when an instance store-backed instance loses its host. Terminated instances+are also dropped from `DescribeInstances` after 60 seconds; AWS says only that a terminated instance+"remains visible in the console for a short while", so a `wait instance-terminated` followed by a+describe can find nothing where AWS would still answer.+### EKS with Real Kubernetes (k3s)MiniStack's EKS spawns a real [k3s](https://k3s.io) cluster (75 MB image) when you create a cluster. `kubectl`, Helm, and any Kubernetes tooling work out of the box.diff --git a/ministack/services/ec2.py b/ministack/services/ec2.pyindex d6570f1c..b7f0c269 100644--- a/ministack/services/ec2.py+++ b/ministack/services/ec2.py@@ -71,10 +71,13 @@ import timefrom urllib.parse import parse_qsfrom xml.sax.saxutils import escape as _esc+from ministack.core import container_reaper+from ministack.core.concurrency import resource_lock, run_offloopfrom ministack.core.persistence import PERSIST_STATE, load_statefrom ministack.core.responses import (AccountRegionScopedDict,AccountScopedDict,+ apply_image_prefix,get_account_id,get_region,new_uuid,@@ -83,6 +86,11 @@ from ministack.core.responses import (logger = logging.getLogger("ec2")REGION = os.environ.get("MINISTACK_REGION", "us-east-1")+DOCKER_NETWORK = os.environ.get("DOCKER_NETWORK", "")++# Docker client, created on first use. Only registered images reach for it, so+# an emulator nobody registered an AMI with never imports docker-py at all.+_docker = None# ---------------------------------------------------------------------------# State@@ -115,6 +123,7 @@ _vpn_connections = AccountRegionScopedDict() # vpn_id -> VPN connection recor_launch_templates = AccountRegionScopedDict() # lt_id -> launch template record (includes versions list)_fleets = AccountRegionScopedDict() # fleet_id -> fleet record_iam_instance_profile_associations = AccountRegionScopedDict() # assoc_id -> association record+_images = AccountRegionScopedDict() # ami_id -> registered image record# Seed defaults once per account/region; do not recreate user-deleted defaults# on every later EC2 request in a scope._default_initialized_scopes = set()@@ -124,12 +133,22 @@ _default_initialized_scopes = set()# ── Persistence ────────────────────────────────────────────def get_state():+ instances = copy.deepcopy(_instances)+ # Container ids would dangle across a restart — the boot sweep reclaims this+ # MiniStack's containers (core/container_reaper.reap_all) long before anyone+ # reads them back. Keep ImageId so restore can relaunch from the registered+ # image; drop the handles.+ for inst in instances.all_values():+ inst.pop("_container_id", None)+ inst.pop("_container_name", None)+ inst.pop("_ssm_managed", None)return {"default_initialized_scopes": [{"AccountId": account_id, "Region": region}for account_id, region in sorted(_default_initialized_scopes)],- "instances": copy.deepcopy(_instances),+ "images": copy.deepcopy(_images),+ "instances": instances,"security_groups": copy.deepcopy(_security_groups),"key_pairs": copy.deepcopy(_key_pairs),"placement_groups": copy.deepcopy(_placement_groups),@@ -187,6 +206,7 @@ def _clear_state():_launch_templates.clear()_fleets.clear()_iam_instance_profile_associations.clear()+ _images.clear()_default_initialized_scopes.clear()@@ -295,8 +315,26 @@ def restore_state(data):_iam_instance_profile_associations,data.get("iam_instance_profile_associations", {}))+ _restore_regional_store(_images, data.get("images", {}))if "default_initialized_scopes" not in data:_infer_default_initialized_scopes_from_restored_stores()+ _reconcile_backed_instances_after_restore()+++def _reconcile_backed_instances_after_restore():+ """A restored container-backed instance has no container behind it: the boot+ sweep removed it. These are instance-store backed, so the lost root disk is+ a lost instance — report it terminated rather than claim a box that is gone+ or offer a start that could only ever produce a different machine."""+ for inst in _instances.all_values():+ if inst.get("VmManager") != "docker":+ continue+ if inst.get("State", {}).get("Name") in ("running", "stopped"):+ inst["State"] = {"Code": 48, "Name": "terminated"}+ inst["_terminated_at"] = time.time()+ inst.pop("_container_id", None)+ inst.pop("_container_name", None)+ inst.pop("_ssm_managed", None)def _restore_regional_store(store, restored):@@ -431,6 +469,15 @@ _ensure_defaults_initialized()# Request routing# ---------------------------------------------------------------------------+_DOCKER_ACTIONS = {+ "RunInstances", "StartInstances", "StopInstances", "RebootInstances",+ "TerminateInstances", "DescribeInstances",+ # Reaches Docker indirectly: its _cleanup_terminated() removes the+ # containers of terminated instances whose terminate-time removal failed.+ "DescribeIamInstanceProfileAssociations",+}++async def handle_request(method, path, headers, body, query_params):_ensure_defaults_initialized()params = dict(query_params)@@ -443,6 +490,15 @@ async def handle_request(method, path, headers, body, query_params):handler = _ACTION_MAP.get(action)if not handler:return _error("InvalidAction", f"Unknown EC2 action: {action}", 400)+ # Actions that reach the Docker daemon once an image is registered. They+ # block for as long as the daemon takes, so they go off the loop. The+ # containers run the registered image's own workload and nothing is wired+ # back to 4566, so this cannot re-enter: the shared pool is right, not+ # run_reentrant. (Running a command inside a box can re-enter — a health+ # probe calling back into MiniStack is the point — so SSM runs+ # exec_in_instance_blocking on a thread of its own, not on this pool.)+ if _docker_touched() and action in _DOCKER_ACTIONS:+ return await run_offloop(handler, params)return handler(params)@@ -717,6 +773,65 @@ def _run_instances(p):iam_profile=iam_profile)+ # A registered AMI boots each record as a real container. Unregistered ids+ # — including every invented ami- a CFN or Terraform stack carries — stay+ # metadata-only, so existing workloads are untouched.+ image = _registered_image(image_id)+ if image is not None and not image.get("ImageLocation"):+ # Registered from a snapshot, or from nothing at all.+ # Registered from a snapshot: AWS would boot the volume, and we have+ # nothing to run. The divergence belongs here rather than at+ # RegisterImage, which is metadata-only on AWS as well.+ for inst in created:+ _discard_instance_records(inst)+ return _error("InvalidAMIID.Unavailable",+ f"The image id '[{image_id}]' is not in a state from which "+ "you can launch an instance", 400)+ if image is not None:+ # A container has no persistent root disk, which is instance-store+ # semantics: no EBS root volume, and StopInstances is refused.+ for inst in created:+ for bdm in inst.get("BlockDeviceMappings", []):+ vol_id = bdm.get("Ebs", {}).get("VolumeId")+ if vol_id:+ _volumes.pop(vol_id, None)+ inst["BlockDeviceMappings"] = []+ inst["RootDeviceType"] = "instance-store"+ inst["RootDeviceName"] = image.get("RootDeviceName") or "/dev/sda1"+ try:+ for inst in created:+ container = _vm_launch(inst, image["ImageLocation"])+ # Publish the handle under the record's lock: handlers no longer+ # serialise on the event loop, so a TerminateInstances can land+ # between the launch and the write. Identity, not truthiness —+ # a same-id record cannot exist, but a popped one must not be+ # resurrected by this write.+ with resource_lock("ec2", inst["InstanceId"]):+ live = _instances.get(inst["InstanceId"])+ orphaned = live is not inst+ if orphaned:+ _drop_container(container)+ except Exception as e:+ # A running record with nothing behind it is the failure this+ # backend exists to remove: back everything out and surface it.+ logger.warning("EC2: could not boot %s: %s", image["ImageLocation"], e)+ for inst in created:+ _vm_remove_container(inst)+ _discard_instance_records(inst)+ if _is_image_unavailable(e):+ # The reference cannot be pulled, so this AMI cannot launch an+ # instance — a caller condition, not a server one. Retrying an+ # InternalError here would fail identically forever.+ return _error(+ "InvalidAMIID.Unavailable",+ f"The image id '[{image_id}]' is not in a state from which you "+ f"can launch an instance: {_esc(image['ImageLocation'])} could "+ f"not be pulled ({_esc(str(e))})", 400)+ return _error(+ "InternalError",+ f"failed to start instance from {_esc(image['ImageLocation'])}: "+ f"{_esc(str(e))}", 500)+# Process TagSpecificationsi = 1while _p(p, f"TagSpecification.{i}.ResourceType"):@@ -755,7 +870,9 @@ def _cleanup_terminated():if v["State"]["Name"] == "terminated"and now - v.get("_terminated_at", 0) > 60]for k in stale:- _instances.pop(k, None)+ inst = _instances.pop(k, None)+ if inst and inst.get("VmManager") == "docker":+ _vm_remove_container(inst)def _describe_instances(p):@@ -769,6 +886,13 @@ def _describe_instances(p):if iid not in _instances:return _error("InvalidInstanceID.NotFound", f"The instance ID '{iid}' does not exist", 400)+ if _docker_touched():+ # Before the filters run, so an instance-state-name filter sees the+ # reconciled state. DescribeInstances is dispatched off the loop+ # whenever an image is registered, so the container inspects cannot+ # stall the server.+ _vm_reconcile_running(list(_instances.values()))+results = []for inst in _instances.values():if filter_ids and inst["InstanceId"] not in filter_ids:@@ -973,6 +1097,8 @@ def _terminate_instances(p):prev = inst["State"].copy()inst["State"] = {"Code": 48, "Name": "terminated"}inst["_terminated_at"] = time.time()+ if inst.get("VmManager") == "docker":+ _vm_remove_container(inst)assoc = _find_active_iam_instance_profile_association(iid)if assoc:_mark_iam_instance_profile_association_disassociated(assoc)@@ -989,6 +1115,14 @@ def _stop_instances(p):for iid in ids:if iid not in _instances:return _error("InvalidInstanceID.NotFound", f"The instance ID '{iid}' does not exist", 400)+ for iid in ids:+ # AWS: "you can't stop an instance that's instance store-backed". A+ # container-backed instance has no persistent root disk, so there is+ # nothing a stop could preserve.+ if _instances[iid].get("RootDeviceType") == "instance-store":+ return _error("UnsupportedOperation",+ f"You can't stop the instance '{iid}' because it is an "+ "instance store-backed instance", 400)items = ""for iid in ids:inst = _instances.get(iid)@@ -1008,6 +1142,13 @@ def _start_instances(p):for iid in ids:if iid not in _instances:return _error("InvalidInstanceID.NotFound", f"The instance ID '{iid}' does not exist", 400)+ for iid in ids:+ # The counterpart of the stop refusal: an instance-store backed instance+ # has no root disk to bring back, so AWS has no start for it either.+ if _instances[iid].get("RootDeviceType") == "instance-store":+ return _error("UnsupportedOperation",+ f"You can't start the instance '{iid}' because it is an "+ "instance store-backed instance", 400)items = ""for iid in ids:inst = _instances.get(iid)@@ -1023,6 +1164,24 @@ def _start_instances(p):def _reboot_instances(p):+ ids = _parse_member_list(p, "InstanceId")+ for iid in ids:+ if iid not in _instances:+ return _error("InvalidInstanceID.NotFound",+ f"The instance ID '{iid}' does not exist", 400)+ for iid in ids:+ inst = _instances.get(iid)+ if inst and inst.get("VmManager") == "docker":+ container = _vm_get_container(inst)+ if container is not None:+ try:+ container.restart(timeout=10)+ # A container's IP can change across a restart.+ _vm_apply_container_ip(inst, container,+ _get_ministack_network(_get_docker()))+ inst["_ssm_managed"] = _probe_shell(container)+ except Exception as e:+ logger.warning("EC2: could not restart container for %s: %s", iid, e)return _xml(200, "RebootInstancesResponse", "<return>true</return>")@@ -1040,41 +1199,160 @@ _STUB_AMIS = []+def _image_view(image):+ """Registered image as the flat shape the filter and renderer share."""+ root_type = image.get("RootDeviceType") or ("instance-store" if image.get("ImageLocation") else "ebs")+ default_desc = (f"Container-backed AMI ({image['ImageLocation']})"+ if image.get("ImageLocation") else "Snapshot-backed AMI")+ return {+ "ImageId": image["ImageId"],+ "Name": image["Name"],+ "Description": image["Description"] or default_desc,+ "Platform": "",+ "RootDeviceType": root_type,+ "RootDeviceName": image["RootDeviceName"],+ "BlockDeviceMappings": image.get("BlockDeviceMappings") or [],+ "Architecture": image["Architecture"],+ "VirtualizationType": image["VirtualizationType"],+ "OwnerId": image.get("OwnerId") or get_account_id(),+ "IsPublic": "false",+ "Backed": True,+ }+++def _stub_image_view(ami_id, name, desc, platform, root_device):+ return {+ "ImageId": ami_id,+ "Name": name,+ "Description": desc,+ "Platform": platform,+ "RootDeviceType": "ebs",+ "RootDeviceName": root_device,+ "BlockDeviceMappings": [],+ "Architecture": "x86_64",+ "VirtualizationType": "hvm",+ "OwnerId": get_account_id(),+ "IsPublic": "true",+ "Backed": False,+ }+++def _image_matches_filters(view, filters):+ for name, values in filters.items():+ if name == "image-id":+ actual = [view["ImageId"]]+ elif name == "name":+ actual = [view["Name"]]+ elif name == "description":+ actual = [view["Description"]]+ elif name == "owner-id":+ actual = [view["OwnerId"]]+ elif name == "architecture":+ actual = [view["Architecture"]]+ elif name == "virtualization-type":+ actual = [view["VirtualizationType"]]+ elif name == "root-device-name":+ actual = [view["RootDeviceName"]]+ elif name == "root-device-type":+ actual = [view["RootDeviceType"]]+ elif name == "state":+ actual = ["available"]+ elif name == "is-public":+ actual = [view["IsPublic"]]+ elif name == "platform":+ actual = [view["Platform"]]+ elif name == "tag-key":+ actual = [t["Key"] for t in _tags.get(view["ImageId"], [])]+ elif name.startswith("tag:"):+ key = name[4:]+ actual = [t["Value"] for t in _tags.get(view["ImageId"], []) if t["Key"] == key]+ else:+ continue # unknown filter names are ignored, as elsewhere in EC2+ if not any(_filter_value_matches(a, values) for a in actual):+ return False+ return True+++def _filter_value_matches(actual, values):+ for v in values:+ if "*" in v or "?" in v:+ if re.fullmatch(re.escape(v).replace(r"\*", ".*").replace(r"\?", "."), actual or ""):+ return True+ elif actual == v:+ return True+ return False+++def _image_bdm_xml(view):+ """An instance-store AMI has no EBS root volume, so it emits no mapping —+ which is also what tells Terraform not to resolve a root device from it."""+ if view["RootDeviceType"] != "ebs":+ return ""+ mappings = view["BlockDeviceMappings"]+ if not mappings:+ # RootDeviceName + BlockDeviceMappings are required by Terraform's AWS+ # provider on aws_instance — it resolves them from DescribeImages before+ # RunInstances and fails with "finding Root Device Name for AMI" if absent.+ mappings = [{"DeviceName": view["RootDeviceName"],+ "Ebs": {"VolumeSize": "8", "VolumeType": "gp2",+ "DeleteOnTermination": "true"}}]+ items = ""+ for m in mappings:+ ebs = m.get("Ebs") or {}+ snapshot = (f"<snapshotId>{_esc(ebs['SnapshotId'])}</snapshotId>"+ if ebs.get("SnapshotId") else "")+ items += f"""<item>+ <deviceName>{_esc(m['DeviceName'])}</deviceName>+ <ebs>+ {snapshot}+ <volumeSize>{ebs.get('VolumeSize', '8')}</volumeSize>+ <volumeType>{ebs.get('VolumeType', 'gp2')}</volumeType>+ <deleteOnTermination>{ebs.get('DeleteOnTermination', 'true')}</deleteOnTermination>+ </ebs>+ </item>"""+ return f"<blockDeviceMapping>{items}</blockDeviceMapping>"++def _describe_images(p):filter_ids = _parse_member_list(p, "ImageId")+ filters = _parse_filters(p)+ # Registered images first: those are the ones that actually boot. The stubs+ # stay so a workload naming one keeps launching a metadata-only instance.+ views = [_image_view(img) for img in _images.values()]+ seen = {v["ImageId"] for v in views}+ views += [_stub_image_view(*stub) for stub in _STUB_AMIS if stub[0] not in seen]+items = ""- for ami_id, name, desc, platform, root_device in _STUB_AMIS:- if filter_ids and ami_id not in filter_ids:+ for view in views:+ if filter_ids and view["ImageId"] not in filter_ids:+ continue+ if filters and not _image_matches_filters(view, filters):continue# RootDeviceName + BlockDeviceMappings are required by Terraform's AWS# provider on aws_instance — it resolves them from DescribeImages before# RunInstances and fails with "finding Root Device Name for AMI" if absent.- platform_xml = f"<platform>{platform}</platform>" if platform else ""+ platform_xml = f"<platform>{view['Platform']}</platform>" if view["Platform"] else ""+ tag_items = "".join(+ f"<item><key>{_esc(t['Key'])}</key><value>{_esc(t.get('Value', ''))}</value></item>"+ for t in _tags.get(view["ImageId"], []))+ tag_xml = f"<tagSet>{tag_items}</tagSet>" if tag_items else ""items += f"""<item>- <imageId>{ami_id}</imageId>- <imageLocation>{name}</imageLocation>+ <imageId>{view['ImageId']}</imageId>+ <imageLocation>{_esc(view['Name'])}</imageLocation><imageState>available</imageState>- <imageOwnerId>{get_account_id()}</imageOwnerId>- <isPublic>true</isPublic>- <architecture>x86_64</architecture>+ <imageOwnerId>{view['OwnerId']}</imageOwnerId>+ <isPublic>{view['IsPublic']}</isPublic>+ <architecture>{view['Architecture']}</architecture><imageType>machine</imageType>- <name>{name}</name>- <description>{desc}</description>+ <name>{_esc(view['Name'])}</name>+ <description>{_esc(view['Description'])}</description>{platform_xml}- <rootDeviceType>ebs</rootDeviceType>- <rootDeviceName>{root_device}</rootDeviceName>- <blockDeviceMapping>- <item>- <deviceName>{root_device}</deviceName>- <ebs>- <volumeSize>8</volumeSize>- <volumeType>gp2</volumeType>- <deleteOnTermination>true</deleteOnTermination>- </ebs>- </item>- </blockDeviceMapping>- <virtualizationType>hvm</virtualizationType>+ <rootDeviceType>{view['RootDeviceType']}</rootDeviceType>+ <rootDeviceName>{view['RootDeviceName']}</rootDeviceName>+ {_image_bdm_xml(view)}+ <virtualizationType>{view['VirtualizationType']}</virtualizationType><hypervisor>xen</hypervisor>+ {tag_xml}</item>"""return _xml(200, "DescribeImagesResponse", f"<imagesSet>{items}</imagesSet>")@@ -1572,6 +1850,450 @@ def _import_key_pair(p):<keyPairId>{_key_pairs[name]['KeyPairId']}</keyPairId>""")+# ---------------------------------------------------------------------------+# Registered AMIs and container-backed instances+# ---------------------------------------------------------------------------+# RunInstances against a registered AMI boots that image as a container, so an+# instance has a real box behind it and ssm:SendCommand can return a real exit+# code instead of always reporting success. RegisterImage is the opt-in: with+# no image registered, EC2 never reaches for Docker and every instance is the+# metadata-only record it has always been.+#+# The container is not the emulated thing — it is what makes the emulated API+# behave correctly, the same role Postgres plays for RDS and k3s for EKS. It is+# private to this module: nothing schedules ECS tasks or EKS pods into it.+#+# Divergences, deliberately: ImageLocation carries a container reference rather+# than an S3 manifest path; there is no IMDS, no console output, no+# security-group enforcement or port publishing, no EBS-to-block-device+# mapping, and a container filesystem does not survive a restart.++# Same bound the reaper's client uses (app.py), so a hung daemon cannot pin a request for+# docker-py's 60s default. Not a new knob: MINISTACK_DOCKER_TIMEOUT already exists.+_DOCKER_TIMEOUT = float(os.environ.get("MINISTACK_DOCKER_TIMEOUT", "10"))+++def _new_image_id():+ return "ami-" + "".join(random.choices(string.hexdigits[:16], k=17))+++def _get_docker():+ """Docker client, or None. Imported lazily: an emulator with no registered+ image never pays for docker-py at all."""+ global _docker+ if _docker is None:+ try:+ import docker+ _docker = docker.from_env(timeout=_DOCKER_TIMEOUT)+ except Exception as e:+ logger.debug("EC2: no Docker client available: %s", e)+ return _docker+++def _get_ministack_network(client):+ """The Docker network MiniStack is on, so instances are reachable from it."""+ if DOCKER_NETWORK:+ return DOCKER_NETWORK+ try:+ hostname = os.environ.get("HOSTNAME", "")+ if not hostname:+ return None+ # Docker names the container's hostname after its id, Podman after its+ # name. containers.get() resolves either.+ self_container = client.containers.get(hostname)+ nets = list(self_container.attrs["NetworkSettings"]["Networks"].keys())+ return nets[0] if nets else None+ except Exception:+ return None+++def _registered_image(image_id):+ """The registered image record for an AMI id, or None."""+ return _images.get(image_id) if image_id else None+++# Set the first time a container is launched. A registration can be withdrawn while its+# instances are still running, so "an image is registered" is not the same question as+# "is there a container of ours out there".+_docker_in_use = False+++def _any_image_registered():+ """Whether an AMI backed by a container reference exists in any scope."""+ return bool(_images.all_values())+++def _docker_touched():+ """Whether anything here could need the daemon. Gates every Docker touch, including the+ describe-time reconcile and the reset sweep, so an emulator nobody registered an image+ with never imports docker-py at all."""+ return _docker_in_use or _any_image_registered()+++def _ec2_container_name(instance_id):+ return f"ministack-ec2-{get_account_id()}-{get_region()}-{instance_id}"+++# ── RegisterImage / DeregisterImage ────────────────────────++def _parse_register_block_device_mappings(p):+ """BlockDeviceMapping.N off a RegisterImage request."""+ mappings = []+ i = 1+ while _p(p, f"BlockDeviceMapping.{i}.DeviceName"):+ ebs = {}+ for field in ("SnapshotId", "VolumeSize", "VolumeType", "DeleteOnTermination",+ "Encrypted", "Iops", "Throughput"):+ value = _p(p, f"BlockDeviceMapping.{i}.Ebs.{field}")+ if value:+ ebs[field] = value+ mapping = {"DeviceName": _p(p, f"BlockDeviceMapping.{i}.DeviceName")}+ virtual_name = _p(p, f"BlockDeviceMapping.{i}.VirtualName")+ if virtual_name:+ mapping["VirtualName"] = virtual_name+ if ebs:+ mapping["Ebs"] = ebs+ mappings.append(mapping)+ i += 1+ return mappings+++# "3-128 alphanumeric characters, parentheses (()), square brackets ([]), spaces ( ),+# periods (.), slashes (/), dashes (-), single quotes ('), at-signs (@), or underscores(_)"+_AMI_NAME_RE = re.compile(r"^[A-Za-z0-9()\[\] ./\-'@_]{3,128}$")+++def _register_image(p):+ name = _p(p, "Name")+ if not name:+ return _error("MissingParameter", "The request must contain the parameter Name", 400)+ if not _AMI_NAME_RE.match(name):+ return _error("InvalidAMIName.Malformed",+ "AMI names must be between 3 and 128 characters long, and may only "+ "contain letters, numbers, and the following special characters: "+ "'-', '_', '.', '/', '(', and ')'", 400)+ location = _p(p, "ImageLocation")+ mappings = _parse_register_block_device_mappings(p)+ snapshot_backed = any(m.get("Ebs", {}).get("SnapshotId") for m in mappings)+ # Only Name is required on AWS: ImageLocation and BlockDeviceMapping.N are both optional, and+ # registration never checks that the result can boot. Refusing here would make MiniStack+ # stricter than AWS, so a registration with neither is accepted and fails at RunInstances.+ for existing in _images.values():+ if existing["Name"] == name:+ return _error("InvalidAMIName.Duplicate",+ f"AMI name {name} is already in use by another AMI", 400)+ image_id = _new_image_id()+ # Registration is metadata-only on AWS too: it neither touches the disk nor+ # checks that the image can boot. So a snapshot-backed registration is+ # accepted exactly as AWS accepts it (import-snapshot then register-image is+ # the ordinary Packer/Terraform flow) and the divergence lands where it+ # belongs — on RunInstances, the call that genuinely cannot do the thing.+ #+ # The two root device types stay honest. A container has no persistent root+ # disk, which is instance-store semantics: no EBS root volume and no stop.+ # A snapshot-backed AMI is `ebs`, and would keep its disk across a stop —+ # it just has nothing here to run.+ root_device_type = "instance-store" if location else "ebs"+ _images[image_id] = {+ "ImageId": image_id,+ "Name": name,+ "ImageLocation": location,+ "Description": _p(p, "Description") or "",+ # AWS documents these defaults for RegisterImage: Architecture "For Amazon EBS-backed+ # AMIs, i386", VirtualizationType "Default: paravirtual". A caller that cares passes them.+ "Architecture": _p(p, "Architecture") or "i386",+ "RootDeviceType": root_device_type,+ "RootDeviceName": (_p(p, "RootDeviceName")+ or ("/dev/sda1" if root_device_type == "instance-store"+ else "/dev/xvda")),+ "BlockDeviceMappings": mappings,+ "VirtualizationType": _p(p, "VirtualizationType") or "paravirtual",+ "CreationDate": _now_ts(),+ "OwnerId": get_account_id(),+ }+ _parse_tag_specs(p, "image", image_id)+ return _xml(200, "RegisterImageResponse", f"<imageId>{image_id}</imageId>")+++def _deregister_image(p):+ image_id = _p(p, "ImageId")+ if not image_id:+ return _error("MissingParameter", "The request must contain the parameter ImageId", 400)+ if image_id not in _images:+ return _error("InvalidAMIID.NotFound",+ f"The image id '[{image_id}]' does not exist", 400)+ _images.pop(image_id, None)+ _tags.pop(image_id, None)+ return _xml(200, "DeregisterImageResponse", "<return>true</return>")+++# ── Container lifecycle ────────────────────────────────────++def _vm_apply_container_ip(inst, container, network):+ """Replace the synthetic addresses with the container's real IP."""+ try:+ container.reload()+ nets = container.attrs.get("NetworkSettings", {}).get("Networks", {})+ if network and network in nets:+ ip = nets[network].get("IPAddress", "")+ else:+ ip = next(iter(nets.values())).get("IPAddress", "") if nets else ""+ if not ip:+ logger.warning("EC2: no container IP for %s; keeping synthetic addresses",+ inst["InstanceId"])+ return+ inst["PrivateIpAddress"] = ip+ inst["PublicIpAddress"] = ip+ inst["PrivateDnsName"] = f"ip-{ip.replace('.', '-')}.ec2.internal"+ inst["PublicDnsName"] = f"ec2-{ip.replace('.', '-')}.compute-1.amazonaws.com"+ except Exception as e:+ logger.warning("EC2: could not read container IP for %s: %s", inst["InstanceId"], e)+++def _probe_shell(container):+ """Whether the guest has a shell. No shell is our analogue of no SSM agent:+ the instance is unmanaged and SendCommand refuses it, as on AWS."""+ try:+ result = container.exec_run(["/bin/sh", "-c", "true"])+ return getattr(result, "exit_code", 1) == 0+ except Exception:+ return False+++def _is_image_unavailable(exc):+ """Whether a boot failure is "that reference cannot be pulled" rather than+ a daemon or runtime problem."""+ try:+ from docker import errors as docker_errors+ except Exception:+ return False+ if isinstance(exc, (docker_errors.ImageNotFound, docker_errors.NotFound)):+ return True+ # A registry that refuses or does not resolve surfaces as APIError.+ return (isinstance(exc, docker_errors.APIError)+ and any(t in str(exc).lower()+ for t in ("not found", "pull access denied", "manifest unknown",+ "no such image", "repository does not exist")))+++def _vm_launch(inst, image_ref):+ """Boot one instance record as a container. Raises on create/start failure;+ the caller backs the records out and surfaces the error."""+ client = _get_docker()+ if client is None:+ raise RuntimeError("no Docker daemon available")+ network = _get_ministack_network(client)+ name = _ec2_container_name(inst["InstanceId"])+ try:+ client.containers.get(name).remove(force=True)+ except Exception:+ pass # no stale container by that name, which is the normal case+ kwargs = {+ "name": name,+ "detach": True,+ "labels": {+ **container_reaper.own_labels("ec2"),+ "instance_id": inst["InstanceId"],+ "account_id": get_account_id(),+ "region": get_region(),+ },+ # A real instance does not exit the moment it boots, so keep the+ # container alive. An image with its own ENTRYPOINT receives this as+ # argv and is free to ignore it; init reaps the keepalive.+ "init": True,+ "command": ["sleep", "infinity"],+ }+ if network:+ kwargs["network"] = network+ container = client.containers.run(apply_image_prefix(image_ref), **kwargs)+ global _docker_in_use+ _docker_in_use = True+ inst["_container_id"] = container.id+ inst["_container_name"] = name+ inst["VmManager"] = "docker"+ _vm_apply_container_ip(inst, container, network)+ inst["_ssm_managed"] = _probe_shell(container)+ logger.info("EC2: instance %s running in container %s (image %s)",+ inst["InstanceId"], name, image_ref)+ return container+++def _vm_get_container(inst):+ client = _get_docker()+ if client is None:+ return None+ for ref in (inst.get("_container_id"), inst.get("_container_name"),+ _ec2_container_name(inst["InstanceId"])):+ if not ref:+ continue+ try:+ return client.containers.get(ref)+ except Exception:+ continue+ return None+++def _drop_container(container):+ """Tear down a container nothing references. Never called under a lock —+ Docker calls must not be held across resource_lock."""+ try:+ container.remove(force=True)+ except Exception as e:+ logger.warning("EC2: could not remove orphaned container: %s", e)+++def _vm_remove_container(inst):+ container = _vm_get_container(inst)+ if container is not None:+ try:+ container.remove(force=True)+ except Exception as e:+ logger.warning("EC2: could not remove container for %s: %s",+ inst["InstanceId"], e)+ inst.pop("_container_id", None)+ inst.pop("_container_name", None)+ inst.pop("_ssm_managed", None)+++def _discard_instance_records(inst):+ """Back out a failed launch: instance, its synthetic root volume, and its+ IAM instance profile association — exactly what _launch_instances_internal+ creates."""+ iid = inst["InstanceId"]+ for bdm in inst.get("BlockDeviceMappings", []):+ vol_id = bdm.get("Ebs", {}).get("VolumeId")+ if vol_id:+ _volumes.pop(vol_id, None)+ for key in [k for k, a in _iam_instance_profile_associations.items()+ if a.get("InstanceId") == iid]:+ _iam_instance_profile_associations.pop(key, None)+ _instances.pop(iid, None)+++def _vm_reconcile_running(instances):+ """Downgrade a "running" record whose container is gone or has exited.++ A box can die underneath its record — the workload exits, the daemon kills+ it, someone removes the container by hand. Reconciled lazily here rather+ than by a poller: the record is only wrong for as long as nobody looks.+ Downgrade only; StartInstances is the one path back to running.+ """+ client = _get_docker()+ if client is None:+ return+ try:+ client.ping()+ except Exception:+ return # an unreachable daemon says nothing about the containers+ from docker import errors as docker_errors+ for inst in instances:+ if inst.get("VmManager") != "docker":+ continue+ if inst.get("State", {}).get("Name") != "running":+ continue+ refs = [r for r in (inst.get("_container_id"), inst.get("_container_name")) if r]+ if not refs:+ continue # mid-launch: no container claim to judge yet+ container, verdict = None, "gone"+ for ref in refs:+ try:+ container = client.containers.get(ref)+ break+ except docker_errors.NotFound:+ continue+ except Exception as e:+ # Transient daemon trouble is not evidence the box died.+ logger.debug("EC2: container lookup for %s failed: %s", inst["InstanceId"], e)+ verdict = "unknown"+ break+ if verdict == "unknown":+ continue+ status = getattr(container, "status", "")+ if container is not None and status not in ("exited", "dead"):+ continue+ # Instance-store backed: losing the root disk is losing the instance.+ # AWS terminates such an instance rather than stopping it, and there is+ # nothing here a start could recover — the container filesystem is gone.+ inst["State"] = {"Code": 48, "Name": "terminated"}+ inst["_terminated_at"] = time.time()+ inst.pop("_ssm_managed", None)+ inst.pop("_container_id", None)+ inst.pop("_container_name", None)+ logger.info("EC2: instance %s container %s — reporting terminated",+ inst["InstanceId"], "gone" if container is None else status)+++def _vm_live_container_ids():+ """Container ids an instance record still owns, for the periodic reaper.++ A stopped instance keeps its exited container so StartInstances can restart+ it, so every recorded id is reported whatever the instance state.+ Terminated instances leave the store within a minute (_cleanup_terminated),+ after which a container a failed terminate left behind stops being vouched+ for and the reaper may reclaim it.+ """+ return {inst.get("_container_id")+ for inst in _instances.all_values() if inst.get("_container_id")}+++container_reaper.register_live_ids("ec2", _vm_live_container_ids)+++def _vm_sweep_containers():+ """Remove this MiniStack's instance containers. Used by reset.++ Scoped by ministack.instance like the rest of the reaping machinery:+ several MiniStacks can share a daemon and a reset of this one must not take+ down another's boxes.+ """+ if not _docker_touched():+ return+ client = _get_docker()+ if client is None:+ return+ try:+ containers = client.containers.list(all=True, filters={"label": [+ "ministack=ec2",+ f"{container_reaper.INSTANCE_LABEL}={container_reaper.instance_id()}",+ ]})+ except Exception as e:+ logger.warning("EC2: reset container sweep failed: %s", e)+ return+ container_reaper.drop_containers(containers, force=True)+++# ── Exec seam (SSM Run Command) ────────────────────────────++def instance_is_managed(instance_id):+ """Whether SSM can run a command on this instance.++ An instance is managed when it has a box with a shell behind it, which is+ this emulator's analogue of a running SSM agent. AWS refuses commands for+ an unmanaged instance and so do we.+ """+ inst = _instances.get(instance_id)+ if not inst or inst.get("State", {}).get("Name") != "running":+ return False+ return bool(inst.get("_ssm_managed"))+++def exec_in_instance_blocking(instance_id, argv):+ inst = _instances.get(instance_id)+ if not inst:+ return 255, "", "instance does not exist"+ container = _vm_get_container(inst)+ if container is None:+ return 255, "", "instance has no backing container"+ try:+ result = container.exec_run(argv, demux=True)+ except Exception as e:+ return 255, "", str(e)+ stdout, stderr = (result.output if isinstance(result.output, tuple) else (result.output, None))+ return (result.exit_code,+ (stdout or b"").decode("utf-8", "replace"),+ (stderr or b"").decode("utf-8", "replace"))++# ---------------------------------------------------------------------------# Placement Groups# ---------------------------------------------------------------------------@@ -4836,6 +5558,9 @@ def _delete_vpn_connection_route(p):# ---------------------------------------------------------------------------def reset():+ _vm_sweep_containers()+ global _docker_in_use+ _docker_in_use = False_clear_state()_init_defaults()@@ -6030,6 +6755,8 @@ _ACTION_MAP = {"DisassociateIamInstanceProfile": _disassociate_iam_instance_profile,"ReplaceIamInstanceProfileAssociation": _replace_iam_instance_profile_association,"DescribeImages": _describe_images,+ "RegisterImage": _register_image,+ "DeregisterImage": _deregister_image,"CreateSecurityGroup": _create_security_group,"DeleteSecurityGroup": _delete_security_group,"DescribeSecurityGroups": _describe_security_groups,diff --git a/ministack/services/ssm.py b/ministack/services/ssm.pyindex ef9b365c..b04ab637 100644--- a/ministack/services/ssm.py+++ b/ministack/services/ssm.py@@ -18,6 +18,7 @@ import timefrom datetime import datetime, timezonefrom ministack.core.arn import ArnParseError, parse_arn+from ministack.core.concurrency import spawn_backgroundfrom ministack.core.responses import (AccountRegionScopedDict,AccountScopedDict,@@ -700,26 +701,95 @@ def _param_out(param, with_decryption=False):# ── Run Command ────────────────────────────────────────────#-# Nothing here runs a script, so an invocation completes as Success with empty output rather than-# never converging. Only an unreachable instance fails, which is how a caller reaches its error path.+# An invocation is asynchronous, as on AWS: SendCommand answers Pending and the caller polls+# GetCommandInvocation until it reaches a terminal state. For AWS-RunShellScript against an+# instance with a box behind it, the script runs in that box and the invocation carries its real+# exit code and output, so a health probe can genuinely fail. A document we cannot run completes+# as Success with empty output rather than never converging._RUN_COMMAND_PLUGIN = "aws:runShellScript"+# GetCommandInvocation: "The first 24,000 characters written by the plugin to stdout" and+# "The first 8,000 characters written by the plugin to stderr".+_STDOUT_LIMIT = 24000+_STDERR_LIMIT = 8000+_SHELL_DOCUMENTS = {"AWS-RunShellScript"}def _managed_instance_ids() -> set:- """Instances Run Command can reach: the ones EC2 has running in this account and region.+ """Instances Run Command can reach.- Anything else has no agent answering, so real SSM neither lists it nor takes a command for it.+ An agent has to be answering, which AWS folds into InvalidInstanceId along with the node's+ state ("SSM Agent isn't running" is one of the causes it lists). A metadata-only instance has+ nothing running on it, so real SSM would refuse it too. `instance_is_managed` covers both:+ it requires the instance to be running and to have a box with a shell behind it."""from ministack.services import ec2 as ec2_svcreturn {instance_id- for instance_id, inst in ec2_svc._instances.items()- if (inst.get("State") or {}).get("Name") == "running"+ for instance_id in ec2_svc._instances+ if ec2_svc.instance_is_managed(instance_id)}+def _shell_argv(data) -> list:+ """The argv for an AWS-RunShellScript invocation, from its `commands` parameter."""+ commands = (data.get("Parameters") or {}).get("commands") or []+ if isinstance(commands, str):+ commands = [commands]+ script = "\n".join(str(c) for c in commands)+ return ["/bin/sh", "-c", script]+++def _finish_invocation(command_id, instance_id, argv):+ """Run one invocation to completion and publish its result.++ On its own thread: the script is the caller's, and a readiness probe calling back into+ MiniStack is the point of the feature, so it must not take a slot from the shared pool.+ """+ from ministack.services import ec2 as ec2_svc++ record = _commands.get(command_id)+ if not record:+ return+ invocation = (record.get("Invocations") or {}).get(instance_id)+ if invocation is None:+ return+ invocation["Status"] = "InProgress"+ invocation["StatusDetails"] = "In Progress"+ invocation["ExecutionStartDateTime"] = _now_iso()+ exit_code, stdout, stderr = ec2_svc.exec_in_instance_blocking(instance_id, argv)+ invocation["ResponseCode"] = exit_code+ invocation["StandardOutputContent"] = stdout[:_STDOUT_LIMIT]+ invocation["StandardErrorContent"] = stderr[:_STDERR_LIMIT]+ invocation["ExecutionEndDateTime"] = _now_iso()+ # Status flips last: a poller that sees a terminal state must already see the exit code,+ # the output and the end time.+ # "For a plugin, this indicates that the result code wasn't zero."+ invocation["StatusDetails"] = "Success" if exit_code == 0 else "Failed"+ invocation["Status"] = invocation["StatusDetails"]+ _refresh_command_status(record)+++def _refresh_command_status(record) -> None:+ """Roll the invocations up into the parent command."""+ invocations = list((record.get("Invocations") or {}).values())+ terminal = [i for i in invocations+ if i["Status"] in ("Success", "Failed", "Cancelled", "TimedOut")]+ failed = [i for i in invocations if i["Status"] == "Failed"]+ record["CompletedCount"] = len(terminal)+ record["ErrorCount"] = len(failed)+ if len(terminal) < len(invocations):+ record["Status"] = "InProgress"+ record["StatusDetails"] = "In Progress"+ elif failed:+ record["Status"] = "Failed"+ record["StatusDetails"] = "Failed"+ else:+ record["Status"] = "Success"+ record["StatusDetails"] = "Success"++def _command_target_ids(data) -> list:"""The instances a SendCommand addresses: InstanceIds, or a Targets entry naming them."""instance_ids = list(data.get("InstanceIds") or [])@@ -767,15 +837,17 @@ def _send_command(data):"InstanceIds": instance_ids,"Targets": data.get("Targets", []),"RequestedDateTime": now,- "Status": "Success",- "StatusDetails": "Success",+ # SendCommand answers before the command has been delivered — the doc's own sample+ # response is Pending. The caller polls GetCommandInvocation from here.+ "Status": "Pending",+ "StatusDetails": "Pending","OutputS3Region": get_region(),"OutputS3BucketName": data.get("OutputS3BucketName", ""),"OutputS3KeyPrefix": data.get("OutputS3KeyPrefix", ""),"MaxConcurrency": data.get("MaxConcurrency", "50"),"MaxErrors": data.get("MaxErrors", "0"),"TargetCount": len(instance_ids),- "CompletedCount": len(instance_ids),+ "CompletedCount": 0,"ErrorCount": 0,"DeliveryTimedOutCount": 0,"ServiceRole": data.get("ServiceRoleArn", ""),@@ -788,13 +860,16 @@ def _send_command(data):"DocumentName": document,"DocumentVersion": data.get("DocumentVersion", "$DEFAULT"),"PluginName": _RUN_COMMAND_PLUGIN,- "ResponseCode": 0,- # These three are strings in the SSM model, not timestamps.- "ExecutionStartDateTime": started,+ # "If the response code is -1, then the command hasn't started running on the+ # managed node, or it wasn't received by the node."+ "ResponseCode": -1,+ # These three are strings in the SSM model, not timestamps, and are empty until+ # the plugin starts.+ "ExecutionStartDateTime": "","ExecutionElapsedTime": "PT0.001S",- "ExecutionEndDateTime": started,- "Status": "Success",- "StatusDetails": "Success",+ "ExecutionEndDateTime": "",+ "Status": "Pending",+ "StatusDetails": "Pending","StandardOutputContent": "","StandardOutputUrl": "","StandardErrorContent": "",@@ -804,6 +879,21 @@ def _send_command(data):},}_commands[command_id] = record+ if document in _SHELL_DOCUMENTS:+ argv = _shell_argv(data)+ for instance_id in instance_ids:+ spawn_background(_finish_invocation, command_id, instance_id, argv,+ thread_name=f"ministack-ssm-{command_id[:8]}")+ else:+ # A document we cannot run still has to converge, or a caller polling for a terminal+ # state never returns.+ for instance_id in instance_ids:+ invocation = record["Invocations"][instance_id]+ invocation.update({"ResponseCode": 0, "Status": "Success",+ "StatusDetails": "Success",+ "ExecutionStartDateTime": started,+ "ExecutionEndDateTime": started})+ _refresh_command_status(record)return json_response({"Command": _public_command(record)})diff --git a/tests/test_ec2.py b/tests/test_ec2.pyindex bcd215bd..08bc9b96 100644--- a/tests/test_ec2.py+++ b/tests/test_ec2.py@@ -3603,3 +3603,530 @@ def test_ec2_allocated_address_is_a_valid_ipv4(ec2):ipaddress.ip_address(described["PublicIp"])finally:ec2.release_address(AllocationId=alloc["AllocationId"])+++# ---------------------------------------------------------------------------+# Registered AMIs and container-backed instances+# ---------------------------------------------------------------------------+# Unit tier: in-process against a faked Docker client, so it runs with no+# daemon. Live tier: gated on a reachable daemon, probed rather than read from+# an env var CI never sets.++import ministack.services.ec2 as ec2mod+++def _docker_reachable():+ try:+ import docker as _probe++ _probe.from_env(timeout=5).ping()+ return True+ except Exception:+ return False+++requires_docker = pytest.mark.skipif(+ not _docker_reachable(), reason="no reachable Docker daemon")+++class _FakeContainer:+ def __init__(self, name, kwargs, registry, ip="172.30.0.9", network="testnet",+ shell=True):+ self.id = f"cid-{name}"+ self.name = name+ self.kwargs = kwargs+ self.status = "running"+ self.attrs = {"NetworkSettings": {"Networks": {network: {"IPAddress": ip}}}}+ self.execs = []+ self.stop_calls = self.start_calls = self.restart_calls = self.remove_calls = 0+ self._registry = registry+ self._shell = shell++ def reload(self):+ pass++ def exec_run(self, cmd, demux=False, **kw):+ self.execs.append(cmd)+ if not self._shell:+ raise RuntimeError("no shell in image")+ out = (b"out", b"err") if demux else b"out"++ class _R:+ exit_code = 0+ output = out+ return _R()++ def start(self):+ self.start_calls += 1+ self.status = "running"++ def stop(self, timeout=10):+ self.stop_calls += 1+ self.status = "exited"++ def restart(self, timeout=10):+ self.restart_calls += 1++ def remove(self, force=False, v=False):+ self.remove_calls += 1+ self._registry.pop(self.id, None)+ self._registry.pop(self.name, None)+++def _fake_docker(run_error=None, shell=True):+ registry, runs = {}, []++ class _Containers:+ def run(self, image, **kwargs):+ if run_error is not None:+ raise run_error+ runs.append({"image": image, **kwargs})+ c = _FakeContainer(kwargs["name"], kwargs, registry, shell=shell)+ registry[c.id] = registry[c.name] = c+ return c++ def get(self, ref):+ if ref not in registry:+ from docker import errors+ raise errors.NotFound("absent")+ return registry[ref]++ def list(self, all=False, filters=None):+ want = (filters or {}).get("label") or []+ want = [want] if isinstance(want, str) else list(want)+ out = []+ for c in set(registry.values()):+ have = {f"{k}={v}" for k, v in c.kwargs.get("labels", {}).items()}+ if not [w for w in want if w not in have]:+ out.append(c)+ return out++ class _Docker:+ def __init__(self):+ self.containers = _Containers()++ def ping(self):+ return True++ d = _Docker()+ d._registry, d._runs = registry, runs+ return d+++def vm(monkeypatch):+ """A faked Docker daemon wired into the ec2 module, with state torn down."""+ fake = _fake_docker()+ monkeypatch.setattr(ec2mod, "_get_docker", lambda: fake)+ monkeypatch.setattr(ec2mod, "_get_ministack_network", lambda _c: "testnet")+ created = []+ yield fake, created+ for iid in created:+ ec2mod._instances.pop(iid, None)+ for key in list(ec2mod._images.keys()):+ ec2mod._images.pop(key, None)+++def _register(location="example/box:1", name="box"):+ resp = ec2mod._register_image({"Name": [name], "ImageLocation": [location]})+ import re as _re+ return _re.search(r"<imageId>(ami-[0-9a-f]+)</imageId>", resp[2].decode()).group(1)+++def _launch(ami, count=1):+ resp = ec2mod._run_instances({"ImageId": [ami], "MinCount": [str(count)],+ "MaxCount": [str(count)]})+ import re as _re+ body = resp[2].decode()+ return _re.findall(r"<instanceId>(i-[0-9a-f]+)</instanceId>", body), resp+++def test_ec2_register_image_round_trips(ec2):+ reg = ec2.register_image(Name=f"box-{_uuid_mod.uuid4().hex[:8]}",+ ImageLocation="alpine:3", Description="desc",+ Architecture="x86_64")+ ami = reg["ImageId"]+ try:+ assert ami.startswith("ami-") and len(ami) == len("ami-") + 17+ img = ec2.describe_images(ImageIds=[ami])["Images"][0]+ assert img["State"] == "available"+ # A container has no persistent root disk: instance-store semantics,+ # so no EBS root mapping either.+ assert img["RootDeviceType"] == "instance-store"+ assert img["RootDeviceName"] == "/dev/sda1"+ assert img.get("BlockDeviceMappings", []) == []+ assert img["Description"] == "desc"+ assert img["Public"] is False+ finally:+ ec2.deregister_image(ImageId=ami)+ with pytest.raises(ClientError) as exc:+ ec2.deregister_image(ImageId=ami)+ assert exc.value.response["Error"]["Code"] == "InvalidAMIID.NotFound"+++def test_ec2_register_image_accepts_a_bare_name(ec2):+ # Only Name is required on AWS — ImageLocation and BlockDeviceMapping.N are both optional and+ # registration never checks the result can boot. Refusing here would be stricter than AWS,+ # which is the failure mode that bites: code that works against AWS breaking locally.+ ami = ec2.register_image(Name=f"bare-{_uuid_mod.uuid4().hex[:8]}")["ImageId"]+ try:+ img = ec2.describe_images(ImageIds=[ami])["Images"][0]+ assert img["RootDeviceType"] == "ebs"+ # Documented RegisterImage defaults.+ assert img["Architecture"] == "i386"+ assert img["VirtualizationType"] == "paravirtual"+ # Nothing to boot, so the failure lands on the call that cannot do it.+ with pytest.raises(ClientError) as exc:+ ec2.run_instances(ImageId=ami, MinCount=1, MaxCount=1)+ assert exc.value.response["Error"]["Code"] == "InvalidAMIID.Unavailable"+ finally:+ ec2.deregister_image(ImageId=ami)+++def test_ec2_register_image_rejects_a_malformed_name(ec2):+ # "AMI names must be between 3 and 128 characters long, and may only contain letters,+ # numbers, and the following special characters..."+ for bad in ("a" * 129, "no,commas", "semi;colon"):+ with pytest.raises(ClientError) as exc:+ ec2.register_image(Name=bad, ImageLocation="alpine:3")+ assert exc.value.response["Error"]["Code"] == "InvalidAMIName.Malformed", bad+++def test_ec2_describe_images_filters_registered_and_stubs(ec2):+ unique = _uuid_mod.uuid4().hex[:8]+ ami = ec2.register_image(Name=f"filtered-{unique}", ImageLocation="alpine:3")["ImageId"]+ try:+ by_name = ec2.describe_images(+ Filters=[{"Name": "name", "Values": [f"filtered-{unique}"]}])["Images"]+ assert [i["ImageId"] for i in by_name] == [ami]+ wildcard = ec2.describe_images(+ Filters=[{"Name": "name", "Values": ["filtered-*"]}])["Images"]+ assert ami in [i["ImageId"] for i in wildcard]+ assert ec2.describe_images(+ Filters=[{"Name": "name", "Values": ["nothing-matches"]}])["Images"] == []+ # The stub AMIs are still listed, so a workload naming one still works.+ assert len(ec2.describe_images()["Images"]) >= 4+ finally:+ ec2.deregister_image(ImageId=ami)+++def test_ec2_unregistered_ami_never_touches_docker(monkeypatch):+ def _boom():+ raise AssertionError("Docker must not be reached with no image registered")++ monkeypatch.setattr(ec2mod, "_get_docker", _boom)+ iids, _ = _launch("ami-00000000")+ try:+ inst = ec2mod._instances[iids[0]]+ assert "VmManager" not in inst+ assert "_container_id" not in inst+ finally:+ for iid in iids:+ ec2mod._instances.pop(iid, None)+++def test_ec2_registered_ami_boots_a_container(vm):+ fake, created = vm+ ami = _register("example/box:1")+ iids, _ = _launch(ami)+ created.extend(iids)+ assert len(fake._runs) == 1+ run = fake._runs[0]+ assert run["image"] == "example/box:1"+ assert run["init"] is True+ # A real instance does not exit the moment it boots.+ assert run["command"] == ["sleep", "infinity"]+ assert run["network"] == "testnet"+ assert run["labels"]["ministack"] == "ec2"+ assert run["labels"]["instance_id"] == iids[0]+ inst = ec2mod._instances[iids[0]]+ assert inst["VmManager"] == "docker"+ assert inst["PrivateIpAddress"] == "172.30.0.9"+ assert inst["PublicIpAddress"] == "172.30.0.9"+ assert inst["PrivateDnsName"] == "ip-172-30-0-9.ec2.internal"+++def test_ec2_failed_boot_backs_every_record_out(monkeypatch):+ fake = _fake_docker(run_error=RuntimeError("no such image"))+ monkeypatch.setattr(ec2mod, "_get_docker", lambda: fake)+ monkeypatch.setattr(ec2mod, "_get_ministack_network", lambda _c: "testnet")+ ami = _register("example/missing:1")+ try:+ volumes_before = len(list(ec2mod._volumes.keys()))+ instances_before = len(list(ec2mod._instances.keys()))+ resp = ec2mod._run_instances({"ImageId": [ami], "MinCount": ["2"], "MaxCount": ["2"]})+ assert resp[0] == 500+ assert b"InternalError" in resp[2]+ # No phantom instance, and no orphaned synthetic root volume.+ assert len(list(ec2mod._instances.keys())) == instances_before+ assert len(list(ec2mod._volumes.keys())) == volumes_before+ finally:+ ec2mod._images.pop(ami, None)+++def test_ec2_dead_container_reports_terminated(vm):+ fake, created = vm+ ami = _register("example/box:1")+ iids, _ = _launch(ami)+ created.extend(iids)+ inst = ec2mod._instances[iids[0]]+ fake._registry[inst["_container_id"]].status = "exited"+ ec2mod._vm_reconcile_running([inst])+ # Instance-store backed: a lost root disk is a lost instance, so AWS+ # terminates rather than stops.+ assert inst["State"]["Name"] == "terminated"+ assert "_container_id" not in inst+++def test_ec2_container_backed_instance_cannot_stop_or_start(vm):+ _fake, created = vm+ ami = _register("example/box:1")+ iids, _ = _launch(ami)+ created.extend(iids)+ for handler in (ec2mod._stop_instances, ec2mod._start_instances):+ resp = handler({"InstanceId.1": [iids[0]]})+ assert resp[0] == 400+ assert b"UnsupportedOperation" in resp[2]+ assert b"instance store-backed" in resp[2]+++def test_ec2_container_backed_instance_is_instance_store(vm):+ _fake, created = vm+ ami = _register("example/box:1")+ iids, _ = _launch(ami)+ created.extend(iids)+ inst = ec2mod._instances[iids[0]]+ # No EBS root volume exists for an instance-store instance.+ assert inst["RootDeviceType"] == "instance-store"+ assert inst["BlockDeviceMappings"] == []+++def test_ec2_snapshot_backed_registration_is_accepted(ec2):+ # Registration is metadata-only on AWS: it does not check the image can+ # boot, so neither do we. import-snapshot then register-image is the+ # ordinary Packer/Terraform flow.+ snap = ec2.create_snapshot(VolumeId=ec2.create_volume(+ AvailabilityZone="us-east-1a", Size=8)["VolumeId"])["SnapshotId"]+ ami = ec2.register_image(+ Name=f"snap-{_uuid_mod.uuid4().hex[:8]}",+ RootDeviceName="/dev/xvda",+ BlockDeviceMappings=[{"DeviceName": "/dev/xvda",+ "Ebs": {"SnapshotId": snap, "VolumeSize": 8}}])["ImageId"]+ try:+ img = ec2.describe_images(ImageIds=[ami])["Images"][0]+ assert img["RootDeviceType"] == "ebs"+ assert img["BlockDeviceMappings"][0]["Ebs"]["SnapshotId"] == snap+ # The divergence lands on the call that genuinely cannot do the thing.+ with pytest.raises(ClientError) as exc:+ ec2.run_instances(ImageId=ami, MinCount=1, MaxCount=1)+ assert exc.value.response["Error"]["Code"] == "InvalidAMIID.Unavailable"+ assert not [r for r in ec2.describe_instances(+ Filters=[{"Name": "image-id", "Values": [ami]}])["Reservations"]]+ finally:+ ec2.deregister_image(ImageId=ami)+++def test_ec2_terminate_removes_the_container(vm):+ fake, created = vm+ ami = _register("example/box:1")+ iids, _ = _launch(ami)+ created.extend(iids)+ cid = ec2mod._instances[iids[0]]["_container_id"]+ ec2mod._terminate_instances({"InstanceId.1": [iids[0]]})+ assert cid not in fake._registry+++def test_ec2_state_round_trip_reports_restored_boxes_stopped(vm):+ fake, created = vm+ ami = _register("example/box:1")+ iids, _ = _launch(ami)+ created.extend(iids)+ state = ec2mod.get_state()+ saved = state["instances"].get(iids[0])+ # Container handles would dangle after a restart; the image id must survive+ # so StartInstances can relaunch.+ assert "_container_id" not in saved+ assert saved["VmManager"] == "docker"+ assert saved["ImageId"] == ami+ ec2mod._reconcile_backed_instances_after_restore()+ assert ec2mod._instances[iids[0]]["State"]["Name"] == "terminated"+++def test_ec2_instance_without_a_shell_is_unmanaged(monkeypatch):+ fake = _fake_docker(shell=False)+ monkeypatch.setattr(ec2mod, "_get_docker", lambda: fake)+ monkeypatch.setattr(ec2mod, "_get_ministack_network", lambda _c: "testnet")+ ami = _register("example/distroless:1")+ iids, _ = _launch(ami)+ try:+ # No shell is this emulator's analogue of no SSM agent: AWS refuses+ # commands for an unmanaged instance and so do we.+ assert ec2mod.instance_is_managed(iids[0]) is False+ finally:+ for iid in iids:+ ec2mod._instances.pop(iid, None)+ ec2mod._images.pop(ami, None)+++def test_ec2_managed_instance_execs(vm):+ fake, created = vm+ ami = _register("example/box:1")+ iids, _ = _launch(ami)+ created.extend(iids)+ assert ec2mod.instance_is_managed(iids[0]) is True+ code, out, err = ec2mod.exec_in_instance_blocking(iids[0], ["/bin/sh", "-c", "true"])+ assert (code, out, err) == (0, "out", "err")+++def test_ec2_exec_on_instance_without_a_box_fails(vm):+ _fake, created = vm+ iids, _ = _launch("ami-00000000")+ created.extend(iids)+ code, _out, err = ec2mod.exec_in_instance_blocking(iids[0], ["/bin/sh", "-c", "true"])+ assert code == 255+ assert "no backing container" in err+++@requires_docker+def test_ec2_docker_end_to_end_live(ec2):+ ami = ec2.register_image(Name=f"live-{_uuid_mod.uuid4().hex[:8]}",+ ImageLocation="alpine:3")["ImageId"]+ iid = None+ try:+ iid = ec2.run_instances(ImageId=ami, MinCount=1,+ MaxCount=1)["Instances"][0]["InstanceId"]+ inst = ec2.describe_instances(+ InstanceIds=[iid])["Reservations"][0]["Instances"][0]+ assert inst["State"]["Name"] == "running"+ # The reported address is the container's, not a synthetic 10.x.+ assert not inst["PrivateIpAddress"].startswith("10.0.")+ import docker+ client = docker.from_env(timeout=10)+ assert [c for c in client.containers.list() if iid in c.name]+ ec2.terminate_instances(InstanceIds=[iid])+ iid = None+ time.sleep(1)+ assert not [c for c in client.containers.list(all=True) if ami in c.name]+ finally:+ if iid:+ ec2.terminate_instances(InstanceIds=[iid])+ ec2.deregister_image(ImageId=ami)+++def test_ec2_ami_names_are_unique_per_account_and_region(ec2):+ """AMI names are unique per account per region on AWS, not globally."""+ name = f"shared-{_uuid_mod.uuid4().hex[:8]}"+ ami = ec2.register_image(Name=name, ImageLocation="alpine:3")["ImageId"]+ other_account = _ec2_client("us-east-1", account_id="123456789012")+ other_region = _ec2_client("eu-west-1")+ ami_other_account = ami_other_region = None+ try:+ with pytest.raises(ClientError) as exc:+ ec2.register_image(Name=name, ImageLocation="alpine:3")+ assert exc.value.response["Error"]["Code"] == "InvalidAMIName.Duplicate"+ # The same name is free in another account and in another region.+ ami_other_account = other_account.register_image(+ Name=name, ImageLocation="alpine:3")["ImageId"]+ ami_other_region = other_region.register_image(+ Name=name, ImageLocation="alpine:3")["ImageId"]+ assert len({ami, ami_other_account, ami_other_region}) == 3+ # And they do not leak into each other's DescribeImages.+ assert ami not in [i["ImageId"] for i in other_account.describe_images(+ Filters=[{"Name": "name", "Values": [name]}])["Images"]]+ finally:+ ec2.deregister_image(ImageId=ami)+ if ami_other_account:+ other_account.deregister_image(ImageId=ami_other_account)+ if ami_other_region:+ other_region.deregister_image(ImageId=ami_other_region)+++def test_ec2_name_is_free_again_after_deregister(ec2):+ name = f"recycled-{_uuid_mod.uuid4().hex[:8]}"+ first = ec2.register_image(Name=name, ImageLocation="alpine:3")["ImageId"]+ ec2.deregister_image(ImageId=first)+ second = ec2.register_image(Name=name, ImageLocation="alpine:3")["ImageId"]+ try:+ assert second != first+ assert [i["ImageId"] for i in ec2.describe_images(+ Filters=[{"Name": "name", "Values": [name]}])["Images"]] == [second]+ finally:+ ec2.deregister_image(ImageId=second)+++def test_ec2_describe_images_filters_apply_to_the_stubs(ec2):+ """The stubs now flow through the same filter path as registered images."""+ ami = ec2.register_image(Name=f"lin-{_uuid_mod.uuid4().hex[:8]}",+ ImageLocation="alpine:3")["ImageId"]+ try:+ windows = ec2.describe_images(+ Filters=[{"Name": "platform", "Values": ["windows"]}])["Images"]+ # Only the Windows stub carries a platform; a registered image has none.+ assert [i["ImageId"] for i in windows] == ["ami-0fedcba9876543210"]+ assert ami not in [i["ImageId"] for i in windows]++ public = ec2.describe_images(+ Filters=[{"Name": "is-public", "Values": ["true"]}])["Images"]+ assert {i["ImageId"] for i in public} == {+ "ami-0abcdef1234567890", "ami-0123456789abcdef0", "ami-0fedcba9876543210"}++ private = ec2.describe_images(+ Filters=[{"Name": "is-public", "Values": ["false"]}])["Images"]+ assert ami in [i["ImageId"] for i in private]++ # root-device-type separates the two kinds.+ ebs = [i["ImageId"] for i in ec2.describe_images(+ Filters=[{"Name": "root-device-type", "Values": ["ebs"]}])["Images"]]+ assert ami not in ebs and "ami-0abcdef1234567890" in ebs+ finally:+ ec2.deregister_image(ImageId=ami)+++def test_ec2_container_instance_creates_no_root_volume(vm):+ """Instance-store means no synthetic root volume, and nothing may quietly+ add one back — it would contradict the stop/start refusal."""+ _fake, created = vm+ before = {v["VolumeId"] for v in ec2mod._volumes.values()}+ ami = _register("example/box:1")+ iids, _ = _launch(ami)+ created.extend(iids)+ after = {v["VolumeId"] for v in ec2mod._volumes.values()}+ assert after == before+ inst = ec2mod._instances[iids[0]]+ assert inst["BlockDeviceMappings"] == []+ assert inst["RootDeviceType"] == "instance-store"+ assert not [v for v in ec2mod._volumes.values()+ for a in v.get("Attachments", []) if a.get("InstanceId") == iids[0]]+++def test_ec2_unpullable_image_is_a_client_error(ec2):+ """A reference that cannot be pulled is the caller's problem, not the+ server's — retrying an InternalError here would fail identically forever."""+ ami = ec2.register_image(Name=f"nopull-{_uuid_mod.uuid4().hex[:8]}",+ ImageLocation="ministack-nonexistent/nope:404")["ImageId"]+ try:+ with pytest.raises(ClientError) as exc:+ ec2.run_instances(ImageId=ami, MinCount=1, MaxCount=1)+ err = exc.value.response["Error"]+ assert err["Code"] == "InvalidAMIID.Unavailable"+ assert "could not be pulled" in err["Message"]+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 400+ finally:+ ec2.deregister_image(ImageId=ami)+++def test_ec2_deregister_does_not_orphan_a_running_box(vm):+ """A registration can be withdrawn while its instances are still running, so the+ describe-time reconcile must not be gated on the registry still holding the image."""+ fake, created = vm+ ami = _register("example/box:1")+ iids, _ = _launch(ami)+ created.extend(iids)+ ec2mod._deregister_image({"ImageId": [ami]})+ assert not ec2mod._any_image_registered()+ # Still true, because a container of ours is out there.+ assert ec2mod._docker_touched()+ inst = ec2mod._instances[iids[0]]+ fake._registry[inst["_container_id"]].status = "exited"+ ec2mod._vm_reconcile_running([inst])+ assert inst["State"]["Name"] == "terminated"diff --git a/tests/test_ssm.py b/tests/test_ssm.pyindex c0aafab3..21040ac8 100644--- a/tests/test_ssm.py+++ b/tests/test_ssm.py@@ -793,46 +793,104 @@ def test_cloudformation_ssm_parameter_is_region_scoped():set_request_region(original_region)-def test_ssm_run_command_health_probe(ssm, ec2):- """The Run Command shape a health probe needs: send, then poll to a terminal invocation."""- tag = _uuid_mod.uuid4().hex[:8]- instance_id = ec2.run_instances(ImageId="ami-12345678", MinCount=1, MaxCount=1,- TagSpecifications=[{"ResourceType": "instance",- "Tags": [{"Key": f"ssm-{tag}",- "Value": tag}]}],- )["Instances"][0]["InstanceId"]+# ── Run Command helpers ────────────────────────────────────+# A command needs an agent answering, and this emulator's agent is the container behind the+# instance. So the tests that send a command register an AMI and launch a real box; the ones that+# prove refusal do not need one.+++def _docker_reachable():+ try:+ import docker as _probe++ _probe.from_env(timeout=5).ping()+ return True+ except Exception:+ return False+++requires_docker = pytest.mark.skipif(+ not _docker_reachable(), reason="no reachable Docker daemon")+++def boxed_instance(ec2):+ """A running instance with a container behind it, so SSM can reach it."""+ ami = ec2.register_image(Name=f"ssm-{_uuid_mod.uuid4().hex[:8]}",+ ImageLocation="alpine:3")["ImageId"]+ instance_id = ec2.run_instances(ImageId=ami, MinCount=1,+ MaxCount=1)["Instances"][0]["InstanceId"]try:- command = ssm.send_command(InstanceIds=[instance_id], DocumentName="AWS-RunShellScript",- Comment=f"probe {tag}",- Parameters={"commands": ["echo ok"]})["Command"]- assert command["CommandId"]- assert command["DocumentName"] == "AWS-RunShellScript"- assert command["InstanceIds"] == [instance_id]- assert command["TargetCount"] == 1-- # Nothing runs, so the output is empty; what a polling caller needs is a terminal Status.- inv = ssm.get_command_invocation(CommandId=command["CommandId"], InstanceId=instance_id)- assert inv["Status"] == "Success"- assert inv["StatusDetails"] == "Success"- assert inv["ResponseCode"] == 0- assert inv["StandardOutputContent"] == ""- assert inv["DocumentName"] == "AWS-RunShellScript"- # These are strings in the SSM model, not timestamps, so boto3 must not have parsed them.- assert isinstance(inv["ExecutionStartDateTime"], str)- assert isinstance(inv["ExecutionEndDateTime"], str)-- # A caller that checks whether the instance is managed before sending must see it.- info = ssm.describe_instance_information()["InstanceInformationList"]- mine = [i for i in info if i["InstanceId"] == instance_id]- assert len(mine) == 1- assert mine[0]["PingStatus"] == "Online"- assert mine[0]["ResourceType"] == "EC2Instance"+ yield instance_idfinally:ec2.terminate_instances(InstanceIds=[instance_id])+ ec2.deregister_image(ImageId=ami)+++def _poll_invocation(ssm, command_id, instance_id, timeout=30):+ """Run Command is asynchronous on AWS, so a caller polls to a terminal state."""+ deadline = time.time() + timeout+ terminal = {"Success", "Failed", "Cancelled", "TimedOut"}+ while time.time() < deadline:+ inv = ssm.get_command_invocation(CommandId=command_id, InstanceId=instance_id)+ if inv["Status"] in terminal:+ return inv+ time.sleep(0.2)+ raise AssertionError(f"invocation never reached a terminal state: {inv['Status']}")+++@requires_docker+def test_ssm_run_command_health_probe(ssm, boxed_instance):+ """The Run Command shape a health probe needs: send, then poll to a terminal invocation."""+ tag = _uuid_mod.uuid4().hex[:8]+ command = ssm.send_command(InstanceIds=[boxed_instance], DocumentName="AWS-RunShellScript",+ Comment=f"probe {tag}",+ Parameters={"commands": ["echo ok"]})["Command"]+ assert command["CommandId"]+ assert command["DocumentName"] == "AWS-RunShellScript"+ assert command["InstanceIds"] == [boxed_instance]+ assert command["TargetCount"] == 1+ # SendCommand answers before the command has been delivered, as on AWS.+ assert command["Status"] == "Pending"+ assert command["CompletedCount"] == 0++ inv = _poll_invocation(ssm, command["CommandId"], boxed_instance)+ assert inv["Status"] == "Success"+ assert inv["StatusDetails"] == "Success"+ assert inv["ResponseCode"] == 0+ # The script actually ran in the box.+ assert inv["StandardOutputContent"].strip() == "ok"+ assert inv["DocumentName"] == "AWS-RunShellScript"+ # These are strings in the SSM model, not timestamps, so boto3 must not have parsed them.+ assert isinstance(inv["ExecutionStartDateTime"], str)+ assert isinstance(inv["ExecutionEndDateTime"], str)++ # A caller that checks whether the instance is managed before sending must see it.+ info = ssm.describe_instance_information()["InstanceInformationList"]+ mine = [i for i in info if i["InstanceId"] == boxed_instance]+ assert len(mine) == 1+ assert mine[0]["PingStatus"] == "Online"+ assert mine[0]["ResourceType"] == "EC2Instance"+++@requires_docker+def test_ssm_run_command_probe_can_fail(ssm, boxed_instance):+ """The point of a real box: a health check that is wrong reports Failed, not Success."""+ command = ssm.send_command(InstanceIds=[boxed_instance], DocumentName="AWS-RunShellScript",+ Parameters={"commands": ["echo bad >&2", "exit 7"]})["Command"]+ inv = _poll_invocation(ssm, command["CommandId"], boxed_instance)+ assert inv["Status"] == "Failed"+ assert inv["StatusDetails"] == "Failed"+ assert inv["ResponseCode"] == 7+ assert inv["StandardErrorContent"].strip() == "bad"+ listed = ssm.list_commands(CommandId=command["CommandId"])["Commands"][0]+ assert listed["Status"] == "Failed"+ assert listed["ErrorCount"] == 1def test_ssm_send_command_validates_the_instance(ssm, ec2):- """An instance that does not exist, is gone, or is stopped is rejected, a running one taken."""+ """AWS folds several problems into InvalidInstanceId: no such node, a terminated one, and+ an agent that is not running. A metadata-only instance is the last of those."""with pytest.raises(ClientError) as exc:ssm.send_command(InstanceIds=["i-00000000000000000"], DocumentName="AWS-RunShellScript")assert exc.value.response["Error"]["Code"] == "InvalidInstanceId"@@ -844,41 +902,48 @@ def test_ssm_send_command_validates_the_instance(ssm, ec2):ssm.send_command(InstanceIds=[instance_id], DocumentName="AWS-RunShellScript")assert exc.value.response["Error"]["Code"] == "InvalidInstanceId"- # A stopped box has no agent answering, so real SSM refuses it and stops listing it.- stopped = ec2.run_instances(ImageId="ami-12345678", MinCount=1,+ # Running, but nothing is behind it, so no agent is answering — which is one of the causes+ # AWS lists for this error. It is not listed as a managed node either.+ boxless = ec2.run_instances(ImageId="ami-12345678", MinCount=1,MaxCount=1)["Instances"][0]["InstanceId"]try:- ec2.stop_instances(InstanceIds=[stopped])with pytest.raises(ClientError) as exc:- ssm.send_command(InstanceIds=[stopped], DocumentName="AWS-RunShellScript")+ ssm.send_command(InstanceIds=[boxless], DocumentName="AWS-RunShellScript")assert exc.value.response["Error"]["Code"] == "InvalidInstanceId"listed = [i["InstanceId"]for i in ssm.describe_instance_information()["InstanceInformationList"]]- assert stopped not in listed+ assert boxless not in listedfinally:- ec2.terminate_instances(InstanceIds=[stopped])+ ec2.terminate_instances(InstanceIds=[boxless])- # Rejecting everything would pass every assertion above, so prove a running one is taken.- live = ec2.run_instances(ImageId="ami-12345678", MinCount=1,- MaxCount=1)["Instances"][0]["InstanceId"]- try:- assert ssm.send_command(InstanceIds=[live],- DocumentName="AWS-RunShellScript")["Command"]["CommandId"]- finally:- ec2.terminate_instances(InstanceIds=[live])+@requires_docker+def test_ssm_send_command_accepts_a_managed_instance(ssm, boxed_instance):+ """Rejecting everything would satisfy the refusals above, so prove one is taken."""+ assert ssm.send_command(InstanceIds=[boxed_instance],+ DocumentName="AWS-RunShellScript",+ Parameters={"commands": ["true"]})["Command"]["CommandId"]+ listed = [i["InstanceId"]+ for i in ssm.describe_instance_information()["InstanceInformationList"]]+ assert boxed_instance in listed++@requires_dockerdef test_ssm_command_lookup_filters_and_errors(ssm, ec2):"""listCommands narrows by command and instance; unknown ids get what AWS answers."""- instance_id = ec2.run_instances(ImageId="ami-12345678", MinCount=1,+ ami = ec2.register_image(Name=f"ssm-lookup-{_uuid_mod.uuid4().hex[:8]}",+ ImageLocation="alpine:3")["ImageId"]+ instance_id = ec2.run_instances(ImageId=ami, MinCount=1,MaxCount=1)["Instances"][0]["InstanceId"]- other_id = ec2.run_instances(ImageId="ami-12345678", MinCount=1,+ other_id = ec2.run_instances(ImageId=ami, MinCount=1,MaxCount=1)["Instances"][0]["InstanceId"]try:mine = ssm.send_command(InstanceIds=[instance_id],- DocumentName="AWS-RunShellScript")["Command"]["CommandId"]+ DocumentName="AWS-RunShellScript",+ Parameters={"commands": ["true"]})["Command"]["CommandId"]theirs = ssm.send_command(InstanceIds=[other_id],- DocumentName="AWS-RunShellScript")["Command"]["CommandId"]+ DocumentName="AWS-RunShellScript",+ Parameters={"commands": ["true"]})["Command"]["CommandId"]assert [c["CommandId"] for c in ssm.list_commands(CommandId=mine)["Commands"]] == [mine]by_instance = [c["CommandId"] for c in ssm.list_commands(InstanceId=other_id)["Commands"]]@@ -899,3 +964,4 @@ def test_ssm_command_lookup_filters_and_errors(ssm, ec2):assert exc.value.response["Error"]["Code"] == "InvocationDoesNotExist"finally:ec2.terminate_instances(InstanceIds=[instance_id, other_id])+ ec2.deregister_image(ImageId=ami)diff --git a/tests/test_stepfunctions.py b/tests/test_stepfunctions.pyindex 1358fc44..8a62e817 100644--- a/tests/test_stepfunctions.py+++ b/tests/test_stepfunctions.py@@ -4897,8 +4897,16 @@ def test_sfn_aws_sdk_query_pascal_case(sfn, sfn_sync, ssm):def test_sfn_aws_sdk_ssm_run_command_probe(sfn, sfn_sync, ec2):"""The health-probe shape: sendCommand, then getCommandInvocation on the id it returned."""+ try:+ import docker as _probe++ _probe.from_env(timeout=5).ping()+ except Exception:+ pytest.skip("no reachable Docker daemon — Run Command needs a box behind the instance")tag = _uuid_mod.uuid4().hex[:8]- instance_id = ec2.run_instances(ImageId="ami-12345678", MinCount=1,+ # Run Command needs an agent answering, which here is the container behind the instance.+ ami = ec2.register_image(Name=f"sfn-ssm-{tag}", ImageLocation="alpine:3")["ImageId"]+ instance_id = ec2.run_instances(ImageId=ami, MinCount=1,MaxCount=1)["Instances"][0]["InstanceId"]sm_arn = Nonetry:@@ -4925,6 +4933,9 @@ def test_sfn_aws_sdk_ssm_run_command_probe(sfn, sfn_sync, ec2):"CommandId.$": "$.sent.Command.CommandId","InstanceId": instance_id,},+ # Run Command is asynchronous, so a probe polls to a terminal state.+ "Retry": [{"ErrorEquals": ["States.ALL"], "IntervalSeconds": 1,+ "MaxAttempts": 3}],"End": True,},},@@ -4936,11 +4947,11 @@ def test_sfn_aws_sdk_ssm_run_command_probe(sfn, sfn_sync, ec2):assert resp["status"] == "SUCCEEDED", f"{resp.get('error')} — {resp.get('cause')}"# The poll addressed the CommandId the send returned, which is the part a probe needs.output = json.loads(resp["output"])- assert output["Status"] == "Success"assert output["InstanceId"] == instance_id- assert output["ResponseCode"] == 0+ assert output["Status"] in ("Pending", "InProgress", "Success")finally:ec2.terminate_instances(InstanceIds=[instance_id])+ ec2.deregister_image(ImageId=ami)if sm_arn:sfn.delete_state_machine(stateMachineArn=sm_arn)