Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(ecs): task record and task metadata parity (#1720)

iot-rocketeebba808299b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6ab301eb..f86ad850 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,10 @@ Versioning follows [Semantic Versioning](https://semver.org/).
## [Unreleased]
### Fixed
+- **ECS — a task's `version` counts its state changes** — the counter was minted at `1` and never moved again, so a consumer holding an older `DescribeTasks` copy could not tell it from the current one, which is the comparison the field exists for. It now increments on each transition the record reports: the move to `ACTIVATING`, the move to `RUNNING`, the move to `STOPPED` whether it came from `StopTask`, an essential container exiting or a failed start, and the restore after a restart that stops a task that was running. A real task counts the same way, measured on a Fargate task polled through `DescribeTasks`: `1` at `PROVISIONING`, `2` at `PENDING`, `3` at `RUNNING`, `4` when `StopTask` set `desiredStatus`, `5` at `DEPROVISIONING`, `6` at `STOPPED`. There are no task state change events here, so the counter is what `DescribeTasks`, `RunTask` and `StopTask` report and nothing else. Contributed by @iot-rocket.
+- **ECS — only an `awsvpc` task reports an elastic network interface** — every task got an `ElasticNetworkInterface` attachment once its container was up, whatever its network mode, while the Task reference describes `attachments` as the adapter a task has "if the task uses the `awsvpc` network mode": a `bridge` or `host` task on AWS reports none and is reached through its container instance. A task in those modes now reports an empty list, an `awsvpc` task's attachment additionally names the subnet the request placed it in, and the address stays available to the target-group sync in every mode. Contributed by @iot-rocket.
+- **ECS — a task no longer reports `attachmentsStatus`** — the task record carried an `attachmentsStatus` member next to its attachment, and there is no such member on the `Task` shape: the API model has it on `Cluster` only, and a live task on AWS, awsvpc and bridge alike, sends no such field. A client reading the raw wire saw an invented member, and the emulator was the only place it existed. The cluster's own `attachmentsStatus` is untouched. Contributed by @iot-rocket.
+- **ECS — the task metadata endpoint reports the status the task is in** — `${ECS_CONTAINER_METADATA_URI_V4}` served `DesiredStatus` and `KnownStatus` as the literal `RUNNING`, captured when the container was registered, so a container that read its own metadata was told the task was running while `DescribeTasks` reported `PENDING` or `ACTIVATING`. The two could not disagree before `RunTask` returned asynchronously; now that the image pull sits inside that window, they disagree for the length of it. Both members are read off the task record on every request, on the container route and the task route, and a task that no longer has a record keeps what was registered. `DesiredStatus` is the task's everywhere, which is how a container learns it is being shut down; `KnownStatus` is the task's on the task payload and the container's own on a container payload, because that is what AWS reports: a live Fargate task served `"KnownStatus": "NONE"` for itself and `"RUNNING"` for the container reading the endpoint, in one payload. The task is looked up by the account and region in its ARN, since a container reaches the endpoint with a path token and no SigV4. Contributed by @iot-rocket.
- **IAM — one `_account_from_arn`, with the contract the condition evaluator documents** — the module carried two top-level definitions of the helper under the same name, so the later one silently won and the guards the documented one describes (a bare `*`, a value that is not an ARN at all, an AWS-owned `arn:aws:iam::aws:policy/...`) were not the ones in force when `aws:ResourceAccount` resolved a resource's account. They are folded into one definition that keeps both contracts: the field is returned only when it holds an account id, twelve digits, and only from a value that is an ARN. No request the emulator serves changes behaviour, because both callers pass an ARN whose account is twelve digits; the parametrized test gained the two cases that tell the former definitions apart. Contributed by @iot-rocket.
- **Test harness — the suite runs on a non-default `GATEWAY_PORT`** — eight tests compared a value a service had built from `GATEWAY_PORT` against a hard-coded `4566`, so the suite passed only on the default port. That is the port a contributor moves off to run a private server rather than resetting a shared one, which is exactly when the failures appear and look like real regressions. The ESM tests now key their queues through `sqs._queue_url()`, the AppSync and Glue assertions read the port from one place in `tests/conftest.py`, and the three files pass on any port. No user-visible behavior changes. Contributed by @iot-rocket.
diff --git a/ministack/services/ecs.py b/ministack/services/ecs.py
index fa434413..9804a363 100644
--- a/ministack/services/ecs.py
+++ b/ministack/services/ecs.py
@@ -77,6 +77,23 @@ _capacity_providers = AccountRegionScopedDict()
# try/except swallows it, and ALL ECS state silently fails to restore.
_attributes = AccountRegionScopedDict()
+
+# Up here for the same reason as `_attributes`: the import-time `load_state`
+# block calls `restore_state`, which counts the tasks it stops, so this has to
+# be bound before that runs. Defined further down it raises NameError there,
+# the surrounding try/except swallows it, and ALL ECS state fails to restore.
+def _bump_task_version(task):
+ """Count one observable change on the task.
+
+ The Task reference calls ``version`` the counter a consumer compares against
+ the version an event carries, to tell a stale copy of the record from the
+ current one. A real task counts its whole lifecycle: a Fargate task polled
+ through DescribeTasks reported 1 at PROVISIONING, 2 at PENDING, 3 at
+ RUNNING, 4 when StopTask set desiredStatus, 5 at DEPROVISIONING, 6 at
+ STOPPED. The emulator counts the states it has.
+ """
+ task["version"] = int(task.get("version") or 1) + 1
+
_docker = None
# ECS exited-container reaper. Every ministack=ecs container we start via
@@ -164,6 +181,8 @@ def get_state():
t = copy.deepcopy(task)
t.pop("_docker_ids", None)
t.pop("_metadata_tokens", None)
+ # The container is gone with the process; its address must not outlive it.
+ t.pop("_container_ip", None)
tasks._data[scoped_key] = t
state["tasks"] = tasks
return state
@@ -236,12 +255,16 @@ def restore_state(data):
for scoped_key, task in tasks_data._data.items():
restored_task = copy.deepcopy(task)
restored_task["_docker_ids"] = []
+ if restored_task.get("lastStatus") != "STOPPED":
+ _bump_task_version(restored_task)
restored_task["lastStatus"] = "STOPPED"
_tasks._data[scoped_key] = restored_task
elif isinstance(tasks_data, AccountScopedDict):
for (account_id, arn), task in tasks_data._data.items():
restored_task = copy.deepcopy(task)
restored_task["_docker_ids"] = []
+ if restored_task.get("lastStatus") != "STOPPED":
+ _bump_task_version(restored_task)
restored_task["lastStatus"] = "STOPPED"
region = _tasks._region_for_legacy_value(arn, restored_task)
_tasks.set_scoped(account_id, region, arn, restored_task)
@@ -249,6 +272,8 @@ def restore_state(data):
for arn, task in tasks_data.items():
restored_task = copy.deepcopy(task)
restored_task["_docker_ids"] = []
+ if restored_task.get("lastStatus") != "STOPPED":
+ _bump_task_version(restored_task)
restored_task["lastStatus"] = "STOPPED"
region = _tasks._region_for_legacy_value(arn, restored_task)
_tasks.set_scoped(get_account_id(), region, arn, restored_task)
@@ -717,29 +742,78 @@ def _make_deployment(task_definition, desired_count, status="PRIMARY"):
}
-def _record_task_ip(task, container, ecs_network):
- """Store the container's address on the task as an ENI attachment.
+def _requested_subnet(data):
+ """The first subnet of a RunTask/CreateService awsvpcConfiguration, if any.
+
+ Nothing else read the request's network configuration until now, so this
+ cannot assume the shape the SDK would have sent: a body that names it as
+ anything but the documented object has no subnet to report.
- Real awsvpc tasks expose it as attachments[].details[privateIPv4Address], which
- is where an ALB target group and DescribeTasks both look for it.
+ Both casings are read. A service created through CloudFormation keeps the
+ template's `NetworkConfiguration` verbatim (provisioners.py, the
+ AWS::ECS::Service handler) and replays it here, so the camelCase lookup
+ alone would miss every CFN-defined service.
"""
- if task.get("attachments"):
- return
+ network = data.get("networkConfiguration")
+ if not isinstance(network, dict):
+ return None
+ config = network.get("awsvpcConfiguration") or network.get("AwsvpcConfiguration")
+ if not isinstance(config, dict):
+ return None
+ subnets = config.get("subnets") or config.get("Subnets")
+ if not isinstance(subnets, list) or not subnets:
+ return None
+ return subnets[0] if isinstance(subnets[0], str) else None
+
+
+def _container_ip(container, ecs_network):
try:
container.reload()
nets = container.attrs["NetworkSettings"]["Networks"]
- ip = (nets.get(ecs_network) or next(iter(nets.values()), {})).get("IPAddress")
+ return (nets.get(ecs_network) or next(iter(nets.values()), {})).get("IPAddress")
except Exception:
- ip = None
+ return None
+
+
+def _record_task_ip(task, container, ecs_network):
+ """Store the container's address on the task.
+
+ The Task reference calls `attachments` "The Elastic Network Adapter that's
+ associated with the task if the task uses the `awsvpc` network mode", so a
+ bridge or host task has none: its ports are published on the container
+ instance and that is where a consumer reads its address. Only an awsvpc task
+ gets the attachment, and it carries the subnet the request placed it in
+ alongside the address, two of the four members the Attachment reference
+ names for an elastic network interface.
+
+ The address is kept on the task in either mode, under an internal key, for
+ the target-group sync. The first container to report one owns it: a
+ two-container task on AWS reports one attachment and both containers name
+ that one address, so the sync must not follow a sidecar around. Each
+ container calls this from its own thread, so the guard and the write it
+ protects share the task lock; the docker round trip stays outside it.
+ """
+ if task.get("_container_ip") or task.get("attachments"):
+ return
+ ip = _container_ip(container, ecs_network)
if not ip:
return
- task["attachments"] = [{
- "id": new_uuid(),
- "type": "ElasticNetworkInterface",
- "status": "ATTACHED",
- "details": [{"name": "privateIPv4Address", "value": ip}],
- }]
- task["attachmentsStatus"] = "ATTACHED"
+ with resource_lock("ecs-task", task.get("taskArn", "")):
+ if task.get("_container_ip") or task.get("attachments"):
+ return
+ task["_container_ip"] = ip
+ if task.get("_network_mode") != "awsvpc":
+ return
+ details = [{"name": "privateIPv4Address", "value": ip}]
+ subnet = task.get("_subnet")
+ if subnet:
+ details.insert(0, {"name": "subnetId", "value": subnet})
+ task["attachments"] = [{
+ "id": new_uuid(),
+ "type": "ElasticNetworkInterface",
+ "status": "ATTACHED",
+ "details": details,
+ }]
def _task_ip(task):
@@ -747,7 +821,7 @@ def _task_ip(task):
for d in att.get("details") or []:
if d.get("name") == "privateIPv4Address":
return d.get("value")
- return None
+ return task.get("_container_ip")
def _sync_service_targets(cluster_name, svc):
@@ -1208,6 +1282,13 @@ def _register_metadata(task_arn, cluster_arn, td, cdef, launch_type, env,
env["AWS_CONTAINER_CREDENTIALS_FULL_URI"] = f"http://{host}:{port}/v2/credentials/{new_uuid()}"
env["AWS_CONTAINER_AUTHORIZATION_TOKEN"] = secrets.token_urlsafe(32)
env["AWS_ENDPOINT_URL"] = f"http://{host}:{port}"
+ # Seeded from the record, not from a literal: the endpoint overlays the
+ # live status on every read, so a literal here would only ever be read for
+ # a task that no longer has one, and it would be wrong then too.
+ desired, known, per_container = (
+ metadata_task_status(task_arn) or ("RUNNING", "PENDING", {})
+ )
+ container_known = per_container.get(cdef["name"]) or known
ecs_metadata.register_container(
token,
task_arn,
@@ -1216,8 +1297,8 @@ def _register_metadata(task_arn, cluster_arn, td, cdef, launch_type, env,
"TaskARN": task_arn,
"Family": td.get("family", ""),
"Revision": str(td.get("revision", 1)),
- "DesiredStatus": "RUNNING",
- "KnownStatus": "RUNNING",
+ "DesiredStatus": desired,
+ "KnownStatus": known,
"AvailabilityZone": f"{get_region()}a",
"LaunchType": launch_type,
},
@@ -1232,8 +1313,8 @@ def _register_metadata(task_arn, cluster_arn, td, cdef, launch_type, env,
"com.amazonaws.ecs.task-definition-version": str(td.get("revision", 1)),
"com.amazonaws.ecs.cluster": cluster_arn,
},
- "DesiredStatus": "RUNNING",
- "KnownStatus": "RUNNING",
+ "DesiredStatus": desired,
+ "KnownStatus": container_known,
"Type": "NORMAL",
},
)
@@ -1384,6 +1465,45 @@ def _resolve_container_secrets(cdef):
return resolved
+def metadata_task_status(task_arn):
+ """Status for the task metadata endpoint, or None when the task is gone.
+
+ Returns ``(desiredStatus, lastStatus, {container name: lastStatus})``. The
+ endpoint reports a task's status, and a task's status changes after the
+ metadata is registered: it is `PENDING` when `RunTask` answers, `ACTIVATING`
+ for the pull and `RUNNING` once the container is up. Serving the record
+ rather than a value captured at registration is what keeps
+ `${ECS_CONTAINER_METADATA_URI_V4}/task` agreeing with `DescribeTasks`.
+
+ A container's own status is carried separately because the two really do
+ differ: a live Fargate task served `"KnownStatus": "NONE"` for the task and
+ `"KnownStatus": "RUNNING"` for the container in the same payload, read by
+ the container itself while it was starting. `DesiredStatus` is the task's
+ on both, which is what flips to STOPPED when the task is being shut down.
+
+ The lookup is scoped by the account and the region in the task ARN, not by
+ the request's: a container reaches this endpoint with a path token and no
+ SigV4, so the request resolves under the default account and region
+ whatever the task was created under.
+ """
+ try:
+ spec = parse_arn(task_arn)
+ except ArnParseError:
+ return None
+ task = _tasks.get_scoped(spec.account_id, spec.region, task_arn)
+ if task is None:
+ return None
+ return (
+ task.get("desiredStatus") or "RUNNING",
+ task.get("lastStatus") or "PENDING",
+ {
+ c["name"]: c.get("lastStatus")
+ for c in task.get("containers", [])
+ if c.get("name")
+ },
+ )
+
+
def _task_is_active(task_arn, task):
return (
_tasks.get(task_arn) is task
@@ -1442,6 +1562,7 @@ def _mark_task_stopped(task_arn, task, reason, stop_code, exit_code=None):
if not _task_is_active(task_arn, task):
return False
now = _iso()
+ _bump_task_version(task)
task["lastStatus"] = "STOPPED"
task["desiredStatus"] = "STOPPED"
task["stoppingAt"] = task.get("stoppingAt") or now
@@ -1473,6 +1594,7 @@ def _mark_task_activating(task_arn, task):
with resource_lock("ecs-task", task_arn):
if not _task_is_active(task_arn, task):
return False
+ _bump_task_version(task)
task["lastStatus"] = "ACTIVATING"
task["pullStartedAt"] = task.get("pullStartedAt") or _iso()
@@ -1491,6 +1613,7 @@ def _mark_task_running(task_arn, task):
if not _task_is_active(task_arn, task):
return False
now = _iso()
+ _bump_task_version(task)
task["lastStatus"] = "RUNNING"
task["pullStoppedAt"] = task.get("pullStoppedAt") or now
task["startedAt"] = task.get("startedAt") or now
@@ -1717,6 +1840,11 @@ def _run_task(data):
docker_client = _get_docker()
docker_backed = bool(docker_client)
initial_status = "PENDING" if docker_backed else "RUNNING"
+ # The network mode decides whether a task reports an ENI attachment at all,
+ # and the subnet is one of the members that attachment carries. Both are read
+ # when the container comes up, so they ride on the record until then.
+ network_mode = td.get("networkMode")
+ subnet = _requested_subnet(data)
tasks = []
failures = []
@@ -1767,6 +1895,8 @@ def _run_task(data):
"group": group,
"startedBy": started_by,
"version": 1,
+ "_network_mode": network_mode,
+ "_subnet": subnet,
"containers": containers,
"attachments": [],
"availabilityZone": f"{region}a",
diff --git a/ministack/services/ecs_metadata.py b/ministack/services/ecs_metadata.py
index 25ca2259..260a5f70 100644
--- a/ministack/services/ecs_metadata.py
+++ b/ministack/services/ecs_metadata.py
@@ -80,6 +80,44 @@ def reset() -> None:
_TOKEN_TO_CONTAINER.clear()
+def _live_status(task_arn: str) -> tuple[str, str, dict] | None:
+ """``(DesiredStatus, KnownStatus, {container name: status})``, or None.
+
+ Imported lazily: services/ecs.py imports this module, so a module-level
+ import back would be circular.
+ """
+ from ministack.services import ecs
+
+ return ecs.metadata_task_status(task_arn)
+
+
+def _with_status(payload: dict, status, container: bool = False) -> dict:
+ """A copy of ``payload`` carrying the current status.
+
+ `DesiredStatus` is the task's everywhere, which is how a container learns
+ it is being shut down. `KnownStatus` is the task's on the task payload and
+ the container's own on a container payload: on AWS the two differ while a
+ task starts, where the container is already RUNNING and the task is not.
+
+ A copy because this writes the two members, and the dicts it is handed are
+ the live registry entries: the task payload every sibling container's
+ `/task` view is built from, and the container payloads inside it. Writing
+ the overlay into those would drift the registry away from what was
+ registered, and the fallback for a task whose record is gone would then
+ serve a value nobody registered.
+ """
+ out = dict(payload)
+ if status is None:
+ return out
+ desired, known, per_container = status
+ out["DesiredStatus"] = desired
+ if container:
+ out["KnownStatus"] = per_container.get(out.get("Name")) or out.get("KnownStatus") or known
+ else:
+ out["KnownStatus"] = known
+ return out
+
+
async def handle_request(method, path, headers, body, query_params):
m = _PATH_RE.match(path)
if not m:
@@ -91,11 +129,20 @@ async def handle_request(method, path, headers, body, query_params):
return json_response({"message": "unknown token"}, status=404)
container = _TOKEN_TO_CONTAINER[token]
task = _TASKS[arn]
+ containers = list(task.get("Containers", []))
+
+ # Resolved outside _LOCK: the ecs service takes its own per-task locks, and
+ # holding this one across that call would be a lock-order inversion.
+ status = _live_status(arn)
rest = (m.group("rest") or "").rstrip("/")
if rest == "":
- return json_response(container)
+ return json_response(_with_status(container, status, container=True))
if rest == "/task":
+ task = _with_status(task, status)
+ task["Containers"] = [
+ _with_status(c, status, container=True) for c in containers
+ ]
return json_response(task)
if rest in ("/stats", "/task/stats"):
return json_response({})
diff --git a/tests/test_ecs.py b/tests/test_ecs.py
index 47a04a35..0ad96d9b 100644
--- a/tests/test_ecs.py
+++ b/tests/test_ecs.py
@@ -237,6 +237,50 @@ def test_ecs_run_task_metadata_v4(ecs):
assert success, "Task should transition to STOPPED"
+def test_ecs_restore_helpers_are_defined_before_the_import_time_restore():
+ """Everything `restore_state` calls must be bound before the module runs it.
+
+ `ecs.py` calls `restore_state(_restored)` at module level, so a helper it
+ reaches that is defined further down the file raises NameError there. The
+ surrounding try/except catches it and ALL ECS state fails to restore, which
+ is only visible under PERSIST_STATE=1 and never in a test that calls
+ `restore_state` after the import. The file already carries `_attributes`
+ at the top for exactly this reason; this keeps the next one honest.
+ """
+ import ast
+ import inspect
+
+ tree = ast.parse(inspect.getsource(ecs_service))
+ defined_at = {
+ node.name: node.lineno
+ for node in tree.body
+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))
+ }
+ restore = next(
+ n for n in tree.body
+ if isinstance(n, ast.FunctionDef) and n.name == "restore_state"
+ )
+ call_line = next(
+ n.lineno for n in ast.walk(tree)
+ if isinstance(n, ast.Call)
+ and isinstance(n.func, ast.Name)
+ and n.func.id == "restore_state"
+ and n.col_offset == 8 # the module-level try: block, not a nested call
+ )
+
+ late = sorted({
+ node.id
+ for node in ast.walk(restore)
+ if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Load)
+ and node.id in defined_at and defined_at[node.id] > call_line
+ })
+ assert not late, (
+ f"restore_state reaches {late}, defined after the import-time call at "
+ f"line {call_line}; move them above it or the warm-boot restore dies "
+ f"silently"
+ )
+
+
def test_ecs_run_task_applies_container_command_overrides(monkeypatch):
"""RunTask containerOverrides.command should reach Docker run kwargs."""
from ministack.services import ecs as _ecs
@@ -1722,6 +1766,113 @@ def test_ecs_reset_during_pending_start_cleans_late_container(monkeypatch):
assert task_arn not in _ecs._tasks
+def _version_probe_container(cid):
+ class FakeContainer:
+ def __init__(self):
+ self.id = cid
+ self.status = "running"
+ self.attrs = {"NetworkSettings": {"Networks": {}}}
+ self.removed = False
+
+ def reload(self):
+ pass
+
+ def wait(self):
+ return {"StatusCode": 0}
+
+ def stop(self, timeout=5):
+ self.status = "exited"
+
+ def remove(self, **kwargs):
+ self.removed = True
+
+ return FakeContainer()
+
+
+def _version_probe_docker(container):
+ class FakeContainers:
+ def get(self, name):
+ if name == container.id:
+ return container
+ raise Exception("not found")
+
+ def list(self, *args, **kwargs):
+ return [container]
+
+ def run(self, image, **kwargs):
+ return container
+
+ return SimpleNamespace(containers=FakeContainers())
+
+
+def test_ecs_task_version_counts_state_changes(monkeypatch):
+ """The version counter moves on every transition the record reports, so a
+ consumer can tell a stale copy from the current one."""
+ from ministack.services import ecs as _ecs
+
+ container = _version_probe_container("version-counter-container")
+ monkeypatch.setattr(_ecs, "_get_docker", lambda: _version_probe_docker(container))
+ _ecs._register_task_definition({
+ "family": "version-counter-td",
+ "containerDefinitions": [{"name": "app", "image": "busybox"}],
+ })
+
+ response = _ecs._run_task({
+ "cluster": "version-counter-c",
+ "taskDefinition": "version-counter-td",
+ })
+ task = json.loads(response[2])["tasks"][0]
+ task_arn = task["taskArn"]
+ assert task["lastStatus"] == "PENDING"
+ assert task["version"] == 1
+
+ # 3, not 2: the task passes through ACTIVATING on its way to RUNNING, and
+ # that is a state this record reports, so it counts like the others.
+ _wait_until(lambda: _ecs._tasks[task_arn]["lastStatus"] == "RUNNING")
+ assert _ecs._tasks[task_arn]["version"] == 3
+
+ stopped = json.loads(_ecs._stop_task({
+ "cluster": "version-counter-c",
+ "task": task_arn,
+ "reason": "done here",
+ })[2])["task"]
+ assert stopped["lastStatus"] == "STOPPED"
+ assert stopped["version"] == 4
+
+
+def test_ecs_task_version_moves_once_for_a_natural_exit(monkeypatch):
+ """The exit is observed by whichever DescribeTasks notices it first; the
+ ones after it describe the same version."""
+ from ministack.services import ecs as _ecs
+
+ container = _version_probe_container("version-exit-container")
+ monkeypatch.setattr(_ecs, "_get_docker", lambda: _version_probe_docker(container))
+ _ecs._register_task_definition({
+ "family": "version-exit-td",
+ "containerDefinitions": [{"name": "app", "image": "busybox"}],
+ })
+
+ task_arn = json.loads(_ecs._run_task({
+ "cluster": "version-exit-c",
+ "taskDefinition": "version-exit-td",
+ })[2])["tasks"][0]["taskArn"]
+ _wait_until(lambda: _ecs._tasks[task_arn]["lastStatus"] == "RUNNING")
+
+ container.status = "exited"
+ described = json.loads(_ecs._describe_tasks({
+ "cluster": "version-exit-c",
+ "tasks": [task_arn],
+ })[2])["tasks"][0]
+ assert described["lastStatus"] == "STOPPED"
+ assert described["version"] == 4
+
+ again = json.loads(_ecs._describe_tasks({
+ "cluster": "version-exit-c",
+ "tasks": [task_arn],
+ })[2])["tasks"][0]
+ assert again["version"] == 4
+
+
def test_ecs_secret_resolution_failure_stops_before_docker_run(monkeypatch):
from ministack.services import ecs as _ecs
@@ -1984,10 +2135,224 @@ def test_ecs_task_records_private_ipv4_attachment(monkeypatch):
_wait_until(lambda: _ecs._task_ip(_ecs._tasks.get(task["taskArn"])) == "172.30.0.9")
task = _ecs._tasks[task["taskArn"]]
assert _ecs._task_ip(task) == "172.30.0.9"
- assert task["attachmentsStatus"] == "ATTACHED"
+ # Not a member of the Task shape; the emulator used to invent it.
+ assert "attachmentsStatus" not in task
assert task["attachments"][0]["type"] == "ElasticNetworkInterface"
+def _eni_probe_docker(ip):
+ class FakeContainer:
+ id = "container-000000000002"
+ attrs = {"NetworkSettings": {"Networks": {"ministack_default": {"IPAddress": ip}}}}
+
+ def reload(self):
+ pass
+
+ class FakeContainers:
+ def get(self, _name):
+ raise Exception("not found")
+
+ def list(self, *a, **k):
+ return []
+
+ def run(self, image, **kwargs):
+ return FakeContainer()
+
+ return SimpleNamespace(containers=FakeContainers())
+
+
+def test_ecs_awsvpc_attachment_carries_the_subnet_it_was_placed_in(monkeypatch):
+ """The attachment names the subnet the request asked for, one of the members
+ the Attachment reference lists for an elastic network interface."""
+ from ministack.services import ecs as _ecs
+
+ monkeypatch.setattr(_ecs, "_get_docker", lambda: _eni_probe_docker("172.30.0.11"))
+ _ecs._register_task_definition({
+ "family": "eni-subnet-td",
+ "networkMode": "awsvpc",
+ "containerDefinitions": [{"name": "web", "image": "busybox"}],
+ })
+ task_arn = json.loads(_ecs._run_task({
+ "cluster": "eni-subnet-c",
+ "taskDefinition": "eni-subnet-td",
+ "networkConfiguration": {"awsvpcConfiguration": {
+ "subnets": ["subnet-0a1b2c3d", "subnet-9f8e7d6c"],
+ }},
+ })[2])["tasks"][0]["taskArn"]
+
+ _wait_until(lambda: _ecs._tasks[task_arn].get("attachments"))
+ details = {d["name"]: d["value"]
+ for d in _ecs._tasks[task_arn]["attachments"][0]["details"]}
+ assert details["subnetId"] == "subnet-0a1b2c3d"
+ assert details["privateIPv4Address"] == "172.30.0.11"
+
+
+def test_ecs_awsvpc_attachment_reads_the_cloudformation_casing(monkeypatch):
+ """A CloudFormation service replays its template's `NetworkConfiguration`.
+
+ The AWS::ECS::Service handler stores the template block verbatim, so it
+ reaches RunTask in PascalCase. Reading only the SDK's casing would leave
+ every CFN-defined service's tasks without a subnet.
+ """
+ from ministack.services import ecs as _ecs
+
+ monkeypatch.setattr(_ecs, "_get_docker", lambda: _eni_probe_docker("172.30.0.41"))
+ _ecs._register_task_definition({
+ "family": "eni-cfncase-td",
+ "networkMode": "awsvpc",
+ "containerDefinitions": [{"name": "web", "image": "busybox"}],
+ })
+ task_arn = json.loads(_ecs._run_task({
+ "cluster": "eni-cfncase-c",
+ "taskDefinition": "eni-cfncase-td",
+ "networkConfiguration": {"AwsvpcConfiguration": {
+ "Subnets": ["subnet-cfn00001"],
+ }},
+ })[2])["tasks"][0]["taskArn"]
+
+ _wait_until(lambda: _ecs._tasks[task_arn].get("attachments"))
+ details = {d["name"]: d["value"]
+ for d in _ecs._tasks[task_arn]["attachments"][0]["details"]}
+ assert details["subnetId"] == "subnet-cfn00001"
+
+
[email protected]("network_configuration", [
+ None,
+ {},
+ {"awsvpcConfiguration": {}},
+ {"awsvpcConfiguration": {"subnets": []}},
+])
+def test_ecs_awsvpc_attachment_without_a_usable_subnet(monkeypatch, network_configuration):
+ """A guard: the request's network configuration is not read anywhere else,
+ so a body that leaves it out or leaves it empty must still start the task
+ and report the attachment, just without a subnet."""
+ from ministack.services import ecs as _ecs
+
+ uid = _uuid_mod.uuid4().hex[:8]
+ monkeypatch.setattr(_ecs, "_get_docker", lambda: _eni_probe_docker("172.30.0.13"))
+ _ecs._register_task_definition({
+ "family": f"eni-nosubnet-td-{uid}",
+ "networkMode": "awsvpc",
+ "containerDefinitions": [{"name": "web", "image": "busybox"}],
+ })
+ request = {"cluster": "eni-nosubnet-c", "taskDefinition": f"eni-nosubnet-td-{uid}"}
+ if network_configuration is not None:
+ request["networkConfiguration"] = network_configuration
+ task_arn = json.loads(_ecs._run_task(request)[2])["tasks"][0]["taskArn"]
+
+ _wait_until(lambda: _ecs._tasks[task_arn].get("attachments"))
+ details = {d["name"]: d["value"]
+ for d in _ecs._tasks[task_arn]["attachments"][0]["details"]}
+ assert details == {"privateIPv4Address": "172.30.0.13"}
+
+
+def test_ecs_restore_stops_a_running_task_and_counts_it(monkeypatch):
+ """A restart stops every restored task, which is a change to the record:
+ without the bump a consumer's pre-restart copy still compares equal to a
+ task that is no longer running.
+
+ The saved state also drops the address the container had. That key is what
+ the live target-group sync reads, and the container it named is gone with
+ the process. The attachment keeps its own copy, which is right: AWS reports
+ a stopped task's attachment too, with the interface already DELETED.
+ """
+ from ministack.services import ecs as _ecs
+
+ container = _version_probe_container("restore-version-container")
+ container.attrs = {"NetworkSettings": {"Networks": {"n": {"IPAddress": "172.30.0.31"}}}}
+ monkeypatch.setattr(_ecs, "_get_docker", lambda: _version_probe_docker(container))
+ _ecs._register_task_definition({
+ "family": "restore-version-td",
+ "networkMode": "bridge",
+ "containerDefinitions": [{"name": "app", "image": "busybox"}],
+ })
+ task_arn = json.loads(_ecs._run_task({
+ "cluster": "restore-version-c",
+ "taskDefinition": "restore-version-td",
+ })[2])["tasks"][0]["taskArn"]
+ _wait_until(lambda: _ecs._tasks[task_arn]["lastStatus"] == "RUNNING")
+ _wait_until(lambda: _ecs._tasks[task_arn].get("_container_ip"))
+ running_version = _ecs._tasks[task_arn]["version"]
+
+ state = _ecs.get_state()
+ saved = [t for t in state["tasks"]._data.values() if t["taskArn"] == task_arn][0]
+ assert _ecs._tasks[task_arn]["_container_ip"] == "172.30.0.31"
+ assert "_container_ip" not in saved
+ _ecs.reset()
+ _ecs.restore_state(state)
+
+ restored = _ecs._tasks[task_arn]
+ assert restored["lastStatus"] == "STOPPED"
+ assert restored["version"] == running_version + 1
+
+ # Restoring an already stopped task is not a transition.
+ state = _ecs.get_state()
+ _ecs.reset()
+ _ecs.restore_state(state)
+ assert _ecs._tasks[task_arn]["version"] == running_version + 1
+
+
+def test_ecs_bridge_task_reports_no_eni_attachment(monkeypatch):
+ """`attachments` is documented as the adapter a task has "if the task uses
+ the awsvpc network mode", so a bridge task reports none. Its address is still
+ known, which is what the target-group sync reads."""
+ from ministack.services import ecs as _ecs
+
+ monkeypatch.setattr(_ecs, "_get_docker", lambda: _eni_probe_docker("172.30.0.12"))
+ _ecs._register_task_definition({
+ "family": "eni-bridge-td",
+ "networkMode": "bridge",
+ "containerDefinitions": [{"name": "web", "image": "busybox"}],
+ })
+ task_arn = json.loads(_ecs._run_task({
+ "cluster": "eni-bridge-c",
+ "taskDefinition": "eni-bridge-td",
+ })[2])["tasks"][0]["taskArn"]
+
+ _wait_until(lambda: _ecs._task_ip(_ecs._tasks[task_arn]) == "172.30.0.12")
+ task = _ecs._tasks[task_arn]
+ assert task["attachments"] == []
+ assert "attachmentsStatus" not in task
+ described = json.loads(_ecs._describe_tasks({
+ "cluster": "eni-bridge-c",
+ "tasks": [task_arn],
+ })[2])["tasks"][0]
+ assert described["attachments"] == []
+ assert "_container_ip" not in described
+
+
[email protected]("mode", ["awsvpc", "bridge"])
+def test_ecs_first_container_to_report_an_address_owns_it(mode):
+ """A task has one address, not one per container.
+
+ A two-container awsvpc task on AWS reports a single attachment, and both
+ containers name that one attachment and that one private address. Each
+ container here calls _record_task_ip from its own thread, so without the
+ guard a sidecar coming up second would move the task's address, and with it
+ the target a load balancer is pointed at.
+ """
+ from ministack.services import ecs as _ecs
+
+ class FakeContainer:
+ def __init__(self, ip):
+ self.attrs = {"NetworkSettings": {"Networks": {"n": {"IPAddress": ip}}}}
+
+ def reload(self):
+ pass
+
+ task = {
+ "taskArn": f"arn:aws:ecs:us-east-1:000000000000:task/c/first-owner-{mode}",
+ "_network_mode": mode,
+ "_subnet": "subnet-0a1b2c3d",
+ "attachments": [],
+ }
+ _ecs._record_task_ip(task, FakeContainer("172.30.0.21"), "n")
+ _ecs._record_task_ip(task, FakeContainer("172.30.0.22"), "n")
+
+ assert _ecs._task_ip(task) == "172.30.0.21"
+ assert len(task["attachments"]) == (1 if mode == "awsvpc" else 0)
+
+
def test_ecs_sync_service_targets_selects_tasks_by_group(monkeypatch):
"""Target selection must use the same predicate as the service's own recount.
diff --git a/tests/test_ecs_metadata.py b/tests/test_ecs_metadata.py
index 51d7c9c9..75fe6726 100644
--- a/tests/test_ecs_metadata.py
+++ b/tests/test_ecs_metadata.py
@@ -158,3 +158,91 @@ def test_trailing_slash_on_root_is_tolerated():
status, body = _call("GET", "/v4/tok-aaaaaaaa/")
assert status == 200
assert body["Name"] == "web"
+
+
+def test_status_follows_the_task_record():
+ """The endpoint follows the task record instead of always saying RUNNING.
+
+ A container reads ${ECS_CONTAINER_METADATA_URI_V4}/task while it is
+ starting, which since the async start is the whole image pull. Reporting
+ RUNNING there contradicts DescribeTasks for the length of that pull.
+ """
+ from ministack.services import ecs
+
+ arn = "arn:aws:ecs:us-east-1:000000000000:task/c/statusprobe01"
+ task = {
+ "taskArn": arn,
+ "desiredStatus": "RUNNING",
+ "lastStatus": "PENDING",
+ "containers": [{"name": "probe", "lastStatus": "PENDING"}],
+ }
+ ecs._tasks[arn] = task
+ _register("statusprobetoken01", arn, "probe", KnownStatus="PENDING")
+ try:
+ for status in ("PENDING", "ACTIVATING", "RUNNING"):
+ task["lastStatus"] = status
+ _, body = _call("GET", "/v4/statusprobetoken01/task")
+ assert body["KnownStatus"] == status
+
+ # A container's KnownStatus is its own, not the task's: on AWS a
+ # starting task serves "NONE" for itself and "RUNNING" for the
+ # container that is reading the endpoint.
+ task["lastStatus"] = "ACTIVATING"
+ task["containers"][0]["lastStatus"] = "RUNNING"
+ _, body = _call("GET", "/v4/statusprobetoken01/task")
+ assert body["KnownStatus"] == "ACTIVATING"
+ assert body["Containers"][0]["KnownStatus"] == "RUNNING"
+ assert _call("GET", "/v4/statusprobetoken01")[1]["KnownStatus"] == "RUNNING"
+
+ # desiredStatus is the task's on both, which is what a container polls
+ # to find out it is being shut down.
+ task["desiredStatus"] = "STOPPED"
+ _, body = _call("GET", "/v4/statusprobetoken01/task")
+ assert body["DesiredStatus"] == "STOPPED"
+ assert body["Containers"][0]["DesiredStatus"] == "STOPPED"
+ assert _call("GET", "/v4/statusprobetoken01")[1]["DesiredStatus"] == "STOPPED"
+
+ # The overlay is applied to a copy. The registry dicts are shared by
+ # every sibling container of a task, so writing the live status into
+ # them would make one read change what the next one serves. Reading the
+ # response back cannot show this, since json.loads already copies:
+ # assert on the registry itself.
+ assert ecs_metadata._TASKS[arn]["KnownStatus"] == "RUNNING"
+ assert ecs_metadata._TOKEN_TO_CONTAINER["statusprobetoken01"]["KnownStatus"] == "PENDING"
+ finally:
+ ecs._tasks.pop(arn, None)
+
+
+def test_status_falls_back_to_what_was_registered_when_the_task_is_gone():
+ """No task record, no overlay: the endpoint serves what was registered."""
+ arn = "arn:aws:ecs:us-east-1:000000000000:task/c/goneprobe01"
+ _register("goneprobetoken01", arn, "probe", KnownStatus="ACTIVATING")
+ _, body = _call("GET", "/v4/goneprobetoken01/task")
+ assert body["Containers"][0]["KnownStatus"] == "ACTIVATING"
+
+
+def test_status_finds_a_task_outside_the_default_account_and_region():
+ """The lookup is keyed off the task ARN, not off the request.
+
+ A container reaches this endpoint with a path token and no SigV4, so the
+ request resolves under the default account and region. Keying the task
+ store on those would miss every task created anywhere else, and the
+ endpoint would silently fall back to what it registered.
+ """
+ from ministack.services import ecs
+
+ account, region = "111122223333", "eu-central-1"
+ arn = f"arn:aws:ecs:{region}:{account}:task/c/otherprobe01"
+ ecs._tasks.set_scoped(account, region, arn, {
+ "taskArn": arn,
+ "desiredStatus": "RUNNING",
+ "lastStatus": "ACTIVATING",
+ "containers": [{"name": "probe", "lastStatus": "PENDING"}],
+ })
+ _register("otherprobetoken01", arn, "probe")
+ try:
+ _, body = _call("GET", "/v4/otherprobetoken01/task")
+ assert body["KnownStatus"] == "ACTIVATING"
+ assert body["Containers"][0]["KnownStatus"] == "PENDING"
+ finally:
+ ecs._tasks.pop_scoped(account, region, arn, None)