Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(iot): registered-certificate event on connect, retired endpoint types, $ topics (#1755)
iot-rocketa0f9fe474252
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7f276494..88f237b7 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,9 @@ Versioning follows [Semantic Versioning](https://semver.org/).### Fixed- **IAM — three calls are authorized against the action and resource AWS uses** — a Lambda Function URL invoke is checked against its function ARN, alias qualifier included, and supplies `lambda:FunctionUrlAuthType`, so the policy `grantInvokeUrl` writes matches. The WebSocket `@connections` API asks for `execute-api:ManageConnections` instead of `execute-api:Invoke`. `iot-jobs-data` operations are authorized under `iotjobsdata:`, except `StartCommandExecution`, which stays on `iot:`. A policy written for the old action names stops matching, as on AWS. Contributed by @iot-rocket.+- **IoT — the registered-certificate event comes from the device connect** — `RegisterCertificate` published `$aws/events/certificates/registered/{caCertificateId}`, and an mTLS connect with an unknown certificate got CONNACK 5 with nothing created. AWS publishes the event only on the connect: a certificate signed by an ACTIVE CA with auto-registration enabled is created `PENDING_ACTIVATION`, the event carries `sourceIp`, and the connection closes without a CONNACK. The listener now does the same, and `RegisterCertificate` with status `PENDING_ACTIVATION` is `CertificateStateException` 406. Contributed by @iot-rocket.+- **IoT — `DescribeEndpoint` refuses the retired `iot:Data` and `iot:Jobs` types** — both returned a hostname, so a client could keep using an endpoint type AWS no longer serves. AWS answers `InvalidRequestException` 400 and points to `iot:Data-ATS`; both types now get the same errors and messages. Contributed by @iot-rocket.+- **IoT Data — HTTPS `Publish` refuses the topics AWS refuses** — any topic starting with `$` was accepted, so a caller could forge `$aws/events/...` lifecycle events and fire the rules on them. AWS answers `InvalidRequestException` 400 "Topic can't start with $" outside the `$aws/rules/`, `$aws/things/`, `$aws/certificates/`, `$aws/provisioning-templates/`, `$aws/device_location/` and `$aws/commands/` families, and "Invalid publish to restricted topic using HTTP" for the jobs, Device Defender and commands topics, which are MQTT-only. MiniStack now returns the same errors; MQTT publishes are unchanged. Contributed by @iot-rocket.- **CloudFormation — `AWS::ECR::Repository` keeps what the template declares** — scanning and encryption settings were stored in the template's PascalCase and read back empty, lifecycle policy, repository policy and tags were dropped, `PutImage` into the repository failed and a deleted repository left its images and policies behind. The repository is now created, updated in place and deleted through the ECR store. Contributed by @iot-rocket.- **CloudFormation — a rolled-back replacement keeps the old resource** — an update handler deleted the predecessor as soon as the replacement existed, so when a later resource failed the rollback removed the replacement too and the stack recorded a resource that was gone. The predecessor is now deleted in the cleanup phase after the update succeeds, and a rollback deletes only the replacement, as measured on AWS. Contributed by @iot-rocket.### Addeddiff --git a/ministack/core/router.py b/ministack/core/router.pyindex 58bfae20..a4bd2fc3 100644--- a/ministack/core/router.py+++ b/ministack/core/router.py@@ -333,9 +333,9 @@ SERVICE_PATTERNS = {# routing would otherwise swallow it — and on the `iot` control plane# `GET /things/{t}/jobs` is a DIFFERENT operation (ListJobExecutionsForThing)# that answers with a different envelope. The pattern must cover both host- # spellings: `DescribeEndpoint(endpointType='iot:Jobs')` hands out- # `{prefix}.jobs.iot.{region}.{host}`, while the AWS Device SDK's jobs- # topics document `{prefix}.data.jobs.iot.{region}.{host}`. The SDK also+ # spellings: `{prefix}.jobs.iot.{region}.{host}`, the legacy iot:Jobs+ # endpoint, and `{prefix}.data.jobs.iot.{region}.{host}`, which the AWS+ # Device SDK's jobs topics document. The SDK also# signs with credential scope `iot-jobs-data` (botocore signingName), which# the scope early-return resolves via this key."iot-jobs-data": {diff --git a/ministack/services/iot.py b/ministack/services/iot.pyindex 965338d0..3e27c72c 100644--- a/ministack/services/iot.py+++ b/ministack/services/iot.py@@ -13,9 +13,10 @@ Implements the JSON/REST APIs under ``iot.{region}.amazonaws.com``:``DeleteCertificate``, ``AttachThingPrincipal`` / ``DetachThingPrincipal``- CA certificates + JITR: ``GetRegistrationCode`` / ``DeleteRegistrationCode``,``RegisterCACertificate``, ``DescribeCACertificate``, ``UpdateCACertificate``,- ``ListCACertificates``, ``DeleteCACertificate``; registering a device- certificate under a CA with auto-registration enabled publishes the AWS- JITR event to ``$aws/events/certificates/registered/{caCertificateId}``+ ``ListCACertificates``, ``DeleteCACertificate``; an mTLS connect with an+ unknown certificate signed by a CA with auto-registration enabled creates+ it PENDING_ACTIVATION and publishes the AWS JITR event to+ ``$aws/events/certificates/registered/{caCertificateId}``- Provisioning templates: ``CreateProvisioningTemplate``,``DescribeProvisioningTemplate``, ``ListProvisioningTemplates``,``UpdateProvisioningTemplate``, ``DeleteProvisioningTemplate`` —@@ -604,6 +605,17 @@ async def handle_request(# ---------------------------------------------------------------------------+# AWS refuses the legacy endpoint types; the jobs data plane is served from+# the iot:Data-ATS endpoint.+_RETIRED_ENDPOINT_TYPES = {+ "iot:Data": "iot:Data is not supported. Please use iot:Data-ATS instead.",+ "iot:Jobs": (+ "IoT Jobs and Commands APIs are now available through iot:Data-ATS "+ "endpoints instead of iot:Jobs endpoints. Please use iot:Data-ATS."+ ),+}++def _describe_endpoint(qp: dict) -> tuple:"""Return a per-account endpoint hostname.@@ -617,13 +629,14 @@ def _describe_endpoint(qp: dict) -> tuple:prefix = hashlib.sha256(account_id.encode("utf-8")).hexdigest()[:14]region = get_region()- if endpoint_type in ("iot:Data-ATS", "iot:Data", None):- suffix = "-ats" if endpoint_type != "iot:Data" else ""- host = f"{prefix}{suffix}.iot.{region}.{_MINISTACK_HOST}:{_GATEWAY_PORT}"+ if endpoint_type in _RETIRED_ENDPOINT_TYPES:+ return error_response_json(+ "InvalidRequestException", _RETIRED_ENDPOINT_TYPES[endpoint_type], 400+ )+ if endpoint_type == "iot:Data-ATS":+ host = f"{prefix}-ats.iot.{region}.{_MINISTACK_HOST}:{_GATEWAY_PORT}"elif endpoint_type == "iot:CredentialProvider":host = f"{prefix}.credentials.iot.{region}.{_MINISTACK_HOST}:{_GATEWAY_PORT}"- elif endpoint_type == "iot:Jobs":- host = f"{prefix}.jobs.iot.{region}.{_MINISTACK_HOST}:{_GATEWAY_PORT}"else:return error_response_json("InvalidRequestException",@@ -1339,6 +1352,64 @@ def _delete_provisioning_template(name: str) -> tuple:# ---------------------------------------------------------------------------+def _certificate_record(+ cert_id: str, cert_pem: str, status: str, ca_id: str | None = None+) -> dict:+ """The registry record of a device certificate, owned by the current scope."""+ return {+ "certificateId": cert_id,+ "certificateArn": _cert_arn(cert_id),+ "certificatePem": cert_pem, # verbatim+ "status": status,+ "creationDate": _now_epoch(),+ "ownedBy": get_account_id(),+ "caCertificateId": ca_id,+ "attachedThings": [],+ "attachedPolicies": [],+ }+++async def _publish_certificate_registered(+ account_id: str,+ region: str,+ record: dict,+ source_ip: str | None,+ registration_timestamp: str | None,+) -> None:+ """Publish the JITR event to ``$aws/events/certificates/registered/{caId}``.++ AWS sends it when a device connects with a certificate its CA+ auto-registers: ``certificateRegistrationTimestamp`` is null on the connect+ that created the certificate and the creation time in epoch milliseconds,+ as a string, on every later connect while it is still PENDING_ACTIVATION.+ """+ ca_id = record["caCertificateId"]+ event = {+ "certificateId": record["certificateId"],+ "caCertificateId": ca_id,+ "timestamp": int(time.time() * 1000),+ "certificateStatus": record["status"],+ "awsAccountId": account_id,+ "certificateRegistrationTimestamp": registration_timestamp,+ "sourceIp": source_ip,+ }+ try:+ await broker_publish(+ account_id,+ region,+ f"$aws/events/certificates/registered/{ca_id}",+ json.dumps(event).encode("utf-8"),+ qos=0,+ )+ except Exception:+ # The certificate is already registered at this point, so a broker+ # failure must not undo it; the iot-data publish path guards the same+ # call the same way.+ logger.warning(+ "JITR registered-event publish failed for CA %s", ca_id, exc_info=True+ )++def _create_keys_and_certificate(qp: dict) -> tuple:"""Generate a fresh keypair and sign a leaf certificate with the Local CA."""set_active = qp.get("setAsActive", "false").lower() == "true"@@ -1352,21 +1423,10 @@ def _create_keys_and_certificate(qp: dict) -> tuple:except RuntimeError as e:return error_response_json("InternalFailureException", str(e), 503)cert_id = get_certificate_id(cert_pem)- arn = _cert_arn(cert_id)- record = {- "certificateId": cert_id,- "certificateArn": arn,- "certificatePem": cert_pem,- "status": "ACTIVE" if set_active else "INACTIVE",- "creationDate": _now_epoch(),- "ownedBy": get_account_id(),- "caCertificateId": None,- "attachedThings": [],- "attachedPolicies": [],- }+ record = _certificate_record(cert_id, cert_pem, "ACTIVE" if set_active else "INACTIVE")_certificates[cert_id] = recordreturn json_response({- "certificateArn": arn,+ "certificateArn": record["certificateArn"],"certificateId": cert_id,"certificatePem": cert_pem,"keyPair": {@@ -1403,11 +1463,10 @@ async def _register_certificate(``caCertificatePem`` must name a CA registered via``RegisterCACertificate`` in this account/region that really signed the- leaf; anything else is a ``CertificateValidationException``. When that CA's- ``autoRegistrationStatus`` is ``ENABLE``, the AWS JITR lifecycle event is- published to ``$aws/events/certificates/registered/{caCertificateId}`` so- just-in-time-registration Lambdas subscribed via topic rules fire exactly- as on AWS.+ leaf; anything else is a ``CertificateValidationException``. Registering+ publishes no JITR event, whatever the CA's ``autoRegistrationStatus``: AWS+ sends that event only from a device connect (see ``_mtls_auto_register``),+ and refuses a registration with status PENDING_ACTIVATION."""cert_pem = payload.get("certificatePem") or qp.get("certificatePem")if not cert_pem:@@ -1415,6 +1474,12 @@ async def _register_certificate("InvalidRequestException", "certificatePem is required", 400)status = payload.get("status")+ if status == "PENDING_ACTIVATION":+ return error_response_json(+ "CertificateStateException",+ "Not allowed to register certificate with PENDING_ACTIVATION status",+ 406,+ )if without_ca:set_active = Falseelse:@@ -1461,55 +1526,10 @@ async def _register_certificate()if cert_id in _certificates:return _certificate_already_exists(cert_id)- record = {- "certificateId": cert_id,- "certificateArn": _cert_arn(cert_id),- "certificatePem": cert_pem, # verbatim- "status": status or ("ACTIVE" if set_active else "INACTIVE"),- "creationDate": _now_epoch(),- "ownedBy": get_account_id(),- "caCertificateId": ca_id,- "attachedThings": [],- "attachedPolicies": [],- }+ record = _certificate_record(+ cert_id, cert_pem, status or ("ACTIVE" if set_active else "INACTIVE"), ca_id+ )_certificates[cert_id] = record-- # JITR: the registered-certificate lifecycle event, fired when the- # referenced CA is ACTIVE and has auto-registration enabled — AWS- # auto-registers nothing under an INACTIVE CA. certificateStatus echoes- # the requested register status; real connect-triggered auto-registration- # carries PENDING_ACTIVATION, but here registration is always explicit —- # there is no mTLS first-connect path to auto-register from.- if (- ca- and ca.get("status") == "ACTIVE"- and ca.get("autoRegistrationStatus") == "ENABLE"- ):- now_ms = int(time.time() * 1000)- event = {- "certificateId": cert_id,- "caCertificateId": ca_id,- "timestamp": now_ms,- "certificateStatus": record["status"],- "awsAccountId": get_account_id(),- "certificateRegistrationTimestamp": str(now_ms),- }- try:- await broker_publish(- get_account_id(),- get_region(),- f"$aws/events/certificates/registered/{ca_id}",- json.dumps(event).encode("utf-8"),- qos=0,- )- except Exception:- # The certificate is already registered at this point, so a broker- # failure must not turn a successful registration into a 500 — the- # iot-data publish path guards the same call the same way.- logger.warning(- "JITR registered-event publish failed for CA %s", ca_id, exc_info=True- )-return json_response({"certificateArn": record["certificateArn"],"certificateId": cert_id,@@ -4981,6 +5001,25 @@ def _validate_publish_topic(topic: str) -> bool:return True+# The reserved topic space a client may publish into. AWS closes the connection+# of an MQTT client that publishes to any other topic starting with ``$``+# (measured for ``$aws/events/...``, ``$aws/jobs/...`` and ``$foo/...``; a+# publish under ``$aws/rules/`` and ``$aws/things/`` is acknowledged). The HTTPS+# Publish in iot_data.py applies the same list and answers 400.+RESERVED_PUBLISH_PREFIXES = (+ "$aws/rules/",+ "$aws/things/",+ "$aws/certificates/",+ "$aws/provisioning-templates/",+ "$aws/device_location/",+ "$aws/commands/",+)+++def _reserved_topic_publish_allowed(topic: str) -> bool:+ return not topic.startswith("$") or topic.startswith(RESERVED_PUBLISH_PREFIXES)++# ---------------------------------------------------------------------------# Broker public API (consumed by iot_data.py and handle_websocket)# ---------------------------------------------------------------------------@@ -6926,6 +6965,13 @@ class _WSSession:if self.protocol_version == MQTT_5:await self.send_bytes(_make_disconnect(RC5_TOPIC_NAME_INVALID))return False+ if not _reserved_topic_publish_allowed(topic):+ # AWS closes the connection without a PUBACK (measured over+ # MQTT 3.1.1; the MQTT 5 reason code is not measured).+ _broker_logger.warning("IoT broker: PUBLISH to a reserved topic closes the connection: %r", topic)+ if self.protocol_version == MQTT_5:+ await self.send_bytes(_make_disconnect(RC5_NOT_AUTHORIZED))+ return Falsepayload = body[off:]delivered = await broker_publish(self.account_id,@@ -7251,6 +7297,12 @@ async def handle_websocket(# mean anything and what real AWS IoT does. The ambiguous case cannot be# attributed at all: two registrations see byte-identical bytes, so any# tie-break would award the session to whoever registered the copy.+# * just-in-time registration, as on AWS: a certificate that is unknown, or+# still PENDING_ACTIVATION, and signed by a registered CA that is ACTIVE+# with auto-registration enabled is created PENDING_ACTIVATION in that CA's+# account and region, the registered event is published, and the connection+# is closed without a CONNACK. As on AWS this happens on the CONNECT packet;+# a handshake that sends no CONNECT registers nothing.## The TLS layer adds one constraint of its own: a presented chain is verified,# so a certificate from a CA the listener does not trust fails the handshake@@ -7613,12 +7665,16 @@ def mtls_schedule_restart() -> None:_mtls_logger.debug("IoT mTLS: could not schedule a restart: %s", e)-def _mtls_active_registrations(cert_id: str) -> list[tuple[str, str, dict]]:- """Every (account_id, region, record) holding this certificate ACTIVE."""+def _mtls_active_registrations(+ cert_id: str, status: str | None = "ACTIVE"+) -> list[tuple[str, str, dict]]:+ """Every (account_id, region, record) holding this certificate in ``status``+ (any status when None)."""return [(account_id, region, record)for (account_id, region, key), record in list(_certificates._data.items())- if key == cert_id and isinstance(record, dict) and record.get("status") == "ACTIVE"+ if key == cert_id and isinstance(record, dict)+ and (status is None or record.get("status") == status)]@@ -7661,21 +7717,84 @@ def _mtls_resolve_identity(der: bytes | None) -> tuple[str, str] | None:return None-async def _mtls_refuse(- reader: asyncio.StreamReader, writer: asyncio.StreamWriter, cert_id: str-) -> None:- """Answer an unusable client certificate with a "not authorized" CONNACK- (0x05, or reason code 0x87 for an MQTT 5 client), then close.+def _mtls_auto_registering_ca(account_id: str, region: str, ca_id: str | None) -> dict | None:+ """The CA record when it would auto-register a device certificate now."""+ ca = _ca_certificates.get_scoped(account_id, region, ca_id) if ca_id else None+ if (+ isinstance(ca, dict)+ and ca.get("status") == "ACTIVE"+ and ca.get("autoRegistrationStatus") == "ENABLE"+ and ca.get("certificateMode", "DEFAULT") == "DEFAULT"+ ):+ return ca+ return None- AWS closes the socket without a CONNACK unless just-in-time registration- applies. Diverging here is deliberate: an emulator's job is- diagnosability, "not authorized" is reported cleanly by every device SDK,- and a TLS-layer rejection could not tell the user "unknown" from- "deactivated".++def _mtls_auto_registering_signers(cert_pem: str) -> list[tuple[str, str, str]]:+ """Every (account_id, region, ca_id) of an auto-registering CA that signed ``cert_pem``."""+ return [+ (account_id, region, ca_id)+ for (account_id, region, ca_id), ca in list(_ca_certificates._data.items())+ if _mtls_auto_registering_ca(account_id, region, ca_id) is not None+ and certificate_is_signed_by(cert_pem, ca.get("certificatePem") or "")+ ]+++async def _mtls_auto_register(der: bytes, peername) -> bool:+ """Just-in-time registration for a presented certificate that no scope+ holds ACTIVE. True when it applied, and the caller closes without a+ CONNACK; False leaves the refusal to ``_mtls_refuse``.++ An unknown certificate is created PENDING_ACTIVATION under the one+ auto-registering CA that signed it. A certificate already+ PENDING_ACTIVATION under such a CA only publishes the event again. AWS+ does both on every such connect, and nothing for an INACTIVE certificate+ or a CA with auto-registration disabled."""- _mtls_logger.warning(- "IoT mTLS: refusing client certificate %s (%s)", cert_id, _mtls_refusal_reason(cert_id)+ cert_id = hashlib.sha256(der).hexdigest()+ held = _mtls_active_registrations(cert_id, status=None)+ if len(held) > 1:+ return False+ if held:+ account_id, region, record = held[0]+ if record.get("status") != "PENDING_ACTIVATION":+ return False+ if _mtls_auto_registering_ca(account_id, region, record.get("caCertificateId")) is None:+ return False+ registration_timestamp = str(int(record["creationDate"] * 1000))+ else:+ cert_pem = ssl.DER_cert_to_PEM_cert(der)+ # One signature check per auto-registering CA, off the broker's loop:+ # a device retrying with a refused certificate must not stall it.+ signers = await asyncio.to_thread(_mtls_auto_registering_signers, cert_pem)+ # The same CA registered in several scopes cannot be attributed, for+ # the same reason an ambiguous ACTIVE certificate is refused.+ if len(signers) != 1:+ return False+ account_id, region, ca_id = signers[0]+ with request_scope(account_id, region):+ record = _certificate_record(cert_id, cert_pem, "PENDING_ACTIVATION", ca_id)+ _certificates[cert_id] = record+ registration_timestamp = None+ _mtls_logger.info(+ "IoT mTLS: certificate %s is PENDING_ACTIVATION under CA %s in %s/%s; "+ "published the registered event and closing",+ cert_id,+ record["caCertificateId"],+ account_id,+ region,)+ source_ip = peername[0] if isinstance(peername, tuple) and peername else None+ await _publish_certificate_registered(+ account_id, region, record, source_ip, registration_timestamp+ )+ return True+++async def _mtls_first_packet(reader: asyncio.StreamReader) -> bytes | None:+ """The client's first complete MQTT packet, or None when the peer closes,+ sends nothing within ``_MTLS_REFUSE_READ_TIMEOUT`` or exceeds+ ``_MTLS_REFUSE_MAX_BYTES`` without completing one."""buffer = bytearray()try:while len(buffer) < _MTLS_REFUSE_MAX_BYTES:@@ -7683,7 +7802,7 @@ async def _mtls_refuse(reader.read(_MTLS_READ_CHUNK), timeout=_MTLS_REFUSE_READ_TIMEOUT)if not data:- return+ return Nonebuffer.extend(data)if len(buffer) < 2:continue@@ -7693,27 +7812,54 @@ async def _mtls_refuse(continueif len(buffer) < header_end + remaining:continue- if (buffer[0] >> 4) & 0x0F == PKT_CONNECT:- # The CONNACK has to match the client's protocol level: a v5- # client handed the two-byte 3.1.1 form dies in its decoder- # instead of reporting the refusal, and 0x05 is not a defined- # v5 reason code.- body = bytes(buffer[header_end:header_end + remaining])- try:- _proto_name, level_at = _read_string(body, 0)- protocol_level = body[level_at]- except (ValueError, IndexError):- protocol_level = MQTT_311- if protocol_level == MQTT_5:- connack = _make_connack(- return_code=RC5_NOT_AUTHORIZED, protocol_version=MQTT_5- )- else:- connack = _make_connack(return_code=CONNACK_311_NOT_AUTHORIZED)- writer.write(connack)- await writer.drain()- return+ return bytes(buffer[: header_end + remaining])except (asyncio.TimeoutError, OSError, ssl.SSLError):+ return None+ return None+++def _connect_protocol_level(packet: bytes) -> int | None:+ """The protocol level of a CONNECT packet; None for any other packet."""+ if (packet[0] >> 4) & 0x0F != PKT_CONNECT:+ return None+ remaining, header_end = _decode_remaining_length(packet, 1)+ body = packet[header_end : header_end + remaining]+ try:+ _proto_name, level_at = _read_string(body, 0)+ return body[level_at]+ except (ValueError, IndexError):+ return MQTT_311+++async def _mtls_refuse(+ writer: asyncio.StreamWriter, cert_id: str, protocol_level: int | None+) -> None:+ """Answer an unusable client certificate with a "not authorized" CONNACK+ (0x05, or reason code 0x87 for an MQTT 5 client) when the client sent a+ CONNECT, then close.++ AWS closes the socket without a CONNACK unless just-in-time registration+ applies. Diverging here is deliberate: an emulator's job is+ diagnosability, "not authorized" is reported cleanly by every device SDK,+ and a TLS-layer rejection could not tell the user "unknown" from+ "deactivated".+ """+ _mtls_logger.warning(+ "IoT mTLS: refusing client certificate %s (%s)", cert_id, _mtls_refusal_reason(cert_id)+ )+ if protocol_level is None:+ return+ # The CONNACK has to match the client's protocol level: a v5 client+ # handed the two-byte 3.1.1 form dies in its decoder instead of reporting+ # the refusal, and 0x05 is not a defined v5 reason code.+ if protocol_level == MQTT_5:+ connack = _make_connack(return_code=RC5_NOT_AUTHORIZED, protocol_version=MQTT_5)+ else:+ connack = _make_connack(return_code=CONNACK_311_NOT_AUTHORIZED)+ try:+ writer.write(connack)+ await writer.drain()+ except (OSError, ssl.SSLError):return@@ -7741,7 +7887,16 @@ async def _mtls_serve_conn(reader: asyncio.StreamReader, writer: asyncio.StreamWowner = _mtls_resolve_identity(der)if owner is None:try:- await _mtls_refuse(reader, writer, hashlib.sha256(der).hexdigest())+ # AWS registers on the CONNECT packet, not on the handshake: a+ # client that closes after the handshake, sends nothing, or sends+ # another packet first registers nothing (measured).+ packet = await _mtls_first_packet(reader)+ protocol_level = _connect_protocol_level(packet) if packet else None+ if protocol_level is not None and await _mtls_auto_register(+ der, writer.get_extra_info("peername")+ ):+ return+ await _mtls_refuse(writer, hashlib.sha256(der).hexdigest(), protocol_level)finally:await _mtls_close(writer)returndiff --git a/ministack/services/iot_data.py b/ministack/services/iot_data.pyindex b027a099..31c932ac 100644--- a/ministack/services/iot_data.py+++ b/ministack/services/iot_data.py@@ -19,6 +19,7 @@ from __future__ import annotationsimport copyimport jsonimport logging+import refrom urllib.parse import unquotefrom ministack.core.responses import (@@ -35,6 +36,14 @@ logger = logging.getLogger("iot_data")# AWS IoT topics: max 7 segments, max 256 UTF-8 bytes total._MAX_TOPIC_BYTES = 256+# HTTPS Publish to a topic starting with "$" is refused unless the topic sits+# under one of these reserved families. Within them, the device-side jobs,+# Device Defender and commands topics are MQTT-only.+_HTTP_RESERVED_PREFIXES = _iot_module.RESERVED_PUBLISH_PREFIXES+_HTTP_RESTRICTED_TOPIC_RE = re.compile(+ r"^\$aws/(things/[^/]+/(jobs|defender)/|commands/)"+)+def _validate_topic(topic: str) -> tuple | None:"""Return an error response if ``topic`` is invalid, else None."""@@ -54,6 +63,17 @@ def _validate_topic(topic: str) -> tuple | None:f"Topic exceeds {_MAX_TOPIC_BYTES} bytes",400,)+ if topic.startswith("$"):+ if not topic.startswith(_HTTP_RESERVED_PREFIXES):+ return error_response_json(+ "InvalidRequestException", "Topic can't start with $", 400+ )+ if _HTTP_RESTRICTED_TOPIC_RE.match(topic):+ return error_response_json(+ "InvalidRequestException",+ "Invalid publish to restricted topic using HTTP",+ 400,+ )return Nonediff --git a/ministack/services/iot_jobs_data.py b/ministack/services/iot_jobs_data.pyindex 814696c4..91eeaf58 100644--- a/ministack/services/iot_jobs_data.py+++ b/ministack/services/iot_jobs_data.py@@ -8,8 +8,8 @@ Implements the device-side REST surface of AWS IoT Jobs:Routing reaches us either through credential-scope detection (the SDK signsrequests with the ``iot-jobs-data`` scope — botocore signingName) or via the-host pattern ``{prefix}.jobs.iot.{region}.{host}``, which is what-``DescribeEndpoint(endpointType='iot:Jobs')`` hands out.+host pattern ``{prefix}.jobs.iot.{region}.{host}``, the shape of the legacy+``iot:Jobs`` endpoint.This module is a wire adapter: it parses requests, shapes responses, and logs.The job store, the execution store, and every rule of the execution statediff --git a/tests/conftest.py b/tests/conftest.pyindex a9db94de..ae4a0610 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -191,11 +191,10 @@ _SERIAL_TESTS = {"tests/test_iot_data.py::test_iot_rule_where_clause_gates_dispatch","tests/test_iot_data.py::test_iot_rule_where_topic_function_under_basic_ingest","tests/test_iot_data.py::test_iot_rule_where_or_clause_dispatches_either_branch",- "tests/test_iot_data.py::test_iot_jitr_registration_event_drives_a_topic_rule",# IoT Jobs data-plane routing test: a raw urllib GET with the advertised# `{prefix}.jobs.iot.{region}` Host and a tight 5s timeout, so cross-file# xdist pressure on the shared event loop makes it time out at random.- "tests/test_iot_jobs.py::test_iot_jobs_advertised_endpoint_host_reaches_the_data_plane",+ "tests/test_iot_jobs.py::test_iot_jobs_endpoint_host_reaches_the_data_plane",# ECS service task-spawn: with a Docker daemon present (CI has one) a task# whose container fails to start/exits under parallel container churn is set# STOPPED, so list_tasks (RUNNING-only) sees fewer than desiredCount. Passes@@ -273,6 +272,7 @@ _SERIAL_TESTS = {"tests/test_iot_data.py::test_mtls_inactive_cert_refused","tests/test_iot_data.py::test_mtls_ambiguous_cert_is_refused","tests/test_iot_data.py::test_mtls_registered_ca_chain_connects",+ "tests/test_iot_data.py::test_mtls_jitr_auto_registers_an_unknown_cert_without_connack","tests/test_iot_data.py::test_mtls_account_scoped_delivery","tests/test_iot_data.py::test_mtls_garbage_bytes_dropped","tests/test_iot_data.py::test_mtls_duplicate_client_id_evicts_first_connection",diff --git a/tests/test_iot.py b/tests/test_iot.pyindex ab607f9a..401b9ca1 100644--- a/tests/test_iot.py+++ b/tests/test_iot.py@@ -36,12 +36,23 @@ def test_iot_describe_endpoint_default_uses_data_ats(iot_client):assert "-ats.iot." in resp["endpointAddress"]-def test_iot_describe_endpoint_data_legacy(iot_client):- resp = iot_client.describe_endpoint(endpointType="iot:Data")- addr = resp["endpointAddress"]- # Legacy endpoint omits the -ats suffix.- assert ".iot." in addr- assert "-ats.iot." not in addr+ ("endpoint_type", "message"),+ [+ ("iot:Data", "iot:Data is not supported. Please use iot:Data-ATS instead."),+ (+ "iot:Jobs",+ "IoT Jobs and Commands APIs are now available through iot:Data-ATS "+ "endpoints instead of iot:Jobs endpoints. Please use iot:Data-ATS.",+ ),+ ],+)+def test_iot_describe_endpoint_retired_types_rejected(iot_client, endpoint_type, message):+ with pytest.raises(ClientError) as ei:+ iot_client.describe_endpoint(endpointType=endpoint_type)+ assert ei.value.response["Error"]["Code"] == "InvalidRequestException"+ assert ei.value.response["Error"]["Message"] == message+ assert ei.value.response["ResponseMetadata"]["HTTPStatusCode"] == 400def test_iot_describe_endpoint_unknown_type_rejected(iot_client):@@ -5281,90 +5292,139 @@ def test_iot_ca_certificate_registration_config_round_trip(iot_client):iot_client.delete_ca_certificate(certificateId=ca_id)-def test_iot_jitr_registered_event_published_when_auto_registration_enabled():- """Register a CA with auto-registration ENABLE, subscribe on- ``$aws/events/certificates/registered/#``, register a device cert under- that CA — the JITR lifecycle event arrives with certificateId +- caCertificateId and DescribeCertificate resolves the signing CA.+_JITR_ACCOUNT = "123456789012"- Runs in-process against the module broker (the WS route to the live- server needs ``*.localhost`` DNS this environment lacks), following the- broker-helper pattern above.- """- from ministack.core.responses import (- set_request_account_id,- set_request_region,++async def _jitr_scope_with_ca(iot_module, ca_pem: str, received: list, allow_auto=True) -> str:+ """Pin the JITR test scope, register ``ca_pem`` ACTIVE and collect every+ registered event into ``received``. Returns the CA id."""+ from ministack.core.responses import set_request_account_id, set_request_region++ set_request_account_id(_JITR_ACCOUNT)+ set_request_region(_TEST_REGION)+ qp = {"setAsActive": "true"}+ if allow_auto:+ qp["allowAutoRegistration"] = "true"+ status, _, body = await iot_module.handle_request(+ "POST", "/cacertificate", {}, json.dumps({"caCertificate": ca_pem}).encode(), qp+ )+ assert status == 200++ async def _collect(topic, payload, qos):+ received.append((topic, json.loads(payload)))++ await iot_module.broker_subscribe(+ _JITR_ACCOUNT, _TEST_REGION, "$aws/events/certificates/registered/#", _collect)+ return json.loads(body)["certificateId"]++[email protected]("without_ca", [False, True], ids=["with-ca", "without-ca"])[email protected]("status", ["ACTIVE", "INACTIVE", "PENDING_ACTIVATION"])+def test_iot_register_certificate_publishes_no_registered_event(status, without_ca):+ """RegisterCertificate under an ACTIVE CA with auto-registration enabled+ publishes nothing, as on AWS, where only a device connect sends the+ registered event. PENDING_ACTIVATION is refused outright, by+ RegisterCertificateWithoutCA as well (measured)."""from ministack.services import iot as iot_moduleca_pem, leaf_pem = _generate_ca_and_leaf()- account_id = "123456789012"received: list = []async def _run():- set_request_account_id(account_id)- set_request_region(_TEST_REGION)-- status, _, body = await iot_module.handle_request(+ await _jitr_scope_with_ca(iot_module, ca_pem, received)+ payload = {"certificatePem": leaf_pem, "status": status}+ if not without_ca:+ payload["caCertificatePem"] = ca_pem+ code, _, body = await iot_module.handle_request("POST",- "/cacertificate",+ "/certificate/register-no-ca" if without_ca else "/certificate/register",+ {},+ json.dumps(payload).encode(),{},- json.dumps({"caCertificate": ca_pem}).encode(),- {"setAsActive": "true", "allowAutoRegistration": "true"},)- assert status == 200- ca_id = json.loads(body)["certificateId"]+ return code, json.loads(body)- async def _collect(topic, payload, qos):- received.append((topic, payload, qos))+ try:+ code, body = asyncio.run(_run())+ finally:+ iot_module.reset()+ iot_module.broker_reset()+ if status == "PENDING_ACTIVATION":+ assert code == 406+ assert body["__type"] == "CertificateStateException"+ assert body["message"] == "Not allowed to register certificate with PENDING_ACTIVATION status"+ else:+ assert code == 200+ assert received == []+++def test_iot_jitr_auto_registration_on_connect(monkeypatch):+ """The registry half of just-in-time registration on an mTLS connect; the+ wire half (closed without a CONNACK) is in test_iot_data.py.++ First connect: the certificate is created PENDING_ACTIVATION under its CA+ and the event carries a null certificateRegistrationTimestamp. A repeat+ connect publishes again with the creation time as a string. An INACTIVE+ certificate, or a CA with auto-registration disabled, gets nothing. A+ broker failure does not undo the registration.+ """+ import ssl- await iot_module.broker_subscribe(- account_id,- _TEST_REGION,- "$aws/events/certificates/registered/#",- _collect,- )+ from ministack.services import iot as iot_module- status, _, body = await iot_module.handle_request(- "POST",- "/certificate/register",- {},- json.dumps({- "certificatePem": leaf_pem,- "caCertificatePem": ca_pem,- "status": "PENDING_ACTIVATION",- }).encode(),- {},- )- assert status == 200- cert_id = json.loads(body)["certificateId"]+ ca_pem, leaf_pem = _generate_ca_and_leaf()+ off_ca_pem, off_leaf_pem = _generate_ca_and_leaf()+ cert_id = iot_module.get_certificate_id(leaf_pem)+ peer = ("192.0.2.10", 50000)+ received: list = []- assert len(received) == 1- topic, payload, _qos = received[0]- assert topic == f"$aws/events/certificates/registered/{ca_id}"- event = json.loads(payload)- assert event["certificateId"] == cert_id- assert event["caCertificateId"] == ca_id- assert event["certificateStatus"] == "PENDING_ACTIVATION"- assert event["awsAccountId"] == account_id- assert isinstance(event["timestamp"], int)- assert event["certificateRegistrationTimestamp"] == str(event["timestamp"])-- # A JITR Lambda resolves the signing CA via DescribeCertificate.- status, _, body = await iot_module.handle_request(- "GET", f"/certificates/{cert_id}", {}, b"", {}- )- assert status == 200- assert (- json.loads(body)["certificateDescription"]["caCertificateId"] == ca_id- )+ async def _boom(*_args, **_kwargs):+ raise RuntimeError("broker down")++ async def _run():+ ca_id = await _jitr_scope_with_ca(iot_module, ca_pem, received)+ await _jitr_scope_with_ca(iot_module, off_ca_pem, [], allow_auto=False)+ der = ssl.PEM_cert_to_DER_cert(leaf_pem)++ assert await iot_module._mtls_auto_register(der, peer)+ record = iot_module._certificates[cert_id]+ assert record["status"] == "PENDING_ACTIVATION"+ assert record["caCertificateId"] == ca_id+ assert await iot_module._mtls_auto_register(der, peer)++ off_der = ssl.PEM_cert_to_DER_cert(off_leaf_pem)+ assert not await iot_module._mtls_auto_register(off_der, peer)+ assert iot_module.get_certificate_id(off_leaf_pem) not in iot_module._certificates++ record["status"] = "INACTIVE"+ assert not await iot_module._mtls_auto_register(der, peer)++ record["status"] = "PENDING_ACTIVATION"+ monkeypatch.setattr(iot_module, "broker_publish", _boom)+ assert await iot_module._mtls_auto_register(der, peer)+ return ca_id, recordtry:- asyncio.run(_run())+ ca_id, record = asyncio.run(_run())finally:iot_module.reset()iot_module.broker_reset()+ topic = f"$aws/events/certificates/registered/{ca_id}"+ assert [t for t, _event in received] == [topic, topic]+ first, second = received[0][1], received[1][1]+ assert isinstance(first.pop("timestamp"), int)+ assert first == {+ "certificateId": cert_id,+ "caCertificateId": ca_id,+ "certificateStatus": "PENDING_ACTIVATION",+ "awsAccountId": _JITR_ACCOUNT,+ "certificateRegistrationTimestamp": None,+ "sourceIp": "192.0.2.10",+ }+ assert second["certificateRegistrationTimestamp"] == str(int(record["creationDate"] * 1000))+# ----------------------------------------------------------------------# Shadow-over-MQTT bridge (white-box).@@ -5499,71 +5559,6 @@ def test_shadow_mqtt_update_reported_in_sync_emits_no_delta():iot_module.broker_reset()-def test_iot_jitr_no_event_when_auto_registration_disabled():- """A CA registered without ``allowAutoRegistration`` links the device cert- (``caCertificateId``) but publishes no registered event."""- from ministack.core.responses import (- set_request_account_id,- set_request_region,- )- from ministack.services import iot as iot_module-- ca_pem, leaf_pem = _generate_ca_and_leaf()- account_id = "123456789012"- received: list = []-- async def _run():- set_request_account_id(account_id)- set_request_region(_TEST_REGION)-- status, _, body = await iot_module.handle_request(- "POST",- "/cacertificate",- {},- json.dumps({"caCertificate": ca_pem}).encode(),- {"setAsActive": "true"},- )- assert status == 200- ca_id = json.loads(body)["certificateId"]-- async def _collect(topic, payload, qos):- received.append((topic, payload, qos))-- await iot_module.broker_subscribe(- account_id,- _TEST_REGION,- "$aws/events/certificates/registered/#",- _collect,- )-- status, _, body = await iot_module.handle_request(- "POST",- "/certificate/register",- {},- json.dumps({- "certificatePem": leaf_pem,- "caCertificatePem": ca_pem,- }).encode(),- {},- )- assert status == 200- cert_id = json.loads(body)["certificateId"]- assert received == []-- status, _, body = await iot_module.handle_request(- "GET", f"/certificates/{cert_id}", {}, b"", {}- )- assert (- json.loads(body)["certificateDescription"]["caCertificateId"] == ca_id- )-- try:- asyncio.run(_run())- finally:- iot_module.reset()- iot_module.broker_reset()--def test_shadow_mqtt_update_rejections():"""Invalid JSON → 400 (no token); version conflict → 409 with the token."""from ministack.services import iot as iot_module@@ -5620,13 +5615,12 @@ def test_iot_register_certificate_under_ca_links_ca_certificate_id(iot_client):cert_id = iot_client.register_certificate(certificatePem=leaf_pem,caCertificatePem=ca_pem,- status="PENDING_ACTIVATION",)["certificateId"]desc = iot_client.describe_certificate(certificateId=cert_id)["certificateDescription"]assert desc["caCertificateId"] == ca_id- assert desc["status"] == "PENDING_ACTIVATION"+ assert desc["status"] == "INACTIVE"iot_client.delete_certificate(certificateId=cert_id)finally:iot_client.update_ca_certificate(certificateId=ca_id, newStatus="INACTIVE")@@ -5694,63 +5688,6 @@ def test_iot_register_certificate_rejects_a_leaf_from_another_ca(iot_client):iot_client.delete_ca_certificate(certificateId=ca_id)-def test_iot_jitr_no_event_for_a_certificate_from_another_ca():- """The rejection happens before anything is published, so no JITR consumer- ever sees an event attributing a foreign certificate to its CA."""- from ministack.core.responses import (- set_request_account_id,- set_request_region,- )- from ministack.services import iot as iot_module-- _ca_x_pem, leaf_pem = _generate_ca_and_leaf()- ca_y_pem, _leaf_y = _generate_ca_and_leaf()- account_id = "123456789012"- received: list = []-- async def _run():- set_request_account_id(account_id)- set_request_region(_TEST_REGION)-- status, _, _body = await iot_module.handle_request(- "POST",- "/cacertificate",- {},- json.dumps({"caCertificate": ca_y_pem}).encode(),- {"setAsActive": "true", "allowAutoRegistration": "true"},- )- assert status == 200-- async def _collect(topic, payload, qos):- received.append((topic, payload, qos))-- await iot_module.broker_subscribe(- account_id,- _TEST_REGION,- "$aws/events/certificates/registered/#",- _collect,- )-- status, _, body = await iot_module.handle_request(- "POST",- "/certificate/register",- {},- json.dumps(- {"certificatePem": leaf_pem, "caCertificatePem": ca_y_pem}- ).encode(),- {},- )- assert status == 400- assert json.loads(body)["__type"] == "CertificateValidationException"- assert received == []-- try:- asyncio.run(_run())- finally:- iot_module.reset()- iot_module.broker_reset()--def test_iot_update_ca_certificate_applies_a_body_only_status(iot_client):"""newStatus / newAutoRegistrationStatus are modeled in the query string,but a raw caller that sends them in the JSON body must not get a 200 that@@ -5795,141 +5732,6 @@ def test_iot_update_ca_certificate_applies_a_body_only_status(iot_client):iot_client.delete_ca_certificate(certificateId=ca_id)-def test_iot_jitr_event_requires_an_active_ca():- """Auto-registration is gated on the CA's own status: an INACTIVE CA with- allowAutoRegistration publishes nothing (AWS registers nothing under one),- and activating the same CA starts the events flowing."""- from ministack.core.responses import (- set_request_account_id,- set_request_region,- )- from ministack.services import iot as iot_module-- ca_pem, (leaf_pem, leaf_pem2) = _generate_ca_and_leaves(2)- account_id = "123456789012"- received: list = []-- async def _run():- set_request_account_id(account_id)- set_request_region(_TEST_REGION)-- # INACTIVE (no setAsActive) but auto-registration ENABLE.- status, _, body = await iot_module.handle_request(- "POST",- "/cacertificate",- {},- json.dumps({"caCertificate": ca_pem}).encode(),- {"allowAutoRegistration": "true"},- )- assert status == 200- ca_id = json.loads(body)["certificateId"]-- async def _collect(topic, payload, qos):- received.append((topic, payload, qos))-- await iot_module.broker_subscribe(- account_id,- _TEST_REGION,- "$aws/events/certificates/registered/#",- _collect,- )-- status, _, _body = await iot_module.handle_request(- "POST",- "/certificate/register",- {},- json.dumps(- {"certificatePem": leaf_pem, "caCertificatePem": ca_pem}- ).encode(),- {},- )- assert status == 200- assert received == []-- # Activate the CA — the very next registration fires.- status, _, _body = await iot_module.handle_request(- "PUT", f"/cacertificate/{ca_id}", {}, b"", {"newStatus": "ACTIVE"}- )- assert status == 200-- status, _, body = await iot_module.handle_request(- "POST",- "/certificate/register",- {},- json.dumps(- {"certificatePem": leaf_pem2, "caCertificatePem": ca_pem}- ).encode(),- {},- )- assert status == 200- cert_id2 = json.loads(body)["certificateId"]-- assert len(received) == 1- topic, payload, _qos = received[0]- assert topic == f"$aws/events/certificates/registered/{ca_id}"- assert json.loads(payload)["certificateId"] == cert_id2-- try:- asyncio.run(_run())- finally:- iot_module.reset()- iot_module.broker_reset()---def test_iot_jitr_registration_survives_a_broker_failure(monkeypatch):- """The certificate is committed before the event is published, so a broker- that raises must not turn a successful registration into a 500."""- from ministack.core.responses import (- set_request_account_id,- set_request_region,- )- from ministack.services import iot as iot_module-- ca_pem, leaf_pem = _generate_ca_and_leaf()- account_id = "123456789012"-- async def _boom(*_args, **_kwargs):- raise RuntimeError("broker down")-- async def _run():- set_request_account_id(account_id)- set_request_region(_TEST_REGION)-- status, _, _body = await iot_module.handle_request(- "POST",- "/cacertificate",- {},- json.dumps({"caCertificate": ca_pem}).encode(),- {"setAsActive": "true", "allowAutoRegistration": "true"},- )- assert status == 200-- monkeypatch.setattr(iot_module, "broker_publish", _boom)- status, _, body = await iot_module.handle_request(- "POST",- "/certificate/register",- {},- json.dumps(- {"certificatePem": leaf_pem, "caCertificatePem": ca_pem}- ).encode(),- {},- )- assert status == 200- cert_id = json.loads(body)["certificateId"]-- status, _, body = await iot_module.handle_request(- "GET", f"/certificates/{cert_id}", {}, b"", {}- )- assert status == 200- assert json.loads(body)["certificateDescription"]["certificateId"] == cert_id-- try:- asyncio.run(_run())- finally:- iot_module.reset()- iot_module.broker_reset()--def test_iot_update_ca_certificate_rejects_invalid_enum_values(iot_client):"""The two update fields validate their enums with a 400.diff --git a/tests/test_iot_data.py b/tests/test_iot_data.pyindex a69933a1..1e9b41f3 100644--- a/tests/test_iot_data.py+++ b/tests/test_iot_data.py@@ -111,6 +111,39 @@ def test_iot_data_publish_oversized_topic_400():assert e.code == 400+_NO_DOLLAR = "Topic can't start with $"+_MQTT_ONLY = "Invalid publish to restricted topic using HTTP"+++ ("topic", "message"),+ [+ ("$aws/events/certificates/registered/x", _NO_DOLLAR),+ ("$AWS/events/certificates/registered/x", _NO_DOLLAR),+ ("$aws/jobs/x", _NO_DOLLAR),+ ("$aws/rules", _NO_DOLLAR),+ ("$foo/bar", _NO_DOLLAR),+ ("$aws/things/t1/jobs/get", _MQTT_ONLY),+ ("$aws/things/t1/defender/metrics/json", _MQTT_ONLY),+ ("$aws/commands/things/t1/executions/1/response/json", _MQTT_ONLY),+ ("$aws/rules/no_such_rule/x", None),+ ("$aws/things/t1/jobs", None),+ ],+)+def test_iot_data_publish_reserved_topics(iot_data_client, topic, message):+ """HTTPS Publish to a "$" topic is refused outside the reserved families+ and for the MQTT-only jobs, Device Defender and commands topics."""+ if message is None:+ resp = iot_data_client.publish(topic=topic, payload=b"{}")+ assert resp["ResponseMetadata"]["HTTPStatusCode"] == 200+ return+ with pytest.raises(ClientError) as ei:+ iot_data_client.publish(topic=topic, payload=b"{}")+ assert ei.value.response["Error"]["Code"] == "InvalidRequestException"+ assert ei.value.response["Error"]["Message"] == message+ assert ei.value.response["ResponseMetadata"]["HTTPStatusCode"] == 400++# ---------------------------------------------------------------------------# MQTT-over-WebSocket round-trip# ---------------------------------------------------------------------------@@ -1369,55 +1402,8 @@ def test_search_index_connectivity_is_isolated_across_accounts_and_regions():finally:owner.delete_thing(thingName=thing)owner_eu.delete_thing(thingName=thing)-def test_iot_jitr_registration_event_drives_a_topic_rule(iot_client, lam, sqs):- """The JITR lifecycle event is a real broker publish, so a topic rule on- ``$aws/events/certificates/registered/{caId}`` hands it to a Lambda — the- shape a just-in-time-registration stack actually deploys.-- The rule names this test's CA id rather than the ``+`` wildcard: the- account's registered-certificate topics are shared, so a wildcard rule- living for the length of this test also catches the registrations other- tests make on other xdist workers, and ``_poll_sink`` would return- whichever event landed first."""- pytest.importorskip("cryptography")- from ministack.core.x509_utils import generate_ca, sign_leaf_certificate- ca_pem, ca_key_pem = generate_ca(common_name=_unique("jitr-rule-ca"))- leaf_pem, _priv, _pub = sign_leaf_certificate(- ca_cert_pem=ca_pem,- ca_key_pem=ca_key_pem,- common_name=_unique("jitr-rule-device"),- )- sink = sqs.create_queue(QueueName=_unique("jitr-sink"))["QueueUrl"]- fn_arn = _make_sink_lambda(lam, sink)- rule = _unique("jitr").replace("-", "_")- ca_id = iot_client.register_ca_certificate(- caCertificate=ca_pem, setAsActive=True, allowAutoRegistration=True- )["certificateId"]- iot_client.create_topic_rule(- ruleName=rule,- topicRulePayload={- "sql": f"SELECT * FROM '$aws/events/certificates/registered/{ca_id}'",- "actions": [{"lambda": {"functionArn": fn_arn}}],- },- )- try:- cert_id = iot_client.register_certificate(- certificatePem=leaf_pem,- caCertificatePem=ca_pem,- status="PENDING_ACTIVATION",- )["certificateId"]- event = _poll_sink(sqs, sink)- assert event is not None, "no JITR event reached the rule's Lambda"- assert event["certificateId"] == cert_id- assert event["caCertificateId"] == ca_id- assert event["certificateStatus"] == "PENDING_ACTIVATION"- iot_client.delete_certificate(certificateId=cert_id)- finally:- iot_client.delete_topic_rule(ruleName=rule)- iot_client.update_ca_certificate(certificateId=ca_id, newStatus="INACTIVE")- iot_client.delete_ca_certificate(certificateId=ca_id)# ---------------------------------------------------------------------------# Topic-rule `sqs` action (publish → rule → SQS queue)# ---------------------------------------------------------------------------@@ -2522,6 +2508,33 @@ def test_mqtt5_qos1_puback_carries_reason_code():assert matched == b"\x00\x08\x00\x00", "packet id 8, Success, no properties"[email protected]("topic,acknowledged", [+ ("$aws/rules/no-such-rule/x", True),+ ("$aws/things/some-thing/shadow/update", True),+ ("$aws/events/certificates/registered/abc", False),+ ("$aws/jobs/abc", False),+ ("$foo/abc", False),+], ids=["rules", "shadow", "events", "jobs", "other-dollar"])+def test_mqtt_publish_to_reserved_topic_closes_the_connection(topic, acknowledged):+ """AWS acknowledges a publish under $aws/rules/ and $aws/things/ and closes+ the connection of a client that publishes to any other $ topic (measured+ over MQTT 3.1.1); the HTTPS Publish applies the same list."""++ async def scenario():+ async with _connect(MQTT_311) as (pub, _c):+ await pub.send(_make_publish(topic, b"x", qos=1, packet_id=5))+ try:+ _flags, body = await pub.await_packet(PKT_PUBACK, timeout=3)+ return ("puback", body)+ except (websockets.exceptions.ConnectionClosed, OSError) as e:+ return ("closed", type(e).__name__)++ kind, detail = _run(scenario())+ assert kind == ("puback" if acknowledged else "closed"), (kind, detail)+ if acknowledged:+ assert detail == b"\x00\x05"++def test_mqtt311_qos1_puback_stays_two_bytes():"""The 3.1.1 PUBACK gains neither a reason code nor properties."""topic = _unique("v311/qos1")@@ -3551,6 +3564,82 @@ def test_mtls_registered_ca_chain_connects(broker, tmp_path):peer.close()+def test_mtls_jitr_auto_registers_an_unknown_cert_without_connack(broker, tmp_path):+ """Just-in-time registration, as on AWS: an unknown certificate signed by a+ CA with auto-registration enabled is created PENDING_ACTIVATION, the+ registered event is published and the connection closes without a CONNACK.+ A repeat connect publishes again, now with the creation time. A TLS+ handshake that sends no CONNECT registers nothing (AWS registers on the+ packet, not on the handshake). With auto-registration disabled the refusal+ stays CONNACK 5 and nothing is created."""+ from ministack.core.x509_utils import generate_ca, get_certificate_id, sign_leaf_certificate++ iot = broker.client("iot")+ ca_pem, ca_key = generate_ca(common_name=_unique("jitr-ca"))+ ca_id = iot.register_ca_certificate(+ caCertificate=ca_pem, setAsActive=True, allowAutoRegistration=True+ )["certificateId"]+ leaf_pem, leaf_key, _public = sign_leaf_certificate(ca_pem, ca_key, common_name="jitr-device")+ cert_id = get_certificate_id(leaf_pem)+ topic = f"$aws/events/certificates/registered/{ca_id}"+ try:++ listener = _Peer(_mtls_connect(broker, None, None, tmp_path))+ events = []+ try:+ _assert_connack(listener.connect(_unique("jitr-listener")))+ listener.subscribe(topic)+ try:+ _Peer(_mtls_connect(broker, leaf_pem, leaf_key, tmp_path)).close()+ except OSError:+ pass+ assert listener.next_publish(timeout=2.0) is None, "handshake alone registered"+ with pytest.raises(ClientError) as exc:+ iot.describe_certificate(certificateId=cert_id)+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"+ for attempt in ("first", "repeat"):+ assert _refused_below_mqtt(+ broker, leaf_pem, leaf_key, tmp_path, _unique(f"jitr-{attempt}")+ ), f"{attempt} connect got a CONNACK"+ events.append(listener.next_publish())+ finally:+ listener.close()++ assert None not in events, events+ assert [t for t, _payload in events] == [topic, topic]+ first, repeat = (json.loads(payload) for _t, payload in events)+ assert first["certificateId"] == cert_id+ assert first["caCertificateId"] == ca_id+ assert first["certificateStatus"] == "PENDING_ACTIVATION"+ assert first["certificateRegistrationTimestamp"] is None+ assert first["sourceIp"] == "127.0.0.1"+ assert isinstance(repeat["certificateRegistrationTimestamp"], str)+ desc = iot.describe_certificate(certificateId=cert_id)["certificateDescription"]+ assert desc["status"] == "PENDING_ACTIVATION"+ assert desc["caCertificateId"] == ca_id++ iot.update_ca_certificate(certificateId=ca_id, newAutoRegistrationStatus="DISABLE")+ off_pem, off_key, _public = sign_leaf_certificate(ca_pem, ca_key, common_name="jitr-off")+ peer = _Peer(_mtls_connect(broker, off_pem, off_key, tmp_path))+ try:+ _assert_connack(peer.connect(_unique("jitr-off")), return_code=5)+ finally:+ peer.close()+ with pytest.raises(ClientError) as exc:+ iot.describe_certificate(certificateId=get_certificate_id(off_pem))+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"+ finally:+ # The auto-registered certificate and the CA must not outlive the test.+ for cid in (cert_id,):+ try:+ iot.update_certificate(certificateId=cid, newStatus="INACTIVE")+ iot.delete_certificate(certificateId=cid, forceDelete=True)+ except ClientError:+ pass+ iot.update_ca_certificate(certificateId=ca_id, newStatus="INACTIVE")+ iot.delete_ca_certificate(certificateId=ca_id)++def test_mtls_account_scoped_delivery(broker, tmp_path):"""The client certificate decides the tenant: a device whose certificate isowned by 111111111111 sees that account's traffic and nobody else's."""diff --git a/tests/test_iot_jobs.py b/tests/test_iot_jobs.pyindex de321e6e..411325dd 100644--- a/tests/test_iot_jobs.py+++ b/tests/test_iot_jobs.py@@ -447,9 +447,8 @@ def test_iot_jobs_fleet_completes_only_when_every_execution_is_terminal(# ----------------------------------------------------------------------------def test_iot_jobs_advertised_endpoint_host_reaches_the_data_plane(iot_client):- """The documented device flow: `DescribeEndpoint(endpointType='iot:Jobs')`- hands out `{prefix}.jobs.iot.{region}`, and a request carrying that Host+def test_iot_jobs_endpoint_host_reaches_the_data_plane(iot_client):+ """A request carrying the jobs endpoint Host `{prefix}.jobs.iot.{region}`must land on the jobs data plane. Routed to the `iot` control planeinstead, `GET /things/{t}/jobs` is ListJobExecutionsForThing and silentlyanswers a different envelope."""@@ -459,10 +458,7 @@ def test_iot_jobs_advertised_endpoint_host_reaches_the_data_plane(iot_client):thing_arn = _create_thing(iot_client, thing)iot_client.create_job(jobId=job_id, targets=[thing_arn], document=_DOCUMENT)- endpoint = iot_client.describe_endpoint(endpointType="iot:Jobs")[- "endpointAddress"- ]- assert ".jobs.iot." in endpoint+ endpoint = "a1b2c3.jobs.iot.us-east-1.localhost"request = urllib.request.Request(f"{ENDPOINT}/things/{quote(thing)}/jobs",method="GET",@@ -1238,11 +1234,36 @@ def test_iot_jobs_lists_do_not_paginate_and_return_no_token(iot_client):# ---------------------------------------------------------------------------# Jobs over MQTT: the reserved $aws/things/<t>/jobs/# bridge# ----------------------------------------------------------------------------from test_iot_data import _collect_shadow_frames # noqa: E402+import asyncio # noqa: E402++from conftest import patch_endpoint_dns # noqa: E402+from test_iot_data import ( # noqa: E402+ _WS_HANDSHAKE_TIMEOUT,+ _collect_shadow_frames,+ _make_publish,+ _mqtt_connect,+ _mqtt_disconnect,+)_DOCUMENT_OBJECT = json.loads(_DOCUMENT)+def _mqtt_publish(topic, payload):+ """Publish at QoS 1 over the MQTT-over-WebSocket broker and wait for the+ PUBACK. The jobs topics are MQTT-only: AWS refuses them over HTTPS."""++ async def _run():+ ws = await _mqtt_connect(_unique("jobs-device"))+ try:+ await ws.send(_make_publish(topic, payload, qos=1, packet_id=1))+ await asyncio.wait_for(ws.recv(), timeout=_WS_HANDSHAKE_TIMEOUT)+ finally:+ await _mqtt_disconnect(ws)++ with patch_endpoint_dns():+ asyncio.run(_run())++def _frames_by_topic(received):return {topic: json.loads(payload) for topic, payload in received}@@ -1285,7 +1306,7 @@ def test_jobs_mqtt_create_job_notifies_and_get_lists(iot_client, iot_data_clientreceived = _collect_shadow_frames(f"{base}/get/accepted",- lambda: iot_data_client.publish(+ lambda: _mqtt_publish(topic=f"{base}/get",payload=json.dumps({"clientToken": "tok-get"}).encode(),),@@ -1351,7 +1372,7 @@ def test_jobs_mqtt_start_next_of_first_job_is_silent(iot_client, iot_data_clientreceived = _collect_shadow_frames(f"{base}/#",- lambda: iot_data_client.publish(+ lambda: _mqtt_publish(topic=f"{base}/start-next",payload=json.dumps({"clientToken": "tok-sn"}).encode(),),@@ -1391,13 +1412,13 @@ def test_jobs_mqtt_update_include_flags_and_terminal_notify()_collect_shadow_frames(f"{base}/start-next/accepted",- lambda: iot_data_client.publish(topic=f"{base}/start-next", payload=b"{}"),+ lambda: _mqtt_publish(topic=f"{base}/start-next", payload=b"{}"),want=1,)received = _collect_shadow_frames(f"{base}/{job_id}/update/accepted",- lambda: iot_data_client.publish(+ lambda: _mqtt_publish(topic=f"{base}/{job_id}/update",payload=json.dumps({@@ -1419,7 +1440,7 @@ def test_jobs_mqtt_update_include_flags_and_terminal_notify(received = _collect_shadow_frames(f"{base}/#",- lambda: iot_data_client.publish(+ lambda: _mqtt_publish(topic=f"{base}/{job_id}/update",payload=json.dumps({@@ -1466,7 +1487,7 @@ def test_jobs_mqtt_version_mismatch_rejected(iot_client, iot_data_client):received = _collect_shadow_frames(f"{base}/{job_id}/update/rejected",- lambda: iot_data_client.publish(+ lambda: _mqtt_publish(topic=f"{base}/{job_id}/update",payload=json.dumps({@@ -1495,7 +1516,7 @@ def test_jobs_mqtt_update_unknown_job_rejected(iot_client, iot_data_client):received = _collect_shadow_frames(f"{base}/no-such-job/update/rejected",- lambda: iot_data_client.publish(+ lambda: _mqtt_publish(topic=f"{base}/no-such-job/update",payload=json.dumps({"status": "SUCCEEDED", "clientToken": "tok-x"}).encode(),),@@ -1518,7 +1539,7 @@ def test_jobs_mqtt_next_sentinel_get(iot_client, iot_data_client):received = _collect_shadow_frames(f"{base}/$next/get/accepted",- lambda: iot_data_client.publish(+ lambda: _mqtt_publish(topic=f"{base}/$next/get",payload=json.dumps({"clientToken": "tok-next"}).encode(),),@@ -1534,7 +1555,7 @@ def test_jobs_mqtt_next_sentinel_get(iot_client, iot_data_client):received = _collect_shadow_frames(f"{base}/get/accepted",- lambda: iot_data_client.publish(topic=f"{base}/get", payload=b"{}"),+ lambda: _mqtt_publish(topic=f"{base}/get", payload=b"{}"),want=1,)doc = _frames_by_topic(received)[f"{base}/get/accepted"]@@ -1551,7 +1572,7 @@ def test_jobs_mqtt_non_object_payload_rejected(iot_client, iot_data_client):received = _collect_shadow_frames(f"{base}/get/rejected",- lambda: iot_data_client.publish(topic=f"{base}/get", payload=b"[1, 2]"),+ lambda: _mqtt_publish(topic=f"{base}/get", payload=b"[1, 2]"),want=1,)doc = _frames_by_topic(received)[f"{base}/get/rejected"]@@ -1560,7 +1581,7 @@ def test_jobs_mqtt_non_object_payload_rejected(iot_client, iot_data_client):def _qa_publish(iot_data_client, topic, payload):- iot_data_client.publish(topic=topic, qos=1, payload=payload)+ _mqtt_publish(topic=topic, payload=payload)def test_jobs_mqtt_malformed_json_is_rejected_invalidjson(iot_client, iot_data_client):diff --git a/tests/test_router.py b/tests/test_router.pyindex 3571ee5a..c2d2b115 100644--- a/tests/test_router.py+++ b/tests/test_router.py@@ -135,8 +135,7 @@ def test_iot_jobs_data_credential_scope_routes():@pytest.mark.parametrize("host",[- # What DescribeEndpoint(endpointType="iot:Jobs") actually hands out —- # a device following the documented flow signs against exactly this.+ # The legacy iot:Jobs endpoint shape."a1b2c3.jobs.iot.us-east-1.localhost:4566",# The spelling the AWS Device SDK's jobs documentation uses."a1b2c3.data.jobs.iot.us-east-1.localhost:4566",