Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(iot): registered-certificate event on connect, retired endpoint types, $ topics (#1755)

iot-rocketa0f9fe474252
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7f276494..88f237b7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,9 @@ Versioning follows [Semantic Versioning](https://semver.org/).
### Fixed
- **IAM — three calls are authorized against the action and resource AWS uses** — a Lambda Function URL invoke is checked against its function ARN, alias qualifier included, and supplies `lambda:FunctionUrlAuthType`, so the policy `grantInvokeUrl` writes matches. The WebSocket `@connections` API asks for `execute-api:ManageConnections` instead of `execute-api:Invoke`. `iot-jobs-data` operations are authorized under `iotjobsdata:`, except `StartCommandExecution`, which stays on `iot:`. A policy written for the old action names stops matching, as on AWS. Contributed by @iot-rocket.
+- **IoT — the registered-certificate event comes from the device connect** — `RegisterCertificate` published `$aws/events/certificates/registered/{caCertificateId}`, and an mTLS connect with an unknown certificate got CONNACK 5 with nothing created. AWS publishes the event only on the connect: a certificate signed by an ACTIVE CA with auto-registration enabled is created `PENDING_ACTIVATION`, the event carries `sourceIp`, and the connection closes without a CONNACK. The listener now does the same, and `RegisterCertificate` with status `PENDING_ACTIVATION` is `CertificateStateException` 406. Contributed by @iot-rocket.
+- **IoT — `DescribeEndpoint` refuses the retired `iot:Data` and `iot:Jobs` types** — both returned a hostname, so a client could keep using an endpoint type AWS no longer serves. AWS answers `InvalidRequestException` 400 and points to `iot:Data-ATS`; both types now get the same errors and messages. Contributed by @iot-rocket.
+- **IoT Data — HTTPS `Publish` refuses the topics AWS refuses** — any topic starting with `$` was accepted, so a caller could forge `$aws/events/...` lifecycle events and fire the rules on them. AWS answers `InvalidRequestException` 400 "Topic can't start with $" outside the `$aws/rules/`, `$aws/things/`, `$aws/certificates/`, `$aws/provisioning-templates/`, `$aws/device_location/` and `$aws/commands/` families, and "Invalid publish to restricted topic using HTTP" for the jobs, Device Defender and commands topics, which are MQTT-only. MiniStack now returns the same errors; MQTT publishes are unchanged. Contributed by @iot-rocket.
- **CloudFormation — `AWS::ECR::Repository` keeps what the template declares** — scanning and encryption settings were stored in the template's PascalCase and read back empty, lifecycle policy, repository policy and tags were dropped, `PutImage` into the repository failed and a deleted repository left its images and policies behind. The repository is now created, updated in place and deleted through the ECR store. Contributed by @iot-rocket.
- **CloudFormation — a rolled-back replacement keeps the old resource** — an update handler deleted the predecessor as soon as the replacement existed, so when a later resource failed the rollback removed the replacement too and the stack recorded a resource that was gone. The predecessor is now deleted in the cleanup phase after the update succeeds, and a rollback deletes only the replacement, as measured on AWS. Contributed by @iot-rocket.
### Added
diff --git a/ministack/core/router.py b/ministack/core/router.py
index 58bfae20..a4bd2fc3 100644
--- a/ministack/core/router.py
+++ b/ministack/core/router.py
@@ -333,9 +333,9 @@ SERVICE_PATTERNS = {
# routing would otherwise swallow it — and on the `iot` control plane
# `GET /things/{t}/jobs` is a DIFFERENT operation (ListJobExecutionsForThing)
# that answers with a different envelope. The pattern must cover both host
- # spellings: `DescribeEndpoint(endpointType='iot:Jobs')` hands out
- # `{prefix}.jobs.iot.{region}.{host}`, while the AWS Device SDK's jobs
- # topics document `{prefix}.data.jobs.iot.{region}.{host}`. The SDK also
+ # spellings: `{prefix}.jobs.iot.{region}.{host}`, the legacy iot:Jobs
+ # endpoint, and `{prefix}.data.jobs.iot.{region}.{host}`, which the AWS
+ # Device SDK's jobs topics document. The SDK also
# signs with credential scope `iot-jobs-data` (botocore signingName), which
# the scope early-return resolves via this key.
"iot-jobs-data": {
diff --git a/ministack/services/iot.py b/ministack/services/iot.py
index 965338d0..3e27c72c 100644
--- a/ministack/services/iot.py
+++ b/ministack/services/iot.py
@@ -13,9 +13,10 @@ Implements the JSON/REST APIs under ``iot.{region}.amazonaws.com``:
``DeleteCertificate``, ``AttachThingPrincipal`` / ``DetachThingPrincipal``
- CA certificates + JITR: ``GetRegistrationCode`` / ``DeleteRegistrationCode``,
``RegisterCACertificate``, ``DescribeCACertificate``, ``UpdateCACertificate``,
- ``ListCACertificates``, ``DeleteCACertificate``; registering a device
- certificate under a CA with auto-registration enabled publishes the AWS
- JITR event to ``$aws/events/certificates/registered/{caCertificateId}``
+ ``ListCACertificates``, ``DeleteCACertificate``; an mTLS connect with an
+ unknown certificate signed by a CA with auto-registration enabled creates
+ it PENDING_ACTIVATION and publishes the AWS JITR event to
+ ``$aws/events/certificates/registered/{caCertificateId}``
- Provisioning templates: ``CreateProvisioningTemplate``,
``DescribeProvisioningTemplate``, ``ListProvisioningTemplates``,
``UpdateProvisioningTemplate``, ``DeleteProvisioningTemplate`` —
@@ -604,6 +605,17 @@ async def handle_request(
# ---------------------------------------------------------------------------
+# AWS refuses the legacy endpoint types; the jobs data plane is served from
+# the iot:Data-ATS endpoint.
+_RETIRED_ENDPOINT_TYPES = {
+ "iot:Data": "iot:Data is not supported. Please use iot:Data-ATS instead.",
+ "iot:Jobs": (
+ "IoT Jobs and Commands APIs are now available through iot:Data-ATS "
+ "endpoints instead of iot:Jobs endpoints. Please use iot:Data-ATS."
+ ),
+}
+
+
def _describe_endpoint(qp: dict) -> tuple:
"""Return a per-account endpoint hostname.
@@ -617,13 +629,14 @@ def _describe_endpoint(qp: dict) -> tuple:
prefix = hashlib.sha256(account_id.encode("utf-8")).hexdigest()[:14]
region = get_region()
- if endpoint_type in ("iot:Data-ATS", "iot:Data", None):
- suffix = "-ats" if endpoint_type != "iot:Data" else ""
- host = f"{prefix}{suffix}.iot.{region}.{_MINISTACK_HOST}:{_GATEWAY_PORT}"
+ if endpoint_type in _RETIRED_ENDPOINT_TYPES:
+ return error_response_json(
+ "InvalidRequestException", _RETIRED_ENDPOINT_TYPES[endpoint_type], 400
+ )
+ if endpoint_type == "iot:Data-ATS":
+ host = f"{prefix}-ats.iot.{region}.{_MINISTACK_HOST}:{_GATEWAY_PORT}"
elif endpoint_type == "iot:CredentialProvider":
host = f"{prefix}.credentials.iot.{region}.{_MINISTACK_HOST}:{_GATEWAY_PORT}"
- elif endpoint_type == "iot:Jobs":
- host = f"{prefix}.jobs.iot.{region}.{_MINISTACK_HOST}:{_GATEWAY_PORT}"
else:
return error_response_json(
"InvalidRequestException",
@@ -1339,6 +1352,64 @@ def _delete_provisioning_template(name: str) -> tuple:
# ---------------------------------------------------------------------------
+def _certificate_record(
+ cert_id: str, cert_pem: str, status: str, ca_id: str | None = None
+) -> dict:
+ """The registry record of a device certificate, owned by the current scope."""
+ return {
+ "certificateId": cert_id,
+ "certificateArn": _cert_arn(cert_id),
+ "certificatePem": cert_pem, # verbatim
+ "status": status,
+ "creationDate": _now_epoch(),
+ "ownedBy": get_account_id(),
+ "caCertificateId": ca_id,
+ "attachedThings": [],
+ "attachedPolicies": [],
+ }
+
+
+async def _publish_certificate_registered(
+ account_id: str,
+ region: str,
+ record: dict,
+ source_ip: str | None,
+ registration_timestamp: str | None,
+) -> None:
+ """Publish the JITR event to ``$aws/events/certificates/registered/{caId}``.
+
+ AWS sends it when a device connects with a certificate its CA
+ auto-registers: ``certificateRegistrationTimestamp`` is null on the connect
+ that created the certificate and the creation time in epoch milliseconds,
+ as a string, on every later connect while it is still PENDING_ACTIVATION.
+ """
+ ca_id = record["caCertificateId"]
+ event = {
+ "certificateId": record["certificateId"],
+ "caCertificateId": ca_id,
+ "timestamp": int(time.time() * 1000),
+ "certificateStatus": record["status"],
+ "awsAccountId": account_id,
+ "certificateRegistrationTimestamp": registration_timestamp,
+ "sourceIp": source_ip,
+ }
+ try:
+ await broker_publish(
+ account_id,
+ region,
+ f"$aws/events/certificates/registered/{ca_id}",
+ json.dumps(event).encode("utf-8"),
+ qos=0,
+ )
+ except Exception:
+ # The certificate is already registered at this point, so a broker
+ # failure must not undo it; the iot-data publish path guards the same
+ # call the same way.
+ logger.warning(
+ "JITR registered-event publish failed for CA %s", ca_id, exc_info=True
+ )
+
+
def _create_keys_and_certificate(qp: dict) -> tuple:
"""Generate a fresh keypair and sign a leaf certificate with the Local CA."""
set_active = qp.get("setAsActive", "false").lower() == "true"
@@ -1352,21 +1423,10 @@ def _create_keys_and_certificate(qp: dict) -> tuple:
except RuntimeError as e:
return error_response_json("InternalFailureException", str(e), 503)
cert_id = get_certificate_id(cert_pem)
- arn = _cert_arn(cert_id)
- record = {
- "certificateId": cert_id,
- "certificateArn": arn,
- "certificatePem": cert_pem,
- "status": "ACTIVE" if set_active else "INACTIVE",
- "creationDate": _now_epoch(),
- "ownedBy": get_account_id(),
- "caCertificateId": None,
- "attachedThings": [],
- "attachedPolicies": [],
- }
+ record = _certificate_record(cert_id, cert_pem, "ACTIVE" if set_active else "INACTIVE")
_certificates[cert_id] = record
return json_response({
- "certificateArn": arn,
+ "certificateArn": record["certificateArn"],
"certificateId": cert_id,
"certificatePem": cert_pem,
"keyPair": {
@@ -1403,11 +1463,10 @@ async def _register_certificate(
``caCertificatePem`` must name a CA registered via
``RegisterCACertificate`` in this account/region that really signed the
- leaf; anything else is a ``CertificateValidationException``. When that CA's
- ``autoRegistrationStatus`` is ``ENABLE``, the AWS JITR lifecycle event is
- published to ``$aws/events/certificates/registered/{caCertificateId}`` so
- just-in-time-registration Lambdas subscribed via topic rules fire exactly
- as on AWS.
+ leaf; anything else is a ``CertificateValidationException``. Registering
+ publishes no JITR event, whatever the CA's ``autoRegistrationStatus``: AWS
+ sends that event only from a device connect (see ``_mtls_auto_register``),
+ and refuses a registration with status PENDING_ACTIVATION.
"""
cert_pem = payload.get("certificatePem") or qp.get("certificatePem")
if not cert_pem:
@@ -1415,6 +1474,12 @@ async def _register_certificate(
"InvalidRequestException", "certificatePem is required", 400
)
status = payload.get("status")
+ if status == "PENDING_ACTIVATION":
+ return error_response_json(
+ "CertificateStateException",
+ "Not allowed to register certificate with PENDING_ACTIVATION status",
+ 406,
+ )
if without_ca:
set_active = False
else:
@@ -1461,55 +1526,10 @@ async def _register_certificate(
)
if cert_id in _certificates:
return _certificate_already_exists(cert_id)
- record = {
- "certificateId": cert_id,
- "certificateArn": _cert_arn(cert_id),
- "certificatePem": cert_pem, # verbatim
- "status": status or ("ACTIVE" if set_active else "INACTIVE"),
- "creationDate": _now_epoch(),
- "ownedBy": get_account_id(),
- "caCertificateId": ca_id,
- "attachedThings": [],
- "attachedPolicies": [],
- }
+ record = _certificate_record(
+ cert_id, cert_pem, status or ("ACTIVE" if set_active else "INACTIVE"), ca_id
+ )
_certificates[cert_id] = record
-
- # JITR: the registered-certificate lifecycle event, fired when the
- # referenced CA is ACTIVE and has auto-registration enabled — AWS
- # auto-registers nothing under an INACTIVE CA. certificateStatus echoes
- # the requested register status; real connect-triggered auto-registration
- # carries PENDING_ACTIVATION, but here registration is always explicit —
- # there is no mTLS first-connect path to auto-register from.
- if (
- ca
- and ca.get("status") == "ACTIVE"
- and ca.get("autoRegistrationStatus") == "ENABLE"
- ):
- now_ms = int(time.time() * 1000)
- event = {
- "certificateId": cert_id,
- "caCertificateId": ca_id,
- "timestamp": now_ms,
- "certificateStatus": record["status"],
- "awsAccountId": get_account_id(),
- "certificateRegistrationTimestamp": str(now_ms),
- }
- try:
- await broker_publish(
- get_account_id(),
- get_region(),
- f"$aws/events/certificates/registered/{ca_id}",
- json.dumps(event).encode("utf-8"),
- qos=0,
- )
- except Exception:
- # The certificate is already registered at this point, so a broker
- # failure must not turn a successful registration into a 500 — the
- # iot-data publish path guards the same call the same way.
- logger.warning(
- "JITR registered-event publish failed for CA %s", ca_id, exc_info=True
- )
-
return json_response({
"certificateArn": record["certificateArn"],
"certificateId": cert_id,
@@ -4981,6 +5001,25 @@ def _validate_publish_topic(topic: str) -> bool:
return True
+# The reserved topic space a client may publish into. AWS closes the connection
+# of an MQTT client that publishes to any other topic starting with ``$``
+# (measured for ``$aws/events/...``, ``$aws/jobs/...`` and ``$foo/...``; a
+# publish under ``$aws/rules/`` and ``$aws/things/`` is acknowledged). The HTTPS
+# Publish in iot_data.py applies the same list and answers 400.
+RESERVED_PUBLISH_PREFIXES = (
+ "$aws/rules/",
+ "$aws/things/",
+ "$aws/certificates/",
+ "$aws/provisioning-templates/",
+ "$aws/device_location/",
+ "$aws/commands/",
+)
+
+
+def _reserved_topic_publish_allowed(topic: str) -> bool:
+ return not topic.startswith("$") or topic.startswith(RESERVED_PUBLISH_PREFIXES)
+
+
# ---------------------------------------------------------------------------
# Broker public API (consumed by iot_data.py and handle_websocket)
# ---------------------------------------------------------------------------
@@ -6926,6 +6965,13 @@ class _WSSession:
if self.protocol_version == MQTT_5:
await self.send_bytes(_make_disconnect(RC5_TOPIC_NAME_INVALID))
return False
+ if not _reserved_topic_publish_allowed(topic):
+ # AWS closes the connection without a PUBACK (measured over
+ # MQTT 3.1.1; the MQTT 5 reason code is not measured).
+ _broker_logger.warning("IoT broker: PUBLISH to a reserved topic closes the connection: %r", topic)
+ if self.protocol_version == MQTT_5:
+ await self.send_bytes(_make_disconnect(RC5_NOT_AUTHORIZED))
+ return False
payload = body[off:]
delivered = await broker_publish(
self.account_id,
@@ -7251,6 +7297,12 @@ async def handle_websocket(
# mean anything and what real AWS IoT does. The ambiguous case cannot be
# attributed at all: two registrations see byte-identical bytes, so any
# tie-break would award the session to whoever registered the copy.
+# * just-in-time registration, as on AWS: a certificate that is unknown, or
+# still PENDING_ACTIVATION, and signed by a registered CA that is ACTIVE
+# with auto-registration enabled is created PENDING_ACTIVATION in that CA's
+# account and region, the registered event is published, and the connection
+# is closed without a CONNACK. As on AWS this happens on the CONNECT packet;
+# a handshake that sends no CONNECT registers nothing.
#
# The TLS layer adds one constraint of its own: a presented chain is verified,
# so a certificate from a CA the listener does not trust fails the handshake
@@ -7613,12 +7665,16 @@ def mtls_schedule_restart() -> None:
_mtls_logger.debug("IoT mTLS: could not schedule a restart: %s", e)
-def _mtls_active_registrations(cert_id: str) -> list[tuple[str, str, dict]]:
- """Every (account_id, region, record) holding this certificate ACTIVE."""
+def _mtls_active_registrations(
+ cert_id: str, status: str | None = "ACTIVE"
+) -> list[tuple[str, str, dict]]:
+ """Every (account_id, region, record) holding this certificate in ``status``
+ (any status when None)."""
return [
(account_id, region, record)
for (account_id, region, key), record in list(_certificates._data.items())
- if key == cert_id and isinstance(record, dict) and record.get("status") == "ACTIVE"
+ if key == cert_id and isinstance(record, dict)
+ and (status is None or record.get("status") == status)
]
@@ -7661,21 +7717,84 @@ def _mtls_resolve_identity(der: bytes | None) -> tuple[str, str] | None:
return None
-async def _mtls_refuse(
- reader: asyncio.StreamReader, writer: asyncio.StreamWriter, cert_id: str
-) -> None:
- """Answer an unusable client certificate with a "not authorized" CONNACK
- (0x05, or reason code 0x87 for an MQTT 5 client), then close.
+def _mtls_auto_registering_ca(account_id: str, region: str, ca_id: str | None) -> dict | None:
+ """The CA record when it would auto-register a device certificate now."""
+ ca = _ca_certificates.get_scoped(account_id, region, ca_id) if ca_id else None
+ if (
+ isinstance(ca, dict)
+ and ca.get("status") == "ACTIVE"
+ and ca.get("autoRegistrationStatus") == "ENABLE"
+ and ca.get("certificateMode", "DEFAULT") == "DEFAULT"
+ ):
+ return ca
+ return None
- AWS closes the socket without a CONNACK unless just-in-time registration
- applies. Diverging here is deliberate: an emulator's job is
- diagnosability, "not authorized" is reported cleanly by every device SDK,
- and a TLS-layer rejection could not tell the user "unknown" from
- "deactivated".
+
+def _mtls_auto_registering_signers(cert_pem: str) -> list[tuple[str, str, str]]:
+ """Every (account_id, region, ca_id) of an auto-registering CA that signed ``cert_pem``."""
+ return [
+ (account_id, region, ca_id)
+ for (account_id, region, ca_id), ca in list(_ca_certificates._data.items())
+ if _mtls_auto_registering_ca(account_id, region, ca_id) is not None
+ and certificate_is_signed_by(cert_pem, ca.get("certificatePem") or "")
+ ]
+
+
+async def _mtls_auto_register(der: bytes, peername) -> bool:
+ """Just-in-time registration for a presented certificate that no scope
+ holds ACTIVE. True when it applied, and the caller closes without a
+ CONNACK; False leaves the refusal to ``_mtls_refuse``.
+
+ An unknown certificate is created PENDING_ACTIVATION under the one
+ auto-registering CA that signed it. A certificate already
+ PENDING_ACTIVATION under such a CA only publishes the event again. AWS
+ does both on every such connect, and nothing for an INACTIVE certificate
+ or a CA with auto-registration disabled.
"""
- _mtls_logger.warning(
- "IoT mTLS: refusing client certificate %s (%s)", cert_id, _mtls_refusal_reason(cert_id)
+ cert_id = hashlib.sha256(der).hexdigest()
+ held = _mtls_active_registrations(cert_id, status=None)
+ if len(held) > 1:
+ return False
+ if held:
+ account_id, region, record = held[0]
+ if record.get("status") != "PENDING_ACTIVATION":
+ return False
+ if _mtls_auto_registering_ca(account_id, region, record.get("caCertificateId")) is None:
+ return False
+ registration_timestamp = str(int(record["creationDate"] * 1000))
+ else:
+ cert_pem = ssl.DER_cert_to_PEM_cert(der)
+ # One signature check per auto-registering CA, off the broker's loop:
+ # a device retrying with a refused certificate must not stall it.
+ signers = await asyncio.to_thread(_mtls_auto_registering_signers, cert_pem)
+ # The same CA registered in several scopes cannot be attributed, for
+ # the same reason an ambiguous ACTIVE certificate is refused.
+ if len(signers) != 1:
+ return False
+ account_id, region, ca_id = signers[0]
+ with request_scope(account_id, region):
+ record = _certificate_record(cert_id, cert_pem, "PENDING_ACTIVATION", ca_id)
+ _certificates[cert_id] = record
+ registration_timestamp = None
+ _mtls_logger.info(
+ "IoT mTLS: certificate %s is PENDING_ACTIVATION under CA %s in %s/%s; "
+ "published the registered event and closing",
+ cert_id,
+ record["caCertificateId"],
+ account_id,
+ region,
)
+ source_ip = peername[0] if isinstance(peername, tuple) and peername else None
+ await _publish_certificate_registered(
+ account_id, region, record, source_ip, registration_timestamp
+ )
+ return True
+
+
+async def _mtls_first_packet(reader: asyncio.StreamReader) -> bytes | None:
+ """The client's first complete MQTT packet, or None when the peer closes,
+ sends nothing within ``_MTLS_REFUSE_READ_TIMEOUT`` or exceeds
+ ``_MTLS_REFUSE_MAX_BYTES`` without completing one."""
buffer = bytearray()
try:
while len(buffer) < _MTLS_REFUSE_MAX_BYTES:
@@ -7683,7 +7802,7 @@ async def _mtls_refuse(
reader.read(_MTLS_READ_CHUNK), timeout=_MTLS_REFUSE_READ_TIMEOUT
)
if not data:
- return
+ return None
buffer.extend(data)
if len(buffer) < 2:
continue
@@ -7693,27 +7812,54 @@ async def _mtls_refuse(
continue
if len(buffer) < header_end + remaining:
continue
- if (buffer[0] >> 4) & 0x0F == PKT_CONNECT:
- # The CONNACK has to match the client's protocol level: a v5
- # client handed the two-byte 3.1.1 form dies in its decoder
- # instead of reporting the refusal, and 0x05 is not a defined
- # v5 reason code.
- body = bytes(buffer[header_end:header_end + remaining])
- try:
- _proto_name, level_at = _read_string(body, 0)
- protocol_level = body[level_at]
- except (ValueError, IndexError):
- protocol_level = MQTT_311
- if protocol_level == MQTT_5:
- connack = _make_connack(
- return_code=RC5_NOT_AUTHORIZED, protocol_version=MQTT_5
- )
- else:
- connack = _make_connack(return_code=CONNACK_311_NOT_AUTHORIZED)
- writer.write(connack)
- await writer.drain()
- return
+ return bytes(buffer[: header_end + remaining])
except (asyncio.TimeoutError, OSError, ssl.SSLError):
+ return None
+ return None
+
+
+def _connect_protocol_level(packet: bytes) -> int | None:
+ """The protocol level of a CONNECT packet; None for any other packet."""
+ if (packet[0] >> 4) & 0x0F != PKT_CONNECT:
+ return None
+ remaining, header_end = _decode_remaining_length(packet, 1)
+ body = packet[header_end : header_end + remaining]
+ try:
+ _proto_name, level_at = _read_string(body, 0)
+ return body[level_at]
+ except (ValueError, IndexError):
+ return MQTT_311
+
+
+async def _mtls_refuse(
+ writer: asyncio.StreamWriter, cert_id: str, protocol_level: int | None
+) -> None:
+ """Answer an unusable client certificate with a "not authorized" CONNACK
+ (0x05, or reason code 0x87 for an MQTT 5 client) when the client sent a
+ CONNECT, then close.
+
+ AWS closes the socket without a CONNACK unless just-in-time registration
+ applies. Diverging here is deliberate: an emulator's job is
+ diagnosability, "not authorized" is reported cleanly by every device SDK,
+ and a TLS-layer rejection could not tell the user "unknown" from
+ "deactivated".
+ """
+ _mtls_logger.warning(
+ "IoT mTLS: refusing client certificate %s (%s)", cert_id, _mtls_refusal_reason(cert_id)
+ )
+ if protocol_level is None:
+ return
+ # The CONNACK has to match the client's protocol level: a v5 client
+ # handed the two-byte 3.1.1 form dies in its decoder instead of reporting
+ # the refusal, and 0x05 is not a defined v5 reason code.
+ if protocol_level == MQTT_5:
+ connack = _make_connack(return_code=RC5_NOT_AUTHORIZED, protocol_version=MQTT_5)
+ else:
+ connack = _make_connack(return_code=CONNACK_311_NOT_AUTHORIZED)
+ try:
+ writer.write(connack)
+ await writer.drain()
+ except (OSError, ssl.SSLError):
return
@@ -7741,7 +7887,16 @@ async def _mtls_serve_conn(reader: asyncio.StreamReader, writer: asyncio.StreamW
owner = _mtls_resolve_identity(der)
if owner is None:
try:
- await _mtls_refuse(reader, writer, hashlib.sha256(der).hexdigest())
+ # AWS registers on the CONNECT packet, not on the handshake: a
+ # client that closes after the handshake, sends nothing, or sends
+ # another packet first registers nothing (measured).
+ packet = await _mtls_first_packet(reader)
+ protocol_level = _connect_protocol_level(packet) if packet else None
+ if protocol_level is not None and await _mtls_auto_register(
+ der, writer.get_extra_info("peername")
+ ):
+ return
+ await _mtls_refuse(writer, hashlib.sha256(der).hexdigest(), protocol_level)
finally:
await _mtls_close(writer)
return
diff --git a/ministack/services/iot_data.py b/ministack/services/iot_data.py
index b027a099..31c932ac 100644
--- a/ministack/services/iot_data.py
+++ b/ministack/services/iot_data.py
@@ -19,6 +19,7 @@ from __future__ import annotations
import copy
import json
import logging
+import re
from urllib.parse import unquote
from ministack.core.responses import (
@@ -35,6 +36,14 @@ logger = logging.getLogger("iot_data")
# AWS IoT topics: max 7 segments, max 256 UTF-8 bytes total.
_MAX_TOPIC_BYTES = 256
+# HTTPS Publish to a topic starting with "$" is refused unless the topic sits
+# under one of these reserved families. Within them, the device-side jobs,
+# Device Defender and commands topics are MQTT-only.
+_HTTP_RESERVED_PREFIXES = _iot_module.RESERVED_PUBLISH_PREFIXES
+_HTTP_RESTRICTED_TOPIC_RE = re.compile(
+ r"^\$aws/(things/[^/]+/(jobs|defender)/|commands/)"
+)
+
def _validate_topic(topic: str) -> tuple | None:
"""Return an error response if ``topic`` is invalid, else None."""
@@ -54,6 +63,17 @@ def _validate_topic(topic: str) -> tuple | None:
f"Topic exceeds {_MAX_TOPIC_BYTES} bytes",
400,
)
+ if topic.startswith("$"):
+ if not topic.startswith(_HTTP_RESERVED_PREFIXES):
+ return error_response_json(
+ "InvalidRequestException", "Topic can't start with $", 400
+ )
+ if _HTTP_RESTRICTED_TOPIC_RE.match(topic):
+ return error_response_json(
+ "InvalidRequestException",
+ "Invalid publish to restricted topic using HTTP",
+ 400,
+ )
return None
diff --git a/ministack/services/iot_jobs_data.py b/ministack/services/iot_jobs_data.py
index 814696c4..91eeaf58 100644
--- a/ministack/services/iot_jobs_data.py
+++ b/ministack/services/iot_jobs_data.py
@@ -8,8 +8,8 @@ Implements the device-side REST surface of AWS IoT Jobs:
Routing reaches us either through credential-scope detection (the SDK signs
requests with the ``iot-jobs-data`` scope — botocore signingName) or via the
-host pattern ``{prefix}.jobs.iot.{region}.{host}``, which is what
-``DescribeEndpoint(endpointType='iot:Jobs')`` hands out.
+host pattern ``{prefix}.jobs.iot.{region}.{host}``, the shape of the legacy
+``iot:Jobs`` endpoint.
This module is a wire adapter: it parses requests, shapes responses, and logs.
The job store, the execution store, and every rule of the execution state
diff --git a/tests/conftest.py b/tests/conftest.py
index a9db94de..ae4a0610 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -191,11 +191,10 @@ _SERIAL_TESTS = {
"tests/test_iot_data.py::test_iot_rule_where_clause_gates_dispatch",
"tests/test_iot_data.py::test_iot_rule_where_topic_function_under_basic_ingest",
"tests/test_iot_data.py::test_iot_rule_where_or_clause_dispatches_either_branch",
- "tests/test_iot_data.py::test_iot_jitr_registration_event_drives_a_topic_rule",
# IoT Jobs data-plane routing test: a raw urllib GET with the advertised
# `{prefix}.jobs.iot.{region}` Host and a tight 5s timeout, so cross-file
# xdist pressure on the shared event loop makes it time out at random.
- "tests/test_iot_jobs.py::test_iot_jobs_advertised_endpoint_host_reaches_the_data_plane",
+ "tests/test_iot_jobs.py::test_iot_jobs_endpoint_host_reaches_the_data_plane",
# ECS service task-spawn: with a Docker daemon present (CI has one) a task
# whose container fails to start/exits under parallel container churn is set
# STOPPED, so list_tasks (RUNNING-only) sees fewer than desiredCount. Passes
@@ -273,6 +272,7 @@ _SERIAL_TESTS = {
"tests/test_iot_data.py::test_mtls_inactive_cert_refused",
"tests/test_iot_data.py::test_mtls_ambiguous_cert_is_refused",
"tests/test_iot_data.py::test_mtls_registered_ca_chain_connects",
+ "tests/test_iot_data.py::test_mtls_jitr_auto_registers_an_unknown_cert_without_connack",
"tests/test_iot_data.py::test_mtls_account_scoped_delivery",
"tests/test_iot_data.py::test_mtls_garbage_bytes_dropped",
"tests/test_iot_data.py::test_mtls_duplicate_client_id_evicts_first_connection",
diff --git a/tests/test_iot.py b/tests/test_iot.py
index ab607f9a..401b9ca1 100644
--- a/tests/test_iot.py
+++ b/tests/test_iot.py
@@ -36,12 +36,23 @@ def test_iot_describe_endpoint_default_uses_data_ats(iot_client):
assert "-ats.iot." in resp["endpointAddress"]
-def test_iot_describe_endpoint_data_legacy(iot_client):
- resp = iot_client.describe_endpoint(endpointType="iot:Data")
- addr = resp["endpointAddress"]
- # Legacy endpoint omits the -ats suffix.
- assert ".iot." in addr
- assert "-ats.iot." not in addr
+ ("endpoint_type", "message"),
+ [
+ ("iot:Data", "iot:Data is not supported. Please use iot:Data-ATS instead."),
+ (
+ "iot:Jobs",
+ "IoT Jobs and Commands APIs are now available through iot:Data-ATS "
+ "endpoints instead of iot:Jobs endpoints. Please use iot:Data-ATS.",
+ ),
+ ],
+)
+def test_iot_describe_endpoint_retired_types_rejected(iot_client, endpoint_type, message):
+ with pytest.raises(ClientError) as ei:
+ iot_client.describe_endpoint(endpointType=endpoint_type)
+ assert ei.value.response["Error"]["Code"] == "InvalidRequestException"
+ assert ei.value.response["Error"]["Message"] == message
+ assert ei.value.response["ResponseMetadata"]["HTTPStatusCode"] == 400
def test_iot_describe_endpoint_unknown_type_rejected(iot_client):
@@ -5281,90 +5292,139 @@ def test_iot_ca_certificate_registration_config_round_trip(iot_client):
iot_client.delete_ca_certificate(certificateId=ca_id)
-def test_iot_jitr_registered_event_published_when_auto_registration_enabled():
- """Register a CA with auto-registration ENABLE, subscribe on
- ``$aws/events/certificates/registered/#``, register a device cert under
- that CA — the JITR lifecycle event arrives with certificateId +
- caCertificateId and DescribeCertificate resolves the signing CA.
+_JITR_ACCOUNT = "123456789012"
- Runs in-process against the module broker (the WS route to the live
- server needs ``*.localhost`` DNS this environment lacks), following the
- broker-helper pattern above.
- """
- from ministack.core.responses import (
- set_request_account_id,
- set_request_region,
+
+async def _jitr_scope_with_ca(iot_module, ca_pem: str, received: list, allow_auto=True) -> str:
+ """Pin the JITR test scope, register ``ca_pem`` ACTIVE and collect every
+ registered event into ``received``. Returns the CA id."""
+ from ministack.core.responses import set_request_account_id, set_request_region
+
+ set_request_account_id(_JITR_ACCOUNT)
+ set_request_region(_TEST_REGION)
+ qp = {"setAsActive": "true"}
+ if allow_auto:
+ qp["allowAutoRegistration"] = "true"
+ status, _, body = await iot_module.handle_request(
+ "POST", "/cacertificate", {}, json.dumps({"caCertificate": ca_pem}).encode(), qp
+ )
+ assert status == 200
+
+ async def _collect(topic, payload, qos):
+ received.append((topic, json.loads(payload)))
+
+ await iot_module.broker_subscribe(
+ _JITR_ACCOUNT, _TEST_REGION, "$aws/events/certificates/registered/#", _collect
)
+ return json.loads(body)["certificateId"]
+
+
[email protected]("without_ca", [False, True], ids=["with-ca", "without-ca"])
[email protected]("status", ["ACTIVE", "INACTIVE", "PENDING_ACTIVATION"])
+def test_iot_register_certificate_publishes_no_registered_event(status, without_ca):
+ """RegisterCertificate under an ACTIVE CA with auto-registration enabled
+ publishes nothing, as on AWS, where only a device connect sends the
+ registered event. PENDING_ACTIVATION is refused outright, by
+ RegisterCertificateWithoutCA as well (measured)."""
from ministack.services import iot as iot_module
ca_pem, leaf_pem = _generate_ca_and_leaf()
- account_id = "123456789012"
received: list = []
async def _run():
- set_request_account_id(account_id)
- set_request_region(_TEST_REGION)
-
- status, _, body = await iot_module.handle_request(
+ await _jitr_scope_with_ca(iot_module, ca_pem, received)
+ payload = {"certificatePem": leaf_pem, "status": status}
+ if not without_ca:
+ payload["caCertificatePem"] = ca_pem
+ code, _, body = await iot_module.handle_request(
"POST",
- "/cacertificate",
+ "/certificate/register-no-ca" if without_ca else "/certificate/register",
+ {},
+ json.dumps(payload).encode(),
{},
- json.dumps({"caCertificate": ca_pem}).encode(),
- {"setAsActive": "true", "allowAutoRegistration": "true"},
)
- assert status == 200
- ca_id = json.loads(body)["certificateId"]
+ return code, json.loads(body)
- async def _collect(topic, payload, qos):
- received.append((topic, payload, qos))
+ try:
+ code, body = asyncio.run(_run())
+ finally:
+ iot_module.reset()
+ iot_module.broker_reset()
+ if status == "PENDING_ACTIVATION":
+ assert code == 406
+ assert body["__type"] == "CertificateStateException"
+ assert body["message"] == "Not allowed to register certificate with PENDING_ACTIVATION status"
+ else:
+ assert code == 200
+ assert received == []
+
+
+def test_iot_jitr_auto_registration_on_connect(monkeypatch):
+ """The registry half of just-in-time registration on an mTLS connect; the
+ wire half (closed without a CONNACK) is in test_iot_data.py.
+
+ First connect: the certificate is created PENDING_ACTIVATION under its CA
+ and the event carries a null certificateRegistrationTimestamp. A repeat
+ connect publishes again with the creation time as a string. An INACTIVE
+ certificate, or a CA with auto-registration disabled, gets nothing. A
+ broker failure does not undo the registration.
+ """
+ import ssl
- await iot_module.broker_subscribe(
- account_id,
- _TEST_REGION,
- "$aws/events/certificates/registered/#",
- _collect,
- )
+ from ministack.services import iot as iot_module
- status, _, body = await iot_module.handle_request(
- "POST",
- "/certificate/register",
- {},
- json.dumps({
- "certificatePem": leaf_pem,
- "caCertificatePem": ca_pem,
- "status": "PENDING_ACTIVATION",
- }).encode(),
- {},
- )
- assert status == 200
- cert_id = json.loads(body)["certificateId"]
+ ca_pem, leaf_pem = _generate_ca_and_leaf()
+ off_ca_pem, off_leaf_pem = _generate_ca_and_leaf()
+ cert_id = iot_module.get_certificate_id(leaf_pem)
+ peer = ("192.0.2.10", 50000)
+ received: list = []
- assert len(received) == 1
- topic, payload, _qos = received[0]
- assert topic == f"$aws/events/certificates/registered/{ca_id}"
- event = json.loads(payload)
- assert event["certificateId"] == cert_id
- assert event["caCertificateId"] == ca_id
- assert event["certificateStatus"] == "PENDING_ACTIVATION"
- assert event["awsAccountId"] == account_id
- assert isinstance(event["timestamp"], int)
- assert event["certificateRegistrationTimestamp"] == str(event["timestamp"])
-
- # A JITR Lambda resolves the signing CA via DescribeCertificate.
- status, _, body = await iot_module.handle_request(
- "GET", f"/certificates/{cert_id}", {}, b"", {}
- )
- assert status == 200
- assert (
- json.loads(body)["certificateDescription"]["caCertificateId"] == ca_id
- )
+ async def _boom(*_args, **_kwargs):
+ raise RuntimeError("broker down")
+
+ async def _run():
+ ca_id = await _jitr_scope_with_ca(iot_module, ca_pem, received)
+ await _jitr_scope_with_ca(iot_module, off_ca_pem, [], allow_auto=False)
+ der = ssl.PEM_cert_to_DER_cert(leaf_pem)
+
+ assert await iot_module._mtls_auto_register(der, peer)
+ record = iot_module._certificates[cert_id]
+ assert record["status"] == "PENDING_ACTIVATION"
+ assert record["caCertificateId"] == ca_id
+ assert await iot_module._mtls_auto_register(der, peer)
+
+ off_der = ssl.PEM_cert_to_DER_cert(off_leaf_pem)
+ assert not await iot_module._mtls_auto_register(off_der, peer)
+ assert iot_module.get_certificate_id(off_leaf_pem) not in iot_module._certificates
+
+ record["status"] = "INACTIVE"
+ assert not await iot_module._mtls_auto_register(der, peer)
+
+ record["status"] = "PENDING_ACTIVATION"
+ monkeypatch.setattr(iot_module, "broker_publish", _boom)
+ assert await iot_module._mtls_auto_register(der, peer)
+ return ca_id, record
try:
- asyncio.run(_run())
+ ca_id, record = asyncio.run(_run())
finally:
iot_module.reset()
iot_module.broker_reset()
+ topic = f"$aws/events/certificates/registered/{ca_id}"
+ assert [t for t, _event in received] == [topic, topic]
+ first, second = received[0][1], received[1][1]
+ assert isinstance(first.pop("timestamp"), int)
+ assert first == {
+ "certificateId": cert_id,
+ "caCertificateId": ca_id,
+ "certificateStatus": "PENDING_ACTIVATION",
+ "awsAccountId": _JITR_ACCOUNT,
+ "certificateRegistrationTimestamp": None,
+ "sourceIp": "192.0.2.10",
+ }
+ assert second["certificateRegistrationTimestamp"] == str(int(record["creationDate"] * 1000))
+
# ----------------------------------------------------------------------
# Shadow-over-MQTT bridge (white-box).
@@ -5499,71 +5559,6 @@ def test_shadow_mqtt_update_reported_in_sync_emits_no_delta():
iot_module.broker_reset()
-def test_iot_jitr_no_event_when_auto_registration_disabled():
- """A CA registered without ``allowAutoRegistration`` links the device cert
- (``caCertificateId``) but publishes no registered event."""
- from ministack.core.responses import (
- set_request_account_id,
- set_request_region,
- )
- from ministack.services import iot as iot_module
-
- ca_pem, leaf_pem = _generate_ca_and_leaf()
- account_id = "123456789012"
- received: list = []
-
- async def _run():
- set_request_account_id(account_id)
- set_request_region(_TEST_REGION)
-
- status, _, body = await iot_module.handle_request(
- "POST",
- "/cacertificate",
- {},
- json.dumps({"caCertificate": ca_pem}).encode(),
- {"setAsActive": "true"},
- )
- assert status == 200
- ca_id = json.loads(body)["certificateId"]
-
- async def _collect(topic, payload, qos):
- received.append((topic, payload, qos))
-
- await iot_module.broker_subscribe(
- account_id,
- _TEST_REGION,
- "$aws/events/certificates/registered/#",
- _collect,
- )
-
- status, _, body = await iot_module.handle_request(
- "POST",
- "/certificate/register",
- {},
- json.dumps({
- "certificatePem": leaf_pem,
- "caCertificatePem": ca_pem,
- }).encode(),
- {},
- )
- assert status == 200
- cert_id = json.loads(body)["certificateId"]
- assert received == []
-
- status, _, body = await iot_module.handle_request(
- "GET", f"/certificates/{cert_id}", {}, b"", {}
- )
- assert (
- json.loads(body)["certificateDescription"]["caCertificateId"] == ca_id
- )
-
- try:
- asyncio.run(_run())
- finally:
- iot_module.reset()
- iot_module.broker_reset()
-
-
def test_shadow_mqtt_update_rejections():
"""Invalid JSON → 400 (no token); version conflict → 409 with the token."""
from ministack.services import iot as iot_module
@@ -5620,13 +5615,12 @@ def test_iot_register_certificate_under_ca_links_ca_certificate_id(iot_client):
cert_id = iot_client.register_certificate(
certificatePem=leaf_pem,
caCertificatePem=ca_pem,
- status="PENDING_ACTIVATION",
)["certificateId"]
desc = iot_client.describe_certificate(certificateId=cert_id)[
"certificateDescription"
]
assert desc["caCertificateId"] == ca_id
- assert desc["status"] == "PENDING_ACTIVATION"
+ assert desc["status"] == "INACTIVE"
iot_client.delete_certificate(certificateId=cert_id)
finally:
iot_client.update_ca_certificate(certificateId=ca_id, newStatus="INACTIVE")
@@ -5694,63 +5688,6 @@ def test_iot_register_certificate_rejects_a_leaf_from_another_ca(iot_client):
iot_client.delete_ca_certificate(certificateId=ca_id)
-def test_iot_jitr_no_event_for_a_certificate_from_another_ca():
- """The rejection happens before anything is published, so no JITR consumer
- ever sees an event attributing a foreign certificate to its CA."""
- from ministack.core.responses import (
- set_request_account_id,
- set_request_region,
- )
- from ministack.services import iot as iot_module
-
- _ca_x_pem, leaf_pem = _generate_ca_and_leaf()
- ca_y_pem, _leaf_y = _generate_ca_and_leaf()
- account_id = "123456789012"
- received: list = []
-
- async def _run():
- set_request_account_id(account_id)
- set_request_region(_TEST_REGION)
-
- status, _, _body = await iot_module.handle_request(
- "POST",
- "/cacertificate",
- {},
- json.dumps({"caCertificate": ca_y_pem}).encode(),
- {"setAsActive": "true", "allowAutoRegistration": "true"},
- )
- assert status == 200
-
- async def _collect(topic, payload, qos):
- received.append((topic, payload, qos))
-
- await iot_module.broker_subscribe(
- account_id,
- _TEST_REGION,
- "$aws/events/certificates/registered/#",
- _collect,
- )
-
- status, _, body = await iot_module.handle_request(
- "POST",
- "/certificate/register",
- {},
- json.dumps(
- {"certificatePem": leaf_pem, "caCertificatePem": ca_y_pem}
- ).encode(),
- {},
- )
- assert status == 400
- assert json.loads(body)["__type"] == "CertificateValidationException"
- assert received == []
-
- try:
- asyncio.run(_run())
- finally:
- iot_module.reset()
- iot_module.broker_reset()
-
-
def test_iot_update_ca_certificate_applies_a_body_only_status(iot_client):
"""newStatus / newAutoRegistrationStatus are modeled in the query string,
but a raw caller that sends them in the JSON body must not get a 200 that
@@ -5795,141 +5732,6 @@ def test_iot_update_ca_certificate_applies_a_body_only_status(iot_client):
iot_client.delete_ca_certificate(certificateId=ca_id)
-def test_iot_jitr_event_requires_an_active_ca():
- """Auto-registration is gated on the CA's own status: an INACTIVE CA with
- allowAutoRegistration publishes nothing (AWS registers nothing under one),
- and activating the same CA starts the events flowing."""
- from ministack.core.responses import (
- set_request_account_id,
- set_request_region,
- )
- from ministack.services import iot as iot_module
-
- ca_pem, (leaf_pem, leaf_pem2) = _generate_ca_and_leaves(2)
- account_id = "123456789012"
- received: list = []
-
- async def _run():
- set_request_account_id(account_id)
- set_request_region(_TEST_REGION)
-
- # INACTIVE (no setAsActive) but auto-registration ENABLE.
- status, _, body = await iot_module.handle_request(
- "POST",
- "/cacertificate",
- {},
- json.dumps({"caCertificate": ca_pem}).encode(),
- {"allowAutoRegistration": "true"},
- )
- assert status == 200
- ca_id = json.loads(body)["certificateId"]
-
- async def _collect(topic, payload, qos):
- received.append((topic, payload, qos))
-
- await iot_module.broker_subscribe(
- account_id,
- _TEST_REGION,
- "$aws/events/certificates/registered/#",
- _collect,
- )
-
- status, _, _body = await iot_module.handle_request(
- "POST",
- "/certificate/register",
- {},
- json.dumps(
- {"certificatePem": leaf_pem, "caCertificatePem": ca_pem}
- ).encode(),
- {},
- )
- assert status == 200
- assert received == []
-
- # Activate the CA — the very next registration fires.
- status, _, _body = await iot_module.handle_request(
- "PUT", f"/cacertificate/{ca_id}", {}, b"", {"newStatus": "ACTIVE"}
- )
- assert status == 200
-
- status, _, body = await iot_module.handle_request(
- "POST",
- "/certificate/register",
- {},
- json.dumps(
- {"certificatePem": leaf_pem2, "caCertificatePem": ca_pem}
- ).encode(),
- {},
- )
- assert status == 200
- cert_id2 = json.loads(body)["certificateId"]
-
- assert len(received) == 1
- topic, payload, _qos = received[0]
- assert topic == f"$aws/events/certificates/registered/{ca_id}"
- assert json.loads(payload)["certificateId"] == cert_id2
-
- try:
- asyncio.run(_run())
- finally:
- iot_module.reset()
- iot_module.broker_reset()
-
-
-def test_iot_jitr_registration_survives_a_broker_failure(monkeypatch):
- """The certificate is committed before the event is published, so a broker
- that raises must not turn a successful registration into a 500."""
- from ministack.core.responses import (
- set_request_account_id,
- set_request_region,
- )
- from ministack.services import iot as iot_module
-
- ca_pem, leaf_pem = _generate_ca_and_leaf()
- account_id = "123456789012"
-
- async def _boom(*_args, **_kwargs):
- raise RuntimeError("broker down")
-
- async def _run():
- set_request_account_id(account_id)
- set_request_region(_TEST_REGION)
-
- status, _, _body = await iot_module.handle_request(
- "POST",
- "/cacertificate",
- {},
- json.dumps({"caCertificate": ca_pem}).encode(),
- {"setAsActive": "true", "allowAutoRegistration": "true"},
- )
- assert status == 200
-
- monkeypatch.setattr(iot_module, "broker_publish", _boom)
- status, _, body = await iot_module.handle_request(
- "POST",
- "/certificate/register",
- {},
- json.dumps(
- {"certificatePem": leaf_pem, "caCertificatePem": ca_pem}
- ).encode(),
- {},
- )
- assert status == 200
- cert_id = json.loads(body)["certificateId"]
-
- status, _, body = await iot_module.handle_request(
- "GET", f"/certificates/{cert_id}", {}, b"", {}
- )
- assert status == 200
- assert json.loads(body)["certificateDescription"]["certificateId"] == cert_id
-
- try:
- asyncio.run(_run())
- finally:
- iot_module.reset()
- iot_module.broker_reset()
-
-
def test_iot_update_ca_certificate_rejects_invalid_enum_values(iot_client):
"""The two update fields validate their enums with a 400.
diff --git a/tests/test_iot_data.py b/tests/test_iot_data.py
index a69933a1..1e9b41f3 100644
--- a/tests/test_iot_data.py
+++ b/tests/test_iot_data.py
@@ -111,6 +111,39 @@ def test_iot_data_publish_oversized_topic_400():
assert e.code == 400
+_NO_DOLLAR = "Topic can't start with $"
+_MQTT_ONLY = "Invalid publish to restricted topic using HTTP"
+
+
+ ("topic", "message"),
+ [
+ ("$aws/events/certificates/registered/x", _NO_DOLLAR),
+ ("$AWS/events/certificates/registered/x", _NO_DOLLAR),
+ ("$aws/jobs/x", _NO_DOLLAR),
+ ("$aws/rules", _NO_DOLLAR),
+ ("$foo/bar", _NO_DOLLAR),
+ ("$aws/things/t1/jobs/get", _MQTT_ONLY),
+ ("$aws/things/t1/defender/metrics/json", _MQTT_ONLY),
+ ("$aws/commands/things/t1/executions/1/response/json", _MQTT_ONLY),
+ ("$aws/rules/no_such_rule/x", None),
+ ("$aws/things/t1/jobs", None),
+ ],
+)
+def test_iot_data_publish_reserved_topics(iot_data_client, topic, message):
+ """HTTPS Publish to a "$" topic is refused outside the reserved families
+ and for the MQTT-only jobs, Device Defender and commands topics."""
+ if message is None:
+ resp = iot_data_client.publish(topic=topic, payload=b"{}")
+ assert resp["ResponseMetadata"]["HTTPStatusCode"] == 200
+ return
+ with pytest.raises(ClientError) as ei:
+ iot_data_client.publish(topic=topic, payload=b"{}")
+ assert ei.value.response["Error"]["Code"] == "InvalidRequestException"
+ assert ei.value.response["Error"]["Message"] == message
+ assert ei.value.response["ResponseMetadata"]["HTTPStatusCode"] == 400
+
+
# ---------------------------------------------------------------------------
# MQTT-over-WebSocket round-trip
# ---------------------------------------------------------------------------
@@ -1369,55 +1402,8 @@ def test_search_index_connectivity_is_isolated_across_accounts_and_regions():
finally:
owner.delete_thing(thingName=thing)
owner_eu.delete_thing(thingName=thing)
-def test_iot_jitr_registration_event_drives_a_topic_rule(iot_client, lam, sqs):
- """The JITR lifecycle event is a real broker publish, so a topic rule on
- ``$aws/events/certificates/registered/{caId}`` hands it to a Lambda — the
- shape a just-in-time-registration stack actually deploys.
-
- The rule names this test's CA id rather than the ``+`` wildcard: the
- account's registered-certificate topics are shared, so a wildcard rule
- living for the length of this test also catches the registrations other
- tests make on other xdist workers, and ``_poll_sink`` would return
- whichever event landed first."""
- pytest.importorskip("cryptography")
- from ministack.core.x509_utils import generate_ca, sign_leaf_certificate
- ca_pem, ca_key_pem = generate_ca(common_name=_unique("jitr-rule-ca"))
- leaf_pem, _priv, _pub = sign_leaf_certificate(
- ca_cert_pem=ca_pem,
- ca_key_pem=ca_key_pem,
- common_name=_unique("jitr-rule-device"),
- )
- sink = sqs.create_queue(QueueName=_unique("jitr-sink"))["QueueUrl"]
- fn_arn = _make_sink_lambda(lam, sink)
- rule = _unique("jitr").replace("-", "_")
- ca_id = iot_client.register_ca_certificate(
- caCertificate=ca_pem, setAsActive=True, allowAutoRegistration=True
- )["certificateId"]
- iot_client.create_topic_rule(
- ruleName=rule,
- topicRulePayload={
- "sql": f"SELECT * FROM '$aws/events/certificates/registered/{ca_id}'",
- "actions": [{"lambda": {"functionArn": fn_arn}}],
- },
- )
- try:
- cert_id = iot_client.register_certificate(
- certificatePem=leaf_pem,
- caCertificatePem=ca_pem,
- status="PENDING_ACTIVATION",
- )["certificateId"]
- event = _poll_sink(sqs, sink)
- assert event is not None, "no JITR event reached the rule's Lambda"
- assert event["certificateId"] == cert_id
- assert event["caCertificateId"] == ca_id
- assert event["certificateStatus"] == "PENDING_ACTIVATION"
- iot_client.delete_certificate(certificateId=cert_id)
- finally:
- iot_client.delete_topic_rule(ruleName=rule)
- iot_client.update_ca_certificate(certificateId=ca_id, newStatus="INACTIVE")
- iot_client.delete_ca_certificate(certificateId=ca_id)
# ---------------------------------------------------------------------------
# Topic-rule `sqs` action (publish → rule → SQS queue)
# ---------------------------------------------------------------------------
@@ -2522,6 +2508,33 @@ def test_mqtt5_qos1_puback_carries_reason_code():
assert matched == b"\x00\x08\x00\x00", "packet id 8, Success, no properties"
[email protected]("topic,acknowledged", [
+ ("$aws/rules/no-such-rule/x", True),
+ ("$aws/things/some-thing/shadow/update", True),
+ ("$aws/events/certificates/registered/abc", False),
+ ("$aws/jobs/abc", False),
+ ("$foo/abc", False),
+], ids=["rules", "shadow", "events", "jobs", "other-dollar"])
+def test_mqtt_publish_to_reserved_topic_closes_the_connection(topic, acknowledged):
+ """AWS acknowledges a publish under $aws/rules/ and $aws/things/ and closes
+ the connection of a client that publishes to any other $ topic (measured
+ over MQTT 3.1.1); the HTTPS Publish applies the same list."""
+
+ async def scenario():
+ async with _connect(MQTT_311) as (pub, _c):
+ await pub.send(_make_publish(topic, b"x", qos=1, packet_id=5))
+ try:
+ _flags, body = await pub.await_packet(PKT_PUBACK, timeout=3)
+ return ("puback", body)
+ except (websockets.exceptions.ConnectionClosed, OSError) as e:
+ return ("closed", type(e).__name__)
+
+ kind, detail = _run(scenario())
+ assert kind == ("puback" if acknowledged else "closed"), (kind, detail)
+ if acknowledged:
+ assert detail == b"\x00\x05"
+
+
def test_mqtt311_qos1_puback_stays_two_bytes():
"""The 3.1.1 PUBACK gains neither a reason code nor properties."""
topic = _unique("v311/qos1")
@@ -3551,6 +3564,82 @@ def test_mtls_registered_ca_chain_connects(broker, tmp_path):
peer.close()
+def test_mtls_jitr_auto_registers_an_unknown_cert_without_connack(broker, tmp_path):
+ """Just-in-time registration, as on AWS: an unknown certificate signed by a
+ CA with auto-registration enabled is created PENDING_ACTIVATION, the
+ registered event is published and the connection closes without a CONNACK.
+ A repeat connect publishes again, now with the creation time. A TLS
+ handshake that sends no CONNECT registers nothing (AWS registers on the
+ packet, not on the handshake). With auto-registration disabled the refusal
+ stays CONNACK 5 and nothing is created."""
+ from ministack.core.x509_utils import generate_ca, get_certificate_id, sign_leaf_certificate
+
+ iot = broker.client("iot")
+ ca_pem, ca_key = generate_ca(common_name=_unique("jitr-ca"))
+ ca_id = iot.register_ca_certificate(
+ caCertificate=ca_pem, setAsActive=True, allowAutoRegistration=True
+ )["certificateId"]
+ leaf_pem, leaf_key, _public = sign_leaf_certificate(ca_pem, ca_key, common_name="jitr-device")
+ cert_id = get_certificate_id(leaf_pem)
+ topic = f"$aws/events/certificates/registered/{ca_id}"
+ try:
+
+ listener = _Peer(_mtls_connect(broker, None, None, tmp_path))
+ events = []
+ try:
+ _assert_connack(listener.connect(_unique("jitr-listener")))
+ listener.subscribe(topic)
+ try:
+ _Peer(_mtls_connect(broker, leaf_pem, leaf_key, tmp_path)).close()
+ except OSError:
+ pass
+ assert listener.next_publish(timeout=2.0) is None, "handshake alone registered"
+ with pytest.raises(ClientError) as exc:
+ iot.describe_certificate(certificateId=cert_id)
+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"
+ for attempt in ("first", "repeat"):
+ assert _refused_below_mqtt(
+ broker, leaf_pem, leaf_key, tmp_path, _unique(f"jitr-{attempt}")
+ ), f"{attempt} connect got a CONNACK"
+ events.append(listener.next_publish())
+ finally:
+ listener.close()
+
+ assert None not in events, events
+ assert [t for t, _payload in events] == [topic, topic]
+ first, repeat = (json.loads(payload) for _t, payload in events)
+ assert first["certificateId"] == cert_id
+ assert first["caCertificateId"] == ca_id
+ assert first["certificateStatus"] == "PENDING_ACTIVATION"
+ assert first["certificateRegistrationTimestamp"] is None
+ assert first["sourceIp"] == "127.0.0.1"
+ assert isinstance(repeat["certificateRegistrationTimestamp"], str)
+ desc = iot.describe_certificate(certificateId=cert_id)["certificateDescription"]
+ assert desc["status"] == "PENDING_ACTIVATION"
+ assert desc["caCertificateId"] == ca_id
+
+ iot.update_ca_certificate(certificateId=ca_id, newAutoRegistrationStatus="DISABLE")
+ off_pem, off_key, _public = sign_leaf_certificate(ca_pem, ca_key, common_name="jitr-off")
+ peer = _Peer(_mtls_connect(broker, off_pem, off_key, tmp_path))
+ try:
+ _assert_connack(peer.connect(_unique("jitr-off")), return_code=5)
+ finally:
+ peer.close()
+ with pytest.raises(ClientError) as exc:
+ iot.describe_certificate(certificateId=get_certificate_id(off_pem))
+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"
+ finally:
+ # The auto-registered certificate and the CA must not outlive the test.
+ for cid in (cert_id,):
+ try:
+ iot.update_certificate(certificateId=cid, newStatus="INACTIVE")
+ iot.delete_certificate(certificateId=cid, forceDelete=True)
+ except ClientError:
+ pass
+ iot.update_ca_certificate(certificateId=ca_id, newStatus="INACTIVE")
+ iot.delete_ca_certificate(certificateId=ca_id)
+
+
def test_mtls_account_scoped_delivery(broker, tmp_path):
"""The client certificate decides the tenant: a device whose certificate is
owned by 111111111111 sees that account's traffic and nobody else's."""
diff --git a/tests/test_iot_jobs.py b/tests/test_iot_jobs.py
index de321e6e..411325dd 100644
--- a/tests/test_iot_jobs.py
+++ b/tests/test_iot_jobs.py
@@ -447,9 +447,8 @@ def test_iot_jobs_fleet_completes_only_when_every_execution_is_terminal(
# ---------------------------------------------------------------------------
-def test_iot_jobs_advertised_endpoint_host_reaches_the_data_plane(iot_client):
- """The documented device flow: `DescribeEndpoint(endpointType='iot:Jobs')`
- hands out `{prefix}.jobs.iot.{region}`, and a request carrying that Host
+def test_iot_jobs_endpoint_host_reaches_the_data_plane(iot_client):
+ """A request carrying the jobs endpoint Host `{prefix}.jobs.iot.{region}`
must land on the jobs data plane. Routed to the `iot` control plane
instead, `GET /things/{t}/jobs` is ListJobExecutionsForThing and silently
answers a different envelope."""
@@ -459,10 +458,7 @@ def test_iot_jobs_advertised_endpoint_host_reaches_the_data_plane(iot_client):
thing_arn = _create_thing(iot_client, thing)
iot_client.create_job(jobId=job_id, targets=[thing_arn], document=_DOCUMENT)
- endpoint = iot_client.describe_endpoint(endpointType="iot:Jobs")[
- "endpointAddress"
- ]
- assert ".jobs.iot." in endpoint
+ endpoint = "a1b2c3.jobs.iot.us-east-1.localhost"
request = urllib.request.Request(
f"{ENDPOINT}/things/{quote(thing)}/jobs",
method="GET",
@@ -1238,11 +1234,36 @@ def test_iot_jobs_lists_do_not_paginate_and_return_no_token(iot_client):
# ---------------------------------------------------------------------------
# Jobs over MQTT: the reserved $aws/things/<t>/jobs/# bridge
# ---------------------------------------------------------------------------
-from test_iot_data import _collect_shadow_frames # noqa: E402
+import asyncio # noqa: E402
+
+from conftest import patch_endpoint_dns # noqa: E402
+from test_iot_data import ( # noqa: E402
+ _WS_HANDSHAKE_TIMEOUT,
+ _collect_shadow_frames,
+ _make_publish,
+ _mqtt_connect,
+ _mqtt_disconnect,
+)
_DOCUMENT_OBJECT = json.loads(_DOCUMENT)
+def _mqtt_publish(topic, payload):
+ """Publish at QoS 1 over the MQTT-over-WebSocket broker and wait for the
+ PUBACK. The jobs topics are MQTT-only: AWS refuses them over HTTPS."""
+
+ async def _run():
+ ws = await _mqtt_connect(_unique("jobs-device"))
+ try:
+ await ws.send(_make_publish(topic, payload, qos=1, packet_id=1))
+ await asyncio.wait_for(ws.recv(), timeout=_WS_HANDSHAKE_TIMEOUT)
+ finally:
+ await _mqtt_disconnect(ws)
+
+ with patch_endpoint_dns():
+ asyncio.run(_run())
+
+
def _frames_by_topic(received):
return {topic: json.loads(payload) for topic, payload in received}
@@ -1285,7 +1306,7 @@ def test_jobs_mqtt_create_job_notifies_and_get_lists(iot_client, iot_data_client
received = _collect_shadow_frames(
f"{base}/get/accepted",
- lambda: iot_data_client.publish(
+ lambda: _mqtt_publish(
topic=f"{base}/get",
payload=json.dumps({"clientToken": "tok-get"}).encode(),
),
@@ -1351,7 +1372,7 @@ def test_jobs_mqtt_start_next_of_first_job_is_silent(iot_client, iot_data_client
received = _collect_shadow_frames(
f"{base}/#",
- lambda: iot_data_client.publish(
+ lambda: _mqtt_publish(
topic=f"{base}/start-next",
payload=json.dumps({"clientToken": "tok-sn"}).encode(),
),
@@ -1391,13 +1412,13 @@ def test_jobs_mqtt_update_include_flags_and_terminal_notify(
)
_collect_shadow_frames(
f"{base}/start-next/accepted",
- lambda: iot_data_client.publish(topic=f"{base}/start-next", payload=b"{}"),
+ lambda: _mqtt_publish(topic=f"{base}/start-next", payload=b"{}"),
want=1,
)
received = _collect_shadow_frames(
f"{base}/{job_id}/update/accepted",
- lambda: iot_data_client.publish(
+ lambda: _mqtt_publish(
topic=f"{base}/{job_id}/update",
payload=json.dumps(
{
@@ -1419,7 +1440,7 @@ def test_jobs_mqtt_update_include_flags_and_terminal_notify(
received = _collect_shadow_frames(
f"{base}/#",
- lambda: iot_data_client.publish(
+ lambda: _mqtt_publish(
topic=f"{base}/{job_id}/update",
payload=json.dumps(
{
@@ -1466,7 +1487,7 @@ def test_jobs_mqtt_version_mismatch_rejected(iot_client, iot_data_client):
received = _collect_shadow_frames(
f"{base}/{job_id}/update/rejected",
- lambda: iot_data_client.publish(
+ lambda: _mqtt_publish(
topic=f"{base}/{job_id}/update",
payload=json.dumps(
{
@@ -1495,7 +1516,7 @@ def test_jobs_mqtt_update_unknown_job_rejected(iot_client, iot_data_client):
received = _collect_shadow_frames(
f"{base}/no-such-job/update/rejected",
- lambda: iot_data_client.publish(
+ lambda: _mqtt_publish(
topic=f"{base}/no-such-job/update",
payload=json.dumps({"status": "SUCCEEDED", "clientToken": "tok-x"}).encode(),
),
@@ -1518,7 +1539,7 @@ def test_jobs_mqtt_next_sentinel_get(iot_client, iot_data_client):
received = _collect_shadow_frames(
f"{base}/$next/get/accepted",
- lambda: iot_data_client.publish(
+ lambda: _mqtt_publish(
topic=f"{base}/$next/get",
payload=json.dumps({"clientToken": "tok-next"}).encode(),
),
@@ -1534,7 +1555,7 @@ def test_jobs_mqtt_next_sentinel_get(iot_client, iot_data_client):
received = _collect_shadow_frames(
f"{base}/get/accepted",
- lambda: iot_data_client.publish(topic=f"{base}/get", payload=b"{}"),
+ lambda: _mqtt_publish(topic=f"{base}/get", payload=b"{}"),
want=1,
)
doc = _frames_by_topic(received)[f"{base}/get/accepted"]
@@ -1551,7 +1572,7 @@ def test_jobs_mqtt_non_object_payload_rejected(iot_client, iot_data_client):
received = _collect_shadow_frames(
f"{base}/get/rejected",
- lambda: iot_data_client.publish(topic=f"{base}/get", payload=b"[1, 2]"),
+ lambda: _mqtt_publish(topic=f"{base}/get", payload=b"[1, 2]"),
want=1,
)
doc = _frames_by_topic(received)[f"{base}/get/rejected"]
@@ -1560,7 +1581,7 @@ def test_jobs_mqtt_non_object_payload_rejected(iot_client, iot_data_client):
def _qa_publish(iot_data_client, topic, payload):
- iot_data_client.publish(topic=topic, qos=1, payload=payload)
+ _mqtt_publish(topic=topic, payload=payload)
def test_jobs_mqtt_malformed_json_is_rejected_invalidjson(iot_client, iot_data_client):
diff --git a/tests/test_router.py b/tests/test_router.py
index 3571ee5a..c2d2b115 100644
--- a/tests/test_router.py
+++ b/tests/test_router.py
@@ -135,8 +135,7 @@ def test_iot_jobs_data_credential_scope_routes():
@pytest.mark.parametrize(
"host",
[
- # What DescribeEndpoint(endpointType="iot:Jobs") actually hands out —
- # a device following the documented flow signs against exactly this.
+ # The legacy iot:Jobs endpoint shape.
"a1b2c3.jobs.iot.us-east-1.localhost:4566",
# The spelling the AWS Device SDK's jobs documentation uses.
"a1b2c3.data.jobs.iot.us-east-1.localhost:4566",