Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(iot): resolve principal() and traceid() in rule SQL (#2142)
iot-rocket475f9ebf05a6
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5197a74c..7a6a53e2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,9 @@ Versioning follows [Semantic Versioning](https://semver.org/).## [Unreleased]+### Fixed++- **IoT — rule SQL `principal()` and `traceid()`** — both resolved to Undefined; `principal()` now returns the client certificate id for mTLS, the caller's `GetCallerIdentity` `UserId` for HTTP Publish and SigV4 WebSocket sessions (also their lifecycle events' `principalIdentifier`) and the rule role's session for a republish, and `traceid()` returns a per-message id that the errorAction document carries as `cloudwatchTraceId`. Contributed by @iot-rocket.### Added- **IoT — retained messages** — `GetRetainedMessage` and `ListRetainedMessages` answered 501 `not yet implemented`; they now return the broker's retained messages (newest first, paged by `maxResults` and `nextToken`, with the user properties of an MQTT 5 publish), and a `PERSIST_STATE=1` restart keeps each message's `lastModifiedTime` and user properties. Contributed by @iot-rocket.diff --git a/README.md b/README.mdindex a2943bcb..78b8525c 100644--- a/README.md+++ b/README.md@@ -647,7 +647,7 @@ MiniStack expands AWS SAM transforms (`Transform: AWS::Serverless-2016-10-31`) i| **AppConfig** | CreateApplication, GetApplication, ListApplications, UpdateApplication, DeleteApplication, CreateEnvironment, GetEnvironment, ListEnvironments, UpdateEnvironment, DeleteEnvironment, CreateConfigurationProfile, GetConfigurationProfile, ListConfigurationProfiles, UpdateConfigurationProfile, DeleteConfigurationProfile, CreateHostedConfigurationVersion, GetHostedConfigurationVersion, ListHostedConfigurationVersions, DeleteHostedConfigurationVersion, CreateDeploymentStrategy, GetDeploymentStrategy, ListDeploymentStrategies, UpdateDeploymentStrategy, DeleteDeploymentStrategy, StartDeployment, GetDeployment, ListDeployments, StopDeployment, TagResource, UntagResource, ListTagsForResource, StartConfigurationSession, GetLatestConfiguration | 33 operations; control plane + data plane; hosted configuration versions; the AWS-predefined deployment strategies (4) are listed and immutable; deployments move through `DEPLOYING`, `BAKING` and `COMPLETE` on the strategy's schedule (`LINEAR` or `EXPONENTIAL` steps, then the bake time), with `PercentageComplete`, the event log and the environment `State` reported along the way; a `Monitors` alarm that goes to `ALARM` before the bake ends rolls the deployment back; one deployment per environment at a time (`ConflictException`, also for a stale `LatestDeploymentNumber`); `StopDeployment` rolls back a `DEPLOYING` deployment, and with `AllowRevert` also a `BAKING` one, or reverts a completed one within 72 hours; during a rollout a session gets the new version once the rollout reaches its fixed slot (MiniStack's choice, since AWS does not document how it picks targets); `GetLatestConfiguration` returns an empty body when the session already has the latest version, plus `Version-Label`, and uses `RequiredMinimumPollIntervalInSeconds` as the poll interval || **Transfer Family** | CreateServer, DescribeServer, DeleteServer, ListServers, StartServer, StopServer, CreateUser, DescribeUser, DeleteUser, ListUsers, ImportSshPublicKey, DeleteSshPublicKey | 12 operations; **real SFTP listener** on `:2222` (override with `SFTP_PORT`) backed by S3 — `ls`, `put`, `get`, `mkdir`, `rename` work end-to-end against the local S3 emulator; `SFTP_PORT_PER_SERVER=1` allocates one port per `CreateServer` from `SFTP_BASE_PORT` (default 2300); SSH key auth scans every user across every server and account; `LOGICAL` and `PATH` home-directory mappings; host key persists across restarts when `PERSIST_STATE=1` || **IoT Jobs** | Control plane (`iot`): CreateJob, UpdateJob, DescribeJob, ListJobs, GetJobDocument, CancelJob, DeleteJob, ListJobExecutionsForThing, DescribeJobExecution, CancelJobExecution, CreateJobTemplate, DescribeJobTemplate, ListJobTemplates, DeleteJobTemplate; data plane (`iot-jobs-data`): GetPendingJobExecutions, StartNextPendingJobExecution, DescribeJobExecution (incl. the `$next` sentinel), UpdateJobExecution | 14 control-plane + 4 device data-plane operations sharing one store. A job created with a `jobTemplateArn` takes the template's document, configuration and maintenance windows, and a request member replaces the template's. Executions walk `QUEUED → IN_PROGRESS → {SUCCEEDED, FAILED, REJECTED, CANCELED, TIMED_OUT}`; an execution past its `timeoutConfig.inProgressTimeoutInMinutes`, or past the `stepTimeoutInMinutes` the device reported, becomes `TIMED_OUT`, and the device plane reports `approximateSecondsBeforeTimedOut` while it runs. `REMOVED` is modeled and counted in `jobProcessDetails` but nothing sets it: deleting a thing DELETES its executions (freeing a job that was only waiting on that thing) rather than marking them `REMOVED`. `versionNumber` gives optimistic concurrency: every transition bumps it and a stale `expectedVersion` is rejected with a 409 whose message carries the current version — coded `VersionConflictException` on the control plane and `InvalidStateTransitionException` on the device plane, each as its own service model declares. `executionNumber` is write-once (always 1: an execution is never re-queued). Devices may only report `IN_PROGRESS` / `SUCCEEDED` / `FAILED` / `REJECTED` via `UpdateJobExecution`; targets are thing or thing-group ARNs — `SNAPSHOT` jobs resolve group membership once at creation, `CONTINUOUS` jobs also pick up things added to a target group later, and a target must name a thing or thing group in the caller's own account and region (anything else is `ResourceNotFoundException`, since it would resolve to no devices at all); a non-continuous job auto-`COMPLETED`s (with `jobProcessDetails` counts) once every execution is terminal. **`documentSource` is not fetched**: a job created with an S3 `documentSource` serves a `{"documentSource": "<url>"}` placeholder as its document rather than the object's content, so the lifecycle still runs — pass `document` when the content matters; a job template with a `documentSource` stores the same placeholder. **Timestamps are epoch seconds on every plane, as AWS documents**: control-plane responses use `timestamp` shapes (boto3 parses them to datetimes) while `iot-jobs-data` responses carry whole-second integer `long`s. **Jobs over MQTT**: the reserved `$aws/things/<thing>/jobs/#` request topics (`get`, `start-next`, `<jobId>/get` incl. `$next`, `<jobId>/update`) answer on their `accepted`/`rejected` topics from the same store as the HTTP plane (`jobDocument` as a JSON object over MQTT, a string over HTTP, as on AWS; rejections carry the string `ErrorCode` values and the execution's state on version and state-transition conflicts); `notify` fires when the pending set changes and `notify-next` when the queue front changes; `stepTimeoutInMinutes` and `executionNumber` are accepted but ignored. Job state persists with the IoT store when `PERSIST_STATE=1` |-| **IoT Core** | CreateThing, DescribeThing, ListThings, UpdateThing, DeleteThing, CreateThingType, CreateThingGroup, AddThingToThingGroup, ListThingsInThingGroup, ListThingGroupsForThing, CreateKeysAndCertificate, RegisterCertificate, RegisterCertificateWithoutCA, CreateCertificateFromCsr, UpdateCertificate, DeleteCertificate, GetRegistrationCode, DeleteRegistrationCode, RegisterCACertificate, DescribeCACertificate, UpdateCACertificate, ListCACertificates, DeleteCACertificate, ListCertificatesByCA, CreateProvisioningTemplate, DescribeProvisioningTemplate, ListProvisioningTemplates, UpdateProvisioningTemplate, DeleteProvisioningTemplate, AttachThingPrincipal, DetachThingPrincipal, ListThingPrincipals, ListPrincipalThings, CreatePolicy, CreatePolicyVersion, AttachPolicy, DetachPolicy, ListAttachedPolicies, ListTargetsForPolicy, AttachPrincipalPolicy, DetachPrincipalPolicy, ListPrincipalPolicies, ListPolicyPrincipals, CreateTopicRule, GetTopicRule, ReplaceTopicRule, DeleteTopicRule, ListTopicRules, UpdateIndexingConfiguration, GetIndexingConfiguration, DescribeIndex, ListIndices, SearchIndex, DescribeEndpoint, Publish (HTTP), GetRetainedMessage, ListRetainedMessages, MQTT-over-WebSocket pub/sub, DescribeEventConfigurations, UpdateEventConfigurations, CreateDomainConfiguration, DescribeDomainConfiguration, ListDomainConfigurations, UpdateDomainConfiguration, DeleteDomainConfiguration | 65 operations + **MQTT 3.1.1 and 5.0 over WebSocket** (negotiated per connection) on the gateway port — clients use the address returned by `DescribeEndpoint`; local CA signs `CreateKeysAndCertificate` certificates (CA persists across restarts when `PERSIST_STATE=1`); multi-tenancy via transparent topic prefixing (same pattern as Transfer Family's shared SFTP listener); no plain TCP 1883 (real AWS IoT requires TLS or SigV4 on every connection), but a **native mTLS MQTT listener on 8883** serves device SDK binaries by default — the client certificate is optional (no certificate means the default account, like an unsigned WebSocket upgrade) and, when presented, must be registered `ACTIVE` in exactly one account, which then serves the session; unknown, inactive or doubly-registered certificates get a "not authorized" CONNACK (see `IOT_MTLS_PORT` in the environment table); local CA cert available at `GET /_ministack/iot/ca.pem` for SDK trust configuration; IoT policies are stored but **not enforced** on the data plane; **fleet indexing** works the way it does on AWS — `UpdateIndexingConfiguration` turns the `AWS_Things` index on (Terraform `aws_iot_indexing_configuration`), and searching while `thingIndexingMode` is `OFF` answers `ResourceNotFoundException`; `SearchIndex` then queries the live registry + classic shadows with `AND`-separated `thingName` / `thingTypeName` / `thingGroupNames` / `attributes.*` / `shadow.reported.*` / `shadow.desired.*` terms (`*` / `?` wildcards, `maxResults` / `nextToken` paging) — there is no projection to build, so hits are always current; the deprecated principal-policy family (`AttachPrincipalPolicy` / `DetachPrincipalPolicy` / `ListPrincipalPolicies` / `ListPolicyPrincipals`) maps 1:1 onto the modern policy-target store; **topic rules fire on matching publishes** (incl. Basic Ingest `$aws/rules/<name>`) with SQL SELECT projection (`topic()`, `timestamp()`, `encode()`, `newuuid()`, `replace()`, `clientid()`, `isUndefined()`), **WHERE-clause evaluation** (comparisons, `IN`, `LIKE`, `BETWEEN`, `IS NULL`, `regexp_matches`, bare boolean attributes, composed with `AND` / `OR` and parenthesised grouping; missing attributes fail closed as on AWS; unparseable SQL → `SqlParseException`), and `lambda` / `republish` / `dynamoDBv2` / `sns` / `sqs` actions dispatched in-process with the rule's `errorAction` on failure; **device shadows over MQTT** — the `$aws/things/<thing>/shadow[/name/<name>]/{get,update,delete}` reserved topics drive the shadow store and reply on `accepted` / `rejected` (with `delta` and `documents` on update); a **CA-certificate registry** (`RegisterCACertificate` and friends) with the just-in-time-registration `registered` event on `$aws/events/certificates/registered/{caCertificateId}`; a **provisioning-template registry** (`CreateProvisioningTemplate` and friends) — storage + CRUD only, so `provisioningRoleArn` is presence-checked but not resolved and the fleet-provisioning MQTT workflow (`$aws/provisioning-templates/...`) is not implemented; and `SearchIndex` answers `connectivity.connected` / `connectivity.timestamp` / `connectivity.disconnectReason` when `thingConnectivityIndexingMode` is `STATUS`, deriving `connected` from the live session registry; the broker publishes the presence and subscription lifecycle events (`$aws/events/presence/connected\|disconnected/{clientId}`, `$aws/events/subscriptions/subscribed\|unsubscribed/{clientId}`) |+| **IoT Core** | CreateThing, DescribeThing, ListThings, UpdateThing, DeleteThing, CreateThingType, CreateThingGroup, AddThingToThingGroup, ListThingsInThingGroup, ListThingGroupsForThing, CreateKeysAndCertificate, RegisterCertificate, RegisterCertificateWithoutCA, CreateCertificateFromCsr, UpdateCertificate, DeleteCertificate, GetRegistrationCode, DeleteRegistrationCode, RegisterCACertificate, DescribeCACertificate, UpdateCACertificate, ListCACertificates, DeleteCACertificate, ListCertificatesByCA, CreateProvisioningTemplate, DescribeProvisioningTemplate, ListProvisioningTemplates, UpdateProvisioningTemplate, DeleteProvisioningTemplate, AttachThingPrincipal, DetachThingPrincipal, ListThingPrincipals, ListPrincipalThings, CreatePolicy, CreatePolicyVersion, AttachPolicy, DetachPolicy, ListAttachedPolicies, ListTargetsForPolicy, AttachPrincipalPolicy, DetachPrincipalPolicy, ListPrincipalPolicies, ListPolicyPrincipals, CreateTopicRule, GetTopicRule, ReplaceTopicRule, DeleteTopicRule, ListTopicRules, UpdateIndexingConfiguration, GetIndexingConfiguration, DescribeIndex, ListIndices, SearchIndex, DescribeEndpoint, Publish (HTTP), MQTT-over-WebSocket pub/sub, DescribeEventConfigurations, UpdateEventConfigurations, CreateDomainConfiguration, DescribeDomainConfiguration, ListDomainConfigurations, UpdateDomainConfiguration, DeleteDomainConfiguration | 63 operations + **MQTT 3.1.1 and 5.0 over WebSocket** (negotiated per connection) on the gateway port — clients use the address returned by `DescribeEndpoint`; local CA signs `CreateKeysAndCertificate` certificates (CA persists across restarts when `PERSIST_STATE=1`); multi-tenancy via transparent topic prefixing (same pattern as Transfer Family's shared SFTP listener); no plain TCP 1883 (real AWS IoT requires TLS or SigV4 on every connection), but a **native mTLS MQTT listener on 8883** serves device SDK binaries by default — the client certificate is optional (no certificate means the default account, like an unsigned WebSocket upgrade) and, when presented, must be registered `ACTIVE` in exactly one account, which then serves the session; unknown, inactive or doubly-registered certificates get a "not authorized" CONNACK (see `IOT_MTLS_PORT` in the environment table); local CA cert available at `GET /_ministack/iot/ca.pem` for SDK trust configuration; IoT policies are stored but **not enforced** on the data plane; **fleet indexing** works the way it does on AWS — `UpdateIndexingConfiguration` turns the `AWS_Things` index on (Terraform `aws_iot_indexing_configuration`), and searching while `thingIndexingMode` is `OFF` answers `ResourceNotFoundException`; `SearchIndex` then queries the live registry + classic shadows with `AND`-separated `thingName` / `thingTypeName` / `thingGroupNames` / `attributes.*` / `shadow.reported.*` / `shadow.desired.*` terms (`*` / `?` wildcards, `maxResults` / `nextToken` paging) — there is no projection to build, so hits are always current; the deprecated principal-policy family (`AttachPrincipalPolicy` / `DetachPrincipalPolicy` / `ListPrincipalPolicies` / `ListPolicyPrincipals`) maps 1:1 onto the modern policy-target store; **topic rules fire on matching publishes** (incl. Basic Ingest `$aws/rules/<name>`) with SQL SELECT projection (`topic()`, `timestamp()`, `encode()`, `newuuid()`, `replace()`, `clientid()`, `principal()`, `traceid()`, `isUndefined()`), **WHERE-clause evaluation** (comparisons, `IN`, `LIKE`, `BETWEEN`, `IS NULL`, `regexp_matches`, bare boolean attributes, composed with `AND` / `OR` and parenthesised grouping; missing attributes fail closed as on AWS; unparseable SQL → `SqlParseException`), and `lambda` / `republish` / `dynamoDBv2` / `sns` / `sqs` actions dispatched in-process with the rule's `errorAction` on failure; **device shadows over MQTT** — the `$aws/things/<thing>/shadow[/name/<name>]/{get,update,delete}` reserved topics drive the shadow store and reply on `accepted` / `rejected` (with `delta` and `documents` on update); a **CA-certificate registry** (`RegisterCACertificate` and friends) with the just-in-time-registration `registered` event on `$aws/events/certificates/registered/{caCertificateId}`; a **provisioning-template registry** (`CreateProvisioningTemplate` and friends) — storage + CRUD only, so `provisioningRoleArn` is presence-checked but not resolved and the fleet-provisioning MQTT workflow (`$aws/provisioning-templates/...`) is not implemented; and `SearchIndex` answers `connectivity.connected` / `connectivity.timestamp` / `connectivity.disconnectReason` when `thingConnectivityIndexingMode` is `STATUS`, deriving `connected` from the live session registry; the broker publishes the presence and subscription lifecycle events (`$aws/events/presence/connected\|disconnected/{clientId}`, `$aws/events/subscriptions/subscribed\|unsubscribed/{clientId}`) || **IoT Wireless** | GetPositionEstimate | Single operation (`POST /position-estimate`, SigV4 scope `iotwireless`, endpoint prefix `api.iotwireless`). **The response is the raw GeoJSON blob** — the output shape's `payload` trait — so boto3 hands back a `StreamingBody` under `GeoJsonPayload`, not a JSON envelope. Resolves from `Ip` only: `WiFiAccessPoints` / `CellTowers` / `Gnss` are accepted and ignored, `Timestamp` and `AdvancedConfiguration` only reach `properties`, and the estimate is synthetic + deterministic — the canonical IP (an IPv4-mapped IPv6 address counts as its IPv4 form) hashes onto lon [-180, 180) / lat [-60, 60), 4 decimals — so the same address always answers the same estimate and consumer tests can assert on it. `properties` carry `horizontalAccuracy` 1000000 (the value of one live call) and `horizontalConfidenceLevel` 0.68 (the default the AWS IoT Core Device Location guide documents — `AdvancedConfiguration.WiFiCellular.ConfidencePercent` overrides it, divided by 100, within the 50 to 99 the model allows) plus `timestamp`, which is always there: AWS documents the request's `Timestamp` as the resolve time and the receive time as its substitute when the caller sends none, so send a `Timestamp` and the whole blob is byte-identical. The real payload's `country` and city/state/postal properties are omitted for that determinism. An input with no resolver hint and an `IpAddress` that `ipaddress.ip_address()` refuses answer `ValidationException` (those two wordings are recorded from the live service; every other message in this service is MiniStack's own, since none is documented); a `Timestamp` that is not a Unix timestamp, and a `ConfidencePercent` outside the model's range, are `ValidationException` too; a valid address that is **not globally routable or is multicast** (private, loopback, TEST-NET, multicast groups, ...) answers `ResourceNotFoundException` `Cannot find position for the IP address <ip>`, mirroring the real service's refusal when its geo database has no entry. A well-formed request whose only measurements are WLAN/cell/GNSS is `ResourceNotFoundException` too — the developer guide documents 404 for exactly that ("no location information was found or solved ... The MAC address or cellular tower information is not sufficient ... The GNSS payload is not sufficient"), and 400 for measurement data formatted incorrectly. Stateless — nothing persists || **EventBridge Scheduler** | CreateSchedule, GetSchedule, UpdateSchedule, DeleteSchedule, ListSchedules, CreateScheduleGroup, GetScheduleGroup, DeleteScheduleGroup, ListScheduleGroups, TagResource, UntagResource, ListTagsForResource | 12 actions; schedules **fire their targets** — a background sweep invokes each enabled schedule's target (Lambda, SQS, SNS, Step Functions) when due, honoring `State`, `StartDate`/`EndDate`, and `ActionAfterCompletion: DELETE`; schedule groups with cascading deletes; `rate()`, `cron()`, `at()` expressions; group/prefix/state filters on list; default group auto-created; CFN `AWS::Scheduler::Schedule` and `AWS::Scheduler::ScheduleGroup` supported || **EKS** | CreateCluster, DescribeCluster, ListClusters, DeleteCluster, CreateNodegroup, DescribeNodegroup, ListNodegroups, DeleteNodegroup, CreateAddon, DescribeAddon, ListAddons, UpdateAddon, DeleteAddon, AssociateEncryptionConfig, CreateAccessEntry, DescribeAccessEntry, ListAccessEntries, UpdateAccessEntry, DeleteAccessEntry, AssociateAccessPolicy, DisassociateAccessPolicy, ListAssociatedAccessPolicies, AssociateIdentityProviderConfig, DescribeIdentityProviderConfig, DisassociateIdentityProviderConfig, ListIdentityProviderConfigs, CreatePodIdentityAssociation, DescribePodIdentityAssociation, ListPodIdentityAssociations, UpdatePodIdentityAssociation, DeletePodIdentityAssociation, TagResource, UntagResource, ListTagsForResource | `CreateCluster` spawns a real **k3s** container (75 MB) with a full Kubernetes API server; `kubectl`, Helm, and any K8s tooling work out of the box; cascading delete removes nodegroups and k3s container; addon CRUD (e.g. `vpc-cni`, `coredns`, `kube-proxy`, `aws-ebs-csi-driver`) at `/clusters/{name}/addons` flips status to `ACTIVE` on create / update; `AssociateEncryptionConfig` at `/clusters/{name}/encryption-config/associate` records KMS encryption config (rejects re-association, AWS-shape `update` envelope); **Access Entries** (modern post-1.29 IAM bindings — replaces aws-auth ConfigMap) at `/clusters/{name}/access-entries[/{principalArn}[/access-policies[/{policyArn}]]]` for Crossplane / Terraform `aws_eks_access_entry` + `aws_eks_access_policy_association`; OIDC discovery + JWKS served at the cluster's `identity.oidc.issuer` URL so Terraform's `aws_iam_openid_connect_provider` (IRSA) works against ministack; **OIDC Identity Provider Config** (`AssociateIdentityProviderConfig` / `DescribeIdentityProviderConfig` / `DisassociateIdentityProviderConfig` / `ListIdentityProviderConfigs`) at `/clusters/{name}/identity-provider-configs[/{verb}]` — one OIDC IdP per cluster, issuer + clientId + claims forwarded to the k3s API server via `--kube-apiserver-arg=oidc-*` flags, tags reachable via `ListTagsForResource`; CFN `AWS::EKS::Cluster` and `AWS::EKS::Nodegroup` supported |diff --git a/ministack/services/iot.py b/ministack/services/iot.pyindex 9f2f68ac..f911217b 100644--- a/ministack/services/iot.py+++ b/ministack/services/iot.py@@ -75,7 +75,9 @@ import jsonimport loggingimport osimport re+import secretsimport ssl+import stringimport structimport timeimport unicodedata@@ -83,6 +85,7 @@ import uuidimport weakreffrom datetime import datetime, timezonefrom typing import Awaitable, Callable+from urllib.parse import parse_qsfrom ministack.core.arn import ArnParseError, parse_arnfrom ministack.core.responses import (@@ -96,6 +99,7 @@ from ministack.core.responses import (new_uuid,request_scope,)+from ministack.core.router import extract_access_key_idfrom ministack.core.x509_utils import (certificate_is_signed_by,generate_ca,@@ -4028,10 +4032,12 @@ _IMPLEMENTED_SQL_FUNCS = frozenset({"encode","isundefined","newuuid",+ "principal","regexp_matches","replace","timestamp","topic",+ "traceid",})_warned_sql_funcs: set[str] = set()@@ -4096,9 +4102,10 @@ def _eval_select_function(if name == "clientid" and not args:# Documented as the client id "or n/a if the message wasn't sent over MQTT".return client_id if client_id else "n/a"- # principal() and traceid() land here too: this publish path carries no- # certificate identity or trace id to report, so they warn like any other- # function the evaluator does not implement.+ if name == "principal" and not args:+ return _rule_message_identity.get()[0] or _MISSING+ if name == "traceid" and not args:+ return _rule_message_identity.get()[1] or _MISSING_warn_unimplemented_sql_function(name)return _MISSING@@ -7264,6 +7271,26 @@ _republish_depth: contextvars.ContextVar[int] = contextvars.ContextVar("_iot_republish_depth", default=0)+_ROLE_SESSION_NAME_CHARS = string.ascii_letters + string.digits+++def _rule_role_principal(account_id: str, role_arn: str) -> str | None:+ """principal() of a republished message: the rule role's session, as GetCallerIdentity names it."""+ from ministack.services import iam as _iam++ try:+ arn = parse_arn(role_arn)+ except ArnParseError:+ return None+ if arn.account_id != account_id or not arn.resource.startswith("role/"):+ return None+ role = _iam._roles.get_scoped(account_id, None, arn.resource.rsplit("/", 1)[-1])+ if not role:+ return None+ # The rules engine names its role sessions with eight random letters and digits.+ session = "".join(secrets.choice(_ROLE_SESSION_NAME_CHARS) for _ in range(8))+ return f"{role['RoleId']}:{session}"+async def _dispatch_rule_republish(account_id: str, region: str, rule_name: str, spec: dict, event@@ -7296,6 +7323,7 @@ async def _dispatch_rule_republish(target_topic,json.dumps(event).encode("utf-8"),qos=qos,+ principal=_rule_role_principal(account_id, spec.get("roleArn") or ""),)finally:_republish_depth.reset(token)@@ -7465,8 +7493,8 @@ async def _dispatch_rule_error_action(error_event = {"ruleName": rule_name,"topic": topic,- # The emulator's publish path carries no CloudWatch trace id.- "cloudwatchTraceId": "",+ # The id traceid() returns for the same message.+ "cloudwatchTraceId": _rule_message_identity.get()[1],# "N/A", not "" — and not clientid()'s lowercase "n/a" (measured eu-north-1 2026-09-19)."clientId": client_id or "N/A","sourceIp": _publish_source_ip.get(),@@ -8046,6 +8074,7 @@ async def broker_publish(retain: bool = False,properties: bytes = b"",client_id: str | None = None,+ principal: str | None = None,) -> int:"""Deliver a message to every matching subscriber.@@ -8060,6 +8089,9 @@ async def broker_publish(its own messages back. A publish with no client identity — iot_data's HTTPPublish, a rule, an in-process caller — matches no subscriber's identityand so is delivered to all of them.++ ``principal`` is who published the message, what the rule SQL+ ``principal()`` returns; rules also see a fresh trace id per message."""delivered = 0# Basic Ingest: a publish to `$aws/rules/<ruleName>` is delivered straight@@ -8067,16 +8099,17 @@ async def broker_publish(if topic.startswith(_BASIC_INGEST_PREFIX):remainder = topic[len(_BASIC_INGEST_PREFIX):].split("/", 1)rule_name = remainder[0]- await _run_rule_actions(- account_id,- region,- _topic_rules.get_scoped(account_id, region, rule_name),- payload,- # Under Basic Ingest `topic()` reports the topic after the- # `$aws/rules/<ruleName>/` prefix, as it does on AWS.- remainder[1] if len(remainder) > 1 else "",- client_id,- )+ with _rule_message_scope(principal):+ await _run_rule_actions(+ account_id,+ region,+ _topic_rules.get_scoped(account_id, region, rule_name),+ payload,+ # Under Basic Ingest `topic()` reports the topic after the+ # `$aws/rules/<ruleName>/` prefix, as it does on AWS.+ remainder[1] if len(remainder) > 1 else "",+ client_id,+ )return deliveredscoped = _scoped_topic(account_id, region, topic)@@ -8138,7 +8171,8 @@ async def broker_publish(delivered += 1break- await _evaluate_topic_rules(account_id, region, topic, payload, client_id)+ with _rule_message_scope(principal):+ await _evaluate_topic_rules(account_id, region, topic, payload, client_id)# Shadow-over-MQTT bridge — AFTER delivery and rule evaluation, so rules# matching `$aws/things/+/shadow/update` fire on the request, and the@@ -8524,6 +8558,32 @@ _delivery_retain: contextvars.ContextVar[bool] = contextvars.ContextVar()+# The principal and trace id of the message the rules engine is evaluating, for+# principal(), traceid() and the errorAction document. Scoped to one message's+# rule evaluation, so a republished message carries its own.+_rule_message_identity: contextvars.ContextVar[tuple[str | None, str]] = contextvars.ContextVar(+ "_iot_rule_message_identity", default=(None, "")+)+++def _rule_message_scope(principal: str | None):+ """Scope a message's principal and a fresh trace id over its rule evaluation."""+ token = _rule_message_identity.set((principal, new_uuid()))+ try:+ yield+ finally:+ _rule_message_identity.reset(token)+++def _caller_principal(access_key: str) -> str:+ """principal() of an IAM-signed publish: the caller's user id as GetCallerIdentity reports it."""+ from ministack.core.iam_evaluator import resolve_caller_identity++ identity = resolve_caller_identity(access_key)+ return identity["userId"] if identity else get_account_id()++def _encode_remaining_length(n: int) -> bytes:out = bytearray()while True:@@ -8772,7 +8832,8 @@ class _WSSession:self._send = send_coroself.account_id = account_idself.region = region- # Lifecycle-event principal: the client certificate id, else the account.+ # The client certificate id or the IAM caller's user id, else the account:+ # lifecycle events' principalIdentifier and the rule SQL principal().self._principal = principal or account_idself._session_identifier = ""self._presence_version = 0@@ -9222,6 +9283,7 @@ class _WSSession:retain=retain,properties=properties,client_id=self._client_id or None,+ principal=self._principal,)if qos == 1 and packet_id is not None:if self.protocol_version == MQTT_5:@@ -9478,7 +9540,9 @@ async def handle_websocket(await send(accept)ctx_token = _request_account_id.set(account_id)- session = _WSSession(send, account_id, region)+ query = parse_qs(scope.get("query_string", b"").decode("latin-1"))+ principal = _caller_principal(extract_access_key_id(sub_headers, query))+ session = _WSSession(send, account_id, region, principal)max_buffer = _max_frame_buffer_bytes()try:diff --git a/ministack/services/iot_data.py b/ministack/services/iot_data.pyindex 675c2447..9185cdb1 100644--- a/ministack/services/iot_data.py+++ b/ministack/services/iot_data.py@@ -30,6 +30,7 @@ from ministack.core.responses import (get_region,json_response,)+from ministack.core.router import extract_access_key_idfrom ministack.services import iot as _iot_modulelogger = logging.getLogger("iot_data")@@ -120,7 +121,8 @@ async def handle_request()if method == "POST" and path.startswith("/topics/"):- return await _publish(path[len("/topics/"):], body, qp)+ access_key = extract_access_key_id(headers, query_params)+ return await _publish(path[len("/topics/"):], body, qp, access_key)# Device Shadow: /things/{thingName}/shadow[?name={shadowName}]if path.startswith("/things/") and path.endswith("/shadow"):@@ -199,7 +201,7 @@ async def _shadow(method: str, path: str, body: bytes, qp: dict) -> tuple:return json_response(doc)-async def _publish(raw_topic: str, body: bytes, qp: dict) -> tuple:+async def _publish(raw_topic: str, body: bytes, qp: dict, access_key: str) -> tuple:"""``POST /topics/{topic}`` — forward body to the broker.``raw_topic`` is what came after ``/topics/`` in the URL. The SDK URL-@@ -237,6 +239,7 @@ async def _publish(raw_topic: str, body: bytes, qp: dict) -> tuple:body or b"",qos=qos,retain=retain,+ principal=_iot_module._caller_principal(access_key),)except Exception as e:logger.exception("iot-data publish failed")diff --git a/tests/test_iot.py b/tests/test_iot.pyindex 74485870..f33fff55 100644--- a/tests/test_iot.py+++ b/tests/test_iot.py@@ -4690,8 +4690,8 @@ def test_rule_event_from_less_basic_ingest_projects():# -----------------------------------------------------------------------# Rule SQL functions: newuuid(), replace(), clientid() (and the accepted-# but unresolvable principal() / traceid()).+# Rule SQL functions: newuuid(), replace(), clientid(), principal(),+# traceid().# ----------------------------------------------------------------------@@ -4757,22 +4757,23 @@ def test_rule_event_replace_handles_escaped_quotes():assert event == {"n": "it is on"}-def test_rule_event_principal_and_traceid_accepted_but_unresolved(caplog):- """Both are real AWS functions, so a rule using them is stored rather than- failing a stack that deploys on AWS — but this publish path carries no- certificate identity or trace id, so they warn like any other function the- evaluator does not implement."""+def test_rule_event_principal_and_traceid_read_the_message_identity(caplog):+ """Both read the message under evaluation, without a warning, and are Undefined outside one."""from ministack.services import iot as iot_module- from ministack.services.iot import _rule_event, _validate_rule_sql+ from ministack.services.iot import _rule_event, _unimplemented_sql_functions, _validate_rule_sqlsql = "SELECT principal() AS p, traceid() AS t, temp FROM 't'"assert _validate_rule_sql(sql) is None- # The evaluator warns once per function name for the life of the process.+ assert _unimplemented_sql_functions(sql) == []iot_module._warned_sql_funcs.clear()with caplog.at_level(logging.WARNING, logger="iot"):assert _rule_event(sql, "t", b'{"temp": 1}') == {"temp": 1}- warned = " ".join(r.message for r in caplog.records)- assert "principal()" in warned and "traceid()" in warned+ token = iot_module._rule_message_identity.set(("c0ffee", "trace-1"))+ try:+ assert _rule_event(sql, "t", b'{"temp": 1}') == {"p": "c0ffee", "t": "trace-1", "temp": 1}+ finally:+ iot_module._rule_message_identity.reset(token)+ assert not caplog.recordsdef test_eval_where_clientid_reaches_every_leaf_form():@@ -4803,6 +4804,31 @@ def test_eval_where_clientid_reaches_every_leaf_form():assert _eval_where("clientid() = 'n/a'", "t", raw, message) is True+def test_eval_where_principal_and_traceid_are_operands():+ """A rule can gate on who published (`WHERE principal() = '<certificate id>'`)."""+ from ministack.services import iot as iot_module+ from ministack.services.iot import _eval_where, _rule_message++ raw = b'{"temp": 1}'+ message = _rule_message(raw)+ token = iot_module._rule_message_identity.set(("c0ffee", "trace-1"))+ try:+ for pred, expected in (+ ("principal() = 'c0ffee'", True),+ ("principal() IN ('beef', 'c0ffee')", True),+ ("principal() LIKE 'c0%'", True),+ ("principal() = 'beef'", False),+ ("traceid() = 'trace-1'", True),+ ("isUndefined(traceid())", False),+ ):+ assert _eval_where(pred, "t", raw, message) is expected+ finally:+ iot_module._rule_message_identity.reset(token)+ # With no message identity both are Undefined, so a comparison fails closed.+ assert _eval_where("principal() = 'c0ffee'", "t", raw, message) is False+ assert _eval_where("isUndefined(traceid())", "t", raw, message) is True++def test_broker_publish_threads_client_id_into_rule_projection(monkeypatch):"""`clientid()` sees the WS client's id via broker_publish, end to end."""from ministack.services import iot as iot_module@@ -4848,6 +4874,56 @@ def test_broker_publish_threads_client_id_into_rule_projection(monkeypatch):reset()+def test_broker_publish_threads_principal_and_trace_id_into_rules(monkeypatch):+ """principal() is the publisher's; traceid() is one id per message, shared by every rule it reaches."""+ from ministack.services import iot as iot_module++ reset()+ iot_module._topic_rules.clear()+ account_id = "123456789012"+ for name in ("pt_a", "pt_b"):+ iot_module._topic_rules.set_scoped(+ account_id,+ _TEST_REGION,+ name,+ {+ "ruleName": name,+ "sql": "SELECT principal() AS p, traceid() AS t FROM 'sensors/#'",+ "ruleDisabled": False,+ "actions": [{"lambda": {"functionArn": f"arn:aws:lambda:us-east-1:123456789012:function:{name}"}}],+ },+ )+ dispatched = []+ monkeypatch.setattr(+ iot_module,+ "_dispatch_rule_to_lambda",+ lambda account, region, arn, event: dispatched.append((arn.rsplit(":", 1)[-1], event)),+ )++ async def _run():+ send, _sent = _mock_send()+ session = _IoTWSSession(send, account_id, _TEST_REGION, "c0ffee")+ await session.handle_packet(PKT_CONNECT, 0, _build_connect_body("sensor-7"))+ for _ in range(2):+ await session.handle_packet(PKT_PUBLISH, 0, _encode_string("sensors/door") + b"{}")+ await session.cleanup()+ # An in-process publish names no principal; it still gets a trace id.+ await publish(account_id, "sensors/door", b"{}")++ try:+ asyncio.run(_run())+ assert [rule for rule, _event in dispatched] == ["pt_a", "pt_b"] * 3+ events = [event for _rule, event in dispatched]+ assert [event.get("p") for event in events] == ["c0ffee"] * 4 + [None] * 2+ trace_ids = [event["t"] for event in events]+ assert all(str(uuid.UUID(t)) == t for t in trace_ids)+ assert trace_ids[0] == trace_ids[1] and trace_ids[2] == trace_ids[3] and trace_ids[4] == trace_ids[5]+ assert len(set(trace_ids)) == 3+ finally:+ iot_module._topic_rules.clear()+ reset()++# ----------------------------------------------------------------------# Rule SQL WHERE clause (white-box tests for _rule_where_clause /# _eval_where / _validate_rule_sql).@@ -5405,6 +5481,54 @@ def test_rule_republish_action_delivers_projected_event():reset()+def test_rule_republish_is_a_new_message_from_the_rule_role_session():+ """A republished message comes from the rule role's session and gets its own trace id."""+ from ministack.services import iam as iam_module+ from ministack.services import iot as iot_module++ reset()+ iot_module._topic_rules.clear()+ account_id = "123456789012"+ iam_module._roles.set_scoped(+ account_id, None, "rule-role", {"RoleName": "rule-role", "RoleId": "AROAEXAMPLEROLEID0001"}+ )+ role_arn = f"arn:aws:iam::{account_id}:role/rule-role"+ _put_rule(+ account_id,+ "chain_first",+ "SELECT traceid() AS t FROM 'chain/in'",+ [{"republish": {"topic": "chain/mid", "roleArn": role_arn}}],+ )+ _put_rule(+ account_id,+ "chain_second",+ "SELECT principal() AS p, traceid() AS t, t AS original FROM 'chain/mid'",+ [{"republish": {"topic": "chain/out", "roleArn": role_arn}}],+ )+ received = []++ async def _run():+ async def _collect(topic, payload, qos):+ received.append(json.loads(payload))++ await subscribe(account_id, "chain/out", _collect)+ await publish(account_id, "chain/in", b"{}", principal="c0ffee")++ try:+ asyncio.run(_run())+ [event] = received+ assert re.fullmatch(r"AROAEXAMPLEROLEID0001:[A-Za-z0-9]{8}", event["p"])+ assert str(uuid.UUID(event["t"])) == event["t"]+ assert event["t"] != event["original"]+ # A role the account does not have names no session.+ assert iot_module._rule_role_principal(account_id, f"arn:aws:iam::{account_id}:role/absent") is None+ assert iot_module._rule_role_principal("210987654321", role_arn) is None+ finally:+ iam_module._roles.pop_scoped(account_id, None, "rule-role", None)+ iot_module._topic_rules.clear()+ reset()++def test_rule_republish_onto_own_topic_terminates():"""A rule republishing onto its own topic filter is user error, but thebroker must cut the chain instead of recursing forever."""@@ -5616,6 +5740,56 @@ def test_rule_error_action_runs_when_an_action_fails(monkeypatch):reset()+def test_rule_error_action_reports_the_message_trace_id(monkeypatch):+ """The errorAction document's cloudwatchTraceId is the message's traceid()."""+ from ministack.services import iot as iot_module++ reset()+ iot_module._topic_rules.clear()+ account_id = "123456789012"+ iot_module._topic_rules.set_scoped(+ account_id,+ _TEST_REGION,+ "err_rule",+ {+ "ruleName": "err_rule",+ "sql": "SELECT * FROM 'err/topic'",+ "ruleDisabled": False,+ "actions": [{"dynamoDBv2": {"putItem": {"tableName": "absent"}}}],+ "errorAction": {"republish": {"topic": "err/dlq"}},+ },+ )+ _put_rule(+ account_id,+ "trace_rule",+ "SELECT traceid() AS t FROM 'err/topic'",+ [{"republish": {"topic": "err/trace"}}],+ )++ def _boom(*args, **kwargs):+ raise RuntimeError("dispatch blew up")++ monkeypatch.setattr(iot_module, "_dispatch_rule_dynamodb", _boom)+ received = {}++ async def _run():+ async def _collect(topic, payload, qos):+ received[topic] = json.loads(payload)++ await subscribe(account_id, "err/dlq", _collect)+ await subscribe(account_id, "err/trace", _collect)+ await publish(account_id, "err/topic", b'{"n": 1}')++ try:+ asyncio.run(_run())+ trace_id = received["err/trace"]["t"]+ assert str(uuid.UUID(trace_id)) == trace_id+ assert received["err/dlq"]["cloudwatchTraceId"] == trace_id+ finally:+ iot_module._topic_rules.clear()+ reset()++@pytest.mark.parametrize(("action", "action_type", "error_fragment"),[diff --git a/tests/test_iot_data.py b/tests/test_iot_data.pyindex 2a01bd3b..44872a81 100644--- a/tests/test_iot_data.py+++ b/tests/test_iot_data.py@@ -1679,6 +1679,141 @@ def test_iot_rule_sqs_action_applies_select_projection(iot_client, iot_data_clieiot_client.delete_topic_rule(ruleName=rule)+def _take_bodies(sqs, url, count, timeout=10):+ """The JSON bodies of the next ``count`` messages, deleted as they are read."""+ bodies = []+ deadline = time.time() + timeout+ while len(bodies) < count and time.time() < deadline:+ msg = _poll_message(sqs, url, timeout=max(0.0, deadline - time.time()))+ if msg:+ sqs.delete_message(QueueUrl=url, ReceiptHandle=msg["ReceiptHandle"])+ bodies.append(json.loads(msg["Body"]))+ return bodies+++def _rule_role(iam):+ """A role the rules engine can assume, so the rule also creates under AUTH=true."""+ name = _unique("rule-role")+ trust = {+ "Version": "2012-10-17",+ "Statement": [+ {"Effect": "Allow", "Principal": {"Service": "iot.amazonaws.com"}, "Action": "sts:AssumeRole"}+ ],+ }+ return iam.create_role(RoleName=name, AssumeRolePolicyDocument=json.dumps(trust))["Role"]+++def _iam_user_key(iam):+ """A fresh IAM user allowed to publish, its access key, and its GetCallerIdentity UserId."""+ user = _unique("publisher")+ iam.create_user(UserName=user)+ iam.put_user_policy(+ UserName=user,+ PolicyName="publish",+ PolicyDocument=json.dumps(+ {"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": "iot:*", "Resource": "*"}]}+ ),+ )+ key = iam.create_access_key(UserName=user)["AccessKey"]+ caller = boto3.client(+ "sts",+ endpoint_url=ENDPOINT,+ aws_access_key_id=key["AccessKeyId"],+ aws_secret_access_key=key["SecretAccessKey"],+ region_name="us-east-1",+ ).get_caller_identity()+ return user, key, caller["UserId"]+++def _drop_iam_user(iam, user, key):+ iam.delete_access_key(UserName=user, AccessKeyId=key["AccessKeyId"])+ iam.delete_user_policy(UserName=user, PolicyName="publish")+ iam.delete_user(UserName=user)+++def test_iot_rule_principal_is_the_http_callers_user_id(iot_client, iam, sqs, sts):+ """principal() of an HTTP Publish is the caller's UserId; every message gets its own trace id."""+ queue = sqs.create_queue(QueueName=_unique("rule-principal"))["QueueUrl"]+ topic = _unique("principal/http")+ rule = _unique("principal").replace("-", "_")+ user, key, user_id = _iam_user_key(iam)+ role = _rule_role(iam)+ iot_client.create_topic_rule(+ ruleName=rule,+ topicRulePayload={+ "sql": f"SELECT principal() AS p, traceid() AS t FROM '{topic}'",+ "actions": [{"sqs": {"queueUrl": queue, "roleArn": role["Arn"]}}],+ },+ )+ try:+ callers = (+ (key["AccessKeyId"], key["SecretAccessKey"], user_id),+ ("test", "test", sts.get_caller_identity()["UserId"]),+ )+ trace_ids = []+ for access_key, secret, expected in callers:+ boto3.client(+ "iot-data",+ endpoint_url=ENDPOINT,+ aws_access_key_id=access_key,+ aws_secret_access_key=secret,+ region_name="us-east-1",+ ).publish(topic=topic, payload=b"{}")+ [event] = _take_bodies(sqs, queue, 1)+ assert event["p"] == expected+ assert str(uuid.UUID(event["t"])) == event["t"]+ trace_ids.append(event["t"])+ assert trace_ids[0] != trace_ids[1]+ finally:+ iot_client.delete_topic_rule(ruleName=rule)+ iam.delete_role(RoleName=role["RoleName"])+ _drop_iam_user(iam, user, key)+++def test_iot_rule_principal_is_the_websocket_signers_user_id(iot_client, iam, sqs):+ """principal() and the connected event's principalIdentifier are the WebSocket signer's UserId."""+ queue = sqs.create_queue(QueueName=_unique("rule-ws-principal"))["QueueUrl"]+ topic = _unique("principal/ws")+ client_id = _unique("ws-signed")+ user, key, user_id = _iam_user_key(iam)+ role = _rule_role(iam)+ rules = {+ _unique("wspub").replace("-", "_"): f"SELECT 'publish' AS kind, principal() AS p FROM '{topic}'",+ _unique("wscon").replace("-", "_"): (+ "SELECT 'connected' AS kind, principalIdentifier AS p "+ f"FROM '$aws/events/presence/connected/{client_id}'"+ ),+ }+ for name, sql in rules.items():+ iot_client.create_topic_rule(+ ruleName=name,+ topicRulePayload={"sql": sql, "actions": [{"sqs": {"queueUrl": queue, "roleArn": role["Arn"]}}]},+ )+ parsed = urlparse(ENDPOINT)+ cred = quote(f"{key['AccessKeyId']}/20240101/us-east-1/iotdevicegateway/aws4_request")+ url = (+ f"ws://prefix-ats.iot.us-east-1.{parsed.hostname or 'localhost'}:{parsed.port or 4566}/mqtt"+ f"?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential={cred}"+ )++ async def _run():+ ws = await _mqtt_connect(client_id, url)+ await ws.send(_make_publish(topic, b"{}"))+ await asyncio.sleep(0.5)+ await _mqtt_disconnect(ws)++ try:+ with patch_endpoint_dns():+ asyncio.run(_run())+ events = {event["kind"]: event["p"] for event in _take_bodies(sqs, queue, 2)}+ assert events == {"publish": user_id, "connected": user_id}+ finally:+ for name in rules:+ iot_client.delete_topic_rule(ruleName=name)+ iam.delete_role(RoleName=role["RoleName"])+ _drop_iam_user(iam, user, key)++def test_iot_rule_sqs_action_missing_queue_does_not_stop_the_rule(iot_client, iot_data_client, sqs):@@ -3571,6 +3706,37 @@ def test_mtls_publish_is_brokered(broker, tmp_path):subscriber.close()+def test_mtls_publish_principal_is_the_certificate_id(broker, tmp_path):+ """principal() of an mTLS publish is the id of the client certificate."""+ cert_id, cert_pem, key_pem = _new_cert(broker)+ topic = _unique("mtls/principal")+ rule = _unique("mtlsprincipal").replace("-", "_")+ role = _rule_role(broker.client("iam"))+ broker.client("iot").create_topic_rule(+ ruleName=rule,+ topicRulePayload={+ "sql": f"SELECT principal() AS p, traceid() AS t FROM '{topic}'",+ "actions": [{"republish": {"topic": f"{topic}/out", "roleArn": role["Arn"]}}],+ },+ )+ peer = _Peer(_mtls_connect(broker, cert_pem, key_pem, tmp_path))+ try:+ _assert_connack(peer.connect(_unique("device")))+ peer.subscribe(f"{topic}/out")+ peer.send(_make_publish(topic, b"{}"))+ received = peer.next_publish()+ assert received is not None, "the rule's republish did not arrive"+ out_topic, payload = received+ event = json.loads(payload)+ assert out_topic == f"{topic}/out"+ assert event["p"] == cert_id+ assert str(uuid.UUID(event["t"])) == event["t"]+ finally:+ peer.close()+ broker.client("iot").delete_topic_rule(ruleName=rule)+ broker.client("iam").delete_role(RoleName=role["RoleName"])++def test_mtls_no_client_cert_uses_default_account(broker, tmp_path):"""No client certificate connects fine, exactly like the WebSocket path,and the session runs under the default account."""