Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

feat(iot): presign S3 URL placeholders in job documents (#2175)

iot-rocket333a062ddffb
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3e783cef..5f0d6cf6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -24,6 +24,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).
### Added
+- **IoT — presigned URLs in job documents** — devices got `${aws:iot:s3-presigned-url:<url>}` and `${aws:iot:s3-presigned-url-v2:<url>}` placeholders as written; the `iot-jobs-data` API and the MQTT jobs topics now replace each with a SigV4 presigned GET URL of MiniStack's S3, signed with a session of `presignedUrlConfig.roleArn` and valid for `expiresInSec`, and `CreateJob` and `CreateJobTemplate` refuse a placeholder without a role or an S3 object URL with AWS's messages. Contributed by @iot-rocket.
- **IoT — job documents from S3** — `CreateJob` and `CreateJobTemplate` read a `documentSource` (`s3://` or an S3 object URL) from MiniStack's S3 once, at creation, where they stored a `{"documentSource": "<url>"}` placeholder. A missing bucket or object, an object that is not JSON or is over 32 KB and a URL that names no S3 host are refused with AWS's messages, and `CreateJob` refuses `document` together with `documentSource`. Devices get every job document re-serialized as AWS sends it, without spaces and without object members that are null, while `GetJobDocument` keeps the stored text. Contributed by @iot-rocket.
- **IoT — `SetDefaultPolicyVersion`** — the call was missing (`PATCH /policies/{policyName}/version/{policyVersionId}` answered `Unsupported policy path`), so only `CreatePolicyVersion` with `setAsDefault` could move a policy's default version; it now sets the default that `GetPolicy`, `GetPolicyVersion`, `ListPolicyVersions` and the MQTT policy evaluation under `AUTH=true` use, deleting the default version answers `DeleteConflictException` (409) instead of 400, and `ListPolicyVersions` lists the newest version first. Contributed by @iot-rocket.
- **IoT — `EnableTopicRule` and `DisableTopicRule`** — both calls were missing (`POST /rules/{ruleName}/enable` answered `Invalid ruleName`), so a rule could only be switched off by replacing it; they now set `ruleDisabled`, which `GetTopicRule` and `ListTopicRules` report and which stops the rule from acting on matching publishes and Basic Ingest until it is enabled again. Contributed by @iot-rocket.
diff --git a/README.md b/README.md
index 0285e03b..008dd79c 100644
--- a/README.md
+++ b/README.md
@@ -646,7 +646,7 @@ MiniStack expands AWS SAM transforms (`Transform: AWS::Serverless-2016-10-31`) i
| **CodeBuild** | CreateProject, BatchGetProjects, ListProjects, UpdateProject, DeleteProject, StartBuild, BatchGetBuilds, StopBuild, ListBuilds, ListBuildsForProject, BatchDeleteBuilds | 11 actions; project and build metadata stored in-memory; builds complete immediately with SUCCEEDED status by default. Set `MINISTACK_CODEBUILD_EXECUTE=1` and mount the Docker socket to **really run them**: the project's inline buildspec goes to the official AWS CodeBuild local agent, which runs the phases in `environment.image`, and `BatchGetBuilds` reports live `phases` with a real `SUCCEEDED`/`FAILED` outcome. Execution needs the workspace shared with the host at the **same path** (`-v /tmp/ministack-codebuild:/tmp/ministack-codebuild`), because the agent bind-mounts it into the build container; a build gets `AWS_ENDPOINT_URL` pointing back at MiniStack, honours `timeoutInMinutes`, and is stopped by `StopBuild`. An executed build streams the agent output into its `logs.groupName` / `logs.streamName`, so `aws logs tail` reads the build log back |
| **AppConfig** | CreateApplication, GetApplication, ListApplications, UpdateApplication, DeleteApplication, CreateEnvironment, GetEnvironment, ListEnvironments, UpdateEnvironment, DeleteEnvironment, CreateConfigurationProfile, GetConfigurationProfile, ListConfigurationProfiles, UpdateConfigurationProfile, DeleteConfigurationProfile, CreateHostedConfigurationVersion, GetHostedConfigurationVersion, ListHostedConfigurationVersions, DeleteHostedConfigurationVersion, CreateDeploymentStrategy, GetDeploymentStrategy, ListDeploymentStrategies, UpdateDeploymentStrategy, DeleteDeploymentStrategy, StartDeployment, GetDeployment, ListDeployments, StopDeployment, TagResource, UntagResource, ListTagsForResource, StartConfigurationSession, GetLatestConfiguration | 33 operations; control plane + data plane; hosted configuration versions; the AWS-predefined deployment strategies (4) are listed and immutable; deployments move through `DEPLOYING`, `BAKING` and `COMPLETE` on the strategy's schedule (`LINEAR` or `EXPONENTIAL` steps, then the bake time), with `PercentageComplete`, the event log and the environment `State` reported along the way; a `Monitors` alarm that goes to `ALARM` before the bake ends rolls the deployment back; one deployment per environment at a time (`ConflictException`, also for a stale `LatestDeploymentNumber`); `StopDeployment` rolls back a `DEPLOYING` deployment, and with `AllowRevert` also a `BAKING` one, or reverts a completed one within 72 hours; during a rollout a session gets the new version once the rollout reaches its fixed slot (MiniStack's choice, since AWS does not document how it picks targets); `GetLatestConfiguration` returns an empty body when the session already has the latest version, plus `Version-Label`, and uses `RequiredMinimumPollIntervalInSeconds` as the poll interval |
| **Transfer Family** | CreateServer, DescribeServer, DeleteServer, ListServers, StartServer, StopServer, CreateUser, DescribeUser, DeleteUser, ListUsers, ImportSshPublicKey, DeleteSshPublicKey | 12 operations; **real SFTP listener** on `:2222` (override with `SFTP_PORT`) backed by S3 — `ls`, `put`, `get`, `mkdir`, `rename` work end-to-end against the local S3 emulator; `SFTP_PORT_PER_SERVER=1` allocates one port per `CreateServer` from `SFTP_BASE_PORT` (default 2300); SSH key auth scans every user across every server and account; `LOGICAL` and `PATH` home-directory mappings; host key persists across restarts when `PERSIST_STATE=1` |
-| **IoT Jobs** | Control plane (`iot`): CreateJob, UpdateJob, DescribeJob, ListJobs, GetJobDocument, CancelJob, DeleteJob, ListJobExecutionsForThing, ListJobExecutionsForJob, DescribeJobExecution, CancelJobExecution, DeleteJobExecution, CreateJobTemplate, DescribeJobTemplate, ListJobTemplates, DeleteJobTemplate; data plane (`iot-jobs-data`): GetPendingJobExecutions, StartNextPendingJobExecution, DescribeJobExecution (incl. the `$next` sentinel), UpdateJobExecution | 16 control-plane + 4 device data-plane operations sharing one store. A job created with a `jobTemplateArn` takes the template's document, configuration and maintenance windows, and a request member replaces the template's. Executions walk `QUEUED → IN_PROGRESS → {SUCCEEDED, FAILED, REJECTED, CANCELED, TIMED_OUT}`; an execution past its `timeoutConfig.inProgressTimeoutInMinutes`, or past the `stepTimeoutInMinutes` the device reported, becomes `TIMED_OUT`, and the device plane reports `approximateSecondsBeforeTimedOut` while it runs. `REMOVED` is counted in `jobProcessDetails`; a thing that leaves every target group of a `CONTINUOUS` job turns its `QUEUED` execution `REMOVED`, or gets a `REMOVED` execution 1 (without an event) if all its executions were deleted; deleting a thing DELETES its executions (freeing a job that was only waiting on that thing) rather than marking them `REMOVED`. `versionNumber` gives optimistic concurrency: every transition bumps it and a stale `expectedVersion` is rejected with a 409 whose message carries the current version — coded `VersionConflictException` on the control plane and `InvalidStateTransitionException` on the device plane, each as its own service model declares. `executionNumber` is 1 for a thing's first execution of a job and one more each time the thing rejoins a `CONTINUOUS` job's target group after its execution finished. Devices may only report `IN_PROGRESS` / `SUCCEEDED` / `FAILED` / `REJECTED` via `UpdateJobExecution`; targets are thing or thing-group ARNs — `SNAPSHOT` jobs resolve group membership once at creation, `CONTINUOUS` jobs also pick up things added to a target group later, and a target must name a thing or thing group in the caller's own account and region (anything else is `ResourceNotFoundException`, since it would resolve to no devices at all); a non-continuous job auto-`COMPLETED`s (with `jobProcessDetails` counts) once every execution is terminal. **`documentSource` is read from MiniStack's S3**: a job or job template created with an `s3://` or S3 object URL keeps the object's content as its document, read once at creation; a missing bucket or object, a non-JSON object or one over 32 KB is refused, and so is a URL whose host is not S3. **Timestamps are epoch seconds on every plane, as AWS documents**: control-plane responses use `timestamp` shapes (boto3 parses them to datetimes) while `iot-jobs-data` responses carry whole-second integer `long`s. **Jobs over MQTT**: the reserved `$aws/things/<thing>/jobs/#` request topics (`get`, `start-next`, `<jobId>/get` incl. `$next`, `<jobId>/update`) answer on their `accepted`/`rejected` topics from the same store as the HTTP plane (`jobDocument` as a JSON object over MQTT, a string over HTTP, as on AWS; rejections carry the string `ErrorCode` values and the execution's state on version and state-transition conflicts); `notify` fires when the pending set changes and `notify-next` when the queue front changes; `stepTimeoutInMinutes` and `executionNumber` are accepted but ignored. Job state persists with the IoT store when `PERSIST_STATE=1` |
+| **IoT Jobs** | Control plane (`iot`): CreateJob, UpdateJob, DescribeJob, ListJobs, GetJobDocument, CancelJob, DeleteJob, ListJobExecutionsForThing, ListJobExecutionsForJob, DescribeJobExecution, CancelJobExecution, DeleteJobExecution, CreateJobTemplate, DescribeJobTemplate, ListJobTemplates, DeleteJobTemplate; data plane (`iot-jobs-data`): GetPendingJobExecutions, StartNextPendingJobExecution, DescribeJobExecution (incl. the `$next` sentinel), UpdateJobExecution | 16 control-plane + 4 device data-plane operations sharing one store. A job created with a `jobTemplateArn` takes the template's document, configuration and maintenance windows, and a request member replaces the template's. Executions walk `QUEUED → IN_PROGRESS → {SUCCEEDED, FAILED, REJECTED, CANCELED, TIMED_OUT}`; an execution past its `timeoutConfig.inProgressTimeoutInMinutes`, or past the `stepTimeoutInMinutes` the device reported, becomes `TIMED_OUT`, and the device plane reports `approximateSecondsBeforeTimedOut` while it runs. `REMOVED` is counted in `jobProcessDetails`; a thing that leaves every target group of a `CONTINUOUS` job turns its `QUEUED` execution `REMOVED`, or gets a `REMOVED` execution 1 (without an event) if all its executions were deleted; deleting a thing DELETES its executions (freeing a job that was only waiting on that thing) rather than marking them `REMOVED`. `versionNumber` gives optimistic concurrency: every transition bumps it and a stale `expectedVersion` is rejected with a 409 whose message carries the current version — coded `VersionConflictException` on the control plane and `InvalidStateTransitionException` on the device plane, each as its own service model declares. `executionNumber` is 1 for a thing's first execution of a job and one more each time the thing rejoins a `CONTINUOUS` job's target group after its execution finished. Devices may only report `IN_PROGRESS` / `SUCCEEDED` / `FAILED` / `REJECTED` via `UpdateJobExecution`; targets are thing or thing-group ARNs — `SNAPSHOT` jobs resolve group membership once at creation, `CONTINUOUS` jobs also pick up things added to a target group later, and a target must name a thing or thing group in the caller's own account and region (anything else is `ResourceNotFoundException`, since it would resolve to no devices at all); a non-continuous job auto-`COMPLETED`s (with `jobProcessDetails` counts) once every execution is terminal. **`documentSource` is read from MiniStack's S3**: a job or job template created with an `s3://` or S3 object URL keeps the object's content as its document, read once at creation; a missing bucket or object, a non-JSON object or one over 32 KB is refused, and so is a URL whose host is not S3. **Presigned URLs**: a `${aws:iot:s3-presigned-url:<url>}` or `-v2` placeholder needs `presignedUrlConfig.roleArn`, and what devices receive (HTTP and MQTT) carries a SigV4 presigned GET URL of MiniStack's S3 instead, signed with a session of that role. The URL is path style on the host the device used, and an IoT endpoint host drops everything up to and including its region label. **Timestamps are epoch seconds on every plane, as AWS documents**: control-plane responses use `timestamp` shapes (boto3 parses them to datetimes) while `iot-jobs-data` responses carry whole-second integer `long`s. **Jobs over MQTT**: the reserved `$aws/things/<thing>/jobs/#` request topics (`get`, `start-next`, `<jobId>/get` incl. `$next`, `<jobId>/update`) answer on their `accepted`/`rejected` topics from the same store as the HTTP plane (`jobDocument` as a JSON object over MQTT, a string over HTTP, as on AWS; rejections carry the string `ErrorCode` values and the execution's state on version and state-transition conflicts); `notify` fires when the pending set changes and `notify-next` when the queue front changes; `stepTimeoutInMinutes` and `executionNumber` are accepted but ignored. Job state persists with the IoT store when `PERSIST_STATE=1` |
| **IoT Core** | CreateThing, DescribeThing, ListThings, UpdateThing, DeleteThing, CreateThingType, CreateThingGroup, AddThingToThingGroup, ListThingsInThingGroup, ListThingGroupsForThing, CreateKeysAndCertificate, RegisterCertificate, RegisterCertificateWithoutCA, CreateCertificateFromCsr, UpdateCertificate, DeleteCertificate, GetRegistrationCode, DeleteRegistrationCode, RegisterCACertificate, DescribeCACertificate, UpdateCACertificate, ListCACertificates, DeleteCACertificate, ListCertificatesByCA, CreateProvisioningTemplate, DescribeProvisioningTemplate, ListProvisioningTemplates, UpdateProvisioningTemplate, DeleteProvisioningTemplate, AttachThingPrincipal, DetachThingPrincipal, ListThingPrincipals, ListPrincipalThings, CreatePolicy, CreatePolicyVersion, SetDefaultPolicyVersion, AttachPolicy, DetachPolicy, ListAttachedPolicies, ListTargetsForPolicy, AttachPrincipalPolicy, DetachPrincipalPolicy, ListPrincipalPolicies, ListPolicyPrincipals, CreateTopicRule, GetTopicRule, ReplaceTopicRule, DeleteTopicRule, ListTopicRules, EnableTopicRule, DisableTopicRule, UpdateIndexingConfiguration, GetIndexingConfiguration, DescribeIndex, ListIndices, SearchIndex, DescribeEndpoint, Publish (HTTP), GetRetainedMessage, ListRetainedMessages, MQTT-over-WebSocket pub/sub, DescribeEventConfigurations, UpdateEventConfigurations, CreateDomainConfiguration, DescribeDomainConfiguration, ListDomainConfigurations, UpdateDomainConfiguration, DeleteDomainConfiguration, TagResource, UntagResource, ListTagsForResource | 71 operations + **MQTT 3.1.1 and 5.0 over WebSocket** (negotiated per connection) on the gateway port — clients use the address returned by `DescribeEndpoint`; local CA signs `CreateKeysAndCertificate` certificates (CA persists across restarts when `PERSIST_STATE=1`); multi-tenancy via transparent topic prefixing (same pattern as Transfer Family's shared SFTP listener); no plain TCP 1883 (real AWS IoT requires TLS or SigV4 on every connection), but a **native mTLS MQTT listener on 8883** serves device SDK binaries by default — the client certificate is optional (no certificate means the default account, like an unsigned WebSocket upgrade) and, when presented, must be registered `ACTIVE` in exactly one account, which then serves the session; unknown, inactive or doubly-registered certificates get a "not authorized" CONNACK (see `IOT_MTLS_PORT` in the environment table); local CA cert available at `GET /_ministack/iot/ca.pem` for SDK trust configuration; IoT policies are stored but **not enforced** on the data plane, except on mTLS MQTT connections with `AUTH=true`; **fleet indexing** works the way it does on AWS — `UpdateIndexingConfiguration` turns the `AWS_Things` index on (Terraform `aws_iot_indexing_configuration`), and searching while `thingIndexingMode` is `OFF` answers `ResourceNotFoundException`; `SearchIndex` then queries the live registry + classic shadows with `AND`-separated `thingName` / `thingTypeName` / `thingGroupNames` / `attributes.*` / `shadow.reported.*` / `shadow.desired.*` terms (`*` / `?` wildcards, `maxResults` / `nextToken` paging) — there is no projection to build, so hits are always current; the deprecated principal-policy family (`AttachPrincipalPolicy` / `DetachPrincipalPolicy` / `ListPrincipalPolicies` / `ListPolicyPrincipals`) maps 1:1 onto the modern policy-target store; **topic rules fire on matching publishes** (incl. Basic Ingest `$aws/rules/<name>`) with SQL SELECT projection (`topic()`, `timestamp()`, `encode()`, `newuuid()`, `replace()`, `clientid()`, `principal()`, `traceid()`, `isUndefined()`), **WHERE-clause evaluation** (comparisons, `IN`, `LIKE`, `BETWEEN`, `IS NULL`, `regexp_matches`, bare boolean attributes, composed with `AND` / `OR` and parenthesised grouping; missing attributes fail closed as on AWS; unparseable SQL → `SqlParseException`), and `lambda` / `republish` / `dynamoDBv2` / `sns` / `sqs` actions dispatched in-process with the rule's `errorAction` on failure; **device shadows over MQTT** — the `$aws/things/<thing>/shadow[/name/<name>]/{get,update,delete}` reserved topics drive the shadow store and reply on `accepted` / `rejected` (with `delta` and `documents` on update); a **CA-certificate registry** (`RegisterCACertificate` and friends) with the just-in-time-registration `registered` event on `$aws/events/certificates/registered/{caCertificateId}`; a **provisioning-template registry** (`CreateProvisioningTemplate` and friends) — storage + CRUD only, so `provisioningRoleArn` is presence-checked but not resolved and the fleet-provisioning MQTT workflow (`$aws/provisioning-templates/...`) is not implemented; and `SearchIndex` answers `connectivity.connected` / `connectivity.timestamp` / `connectivity.disconnectReason` when `thingConnectivityIndexingMode` is `STATUS`, deriving `connected` from the live session registry; the broker publishes the presence and subscription lifecycle events (`$aws/events/presence/connected\|disconnected/{clientId}`, `$aws/events/subscriptions/subscribed\|unsubscribed/{clientId}`) |
| **IoT Wireless** | GetPositionEstimate | Single operation (`POST /position-estimate`, SigV4 scope `iotwireless`, endpoint prefix `api.iotwireless`). **The response is the raw GeoJSON blob** — the output shape's `payload` trait — so boto3 hands back a `StreamingBody` under `GeoJsonPayload`, not a JSON envelope. Resolves from `Ip` only: `WiFiAccessPoints` / `CellTowers` / `Gnss` are accepted and ignored, `Timestamp` and `AdvancedConfiguration` only reach `properties`, and the estimate is synthetic + deterministic — the canonical IP (an IPv4-mapped IPv6 address counts as its IPv4 form) hashes onto lon [-180, 180) / lat [-60, 60), 4 decimals — so the same address always answers the same estimate and consumer tests can assert on it. `properties` carry `horizontalAccuracy` 1000000 (the value of one live call) and `horizontalConfidenceLevel` 0.68 (the default the AWS IoT Core Device Location guide documents — `AdvancedConfiguration.WiFiCellular.ConfidencePercent` overrides it, divided by 100, within the 50 to 99 the model allows) plus `timestamp`, which is always there: AWS documents the request's `Timestamp` as the resolve time and the receive time as its substitute when the caller sends none, so send a `Timestamp` and the whole blob is byte-identical. The real payload's `country` and city/state/postal properties are omitted for that determinism. An input with no resolver hint and an `IpAddress` that `ipaddress.ip_address()` refuses answer `ValidationException` (those two wordings are recorded from the live service; every other message in this service is MiniStack's own, since none is documented); a `Timestamp` that is not a Unix timestamp, and a `ConfidencePercent` outside the model's range, are `ValidationException` too; a valid address that is **not globally routable or is multicast** (private, loopback, TEST-NET, multicast groups, ...) answers `ResourceNotFoundException` `Cannot find position for the IP address <ip>`, mirroring the real service's refusal when its geo database has no entry. A well-formed request whose only measurements are WLAN/cell/GNSS is `ResourceNotFoundException` too — the developer guide documents 404 for exactly that ("no location information was found or solved ... The MAC address or cellular tower information is not sufficient ... The GNSS payload is not sufficient"), and 400 for measurement data formatted incorrectly. Stateless — nothing persists |
| **EventBridge Scheduler** | CreateSchedule, GetSchedule, UpdateSchedule, DeleteSchedule, ListSchedules, CreateScheduleGroup, GetScheduleGroup, DeleteScheduleGroup, ListScheduleGroups, TagResource, UntagResource, ListTagsForResource | 12 actions; schedules **fire their targets** — a background sweep invokes each enabled schedule's target (Lambda, SQS, SNS, Step Functions) when due, honoring `State`, `StartDate`/`EndDate`, and `ActionAfterCompletion: DELETE`; schedule groups with cascading deletes; `rate()`, `cron()`, `at()` expressions; group/prefix/state filters on list; default group auto-created; CFN `AWS::Scheduler::Schedule` and `AWS::Scheduler::ScheduleGroup` supported |
diff --git a/ministack/services/iot.py b/ministack/services/iot.py
index ccacf48e..e8bbe468 100644
--- a/ministack/services/iot.py
+++ b/ministack/services/iot.py
@@ -5723,12 +5723,10 @@ _JOB_DOCUMENT_S3_HOST_RE = re.compile(r"(?:(?P<bucket>.+)\.)?s3(?:[.-][a-z0-9-]+
_JOB_DOCUMENT_MAX_BYTES = 32768
-def _job_document_from_source(source: str) -> tuple[str | None, tuple | None]:
- """The job document a documentSource names in this account's S3, or an error."""
+def _job_document_s3_location(source: str) -> tuple[str, str]:
+ """The bucket and key an s3:// or S3 object URL names, empty where it names none."""
from urllib.parse import unquote, urlsplit
- from ministack.services import s3 as _s3
-
bucket = key = ""
if isinstance(source, str) and source.startswith("s3://"):
bucket, _, key = source[len("s3://"):].partition("/")
@@ -5741,6 +5739,14 @@ def _job_document_from_source(source: str) -> tuple[str | None, tuple | None]:
elif match:
bucket, _, key = parts.path[1:].partition("/")
key = unquote(key)
+ return bucket, key
+
+
+def _job_document_from_source(source: str) -> tuple[str | None, tuple | None]:
+ """The job document a documentSource names in this account's S3, or an error."""
+ from ministack.services import s3 as _s3
+
+ bucket, key = _job_document_s3_location(source)
if not bucket:
return None, error_response_json(
"InvalidRequestException", f"Job document source {source} is invalid.", 400
@@ -5778,6 +5784,37 @@ def _job_document_from_source(source: str) -> tuple[str | None, tuple | None]:
return document, None
+# ${aws:iot:s3-presigned-url:<url>}; the -v2 form takes the URL encoded.
+_JOB_PRESIGN_RE = re.compile(r"\$\{aws:iot:s3-presigned-url(?P<v2>-v2)?:(?P<url>[^}]*)\}")
+
+
+def _job_presign_error(document: str | None, config: dict | None, job_id: str) -> tuple | None:
+ """AWS's refusal of presigned-URL placeholders without a role or an S3 object URL."""
+ placeholders = list(_JOB_PRESIGN_RE.finditer(document or ""))
+ if not placeholders:
+ return None
+ if not (isinstance(config, dict) and config.get("roleArn")):
+ message = (
+ "presignedUrlConfig role ARN is required to generate an S3 presigned url in job document."
+ if all(m.group("v2") for m in placeholders)
+ else f"Role must be provided for pre-sign S3 urls in job document of job {job_id}."
+ )
+ return error_response_json("InvalidRequestException", message, 400)
+ for m in placeholders:
+ url = m.group("url")
+ bucket, key = _job_document_s3_location(url)
+ if not bucket or (m.group("v2") and re.search(r'[\s"<>\\^`{|}]', url)):
+ return error_response_json(
+ "InvalidRequestException", f"Given S3 url {url} is invalid.", 400
+ )
+ if not key:
+ return error_response_json(
+ "InvalidRequestException",
+ f"Provided S3 url is not valid: {url}. Error: Key cannot be null", 400,
+ )
+ return None
+
+
async def _create_job(job_id: str, payload: dict) -> tuple:
if not job_id or not _JOB_ID_RE.match(job_id):
return error_response_json(
@@ -5879,6 +5916,9 @@ async def _create_job(job_id: str, payload: dict) -> tuple:
document, err = _job_document_from_source(payload["documentSource"])
if err:
return err
+ err = _job_presign_error(document, payload.get("presignedUrlConfig"), job_id)
+ if err:
+ return err
# Captured BEFORE the job record exists: jobs_first_pending_job_id
# materializes, and once the new job is in the store it would already
# count as the front of the queue.
@@ -6432,6 +6472,9 @@ def _create_job_template(template_id: str, payload: dict) -> tuple:
if payload.get(member) is not None:
record[member] = copy.deepcopy(payload[member])
_job_config_floats(record)
+ err = _job_presign_error(record.get("document"), record.get("presignedUrlConfig"), template_id)
+ if err:
+ return err
_job_templates[template_id] = record
_set_created_tags(record["jobTemplateArn"], payload.get("tags"))
return json_response({
@@ -7009,10 +7052,106 @@ def _jobs_device_json(value) -> str:
return json.dumps(value)
+# The host a device reached MiniStack on: its presigned URLs name it.
+jobs_device_host: contextvars.ContextVar[str] = contextvars.ContextVar(
+ "_iot_jobs_device_host", default=""
+)
+_JOB_PRESIGN_HOST_RE = re.compile(r"[A-Za-z0-9.\-\[\]:_]+")
+# The labels of an IoT endpoint host up to its region; S3 is not routed under them.
+_JOB_PRESIGN_IOT_LABELS_RE = re.compile(r"(?:[^.:]+\.)*?iot\.[a-z0-9-]+\.(?=[^:])")
+# A role session signs an hour of URLs, each valid for its full expiresInSec.
+_JOB_PRESIGN_SESSION_REUSE_SECONDS = 3600
+# (account, role ARN) -> the access key of that role's current session.
+_jobs_presign_sessions: dict[tuple[str, str], str] = {}
+
+
+def _jobs_presign_session(role_arn: str, expires: int) -> tuple[str, dict]:
+ """A session of the presign role, registered with STS, that outlives a URL signed now."""
+ from ministack.services import iam as _iam
+ from ministack.services import sts as _sts
+
+ cache_key = (get_account_id(), role_arn)
+ access_key = _jobs_presign_sessions.get(cache_key, "")
+ session = _sts._sessions.get(access_key)
+ if session is None or session["Expiration"] < time.time() + expires:
+ partition = role_arn.split(":")[1] if _JOB_ROLE_ARN_RE.match(role_arn) else "aws"
+ account = get_account_id() # the job's account, whatever account the role ARN names
+ name = uuid.uuid4().hex
+ access_key = _iam._gen_session_access_key()
+ session = {
+ "Arn": f"arn:{partition}:sts::{account}:assumed-role/{role_arn.rsplit('/', 1)[-1]}/{name}",
+ "UserId": f"AROA{uuid.uuid4().hex[:17].upper()}:{name}",
+ "SecretAccessKey": _iam._gen_secret(),
+ "SessionToken": _iam._gen_session_token(),
+ "Expiration": time.time() + expires + _JOB_PRESIGN_SESSION_REUSE_SECONDS,
+ "AccountId": account,
+ "PrincipalType": "AssumedRole",
+ }
+ _sts.register_session(access_key, session)
+ _jobs_presign_sessions[cache_key] = access_key
+ return access_key, session
+
+
+def _jobs_presigned_document(job: dict) -> str:
+ """The job document with each placeholder replaced by a presigned GET URL of MiniStack's S3."""
+ document = job.get("document") or "{}"
+ config = job.get("presignedUrlConfig")
+ if "${aws:iot:s3-presigned-url" not in document or not (
+ isinstance(config, dict) and config.get("roleArn")
+ ):
+ return document
+ from ministack.core.sigv4 import (
+ build_canonical_query_string,
+ build_canonical_request,
+ build_string_to_sign,
+ calculate_signature,
+ uri_encode,
+ )
+ from ministack.core.tls import use_ssl_enabled
+
+ expires = config.get("expiresInSec")
+ expires = int(expires) if isinstance(expires, (int, float)) and expires > 0 else 3600
+ access_key, session = _jobs_presign_session(config["roleArn"], expires)
+ host = jobs_device_host.get()
+ if not _JOB_PRESIGN_HOST_RE.fullmatch(host):
+ host = f"{_MINISTACK_HOST}:{_GATEWAY_PORT}"
+ elif labels := _JOB_PRESIGN_IOT_LABELS_RE.match(host):
+ host = host[labels.end():]
+ base = f"{'https' if use_ssl_enabled() else 'http'}://{host}"
+ amz_date = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
+ region = get_region()
+ query = {
+ "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
+ "X-Amz-Credential": f"{access_key}/{amz_date[:8]}/{region}/s3/aws4_request",
+ "X-Amz-Date": amz_date,
+ "X-Amz-Expires": str(expires),
+ "X-Amz-Security-Token": session["SessionToken"],
+ "X-Amz-SignedHeaders": "host",
+ }
+ canonical_query = build_canonical_query_string(query)
+
+ def presign(match: re.Match) -> str:
+ bucket, key = _job_document_s3_location(match.group("url"))
+ if not (bucket and key):
+ return match.group(0)
+ path = f"/{bucket}/{key}"
+ string_to_sign = build_string_to_sign(
+ amz_date, amz_date[:8], region, "s3",
+ build_canonical_request("GET", path, {"host": host}, query, "host"),
+ )
+ signature = calculate_signature(
+ session["SecretAccessKey"], amz_date[:8], region, "s3", string_to_sign
+ )
+ return (f"{base}{uri_encode(path, encode_slash=False)}?{canonical_query}"
+ f"&X-Amz-Signature={signature}")
+
+ return _JOB_PRESIGN_RE.sub(presign, document)
+
+
def jobs_job_document(job_id: str) -> str:
"""The document a device should run for a job, re-serialized as AWS sends it — ``{}`` if there is none."""
job = _jobs.get(job_id)
- document = (job or {}).get("document") or "{}"
+ document = _jobs_presigned_document(job) if job else "{}"
try:
return _jobs_device_json(json.loads(document))
except (ValueError, RecursionError):
@@ -10170,6 +10309,8 @@ async def handle_websocket(
chosen = proto
break
+ jobs_device_host.set(sub_headers.get("host", ""))
+
accept: dict = {"type": "websocket.accept"}
if chosen:
accept["subprotocol"] = chosen
@@ -10946,6 +11087,10 @@ async def _mtls_serve_conn(reader: asyncio.StreamReader, writer: asyncio.StreamW
account_token = _request_account_id.set(account_id)
region_token = _request_region.set(region)
+ # The name the device dialled (its SNI), or the address, on the gateway port.
+ local = str((writer.get_extra_info("sockname") or ("",))[0])
+ dialled = _mtls_server_names.get(ssl_object) if ssl_object is not None else None
+ jobs_device_host.set(f"{dialled or (f'[{local}]' if ':' in local else local)}:{_GATEWAY_PORT}")
certificate_id = hashlib.sha256(der).hexdigest() if der else None
session = _WSSession(_tcp_send, account_id, region, certificate_id)
session.certificate_id = certificate_id
diff --git a/ministack/services/iot_jobs_data.py b/ministack/services/iot_jobs_data.py
index dcd3ef96..b1e9ea0d 100644
--- a/ministack/services/iot_jobs_data.py
+++ b/ministack/services/iot_jobs_data.py
@@ -71,6 +71,7 @@ async def handle_request(
method: str, path: str, headers: dict, body: bytes, query_params: dict
) -> tuple:
"""Route a device-plane request, then publish the job events it raised."""
+ _iot_module.jobs_device_host.set(headers.get("host", ""))
async with _iot_module.publishing_events():
return await _route_request(method, path, body, query_params)
diff --git a/tests/test_cfn.py b/tests/test_cfn.py
index a5b16016..7492bc7b 100644
--- a/tests/test_cfn.py
+++ b/tests/test_cfn.py
@@ -1662,6 +1662,29 @@ def test_cfn_iot_job_template_document_source(cfn, iot_client, s3):
s3.delete_bucket(Bucket=bucket)
+def test_cfn_iot_job_template_presigned_url_placeholder(cfn, iot_client):
+ """A Document placeholder needs PresignedUrlConfig.RoleArn; without one the stack fails."""
+ uid = _uuid_mod.uuid4().hex[:8]
+ stack, failing, template_id = f"cfn-iot-jobtmpl-pre-{uid}", f"cfn-iot-jobtmpl-nor-{uid}", f"cfn-jt-pre-{uid}"
+ document = json.dumps({"fw": "${aws:iot:s3-presigned-url:https://s3.amazonaws.com/b/fw.bin}"})
+ try:
+ cfn.create_stack(StackName=stack, TemplateBody=_cfn_job_template(template_id, Document=document))
+ assert _wait_stack(cfn, stack)["StackStatus"] == "CREATE_COMPLETE"
+ assert iot_client.describe_job_template(jobTemplateId=template_id)["document"] == document
+
+ cfn.create_stack(StackName=failing, TemplateBody=_cfn_job_template(
+ f"{template_id}-2", Document=document, PresignedUrlConfig=None))
+ assert _wait_stack(cfn, failing)["StackStatus"] == "ROLLBACK_COMPLETE"
+ reasons = [e.get("ResourceStatusReason", "") for e in
+ cfn.describe_stack_events(StackName=failing)["StackEvents"]
+ if e["ResourceStatus"] == "CREATE_FAILED"]
+ assert any(f"Role must be provided for pre-sign S3 urls in job document of job {template_id}-2."
+ in reason for reason in reasons), reasons
+ finally:
+ _delete_cfn_test_stack(cfn, stack)
+ _delete_cfn_test_stack(cfn, failing)
+
+
def test_cfn_iot_ca_certificate_lifecycle(cfn, iot_client):
"""AWS::IoT::CACertificate provisions onto the real CA registry instead of
rolling the stack back: create registers the PEM (readable back through
diff --git a/tests/test_iot_data.py b/tests/test_iot_data.py
index fb26dd39..17bd3065 100644
--- a/tests/test_iot_data.py
+++ b/tests/test_iot_data.py
@@ -3766,6 +3766,30 @@ def test_mtls_publish_principal_is_the_certificate_id(broker, tmp_path):
broker.client("iam").delete_role(RoleName=role["RoleName"])
+def test_mtls_job_document_urls_name_the_dialled_host(broker, tmp_path):
+ """Presigned URLs a device gets over TLS name the host it dialled, on the gateway port."""
+ iot = broker.client("iot")
+ thing, job_id = _unique("mtls-jobs"), _unique("job")
+ iot.create_job(
+ jobId=job_id, targets=[iot.create_thing(thingName=thing)["thingArn"]],
+ document=json.dumps({"fw": "${aws:iot:s3-presigned-url-v2:s3://bucket/fw.bin}"}),
+ presignedUrlConfig={"roleArn": "arn:aws:iam::000000000000:role/presign"},
+ )
+ port = int(os.environ.get("GATEWAY_PORT", os.environ.get("EDGE_PORT", "4566")))
+ topic = f"$aws/things/{thing}/jobs/{job_id}/get"
+ for server_name, host in (("localhost", "localhost"), (None, "127.0.0.1")):
+ peer = _Peer(_mtls_connect(broker, None, None, tmp_path, server_name=server_name))
+ try:
+ _assert_connack(peer.connect(_unique("device")))
+ peer.subscribe(f"{topic}/accepted")
+ peer.send(_make_publish(topic, b"{}"))
+ _topic, payload = peer.next_publish()
+ url = urlparse(json.loads(payload)["execution"]["jobDocument"]["fw"])
+ assert (url.hostname, url.port, url.path) == (host, port, "/bucket/fw.bin")
+ finally:
+ peer.close()
+
+
def test_mtls_no_client_cert_uses_default_account(broker, tmp_path):
"""No client certificate connects fine, exactly like the WebSocket path,
and the session runs under the default account."""
diff --git a/tests/test_iot_jobs_data.py b/tests/test_iot_jobs_data.py
index e2a99417..78f5ff13 100644
--- a/tests/test_iot_jobs_data.py
+++ b/tests/test_iot_jobs_data.py
@@ -373,6 +373,237 @@ def test_iot_jobs_document_source_is_read_in_the_callers_account():
_delete_bucket(owner_s3, bucket)
+def _presign_role(iam, bucket):
+ """A role IoT may assume that reads the bucket's objects: (name, ARN)."""
+ name = _unique("presign")
+ trust = {"Effect": "Allow", "Principal": {"Service": "iot.amazonaws.com"},
+ "Action": "sts:AssumeRole"}
+ arn = iam.create_role(
+ RoleName=name,
+ AssumeRolePolicyDocument=json.dumps({"Version": "2012-10-17", "Statement": [trust]}),
+ )["Role"]["Arn"]
+ read = {"Effect": "Allow", "Action": "s3:GetObject", "Resource": f"arn:aws:s3:::{bucket}/*"}
+ iam.put_role_policy(
+ RoleName=name, PolicyName="read",
+ PolicyDocument=json.dumps({"Version": "2012-10-17", "Statement": [read]}),
+ )
+ return name, arn
+
+
+def _delete_presign_role(iam, name):
+ try:
+ iam.delete_role_policy(RoleName=name, PolicyName="read")
+ iam.delete_role(RoleName=name)
+ except ClientError:
+ pass
+
+
+def _presigned_get(url):
+ """A presigned URL's netloc, path and query, and the body a GET of it returns."""
+ from urllib.parse import parse_qs, urlsplit
+
+ parts = urlsplit(url)
+ with urllib.request.urlopen(url, timeout=5) as resp:
+ return parts.netloc, parts.path, parse_qs(parts.query), resp.read()
+
+
+def _presign(url, v2=True):
+ return "${aws:iot:s3-presigned-url" + ("-v2" if v2 else "") + ":" + url + "}"
+
+
+def test_iot_jobs_presigned_url_placeholders_reach_the_device(
+ iot_client, iot_jobs_data, s3, iam
+):
+ """The device gets each placeholder as a presigned GET URL of MiniStack's S3 on the host it called."""
+ from urllib.parse import urlsplit
+
+ thing, bucket, role = _unique("jobs-thing"), _unique("jobs-presign"), None
+ jobs = [_unique("job"), _unique("job")]
+ netloc = ENDPOINT.split("://", 1)[1]
+ try:
+ thing_arn = _create_thing(iot_client, thing)
+ s3.create_bucket(Bucket=bucket)
+ s3.put_object(Bucket=bucket, Key="fw.bin", Body=b"firmware")
+ s3.put_object(Bucket=bucket, Key="dir/fw 1.bin", Body=b"spaced")
+ role, role_arn = _presign_role(iam, bucket)
+ url = f"https://s3.us-east-1.amazonaws.com/{bucket}"
+ document = json.dumps({
+ "classic": _presign(f"{url}/fw.bin", v2=False),
+ "v2": _presign(f"{url}/fw.bin"),
+ "classicRaw": _presign(f"{url}/dir/fw 1.bin", v2=False),
+ "v2Encoded": _presign(f"{url}/dir/fw%201.bin"),
+ "hosted": _presign(f"https://{bucket}.s3.us-east-1.amazonaws.com/fw.bin"),
+ "s3": _presign(f"s3://{bucket}/fw.bin"),
+ "inText": f"get {_presign(f'{url}/fw.bin')} now",
+ })
+ iot_client.create_job(jobId=jobs[0], targets=[thing_arn], document=document,
+ presignedUrlConfig={"roleArn": role_arn, "expiresInSec": 300})
+ assert iot_client.get_job_document(jobId=jobs[0])["document"] == document
+
+ served = json.loads(iot_jobs_data.describe_job_execution(
+ jobId=jobs[0], thingName=thing, includeJobDocument=True
+ )["execution"]["jobDocument"])
+ bodies = {"classic": b"firmware", "v2": b"firmware", "classicRaw": b"spaced",
+ "v2Encoded": b"spaced", "hosted": b"firmware", "s3": b"firmware"}
+ for member, body in bodies.items():
+ key = "dir/fw%201.bin" if body == b"spaced" else "fw.bin"
+ got_netloc, path, query, got = _presigned_get(served[member])
+ assert (got_netloc, path, got) == (netloc, f"/{bucket}/{key}", body), member
+ assert sorted(query) == [
+ "X-Amz-Algorithm", "X-Amz-Credential", "X-Amz-Date", "X-Amz-Expires",
+ "X-Amz-Security-Token", "X-Amz-Signature", "X-Amz-SignedHeaders",
+ ]
+ assert query["X-Amz-Expires"] == ["300"]
+ assert query["X-Amz-SignedHeaders"] == ["host"]
+ assert query["X-Amz-Credential"][0].startswith("ASIA")
+ assert query["X-Amz-Credential"][0].endswith("/us-east-1/s3/aws4_request")
+ before, _, rest = served["inText"].partition("http://")
+ assert (before, rest.endswith(" now")) == ("get ", True)
+
+ execution = iot_jobs_data.start_next_pending_job_execution(thingName=thing)["execution"]
+ assert _presigned_get(json.loads(execution["jobDocument"])["v2"])[3] == b"firmware"
+ updated = iot_jobs_data.update_job_execution(
+ jobId=jobs[0], thingName=thing, status="IN_PROGRESS", includeJobDocument=True
+ )
+ assert _presigned_get(json.loads(updated["jobDocument"])["v2"])[3] == b"firmware"
+
+ # The device's host names the URL; an IoT endpoint host loses its IoT labels.
+ port = netloc.rsplit(":", 1)[1]
+ for host, expected in ((f"a1b2c3.jobs.iot.us-east-1.{netloc}", netloc),
+ (f"ministack:{port}", f"ministack:{port}")):
+ request = urllib.request.Request(
+ f"{ENDPOINT}/things/{quote(thing)}/jobs/{jobs[0]}?includeJobDocument=true",
+ headers={"Host": host, "Authorization": _JOBS_DATA_AUTH},
+ )
+ with urllib.request.urlopen(request, timeout=5) as resp:
+ document = json.loads(json.loads(resp.read())["execution"]["jobDocument"])
+ signed = urlsplit(document["v2"])
+ fetch = urllib.request.Request(f"{ENDPOINT}{signed.path}?{signed.query}",
+ headers={"Host": signed.netloc})
+ with urllib.request.urlopen(fetch, timeout=5) as resp:
+ assert (signed.netloc, resp.read()) == (expected, b"firmware"), host
+
+ # Without expiresInSec a URL lasts an hour.
+ iot_client.cancel_job(jobId=jobs[0], force=True)
+ iot_client.create_job(jobId=jobs[1], targets=[thing_arn],
+ document=json.dumps({"v2": _presign(f"{url}/fw.bin")}),
+ presignedUrlConfig={"roleArn": role_arn})
+ execution = iot_jobs_data.describe_job_execution(
+ jobId=jobs[1], thingName=thing, includeJobDocument=True)["execution"]
+ assert _presigned_get(json.loads(execution["jobDocument"])["v2"])[2][
+ "X-Amz-Expires"] == ["3600"]
+ finally:
+ _cleanup(iot_client, jobs=jobs, things=[thing])
+ _delete_bucket(s3, bucket)
+ if role:
+ _delete_presign_role(iam, role)
+
+
+def test_iot_jobs_presigned_url_signs_in_the_jobs_account():
+ """A role ARN of another account signs in the job's account: the URL reaches none of its objects."""
+ owner_s3, owner_iam = _account_client("s3", "111111111111"), _account_client("iam", "111111111111")
+ stranger = _account_client("iot", "222222222222")
+ stranger_jobs = _account_client("iot-jobs-data", "222222222222")
+ thing, bucket, job_id, role = _unique("jobs-thing"), _unique("jobs-presign"), _unique("job"), None
+ try:
+ owner_s3.create_bucket(Bucket=bucket)
+ owner_s3.put_object(Bucket=bucket, Key="fw.bin", Body=b"firmware")
+ role, role_arn = _presign_role(owner_iam, bucket)
+ stranger.create_job(jobId=job_id, targets=[_create_thing(stranger, thing)],
+ document=json.dumps({"fw": _presign(f"s3://{bucket}/fw.bin")}),
+ presignedUrlConfig={"roleArn": role_arn})
+ url = json.loads(stranger_jobs.describe_job_execution(
+ jobId=job_id, thingName=thing, includeJobDocument=True)["execution"]["jobDocument"])["fw"]
+ with pytest.raises(urllib.error.HTTPError) as ei:
+ _presigned_get(url)
+ assert ei.value.code == 403
+ finally:
+ _cleanup(stranger, jobs=[job_id], things=[thing])
+ _delete_bucket(owner_s3, bucket)
+ if role:
+ _delete_presign_role(owner_iam, role)
+
+
[email protected]("host, expected", [
+ ("ministack:4566", "ministack:4566"),
+ ("host.docker.internal:4566", "host.docker.internal:4566"),
+ ("a1b2c3-ats.iot.us-east-1.localhost:4566", "localhost:4566"),
+ ("a1b2c3.jobs.iot.us-east-1.localhost:4566", "localhost:4566"),
+ ("iot.eu-west-1.ministack:4566", "ministack:4566"),
+ ("", None),
+ ('bad"host', None),
+])
+def test_iot_jobs_presigned_url_host(host, expected):
+ """The URL names the device's host, an IoT endpoint host without its IoT labels, else the configured one."""
+ from urllib.parse import urlsplit
+
+ from ministack.services import iot
+
+ job = {"document": json.dumps({"fw": _presign("s3://bucket/fw.bin")}),
+ "presignedUrlConfig": {"roleArn": "arn:aws:iam::000000000000:role/presign"}}
+ token = iot.jobs_device_host.set(host)
+ try:
+ url = json.loads(iot._jobs_presigned_document(job))["fw"]
+ finally:
+ iot.jobs_device_host.reset(token)
+ assert urlsplit(url).netloc == (expected or f"{iot._MINISTACK_HOST}:{iot._GATEWAY_PORT}")
+
+
+def test_iot_jobs_presigned_url_placeholders_refusals(iot_client, s3):
+ """A placeholder without presignedUrlConfig.roleArn, or naming no S3 object, refuses the job."""
+ thing, bucket = _unique("jobs-thing"), _unique("jobs-presign")
+ url = "https://s3.us-east-1.amazonaws.com/bucket/fw.bin"
+ classic, v2 = _presign(url, v2=False), _presign(url)
+ role_message = "Role must be provided for pre-sign S3 urls in job document of job {}."
+ v2_message = (
+ "presignedUrlConfig role ARN is required to generate an S3 presigned url in job document."
+ )
+ role = {"roleArn": "arn:aws:iam::000000000000:role/presign"}
+ spaced, keyless = (
+ "https://s3.us-east-1.amazonaws.com/bucket/fw 1.bin",
+ "https://s3.us-east-1.amazonaws.com/bucket",
+ )
+ try:
+ thing_arn = _create_thing(iot_client, thing)
+ s3.create_bucket(Bucket=bucket)
+ s3.put_object(Bucket=bucket, Key="doc.json", Body=json.dumps({"a": classic}))
+ cases = [
+ ({"document": json.dumps({"a": classic})}, role_message),
+ ({"document": json.dumps({"a": v2})}, v2_message),
+ ({"document": json.dumps({"a": v2, "b": classic})}, role_message),
+ ({"document": json.dumps({"a": classic}),
+ "presignedUrlConfig": {"expiresInSec": 300}}, role_message),
+ ({"document": json.dumps({"a": v2}),
+ "presignedUrlConfig": {"expiresInSec": 300}}, v2_message),
+ ({"documentSource": f"s3://{bucket}/doc.json"}, role_message),
+ ({"document": json.dumps({"a": _presign("https://example.com/fw.bin", v2=False)}),
+ "presignedUrlConfig": role}, "Given S3 url https://example.com/fw.bin is invalid."),
+ ({"document": json.dumps({"a": _presign(spaced)}), "presignedUrlConfig": role},
+ f"Given S3 url {spaced} is invalid."),
+ ({"document": json.dumps({"a": _presign(keyless)}), "presignedUrlConfig": role},
+ f"Provided S3 url is not valid: {keyless}. Error: Key cannot be null"),
+ ]
+ for kwargs, message in cases:
+ job_id = _unique("job")
+ with pytest.raises(ClientError) as ei:
+ iot_client.create_job(jobId=job_id, targets=[thing_arn], **kwargs)
+ assert _error(ei) == (400, "InvalidRequestException", message.format(job_id)), kwargs
+ with pytest.raises(ClientError) as ei:
+ iot_client.describe_job(jobId=job_id)
+ assert ei.value.response["Error"]["Code"] == "ResourceNotFoundException"
+
+ # A raw space is fine in the classic form, and no placeholder needs no role.
+ job_ids = [_unique("job"), _unique("job")]
+ iot_client.create_job(jobId=job_ids[0], targets=[thing_arn], presignedUrlConfig=role,
+ document=json.dumps({"a": _presign(spaced, v2=False)}))
+ iot_client.create_job(jobId=job_ids[1], targets=[thing_arn],
+ document=json.dumps({"a": url}))
+ _cleanup(iot_client, jobs=job_ids)
+ finally:
+ _cleanup(iot_client, things=[thing])
+ _delete_bucket(s3, bucket)
+
+
def test_iot_jobs_create_duplicate_and_unknown_target_rejected(iot_client):
thing = _unique("jobs-thing")
job_id = _unique("job")
@@ -610,6 +841,35 @@ def test_iot_job_template_document_source_reads_the_s3_object(iot_client, s3):
_delete_bucket(s3, bucket)
+def test_iot_job_template_presigned_url_placeholders(iot_client, iot_jobs_data):
+ """CreateJobTemplate needs a role for a placeholder; a job made from the template hands out URLs."""
+ thing, template_id, refused, job_id = (
+ _unique("jobs-thing"), _unique("tmpl"), _unique("tmpl"), _unique("job")
+ )
+ document = json.dumps({"fw": _presign("s3://bucket/fw.bin", v2=False)})
+ try:
+ thing_arn = _create_thing(iot_client, thing)
+ with pytest.raises(ClientError) as ei:
+ iot_client.create_job_template(jobTemplateId=refused, description="d", document=document)
+ assert _error(ei) == (
+ 400, "InvalidRequestException",
+ f"Role must be provided for pre-sign S3 urls in job document of job {refused}.",
+ )
+ template_arn = iot_client.create_job_template(
+ jobTemplateId=template_id, description="d", document=document,
+ presignedUrlConfig={"roleArn": "arn:aws:iam::000000000000:role/presign",
+ "expiresInSec": 120},
+ )["jobTemplateArn"]
+ iot_client.create_job(jobId=job_id, targets=[thing_arn], jobTemplateArn=template_arn)
+ execution = iot_jobs_data.describe_job_execution(
+ jobId=job_id, thingName=thing, includeJobDocument=True)["execution"]
+ url = json.loads(execution["jobDocument"])["fw"]
+ assert url.startswith(f"{ENDPOINT}/bucket/fw.bin?") and "&X-Amz-Expires=120&" in url
+ finally:
+ _delete_templates(iot_client, template_id, refused)
+ _cleanup(iot_client, jobs=[job_id], things=[thing])
+
+
def test_iot_job_template_rejects_what_aws_rejects(iot_client):
template_id = _unique("tmpl")
try:
@@ -3076,6 +3336,42 @@ def test_jobs_mqtt_update_include_flags_and_terminal_notify(
) < topics.index(f"{base}/notify-next")
+def test_jobs_mqtt_presigned_urls(iot_client, iot_data_client, s3, iam):
+ """Over MQTT the document carries presigned URLs too, on the broker host without its IoT labels."""
+ thing, bucket, job_id = _unique("jobs-mqtt-url"), _unique("jobs-presign"), _unique("job")
+ base, role = f"$aws/things/{thing}/jobs", None
+ try:
+ thing_arn = _create_thing(iot_client, thing)
+ s3.create_bucket(Bucket=bucket)
+ s3.put_object(Bucket=bucket, Key="fw.bin", Body=b"firmware")
+ role, role_arn = _presign_role(iam, bucket)
+ received = _collect_shadow_frames(
+ f"{base}/notify-next",
+ lambda: iot_client.create_job(
+ jobId=job_id, targets=[thing_arn], presignedUrlConfig={"roleArn": role_arn},
+ document=json.dumps({"fw": _presign(f"s3://{bucket}/fw.bin")}),
+ ),
+ want=1,
+ )
+ url = _frames_by_topic(received)[f"{base}/notify-next"]["execution"]["jobDocument"]["fw"]
+ assert _presigned_get(url)[3] == b"firmware"
+
+ received = _collect_shadow_frames(
+ f"{base}/{job_id}/get/accepted",
+ lambda: _mqtt_publish(topic=f"{base}/{job_id}/get", payload=b"{}"),
+ want=1,
+ )
+ url = _frames_by_topic(received)[f"{base}/{job_id}/get/accepted"]["execution"][
+ "jobDocument"]["fw"]
+ netloc, _path, _query, body = _presigned_get(url)
+ assert (netloc, body) == (ENDPOINT.split("://", 1)[1], b"firmware")
+ finally:
+ _cleanup(iot_client, jobs=[job_id], things=[thing])
+ _delete_bucket(s3, bucket)
+ if role:
+ _delete_presign_role(iam, role)
+
+
def test_jobs_mqtt_leaving_continuous_job_group_notifies(iot_client, iot_data_client):
"""A thing leaving a CONTINUOUS job's target group loses its QUEUED
execution (REMOVED): its pending set and front changed, so notify carries