Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

feat(iot): fleet indexing — SearchIndex over the live registry, behind the index switch (#1382)

iot-rocketd6fc4622b2d5
diff --git a/README.md b/README.md
index c14c69b1..01eea506 100644
--- a/README.md
+++ b/README.md
@@ -583,7 +583,7 @@ MiniStack expands AWS SAM transforms (`Transform: AWS::Serverless-2016-10-31`) i
| **CodeBuild** | CreateProject, BatchGetProjects, ListProjects, UpdateProject, DeleteProject, StartBuild, BatchGetBuilds, StopBuild, ListBuilds, ListBuildsForProject, BatchDeleteBuilds | 11 actions; project and build metadata stored in-memory; builds complete immediately with SUCCEEDED status by default. Set `MINISTACK_CODEBUILD_EXECUTE=1` and mount the Docker socket to **really run them**: the project's inline buildspec goes to the official AWS CodeBuild local agent, which runs the phases in `environment.image`, and `BatchGetBuilds` reports live `phases` with a real `SUCCEEDED`/`FAILED` outcome. Execution needs the workspace shared with the host at the **same path** (`-v /tmp/ministack-codebuild:/tmp/ministack-codebuild`), because the agent bind-mounts it into the build container; a build gets `AWS_ENDPOINT_URL` pointing back at MiniStack, honours `timeoutInMinutes`, and is stopped by `StopBuild`. An executed build streams the agent output into its `logs.groupName` / `logs.streamName`, so `aws logs tail` reads the build log back |
| **AppConfig** | CreateApplication, GetApplication, ListApplications, UpdateApplication, DeleteApplication, CreateEnvironment, GetEnvironment, ListEnvironments, UpdateEnvironment, DeleteEnvironment, CreateConfigurationProfile, GetConfigurationProfile, ListConfigurationProfiles, UpdateConfigurationProfile, DeleteConfigurationProfile, CreateHostedConfigurationVersion, GetHostedConfigurationVersion, ListHostedConfigurationVersions, DeleteHostedConfigurationVersion, CreateDeploymentStrategy, GetDeploymentStrategy, ListDeploymentStrategies, UpdateDeploymentStrategy, DeleteDeploymentStrategy, StartDeployment, GetDeployment, ListDeployments, StopDeployment, TagResource, UntagResource, ListTagsForResource, StartConfigurationSession, GetLatestConfiguration | 33 operations; control plane + data plane; hosted configuration versions; deployments complete immediately; session-based configuration retrieval with token rotation |
| **Transfer Family** | CreateServer, DescribeServer, DeleteServer, ListServers, StartServer, StopServer, CreateUser, DescribeUser, DeleteUser, ListUsers, ImportSshPublicKey, DeleteSshPublicKey | 12 operations; **real SFTP listener** on `:2222` (override with `SFTP_PORT`) backed by S3 — `ls`, `put`, `get`, `mkdir`, `rename` work end-to-end against the local S3 emulator; `SFTP_PORT_PER_SERVER=1` allocates one port per `CreateServer` from `SFTP_BASE_PORT` (default 2300); SSH key auth scans every user across every server and account; `LOGICAL` and `PATH` home-directory mappings; host key persists across restarts when `PERSIST_STATE=1` |
-| **IoT Core** | CreateThing, DescribeThing, ListThings, UpdateThing, DeleteThing, CreateThingType, CreateThingGroup, AddThingToThingGroup, ListThingsInThingGroup, CreateKeysAndCertificate, RegisterCertificate, UpdateCertificate, DeleteCertificate, AttachThingPrincipal, DetachThingPrincipal, ListThingPrincipals, ListPrincipalThings, CreatePolicy, CreatePolicyVersion, AttachPolicy, DetachPolicy, ListAttachedPolicies, ListTargetsForPolicy, DescribeEndpoint, Publish (HTTP), MQTT-over-WebSocket pub/sub | 24 operations + **real MQTT 3.1.1 over WebSocket** on the gateway port — clients use the address returned by `DescribeEndpoint`; local CA signs `CreateKeysAndCertificate` certificates (CA persists across restarts when `PERSIST_STATE=1`); multi-tenancy via transparent topic prefixing (same pattern as Transfer Family's shared SFTP listener); no plain TCP 1883 (real AWS IoT requires TLS or SigV4 on every connection); local CA cert available at `GET /_ministack/iot/ca.pem` for SDK trust configuration; IoT policies are stored but **not enforced** on the data plane |
+| **IoT Core** | CreateThing, DescribeThing, ListThings, UpdateThing, DeleteThing, CreateThingType, CreateThingGroup, AddThingToThingGroup, ListThingsInThingGroup, CreateKeysAndCertificate, RegisterCertificate, UpdateCertificate, DeleteCertificate, AttachThingPrincipal, DetachThingPrincipal, ListThingPrincipals, ListPrincipalThings, CreatePolicy, CreatePolicyVersion, AttachPolicy, DetachPolicy, ListAttachedPolicies, ListTargetsForPolicy, UpdateIndexingConfiguration, GetIndexingConfiguration, DescribeIndex, ListIndices, SearchIndex, DescribeEndpoint, Publish (HTTP), MQTT-over-WebSocket pub/sub | 29 operations + **real MQTT 3.1.1 over WebSocket** on the gateway port — clients use the address returned by `DescribeEndpoint`; local CA signs `CreateKeysAndCertificate` certificates (CA persists across restarts when `PERSIST_STATE=1`); multi-tenancy via transparent topic prefixing (same pattern as Transfer Family's shared SFTP listener); no plain TCP 1883 (real AWS IoT requires TLS or SigV4 on every connection); local CA cert available at `GET /_ministack/iot/ca.pem` for SDK trust configuration; IoT policies are stored but **not enforced** on the data plane; **fleet indexing** works the way it does on AWS — `UpdateIndexingConfiguration` turns the `AWS_Things` index on (Terraform `aws_iot_indexing_configuration`), and searching while `thingIndexingMode` is `OFF` answers `ResourceNotFoundException`; `SearchIndex` then queries the live registry + classic shadows with `AND`-separated `thingName` / `thingTypeName` / `thingGroupNames` / `attributes.*` / `shadow.reported.*` / `shadow.desired.*` terms (`*` / `?` wildcards, `maxResults` / `nextToken` paging) — there is no projection to build, so hits are always current |
| **EventBridge Scheduler** | CreateSchedule, GetSchedule, UpdateSchedule, DeleteSchedule, ListSchedules, CreateScheduleGroup, GetScheduleGroup, DeleteScheduleGroup, ListScheduleGroups, TagResource, UntagResource, ListTagsForResource | 12 actions; schedules **fire their targets** — a background sweep invokes each enabled schedule's target (Lambda, SQS, SNS, Step Functions) when due, honoring `State`, `StartDate`/`EndDate`, and `ActionAfterCompletion: DELETE`; schedule groups with cascading deletes; `rate()`, `cron()`, `at()` expressions; group/prefix/state filters on list; default group auto-created; CFN `AWS::Scheduler::Schedule` and `AWS::Scheduler::ScheduleGroup` supported |
| **EKS** | CreateCluster, DescribeCluster, ListClusters, DeleteCluster, CreateNodegroup, DescribeNodegroup, ListNodegroups, DeleteNodegroup, CreateAddon, DescribeAddon, ListAddons, UpdateAddon, DeleteAddon, AssociateEncryptionConfig, CreateAccessEntry, DescribeAccessEntry, ListAccessEntries, UpdateAccessEntry, DeleteAccessEntry, AssociateAccessPolicy, DisassociateAccessPolicy, ListAssociatedAccessPolicies, AssociateIdentityProviderConfig, DescribeIdentityProviderConfig, DisassociateIdentityProviderConfig, ListIdentityProviderConfigs, TagResource, UntagResource, ListTagsForResource | `CreateCluster` spawns a real **k3s** container (75 MB) with a full Kubernetes API server; `kubectl`, Helm, and any K8s tooling work out of the box; cascading delete removes nodegroups and k3s container; addon CRUD (e.g. `vpc-cni`, `coredns`, `kube-proxy`, `aws-ebs-csi-driver`) at `/clusters/{name}/addons` flips status to `ACTIVE` on create / update; `AssociateEncryptionConfig` at `/clusters/{name}/encryption-config/associate` records KMS encryption config (rejects re-association, AWS-shape `update` envelope); **Access Entries** (modern post-1.29 IAM bindings — replaces aws-auth ConfigMap) at `/clusters/{name}/access-entries[/{principalArn}[/access-policies[/{policyArn}]]]` for Crossplane / Terraform `aws_eks_access_entry` + `aws_eks_access_policy_association`; OIDC discovery + JWKS served at the cluster's `identity.oidc.issuer` URL so Terraform's `aws_iam_openid_connect_provider` (IRSA) works against ministack; **OIDC Identity Provider Config** (`AssociateIdentityProviderConfig` / `DescribeIdentityProviderConfig` / `DisassociateIdentityProviderConfig` / `ListIdentityProviderConfigs`) at `/clusters/{name}/identity-provider-configs[/{verb}]` — one OIDC IdP per cluster, issuer + clientId + claims forwarded to the k3s API server via `--kube-apiserver-arg=oidc-*` flags, tags reachable via `ListTagsForResource`; CFN `AWS::EKS::Cluster` and `AWS::EKS::Nodegroup` supported |
| **OpenSearch Service** | CreateDomain, DescribeDomain, DescribeDomains, DeleteDomain, ListDomainNames, UpdateDomainConfig, DescribeDomainConfig, DescribeDomainChangeProgress, ListVersions, GetCompatibleVersions, AddTags, ListTags, RemoveTags | Management plane on `/2021-01-01/*` (`boto3.client("opensearch")`, SigV4 scope `es`). Default data plane is a stub endpoint (`<domain>.ministack.local:9200`) — set `OPENSEARCH_DATAPLANE=1` to spawn one real `opensearchproject/opensearch` container per `CreateDomain` (same pattern as ElastiCache/RDS); `DescribeDomain.Endpoint` then points at that container and `_cluster/health`/`/_search` work end-to-end. Add `OPENSEARCH_DASHBOARDS=1` (with dataplane on) to also spawn a per-domain `opensearch-dashboards` sidecar; `DescribeDomain.DashboardEndpoint` is populated. ARNs follow `arn:aws:es:<region>:<account>:domain/<name>`; Terraform `aws_opensearch_domain` resource compatible. |
diff --git a/ministack/services/iot.py b/ministack/services/iot.py
index 27a46859..f10e5264 100644
--- a/ministack/services/iot.py
+++ b/ministack/services/iot.py
@@ -11,6 +11,9 @@ Implements the JSON/REST APIs under ``iot.{region}.amazonaws.com``:
``AttachThingPrincipal`` / ``DetachThingPrincipal``
- Policies: ``CreatePolicy``, ``CreatePolicyVersion``, ``AttachPolicy``,
``DetachPolicy``, etc.
+ - Fleet indexing: ``UpdateIndexingConfiguration`` /
+ ``GetIndexingConfiguration`` / ``DescribeIndex`` / ``ListIndices``, and
+ ``SearchIndex`` over the live registry + shadows
- ``DescribeEndpoint`` returning a per-account hostname
This is the control plane — pure HTTP/JSON, no MQTT broker
@@ -78,6 +81,8 @@ _certificates: AccountRegionScopedDict = AccountRegionScopedDict()
_policies: AccountRegionScopedDict = AccountRegionScopedDict()
_topic_rules: AccountRegionScopedDict = AccountRegionScopedDict()
_shadows: AccountRegionScopedDict = AccountRegionScopedDict()
+# Fleet-indexing configuration, one entry per account+region.
+_indexing_config: AccountRegionScopedDict = AccountRegionScopedDict()
# Local CA state — lazily generated on first use, persisted across restarts.
import threading
@@ -167,6 +172,7 @@ def get_state() -> dict:
"policies": copy.deepcopy(_policies),
"topic_rules": copy.deepcopy(_topic_rules),
"shadows": copy.deepcopy(_shadows),
+ "indexing_config": copy.deepcopy(_indexing_config),
"ca": {"ca_cert_pem": _ca_cert_pem, "ca_key_pem": _ca_key_pem}
if _ca_cert_pem and _ca_key_pem
else {},
@@ -185,6 +191,7 @@ def restore_state(data: dict | None) -> None:
_policies.update(data.get("policies", {}))
_topic_rules.update(data.get("topic_rules", {}))
_shadows.update(data.get("shadows", {}))
+ _indexing_config.update(data.get("indexing_config", {}))
ca_data = data.get("ca")
if ca_data:
cert = ca_data.get("ca_cert_pem")
@@ -206,6 +213,7 @@ def reset() -> None:
_policies.clear()
_topic_rules.clear()
_shadows.clear()
+ _indexing_config.clear()
with _CA_LOCK:
_ca_cert_pem = None
_ca_key_pem = None
@@ -387,6 +395,18 @@ async def handle_request(
if path.startswith("/policies/"):
return _handle_policy(method, path, body, qp)
+ # Fleet indexing — the search path must precede ``/indices/{indexName}``
+ if path == "/indices/search" and method == "POST":
+ return _search_index(_parse_body(body))
+ if path == "/indexing/config" and method == "POST":
+ return _update_indexing_configuration(_parse_body(body))
+ if path == "/indexing/config" and method == "GET":
+ return _get_indexing_configuration()
+ if path == "/indices" and method == "GET":
+ return _list_indices()
+ if path.startswith("/indices/") and method == "GET":
+ return _describe_index(path)
+
# Topic rules
if path == "/rules" and method == "GET":
return _list_topic_rules(qp)
@@ -1389,6 +1409,430 @@ def _list_attached_policies(path: str, qp: dict) -> tuple:
return json_response({"policies": out})
+# ---------------------------------------------------------------------------
+# Fleet indexing (indexing configuration + SearchIndex)
+# ---------------------------------------------------------------------------
+
+# One token: either a ``field:value`` term (quoted values may contain spaces)
+# or a bare word, which the grammar only accepts as the ``AND`` separator.
+_SEARCH_TOKEN_RE = re.compile(
+ r'(?P<field>[\w.]+)\s*:\s*(?P<value>"[^"]*"|\S+)|(?P<bare>"[^"]*"|\S+)'
+)
+_SEARCH_TOP_FIELDS = ("thingName", "thingTypeName", "thingGroupNames")
+# Only the classic shadow is indexed, and only its two state halves are
+# addressable: shadow.metadata / shadow.version / shadow.name.<name> (named
+# shadows) are AWS fields this emulator does not project.
+_SEARCH_SHADOW_HALVES = ("desired", "reported")
+
+# AWS names the single thing index ``AWS_Things``; it exists only while thing
+# indexing is enabled, which is why every fleet stack starts with an
+# ``UpdateIndexingConfiguration`` call (Terraform: aws_iot_indexing_configuration).
+_THING_INDEX_NAME = "AWS_Things"
+_INDEXING_CONFIG_KEY = "indexing"
+_THING_INDEXING_MODES = ("OFF", "REGISTRY", "REGISTRY_AND_SHADOW")
+_THING_CONNECTIVITY_INDEXING_MODES = ("OFF", "STATUS")
+_THING_GROUP_INDEXING_MODES = ("OFF", "ON")
+# AWS's defaults for an account that has never called
+# UpdateIndexingConfiguration: nothing is indexed.
+_DEFAULT_INDEXING_CONFIG = {
+ "thingIndexingConfiguration": {
+ "thingIndexingMode": "OFF",
+ "thingConnectivityIndexingMode": "OFF",
+ },
+ "thingGroupIndexingConfiguration": {"thingGroupIndexingMode": "OFF"},
+}
+# SearchIndex's documented ceiling ("this maximum number cannot exceed 100").
+_SEARCH_MAX_RESULTS_LIMIT = 100
+
+
+def _indexing_configuration() -> dict:
+ """The account+region's indexing configuration, defaulting to all-OFF."""
+ stored = _indexing_config.get(_INDEXING_CONFIG_KEY)
+ return copy.deepcopy(stored) if stored else copy.deepcopy(_DEFAULT_INDEXING_CONFIG)
+
+
+def _thing_indexing_mode() -> str:
+ return (
+ _indexing_configuration()
+ .get("thingIndexingConfiguration", {})
+ .get("thingIndexingMode", "OFF")
+ )
+
+
+def _update_indexing_configuration(payload: dict) -> tuple:
+ """``POST /indexing/config`` — enable or disable fleet indexing.
+
+ Each sub-configuration that the request carries replaces the stored one
+ wholesale (AWS's own semantics: the mode is required, so a partial update
+ of a sub-configuration is not expressible); an omitted sub-configuration is
+ left alone.
+
+ Only the modes this emulator can honor are interpreted —
+ ``thingIndexingMode`` gates ``SearchIndex`` and whether shadow fields are
+ queryable. ``deviceDefenderIndexingMode``, ``namedShadowIndexingMode``,
+ ``customFields`` and ``filter`` are stored and echoed back by
+ ``GetIndexingConfiguration`` so IaC round-trips cleanly, but nothing here
+ projects those fields; querying one is an out-of-grammar
+ ``InvalidQueryException`` rather than a silent miss.
+ """
+ config = _indexing_configuration()
+
+ thing_cfg = payload.get("thingIndexingConfiguration")
+ if thing_cfg is not None:
+ if not isinstance(thing_cfg, dict):
+ return error_response_json(
+ "InvalidRequestException",
+ "thingIndexingConfiguration must be an object",
+ 400,
+ )
+ mode = thing_cfg.get("thingIndexingMode")
+ if mode not in _THING_INDEXING_MODES:
+ return error_response_json(
+ "InvalidRequestException",
+ f"thingIndexingMode must be one of {list(_THING_INDEXING_MODES)}",
+ 400,
+ )
+ connectivity_mode = thing_cfg.get("thingConnectivityIndexingMode") or "OFF"
+ if connectivity_mode not in _THING_CONNECTIVITY_INDEXING_MODES:
+ return error_response_json(
+ "InvalidRequestException",
+ "thingConnectivityIndexingMode must be one of "
+ f"{list(_THING_CONNECTIVITY_INDEXING_MODES)}",
+ 400,
+ )
+ if connectivity_mode == "STATUS" and mode == "OFF":
+ return error_response_json(
+ "InvalidRequestException",
+ "thingIndexingMode must not be OFF to enable thing connectivity "
+ "indexing",
+ 400,
+ )
+ config["thingIndexingConfiguration"] = {
+ **thing_cfg,
+ "thingIndexingMode": mode,
+ "thingConnectivityIndexingMode": connectivity_mode,
+ }
+
+ group_cfg = payload.get("thingGroupIndexingConfiguration")
+ if group_cfg is not None:
+ if not isinstance(group_cfg, dict):
+ return error_response_json(
+ "InvalidRequestException",
+ "thingGroupIndexingConfiguration must be an object",
+ 400,
+ )
+ group_mode = group_cfg.get("thingGroupIndexingMode")
+ if group_mode not in _THING_GROUP_INDEXING_MODES:
+ return error_response_json(
+ "InvalidRequestException",
+ f"thingGroupIndexingMode must be one of {list(_THING_GROUP_INDEXING_MODES)}",
+ 400,
+ )
+ config["thingGroupIndexingConfiguration"] = {
+ **group_cfg,
+ "thingGroupIndexingMode": group_mode,
+ }
+
+ _indexing_config[_INDEXING_CONFIG_KEY] = config
+ logger.info(
+ "IoT fleet indexing configured: thingIndexingMode=%s",
+ config["thingIndexingConfiguration"]["thingIndexingMode"],
+ )
+ return json_response({})
+
+
+def _get_indexing_configuration() -> tuple:
+ """``GET /indexing/config`` — the stored configuration, all-OFF by default."""
+ return json_response(_indexing_configuration())
+
+
+def _index_schema(config: dict) -> str:
+ """``DescribeIndex``'s schema string for the configuration in force."""
+ thing_cfg = config.get("thingIndexingConfiguration", {})
+ return (
+ "REGISTRY_AND_SHADOW"
+ if thing_cfg.get("thingIndexingMode") == "REGISTRY_AND_SHADOW"
+ else "REGISTRY"
+ )
+
+
+def _describe_index(path: str) -> tuple:
+ """``GET /indices/{indexName}``.
+
+ The index is a consequence of the configuration, not a resource of its
+ own: while thing indexing is OFF there is nothing to describe, which is
+ the same ``ResourceNotFoundException`` AWS answers with. It is never
+ ``BUILDING`` here — the registry *is* the index, so it is ready the moment
+ it is enabled.
+ """
+ name = path[len("/indices/"):]
+ config = _indexing_configuration()
+ if (
+ name != _THING_INDEX_NAME
+ or config["thingIndexingConfiguration"]["thingIndexingMode"] == "OFF"
+ ):
+ return error_response_json(
+ "ResourceNotFoundException", f"Index {name!r} does not exist", 404
+ )
+ return json_response({
+ "indexName": _THING_INDEX_NAME,
+ "indexStatus": "ACTIVE",
+ "schema": _index_schema(config),
+ })
+
+
+def _list_indices() -> tuple:
+ """``GET /indices`` — ``AWS_Things`` once thing indexing is on, else empty.
+
+ ``maxResults`` / ``nextToken`` are not honored because a page can never
+ overflow: there is exactly one index, or none.
+ """
+ enabled = _thing_indexing_mode() != "OFF"
+ return json_response({"indexNames": [_THING_INDEX_NAME] if enabled else []})
+
+
+def _parse_search_query(query: str) -> list[tuple[str, str]] | None:
+ """Parse ``field:value`` terms separated by ``AND`` (or by a bare space).
+
+ Returns the terms, or ``None`` for anything outside that grammar — a bare
+ word that is not ``AND``, a leading or doubled ``AND``, and a *dangling*
+ ``AND`` with no term after it. AWS rejects a dangling ``AND``, so ignoring
+ it here would answer a malformed query with results.
+ """
+ terms: list[tuple[str, str]] = []
+ pending_and = False
+ for m in _SEARCH_TOKEN_RE.finditer(query):
+ bare = m.group("bare")
+ if bare is not None:
+ if bare.upper() != "AND" or pending_and or not terms:
+ return None
+ pending_and = True
+ continue
+ terms.append((m.group("field"), m.group("value").strip('"')))
+ pending_and = False
+ if pending_and or not terms:
+ return None
+ return terms
+
+
+def _classic_shadow_state(thing_name: str) -> dict | None:
+ """The classic (unnamed) shadow's ``state`` document, or None."""
+ rec = _shadows.get((thing_name, ""))
+ if rec is None or rec.get("deleted"):
+ return None
+ return rec.get("state") or {}
+
+
+def _search_field_error(field: str, thing_mode: str) -> str | None:
+ """Why ``field`` is not queryable, or None when it is.
+
+ Every rejection is a 400 rather than a query that matches nothing: a
+ mistyped field and a field that no thing happens to carry are
+ indistinguishable in the results, and only one of them is the caller's bug.
+ """
+ if field in _SEARCH_TOP_FIELDS:
+ return None
+ if field.startswith("attributes.") and field != "attributes.":
+ return None
+ if field.startswith("shadow."):
+ parts = field.split(".")
+ if len(parts) < 3 or parts[1] not in _SEARCH_SHADOW_HALVES or not parts[-1]:
+ return (
+ f"Unsupported query field: {field!r} — only "
+ "shadow.desired.<path> and shadow.reported.<path> of the classic "
+ "shadow are indexed"
+ )
+ if thing_mode != "REGISTRY_AND_SHADOW":
+ return (
+ f"Shadow field {field!r} is not indexed: thingIndexingMode is "
+ f"{thing_mode}, and shadow data needs REGISTRY_AND_SHADOW"
+ )
+ return None
+ return f"Unsupported query field: {field!r}"
+
+
+def _search_field_value(thing: dict, field: str, shadow_state: dict | None):
+ if field == "thingName":
+ return thing.get("thingName")
+ if field == "thingTypeName":
+ return thing.get("thingTypeName")
+ if field == "thingGroupNames":
+ return thing.get("thingGroupNames") or None
+ if field.startswith("attributes."):
+ return (thing.get("attributes") or {}).get(field.split(".", 1)[1])
+ if field.startswith("shadow.") and shadow_state is not None:
+ node = shadow_state
+ for part in field.split(".")[1:]:
+ if not isinstance(node, dict):
+ return None
+ node = node.get(part)
+ return node
+ return None
+
+
+def _as_number(value: str) -> float | None:
+ try:
+ return float(value)
+ except (TypeError, ValueError):
+ return None
+
+
+def _search_value_matches(actual, expected: str) -> bool:
+ """Compare an indexed value against a term value, case-insensitively.
+
+ ``*`` (any run of characters) and ``?`` (exactly one) are honored — a
+ prefix query like ``thingName:my-fleet-*`` is what fleet dashboards
+ actually send. Every other character matches literally, so a ``[`` in a
+ thing name is not read as a character class the way ``fnmatch`` would.
+
+ A JSON *number* in a shadow compares numerically, so a reported
+ ``{"temp": 10.0}`` answers ``shadow.reported.temp:10`` — as an index that
+ knows the field's type does. Registry strings keep comparing as strings:
+ a thing named ``007`` is not a hit for ``thingName:7``.
+
+ A list field (``thingGroupNames``) matches when any of its members does,
+ which is what makes ``thingGroupNames:production`` a membership test.
+ """
+ if isinstance(actual, list):
+ return any(_search_value_matches(item, expected) for item in actual)
+ if actual is None:
+ # A missing field never matches — otherwise str(None) would compare as
+ # "none" and thingTypeName:none would match every untyped thing.
+ return False
+ if isinstance(actual, (int, float)) and not isinstance(actual, bool):
+ wanted = _as_number(expected)
+ if wanted is not None:
+ return float(actual) == wanted
+ # Not a number on the query side: fall through so wildcards still work.
+ actual = str(actual).lower()
+ expected = expected.lower()
+ if "*" not in expected and "?" not in expected:
+ return actual == expected
+ pattern = "".join(
+ "." if ch == "?" else ".*" if ch == "*" else re.escape(ch) for ch in expected
+ )
+ return re.fullmatch(pattern, actual, re.DOTALL) is not None
+
+
+def _search_next_token(offset: int) -> str:
+ """An opaque page cursor. It carries an offset, as AWS's own tokens do."""
+ raw = f"{_THING_INDEX_NAME}:{offset}".encode()
+ return base64.urlsafe_b64encode(raw).decode().rstrip("=")
+
+
+def _search_token_offset(token: str) -> int | None:
+ """The offset a token stands for, or None if it was not one of ours."""
+ try:
+ raw = base64.urlsafe_b64decode(token + "=" * (-len(token) % 4)).decode()
+ prefix, sep, value = raw.partition(":")
+ if not sep or prefix != _THING_INDEX_NAME:
+ return None
+ offset = int(value)
+ except Exception:
+ return None
+ return offset if offset >= 0 else None
+
+
+def _search_index(payload: dict) -> tuple:
+ """``POST /indices/search`` — fleet indexing over the live registry.
+
+ Real AWS queries a fleet-index projection; here the thing registry and
+ shadow stores ARE the index, so results are always current and the index
+ is never ``BUILDING``. It still has to be *enabled*: while
+ ``thingIndexingMode`` is OFF the ``AWS_Things`` index does not exist, and
+ querying it answers ``ResourceNotFoundException`` exactly as AWS does —
+ otherwise fleet code would work here and fail in the cloud against the
+ account nobody ran ``UpdateIndexingConfiguration`` on.
+
+ Supported grammar (the subset device fleets actually use):
+ ``AND``-separated ``field:value`` terms over ``thingName``,
+ ``thingTypeName``, ``thingGroupNames``, ``attributes.<name>`` and — once
+ ``thingIndexingMode`` is REGISTRY_AND_SHADOW —
+ ``shadow.desired|reported.<path>``, compared case-insensitively, with
+ ``*`` / ``?`` wildcards in the value. Anything outside that grammar is
+ rejected with ``InvalidQueryException``.
+
+ Results page through ``maxResults`` + ``nextToken``; the token is an
+ offset into the live match list, so a thing created between pages can
+ shift a later page, the same way a live index does.
+ """
+ index_name = payload.get("indexName") or _THING_INDEX_NAME
+ config = _indexing_configuration()
+ thing_mode = config["thingIndexingConfiguration"]["thingIndexingMode"]
+ if index_name != _THING_INDEX_NAME or thing_mode == "OFF":
+ return error_response_json(
+ "ResourceNotFoundException", f"Index {index_name!r} does not exist", 404
+ )
+
+ query = (payload.get("queryString") or "").strip()
+ if not query:
+ return error_response_json(
+ "InvalidRequestException", "queryString is required", 400
+ )
+ terms = _parse_search_query(query)
+ if terms is None:
+ return error_response_json(
+ "InvalidQueryException",
+ f"Unsupported query syntax: {query!r}",
+ 400,
+ )
+ for field, _ in terms:
+ problem = _search_field_error(field, thing_mode)
+ if problem is not None:
+ return error_response_json("InvalidQueryException", problem, 400)
+
+ max_results = payload.get("maxResults", _SEARCH_MAX_RESULTS_LIMIT)
+ if (
+ not isinstance(max_results, int)
+ or isinstance(max_results, bool)
+ or not 1 <= max_results <= _SEARCH_MAX_RESULTS_LIMIT
+ ):
+ return error_response_json(
+ "InvalidRequestException",
+ f"maxResults must be between 1 and {_SEARCH_MAX_RESULTS_LIMIT}",
+ 400,
+ )
+ token = payload.get("nextToken")
+ start = 0 if token is None else _search_token_offset(token)
+ if start is None:
+ return error_response_json(
+ "InvalidRequestException", "nextToken is not a valid page token", 400
+ )
+
+ shadows_indexed = thing_mode == "REGISTRY_AND_SHADOW"
+ matched = []
+ for name, thing in _things.items():
+ shadow_state = _classic_shadow_state(name) if shadows_indexed else None
+ if not all(
+ _search_value_matches(_search_field_value(thing, f, shadow_state), v)
+ for f, v in terms
+ ):
+ continue
+ entry = {
+ "thingName": thing["thingName"],
+ "thingId": thing.get("thingId", name),
+ "attributes": thing.get("attributes") or {},
+ "thingGroupNames": list(thing.get("thingGroupNames") or []),
+ }
+ if thing.get("thingTypeName"):
+ entry["thingTypeName"] = thing["thingTypeName"]
+ if shadow_state:
+ # AWS returns the shadow as a JSON *string* document, and both
+ # halves of the state are searchable, so both are in the document.
+ entry["shadow"] = json.dumps({
+ half: shadow_state[half]
+ for half in ("desired", "reported")
+ if half in shadow_state
+ })
+ matched.append(entry)
+
+ end = start + max_results
+ body = {"things": matched[start:end]}
+ if end < len(matched):
+ body["nextToken"] = _search_next_token(end)
+ return json_response(body)
+
+
# ---------------------------------------------------------------------------
# Topic rules
# ---------------------------------------------------------------------------
diff --git a/tests/test_iot.py b/tests/test_iot.py
index f34b52b9..f1ff11b1 100644
--- a/tests/test_iot.py
+++ b/tests/test_iot.py
@@ -488,6 +488,428 @@ def test_iot_attach_detach_policy(iot_client):
iot_client.delete_certificate(certificateId=cert["certificateId"])
+# ---------------------------------------------------------------------------
+# Fleet indexing (indexing configuration + SearchIndex)
+# ---------------------------------------------------------------------------
+
+
+def _enable_fleet_indexing(client, mode: str = "REGISTRY_AND_SHADOW") -> None:
+ """Turn the AWS_Things index on, as every fleet stack has to on AWS."""
+ client.update_indexing_configuration(
+ thingIndexingConfiguration={"thingIndexingMode": mode}
+ )
+
+
+def _iot_client_for_fresh_account():
+ """An IoT client on an account of its own, so its index starts OFF."""
+ import os
+
+ import boto3
+ from botocore.config import Config
+
+ return boto3.client(
+ "iot",
+ endpoint_url=os.environ.get("MINISTACK_ENDPOINT", "http://localhost:4566"),
+ aws_access_key_id=f"{uuid.uuid4().int % 10**12:012d}",
+ aws_secret_access_key="test",
+ region_name="us-east-1",
+ config=Config(retries={"mode": "standard"}),
+ )
+
+
+def test_iot_indexing_configuration_round_trip():
+ client = _iot_client_for_fresh_account()
+ # AWS's default for an account that never configured indexing.
+ default = client.get_indexing_configuration()
+ assert default["thingIndexingConfiguration"]["thingIndexingMode"] == "OFF"
+ assert client.list_indices()["indexNames"] == []
+ with pytest.raises(ClientError) as ei:
+ client.describe_index(indexName="AWS_Things")
+ assert ei.value.response["Error"]["Code"] == "ResourceNotFoundException"
+
+ client.update_indexing_configuration(
+ thingIndexingConfiguration={"thingIndexingMode": "REGISTRY_AND_SHADOW"},
+ thingGroupIndexingConfiguration={"thingGroupIndexingMode": "ON"},
+ )
+ stored = client.get_indexing_configuration()
+ assert stored["thingIndexingConfiguration"]["thingIndexingMode"] == (
+ "REGISTRY_AND_SHADOW"
+ )
+ assert stored["thingGroupIndexingConfiguration"]["thingGroupIndexingMode"] == "ON"
+ assert client.list_indices()["indexNames"] == ["AWS_Things"]
+ index = client.describe_index(indexName="AWS_Things")
+ # The registry *is* the index, so it is never BUILDING.
+ assert index["indexStatus"] == "ACTIVE"
+ assert index["schema"] == "REGISTRY_AND_SHADOW"
+
+ # Registry-only indexing narrows the schema back.
+ _enable_fleet_indexing(client, "REGISTRY")
+ assert client.describe_index(indexName="AWS_Things")["schema"] == "REGISTRY"
+
+ # ...and turning it off retires the index.
+ _enable_fleet_indexing(client, "OFF")
+ assert client.list_indices()["indexNames"] == []
+
+
+def test_iot_indexing_configuration_rejects_impossible_modes():
+ client = _iot_client_for_fresh_account()
+ for kwargs in (
+ {"thingIndexingConfiguration": {"thingIndexingMode": "REGISTRY_AND_LOGS"}},
+ # AWS: connectivity indexing requires thing indexing to be on.
+ {
+ "thingIndexingConfiguration": {
+ "thingIndexingMode": "OFF",
+ "thingConnectivityIndexingMode": "STATUS",
+ }
+ },
+ {"thingGroupIndexingConfiguration": {"thingGroupIndexingMode": "SOMETIMES"}},
+ ):
+ with pytest.raises(ClientError) as ei:
+ client.update_indexing_configuration(**kwargs)
+ assert ei.value.response["Error"]["Code"] == "InvalidRequestException", kwargs
+ # A rejected update leaves the configuration untouched.
+ assert client.get_indexing_configuration()["thingIndexingConfiguration"][
+ "thingIndexingMode"
+ ] == "OFF"
+
+
+def test_iot_search_index_requires_indexing_enabled():
+ """Searching an account that never enabled indexing is a 404 on AWS."""
+ client = _iot_client_for_fresh_account()
+ name = _unique("unindexed")
+ client.create_thing(thingName=name)
+ try:
+ with pytest.raises(ClientError) as ei:
+ client.search_index(queryString=f"thingName:{name}")
+ assert ei.value.response["Error"]["Code"] == "ResourceNotFoundException"
+
+ _enable_fleet_indexing(client)
+ assert [
+ t["thingName"]
+ for t in client.search_index(queryString=f"thingName:{name}")["things"]
+ ] == [name]
+
+ # An index this emulator does not have is the same 404.
+ with pytest.raises(ClientError) as ei:
+ client.search_index(indexName="AWS_Fleet", queryString="thingName:*")
+ assert ei.value.response["Error"]["Code"] == "ResourceNotFoundException"
+ finally:
+ client.delete_thing(thingName=name)
+
+
+def test_iot_search_index_shadow_terms_need_shadow_indexing():
+ """REGISTRY-only indexing does not project shadows, so shadow terms 400."""
+ client = _iot_client_for_fresh_account()
+ _enable_fleet_indexing(client, "REGISTRY")
+ with pytest.raises(ClientError) as ei:
+ client.search_index(queryString="shadow.reported.firmware:fw-1")
+ assert ei.value.response["Error"]["Code"] == "InvalidQueryException"
+ # Registry fields keep working in the narrower mode.
+ client.search_index(queryString="thingName:whatever")
+
+
+def test_iot_search_index_registry_and_shadow(iot_client, iot_data_client):
+ _enable_fleet_indexing(iot_client)
+ suffix = uuid.uuid4().hex[:8]
+ type_name = f"searchtype-{suffix}"
+ hit = f"searchthing-{suffix}-hit"
+ miss = f"searchthing-{suffix}-miss"
+ iot_client.create_thing_type(thingTypeName=type_name)
+ iot_client.create_thing(
+ thingName=hit,
+ thingTypeName=type_name,
+ attributePayload={"attributes": {"fleet": f"fleet-{suffix}"}},
+ )
+ iot_client.create_thing(
+ thingName=miss, attributePayload={"attributes": {"fleet": "other"}}
+ )
+ iot_data_client.update_thing_shadow(
+ thingName=hit,
+ payload=json.dumps(
+ {"state": {"reported": {"firmware": f"fw-{suffix}"}}}
+ ).encode(),
+ )
+ try:
+ by_name = iot_client.search_index(queryString=f"thingName:{hit}")["things"]
+ assert [t["thingName"] for t in by_name] == [hit]
+ doc = by_name[0]
+ assert doc["thingId"]
+ assert doc["thingTypeName"] == type_name
+ assert doc["attributes"] == {"fleet": f"fleet-{suffix}"}
+ shadow = json.loads(doc["shadow"]) # AWS returns a JSON *string*
+ assert shadow["reported"]["firmware"] == f"fw-{suffix}"
+
+ by_type = iot_client.search_index(queryString=f"thingTypeName:{type_name}")[
+ "things"
+ ]
+ assert [t["thingName"] for t in by_type] == [hit]
+
+ by_attr = iot_client.search_index(
+ queryString=f"attributes.fleet:fleet-{suffix}"
+ )["things"]
+ assert [t["thingName"] for t in by_attr] == [hit]
+
+ by_shadow = iot_client.search_index(
+ queryString=(
+ f"shadow.reported.firmware:fw-{suffix} AND thingTypeName:{type_name}"
+ ),
+ maxResults=10,
+ )["things"]
+ assert [t["thingName"] for t in by_shadow] == [hit]
+
+ empty = iot_client.search_index(queryString=f"thingName:absent-{suffix}")
+ assert empty["things"] == []
+ finally:
+ iot_data_client.delete_thing_shadow(thingName=hit)
+ iot_client.delete_thing(thingName=hit)
+ iot_client.delete_thing(thingName=miss)
+ iot_client.deprecate_thing_type(thingTypeName=type_name)
+ iot_client.delete_thing_type(thingTypeName=type_name)
+
+
+def test_iot_search_index_wildcards(iot_client):
+ _enable_fleet_indexing(iot_client)
+ # The single most common fleet query: every thing under a name prefix.
+ suffix = uuid.uuid4().hex[:8]
+ fleet = [f"fleet-{suffix}-a", f"fleet-{suffix}-b"]
+ outside = f"other-{suffix}"
+ for name in (*fleet, outside):
+ iot_client.create_thing(
+ thingName=name, attributePayload={"attributes": {"site": f"berlin-{suffix}"}}
+ )
+ try:
+ prefix = iot_client.search_index(queryString=f"thingName:fleet-{suffix}-*")
+ assert sorted(t["thingName"] for t in prefix["things"]) == fleet
+
+ suffix_hits = iot_client.search_index(queryString=f"thingName:*-{suffix}-a")
+ assert [t["thingName"] for t in suffix_hits["things"]] == [fleet[0]]
+
+ contains = iot_client.search_index(queryString=f"thingName:*{suffix}*")
+ assert sorted(t["thingName"] for t in contains["things"]) == sorted(
+ [*fleet, outside]
+ )
+
+ # ? matches exactly one character.
+ single = iot_client.search_index(queryString=f"thingName:fleet-{suffix}-?")
+ assert sorted(t["thingName"] for t in single["things"]) == fleet
+ assert (
+ iot_client.search_index(queryString=f"thingName:fleet-{suffix}-??")["things"]
+ == []
+ )
+
+ # Wildcards work on attributes too, and combine with AND.
+ combined = iot_client.search_index(
+ queryString=f"thingName:fleet-{suffix}-* AND attributes.site:berlin-*"
+ )
+ assert sorted(t["thingName"] for t in combined["things"]) == fleet
+ finally:
+ for name in (*fleet, outside):
+ iot_client.delete_thing(thingName=name)
+
+
+def test_iot_search_index_returns_desired_and_reported_shadow(
+ iot_client, iot_data_client
+):
+ _enable_fleet_indexing(iot_client)
+ # shadow.desired.<path> is queryable, so the returned document has to
+ # carry the desired half too — not reported only.
+ suffix = uuid.uuid4().hex[:8]
+ name = f"searchthing-{suffix}-desired"
+ iot_client.create_thing(thingName=name)
+ iot_data_client.update_thing_shadow(
+ thingName=name,
+ payload=json.dumps(
+ {
+ "state": {
+ "desired": {"firmware": f"want-{suffix}"},
+ "reported": {"firmware": f"have-{suffix}"},
+ }
+ }
+ ).encode(),
+ )
+ try:
+ hits = iot_client.search_index(
+ queryString=f"shadow.desired.firmware:want-{suffix}"
+ )["things"]
+ assert [t["thingName"] for t in hits] == [name]
+ shadow = json.loads(hits[0]["shadow"])
+ assert shadow["desired"]["firmware"] == f"want-{suffix}"
+ assert shadow["reported"]["firmware"] == f"have-{suffix}"
+ finally:
+ iot_data_client.delete_thing_shadow(thingName=name)
+ iot_client.delete_thing(thingName=name)
+
+
+def test_iot_search_index_unsupported_query_rejected(iot_client):
+ _enable_fleet_indexing(iot_client)
+ for query in (
+ "connectivity.connected:true",
+ "NOT thingName:foo",
+ "thingName:foo AND", # a dangling AND is an error on AWS, not a no-op
+ "AND thingName:foo",
+ "thingName:foo AND AND thingTypeName:bar",
+ "thingName:foo OR thingTypeName:bar",
+ ):
+ with pytest.raises(ClientError) as ei:
+ iot_client.search_index(queryString=query)
+ assert ei.value.response["Error"]["Code"] == "InvalidQueryException", query
+
+
+def test_iot_search_index_missing_field_never_matches(iot_client):
+ _enable_fleet_indexing(iot_client)
+ # An untyped thing has no thingTypeName; str(None) == "none" must NOT
+ # make it match thingTypeName:none, and a bare * must not resurrect it.
+ name = f"searchthing-{uuid.uuid4().hex[:8]}-untyped"
+ iot_client.create_thing(thingName=name)
+ try:
+ for query in ("thingTypeName:none", "thingTypeName:*"):
+ hits = iot_client.search_index(queryString=query)["things"]
+ assert name not in [t["thingName"] for t in hits], query
+ finally:
+ iot_client.delete_thing(thingName=name)
+
+
+def test_iot_search_index_unknown_shadow_subfield_rejected(iot_client):
+ """A shadow field outside desired/reported is a typo, not an empty result."""
+ _enable_fleet_indexing(iot_client)
+ for query in (
+ "shadow.metadata.reported.firmware:1", # AWS field, not projected here
+ "shadow.name.telemetry.reported.x:1", # named shadows are not indexed
+ "shadow.reported:anything", # a whole half is not a leaf
+ "shadow.version:2",
+ ):
+ with pytest.raises(ClientError) as ei:
+ iot_client.search_index(queryString=query)
+ assert ei.value.response["Error"]["Code"] == "InvalidQueryException", query
+
+
+def test_iot_search_index_thing_group_terms(iot_client):
+ """`thingGroupNames:<group>` is the membership query a fleet console runs."""
+ _enable_fleet_indexing(iot_client)
+ suffix = uuid.uuid4().hex[:8]
+ groups = [f"group-{suffix}-eu", f"group-{suffix}-canary"]
+ member = f"grouped-{suffix}"
+ outsider = f"ungrouped-{suffix}"
+ for group in groups:
+ iot_client.create_thing_group(thingGroupName=group)
+ iot_client.create_thing(thingName=member)
+ iot_client.create_thing(thingName=outsider)
+ for group in groups:
+ iot_client.add_thing_to_thing_group(thingGroupName=group, thingName=member)
+ try:
+ for group in groups:
+ hits = iot_client.search_index(queryString=f"thingGroupNames:{group}")[
+ "things"
+ ]
+ assert [t["thingName"] for t in hits] == [member], group
+ assert sorted(hits[0]["thingGroupNames"]) == sorted(groups)
+
+ # Both memberships are visible at once, and wildcards work on them.
+ both = iot_client.search_index(
+ queryString=f"thingGroupNames:{groups[0]} AND thingGroupNames:{groups[1]}"
+ )["things"]
+ assert [t["thingName"] for t in both] == [member]
+ assert [
+ t["thingName"]
+ for t in iot_client.search_index(
+ queryString=f"thingGroupNames:group-{suffix}-*"
+ )["things"]
+ ] == [member]
+
+ # A thing in no group carries the field as an empty list, and a group
+ # it does not belong to never matches it.
+ hit = iot_client.search_index(queryString=f"thingName:{outsider}")["things"][0]
+ assert hit["thingGroupNames"] == []
+ finally:
+ iot_client.delete_thing(thingName=member)
+ iot_client.delete_thing(thingName=outsider)
+ for group in groups:
+ iot_client.delete_thing_group(thingGroupName=group)
+
+
+def test_iot_search_index_paginates(iot_client):
+ _enable_fleet_indexing(iot_client)
+ suffix = uuid.uuid4().hex[:8]
+ names = sorted(f"paged-{suffix}-{i}" for i in range(3))
+ for name in names:
+ iot_client.create_thing(thingName=name)
+ try:
+ query = f"thingName:paged-{suffix}-*"
+ seen = []
+ token = None
+ for _ in range(len(names) + 1):
+ kwargs = {"nextToken": token} if token else {}
+ page = iot_client.search_index(queryString=query, maxResults=1, **kwargs)
+ seen.extend(t["thingName"] for t in page["things"])
+ token = page.get("nextToken")
+ if token is None:
+ break
+ assert token is None, "pagination did not terminate"
+ assert sorted(seen) == names
+
+ # The last page of an exact fit carries no token.
+ full = iot_client.search_index(queryString=query, maxResults=len(names))
+ assert "nextToken" not in full
+ assert len(full["things"]) == len(names)
+ finally:
+ for name in names:
+ iot_client.delete_thing(thingName=name)
+
+
+def test_iot_search_index_rejects_bad_paging_arguments(iot_client):
+ _enable_fleet_indexing(iot_client)
+ # (maxResults=0 never reaches the server: botocore enforces the model's
+ # min=1 client-side. The ceiling is not in the model, so it is ours.)
+ for kwargs in (
+ {"maxResults": 101}, # AWS's documented ceiling
+ {"nextToken": "not-a-real-cursor"},
+ ):
+ with pytest.raises(ClientError) as ei:
+ iot_client.search_index(queryString="thingName:*", **kwargs)
+ assert ei.value.response["Error"]["Code"] == "InvalidRequestException", kwargs
+
+
+def test_iot_search_index_numeric_shadow_values(iot_client, iot_data_client):
+ """A JSON number in a shadow compares as a number, not as its repr."""
+ _enable_fleet_indexing(iot_client)
+ suffix = uuid.uuid4().hex[:8]
+ name = f"numeric-{suffix}"
+ iot_client.create_thing(thingName=name)
+ iot_data_client.update_thing_shadow(
+ thingName=name,
+ payload=json.dumps(
+ {"state": {"reported": {"temp": 10.0, "errors": 0}}}
+ ).encode(),
+ )
+ try:
+ for query in (
+ f"thingName:{name} AND shadow.reported.temp:10",
+ f"thingName:{name} AND shadow.reported.temp:10.0",
+ f"thingName:{name} AND shadow.reported.errors:0",
+ ):
+ hits = iot_client.search_index(queryString=query)["things"]
+ assert [t["thingName"] for t in hits] == [name], query
+ for query in (
+ f"thingName:{name} AND shadow.reported.temp:11",
+ f"thingName:{name} AND shadow.reported.temp:1",
+ ):
+ assert iot_client.search_index(queryString=query)["things"] == [], query
+
+ # A registry string stays a string: 007 is not 7.
+ zeros = f"0007{suffix}"
+ iot_client.create_thing(thingName=zeros)
+ try:
+ assert (
+ iot_client.search_index(queryString=f"thingName:7{suffix}")["things"]
+ == []
+ )
+ finally:
+ iot_client.delete_thing(thingName=zeros)
+ finally:
+ iot_data_client.delete_thing_shadow(thingName=name)
+ iot_client.delete_thing(thingName=name)
+
+
# ---------------------------------------------------------------------------
# Topic rules
# ---------------------------------------------------------------------------