Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
feat(iot): fleet indexing — SearchIndex over the live registry, behind the index switch (#1382)
iot-rocketd6fc4622b2d5
diff --git a/README.md b/README.mdindex c14c69b1..01eea506 100644--- a/README.md+++ b/README.md@@ -583,7 +583,7 @@ MiniStack expands AWS SAM transforms (`Transform: AWS::Serverless-2016-10-31`) i| **CodeBuild** | CreateProject, BatchGetProjects, ListProjects, UpdateProject, DeleteProject, StartBuild, BatchGetBuilds, StopBuild, ListBuilds, ListBuildsForProject, BatchDeleteBuilds | 11 actions; project and build metadata stored in-memory; builds complete immediately with SUCCEEDED status by default. Set `MINISTACK_CODEBUILD_EXECUTE=1` and mount the Docker socket to **really run them**: the project's inline buildspec goes to the official AWS CodeBuild local agent, which runs the phases in `environment.image`, and `BatchGetBuilds` reports live `phases` with a real `SUCCEEDED`/`FAILED` outcome. Execution needs the workspace shared with the host at the **same path** (`-v /tmp/ministack-codebuild:/tmp/ministack-codebuild`), because the agent bind-mounts it into the build container; a build gets `AWS_ENDPOINT_URL` pointing back at MiniStack, honours `timeoutInMinutes`, and is stopped by `StopBuild`. An executed build streams the agent output into its `logs.groupName` / `logs.streamName`, so `aws logs tail` reads the build log back || **AppConfig** | CreateApplication, GetApplication, ListApplications, UpdateApplication, DeleteApplication, CreateEnvironment, GetEnvironment, ListEnvironments, UpdateEnvironment, DeleteEnvironment, CreateConfigurationProfile, GetConfigurationProfile, ListConfigurationProfiles, UpdateConfigurationProfile, DeleteConfigurationProfile, CreateHostedConfigurationVersion, GetHostedConfigurationVersion, ListHostedConfigurationVersions, DeleteHostedConfigurationVersion, CreateDeploymentStrategy, GetDeploymentStrategy, ListDeploymentStrategies, UpdateDeploymentStrategy, DeleteDeploymentStrategy, StartDeployment, GetDeployment, ListDeployments, StopDeployment, TagResource, UntagResource, ListTagsForResource, StartConfigurationSession, GetLatestConfiguration | 33 operations; control plane + data plane; hosted configuration versions; deployments complete immediately; session-based configuration retrieval with token rotation || **Transfer Family** | CreateServer, DescribeServer, DeleteServer, ListServers, StartServer, StopServer, CreateUser, DescribeUser, DeleteUser, ListUsers, ImportSshPublicKey, DeleteSshPublicKey | 12 operations; **real SFTP listener** on `:2222` (override with `SFTP_PORT`) backed by S3 — `ls`, `put`, `get`, `mkdir`, `rename` work end-to-end against the local S3 emulator; `SFTP_PORT_PER_SERVER=1` allocates one port per `CreateServer` from `SFTP_BASE_PORT` (default 2300); SSH key auth scans every user across every server and account; `LOGICAL` and `PATH` home-directory mappings; host key persists across restarts when `PERSIST_STATE=1` |-| **IoT Core** | CreateThing, DescribeThing, ListThings, UpdateThing, DeleteThing, CreateThingType, CreateThingGroup, AddThingToThingGroup, ListThingsInThingGroup, CreateKeysAndCertificate, RegisterCertificate, UpdateCertificate, DeleteCertificate, AttachThingPrincipal, DetachThingPrincipal, ListThingPrincipals, ListPrincipalThings, CreatePolicy, CreatePolicyVersion, AttachPolicy, DetachPolicy, ListAttachedPolicies, ListTargetsForPolicy, DescribeEndpoint, Publish (HTTP), MQTT-over-WebSocket pub/sub | 24 operations + **real MQTT 3.1.1 over WebSocket** on the gateway port — clients use the address returned by `DescribeEndpoint`; local CA signs `CreateKeysAndCertificate` certificates (CA persists across restarts when `PERSIST_STATE=1`); multi-tenancy via transparent topic prefixing (same pattern as Transfer Family's shared SFTP listener); no plain TCP 1883 (real AWS IoT requires TLS or SigV4 on every connection); local CA cert available at `GET /_ministack/iot/ca.pem` for SDK trust configuration; IoT policies are stored but **not enforced** on the data plane |+| **IoT Core** | CreateThing, DescribeThing, ListThings, UpdateThing, DeleteThing, CreateThingType, CreateThingGroup, AddThingToThingGroup, ListThingsInThingGroup, CreateKeysAndCertificate, RegisterCertificate, UpdateCertificate, DeleteCertificate, AttachThingPrincipal, DetachThingPrincipal, ListThingPrincipals, ListPrincipalThings, CreatePolicy, CreatePolicyVersion, AttachPolicy, DetachPolicy, ListAttachedPolicies, ListTargetsForPolicy, UpdateIndexingConfiguration, GetIndexingConfiguration, DescribeIndex, ListIndices, SearchIndex, DescribeEndpoint, Publish (HTTP), MQTT-over-WebSocket pub/sub | 29 operations + **real MQTT 3.1.1 over WebSocket** on the gateway port — clients use the address returned by `DescribeEndpoint`; local CA signs `CreateKeysAndCertificate` certificates (CA persists across restarts when `PERSIST_STATE=1`); multi-tenancy via transparent topic prefixing (same pattern as Transfer Family's shared SFTP listener); no plain TCP 1883 (real AWS IoT requires TLS or SigV4 on every connection); local CA cert available at `GET /_ministack/iot/ca.pem` for SDK trust configuration; IoT policies are stored but **not enforced** on the data plane; **fleet indexing** works the way it does on AWS — `UpdateIndexingConfiguration` turns the `AWS_Things` index on (Terraform `aws_iot_indexing_configuration`), and searching while `thingIndexingMode` is `OFF` answers `ResourceNotFoundException`; `SearchIndex` then queries the live registry + classic shadows with `AND`-separated `thingName` / `thingTypeName` / `thingGroupNames` / `attributes.*` / `shadow.reported.*` / `shadow.desired.*` terms (`*` / `?` wildcards, `maxResults` / `nextToken` paging) — there is no projection to build, so hits are always current || **EventBridge Scheduler** | CreateSchedule, GetSchedule, UpdateSchedule, DeleteSchedule, ListSchedules, CreateScheduleGroup, GetScheduleGroup, DeleteScheduleGroup, ListScheduleGroups, TagResource, UntagResource, ListTagsForResource | 12 actions; schedules **fire their targets** — a background sweep invokes each enabled schedule's target (Lambda, SQS, SNS, Step Functions) when due, honoring `State`, `StartDate`/`EndDate`, and `ActionAfterCompletion: DELETE`; schedule groups with cascading deletes; `rate()`, `cron()`, `at()` expressions; group/prefix/state filters on list; default group auto-created; CFN `AWS::Scheduler::Schedule` and `AWS::Scheduler::ScheduleGroup` supported || **EKS** | CreateCluster, DescribeCluster, ListClusters, DeleteCluster, CreateNodegroup, DescribeNodegroup, ListNodegroups, DeleteNodegroup, CreateAddon, DescribeAddon, ListAddons, UpdateAddon, DeleteAddon, AssociateEncryptionConfig, CreateAccessEntry, DescribeAccessEntry, ListAccessEntries, UpdateAccessEntry, DeleteAccessEntry, AssociateAccessPolicy, DisassociateAccessPolicy, ListAssociatedAccessPolicies, AssociateIdentityProviderConfig, DescribeIdentityProviderConfig, DisassociateIdentityProviderConfig, ListIdentityProviderConfigs, TagResource, UntagResource, ListTagsForResource | `CreateCluster` spawns a real **k3s** container (75 MB) with a full Kubernetes API server; `kubectl`, Helm, and any K8s tooling work out of the box; cascading delete removes nodegroups and k3s container; addon CRUD (e.g. `vpc-cni`, `coredns`, `kube-proxy`, `aws-ebs-csi-driver`) at `/clusters/{name}/addons` flips status to `ACTIVE` on create / update; `AssociateEncryptionConfig` at `/clusters/{name}/encryption-config/associate` records KMS encryption config (rejects re-association, AWS-shape `update` envelope); **Access Entries** (modern post-1.29 IAM bindings — replaces aws-auth ConfigMap) at `/clusters/{name}/access-entries[/{principalArn}[/access-policies[/{policyArn}]]]` for Crossplane / Terraform `aws_eks_access_entry` + `aws_eks_access_policy_association`; OIDC discovery + JWKS served at the cluster's `identity.oidc.issuer` URL so Terraform's `aws_iam_openid_connect_provider` (IRSA) works against ministack; **OIDC Identity Provider Config** (`AssociateIdentityProviderConfig` / `DescribeIdentityProviderConfig` / `DisassociateIdentityProviderConfig` / `ListIdentityProviderConfigs`) at `/clusters/{name}/identity-provider-configs[/{verb}]` — one OIDC IdP per cluster, issuer + clientId + claims forwarded to the k3s API server via `--kube-apiserver-arg=oidc-*` flags, tags reachable via `ListTagsForResource`; CFN `AWS::EKS::Cluster` and `AWS::EKS::Nodegroup` supported || **OpenSearch Service** | CreateDomain, DescribeDomain, DescribeDomains, DeleteDomain, ListDomainNames, UpdateDomainConfig, DescribeDomainConfig, DescribeDomainChangeProgress, ListVersions, GetCompatibleVersions, AddTags, ListTags, RemoveTags | Management plane on `/2021-01-01/*` (`boto3.client("opensearch")`, SigV4 scope `es`). Default data plane is a stub endpoint (`<domain>.ministack.local:9200`) — set `OPENSEARCH_DATAPLANE=1` to spawn one real `opensearchproject/opensearch` container per `CreateDomain` (same pattern as ElastiCache/RDS); `DescribeDomain.Endpoint` then points at that container and `_cluster/health`/`/_search` work end-to-end. Add `OPENSEARCH_DASHBOARDS=1` (with dataplane on) to also spawn a per-domain `opensearch-dashboards` sidecar; `DescribeDomain.DashboardEndpoint` is populated. ARNs follow `arn:aws:es:<region>:<account>:domain/<name>`; Terraform `aws_opensearch_domain` resource compatible. |diff --git a/ministack/services/iot.py b/ministack/services/iot.pyindex 27a46859..f10e5264 100644--- a/ministack/services/iot.py+++ b/ministack/services/iot.py@@ -11,6 +11,9 @@ Implements the JSON/REST APIs under ``iot.{region}.amazonaws.com``:``AttachThingPrincipal`` / ``DetachThingPrincipal``- Policies: ``CreatePolicy``, ``CreatePolicyVersion``, ``AttachPolicy``,``DetachPolicy``, etc.+ - Fleet indexing: ``UpdateIndexingConfiguration`` /+ ``GetIndexingConfiguration`` / ``DescribeIndex`` / ``ListIndices``, and+ ``SearchIndex`` over the live registry + shadows- ``DescribeEndpoint`` returning a per-account hostnameThis is the control plane — pure HTTP/JSON, no MQTT broker@@ -78,6 +81,8 @@ _certificates: AccountRegionScopedDict = AccountRegionScopedDict()_policies: AccountRegionScopedDict = AccountRegionScopedDict()_topic_rules: AccountRegionScopedDict = AccountRegionScopedDict()_shadows: AccountRegionScopedDict = AccountRegionScopedDict()+# Fleet-indexing configuration, one entry per account+region.+_indexing_config: AccountRegionScopedDict = AccountRegionScopedDict()# Local CA state — lazily generated on first use, persisted across restarts.import threading@@ -167,6 +172,7 @@ def get_state() -> dict:"policies": copy.deepcopy(_policies),"topic_rules": copy.deepcopy(_topic_rules),"shadows": copy.deepcopy(_shadows),+ "indexing_config": copy.deepcopy(_indexing_config),"ca": {"ca_cert_pem": _ca_cert_pem, "ca_key_pem": _ca_key_pem}if _ca_cert_pem and _ca_key_pemelse {},@@ -185,6 +191,7 @@ def restore_state(data: dict | None) -> None:_policies.update(data.get("policies", {}))_topic_rules.update(data.get("topic_rules", {}))_shadows.update(data.get("shadows", {}))+ _indexing_config.update(data.get("indexing_config", {}))ca_data = data.get("ca")if ca_data:cert = ca_data.get("ca_cert_pem")@@ -206,6 +213,7 @@ def reset() -> None:_policies.clear()_topic_rules.clear()_shadows.clear()+ _indexing_config.clear()with _CA_LOCK:_ca_cert_pem = None_ca_key_pem = None@@ -387,6 +395,18 @@ async def handle_request(if path.startswith("/policies/"):return _handle_policy(method, path, body, qp)+ # Fleet indexing — the search path must precede ``/indices/{indexName}``+ if path == "/indices/search" and method == "POST":+ return _search_index(_parse_body(body))+ if path == "/indexing/config" and method == "POST":+ return _update_indexing_configuration(_parse_body(body))+ if path == "/indexing/config" and method == "GET":+ return _get_indexing_configuration()+ if path == "/indices" and method == "GET":+ return _list_indices()+ if path.startswith("/indices/") and method == "GET":+ return _describe_index(path)+# Topic rulesif path == "/rules" and method == "GET":return _list_topic_rules(qp)@@ -1389,6 +1409,430 @@ def _list_attached_policies(path: str, qp: dict) -> tuple:return json_response({"policies": out})+# ---------------------------------------------------------------------------+# Fleet indexing (indexing configuration + SearchIndex)+# ---------------------------------------------------------------------------++# One token: either a ``field:value`` term (quoted values may contain spaces)+# or a bare word, which the grammar only accepts as the ``AND`` separator.+_SEARCH_TOKEN_RE = re.compile(+ r'(?P<field>[\w.]+)\s*:\s*(?P<value>"[^"]*"|\S+)|(?P<bare>"[^"]*"|\S+)'+)+_SEARCH_TOP_FIELDS = ("thingName", "thingTypeName", "thingGroupNames")+# Only the classic shadow is indexed, and only its two state halves are+# addressable: shadow.metadata / shadow.version / shadow.name.<name> (named+# shadows) are AWS fields this emulator does not project.+_SEARCH_SHADOW_HALVES = ("desired", "reported")++# AWS names the single thing index ``AWS_Things``; it exists only while thing+# indexing is enabled, which is why every fleet stack starts with an+# ``UpdateIndexingConfiguration`` call (Terraform: aws_iot_indexing_configuration).+_THING_INDEX_NAME = "AWS_Things"+_INDEXING_CONFIG_KEY = "indexing"+_THING_INDEXING_MODES = ("OFF", "REGISTRY", "REGISTRY_AND_SHADOW")+_THING_CONNECTIVITY_INDEXING_MODES = ("OFF", "STATUS")+_THING_GROUP_INDEXING_MODES = ("OFF", "ON")+# AWS's defaults for an account that has never called+# UpdateIndexingConfiguration: nothing is indexed.+_DEFAULT_INDEXING_CONFIG = {+ "thingIndexingConfiguration": {+ "thingIndexingMode": "OFF",+ "thingConnectivityIndexingMode": "OFF",+ },+ "thingGroupIndexingConfiguration": {"thingGroupIndexingMode": "OFF"},+}+# SearchIndex's documented ceiling ("this maximum number cannot exceed 100").+_SEARCH_MAX_RESULTS_LIMIT = 100+++def _indexing_configuration() -> dict:+ """The account+region's indexing configuration, defaulting to all-OFF."""+ stored = _indexing_config.get(_INDEXING_CONFIG_KEY)+ return copy.deepcopy(stored) if stored else copy.deepcopy(_DEFAULT_INDEXING_CONFIG)+++def _thing_indexing_mode() -> str:+ return (+ _indexing_configuration()+ .get("thingIndexingConfiguration", {})+ .get("thingIndexingMode", "OFF")+ )+++def _update_indexing_configuration(payload: dict) -> tuple:+ """``POST /indexing/config`` — enable or disable fleet indexing.++ Each sub-configuration that the request carries replaces the stored one+ wholesale (AWS's own semantics: the mode is required, so a partial update+ of a sub-configuration is not expressible); an omitted sub-configuration is+ left alone.++ Only the modes this emulator can honor are interpreted —+ ``thingIndexingMode`` gates ``SearchIndex`` and whether shadow fields are+ queryable. ``deviceDefenderIndexingMode``, ``namedShadowIndexingMode``,+ ``customFields`` and ``filter`` are stored and echoed back by+ ``GetIndexingConfiguration`` so IaC round-trips cleanly, but nothing here+ projects those fields; querying one is an out-of-grammar+ ``InvalidQueryException`` rather than a silent miss.+ """+ config = _indexing_configuration()++ thing_cfg = payload.get("thingIndexingConfiguration")+ if thing_cfg is not None:+ if not isinstance(thing_cfg, dict):+ return error_response_json(+ "InvalidRequestException",+ "thingIndexingConfiguration must be an object",+ 400,+ )+ mode = thing_cfg.get("thingIndexingMode")+ if mode not in _THING_INDEXING_MODES:+ return error_response_json(+ "InvalidRequestException",+ f"thingIndexingMode must be one of {list(_THING_INDEXING_MODES)}",+ 400,+ )+ connectivity_mode = thing_cfg.get("thingConnectivityIndexingMode") or "OFF"+ if connectivity_mode not in _THING_CONNECTIVITY_INDEXING_MODES:+ return error_response_json(+ "InvalidRequestException",+ "thingConnectivityIndexingMode must be one of "+ f"{list(_THING_CONNECTIVITY_INDEXING_MODES)}",+ 400,+ )+ if connectivity_mode == "STATUS" and mode == "OFF":+ return error_response_json(+ "InvalidRequestException",+ "thingIndexingMode must not be OFF to enable thing connectivity "+ "indexing",+ 400,+ )+ config["thingIndexingConfiguration"] = {+ **thing_cfg,+ "thingIndexingMode": mode,+ "thingConnectivityIndexingMode": connectivity_mode,+ }++ group_cfg = payload.get("thingGroupIndexingConfiguration")+ if group_cfg is not None:+ if not isinstance(group_cfg, dict):+ return error_response_json(+ "InvalidRequestException",+ "thingGroupIndexingConfiguration must be an object",+ 400,+ )+ group_mode = group_cfg.get("thingGroupIndexingMode")+ if group_mode not in _THING_GROUP_INDEXING_MODES:+ return error_response_json(+ "InvalidRequestException",+ f"thingGroupIndexingMode must be one of {list(_THING_GROUP_INDEXING_MODES)}",+ 400,+ )+ config["thingGroupIndexingConfiguration"] = {+ **group_cfg,+ "thingGroupIndexingMode": group_mode,+ }++ _indexing_config[_INDEXING_CONFIG_KEY] = config+ logger.info(+ "IoT fleet indexing configured: thingIndexingMode=%s",+ config["thingIndexingConfiguration"]["thingIndexingMode"],+ )+ return json_response({})+++def _get_indexing_configuration() -> tuple:+ """``GET /indexing/config`` — the stored configuration, all-OFF by default."""+ return json_response(_indexing_configuration())+++def _index_schema(config: dict) -> str:+ """``DescribeIndex``'s schema string for the configuration in force."""+ thing_cfg = config.get("thingIndexingConfiguration", {})+ return (+ "REGISTRY_AND_SHADOW"+ if thing_cfg.get("thingIndexingMode") == "REGISTRY_AND_SHADOW"+ else "REGISTRY"+ )+++def _describe_index(path: str) -> tuple:+ """``GET /indices/{indexName}``.++ The index is a consequence of the configuration, not a resource of its+ own: while thing indexing is OFF there is nothing to describe, which is+ the same ``ResourceNotFoundException`` AWS answers with. It is never+ ``BUILDING`` here — the registry *is* the index, so it is ready the moment+ it is enabled.+ """+ name = path[len("/indices/"):]+ config = _indexing_configuration()+ if (+ name != _THING_INDEX_NAME+ or config["thingIndexingConfiguration"]["thingIndexingMode"] == "OFF"+ ):+ return error_response_json(+ "ResourceNotFoundException", f"Index {name!r} does not exist", 404+ )+ return json_response({+ "indexName": _THING_INDEX_NAME,+ "indexStatus": "ACTIVE",+ "schema": _index_schema(config),+ })+++def _list_indices() -> tuple:+ """``GET /indices`` — ``AWS_Things`` once thing indexing is on, else empty.++ ``maxResults`` / ``nextToken`` are not honored because a page can never+ overflow: there is exactly one index, or none.+ """+ enabled = _thing_indexing_mode() != "OFF"+ return json_response({"indexNames": [_THING_INDEX_NAME] if enabled else []})+++def _parse_search_query(query: str) -> list[tuple[str, str]] | None:+ """Parse ``field:value`` terms separated by ``AND`` (or by a bare space).++ Returns the terms, or ``None`` for anything outside that grammar — a bare+ word that is not ``AND``, a leading or doubled ``AND``, and a *dangling*+ ``AND`` with no term after it. AWS rejects a dangling ``AND``, so ignoring+ it here would answer a malformed query with results.+ """+ terms: list[tuple[str, str]] = []+ pending_and = False+ for m in _SEARCH_TOKEN_RE.finditer(query):+ bare = m.group("bare")+ if bare is not None:+ if bare.upper() != "AND" or pending_and or not terms:+ return None+ pending_and = True+ continue+ terms.append((m.group("field"), m.group("value").strip('"')))+ pending_and = False+ if pending_and or not terms:+ return None+ return terms+++def _classic_shadow_state(thing_name: str) -> dict | None:+ """The classic (unnamed) shadow's ``state`` document, or None."""+ rec = _shadows.get((thing_name, ""))+ if rec is None or rec.get("deleted"):+ return None+ return rec.get("state") or {}+++def _search_field_error(field: str, thing_mode: str) -> str | None:+ """Why ``field`` is not queryable, or None when it is.++ Every rejection is a 400 rather than a query that matches nothing: a+ mistyped field and a field that no thing happens to carry are+ indistinguishable in the results, and only one of them is the caller's bug.+ """+ if field in _SEARCH_TOP_FIELDS:+ return None+ if field.startswith("attributes.") and field != "attributes.":+ return None+ if field.startswith("shadow."):+ parts = field.split(".")+ if len(parts) < 3 or parts[1] not in _SEARCH_SHADOW_HALVES or not parts[-1]:+ return (+ f"Unsupported query field: {field!r} — only "+ "shadow.desired.<path> and shadow.reported.<path> of the classic "+ "shadow are indexed"+ )+ if thing_mode != "REGISTRY_AND_SHADOW":+ return (+ f"Shadow field {field!r} is not indexed: thingIndexingMode is "+ f"{thing_mode}, and shadow data needs REGISTRY_AND_SHADOW"+ )+ return None+ return f"Unsupported query field: {field!r}"+++def _search_field_value(thing: dict, field: str, shadow_state: dict | None):+ if field == "thingName":+ return thing.get("thingName")+ if field == "thingTypeName":+ return thing.get("thingTypeName")+ if field == "thingGroupNames":+ return thing.get("thingGroupNames") or None+ if field.startswith("attributes."):+ return (thing.get("attributes") or {}).get(field.split(".", 1)[1])+ if field.startswith("shadow.") and shadow_state is not None:+ node = shadow_state+ for part in field.split(".")[1:]:+ if not isinstance(node, dict):+ return None+ node = node.get(part)+ return node+ return None+++def _as_number(value: str) -> float | None:+ try:+ return float(value)+ except (TypeError, ValueError):+ return None+++def _search_value_matches(actual, expected: str) -> bool:+ """Compare an indexed value against a term value, case-insensitively.++ ``*`` (any run of characters) and ``?`` (exactly one) are honored — a+ prefix query like ``thingName:my-fleet-*`` is what fleet dashboards+ actually send. Every other character matches literally, so a ``[`` in a+ thing name is not read as a character class the way ``fnmatch`` would.++ A JSON *number* in a shadow compares numerically, so a reported+ ``{"temp": 10.0}`` answers ``shadow.reported.temp:10`` — as an index that+ knows the field's type does. Registry strings keep comparing as strings:+ a thing named ``007`` is not a hit for ``thingName:7``.++ A list field (``thingGroupNames``) matches when any of its members does,+ which is what makes ``thingGroupNames:production`` a membership test.+ """+ if isinstance(actual, list):+ return any(_search_value_matches(item, expected) for item in actual)+ if actual is None:+ # A missing field never matches — otherwise str(None) would compare as+ # "none" and thingTypeName:none would match every untyped thing.+ return False+ if isinstance(actual, (int, float)) and not isinstance(actual, bool):+ wanted = _as_number(expected)+ if wanted is not None:+ return float(actual) == wanted+ # Not a number on the query side: fall through so wildcards still work.+ actual = str(actual).lower()+ expected = expected.lower()+ if "*" not in expected and "?" not in expected:+ return actual == expected+ pattern = "".join(+ "." if ch == "?" else ".*" if ch == "*" else re.escape(ch) for ch in expected+ )+ return re.fullmatch(pattern, actual, re.DOTALL) is not None+++def _search_next_token(offset: int) -> str:+ """An opaque page cursor. It carries an offset, as AWS's own tokens do."""+ raw = f"{_THING_INDEX_NAME}:{offset}".encode()+ return base64.urlsafe_b64encode(raw).decode().rstrip("=")+++def _search_token_offset(token: str) -> int | None:+ """The offset a token stands for, or None if it was not one of ours."""+ try:+ raw = base64.urlsafe_b64decode(token + "=" * (-len(token) % 4)).decode()+ prefix, sep, value = raw.partition(":")+ if not sep or prefix != _THING_INDEX_NAME:+ return None+ offset = int(value)+ except Exception:+ return None+ return offset if offset >= 0 else None+++def _search_index(payload: dict) -> tuple:+ """``POST /indices/search`` — fleet indexing over the live registry.++ Real AWS queries a fleet-index projection; here the thing registry and+ shadow stores ARE the index, so results are always current and the index+ is never ``BUILDING``. It still has to be *enabled*: while+ ``thingIndexingMode`` is OFF the ``AWS_Things`` index does not exist, and+ querying it answers ``ResourceNotFoundException`` exactly as AWS does —+ otherwise fleet code would work here and fail in the cloud against the+ account nobody ran ``UpdateIndexingConfiguration`` on.++ Supported grammar (the subset device fleets actually use):+ ``AND``-separated ``field:value`` terms over ``thingName``,+ ``thingTypeName``, ``thingGroupNames``, ``attributes.<name>`` and — once+ ``thingIndexingMode`` is REGISTRY_AND_SHADOW —+ ``shadow.desired|reported.<path>``, compared case-insensitively, with+ ``*`` / ``?`` wildcards in the value. Anything outside that grammar is+ rejected with ``InvalidQueryException``.++ Results page through ``maxResults`` + ``nextToken``; the token is an+ offset into the live match list, so a thing created between pages can+ shift a later page, the same way a live index does.+ """+ index_name = payload.get("indexName") or _THING_INDEX_NAME+ config = _indexing_configuration()+ thing_mode = config["thingIndexingConfiguration"]["thingIndexingMode"]+ if index_name != _THING_INDEX_NAME or thing_mode == "OFF":+ return error_response_json(+ "ResourceNotFoundException", f"Index {index_name!r} does not exist", 404+ )++ query = (payload.get("queryString") or "").strip()+ if not query:+ return error_response_json(+ "InvalidRequestException", "queryString is required", 400+ )+ terms = _parse_search_query(query)+ if terms is None:+ return error_response_json(+ "InvalidQueryException",+ f"Unsupported query syntax: {query!r}",+ 400,+ )+ for field, _ in terms:+ problem = _search_field_error(field, thing_mode)+ if problem is not None:+ return error_response_json("InvalidQueryException", problem, 400)++ max_results = payload.get("maxResults", _SEARCH_MAX_RESULTS_LIMIT)+ if (+ not isinstance(max_results, int)+ or isinstance(max_results, bool)+ or not 1 <= max_results <= _SEARCH_MAX_RESULTS_LIMIT+ ):+ return error_response_json(+ "InvalidRequestException",+ f"maxResults must be between 1 and {_SEARCH_MAX_RESULTS_LIMIT}",+ 400,+ )+ token = payload.get("nextToken")+ start = 0 if token is None else _search_token_offset(token)+ if start is None:+ return error_response_json(+ "InvalidRequestException", "nextToken is not a valid page token", 400+ )++ shadows_indexed = thing_mode == "REGISTRY_AND_SHADOW"+ matched = []+ for name, thing in _things.items():+ shadow_state = _classic_shadow_state(name) if shadows_indexed else None+ if not all(+ _search_value_matches(_search_field_value(thing, f, shadow_state), v)+ for f, v in terms+ ):+ continue+ entry = {+ "thingName": thing["thingName"],+ "thingId": thing.get("thingId", name),+ "attributes": thing.get("attributes") or {},+ "thingGroupNames": list(thing.get("thingGroupNames") or []),+ }+ if thing.get("thingTypeName"):+ entry["thingTypeName"] = thing["thingTypeName"]+ if shadow_state:+ # AWS returns the shadow as a JSON *string* document, and both+ # halves of the state are searchable, so both are in the document.+ entry["shadow"] = json.dumps({+ half: shadow_state[half]+ for half in ("desired", "reported")+ if half in shadow_state+ })+ matched.append(entry)++ end = start + max_results+ body = {"things": matched[start:end]}+ if end < len(matched):+ body["nextToken"] = _search_next_token(end)+ return json_response(body)++# ---------------------------------------------------------------------------# Topic rules# ---------------------------------------------------------------------------diff --git a/tests/test_iot.py b/tests/test_iot.pyindex f34b52b9..f1ff11b1 100644--- a/tests/test_iot.py+++ b/tests/test_iot.py@@ -488,6 +488,428 @@ def test_iot_attach_detach_policy(iot_client):iot_client.delete_certificate(certificateId=cert["certificateId"])+# ---------------------------------------------------------------------------+# Fleet indexing (indexing configuration + SearchIndex)+# ---------------------------------------------------------------------------+++def _enable_fleet_indexing(client, mode: str = "REGISTRY_AND_SHADOW") -> None:+ """Turn the AWS_Things index on, as every fleet stack has to on AWS."""+ client.update_indexing_configuration(+ thingIndexingConfiguration={"thingIndexingMode": mode}+ )+++def _iot_client_for_fresh_account():+ """An IoT client on an account of its own, so its index starts OFF."""+ import os++ import boto3+ from botocore.config import Config++ return boto3.client(+ "iot",+ endpoint_url=os.environ.get("MINISTACK_ENDPOINT", "http://localhost:4566"),+ aws_access_key_id=f"{uuid.uuid4().int % 10**12:012d}",+ aws_secret_access_key="test",+ region_name="us-east-1",+ config=Config(retries={"mode": "standard"}),+ )+++def test_iot_indexing_configuration_round_trip():+ client = _iot_client_for_fresh_account()+ # AWS's default for an account that never configured indexing.+ default = client.get_indexing_configuration()+ assert default["thingIndexingConfiguration"]["thingIndexingMode"] == "OFF"+ assert client.list_indices()["indexNames"] == []+ with pytest.raises(ClientError) as ei:+ client.describe_index(indexName="AWS_Things")+ assert ei.value.response["Error"]["Code"] == "ResourceNotFoundException"++ client.update_indexing_configuration(+ thingIndexingConfiguration={"thingIndexingMode": "REGISTRY_AND_SHADOW"},+ thingGroupIndexingConfiguration={"thingGroupIndexingMode": "ON"},+ )+ stored = client.get_indexing_configuration()+ assert stored["thingIndexingConfiguration"]["thingIndexingMode"] == (+ "REGISTRY_AND_SHADOW"+ )+ assert stored["thingGroupIndexingConfiguration"]["thingGroupIndexingMode"] == "ON"+ assert client.list_indices()["indexNames"] == ["AWS_Things"]+ index = client.describe_index(indexName="AWS_Things")+ # The registry *is* the index, so it is never BUILDING.+ assert index["indexStatus"] == "ACTIVE"+ assert index["schema"] == "REGISTRY_AND_SHADOW"++ # Registry-only indexing narrows the schema back.+ _enable_fleet_indexing(client, "REGISTRY")+ assert client.describe_index(indexName="AWS_Things")["schema"] == "REGISTRY"++ # ...and turning it off retires the index.+ _enable_fleet_indexing(client, "OFF")+ assert client.list_indices()["indexNames"] == []+++def test_iot_indexing_configuration_rejects_impossible_modes():+ client = _iot_client_for_fresh_account()+ for kwargs in (+ {"thingIndexingConfiguration": {"thingIndexingMode": "REGISTRY_AND_LOGS"}},+ # AWS: connectivity indexing requires thing indexing to be on.+ {+ "thingIndexingConfiguration": {+ "thingIndexingMode": "OFF",+ "thingConnectivityIndexingMode": "STATUS",+ }+ },+ {"thingGroupIndexingConfiguration": {"thingGroupIndexingMode": "SOMETIMES"}},+ ):+ with pytest.raises(ClientError) as ei:+ client.update_indexing_configuration(**kwargs)+ assert ei.value.response["Error"]["Code"] == "InvalidRequestException", kwargs+ # A rejected update leaves the configuration untouched.+ assert client.get_indexing_configuration()["thingIndexingConfiguration"][+ "thingIndexingMode"+ ] == "OFF"+++def test_iot_search_index_requires_indexing_enabled():+ """Searching an account that never enabled indexing is a 404 on AWS."""+ client = _iot_client_for_fresh_account()+ name = _unique("unindexed")+ client.create_thing(thingName=name)+ try:+ with pytest.raises(ClientError) as ei:+ client.search_index(queryString=f"thingName:{name}")+ assert ei.value.response["Error"]["Code"] == "ResourceNotFoundException"++ _enable_fleet_indexing(client)+ assert [+ t["thingName"]+ for t in client.search_index(queryString=f"thingName:{name}")["things"]+ ] == [name]++ # An index this emulator does not have is the same 404.+ with pytest.raises(ClientError) as ei:+ client.search_index(indexName="AWS_Fleet", queryString="thingName:*")+ assert ei.value.response["Error"]["Code"] == "ResourceNotFoundException"+ finally:+ client.delete_thing(thingName=name)+++def test_iot_search_index_shadow_terms_need_shadow_indexing():+ """REGISTRY-only indexing does not project shadows, so shadow terms 400."""+ client = _iot_client_for_fresh_account()+ _enable_fleet_indexing(client, "REGISTRY")+ with pytest.raises(ClientError) as ei:+ client.search_index(queryString="shadow.reported.firmware:fw-1")+ assert ei.value.response["Error"]["Code"] == "InvalidQueryException"+ # Registry fields keep working in the narrower mode.+ client.search_index(queryString="thingName:whatever")+++def test_iot_search_index_registry_and_shadow(iot_client, iot_data_client):+ _enable_fleet_indexing(iot_client)+ suffix = uuid.uuid4().hex[:8]+ type_name = f"searchtype-{suffix}"+ hit = f"searchthing-{suffix}-hit"+ miss = f"searchthing-{suffix}-miss"+ iot_client.create_thing_type(thingTypeName=type_name)+ iot_client.create_thing(+ thingName=hit,+ thingTypeName=type_name,+ attributePayload={"attributes": {"fleet": f"fleet-{suffix}"}},+ )+ iot_client.create_thing(+ thingName=miss, attributePayload={"attributes": {"fleet": "other"}}+ )+ iot_data_client.update_thing_shadow(+ thingName=hit,+ payload=json.dumps(+ {"state": {"reported": {"firmware": f"fw-{suffix}"}}}+ ).encode(),+ )+ try:+ by_name = iot_client.search_index(queryString=f"thingName:{hit}")["things"]+ assert [t["thingName"] for t in by_name] == [hit]+ doc = by_name[0]+ assert doc["thingId"]+ assert doc["thingTypeName"] == type_name+ assert doc["attributes"] == {"fleet": f"fleet-{suffix}"}+ shadow = json.loads(doc["shadow"]) # AWS returns a JSON *string*+ assert shadow["reported"]["firmware"] == f"fw-{suffix}"++ by_type = iot_client.search_index(queryString=f"thingTypeName:{type_name}")[+ "things"+ ]+ assert [t["thingName"] for t in by_type] == [hit]++ by_attr = iot_client.search_index(+ queryString=f"attributes.fleet:fleet-{suffix}"+ )["things"]+ assert [t["thingName"] for t in by_attr] == [hit]++ by_shadow = iot_client.search_index(+ queryString=(+ f"shadow.reported.firmware:fw-{suffix} AND thingTypeName:{type_name}"+ ),+ maxResults=10,+ )["things"]+ assert [t["thingName"] for t in by_shadow] == [hit]++ empty = iot_client.search_index(queryString=f"thingName:absent-{suffix}")+ assert empty["things"] == []+ finally:+ iot_data_client.delete_thing_shadow(thingName=hit)+ iot_client.delete_thing(thingName=hit)+ iot_client.delete_thing(thingName=miss)+ iot_client.deprecate_thing_type(thingTypeName=type_name)+ iot_client.delete_thing_type(thingTypeName=type_name)+++def test_iot_search_index_wildcards(iot_client):+ _enable_fleet_indexing(iot_client)+ # The single most common fleet query: every thing under a name prefix.+ suffix = uuid.uuid4().hex[:8]+ fleet = [f"fleet-{suffix}-a", f"fleet-{suffix}-b"]+ outside = f"other-{suffix}"+ for name in (*fleet, outside):+ iot_client.create_thing(+ thingName=name, attributePayload={"attributes": {"site": f"berlin-{suffix}"}}+ )+ try:+ prefix = iot_client.search_index(queryString=f"thingName:fleet-{suffix}-*")+ assert sorted(t["thingName"] for t in prefix["things"]) == fleet++ suffix_hits = iot_client.search_index(queryString=f"thingName:*-{suffix}-a")+ assert [t["thingName"] for t in suffix_hits["things"]] == [fleet[0]]++ contains = iot_client.search_index(queryString=f"thingName:*{suffix}*")+ assert sorted(t["thingName"] for t in contains["things"]) == sorted(+ [*fleet, outside]+ )++ # ? matches exactly one character.+ single = iot_client.search_index(queryString=f"thingName:fleet-{suffix}-?")+ assert sorted(t["thingName"] for t in single["things"]) == fleet+ assert (+ iot_client.search_index(queryString=f"thingName:fleet-{suffix}-??")["things"]+ == []+ )++ # Wildcards work on attributes too, and combine with AND.+ combined = iot_client.search_index(+ queryString=f"thingName:fleet-{suffix}-* AND attributes.site:berlin-*"+ )+ assert sorted(t["thingName"] for t in combined["things"]) == fleet+ finally:+ for name in (*fleet, outside):+ iot_client.delete_thing(thingName=name)+++def test_iot_search_index_returns_desired_and_reported_shadow(+ iot_client, iot_data_client+):+ _enable_fleet_indexing(iot_client)+ # shadow.desired.<path> is queryable, so the returned document has to+ # carry the desired half too — not reported only.+ suffix = uuid.uuid4().hex[:8]+ name = f"searchthing-{suffix}-desired"+ iot_client.create_thing(thingName=name)+ iot_data_client.update_thing_shadow(+ thingName=name,+ payload=json.dumps(+ {+ "state": {+ "desired": {"firmware": f"want-{suffix}"},+ "reported": {"firmware": f"have-{suffix}"},+ }+ }+ ).encode(),+ )+ try:+ hits = iot_client.search_index(+ queryString=f"shadow.desired.firmware:want-{suffix}"+ )["things"]+ assert [t["thingName"] for t in hits] == [name]+ shadow = json.loads(hits[0]["shadow"])+ assert shadow["desired"]["firmware"] == f"want-{suffix}"+ assert shadow["reported"]["firmware"] == f"have-{suffix}"+ finally:+ iot_data_client.delete_thing_shadow(thingName=name)+ iot_client.delete_thing(thingName=name)+++def test_iot_search_index_unsupported_query_rejected(iot_client):+ _enable_fleet_indexing(iot_client)+ for query in (+ "connectivity.connected:true",+ "NOT thingName:foo",+ "thingName:foo AND", # a dangling AND is an error on AWS, not a no-op+ "AND thingName:foo",+ "thingName:foo AND AND thingTypeName:bar",+ "thingName:foo OR thingTypeName:bar",+ ):+ with pytest.raises(ClientError) as ei:+ iot_client.search_index(queryString=query)+ assert ei.value.response["Error"]["Code"] == "InvalidQueryException", query+++def test_iot_search_index_missing_field_never_matches(iot_client):+ _enable_fleet_indexing(iot_client)+ # An untyped thing has no thingTypeName; str(None) == "none" must NOT+ # make it match thingTypeName:none, and a bare * must not resurrect it.+ name = f"searchthing-{uuid.uuid4().hex[:8]}-untyped"+ iot_client.create_thing(thingName=name)+ try:+ for query in ("thingTypeName:none", "thingTypeName:*"):+ hits = iot_client.search_index(queryString=query)["things"]+ assert name not in [t["thingName"] for t in hits], query+ finally:+ iot_client.delete_thing(thingName=name)+++def test_iot_search_index_unknown_shadow_subfield_rejected(iot_client):+ """A shadow field outside desired/reported is a typo, not an empty result."""+ _enable_fleet_indexing(iot_client)+ for query in (+ "shadow.metadata.reported.firmware:1", # AWS field, not projected here+ "shadow.name.telemetry.reported.x:1", # named shadows are not indexed+ "shadow.reported:anything", # a whole half is not a leaf+ "shadow.version:2",+ ):+ with pytest.raises(ClientError) as ei:+ iot_client.search_index(queryString=query)+ assert ei.value.response["Error"]["Code"] == "InvalidQueryException", query+++def test_iot_search_index_thing_group_terms(iot_client):+ """`thingGroupNames:<group>` is the membership query a fleet console runs."""+ _enable_fleet_indexing(iot_client)+ suffix = uuid.uuid4().hex[:8]+ groups = [f"group-{suffix}-eu", f"group-{suffix}-canary"]+ member = f"grouped-{suffix}"+ outsider = f"ungrouped-{suffix}"+ for group in groups:+ iot_client.create_thing_group(thingGroupName=group)+ iot_client.create_thing(thingName=member)+ iot_client.create_thing(thingName=outsider)+ for group in groups:+ iot_client.add_thing_to_thing_group(thingGroupName=group, thingName=member)+ try:+ for group in groups:+ hits = iot_client.search_index(queryString=f"thingGroupNames:{group}")[+ "things"+ ]+ assert [t["thingName"] for t in hits] == [member], group+ assert sorted(hits[0]["thingGroupNames"]) == sorted(groups)++ # Both memberships are visible at once, and wildcards work on them.+ both = iot_client.search_index(+ queryString=f"thingGroupNames:{groups[0]} AND thingGroupNames:{groups[1]}"+ )["things"]+ assert [t["thingName"] for t in both] == [member]+ assert [+ t["thingName"]+ for t in iot_client.search_index(+ queryString=f"thingGroupNames:group-{suffix}-*"+ )["things"]+ ] == [member]++ # A thing in no group carries the field as an empty list, and a group+ # it does not belong to never matches it.+ hit = iot_client.search_index(queryString=f"thingName:{outsider}")["things"][0]+ assert hit["thingGroupNames"] == []+ finally:+ iot_client.delete_thing(thingName=member)+ iot_client.delete_thing(thingName=outsider)+ for group in groups:+ iot_client.delete_thing_group(thingGroupName=group)+++def test_iot_search_index_paginates(iot_client):+ _enable_fleet_indexing(iot_client)+ suffix = uuid.uuid4().hex[:8]+ names = sorted(f"paged-{suffix}-{i}" for i in range(3))+ for name in names:+ iot_client.create_thing(thingName=name)+ try:+ query = f"thingName:paged-{suffix}-*"+ seen = []+ token = None+ for _ in range(len(names) + 1):+ kwargs = {"nextToken": token} if token else {}+ page = iot_client.search_index(queryString=query, maxResults=1, **kwargs)+ seen.extend(t["thingName"] for t in page["things"])+ token = page.get("nextToken")+ if token is None:+ break+ assert token is None, "pagination did not terminate"+ assert sorted(seen) == names++ # The last page of an exact fit carries no token.+ full = iot_client.search_index(queryString=query, maxResults=len(names))+ assert "nextToken" not in full+ assert len(full["things"]) == len(names)+ finally:+ for name in names:+ iot_client.delete_thing(thingName=name)+++def test_iot_search_index_rejects_bad_paging_arguments(iot_client):+ _enable_fleet_indexing(iot_client)+ # (maxResults=0 never reaches the server: botocore enforces the model's+ # min=1 client-side. The ceiling is not in the model, so it is ours.)+ for kwargs in (+ {"maxResults": 101}, # AWS's documented ceiling+ {"nextToken": "not-a-real-cursor"},+ ):+ with pytest.raises(ClientError) as ei:+ iot_client.search_index(queryString="thingName:*", **kwargs)+ assert ei.value.response["Error"]["Code"] == "InvalidRequestException", kwargs+++def test_iot_search_index_numeric_shadow_values(iot_client, iot_data_client):+ """A JSON number in a shadow compares as a number, not as its repr."""+ _enable_fleet_indexing(iot_client)+ suffix = uuid.uuid4().hex[:8]+ name = f"numeric-{suffix}"+ iot_client.create_thing(thingName=name)+ iot_data_client.update_thing_shadow(+ thingName=name,+ payload=json.dumps(+ {"state": {"reported": {"temp": 10.0, "errors": 0}}}+ ).encode(),+ )+ try:+ for query in (+ f"thingName:{name} AND shadow.reported.temp:10",+ f"thingName:{name} AND shadow.reported.temp:10.0",+ f"thingName:{name} AND shadow.reported.errors:0",+ ):+ hits = iot_client.search_index(queryString=query)["things"]+ assert [t["thingName"] for t in hits] == [name], query+ for query in (+ f"thingName:{name} AND shadow.reported.temp:11",+ f"thingName:{name} AND shadow.reported.temp:1",+ ):+ assert iot_client.search_index(queryString=query)["things"] == [], query++ # A registry string stays a string: 007 is not 7.+ zeros = f"0007{suffix}"+ iot_client.create_thing(thingName=zeros)+ try:+ assert (+ iot_client.search_index(queryString=f"thingName:7{suffix}")["things"]+ == []+ )+ finally:+ iot_client.delete_thing(thingName=zeros)+ finally:+ iot_data_client.delete_thing_shadow(thingName=name)+ iot_client.delete_thing(thingName=name)++# ---------------------------------------------------------------------------# Topic rules# ---------------------------------------------------------------------------