Light, fluffy, and always free - The AWS Local Emulator alternative
fix(iot): merge nested shadow state on update (#6217)
Mario Toffia71cc70431913
diff --git a/docs/services/iot.md b/docs/services/iot.mdindex 0e104aa54d..875a4aeea2 100644--- a/docs/services/iot.md+++ b/docs/services/iot.md@@ -324,6 +324,8 @@ A successful classic or named shadow change reaches MQTT subscribers and the top- `delete/accepted`: the deleted `version`, `timestamp`, and `clientToken`. `DeleteThingShadow` returns `version` and `timestamp`.- `rejected`, for an MQTT request only: `code` (the HTTP status as a number), `message`, and `clientToken` once the request parsed. A version conflict is `409` with `Version conflict`, a payload that is not JSON is `400` with `Invalid JSON`.+An update merges into the stored shadow as on AWS. A nested object merges key by key at any depth: every leaf the update does not name keeps its value and its metadata timestamp, and `{}` sent onto a stored object or section changes nothing. A `null` deletes the key and its metadata. A nested object emptied that way stays as `{}`, but a `desired` or `reported` section that ends up empty is not kept. An object sent for a missing key adds its non-null values: `{}` adds `{}`, and an object holding only nulls adds nothing. An array or a scalar replaces the stored value whole, and so does an object sent onto a scalar or an array, which AWS stores as sent, null leaves included.+An empty `name` is the classic shadow. A name that is not 1 to 64 of `$`, `a-z`, `A-Z`, `0-9`, `:`, `_` and `-` fails with `InvalidRequestException` (`Bad Request`), and an MQTT request on such a named topic, empty name included, is ignored as on AWS, with no `rejected`. A name containing `$` other than the reserved `$package` fails with `InvalidRequestException` (`Invalid shadow name : <name>`), which an MQTT request receives as a `400` `rejected`. `$package` is served as an ordinary named shadow: AWS's reserved document schema, and its omission of `metadata`, are not emulated.Shadow events are evaluated only against the rules of the shadow's region. A REST change belongs to the region of its SigV4 credential, and MQTT shadows live in the default region (`FLOCI_DEFAULT_REGION`). `clientid()` is `N/A` for every shadow event, as on AWS. The MQTT broker itself has no region or account, so its subscribers receive the shadow events of every region and account, as they receive a republish.@@ -345,7 +347,6 @@ Current accepted limitation:- Persistent offline sessions are not modeled yet.- QoS 2 and advanced MQTT 5 property semantics remain follow-up scope.- At most 1,000 MQTT publishes are pending rule evaluation, running or waiting. While that many are pending, further publishes are still delivered to subscribers but their topic rules are skipped, with one warning logged until the backlog drains.-- Shadow state is merged shallowly: a nested object in `desired` or `reported` replaces the stored value of its top-level key instead of being merged into it.- REST shadow errors are not published to `rejected`.- MQTT shadow request topics are not themselves delivered to rules or subscribers.- An IoT Data `Publish` to a shadow request topic does not update the shadow, and a `republish` that targets a shadow request topic is not processed as a shadow request.diff --git a/src/main/java/io/github/hectorvent/floci/services/iot/IotService.java b/src/main/java/io/github/hectorvent/floci/services/iot/IotService.javaindex 710b0d30f7..32663d1231 100644--- a/src/main/java/io/github/hectorvent/floci/services/iot/IotService.java+++ b/src/main/java/io/github/hectorvent/floci/services/iot/IotService.java@@ -2081,8 +2081,9 @@ public class IotService {}/**- * Applies one section of an update shallowly, to the state and its metadata alike: a value sets- * the top-level key, a null removes it, and a null section removes the whole section.+ * Applies one section of an update to the state and its metadata alike. A null section removes the+ * whole section, an object merges through {@link #mergeShadowObject}, and a section left empty is+ * removed, since AWS never keeps an empty {@code desired} or {@code reported}.*/private void mergeShadowSection(ObjectNode state, ObjectNode metadata, String section, JsonNode patch, long timestamp) {if (patch.isMissingNode()) {@@ -2098,13 +2099,41 @@ public class IotService {}ObjectNode stateSection = state.withObject("/" + section);ObjectNode metadataSection = metadata.withObject("/" + section);+ mergeShadowObject(stateSection, metadataSection, patch, timestamp);+ if (stateSection.isEmpty()) {+ state.remove(section);+ metadata.remove(section);+ }+ }++ /**+ * Merges an update object into a stored object and its metadata key by key, as AWS does. A null+ * removes the key and its metadata. An object onto a stored object merges into it at any depth, and+ * one emptied that way stays as {@code {}}. An object onto a missing key adds its non-null values, so+ * one holding only nulls adds nothing while {@code {}} adds {@code {}}. Anything else, an object onto a+ * scalar or an array included, replaces the stored value as sent, null leaves and all, and stamps+ * each of its leaves.+ */+ private void mergeShadowObject(ObjectNode state, ObjectNode metadata, JsonNode patch, long timestamp) {patch.fields().forEachRemaining(entry -> {- if (entry.getValue().isNull()) {- stateSection.remove(entry.getKey());- metadataSection.remove(entry.getKey());+ String key = entry.getKey();+ JsonNode value = entry.getValue();+ JsonNode stored = state.get(key);+ if (value.isNull()) {+ state.remove(key);+ metadata.remove(key);+ } else if (value.isObject() && (stored == null || stored.isObject())) {+ ObjectNode storedObject = stored instanceof ObjectNode existing ? existing : objectMapper.createObjectNode();+ ObjectNode storedMetadata = metadata.get(key) instanceof ObjectNode existingMetadata+ ? existingMetadata : objectMapper.createObjectNode();+ mergeShadowObject(storedObject, storedMetadata, value, timestamp);+ if (stored != null || !storedObject.isEmpty() || value.isEmpty()) {+ state.set(key, storedObject);+ metadata.set(key, storedMetadata);+ }} else {- stateSection.set(entry.getKey(), entry.getValue());- metadataSection.set(entry.getKey(), shadowMetadata(entry.getValue(), timestamp));+ state.set(key, value);+ metadata.set(key, shadowMetadata(value, timestamp));}});}diff --git a/src/test/java/io/github/hectorvent/floci/services/iot/IotDataIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/iot/IotDataIntegrationTest.javaindex 9ff3d64e05..50848ba7b2 100644--- a/src/test/java/io/github/hectorvent/floci/services/iot/IotDataIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/iot/IotDataIntegrationTest.java@@ -3,9 +3,13 @@ package io.github.hectorvent.floci.services.iot;import io.quarkus.test.junit.QuarkusTest;import org.junit.jupiter.api.Test;+import java.util.List;+import static io.restassured.RestAssured.given;import static org.hamcrest.Matchers.equalTo;import static org.hamcrest.Matchers.hasItem;+import static org.hamcrest.Matchers.hasKey;+import static org.hamcrest.Matchers.not;import static org.hamcrest.Matchers.nullValue;@QuarkusTest@@ -258,4 +262,74 @@ class IotDataIntegrationTest {.body("version", equalTo(1)).body("state.desired.mode", equalTo("auto"));}++ @Test+ void nestedShadowUpdateMergesKeyByKey() {+ int first = given()+ .contentType("application/json")+ .body("{\"state\":{\"reported\":{\"sensors\":{\"IDT_O1\":22.3,\"ODT\":-14.1},\"cfg\":{\"modes\":[\"eco\",\"boost\"]}}}}")+ .when()+ .post("/things/teos-merge-shadow/shadow")+ .then()+ .statusCode(200)+ .body("version", equalTo(1))+ .extract().path("timestamp");++ int second = given()+ .contentType("application/json")+ .body("{\"state\":{\"reported\":{\"sensors\":{\"IDT_O1\":26.3},\"tz\":\"Europe/Stockholm\"}}}")+ .when()+ .post("/things/teos-merge-shadow/shadow")+ .then()+ .statusCode(200)+ .body("state.reported.sensors", not(hasKey("ODT")))+ .body("version", equalTo(2))+ .extract().path("timestamp");++ given()+ .when()+ .get("/things/teos-merge-shadow/shadow")+ .then()+ .statusCode(200)+ .body("state.reported.sensors.ODT", equalTo(-14.1f))+ .body("state.reported.sensors.IDT_O1", equalTo(26.3f))+ .body("state.reported.tz", equalTo("Europe/Stockholm"))+ .body("state.reported.cfg.modes", equalTo(List.of("eco", "boost")))+ .body("metadata.reported.sensors.ODT.timestamp", equalTo(first))+ .body("metadata.reported.sensors.IDT_O1.timestamp", equalTo(second))+ .body("metadata.reported.tz.timestamp", equalTo(second))+ .body("version", equalTo(2));++ given()+ .contentType("application/json")+ .body("{\"version\":2,\"state\":{\"reported\":{\"sensors\":{\"ODT\":null},\"cfg\":{\"modes\":[\"off\"]}}}}")+ .when()+ .post("/things/teos-merge-shadow/shadow")+ .then()+ .statusCode(200)+ .body("version", equalTo(3));++ given()+ .contentType("application/json")+ .body("{\"version\":2,\"state\":{\"reported\":{\"tz\":null}}}")+ .when()+ .post("/things/teos-merge-shadow/shadow")+ .then()+ .statusCode(409)+ .header("x-amzn-ErrorType", equalTo("ConflictException"))+ .body("__type", equalTo("ConflictException"));++ given()+ .when()+ .get("/things/teos-merge-shadow/shadow")+ .then()+ .statusCode(200)+ .body("state.reported.sensors", not(hasKey("ODT")))+ .body("metadata.reported.sensors", not(hasKey("ODT")))+ .body("state.reported.sensors.IDT_O1", equalTo(26.3f))+ .body("state.reported.cfg.modes", equalTo(List.of("off")))+ .body("metadata.reported.cfg.modes.size()", equalTo(1))+ .body("state.reported.tz", equalTo("Europe/Stockholm"))+ .body("version", equalTo(3));+ }}diff --git a/src/test/java/io/github/hectorvent/floci/services/iot/IotServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/iot/IotServiceTest.javaindex a4b684b522..c1416510a7 100644--- a/src/test/java/io/github/hectorvent/floci/services/iot/IotServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/iot/IotServiceTest.java@@ -91,6 +91,13 @@ class IotServiceTest {"reported": {"lights": {"color": {"r": 255, "g": 0, "b": 255}, "on": true}, "arr": [1, 3], "same": {"x": 1}}}}""";private static final String NESTED_DELTA = "{\"lights\":{\"color\":{\"g\":255}},\"arr\":[1,2]}";+ private static final String TEOS_STORED_SENSORS = """+ {"state": {"reported": {"sensors": {"IDT_O1": 22.3, "ODT": -14.1}}},+ "metadata": {"reported": {"sensors": {"IDT_O1": {"timestamp": 1000}, "ODT": {"timestamp": 1000}}}},+ "version": 1, "timestamp": 1000}+ """;+ private static final String TEOS_SENSOR_UPDATE =+ "{\"state\":{\"reported\":{\"sensors\":{\"IDT_O1\":26.3},\"tz\":\"Europe/Stockholm\"}}}";private final ObjectMapper mapper = new ObjectMapper();private final SqsService sqs = mock(SqsService.class);@@ -958,6 +965,181 @@ class IotServiceTest {assertEquals(json("{\"desired\":{\"color\":{\"timestamp\":1000}}}"), reparse(stored.get("metadata")));}+ /** The documents current and the stored shadow after TEOS_SENSOR_UPDATE onto TEOS_STORED_SENSORS. */+ private void assertTeosSensorUpdateMerged(String thingName, long timestamp) throws Exception {+ JsonNode current = json("""+ {"state": {"reported": {"sensors": {"IDT_O1": 26.3, "ODT": -14.1}, "tz": "Europe/Stockholm"}},+ "metadata": {"reported": {"sensors": {"IDT_O1": %1$s, "ODT": {"timestamp": 1000}}, "tz": %1$s}},+ "version": 2}+ """.formatted(leaf(timestamp)));+ assertEquals(current, recordedPayload("$aws/things/" + thingName + "/shadow/update/documents").get("current"));+ JsonNode shadow = service.getThingShadow(thingName, null, REGION);+ assertEquals(current.get("state"), reparse(shadow.get("state")));+ assertEquals(current.get("metadata"), reparse(shadow.get("metadata")));+ assertEquals(2, shadow.get("version").asLong());+ }++ @Test+ void aNestedUpdateMergesKeyByKeyAndKeepsTheTimestampOfEveryUntouchedLeaf() throws Exception {+ storeShadow("teos-1", TEOS_STORED_SENSORS);++ JsonNode accepted = service.updateThingShadow("teos-1", null, json(TEOS_SENSOR_UPDATE), REGION);++ assertEquals(json(TEOS_SENSOR_UPDATE).get("state"), accepted.get("state"));+ assertTeosSensorUpdateMerged("teos-1", accepted.get("timestamp").asLong());+ }++ @Test+ void anMqttNestedUpdateMergesKeyByKeyAsTheRestUpdateDoes() throws Exception {+ storeShadow("teos-2", TEOS_STORED_SENSORS);++ mqttShadowRequest("$aws/things/teos-2/shadow/update", TEOS_SENSOR_UPDATE);++ long timestamp = recordedPayload("$aws/things/teos-2/shadow/update/accepted").get("timestamp").asLong();+ assertTeosSensorUpdateMerged("teos-2", timestamp);+ }++ @Test+ void aNestedNullRemovesTheLeafAndItsMetadataAndAnEmptiedObjectStays() throws Exception {+ storeShadow("teos-3", TEOS_STORED_SENSORS);++ service.updateThingShadow("teos-3", null, json("{\"state\":{\"reported\":{\"sensors\":{\"ODT\":null}}}}"), REGION);++ JsonNode afterOne = service.getThingShadow("teos-3", null, REGION);+ assertEquals(json("{\"reported\":{\"sensors\":{\"IDT_O1\":22.3}}}"), reparse(afterOne.get("state")));+ assertEquals(json("{\"reported\":{\"sensors\":{\"IDT_O1\":{\"timestamp\":1000}}}}"), reparse(afterOne.get("metadata")));++ service.updateThingShadow("teos-3", null, json("{\"state\":{\"reported\":{\"sensors\":{\"IDT_O1\":null}}}}"), REGION);++ JsonNode afterBoth = service.getThingShadow("teos-3", null, REGION);+ assertEquals(json("{\"reported\":{\"sensors\":{}}}"), reparse(afterBoth.get("state")));+ assertEquals(json("{\"reported\":{\"sensors\":{}}}"), reparse(afterBoth.get("metadata")));+ }++ @Test+ void anArrayReplacesTheStoredArrayWholeWithFreshMetadata() throws Exception {+ JsonNode first = service.updateThingShadow("teos-4", null, json("""+ {"state": {"reported": {"arr": [1, 2, 3], "cfg": {"objs": [{"a": 1}, {"b": 2}], "keep": 1}}}}+ """), REGION);++ JsonNode second = service.updateThingShadow("teos-4", null,+ json("{\"state\":{\"reported\":{\"arr\":[4],\"cfg\":{\"objs\":[{\"c\":3}]}}}}"), REGION);++ long timestamp = second.get("timestamp").asLong();+ JsonNode shadow = service.getThingShadow("teos-4", null, REGION);+ assertEquals(json("{\"reported\":{\"arr\":[4],\"cfg\":{\"objs\":[{\"c\":3}],\"keep\":1}}}"), reparse(shadow.get("state")));+ assertEquals(json("{\"reported\":{\"arr\":[%1$s],\"cfg\":{\"objs\":[{\"c\":%1$s}],\"keep\":%2$s}}}"+ .formatted(leaf(timestamp), leaf(first.get("timestamp").asLong()))), reparse(shadow.get("metadata")));+ }++ @Test+ void anObjectOntoAScalarOrAnArrayReplacesItAsSentNullsIncluded() throws Exception {+ storeShadow("teos-5", """+ {"state": {"reported": {"cfg": {"s": 5, "arr": [1], "keep": 1}}},+ "metadata": {"reported": {"cfg": {"s": {"timestamp": 1000}, "arr": [{"timestamp": 1000}], "keep": {"timestamp": 1000}}}},+ "version": 1, "timestamp": 1000}+ """);++ JsonNode accepted = service.updateThingShadow("teos-5", null,+ json("{\"state\":{\"reported\":{\"cfg\":{\"s\":{\"a\":null},\"arr\":{\"a\":null}}}}}"), REGION);++ long timestamp = accepted.get("timestamp").asLong();+ JsonNode shadow = service.getThingShadow("teos-5", null, REGION);+ assertEquals(json("{\"reported\":{\"cfg\":{\"s\":{\"a\":null},\"arr\":{\"a\":null},\"keep\":1}}}"),+ reparse(shadow.get("state")));+ assertEquals(json("{\"reported\":{\"cfg\":{\"s\":{\"a\":%1$s},\"arr\":{\"a\":%1$s},\"keep\":{\"timestamp\":1000}}}}"+ .formatted(leaf(timestamp))), reparse(shadow.get("metadata")));+ }++ @Test+ void anObjectOntoAMissingKeyAddsOnlyItsValuesAndNullsAloneAddNothing() throws Exception {+ storeShadow("teos-6", """+ {"state": {"reported": {"keep": 1}}, "metadata": {"reported": {"keep": {"timestamp": 1000}}},+ "version": 1, "timestamp": 1000}+ """);++ JsonNode accepted = service.updateThingShadow("teos-6", null, json("""+ {"state": {"reported": {"n": {"a": null, "b": 1}, "m": {"a": null}, "t": {"u": {"v": null}},+ "t2": {"u": {}}, "w": {"a": null, "b": {}}, "e": {}, "missing": null}}}+ """), REGION);++ long timestamp = accepted.get("timestamp").asLong();+ JsonNode shadow = service.getThingShadow("teos-6", null, REGION);+ assertEquals(json("{\"reported\":{\"keep\":1,\"n\":{\"b\":1},\"t2\":{\"u\":{}},\"w\":{\"b\":{}},\"e\":{}}}"),+ reparse(shadow.get("state")));+ assertEquals(json("{\"reported\":{\"keep\":{\"timestamp\":1000},\"n\":{\"b\":%s},\"t2\":{\"u\":{}},\"w\":{\"b\":{}},\"e\":{}}}"+ .formatted(leaf(timestamp))), reparse(shadow.get("metadata")));+ }++ @Test+ void anEmptyObjectOntoAStoredObjectChangesNothingButTheVersion() throws Exception {+ storeShadow("teos-7", """+ {"state": {"reported": {"x": {"y": 2}}}, "metadata": {"reported": {"x": {"y": {"timestamp": 1000}}}},+ "version": 1, "timestamp": 1000}+ """);++ service.updateThingShadow("teos-7", null, json("{\"state\":{\"reported\":{\"x\":{}}}}"), REGION);++ JsonNode shadow = service.getThingShadow("teos-7", null, REGION);+ assertEquals(json("{\"reported\":{\"x\":{\"y\":2}}}"), reparse(shadow.get("state")));+ assertEquals(json("{\"reported\":{\"x\":{\"y\":{\"timestamp\":1000}}}}"), reparse(shadow.get("metadata")));+ assertEquals(2, shadow.get("version").asLong());+ }++ @Test+ void aSectionSentEmptyOrLeftEmptyIsNotStored() throws Exception {+ service.updateThingShadow("teos-8", null, json("{\"state\":{\"desired\":{}}}"), REGION);++ JsonNode empty = service.getThingShadow("teos-8", null, REGION);+ assertEquals(json("{}"), reparse(empty.get("state")));+ assertEquals(json("{}"), reparse(empty.get("metadata")));++ service.updateThingShadow("teos-8", null, json("{\"state\":{\"desired\":{\"b\":1},\"reported\":{\"a\":1}}}"), REGION);+ recorder.clear();+ service.updateThingShadow("teos-8", null, json("{\"state\":{\"reported\":{\"a\":null}}}"), REGION);++ JsonNode current = recordedPayload("$aws/things/teos-8/shadow/update/documents").get("current");+ assertEquals(json("{\"desired\":{\"b\":1}}"), current.get("state"));+ assertEquals(Set.of("desired"), fieldNames(current.get("metadata")));++ recorder.clear();+ service.updateThingShadow("teos-8", null, json("{\"state\":{\"desired\":{\"b\":null}}}"), REGION);++ assertEquals(json("{\"state\":{},\"metadata\":{},\"version\":4}"),+ recordedPayload("$aws/things/teos-8/shadow/update/documents").get("current"));+ }++ @Test+ void aNestedDesiredChangeGivesTheDeltaAndDocumentsOfTheMergedShadow() throws Exception {+ String stored = """+ {"state": {"desired": {"lights": {"color": {"r": 1, "g": 2}}}, "reported": {"lights": {"color": {"r": 1, "g": 3}}}},+ "metadata": {"desired": {"lights": {"color": {"r": {"timestamp": 1000}, "g": {"timestamp": 1000}}}},+ "reported": {"lights": {"color": {"r": {"timestamp": 1000}, "g": {"timestamp": 1000}}}}},+ "version": 5, "timestamp": 1000}+ """;+ storeShadow("teos-9", stored);++ JsonNode accepted = service.updateThingShadow("teos-9", null,+ json("{\"state\":{\"desired\":{\"lights\":{\"color\":{\"b\":9}}}}}"), REGION);++ long timestamp = accepted.get("timestamp").asLong();+ JsonNode delta = recordedPayload("$aws/things/teos-9/shadow/update/delta");+ assertEquals(json("{\"lights\":{\"color\":{\"b\":9,\"g\":2}}}"), delta.get("state"));+ assertEquals(json("{\"lights\":{\"color\":{\"b\":%s,\"g\":{\"timestamp\":1000}}}}".formatted(leaf(timestamp))),+ delta.get("metadata"));+ assertEquals(6, delta.get("version").asLong());+ JsonNode documents = recordedPayload("$aws/things/teos-9/shadow/update/documents");+ JsonNode previous = json(stored);+ ((ObjectNode) previous).remove("timestamp");+ assertEquals(previous, documents.get("previous"));+ assertEquals(json("""+ {"state": {"desired": {"lights": {"color": {"r": 1, "g": 2, "b": 9}}}, "reported": {"lights": {"color": {"r": 1, "g": 3}}}},+ "metadata": {"desired": {"lights": {"color": {"r": {"timestamp": 1000}, "g": {"timestamp": 1000}, "b": %s}}},+ "reported": {"lights": {"color": {"r": {"timestamp": 1000}, "g": {"timestamp": 1000}}}}},+ "version": 6}+ """.formatted(leaf(timestamp))), documents.get("current"));+ }+@Testvoid aStoredShadowWithoutMetadataIsUpdatedWithMetadataFromItsStoredTimestamp() throws Exception {storeShadow("sensor-3", "{\"state\":{\"desired\":{\"color\":\"red\"}},\"version\":2,\"timestamp\":1000}");