Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
feat(ddb): add ddb replica (#1924)
Nahuel Nucerab4232ed7174a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 71195749..cdefdcf9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,19 +7,9 @@ Versioning follows [Semantic Versioning](https://semver.org/).## [Unreleased]-### Changed--- **Bedrock AgentCore — runtime version history and endpoint pinning** — runtime updates now retain version snapshots, `GetAgentRuntime` retrieves a selected version, and `ListAgentRuntimeVersions` paginates the history. `DEFAULT` advances with the latest version while named endpoints stay pinned until updated; invocations use the selected snapshot, with containers isolated by version. Contributed by @pingedbrain.-- **CloudFormation — `AWS::Lambda::Version` updates `FunctionScalingConfig` in place** — a stack update re-ran the create, which published a new version for every change; a `FunctionScalingConfig` change now keeps the version, also when the update rolls back. Contributed by @iot-rocket.-- **botocore 1.43.106** — the service models MiniStack reads move from 1.43.63 to 1.43.106; the images keep `awscli` 1.45.63, installed on the same botocore instead of its pinned one.--### Fixed--- **SES v2 — `ListEmailIdentities` and `ListConfigurationSets` answer the routes newer SDKs use** — botocore 1.43.106 sends them as `POST /v2/email/list-identities` and `POST /v2/email/list-configuration-sets` with `NextToken`, `PageSize` and `Filter` in the body; those paths answered `NotFoundException`. Both forms page, and the `Filter` keys are applied.-- **Kinesis — `ApproximateArrivalTimestamp` keeps milliseconds** — it was truncated to whole seconds, so an `AT_TIMESTAMP` iterator from an SDK that sends fractional seconds skipped records written earlier in the same second.-- **CloudFormation — an empty `Capabilities` list is accepted** — botocore sends it as a bare `Capabilities=`, which was read as one empty value and refused, so `aws cloudformation deploy` without `--capabilities` and `Capabilities=[]` from an SDK failed with a `ValidationError` since 1.5.11.### Added+- **DynamoDB — global table replicas through `UpdateTable` `ReplicaUpdates`** — a `Create` was accepted and ignored, so `DescribeTable` never listed the replica and Terraform / OpenTofu `aws_dynamodb_table` with a `replica` block waited forever for it. A `Create` now copies the table and its items into the other region, enables `NEW_AND_OLD_IMAGES` streams if they were off, and syncs TTL; each region's `DescribeTable` reports `GlobalTableVersion` `2019.11.21` and the other regions under `Replicas` as `ACTIVE`; writes in any region replicate to the rest; `Delete` removes the replica. A `ReplicaUpdates` entry for the table's own region, or a `Create` into a region that already has the table, fails with the AWS `ValidationException` message. Reported by @wparad.- **IoT — registry events** — with a type enabled through `UpdateEventConfigurations`, the thing, thing type, thing type association, thing group, thing group hierarchy and thing group membership operations publish the AWS payload to `$aws/events/...`, where MQTT subscribers and topic rules receive it. Contributed by @iot-rocket.- **ElastiCache — serverless caches** — `CreateServerlessCache`, `DescribeServerlessCaches`, `ModifyServerlessCache` and `DeleteServerlessCache` for `valkey` (major 7, 8 or 9) and `redis` (major 7), with tags on the `serverlesscache` ARN. The name is stored lowercase. With Docker each cache gets a TLS-only Valkey or Redis container, `creating` until it answers a TLS `PING`; its certificate chains to the CA at `GET /_ministack/elasticache/ca.pem`. `ModifyServerlessCache` accepts a same-engine major upgrade, redis to valkey, and valkey 7 to redis 7, and rejects other engine changes with `InvalidParameterCombination`. Memcached serverless caches and serverless snapshots are refused with `InvalidParameterValue`; without Docker the endpoint is the plaintext Redis sidecar. Contributed by @skialpine.- **CloudFormation — `AWS::Lambda::Version` updates `FunctionScalingConfig` in place** — a stack update re-ran the create, which published a new version for every change; a `FunctionScalingConfig` change now keeps the version, also when the update rolls back. Contributed by @iot-rocket.@@ -28,8 +18,17 @@ Versioning follows [Semantic Versioning](https://semver.org/).- **IoT — `DescribeEventConfigurations` and `UpdateEventConfigurations`** — both answered `Unsupported IoT path`; they now store the registry event switches per account and region, every type starting disabled, an update changing only the types it names, and `creationDate` / `lastModifiedDate` set from the first update on, as on AWS. Contributed by @iot-rocket.- **CloudFormation — `AWS::ServiceDiscovery::HttpNamespace`, `PrivateDnsNamespace`, `PublicDnsNamespace`, `Service` and `Instance`** — a template declaring any of them, as the CDK ECS constructs do for Cloud Map, failed validation with `Unrecognized resource types`. They now create, update and delete through the Cloud Map service, so a DNS namespace from a stack gets its Route 53 hosted zone. `Ref` and `Fn::GetAtt` return the values in the CloudFormation reference; Description, the SOA TTL, a service's `DnsConfig`, `HealthCheckConfig` and `ServiceAttributes`, and instance attributes change in place, while the properties the reference marks as replacing (including `HealthCheckCustomConfig`) replace the resource. Two Cloud Map API changes come with this: namespaces store and update `Properties.DnsProperties.SOA.TTL`, and `DeleteNamespace` removes the namespace's hosted zone, as on AWS. Contributed by @fabio-andre-rodrigues.+### Changed++- **Bedrock AgentCore — runtime version history and endpoint pinning** — runtime updates now retain version snapshots, `GetAgentRuntime` retrieves a selected version, and `ListAgentRuntimeVersions` paginates the history. `DEFAULT` advances with the latest version while named endpoints stay pinned until updated; invocations use the selected snapshot, with containers isolated by version. Contributed by @pingedbrain.+- **CloudFormation — `AWS::Lambda::Version` updates `FunctionScalingConfig` in place** — a stack update re-ran the create, which published a new version for every change; a `FunctionScalingConfig` change now keeps the version, also when the update rolls back. Contributed by @iot-rocket.+- **botocore 1.43.106** — the service models MiniStack reads move from 1.43.63 to 1.43.106; the images keep `awscli` 1.45.63, installed on the same botocore instead of its pinned one.+### Fixed+- **SES v2 — `ListEmailIdentities` and `ListConfigurationSets` answer the routes newer SDKs use** — botocore 1.43.106 sends them as `POST /v2/email/list-identities` and `POST /v2/email/list-configuration-sets` with `NextToken`, `PageSize` and `Filter` in the body; those paths answered `NotFoundException`. Both forms page, and the `Filter` keys are applied.+- **Kinesis — `ApproximateArrivalTimestamp` keeps milliseconds** — it was truncated to whole seconds, so an `AT_TIMESTAMP` iterator from an SDK that sends fractional seconds skipped records written earlier in the same second.+- **CloudFormation — an empty `Capabilities` list is accepted** — botocore sends it as a bare `Capabilities=`, which was read as one empty value and refused, so `aws cloudformation deploy` without `--capabilities` and `Capabilities=[]` from an SDK failed with a `ValidationError` since 1.5.11.- **API Gateway v2 (HTTP API) — the request path keeps a `%25` escape** — `/items/a%252Eb` reached the Lambda's `rawPath` as `/items/a%2Eb`, because the path was fully percent-decoded. The HTTP API path now keeps `%25` and decodes every other escape as before; `rawPath`, `requestContext.http.path`, `pathParameters` and route matching all use it. Contributed by @skialpine.- **Step Functions — JSONPath `$$.` paths read the context object** — a Choice rule's `Variable` and its variable-to-variable comparison paths (including inside `And`, `Or` and `Not`), and a state's `InputPath`, `OutputPath` and Map `ItemsPath`, resolved `$$.` against the state input, so a Choice on `$$.Execution.Input` silently took its default branch. They now read the context object, as the AWS context object reference lists for those fields. Contributed by @jayjanssen.- **CloudFormation — `AWS::EC2::VPCGatewayAttachment` updates in place** — a changed `InternetGatewayId` or `VpnGatewayId` moves the attachment under the same `IGW|vpc-…` / `VGW|vpc-…` physical id, `VpnGatewayId` is attached at all, and a `VpcId` change leaves the gateway attached to the new VPC, or to the old one when the update rolls back. `AttachInternetGateway` on a gateway already attached to a VPC answers `Resource.AlreadyAssociated`, and a stack attaching it to a second VPC leaves it on the first, as AWS does. Contributed by @iot-rocket.diff --git a/ministack/services/dynamodb.py b/ministack/services/dynamodb.pyindex cdc6bf76..a376a8da 100644--- a/ministack/services/dynamodb.py+++ b/ministack/services/dynamodb.py@@ -42,6 +42,7 @@ from ministack.core.responses import (json_response,new_uuid,now_iso,+ request_scope,)from ministack.services._dynamodb_keywords import AWS_KEYWORDS@@ -708,7 +709,8 @@ def _build_change_record(table: dict, event_name: str, old_item: dict | None, nereturn record-def _emit_stream_event(table_name: str, event_name: str, old_item: dict | None, new_item: dict | None):+def _emit_stream_event(table_name: str, event_name: str, old_item: dict | None, new_item: dict | None,+ replicate: bool = True):"""Emit a change to DynamoDB Streams (if enabled) and to any ACTIVE Kinesisstreaming destinations registered for this table.@@ -719,6 +721,8 @@ def _emit_stream_event(table_name: str, event_name: str, old_item: dict | None,table = _tables.get(table_name)if not table:return+ if replicate:+ _replicate_write(table, event_name, old_item, new_item)spec = table.get("StreamSpecification") or {}streams_enabled = bool(spec.get("StreamEnabled"))@@ -1267,7 +1271,10 @@ def _delete_table(data):"Table is protected against deletion. To delete the table, disable deletion protection.", 400)desc = _table_description(name)desc["TableStatus"] = "DELETING"+ remaining = [r for r in _replica_group(_tables[name]) if r != get_region()]del _tables[name]+ if remaining:+ _set_replica_group(name, remaining)_tags.pop(desc.get("TableArn", ""), None)_ttl_settings.pop(name, None)_pitr_settings.pop(name, None)@@ -1296,6 +1303,115 @@ def _list_tables(data):return json_response(result)+_GLOBAL_TABLE_VERSION = "2019.11.21"+++def _replica_group(table) -> list:+ """Every region of the global table ``table`` belongs to, its own included."""+ return table.get("_global_regions") or []+++def _set_replica_group(name, regions):+ account = get_account_id()+ for region in regions:+ member = _tables.get_scoped(account, region, name)+ if member is not None:+ member["_global_regions"] = sorted(regions) if len(regions) > 1 else []+++def _create_replica(name, table, region):+ """Copy ``table`` into ``region`` as a replica, with its items."""+ account = get_account_id()+ replica = copy.deepcopy({k: v for k, v in table.items() if k != "items"})+ replica["items"] = defaultdict(dict, copy.deepcopy(dict(table["items"])))+ arn = f"arn:aws:dynamodb:{region}:{account}:table/{name}"+ replica.update({+ "TableArn": arn, "TableId": new_uuid(), "CreationDateTime": int(time.time()),+ "TableStatus": "ACTIVE", "DeletionProtectionEnabled": False,+ })+ for index in replica.get("GlobalSecondaryIndexes", []) + replica.get("LocalSecondaryIndexes", []):+ index["IndexArn"] = f"{arn}/index/{index['IndexName']}"+ label = _stream_label()+ replica["LatestStreamLabel"] = label+ replica["LatestStreamArn"] = f"{arn}/stream/{label}"+ _tables.set_scoped(account, region, name, replica)+ ttl = _ttl_settings.get(name)+ if ttl:+ _ttl_settings.set_scoped(account, region, name, copy.deepcopy(ttl))+++def _apply_replica_updates(name, table, updates):+ """UpdateTable ReplicaUpdates (global tables version 2019.11.21)."""+ account, home = get_account_id(), get_region()+ regions = set(_replica_group(table)) or {home}+ for update in updates:+ action = next(iter(update), None)+ region = (update.get(action) or {}).get("RegionName", "")+ if region == home:+ return error_response_json("ValidationException",+ "Cannot add, delete, or update the local region through ReplicaUpdates. "+ "Use CreateTable, DeleteTable, or UpdateTable as required.", 400)+ if action == "Create":+ if _tables.get_scoped(account, region, name) is not None:+ return error_response_json("ValidationException",+ f"Failed to create a the new replica of table with name: '{name}' "+ "because one or more replicas already existed as tables.", 400)+ # MREC replicates through Streams, so they are on for every replica.+ if not (table.get("StreamSpecification") or {}).get("StreamEnabled"):+ table["StreamSpecification"] = {"StreamEnabled": True, "StreamViewType": "NEW_AND_OLD_IMAGES"}+ label = _stream_label()+ table["LatestStreamLabel"] = label+ table["LatestStreamArn"] = f"{table['TableArn']}/stream/{label}"+ _create_replica(name, table, region)+ regions.add(region)+ elif action in ("Update", "Delete"):+ if region not in regions:+ return error_response_json("ValidationException",+ "Replica specified in the Replica Update or Replica Delete action of the request was not found.", 400)+ if action == "Delete":+ _tables.pop_scoped(account, region, name, None)+ _ttl_settings.pop_scoped(account, region, name, None)+ _pitr_settings.pop_scoped(account, region, name, None)+ with request_scope(account, region):+ drop_stream_records(name)+ regions.discard(region)+ _set_replica_group(name, regions)+ if len(regions) <= 1:+ table["_global_regions"] = []+ return None+++def _replicate_write(table, event_name, old_item, new_item):+ """Apply a write to the table's other replicas, as MREC global tables do."""+ regions = _replica_group(table)+ if not regions:+ return+ account, home, name = get_account_id(), get_region(), table["TableName"]+ item = new_item if new_item is not None else old_item+ for region in regions:+ if region == home:+ continue+ with request_scope(account, region):+ replica = _tables.get(name)+ if replica is None:+ continue+ pk_val = _extract_key_val(item.get(replica["pk_name"]))+ sk_val = _extract_key_val(item.get(replica["sk_name"])) if replica["sk_name"] else "__no_sort__"+ previous = replica["items"].get(pk_val, {}).get(sk_val)+ if new_item is None:+ replica["items"].get(pk_val, {}).pop(sk_val, None)+ if pk_val in replica["items"] and not replica["items"][pk_val]:+ del replica["items"][pk_val]+ else:+ replica["items"][pk_val][sk_val] = copy.deepcopy(new_item)+ _update_counts(replica)+ if new_item is None and previous is None:+ continue+ replica_event = "REMOVE" if new_item is None else ("MODIFY" if previous else "INSERT")+ _emit_stream_event(name, replica_event, previous,+ copy.deepcopy(new_item) if new_item is not None else None, replicate=False)++def _update_table(data):name = _normalize_table_name(data.get("TableName"))if name not in _tables:@@ -1449,6 +1565,11 @@ def _update_table(data):if ad["AttributeName"] in referenced]+ if data.get("ReplicaUpdates"):+ err = _apply_replica_updates(name, table, data["ReplicaUpdates"])+ if err:+ return err+return json_response({"TableDescription": _table_description(name)})@@ -1483,6 +1604,10 @@ def _table_description(name):if t.get("OnDemandThroughput"):desc["OnDemandThroughput"] = t["OnDemandThroughput"]desc["DeletionProtectionEnabled"] = t.get("DeletionProtectionEnabled", False)+ if _replica_group(t):+ desc["GlobalTableVersion"] = _GLOBAL_TABLE_VERSION+ desc["Replicas"] = [{"RegionName": r, "ReplicaStatus": "ACTIVE"}+ for r in _replica_group(t) if r != get_region()]desc["WarmThroughput"] = t.get("WarmThroughput", {"ReadUnitsPerSecond": 0,"WriteUnitsPerSecond": 0,@@ -4449,6 +4574,10 @@ def _update_ttl(data):"TimeToLiveStatus": "ENABLED" if enabled else "DISABLED","AttributeName": attr_name,}+ # A global table's TTL settings are synchronized to every replica.+ for region in _replica_group(_tables[name]):+ if region != get_region():+ _ttl_settings.set_scoped(get_account_id(), region, name, dict(_ttl_settings[name]))return json_response({"TimeToLiveSpecification": spec})diff --git a/tests/test_dynamodb.py b/tests/test_dynamodb.pyindex ad0dbb41..ff7364e9 100644--- a/tests/test_dynamodb.py+++ b/tests/test_dynamodb.py@@ -7345,3 +7345,78 @@ def test_dynamodb_restore_rebuilds_item_store_for_every_account_and_region():assert body["Item"] == {"pk": {"S": "existing"}}finally:ddb_service.reset()+++def test_dynamodb_global_table_replica_lifecycle():+ home, other = _ddb_client("us-east-1"), _ddb_client("us-west-2")+ name = f"gt-{_uuid_mod.uuid4().hex[:8]}"+ home.create_table(+ TableName=name, BillingMode="PAY_PER_REQUEST",+ AttributeDefinitions=[{"AttributeName": "pk", "AttributeType": "S"}],+ KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}])+ home.put_item(TableName=name, Item={"pk": {"S": "before"}})+ home.update_time_to_live(TableName=name, TimeToLiveSpecification={"Enabled": True, "AttributeName": "ttl"})+ try:+ home.update_table(TableName=name, ReplicaUpdates=[{"Create": {"RegionName": "us-west-2"}}])++ table = home.describe_table(TableName=name)["Table"]+ assert table["GlobalTableVersion"] == "2019.11.21"+ assert table["Replicas"] == [{"RegionName": "us-west-2", "ReplicaStatus": "ACTIVE"}]+ assert table["StreamSpecification"] == {"StreamEnabled": True, "StreamViewType": "NEW_AND_OLD_IMAGES"}+ replica = other.describe_table(TableName=name)["Table"]+ assert replica["Replicas"] == [{"RegionName": "us-east-1", "ReplicaStatus": "ACTIVE"}]+ assert replica["TableArn"].startswith("arn:aws:dynamodb:us-west-2:")+ assert replica["DeletionProtectionEnabled"] is False+ assert other.describe_time_to_live(TableName=name)["TimeToLiveDescription"]["AttributeName"] == "ttl"++ # Existing items are copied; writes replicate both ways.+ assert other.get_item(TableName=name, Key={"pk": {"S": "before"}})["Item"]+ home.put_item(TableName=name, Item={"pk": {"S": "a"}, "v": {"N": "1"}})+ assert other.get_item(TableName=name, Key={"pk": {"S": "a"}})["Item"]["v"] == {"N": "1"}+ other.update_item(TableName=name, Key={"pk": {"S": "a"}}, UpdateExpression="SET v = :v",+ ExpressionAttributeValues={":v": {"N": "2"}})+ assert home.get_item(TableName=name, Key={"pk": {"S": "a"}})["Item"]["v"] == {"N": "2"}+ other.delete_item(TableName=name, Key={"pk": {"S": "a"}})+ assert "Item" not in home.get_item(TableName=name, Key={"pk": {"S": "a"}})++ home.update_table(TableName=name, ReplicaUpdates=[{"Delete": {"RegionName": "us-west-2"}}])+ with pytest.raises(ClientError) as exc:+ other.describe_table(TableName=name)+ assert exc.value.response["Error"]["Code"] == "ResourceNotFoundException"+ table = home.describe_table(TableName=name)["Table"]+ assert "Replicas" not in table and "GlobalTableVersion" not in table+ finally:+ for client in (other, home):+ with contextlib.suppress(ClientError):+ client.delete_table(TableName=name)+++def test_dynamodb_global_table_replica_update_errors():+ home, other = _ddb_client("us-east-1"), _ddb_client("us-west-2")+ name = f"gt-err-{_uuid_mod.uuid4().hex[:8]}"+ spec = dict(BillingMode="PAY_PER_REQUEST",+ AttributeDefinitions=[{"AttributeName": "pk", "AttributeType": "S"}],+ KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}])+ home.create_table(TableName=name, **spec)+ other.create_table(TableName=name, **spec)+ try:+ cases = [+ ([{"Create": {"RegionName": "us-east-1"}}],+ "Cannot add, delete, or update the local region through ReplicaUpdates. "+ "Use CreateTable, DeleteTable, or UpdateTable as required."),+ ([{"Create": {"RegionName": "us-west-2"}}],+ f"Failed to create a the new replica of table with name: '{name}' "+ "because one or more replicas already existed as tables."),+ ([{"Delete": {"RegionName": "eu-west-1"}}],+ "Replica specified in the Replica Update or Replica Delete action of the request was not found."),+ ]+ for updates, message in cases:+ with pytest.raises(ClientError) as exc:+ home.update_table(TableName=name, ReplicaUpdates=updates)+ assert exc.value.response["Error"]["Code"] == "ValidationException"+ assert exc.value.response["Error"]["Message"] == message+ assert "Replicas" not in home.describe_table(TableName=name)["Table"]+ finally:+ for client in (other, home):+ with contextlib.suppress(ClientError):+ client.delete_table(TableName=name)