Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

release v1.5.28 (#2225)

Nahuel Nucerac6eedf298651
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 56a36381..4da11f9d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,8 @@ Versioning follows [Semantic Versioning](https://semver.org/).
## [Unreleased]
+## [1.5.28] — 2026-10-16
+
### Added
- **RDS — `DescribeEvents` with reboot events** — `DescribeEvents` returns the `availability` event `DB instance restarted.` once a `RebootDBInstance` has finished, filtered by source, source type, category and time, paged with `Marker` and kept for 14 days; other RDS lifecycle events are not recorded yet. Contributed by @jayjanssen.
- **S3 Control — Multi-Region Access Points** — `CreateMultiRegionAccessPoint`, `DeleteMultiRegionAccessPoint`, `PutMultiRegionAccessPointPolicy`, `DescribeMultiRegionAccessPointOperation`, `GetMultiRegionAccessPoint`, `GetMultiRegionAccessPointPolicy`, `ListMultiRegionAccessPoints`, `GetMultiRegionAccessPointRoutes` and `SubmitMultiRegionAccessPointRoutes`. The async operations apply at once and `Describe` reports `SUCCEEDED`, or `FAILED` with AWS's `ErrorDetails` for a missing bucket, two buckets in one region or a name already in use. Management is served from `us-west-2` and route operations from `us-east-1`, `us-west-2`, `eu-west-1`, `ap-northeast-1` and `ap-southeast-2`, as on AWS; other regions get `301 PermanentRedirect`. `DeleteBucket` on a member returns `BucketHasAccessPointsAttached`. Under `AUTH=true` each operation needs its `s3:` IAM action and `x-amz-account-id` must name the caller's account. Access points from `AWS::S3::MultiRegionAccessPoint` appear in the API. Contributed by @fabio-andre-rodrigues.
@@ -78,7 +80,8 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- **IAM — caller and transport condition keys under `AUTH=true`** — identity policies saw none of the caller's keys and a fixed `127.0.0.1` over plain HTTP; they now see `aws:username`, `aws:userid`, `aws:PrincipalTag/*` (the user's or role's tags, `AssumeRole` session tags taking precedence; in `rds-db:connect` too), for temporary credentials `aws:MultiFactorAuthPresent` and `aws:TokenIssueTime`, the client's address as `aws:SourceIp` and `aws:SecureTransport` as `true` over HTTPS or behind a proxy that sends `X-Forwarded-Proto: https`, and `AssumedRoleId` starts with the role's `RoleId` with or without `AUTH`. Contributed by @iot-rocket.
- **S3 — requests without credentials are anonymous under `AUTH=true`** — an unsigned request (no `Authorization` header, presigned query or POST form credential) ran as the root key, so a private bucket in the caller's own account answered 200 to anyone; it is now anonymous, and only the owner's public grants let it in: an `AllUsers` ACL grant or a bucket policy that allows `*`, unless Public Access Block shuts them off. Anything else answers 403 `AccessDenied`. A `public-read` object ACL is now also found for an unversioned object, which the grant check missed for cross-account callers too. Without `AUTH` nothing changes. Contributed by @prandogabriel.
- **Cognito — `InboundFederation` `userName`** — the trigger received `<provider>_<NameID>` for SAML and `<provider>_<sub>` for OIDC; it now receives the bare SAML `NameID` or OIDC `sub`. Contributed by @kjdev.
-- **S3 Tables — `CreateTable` keeps Iceberg column types** — a `schema` or `schemaV2` field type other than `string`, `int`, `long`, `boolean`, `date`, `timestamp`, `float` or `double` was stored as `string`, and `timestamp` as `timestamptz`; types such as `decimal(19, 2)`, `uuid`, `fixed[16]` and nested `list`, `struct` and `map` are now stored as given. Reported by @anatollius.
+- **S3 Tables — `CreateTable` keeps Iceberg column types** — a `schema` or `schemaV2` field type other than `string`, `int`, `long`, `boolean`, `date`, `timestamp`, `float` or `double` was stored as `string`, and `timestamp` as `timestamptz`; every Iceberg v2 type, nested `list`, `struct` and `map` included, is now stored as AWS stores it (lowercase, `decimal(P, S)`), and any other type answers `BadRequestException` `The specified metadata is not valid`. Reported by @anatollius.
+- **S3 Tables — the Iceberg REST catalog takes the table bucket prefix** — a request without the `{prefix}` searched every table bucket, so two buckets with the same namespace and table name answered with the first one; such a request now answers 404 `<UnknownOperationException/>`, `GET /v1/config` without a table bucket ARN `warehouse` answers 400 `The specified table bucket ARN is not valid.`, and the url-encoded ARN comes back as `prefix` in `defaults`.
## [1.5.27] — 2026-10-13
diff --git a/ministack/services/cloudformation/provisioners.py b/ministack/services/cloudformation/provisioners.py
index 19a0b85a..cad6328f 100644
--- a/ministack/services/cloudformation/provisioners.py
+++ b/ministack/services/cloudformation/provisioners.py
@@ -12342,10 +12342,14 @@ def _s3tables_table_create(logical_id, props, stack_name):
schema_field_list = (
props.get("IcebergMetadata", {}).get("IcebergSchema", {}).get("SchemaFieldList", [])
)
- schema_fields = [
- {"name": f["Name"], "type": f.get("Type", "string"), "required": f.get("Required", False)}
- for f in schema_field_list
- ]
+ try:
+ schema_fields = [
+ {"name": f["Name"], "type": _s3tables._iceberg_field_type(f.get("Type", "string")),
+ "required": f.get("Required", False)}
+ for f in schema_field_list
+ ]
+ except ValueError:
+ raise ValueError("The specified metadata is not valid") from None
iceberg_metadata = _s3tables._initial_iceberg_metadata(table_name, schema_fields, location)
metadata_location = f"s3://{bucket_name}/{namespace}/{table_name}/metadata/v0.metadata.json"
table_arn = _s3tables._table_arn(bucket_arn, namespace, table_name)
diff --git a/ministack/services/s3tables.py b/ministack/services/s3tables.py
index db6e8741..232187a9 100644
--- a/ministack/services/s3tables.py
+++ b/ministack/services/s3tables.py
@@ -35,7 +35,7 @@ import logging
import os
import re
import time
-from urllib.parse import unquote
+from urllib.parse import quote, unquote
from ministack.core.arn import ArnParseError, parse_arn
from ministack.core.responses import (
@@ -186,6 +186,42 @@ def _set_bucket_region_value(store, bucket_arn, key, value):
store.set_scoped(spec.account_id, spec.region, key, value)
+_ICEBERG_V2_PRIMITIVES = {
+ "boolean", "int", "long", "float", "double", "date", "time", "timestamp", "timestamptz", "string", "uuid", "binary",
+}
+_DECIMAL_TYPE_RE = re.compile(r"decimal\(\s*(\d+)\s*,\s*(\d+)\s*\)")
+_FIXED_TYPE_RE = re.compile(r"fixed\[\s*(\d+)\s*\]")
+
+
+def _iceberg_field_type(kind):
+ """An Iceberg v2 type as the table metadata stores it; ValueError when invalid."""
+ if isinstance(kind, dict):
+ nested = dict(kind)
+ if kind.get("type") == "struct" and isinstance(kind.get("fields"), list):
+ nested["fields"] = [{**f, "type": _iceberg_field_type(f.get("type"))} for f in kind["fields"]]
+ elif kind.get("type") == "list" and "element" in kind:
+ nested["element"] = _iceberg_field_type(kind["element"])
+ elif kind.get("type") == "map" and "key" in kind and "value" in kind:
+ nested["key"] = _iceberg_field_type(kind["key"])
+ nested["value"] = _iceberg_field_type(kind["value"])
+ else:
+ raise ValueError(kind)
+ return nested
+ if not isinstance(kind, str):
+ raise ValueError(kind)
+ lowered = kind.strip().lower()
+ if lowered in _ICEBERG_V2_PRIMITIVES:
+ return lowered
+ if match := _DECIMAL_TYPE_RE.fullmatch(lowered):
+ precision, scale = int(match.group(1)), int(match.group(2))
+ if precision > 38:
+ raise ValueError(kind)
+ return f"decimal({precision}, {scale})"
+ if match := _FIXED_TYPE_RE.fullmatch(lowered):
+ return f"fixed[{int(match.group(1))}]"
+ raise ValueError(kind)
+
+
def _initial_iceberg_metadata(table_name, schema_fields, location):
table_uuid = new_uuid()
fields = []
@@ -372,8 +408,12 @@ def _create_table(bucket_arn, namespace, data):
metadata = data.get("metadata", {})
iceberg_meta = metadata.get("iceberg", {})
schema_def = iceberg_meta.get("schema", iceberg_meta.get("schemaV2", {}))
- for f in schema_def.get("field", schema_def.get("fields", [])):
- schema_fields.append({"name": f["name"], "type": f.get("type", "string"), "required": f.get("required", False)})
+ try:
+ for f in schema_def.get("field", schema_def.get("fields", [])):
+ schema_fields.append({"name": f["name"], "type": _iceberg_field_type(f.get("type", "string")),
+ "required": f.get("required", False)})
+ except ValueError:
+ return error_response_json("BadRequestException", "The specified metadata is not valid", 400)
logger.debug(
"S3Tables: create_table %s/%s schema_fields=%d metadata_keys=%s",
namespace,
@@ -396,7 +436,7 @@ def _create_table(bucket_arn, namespace, data):
{
"id": f.get("id", i + 1),
"name": f["name"],
- "type": f.get("type", "string"),
+ "type": schema_fields[i]["type"],
"required": f.get("required", False),
}
)
@@ -526,6 +566,9 @@ def _update_table_metadata_location(bucket_arn, namespace, table_name, data):
# ── Iceberg REST catalog (data plane for Spark) ───────────
+_TABLE_BUCKET_ARN_RE = re.compile(r"arn:aws[\w-]*:s3tables:[\w-]+:\d{12}:bucket/[a-z0-9-]+")
+
+
def _iceberg_config(query_params=None, headers=None):
warehouse = ""
if query_params:
@@ -545,8 +588,19 @@ def _iceberg_config(query_params=None, headers=None):
"s3.path-style-access": "true",
}
overrides = {}
- if warehouse:
+ if ":s3tablescatalog/" in warehouse:
overrides["prefix"] = warehouse
+ elif _TABLE_BUCKET_ARN_RE.fullmatch(warehouse):
+ defaults.update({
+ "write.object-storage.partitioned-paths": "false",
+ "s3.delete-enabled": "false",
+ "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
+ "write.object-storage.enabled": "true",
+ "prefix": quote(warehouse, safe=""),
+ "rest-metrics-reporting-enabled": "false",
+ })
+ else:
+ return _iceberg_error("The specified table bucket ARN is not valid.", "bad_request", 400)
return json_response({"defaults": defaults, "overrides": overrides})
@@ -1044,6 +1098,10 @@ def _iceberg_create_table(namespace, data, allow_cross_region, bucket_filter=Non
# ── Iceberg REST router ───────────────────────────────────
+def _unknown_operation():
+ return 404, {"Content-Type": "application/xml"}, b"<UnknownOperationException/>"
+
+
async def _handle_iceberg_request(method, path, headers, body, query_params):
parts = [p for p in path.strip("/").split("/") if p]
allow_cross_region = _iceberg_allows_cross_region(headers)
@@ -1062,7 +1120,9 @@ async def _handle_iceberg_request(method, path, headers, body, query_params):
if parts[1] == "v1" and method == "POST" and "transactions" in parts[2:]:
data = json.loads(body) if body else {}
tx_idx = parts.index("transactions", 2)
- tx_filter = _bucket_scope_from_prefix(unquote("/".join(parts[2:tx_idx]))) if tx_idx > 2 else None
+ if tx_idx == 2:
+ return _unknown_operation()
+ tx_filter = _bucket_scope_from_prefix(unquote("/".join(parts[2:tx_idx])))
for change in data.get("table-changes", []):
ident = change.get("identifier", {})
ns = ident.get("namespace", [""])
@@ -1076,18 +1136,8 @@ async def _handle_iceberg_request(method, path, headers, body, query_params):
if len(parts) < 3:
return None
- # Support two URL formats:
- # Standard Iceberg REST: /iceberg/v1/{prefix}/namespaces/...
- # S3 Tables (no prefix): /iceberg/v1/namespaces/... (warehouse in query param)
- #
- # The prefix carries the warehouse value from /v1/config, which for S3
- # Tables is ``{account}:s3tablescatalog/{bucket-name}``. Extract the
- # bucket name to scope table lookups -- without it, tables with the same
- # name in different buckets collide and the wrong schema is returned.
- # The prefix from /v1/config can contain ``/`` (e.g.
- # ``000000000000:s3tablescatalog/bucket-name``), which splits into
- # multiple URL segments. Scan for ``namespaces`` and treat everything
- # between ``v1`` and ``namespaces`` as the prefix.
+ # Every route but /v1/config takes the {prefix}, which "for S3 Tables is
+ # always your url-encoded table bucket ARN"; it can span several segments.
bucket_filter = None
ns_idx = None
for i in range(2, len(parts)):
@@ -1096,8 +1146,9 @@ async def _handle_iceberg_request(method, path, headers, body, query_params):
break
if ns_idx is None:
return None
- if ns_idx > 2:
- bucket_filter = _bucket_scope_from_prefix(unquote("/".join(parts[2:ns_idx])))
+ if ns_idx == 2:
+ return _unknown_operation()
+ bucket_filter = _bucket_scope_from_prefix(unquote("/".join(parts[2:ns_idx])))
rest = parts[ns_idx + 1 :] # segments after "namespaces"
diff --git a/pyproject.toml b/pyproject.toml
index d467664a..2628f447 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "ministack"
-version = "1.5.27"
+version = "1.5.28"
description = "Free, open-source local AWS emulator — drop-in LocalStack replacement"
readme = "README.md"
license = { file = "LICENSE" }
diff --git a/tests/test_cfn.py b/tests/test_cfn.py
index 6aefd4a8..d7750b93 100644
--- a/tests/test_cfn.py
+++ b/tests/test_cfn.py
@@ -18523,7 +18523,7 @@ def test_cfn_s3tables_table_schema_from_iceberg_metadata(cfn, s3tables):
stack = _wait_stack(cfn, stack_name)
assert stack["StackStatus"] == "CREATE_COMPLETE", stack.get("StackStatusReason")
- resp = _cfn_iceberg_json("/iceberg/v1/namespaces/myns/tables/mytable")
+ resp = _cfn_iceberg_json("/iceberg/v1/arn%3Aaws%3As3tables%3Aus-east-1%3A000000000000%3Abucket%2Fcfn-s3tables-schema-test/namespaces/myns/tables/mytable")
fields = resp.get("metadata", {}).get("schemas", [{}])[0].get("fields", [])
field_names = {f["name"] for f in fields}
assert field_names == {"id", "value"}, f"expected columns id/value, got {field_names}"
diff --git a/tests/test_s3tables.py b/tests/test_s3tables.py
index 563f5430..78b4f960 100644
--- a/tests/test_s3tables.py
+++ b/tests/test_s3tables.py
@@ -495,62 +495,54 @@ def test_iceberg_rest_error_uses_spec_envelope():
assert body["error"]["message"]
-def test_s3tables_iceberg_catalog_no_prefix_url_format(s3tables):
- """S3 Tables uses /iceberg/v1/namespaces/... (no catalog prefix in path,
- warehouse in query param) — the format DuckDB sends with ENDPOINT_TYPE s3_tables
- or an explicit ENDPOINT pointing at the catalog root."""
- bucket_name = f"tb-noprefix-{_uuid_mod.uuid4().hex[:6]}"
- bucket_arn = s3tables.create_table_bucket(name=bucket_name)["arn"]
+def test_s3tables_iceberg_routes_without_prefix_are_unknown(s3tables):
+ """Every route but /v1/config takes the table bucket ARN prefix; real AWS
+ answers one without it 404 <UnknownOperationException/>."""
+ bucket_arn = s3tables.create_table_bucket(name=f"tb-noprefix-{_uuid_mod.uuid4().hex[:6]}")["arn"]
ns = f"ns_{_uuid_mod.uuid4().hex[:6]}"
table = f"t_{_uuid_mod.uuid4().hex[:6]}"
try:
s3tables.create_namespace(tableBucketARN=bucket_arn, namespace=[ns])
s3tables.create_table(
- tableBucketARN=bucket_arn,
- namespace=ns,
- name=table,
- format="ICEBERG",
+ tableBucketARN=bucket_arn, namespace=ns, name=table, format="ICEBERG",
metadata={"iceberg": {"schema": {"fields": [{"name": "id", "type": "long"}]}}},
)
-
- # List namespaces — no prefix
- resp = _iceberg_json("/iceberg/v1/namespaces")
- ns_names = [
- (n[0] if isinstance(n, list) else n)
- for n in resp.get("namespaces", [])
- ]
- assert ns in ns_names
-
- # List tables — no prefix
- resp = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables")
- assert {"namespace": [ns], "name": table} in resp.get("identifiers", [])
-
- # Load table — no prefix
- resp = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
- assert resp.get("metadata-location", "").startswith(f"s3://{bucket_name}/{ns}/{table}/")
- assert resp.get("metadata", {}).get("table-uuid")
+ for method, path in (
+ ("GET", "/iceberg/v1/namespaces"),
+ ("GET", f"/iceberg/v1/namespaces/{ns}/tables"),
+ ("GET", f"/iceberg/v1/namespaces/{ns}/tables/{table}"),
+ ("POST", f"/iceberg/v1/namespaces/{ns}/tables"),
+ ("POST", "/iceberg/v1/transactions/commit"),
+ ):
+ req = urllib.request.Request(f"{_ENDPOINT}{path}", method=method, data=b"{}" if method == "POST" else None)
+ with pytest.raises(urllib.error.HTTPError) as exc:
+ urllib.request.urlopen(req)
+ assert (exc.value.code, exc.value.read()) == (404, b"<UnknownOperationException/>")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
+ assert loaded["metadata-location"].startswith(f"s3://{bucket_arn.rsplit('/', 1)[1]}/{ns}/{table}/")
finally:
- try:
- s3tables.delete_table(tableBucketARN=bucket_arn, namespace=ns, name=table)
- except Exception:
- pass
- try:
- s3tables.delete_namespace(tableBucketARN=bucket_arn, namespace=ns)
- except Exception:
- pass
+ s3tables.delete_table(tableBucketARN=bucket_arn, namespace=ns, name=table)
+ s3tables.delete_namespace(tableBucketARN=bucket_arn, namespace=ns)
s3tables.delete_table_bucket(tableBucketARN=bucket_arn)
def test_s3tables_iceberg_config_returns_s3_defaults():
- """GET /iceberg/v1/config must return S3 connection defaults so clients
- don't need out-of-band credential / endpoint configuration."""
- resp = _iceberg_json("/iceberg/v1/config")
- defaults = resp.get("defaults", {})
- assert "s3.endpoint" in defaults, "s3.endpoint missing from config defaults"
- assert "s3.access-key-id" in defaults, "s3.access-key-id missing from config defaults"
- assert "s3.secret-access-key" in defaults, "s3.secret-access-key missing from config defaults"
- assert "s3.path-style-access" in defaults, "s3.path-style-access missing from config defaults"
- assert "client.region" in defaults, "client.region missing from config defaults"
+ """GET /iceberg/v1/config hands back the url-encoded table bucket ARN as the
+ prefix in defaults, as AWS does, plus the S3 connection defaults."""
+ arn = "arn:aws:s3tables:us-east-1:000000000000:bucket/cfg-bucket"
+ resp = _iceberg_json(f"/iceberg/v1/config?warehouse={_arn_prefix(arn)}")
+ defaults = resp["defaults"]
+ assert defaults["prefix"] == _arn_prefix(arn)
+ assert resp["overrides"] == {}
+ for key in ("s3.endpoint", "s3.access-key-id", "s3.secret-access-key", "s3.path-style-access", "client.region"):
+ assert key in defaults
+
+
[email protected]("query", ["", "?warehouse=arn:aws:s3tables:us-east-1:000000000000:bucket"])
+def test_s3tables_iceberg_config_needs_a_table_bucket_arn(query):
+ status, body = _iceberg_call(f"/iceberg/v1/config{query}")
+ assert (status, body) == (400, {"error": {
+ "message": "The specified table bucket ARN is not valid.", "type": "bad_request", "code": 400}})
def test_s3tables_iceberg_load_table_includes_region(s3tables):
@@ -564,7 +556,7 @@ def test_s3tables_iceberg_load_table_includes_region(s3tables):
s3tables.create_namespace(tableBucketARN=bucket_arn, namespace=[ns])
s3tables.create_table(tableBucketARN=bucket_arn, namespace=ns, name=table, format="ICEBERG")
- resp = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ resp = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
config = resp.get("config", {})
assert "s3.region" in config, "s3.region missing from LoadTable config"
assert "client.region" in config, "client.region missing from LoadTable config"
@@ -593,7 +585,7 @@ def test_s3tables_iceberg_transactions_commit(s3tables):
snapshot_id = 1234567890
resp = _iceberg_json(
- "/iceberg/v1/transactions/commit",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/transactions/commit",
method="POST",
payload={
"table-changes": [
@@ -626,7 +618,7 @@ def test_s3tables_iceberg_transactions_commit(s3tables):
assert resp is not None
# Verify the snapshot was recorded in the table metadata
- loaded = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
metadata = loaded.get("metadata", {})
snap_ids = [s.get("snapshot-id") for s in metadata.get("snapshots", [])]
assert snapshot_id in snap_ids, f"snapshot {snapshot_id} not found after commit"
@@ -658,21 +650,21 @@ def test_s3tables_iceberg_upgrade_format_version(s3tables):
s3tables.create_namespace(tableBucketARN=bucket_arn, namespace=[ns])
s3tables.create_table(tableBucketARN=bucket_arn, namespace=ns, name=table, format="ICEBERG")
- loaded = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
assert loaded["metadata"]["format-version"] == 2
resp = _iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables/{table}", method="POST",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}", method="POST",
payload={"requirements": [],
"updates": [{"action": "upgrade-format-version", "format-version": 3}]},
)
assert resp["metadata"]["format-version"] == 3
- loaded = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
assert loaded["metadata"]["format-version"] == 3
# Re-asserting the version the table already has is a no-op commit.
resp = _iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables/{table}", method="POST",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}", method="POST",
payload={"requirements": [],
"updates": [{"action": "upgrade-format-version", "format-version": 3}]},
)
@@ -681,13 +673,13 @@ def test_s3tables_iceberg_upgrade_format_version(s3tables):
# A downgrade is refused, and the table keeps its version.
with pytest.raises(urllib.error.HTTPError) as exc_info:
_iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables/{table}", method="POST",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}", method="POST",
payload={"requirements": [],
"updates": [{"action": "upgrade-format-version", "format-version": 2}]},
)
assert exc_info.value.code == 400
assert b"downgrade" in exc_info.value.read()
- loaded = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
assert loaded["metadata"]["format-version"] == 3
finally:
try:
@@ -716,14 +708,14 @@ def test_s3tables_iceberg_create_table_rejects_resend_instead_of_wiping_data(s3t
s3tables.create_namespace(tableBucketARN=bucket_arn, namespace=[ns])
_iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables",
method="POST",
payload={"name": table, "schema": {"type": "struct", "schema-id": 0, "fields": []}},
)
# Commit a snapshot so there's real state a resent create could destroy.
snapshot_id = 42
_iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables/{table}",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}",
method="POST",
payload={
"requirements": [],
@@ -746,7 +738,7 @@ def test_s3tables_iceberg_create_table_rejects_resend_instead_of_wiping_data(s3t
# response was lost.
try:
_iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables",
method="POST",
payload={"name": table, "schema": {"type": "struct", "schema-id": 0, "fields": []}},
)
@@ -757,7 +749,7 @@ def test_s3tables_iceberg_create_table_rejects_resend_instead_of_wiping_data(s3t
assert body["error"]["type"] == "AlreadyExistsException"
# The original snapshot must still be there -- not wiped by the resend.
- loaded = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
snap_ids = [s.get("snapshot-id") for s in loaded["metadata"]["snapshots"]]
assert snapshot_id in snap_ids, "resent create-table must not wipe existing table state"
finally:
@@ -786,7 +778,7 @@ def test_s3tables_iceberg_add_schema_advances_last_column_id(s3tables):
s3tables.create_table(tableBucketARN=bucket_arn, namespace=ns, name=table, format="ICEBERG")
_iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables/{table}",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}",
method="POST",
payload={
"requirements": [],
@@ -802,11 +794,11 @@ def test_s3tables_iceberg_add_schema_advances_last_column_id(s3tables):
],
},
)
- first = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ first = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
assert first["metadata"]["last-column-id"] == 1
_iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables/{table}",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}",
method="POST",
payload={
"requirements": [],
@@ -825,7 +817,7 @@ def test_s3tables_iceberg_add_schema_advances_last_column_id(s3tables):
],
},
)
- second = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ second = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
metadata = second["metadata"]
assert metadata["last-column-id"] == 2
@@ -858,7 +850,7 @@ def test_s3tables_iceberg_add_spec_uses_real_wire_action_name(s3tables):
s3tables.create_table(tableBucketARN=bucket_arn, namespace=ns, name=table, format="ICEBERG")
_iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables/{table}",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}",
method="POST",
payload={
"requirements": [],
@@ -874,7 +866,7 @@ def test_s3tables_iceberg_add_spec_uses_real_wire_action_name(s3tables):
],
},
)
- loaded = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
metadata = loaded["metadata"]
spec_ids = [s["spec-id"] for s in metadata["partition-specs"]]
assert 1 in spec_ids, "add-spec must append the new partition spec"
@@ -906,7 +898,7 @@ def test_s3tables_iceberg_create_table_honors_requested_partition_spec(s3tables)
s3tables.create_namespace(tableBucketARN=bucket_arn, namespace=[ns])
created = _iceberg_json(
- f"/iceberg/v1/namespaces/{ns}/tables",
+ f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables",
method="POST",
payload={
"name": table,
@@ -965,9 +957,9 @@ def test_s3tables_iceberg_add_schema_is_idempotent_by_schema_id(s3tables):
# Send the identical commit twice, simulating Spark re-declaring its
# current schema on a second, unrelated write.
for _ in range(2):
- _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}", method="POST", payload=payload)
+ _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}", method="POST", payload=payload)
- loaded = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
schema_ids = [s.get("schema-id") for s in loaded["metadata"]["schemas"]]
assert schema_ids.count(0) == 1, "resent add-schema must not duplicate the schema-id entry"
finally:
@@ -1008,9 +1000,9 @@ def test_s3tables_iceberg_add_spec_is_idempotent_by_spec_id(s3tables):
],
}
for _ in range(2):
- _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}", method="POST", payload=payload)
+ _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}", method="POST", payload=payload)
- loaded = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
spec_ids = [s.get("spec-id") for s in loaded["metadata"]["partition-specs"]]
assert spec_ids.count(1) == 1, "resent add-spec must not duplicate the spec-id entry"
finally:
@@ -1051,9 +1043,9 @@ def test_s3tables_iceberg_add_sort_order_is_idempotent_by_order_id(s3tables):
],
}
for _ in range(2):
- _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}", method="POST", payload=payload)
+ _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}", method="POST", payload=payload)
- loaded = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ loaded = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
order_ids = [o.get("order-id") for o in loaded["metadata"]["sort-orders"]]
assert order_ids.count(1) == 1, "resent add-sort-order must not duplicate the order-id entry"
finally:
@@ -1093,7 +1085,7 @@ def test_s3tables_iceberg_load_table_serves_the_numerically_latest_metadata(s3ta
s3tables.create_namespace(tableBucketARN=bucket_arn, namespace=[ns])
s3tables.create_table(tableBucketARN=bucket_arn, namespace=ns, name=table, format="ICEBERG")
- base = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")["metadata"]
+ base = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")["metadata"]
for v in range(1, 12):
doc = dict(base, **{"last-sequence-number": v})
s3.put_object(
@@ -1102,7 +1094,7 @@ def test_s3tables_iceberg_load_table_serves_the_numerically_latest_metadata(s3ta
Body=json.dumps(doc).encode(),
)
- resp = _iceberg_json(f"/iceberg/v1/namespaces/{ns}/tables/{table}")
+ resp = _iceberg_json(f"/iceberg/v1/{_arn_prefix(bucket_arn)}/namespaces/{ns}/tables/{table}")
assert resp["metadata-location"].endswith("/v11.metadata.json"), resp["metadata-location"]
assert resp["metadata"]["last-sequence-number"] == 11
finally:
@@ -1351,7 +1343,10 @@ def test_s3tables_table_subresources_and_rename(s3tables):
def test_s3tables_create_table_keeps_iceberg_types(s3tables):
bucket = s3tables.create_table_bucket(name=f"types-{_uuid_mod.uuid4().hex[:8]}")["arn"]
s3tables.create_namespace(tableBucketARN=bucket, namespace=["ns"])
- primitives = ["decimal(19, 2)", "timestamp", "timestamptz", "time", "uuid", "fixed[16]", "binary", "long"]
+ primitives = ["decimal(19, 2)", "decimal(19,2)", "timestamp", "timestamptz", "time", "uuid", "fixed[16]", "binary",
+ "STRING"]
+ stored = ["decimal(19, 2)", "decimal(19, 2)", "timestamp", "timestamptz", "time", "uuid", "fixed[16]", "binary",
+ "string"]
nested = {"type": "list", "element-id": 3, "element": "string", "element-required": False}
try:
s3tables.create_table(
@@ -1367,12 +1362,32 @@ def test_s3tables_create_table_keeps_iceberg_types(s3tables):
)
prefix = _arn_prefix(bucket)
flat = _iceberg_json(f"/iceberg/v1/{prefix}/namespaces/ns/tables/flat")["metadata"]
- assert [f["type"] for f in flat["schemas"][0]["fields"]] == primitives
+ assert [f["type"] for f in flat["schemas"][0]["fields"]] == stored
fields = _iceberg_json(f"/iceberg/v1/{prefix}/namespaces/ns/tables/nested")["metadata"]["schemas"][0]["fields"]
- assert [f["type"] for f in fields] == ["decimal(9,2)", nested]
+ assert [f["type"] for f in fields] == ["decimal(9, 2)", nested]
finally:
for name in ("flat", "nested"):
with contextlib.suppress(ClientError):
s3tables.delete_table(tableBucketARN=bucket, namespace="ns", name=name)
s3tables.delete_namespace(tableBucketARN=bucket, namespace="ns")
s3tables.delete_table_bucket(tableBucketARN=bucket)
+
+
[email protected]("kind", ["timestamp_ns", "varchar", "decimal(x,2)"])
+def test_s3tables_create_table_refuses_an_invalid_iceberg_type(s3tables, kind):
+ bucket = s3tables.create_table_bucket(name=f"badtype-{_uuid_mod.uuid4().hex[:8]}")["arn"]
+ s3tables.create_namespace(tableBucketARN=bucket, namespace=["ns"])
+ try:
+ with pytest.raises(ClientError) as exc:
+ s3tables.create_table(
+ tableBucketARN=bucket, namespace="ns", name="t", format="ICEBERG",
+ metadata={"iceberg": {"schema": {"fields": [{"name": "c", "type": kind}]}}},
+ )
+ error = exc.value.response
+ assert error["ResponseMetadata"]["HTTPStatusCode"] == 400
+ assert (error["Error"]["Code"], error["Error"]["Message"]) == (
+ "BadRequestException", "The specified metadata is not valid")
+ assert s3tables.list_tables(tableBucketARN=bucket)["tables"] == []
+ finally:
+ s3tables.delete_namespace(tableBucketARN=bucket, namespace="ns")
+ s3tables.delete_table_bucket(tableBucketARN=bucket)