Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

feat(auth): authorize S3 Tables and Glue Iceberg REST calls under AUTH=true (#2289)

iot-rockete2302e95ccde
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7729ea19..fd2269f5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,10 +8,12 @@ Versioning follows [Semantic Versioning](https://semver.org/).
## [Unreleased]
### Added
-- **Auth — WebSocket handshakes are verified for IAM user and STS session keys** — under `AUTH=true` an MQTT-over-WebSocket upgrade or an `AWS_IAM` WebSocket `$connect` signed with such a key was accepted with a wrong secret, a date more than 5 minutes off or a wrong session token; the signature in the query string or `Authorization` header is now checked like that of other requests, IoT answers 403 `Forbidden` and expects the session token after the signed part of the URL, and API Gateway names the canonical string it expected. Contributed by @iot-rocket.
+- **S3 Tables, Glue — Iceberg REST calls are authorized under `AUTH=true`** — the Iceberg REST catalog at `/iceberg` served S3 Tables and Glue catalog calls without checking their credentials; each call is now authorized for the catalog that serves it, whatever service it is signed for: on a table bucket as the `s3tables` actions AWS lists for its operation (`loadTable` needs `GetTableMetadataLocation`, then `GetTableData`) on the bucket or table (`table/*` for one that does not exist), on the Glue catalog as its `glue` action on the catalog, then the database, then the table, and `transactions/commit` as `updateTable` on each table it changes. A denial answers 403 in the Iceberg error shape naming the action and resource ("Access Denied" for a later S3 Tables action, no body for `HEAD`), and a `{prefix}` that is neither a table bucket ARN nor `<account>:s3tablescatalog/<bucket>` answers 400 `bad_request`, as AWS does. Contributed by @iot-rocket.
+- **RDS Data, S3 Control — calls are authorized under `AUTH=true`** — RDS Data calls and S3 Control tagging calls were served whatever the caller's policies allowed; each RDS Data call is now authorized as `rds-data:<operation>` on the cluster its `resourceArn` names, and `ListTagsForResource`, `TagResource` and `UntagResource` as `s3:<operation>` on the bucket, with a 403 denial (`AccessDeniedException` from RDS Data, `AccessDenied` from S3 Control) naming the action and resource, as on AWS. Contributed by @iot-rocket.
+- **SES v2 — calls are authorized under `AUTH=true`** — SES v2 calls were served whatever the caller's policies allowed; each is now authorized as `ses:<operation>` on the configuration set, contact list, custom verification email template, dedicated IP pool, identity or template its path or create body names (`*` otherwise), and a denial answers 403 `AccessDeniedException` naming the action and resource, as on AWS. Contributed by @iot-rocket.
### Fixed
-- **Auth — SigV4 signatures that name no access key are refused** — under `AUTH=true` an `AWS4-HMAC-SHA256` `Authorization` header or a presigned query string without its credential, or with an empty access key, was served as the account root; it now answers `IncompleteSignature` naming every missing part, or the unknown-key error for an empty key, in each service's code and status, and Lambda answers a part that is left out as `MissingAuthenticationTokenException`. S3, API Gateway and Lambda function URLs, which take anonymous calls, are unchanged. Contributed by @iot-rocket.
+- **Auth — requests that map to no IAM action are authenticated** — under `AUTH=true` a request MiniStack maps to no IAM action (SES v2, RDS Data, S3 Control, or a service MiniStack does not implement such as X-Ray) was served with a wrong secret, an unknown key, a session key without its token or a stale signature; it now answers the credential error AWS sends for the service, for example `InvalidSignatureException` 403 from SES v2 and RDS Data and `SignatureDoesNotMatch` 403 from S3 Control. Contributed by @iot-rocket.
## [1.5.29] — 2026-10-17
diff --git a/ministack/app.py b/ministack/app.py
index 74c79f8d..f739eec6 100644
--- a/ministack/app.py
+++ b/ministack/app.py
@@ -2428,6 +2428,63 @@ def _enforce_ses_v2(method: str, path: str, headers: dict, body: bytes, query_pa
named_resource=resource_arn)
+def _iceberg_json_failure(failure):
+ """A credential failure as both Iceberg REST catalogs send it, in application/json."""
+ status, resp_headers, resp_body = failure
+ return status, {**resp_headers, "Content-Type": "application/json"}, resp_body
+
+
+def _enforce_iceberg(method: str, path: str, headers: dict, body: bytes, query_params: dict, request_id: str):
+ """Authorize an Iceberg REST call for the catalog the router serves it from, whatever it is signed for."""
+ from ministack.core.iam_actions import identity_denial_message
+ from ministack.core.iam_evaluator import AuthError, enforce
+
+ glue = _iceberg_targets_glue_catalog(path, query_params)
+ service = "glue" if glue else "s3tables"
+ catalog = _get_module(service)
+ # A route without the table bucket prefix answers 404 before any credential check.
+ if not glue and catalog.iceberg_lacks_prefix(method, path):
+ return None
+ checks = catalog.iceberg_iam_checks(method, path, body, query_params)
+ if not checks:
+ if failure := _authenticate_only(service, headers, query_params, request_id):
+ return _iceberg_json_failure(failure)
+ return None if glue else catalog.iceberg_prefix_error(method, path)
+ access_key, region = extract_access_key_id(headers, query_params), extract_region(headers, query_params)
+ for position, (iam_action, resource_arn) in enumerate(checks):
+ denied = enforce(access_key, iam_action, service, region, resource_arn=resource_arn)
+ if isinstance(denied, AuthError):
+ return _iceberg_json_failure(_credential_failure(service, iam_action, request_id, denied, headers))
+ if not denied:
+ continue
+ resp_headers = {"Content-Type": "application/json",
+ "x-amzn-errortype": "ForbiddenResponse" if glue else "IrcForbiddenResponse"}
+ if method == "HEAD":
+ return 403, resp_headers, b""
+ # S3 Tables names only the first action's denial; a later one answers "Access Denied".
+ message = identity_denial_message(iam_action, denied.principal_arn, resource_arn,
+ denied.decision == "Deny") if glue or position == 0 else "Access Denied"
+ error = {"message": message, "type": "AccessDeniedException" if glue else "forbidden", "code": 403}
+ return 403, resp_headers, json.dumps({"error": error}).encode()
+ return None
+
+
+def _enforce_ses_v2(method: str, path: str, headers: dict, body: bytes, query_params: dict, request_id: str):
+ """Authorize an SES v2 call as ses:<operation> on the resource it names, which its denial names too."""
+ if not AUTH:
+ return None
+ from ministack.core.iam_actions import extract_iam_action, extract_resource_arn
+ from ministack.core.responses import get_account_id
+
+ iam_action = extract_iam_action("ses", method, path, headers, body, query_params)
+ if iam_action is None:
+ return _authenticate_only("sesv2", headers, query_params, request_id)
+ resource_arn = extract_resource_arn("ses", method, path, headers, body, query_params,
+ extract_region(headers, query_params), get_account_id())
+ return _enforce_data_plane("sesv2", iam_action, headers, query_params, request_id, resource_arn=resource_arn,
+ named_resource=resource_arn)
+
+
def _iceberg_targets_glue_catalog(path, query_params):
"""Decide whether an ``/iceberg`` REST request belongs to the Glue Data
Catalog (Glue jobs + Firehose lakehouse) or to S3 Tables.
@@ -2477,6 +2534,8 @@ async def _handle_special_data_plane_request(
# the other reads. Must not fall through to S3 (which reads "iceberg" as a
# bucket name).
if path.startswith("/iceberg"):
+ if AUTH and (denied := _enforce_iceberg(method, path, headers, body, query_params, request_id)):
+ return denied
try:
if _iceberg_targets_glue_catalog(path, query_params):
result = _get_module("glue")._handle_iceberg_rest(method, path, query_params, body=body)
diff --git a/ministack/services/glue.py b/ministack/services/glue.py
index 33c3cfb7..c67dc312 100644
--- a/ministack/services/glue.py
+++ b/ministack/services/glue.py
@@ -711,19 +711,91 @@ def _iceberg_rest_commit_table(namespace, table_name, body):
def _iceberg_rest_commit_transaction(body):
"""POST transactions/commit -- DuckDB's atomic multi-table commit."""
for change in body.get("table-changes", []):
- ident = change.get("identifier", {})
- ns_field = ident.get("namespace", [""])
- ns = ns_field[0] if isinstance(ns_field, list) and ns_field else (ns_field or "")
- tbl = ident.get("name", "")
+ ns, tbl = _iceberg_rest_change_identifier(change)
result = _iceberg_rest_commit_table(ns, tbl, change)
if result and result[0] not in (200, 204):
return result
return _iceberg_json({})
+# Iceberg REST calls below namespaces/ -> the glue action they need and how many of catalog, database, table it names.
+_ICEBERG_IAM_ACTIONS = {
+ (0, "GET"): ("GetDatabases", 1),
+ (0, "POST"): ("CreateDatabase", 2),
+ (1, "GET"): ("GetDatabase", 2),
+ (1, "HEAD"): ("GetDatabase", 2),
+ (1, "DELETE"): ("DeleteDatabase", 2),
+ (2, "GET"): ("GetTables", 2),
+ (2, "POST"): ("CreateTable", 3),
+ (3, "GET"): ("GetTable", 3),
+ (3, "HEAD"): ("GetTable", 3),
+ (3, "POST"): ("UpdateTable", 3),
+ (3, "DELETE"): ("DeleteTable", 3),
+}
+
+
+def iceberg_iam_checks(method, path, body, query_params):
+ """The glue actions an Iceberg REST call on the Glue catalog needs, with resources, in order; else None."""
+ parts = _iceberg_rest_parts(path)[2:]
+ catalog = f"arn:aws:glue:{get_region()}:{get_account_id()}:catalog"
+ if parts == ["config"] and method == "GET":
+ return [("glue:GetCatalog", catalog)]
+ # Every {catalog} is served from the account's default catalog, so it is authorized as that one.
+ if parts[:1] == ["catalogs"] and parts[2:3] == ["transactions"] and method == "POST":
+ # Each table change is an updateTable; a body whose changes cannot be read needs it on every resource.
+ try:
+ tables = [_iceberg_rest_change_identifier(change)
+ for change in json.loads(body or b"{}").get("table-changes", [])]
+ except (ValueError, TypeError, AttributeError):
+ tables = []
+ if not tables:
+ return [("glue:UpdateTable", "*")]
+ return [("glue:UpdateTable", arn) for namespace, table in tables
+ for arn in (catalog, _arn("database", namespace), _arn("table", f"{namespace}/{table}"))]
+ if parts[:1] != ["catalogs"] or parts[2:3] != ["namespaces"]:
+ return None
+ rest = parts[3:]
+ if rest[1:] == ["properties"] and method == "POST":
+ resources = [catalog, _arn("database", rest[0])]
+ return [(f"glue:{action}", arn) for action in ("GetDatabase", "UpdateDatabase") for arn in resources]
+ entry = _ICEBERG_IAM_ACTIONS.get((len(rest), method))
+ if not entry or rest[1:2] not in ([], ["tables"]):
+ return None
+ action, depth = entry
+ namespace, _, table = rest + [""] * (3 - len(rest))
+ if method == "POST" and len(rest) in (0, 2):
+ named = _iceberg_body_name(body, "namespace" if not rest else "name")
+ namespace, table = (named, table) if not rest else (namespace, named)
+ resources = [catalog, _arn("database", namespace), _arn("table", f"{namespace}/{table}")]
+ return [(f"glue:{action}", arn) for arn in resources[:depth]]
+
+
+def _iceberg_rest_parts(path):
+ """The segments of an Iceberg REST path as the Glue router reads them, each decoded on its own."""
+ return [unquote(p) for p in path.strip("/").split("/") if p]
+
+
+def _iceberg_rest_change_identifier(change):
+ """The namespace and table name a transactions/commit table change names, read as the commit reads them."""
+ ident = change.get("identifier", {})
+ ns_field = ident.get("namespace", [""])
+ return (ns_field[0] if isinstance(ns_field, list) and ns_field else (ns_field or "")), ident.get("name", "")
+
+
+def _iceberg_body_name(body, field):
+ """The namespace or table name an Iceberg REST request body names, read as the handlers read it, or ''."""
+ try:
+ name = json.loads(body or b"{}").get(field, "")
+ except (ValueError, TypeError, AttributeError):
+ return ""
+ if field == "namespace":
+ name = name[0] if isinstance(name, list) and name else (name or "")
+ return str(name)
+
+
def _handle_iceberg_rest(method, path, query_params, body=None):
_iceberg_maybe_log_tls_hint()
- parts = [unquote(p) for p in path.strip("/").split("/") if p]
+ parts = _iceberg_rest_parts(path)
if len(parts) < 3 or parts[0] != "iceberg" or parts[1] != "v1":
return _iceberg_error(f"Unknown Iceberg REST path: {path}", "NotFoundException", 404)
diff --git a/ministack/services/s3tables.py b/ministack/services/s3tables.py
index 232187a9..4d0f6932 100644
--- a/ministack/services/s3tables.py
+++ b/ministack/services/s3tables.py
@@ -689,7 +689,13 @@ def _in_bucket(bucket_arn: str, bucket_filter) -> bool:
return bucket_arn.endswith("/" + bucket_filter)
-def _iceberg_allows_cross_region(headers):
+def _iceberg_allows_cross_region(headers, query_params=None):
+ """Whether a call names no signing region; under AUTH a presigned call names it in its query, as checks read it."""
+ if query_params and "X-Amz-Credential" in query_params:
+ from ministack.app import AUTH
+
+ if AUTH:
+ return False
return _SIGV4_CREDENTIAL_REGION_RE.search(headers.get("authorization", "")) is None
@@ -1102,14 +1108,141 @@ def _unknown_operation():
return 404, {"Content-Type": "application/xml"}, b"<UnknownOperationException/>"
+# Iceberg REST calls on a namespace or table -> the s3tables actions they need, in order.
+_ICEBERG_NAMESPACE_ACTIONS = {
+ (0, "GET"): "ListNamespaces",
+ (0, "POST"): "CreateNamespace",
+ (1, "GET"): "GetNamespace",
+ (1, "HEAD"): "GetNamespace",
+ (1, "DELETE"): "DeleteNamespace",
+}
+_ICEBERG_TABLE_ACTIONS = {
+ (2, "GET"): ("ListTables",),
+ (2, "POST"): ("CreateTable", "PutTableData"),
+ (3, "GET"): ("GetTableMetadataLocation", "GetTableData"),
+ (3, "POST"): ("UpdateTableMetadataLocation", "PutTableData", "GetTableData"),
+ (3, "DELETE"): ("DeleteTable",),
+ (3, "HEAD"): ("GetTable",),
+}
+_ICEBERG_BUCKET_ACTIONS = frozenset({"ListTables", "CreateTable"})
+_TABLE_BUCKET_CATALOG_RE = re.compile(r"\d{12}:s3tablescatalog/([a-z0-9-]+)")
+
+
+def _iceberg_route(method, path):
+ """The route the Iceberg REST router reads from a path: (route, raw {prefix} segments, raw segments after it)."""
+ parts = [p for p in path.strip("/").split("/") if p]
+ if parts[1:2] == ["v1"] and parts[2:] == ["config"] and method == "GET":
+ return "config", [], []
+ if parts[1:2] == ["v1"] and method == "POST" and "transactions" in parts[2:]:
+ index = parts.index("transactions", 2)
+ return "transactions", parts[2:index], parts[index + 1:]
+ if "namespaces" in parts[2:]:
+ index = parts.index("namespaces", 2)
+ return "namespaces", parts[2:index], parts[index + 1:]
+ if method == "POST" and len(parts) > 3 and parts[-2:] == ["tables", "rename"]:
+ return "rename", parts[2:-2], []
+ return None
+
+
+def _iceberg_change_identifier(change):
+ """The namespace and table name a transactions/commit table change names, read as the commit reads them."""
+ ident = change.get("identifier", {})
+ ns = ident.get("namespace", [""])
+ return (ns[0] if isinstance(ns, list) else ns), ident.get("name", "")
+
+
+def iceberg_iam_checks(method, path, body, query_params):
+ """The s3tables actions an Iceberg REST call on a table bucket needs, with resources, in AWS's order; else None."""
+ route = _iceberg_route(method, path)
+ if not route:
+ return None
+ kind, prefix, rest = route
+ if kind == "config":
+ warehouse = query_params.get("warehouse", "") if query_params else ""
+ warehouse = (warehouse[0] if warehouse else "") if isinstance(warehouse, list) else warehouse
+ bucket_arn = _iceberg_bucket_arn(warehouse)
+ # getConfig serves any account:s3tablescatalog/ warehouse; one that names no bucket needs it on every bucket.
+ if bucket_arn != "*" or ":s3tablescatalog/" in warehouse:
+ return [("s3tables:GetTableBucket", bucket_arn)]
+ return None
+ bucket_arn = _iceberg_bucket_arn(unquote("/".join(prefix)))
+ if bucket_arn == "*":
+ return None
+ if kind == "rename":
+ try:
+ source = json.loads(body or b"{}").get("source") or {}
+ namespace, name = source.get("namespace") or [""], source.get("name", "")
+ except (ValueError, AttributeError):
+ namespace, name = [""], ""
+ namespace = namespace[0] if isinstance(namespace, list) else namespace
+ return [("s3tables:RenameTable", _iceberg_table_arn(bucket_arn, namespace, name))]
+ if kind == "transactions":
+ # Each table change is an updateTable; a body whose changes cannot be read needs them on every resource.
+ try:
+ data = json.loads(body) if body else {}
+ tables = [_iceberg_change_identifier(change) for change in data.get("table-changes", [])]
+ except (ValueError, TypeError, AttributeError, IndexError):
+ tables = []
+ arns = [_iceberg_table_arn(bucket_arn, ns, name) for ns, name in tables] or ["*"]
+ return [(f"s3tables:{name}", arn) for arn in arns for name in _ICEBERG_TABLE_ACTIONS[3, "POST"]]
+ if len(rest) < 2:
+ action = _ICEBERG_NAMESPACE_ACTIONS.get((len(rest), method))
+ return [(f"s3tables:{action}", bucket_arn)] if action else None
+ if rest[1] != "tables" or len(rest) > 3:
+ return None
+ table_arn = _iceberg_table_arn(bucket_arn, rest[0], rest[2] if len(rest) == 3 else "")
+ names = _ICEBERG_TABLE_ACTIONS.get((len(rest), method), ())
+ return [(f"s3tables:{name}", bucket_arn if name in _ICEBERG_BUCKET_ACTIONS else table_arn)
+ for name in names] or None
+
+
+def iceberg_lacks_prefix(method, path):
+ """Whether an Iceberg REST route leaves out the table bucket {prefix}, which the router answers 404 unread."""
+ route = _iceberg_route(method, path)
+ return bool(route) and route[0] != "config" and not route[1]
+
+
+def iceberg_prefix_error(method, path):
+ """The 400 for a route whose {prefix} names no table bucket, which AWS answers before any policy check."""
+ route = _iceberg_route(method, path)
+ if not route or route[0] == "config" or not route[1]:
+ return None
+ prefix = unquote("/".join(route[1]))
+ if _iceberg_bucket_arn(prefix) != "*":
+ return None
+ if _TABLE_BUCKET_ARN_RE.match(prefix):
+ message = "The specified bucket name isn't valid. Specify a different bucket name, and then try again."
+ else:
+ message = "The specified table bucket ARN is not valid."
+ status, headers, body = _iceberg_error(message, "bad_request", 400)
+ return status, headers, b"" if method == "HEAD" else body
+
+
+def _iceberg_bucket_arn(prefix):
+ """The table bucket a {prefix} or warehouse names by its ARN or as account:s3tablescatalog/bucket, or '*'."""
+ if _TABLE_BUCKET_ARN_RE.fullmatch(prefix):
+ return prefix
+ named = _TABLE_BUCKET_CATALOG_RE.fullmatch(prefix)
+ return _bucket_arn(named.group(1)) if named else "*"
+
+
+def _iceberg_table_arn(bucket_arn, namespace, name):
+ """A table's ARN; AWS names table/* for a table that does not exist."""
+ if bucket_arn == "*":
+ return "*"
+ table = _tables.get(_table_key(bucket_arn, namespace, name))
+ return table["tableARN"] if table else f"{bucket_arn}/table/*"
+
+
async def _handle_iceberg_request(method, path, headers, body, query_params):
parts = [p for p in path.strip("/").split("/") if p]
- allow_cross_region = _iceberg_allows_cross_region(headers)
+ allow_cross_region = _iceberg_allows_cross_region(headers, query_params)
# parts: ["iceberg", "v1", ...]
if len(parts) < 2:
return None
- if parts[1] == "v1" and len(parts) == 3 and parts[2] == "config" and method == "GET":
+ route = _iceberg_route(method, path)
+ if route and route[0] == "config":
return _iceberg_config(query_params, headers)
# POST /iceberg/v1/transactions/commit — DuckDB atomic multi-table commit.
@@ -1117,40 +1250,26 @@ async def _handle_iceberg_request(method, path, headers, body, query_params):
# this arrives as /iceberg/v1/{prefix}/transactions/commit. Match on the
# `transactions` segment wherever it lands, not a fixed position — otherwise
# a prefixed commit is silently dropped and the delivery writes nothing.
- if parts[1] == "v1" and method == "POST" and "transactions" in parts[2:]:
+ if route and route[0] == "transactions":
data = json.loads(body) if body else {}
- tx_idx = parts.index("transactions", 2)
- if tx_idx == 2:
+ if not route[1]:
return _unknown_operation()
- tx_filter = _bucket_scope_from_prefix(unquote("/".join(parts[2:tx_idx])))
+ tx_filter = _bucket_scope_from_prefix(unquote("/".join(route[1])))
for change in data.get("table-changes", []):
- ident = change.get("identifier", {})
- ns = ident.get("namespace", [""])
- ns = ns[0] if isinstance(ns, list) else ns
- tbl = ident.get("name", "")
+ ns, tbl = _iceberg_change_identifier(change)
result = _iceberg_commit_table(ns, tbl, change, allow_cross_region, tx_filter)
if result and result[0] not in (200, 204):
return result
return json_response({})
- if len(parts) < 3:
- return None
-
# Every route but /v1/config takes the {prefix}, which "for S3 Tables is
# always your url-encoded table bucket ARN"; it can span several segments.
- bucket_filter = None
- ns_idx = None
- for i in range(2, len(parts)):
- if parts[i] == "namespaces":
- ns_idx = i
- break
- if ns_idx is None:
+ if not route or route[0] != "namespaces":
return None
- if ns_idx == 2:
+ prefix, rest = route[1], route[2] # rest: segments after "namespaces"
+ if not prefix:
return _unknown_operation()
- bucket_filter = _bucket_scope_from_prefix(unquote("/".join(parts[2:ns_idx])))
-
- rest = parts[ns_idx + 1 :] # segments after "namespaces"
+ bucket_filter = _bucket_scope_from_prefix(unquote("/".join(prefix)))
if len(rest) == 0 and method == "GET":
return _iceberg_list_namespaces(allow_cross_region, bucket_filter)
diff --git a/tests/test_glue.py b/tests/test_glue.py
index 45bc6036..34fdb4f8 100644
--- a/tests/test_glue.py
+++ b/tests/test_glue.py
@@ -2445,6 +2445,228 @@ def test_iceberg_dispatch_routes_glue_vs_s3tables():
assert targets_glue("/iceberg/v1/config", {}) is False
+def test_iceberg_iam_checks_read_body_names_as_the_handlers_do():
+ """The checks read createNamespace, createTable and transaction bodies as the handlers do."""
+ from ministack.services.glue import iceberg_iam_checks
+
+ arn = "arn:aws:glue:us-east-1:000000000000"
+ catalog = "/iceberg/v1/catalogs/000000000000"
+ assert iceberg_iam_checks("POST", f"{catalog}/namespaces", b'{"namespace": ["db"]}', {})[-1] == (
+ "glue:CreateDatabase", f"{arn}:database/db")
+ assert iceberg_iam_checks("POST", f"{catalog}/namespaces/db/tables", b'{"name": ["x"]}', {})[-1] == (
+ "glue:CreateTable", f"{arn}:table/db/['x']")
+ change = b'{"table-changes": [{"identifier": {"namespace": "db", "name": "t"}}]}'
+ assert iceberg_iam_checks("POST", f"{catalog}/transactions/commit", change, {})[-1] == (
+ "glue:UpdateTable", f"{arn}:table/db/t")
+ for unread in (b"", b"[]", b'{"table-changes": [[]]}'):
+ assert iceberg_iam_checks("POST", f"{catalog}/transactions/commit", unread, {}) == [("glue:UpdateTable", "*")]
+
+
+def _glue_iceberg_signed(credentials, method, path, body=None, query=None, service="glue"):
+ """One Iceberg REST call signed for glue (or another service) over HTTP: status, body (JSON or None), headers."""
+ import urllib.error
+ import urllib.request
+
+ from botocore.auth import S3SigV4Auth, SigV4Auth
+ from botocore.awsrequest import AWSRequest
+ from conftest import ENDPOINT
+
+ request = AWSRequest(method=method, url=f"{ENDPOINT}/iceberg/v1/{path}", params=query or {},
+ data=json.dumps(body).encode() if body is not None else None)
+ (S3SigV4Auth if service == "s3" else SigV4Auth)(credentials, service, "us-east-1").add_auth(request)
+ prepared = request.prepare()
+ try:
+ with urllib.request.urlopen(urllib.request.Request(
+ prepared.url, data=prepared.body, method=method, headers=dict(prepared.headers))) as response:
+ status, content, headers = response.status, response.read(), response.headers
+ except urllib.error.HTTPError as error:
+ status, content, headers = error.code, error.read(), error.headers
+ return status, json.loads(content) if content else None, {k.lower(): v for k, v in headers.items()}
+
+
+def glue_iceberg_user(glue, iam_user):
+ """A Glue database with Iceberg tables t and u, and a factory for users of it signing with a second access key."""
+ import contextlib
+ import uuid
+
+ from botocore.credentials import Credentials
+ from conftest import make_client
+
+ database = f"irc_{uuid.uuid4().hex[:8]}"
+ glue.create_database(DatabaseInput={"Name": database})
+ root = Credentials("test", "test")
+ for table in ("t", "u"):
+ assert _glue_iceberg_signed(root, "POST", f"catalogs/000000000000/namespaces/{database}/tables", {
+ "name": table, "schema": {"type": "struct", "schema-id": 0, "fields": []}})[0] == 200
+ iam = make_client("iam")
+ with contextlib.ExitStack() as cleanup:
+ def drop_databases():
+ for found in glue.get_databases()["DatabaseList"]:
+ if found["Name"].startswith(database):
+ glue.delete_database(Name=found["Name"])
+
+ cleanup.callback(drop_databases)
+
+ def create(statements=None):
+ user = iam_user(policy={"Version": "2012-10-17", "Statement": statements} if statements else None)
+ key = iam.create_access_key(UserName=user.name)["AccessKey"]
+ cleanup.callback(iam.delete_access_key, UserName=user.name, AccessKeyId=key["AccessKeyId"])
+ return user, Credentials(key["AccessKeyId"], key["SecretAccessKey"])
+
+ yield database, create
+
+
+def _glue_iceberg_denial(user, action, resource, reason=None):
+ """The Iceberg error body of a glue identity-policy denial."""
+ reason = reason or f"because no identity-based policy allows the glue:{action} action"
+ resource = resource if resource == "*" else f"arn:aws:glue:us-east-1:000000000000:{resource}"
+ return {"error": {"message": f"User: {user.arn} is not authorized to perform: glue:{action} on resource: "
+ f"{resource} {reason}", "type": "AccessDeniedException", "code": 403}}
+
+
+# Iceberg REST routes on the Glue catalog: method, path, body, the glue actions each needs with the resources it
+# checks them on (catalog, then database, then table), and MiniStack's status once they are allowed.
+_DATABASE = ["catalog", "database/{db}"]
+_TABLE_T = ["catalog", "database/{db}", "table/{db}/t"]
+_GLUE_ICEBERG_ROUTES = {
+ "config": ("GET", "config", None, [("GetCatalog", ["catalog"])], 200),
+ "list-namespaces": ("GET", "namespaces", None, [("GetDatabases", ["catalog"])], 200),
+ "create-namespace": ("POST", "namespaces", {"namespace": ["{db}_{new}"]},
+ [("CreateDatabase", ["catalog", "database/{db}_{new}"])], 200),
+ "load-namespace": ("GET", "namespaces/{db}", None, [("GetDatabase", _DATABASE)], 200),
+ "namespace-exists": ("HEAD", "namespaces/{db}", None, [("GetDatabase", _DATABASE)], 200),
+ "drop-namespace": ("DELETE", "namespaces/{db}", None, [("DeleteDatabase", _DATABASE)], 501),
+ "namespace-properties": ("POST", "namespaces/{db}/properties", {"updates": {"k": "v"}},
+ [("GetDatabase", _DATABASE), ("UpdateDatabase", _DATABASE)], 501),
+ "list-tables": ("GET", "namespaces/{db}/tables", None, [("GetTables", _DATABASE)], 200),
+ "create-table": ("POST", "namespaces/{db}/tables", {"name": "{new}"},
+ [("CreateTable", ["catalog", "database/{db}", "table/{db}/{new}"])], 200),
+ "load-table": ("GET", "namespaces/{db}/tables/t", None, [("GetTable", _TABLE_T)], 200),
+ "table-exists": ("HEAD", "namespaces/{db}/tables/t", None, [("GetTable", _TABLE_T)], 200),
+ "update-table": ("POST", "namespaces/{db}/tables/t", {"requirements": [], "updates": []},
+ [("UpdateTable", _TABLE_T)], 200),
+ "drop-table": ("DELETE", "namespaces/{db}/tables/t", None, [("DeleteTable", _TABLE_T)], 501),
+ "commit-transaction": ("POST", "transactions/commit", {"table-changes": [
+ {"identifier": {"namespace": ["{db}"], "name": "t"}, "requirements": [], "updates": []}]},
+ [("UpdateTable", _TABLE_T)], 200),
+}
+
+
+def _glue_iceberg_fill(value, names):
+ """A path, body or resource template with {name} placeholders filled from ``names``."""
+ text = json.dumps(value)
+ for key, name in names.items():
+ text = text.replace("{" + key + "}", name)
+ return json.loads(text)
+
+
[email protected]("service", ["glue", "s3tables", "s3", "sts"])
[email protected]("route", list(_GLUE_ICEBERG_ROUTES))
+def test_glue_iceberg_route_contract(auth_enabled, glue_iceberg_user, route, service):
+ """Each Iceberg route on the Glue catalog needs exactly its glue actions, whatever service it is signed for."""
+ database, create = glue_iceberg_user
+ method, path, body, actions, served = _GLUE_ICEBERG_ROUTES[route]
+ glue_arn = "arn:aws:glue:us-east-1:000000000000"
+ allowed_names = {"db": database, "new": "a"}
+ user, signer = create([
+ {"Effect": "Allow", "Action": f"glue:{action}",
+ "Resource": [f"{glue_arn}:{resource}" for resource in _glue_iceberg_fill(resources, allowed_names)]}
+ for action, resources in actions])
+ nobody, refused_signer = create()
+ query = {"warehouse": "000000000000"} if path == "config" else None
+ path = path if path == "config" else f"catalogs/000000000000/{path}"
+ calls = [_glue_iceberg_signed(credentials, method, *_glue_iceberg_fill([path, body], names), query=query,
+ service=service)
+ for credentials, names in ((signer, allowed_names), (refused_signer, {"db": database, "new": "n"}))]
+ assert calls[0][0] == served
+ if not auth_enabled:
+ assert calls[1][0] == served
+ return
+ status, error, headers = calls[1]
+ assert (status, headers["content-type"], headers["x-amzn-errortype"]) == (403, "application/json",
+ "ForbiddenResponse")
+ assert error == (None if method == "HEAD" else _glue_iceberg_denial(nobody, actions[0][0], "catalog"))
+
+
[email protected]("method,path,body,action,resource,reason", [
+ ("GET", "namespaces/{db}_new", None, "GetDatabase", "database/{db}_new", None),
+ ("POST", "namespaces/{db}/properties", {"updates": {"k": "v"}}, "UpdateDatabase", "catalog", None),
+ ("GET", "namespaces/{db}/tables/u", None, "GetTable", "table/{db}/u", None),
+ ("POST", "namespaces/{db}/tables/t", {"requirements": [], "updates": []}, "UpdateTable", "table/{db}/t",
+ "with an explicit deny in an identity-based policy"),
+], ids=["absent-namespace", "namespace-properties", "other-table", "explicit-deny"])
+def test_glue_iceberg_authorization_contract(auth_enabled, glue_iceberg_user, method, path, body, action, resource,
+ reason):
+ """A denial names the first resource the policy does not cover: the catalog, then the database, then the table."""
+ database, create = glue_iceberg_user
+ glue_arn = "arn:aws:glue:us-east-1:000000000000"
+ scope = [f"{glue_arn}:catalog", f"{glue_arn}:database/{database}"]
+ user, signer = create([
+ {"Effect": "Allow", "Resource": scope,
+ "Action": ["glue:GetCatalog", "glue:GetDatabases", "glue:GetDatabase", "glue:UpdateTable"]},
+ {"Effect": "Allow", "Action": "glue:GetTable", "Resource": [*scope, f"{glue_arn}:table/{database}/t"]},
+ {"Effect": "Deny", "Action": "glue:UpdateTable", "Resource": f"{glue_arn}:table/{database}/t"},
+ ])
+ assert _glue_iceberg_signed(signer, "GET", "config", query={"warehouse": "000000000000"})[0] == 200
+ assert [database] in _glue_iceberg_signed(signer, "GET", "catalogs/:/namespaces")[1]["namespaces"]
+ assert _glue_iceberg_signed(signer, "GET", f"catalogs/000000000000/namespaces/{database}/tables/t")[0] == 200
+ status, error, headers = _glue_iceberg_signed(
+ signer, method, "catalogs/000000000000/" + path.format(db=database), body)
+ if not auth_enabled:
+ assert status != 403
+ return
+ assert (status, headers["content-type"], headers["x-amzn-errortype"]) == (
+ 403, "application/json", "ForbiddenResponse")
+ assert error == _glue_iceberg_denial(user, action, resource.format(db=database), reason)
+
+
[email protected]("catalog", [":", "arn:aws:glue:us-east-1:000000000000:catalog"], ids=["colon", "arn"])
+def test_glue_iceberg_catalog_forms_contract(auth_enabled, glue_iceberg_user, catalog):
+ """MiniStack serves its default catalog for any {catalog}, so every form is authorized as that catalog."""
+ database, create = glue_iceberg_user
+ allowed, signer = create([{"Effect": "Allow", "Action": "glue:GetDatabases",
+ "Resource": "arn:aws:glue:us-east-1:000000000000:catalog"}])
+ nobody, refused_signer = create([{"Effect": "Allow", "Action": "glue:GetDatabase", "Resource": "*"}])
+ assert [database] in _glue_iceberg_signed(signer, "GET", f"catalogs/{catalog}/namespaces")[1]["namespaces"]
+ status, error, _ = _glue_iceberg_signed(refused_signer, "GET", f"catalogs/{catalog}/namespaces")
+ if not auth_enabled:
+ assert status == 200
+ return
+ assert (status, error) == (403, _glue_iceberg_denial(nobody, "GetDatabases", "catalog"))
+
+
[email protected]("body", ["t", "t-and-u", "no-changes", "no-object"])
+def test_glue_iceberg_transaction_commit_contract(auth_enabled, glue_iceberg_user, body):
+ """transactions/commit needs glue:UpdateTable on each table it changes, on * when it names none."""
+ database, create = glue_iceberg_user
+ glue_arn = "arn:aws:glue:us-east-1:000000000000"
+ change = {"identifier": {"namespace": [database], "name": "t"}, "requirements": [], "updates": []}
+ bodies = {"t": {"table-changes": [change]},
+ "t-and-u": {"table-changes": [change, {**change, "identifier": {"namespace": [database], "name": "u"}}]},
+ "no-changes": {}, "no-object": "table-changes"}
+ allowed, signer = create([{"Effect": "Allow", "Action": "glue:UpdateTable", "Resource": [
+ f"{glue_arn}:catalog", f"{glue_arn}:database/{database}", f"{glue_arn}:table/{database}/t"]}])
+ nobody, refused_signer = create()
+ path = "catalogs/000000000000/transactions/commit"
+ committed = _glue_iceberg_signed(signer, "POST", path, bodies[body])
+ refused = _glue_iceberg_signed(refused_signer, "POST", path, bodies[body])
+ if not auth_enabled:
+ assert (committed[0], refused[0]) == ((500, 500) if body == "no-object" else (200, 200))
+ return
+ unread = body in ("no-changes", "no-object")
+ assert refused[:2] == (403, _glue_iceberg_denial(nobody, "UpdateTable", "*" if unread else "catalog"))
+ if body == "t":
+ assert committed[0] == 200
+ else:
+ assert committed[:2] == (403, _glue_iceberg_denial(allowed, "UpdateTable", "*" if unread else
+ f"table/{database}/u"))
+
+
def test_non_catalogs_prefix_returns_501():
"""Bare-warehouse prefixes (`/v1/{warehouse}/namespaces`) are the
S3 Tables shape, not Glue's — reject rather than guess."""
diff --git a/tests/test_s3tables.py b/tests/test_s3tables.py
index 78b4f960..ffe6a00c 100644
--- a/tests/test_s3tables.py
+++ b/tests/test_s3tables.py
@@ -1242,6 +1242,424 @@ def test_s3tables_iceberg_rest_sequence_numbers_and_requirements(s3tables):
s3tables.delete_table_bucket(tableBucketARN=arn)
+def test_iceberg_route_splits_paths_as_the_router():
+ """The router and the IAM checks read one split: raw segments, the {prefix} decoded only afterwards."""
+ from ministack.services.s3tables import _iceberg_route, iceberg_lacks_prefix
+
+ bucket = "arn%3Aaws%3As3tables%3Aus-east-1%3A000000000000%3Abucket%2Fb"
+ assert _iceberg_route("GET", "/iceberg/v1/config") == ("config", [], [])
+ assert _iceberg_route("GET", f"/iceberg/v1/{bucket}/namespaces/ns") == ("namespaces", [bucket], ["ns"])
+ # a decoded ARN spans two segments; an encoded keyword is a segment like any other
+ assert _iceberg_route("GET", "/iceberg/v1/arn:aws:s3tables:us-east-1:000000000000:bucket/b/namespaces") == (
+ "namespaces", ["arn:aws:s3tables:us-east-1:000000000000:bucket", "b"], [])
+ assert _iceberg_route("GET", f"/iceberg/v1/%6Eamespaces/{bucket}/namespaces") == (
+ "namespaces", ["%6Eamespaces", bucket], [])
+ assert _iceberg_route("GET", f"/iceberg/v1/{bucket}/%6Eamespaces") is None
+ # tables/rename after a namespace is updateTable on a table named rename
+ assert _iceberg_route("POST", f"/iceberg/v1/{bucket}/tables/rename") == ("rename", [bucket], [])
+ assert _iceberg_route("POST", f"/iceberg/v1/{bucket}/namespaces/ns/tables/rename") == (
+ "namespaces", [bucket], ["ns", "tables", "rename"])
+ assert _iceberg_route("POST", f"/iceberg/v1/{bucket}/transactions/commit") == (
+ "transactions", [bucket], ["commit"])
+ for method, path in (("GET", "/iceberg/v1/namespaces"), ("GET", "/iceberg/v1/namespaces/ns/tables/t"),
+ ("POST", "/iceberg/v1/tables/rename"), ("POST", "/iceberg/v1/transactions/commit")):
+ assert iceberg_lacks_prefix(method, path)
+ # a first segment before "namespaces" is the {prefix}, whatever it reads
+ for method, path in (("GET", "/iceberg/v1/config"), ("GET", f"/iceberg/v1/{bucket}/namespaces"),
+ ("POST", f"/iceberg/v1/{bucket}/tables/rename"), ("GET", "/iceberg/v1/tables/namespaces"),
+ ("GET", "/iceberg/v1/transactions/namespaces/ns"),
+ ("POST", f"/iceberg/v1/{bucket}/transactions/commit")):
+ assert not iceberg_lacks_prefix(method, path)
+
+
+def test_iceberg_iam_checks_name_the_bucket_the_prefix_names():
+ """A bucket ARN or account:s3tablescatalog/bucket names a table bucket; any other {prefix} names none."""
+ from ministack.services.s3tables import iceberg_iam_checks, iceberg_prefix_error
+
+ bucket = "arn:aws:s3tables:us-east-1:000000000000:bucket/b"
+ for prefix in (_arn_prefix(bucket), "000000000000:s3tablescatalog/b"):
+ assert iceberg_iam_checks("GET", f"/iceberg/v1/{prefix}/namespaces", b"", {}) == [
+ ("s3tables:ListNamespaces", bucket)]
+ assert iceberg_prefix_error("GET", f"/iceberg/v1/{prefix}/namespaces") is None
+ no_arn = "The specified table bucket ARN is not valid."
+ for prefix, message in (("catalog", no_arn), ("%6Eamespaces/000000000000:s3tablescatalog/b", no_arn),
+ (_arn_prefix(f"{bucket}/x"), "The specified bucket name isn't valid. Specify a different "
+ "bucket name, and then try again.")):
+ assert iceberg_iam_checks("POST", f"/iceberg/v1/{prefix}/namespaces", b"{}", {}) is None
+ status, _, body = iceberg_prefix_error("POST", f"/iceberg/v1/{prefix}/namespaces")
+ assert (status, json.loads(body)) == (400, {"error": {"message": message, "type": "bad_request", "code": 400}})
+ # a rename body whose source is no object names no table
+ assert iceberg_iam_checks("POST", f"/iceberg/v1/{_arn_prefix(bucket)}/tables/rename", b'{"source": "t"}', {}) == [
+ ("s3tables:RenameTable", f"{bucket}/table/*")]
+
+
+def iceberg_auth_catalog(s3tables):
+ """A table bucket with namespace ns and tables t and u, removed with whatever a test left in it."""
+ name = f"irc-{_uuid_mod.uuid4().hex[:8]}"
+ bucket = s3tables.create_table_bucket(name=name)["arn"]
+ s3tables.create_namespace(tableBucketARN=bucket, namespace=["ns"])
+ tables = {t: s3tables.create_table(tableBucketARN=bucket, namespace="ns", name=t, format="ICEBERG")["tableARN"]
+ for t in ("t", "u")}
+ yield {"bucket": bucket, "name": name, "prefix": _arn_prefix(bucket), **tables}
+ for namespace in s3tables.list_namespaces(tableBucketARN=bucket)["namespaces"]:
+ ns = namespace["namespace"][0]
+ for table in s3tables.list_tables(tableBucketARN=bucket, namespace=ns)["tables"]:
+ s3tables.delete_table(tableBucketARN=bucket, namespace=ns, name=table["name"])
+ s3tables.delete_namespace(tableBucketARN=bucket, namespace=ns)
+ s3tables.delete_table_bucket(tableBucketARN=bucket)
+
+
+def _iceberg_contract_signer(user):
+ """A second access key of an ``iam_user`` for signed Iceberg REST calls, deleted before the fixture's cleanup."""
+ from botocore.credentials import Credentials
+ from conftest import make_client
+
+ iam = make_client("iam")
+ key = iam.create_access_key(UserName=user.name)["AccessKey"]
+ try:
+ yield Credentials(key["AccessKeyId"], key["SecretAccessKey"])
+ finally:
+ iam.delete_access_key(UserName=user.name, AccessKeyId=key["AccessKeyId"])
+
+
+def _iceberg_signed(credentials, method, path, query=None, body=None, service="s3tables", region="us-east-1",
+ presigned=False):
+ """One SigV4-signed Iceberg REST call over HTTP: status, body (parsed when JSON, None when empty) and headers."""
+ from botocore.auth import S3SigV4Auth, SigV4Auth, SigV4QueryAuth
+ from botocore.awsrequest import AWSRequest
+
+ request = AWSRequest(method=method, url=f"{_ENDPOINT}/iceberg/v1/{path}", params=query or {},
+ data=json.dumps(body).encode() if body is not None else None)
+ if presigned:
+ SigV4QueryAuth(credentials, service, region, expires=300).add_auth(request)
+ else:
+ (S3SigV4Auth if service == "s3" else SigV4Auth)(credentials, service, region).add_auth(request)
+ prepared = request.prepare()
+ try:
+ with urllib.request.urlopen(urllib.request.Request(
+ prepared.url, data=prepared.body, method=method, headers=dict(prepared.headers))) as response:
+ status, content, headers = response.status, response.read(), response.headers
+ except urllib.error.HTTPError as error:
+ status, content, headers = error.code, error.read(), error.headers
+ headers = {k.lower(): v for k, v in headers.items()}
+ if content and "json" in headers.get("content-type", ""):
+ content = json.loads(content)
+ return status, content or None, headers
+
+
+def _iceberg_fill(value, names):
+ """A path, query or body template with {name} placeholders filled from ``names``."""
+ text = json.dumps(value)
+ for key, name in names.items():
+ text = text.replace("{" + key + "}", name)
+ return json.loads(text)
+
+
+def _iceberg_denial(user, action, resource, reason=None):
+ """The Iceberg error body of an s3tables identity-policy denial."""
+ reason = reason or f"because no identity-based policy allows the s3tables:{action} action"
+ return {"error": {"message": f"User: {user.arn} is not authorized to perform: s3tables:{action} "
+ f"on resource: {resource} {reason}", "type": "forbidden", "code": 403}}
+
+
+# Iceberg REST routes on the S3 Tables catalog: method, path, query, body, the actions AWS lists for each (on the
+# table bucket or table), and MiniStack's status once they are allowed.
+_UPDATE_TABLE_T = [("UpdateTableMetadataLocation", "{t}"), ("PutTableData", "{t}"), ("GetTableData", "{t}")]
+_S3TABLES_ICEBERG_ROUTES = {
+ "config": ("GET", "config", {"warehouse": "{bucket}"}, None, [("GetTableBucket", "{bucket}")], 200),
+ "list-namespaces": ("GET", "{prefix}/namespaces", None, None, [("ListNamespaces", "{bucket}")], 200),
+ "create-namespace": ("POST", "{prefix}/namespaces", None, {"namespace": ["{new}"]},
+ [("CreateNamespace", "{bucket}")], 200),
+ "load-namespace": ("GET", "{prefix}/namespaces/ns", None, None, [("GetNamespace", "{bucket}")], 200),
+ "drop-namespace": ("DELETE", "{prefix}/namespaces/ns", None, None, [("DeleteNamespace", "{bucket}")], 404),
+ "list-tables": ("GET", "{prefix}/namespaces/ns/tables", None, None, [("ListTables", "{bucket}")], 200),
+ "create-table": ("POST", "{prefix}/namespaces/ns/tables", None, {"name": "{new}"},
+ [("CreateTable", "{bucket}"), ("PutTableData", "{bucket}/table/*")], 200),
+ "load-table": ("GET", "{prefix}/namespaces/ns/tables/t", None, None,
+ [("GetTableMetadataLocation", "{t}"), ("GetTableData", "{t}")], 200),
+ "update-table": ("POST", "{prefix}/namespaces/ns/tables/t", None, {"requirements": [], "updates": []},
+ _UPDATE_TABLE_T, 200),
+ "drop-table": ("DELETE", "{prefix}/namespaces/ns/tables/t", None, None, [("DeleteTable", "{t}")], 404),
+ "table-exists": ("HEAD", "{prefix}/namespaces/ns/tables/t", None, None, [("GetTable", "{t}")], 200),
+ "rename-table": ("POST", "{prefix}/tables/rename", None, {"source": {"namespace": ["ns"], "name": "t"},
+ "destination": {"namespace": ["ns"], "name": "{new}"}},
+ [("RenameTable", "{t}")], 404),
+ "commit-transaction": ("POST", "{prefix}/transactions/commit", None, {"table-changes": [
+ {"identifier": {"namespace": ["ns"], "name": "t"}, "requirements": [], "updates": []}]}, _UPDATE_TABLE_T, 200),
+}
+
+
[email protected]("service", ["s3tables", "glue", "s3", "sts"])
[email protected]("route", list(_S3TABLES_ICEBERG_ROUTES))
+def test_s3tables_iceberg_route_contract(auth_enabled, iam_user, iceberg_auth_catalog, route, service):
+ """Each Iceberg route on a table bucket needs exactly its s3tables actions, whatever service it is signed for."""
+ method, path, query, body, actions, served = _S3TABLES_ICEBERG_ROUTES[route]
+ actions = _iceberg_fill(actions, iceberg_auth_catalog)
+ allowed = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": f"s3tables:{action}", "Resource": resource} for action, resource in actions]})
+ nobody = iam_user()
+ with _iceberg_contract_signer(allowed) as signer, _iceberg_contract_signer(nobody) as refused_signer:
+ calls = [_iceberg_signed(credentials, method, *_iceberg_fill([path, query, body], {
+ **iceberg_auth_catalog, "new": f"by-{label}"}), service=service)
+ for label, credentials in (("allowed", signer), ("nobody", refused_signer))]
+ assert calls[0][0] == served
+ if not auth_enabled:
+ assert calls[1][0] == served
+ return
+ status, error, headers = calls[1]
+ assert (status, headers["content-type"], headers["x-amzn-errortype"]) == (
+ 403, "application/json", "IrcForbiddenResponse")
+ assert error == (None if method == "HEAD" else _iceberg_denial(nobody, *actions[0]))
+
+
[email protected]("method,path,body,action,resource,reason", [
+ ("GET", "config?warehouse={absent}", None, "GetTableBucket", "{absent}", None),
+ ("POST", "{prefix}/namespaces", {"namespace": ["ns2"]}, "CreateNamespace", "{bucket}",
+ "with an explicit deny in an identity-based policy"),
+ ("GET", "{prefix}/namespaces/ns/tables/u", None, "GetTableMetadataLocation", "{u}", None),
+ ("GET", "{prefix}/namespaces/ns/tables/absent", None, "GetTableMetadataLocation", "{bucket}/table/*", None),
+], ids=["config-absent-bucket", "explicit-deny", "other-table", "absent-table"])
+def test_s3tables_iceberg_authorization_contract(auth_enabled, iam_user, iceberg_auth_catalog,
+ method, path, body, action, resource, reason):
+ """A denial names the bucket or table the call names, table/* for a table that does not exist, and a Deny."""
+ catalog = iceberg_auth_catalog
+ user = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Resource": catalog["bucket"],
+ "Action": ["s3tables:GetTableBucket", "s3tables:ListNamespaces", "s3tables:GetNamespace"]},
+ {"Effect": "Allow", "Action": "s3tables:GetTableMetadataLocation", "Resource": catalog["t"]},
+ {"Effect": "Deny", "Action": "s3tables:CreateNamespace", "Resource": catalog["bucket"]},
+ ]})
+ names = {**catalog, "absent": catalog["bucket"].rsplit("/", 1)[0] + "/absent"}
+ path, query = _iceberg_fill(path, names), None
+ if "?warehouse=" in path:
+ path, warehouse = path.split("?warehouse=")
+ query = {"warehouse": warehouse}
+ with _iceberg_contract_signer(user) as signer:
+ assert _iceberg_signed(signer, "GET", "config", {"warehouse": catalog["bucket"]})[0] == 200
+ assert _iceberg_signed(signer, "GET", f"{catalog['prefix']}/namespaces")[1] == {"namespaces": [["ns"]]}
+ status, error, headers = _iceberg_signed(signer, method, path, query, body)
+ if not auth_enabled:
+ assert status != 403
+ return
+ assert (status, headers["content-type"]) == (403, "application/json")
+ assert error == _iceberg_denial(user, action, _iceberg_fill(resource, names), reason)
+
+
+def test_s3tables_iceberg_later_action_contract(auth_enabled, iam_user, iceberg_auth_catalog):
+ """loadTable's second action answers Access Denied."""
+ catalog = iceberg_auth_catalog
+ user = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": "s3tables:GetTableMetadataLocation", "Resource": catalog["t"]}]})
+ with _iceberg_contract_signer(user) as signer:
+ loaded = _iceberg_signed(signer, "GET", f"{catalog['prefix']}/namespaces/ns/tables/t")
+ if not auth_enabled:
+ assert loaded[0] == 200
+ return
+ assert loaded[:2] == (403, {"error": {"message": "Access Denied", "type": "forbidden", "code": 403}})
+
+
[email protected]("service", ["s3tables", "glue"])
+def test_s3tables_iceberg_wrong_secret_contract(auth_enabled, iam_user, iceberg_auth_catalog, service):
+ """Iceberg REST calls check their signature, for S3 Tables and for the Glue catalog alike, in JSON."""
+ from botocore.credentials import Credentials
+
+ catalog = iceberg_auth_catalog
+ warehouse = catalog["bucket"] if service == "s3tables" else "000000000000"
+ action = "s3tables:GetTableBucket" if service == "s3tables" else "glue:GetCatalog"
+ user = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": action, "Resource": "*"}]})
+ with _iceberg_contract_signer(user) as signer:
+ assert _iceberg_signed(signer, "GET", "config", {"warehouse": warehouse}, service=service)[0] == 200
+ status, _, headers = _iceberg_signed(Credentials(signer.access_key, "wrong-secret"), "GET", "config",
+ {"warehouse": warehouse}, service=service)
+ if not auth_enabled:
+ assert status == 200
+ return
+ assert (status, headers["x-amzn-errortype"], headers["content-type"]) == (
+ 403, "InvalidSignatureException", "application/json")
+
+
+def test_s3tables_iceberg_calls_naming_no_table_bucket_contract(auth_enabled, iam_user):
+ """An unprefixed route answers 404 whatever its signature; config without a table bucket answers 400."""
+ from botocore.credentials import Credentials
+
+ with _iceberg_contract_signer(iam_user()) as signer:
+ assert _iceberg_signed(signer, "GET", "config")[0] == 400
+ wrong = Credentials(signer.access_key, "wrong-secret")
+ assert _iceberg_signed(wrong, "GET", "namespaces")[0] == 404
+ assert _iceberg_signed(wrong, "HEAD", "namespaces/ns")[0] == 404
+ status, _, headers = _iceberg_signed(wrong, "GET", "config")
+ read_as_prefix = _iceberg_signed(wrong, "GET", "tables/namespaces", service="glue")
+ assert (status, headers.get("x-amzn-errortype")) == (
+ (403, "InvalidSignatureException") if auth_enabled else (400, None))
+ # "tables" before "namespaces" is the {prefix}, so that call is authenticated however it is signed
+ assert read_as_prefix[0] == (403 if auth_enabled else 200)
+
+
[email protected]("method,prefix,message,served", [
+ ("GET", "somewhere", "The specified table bucket ARN is not valid.", [200, 200, 200]),
+ ("POST", "%256Eamespaces/000000000000:s3tablescatalog/{name}", "The specified table bucket ARN is not valid.",
+ [200, 409, 409]),
+ ("POST", "{bucket}/x", "The specified bucket name isn't valid. Specify a different bucket name, and then try "
+ "again.", [404, 404, 404]),
+], ids=["list-namespaces", "encoded-namespaces-segment", "bucket-arn-and-more"])
+def test_s3tables_iceberg_prefix_naming_no_table_bucket_contract(auth_enabled, iam_user, iceberg_auth_catalog,
+ s3tables, method, prefix, message, served):
+ """A {prefix} that is no table bucket answers 400 after the credential check, before any policy, as on AWS."""
+ from botocore.credentials import Credentials
+
+ catalog = iceberg_auth_catalog
+ prefix = _iceberg_fill(prefix, catalog)
+ prefix = _arn_prefix(prefix) if prefix.startswith("arn:") else prefix
+ everything = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": "s3tables:*", "Resource": "*"}]})
+ body = {"namespace": ["created"]} if method == "POST" else None
+ with _iceberg_contract_signer(iam_user()) as nobody, _iceberg_contract_signer(everything) as signer:
+ calls = [_iceberg_signed(credentials, method, f"{prefix}/namespaces", body=body)
+ for credentials in (nobody, signer, Credentials(signer.access_key, "wrong-secret"))]
+ if not auth_enabled:
+ assert [status for status, _, _ in calls] == served
+ return
+ for status, error, _ in calls[:2]:
+ assert (status, error) == (400, {"error": {"message": message, "type": "bad_request", "code": 400}})
+ assert (calls[2][0], calls[2][2]["x-amzn-errortype"]) == (403, "InvalidSignatureException")
+ assert [n["namespace"] for n in s3tables.list_namespaces(tableBucketARN=catalog["bucket"])["namespaces"]] == [
+ ["ns"]]
+
+
[email protected]("prefix", ["{prefix}/%6Eamespaces", "000000000000:s3tablescatalog/{name}/namespaces"],
+ ids=["encoded-namespaces-keyword", "account-catalog-prefix"])
+def test_s3tables_iceberg_prefix_forms_served_are_authorized_contract(auth_enabled, iam_user, iceberg_auth_catalog,
+ prefix):
+ """A path MiniStack serves as listNamespaces is authorized as one, whichever form names the bucket."""
+ catalog = iceberg_auth_catalog
+ path = _iceberg_fill(prefix, catalog)
+ allowed = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": "s3tables:ListNamespaces", "Resource": catalog["bucket"]}]})
+ nobody = iam_user()
+ with _iceberg_contract_signer(allowed) as signer, _iceberg_contract_signer(nobody) as refused_signer:
+ listed = _iceberg_signed(signer, "GET", path)
+ refused = _iceberg_signed(refused_signer, "GET", path)
+ assert listed[:2] == (200, {"namespaces": [["ns"]]})
+ if not auth_enabled:
+ assert refused[:2] == listed[:2]
+ return
+ assert refused[:2] == (403, _iceberg_denial(nobody, "ListNamespaces", catalog["bucket"]))
+
+
+def test_s3tables_iceberg_presigned_region_contract(auth_enabled, iam_user, iceberg_auth_catalog):
+ """Under AUTH a presigned call serves its credential region, whose bucket and table the checks read."""
+ catalog = iceberg_auth_catalog
+ west = _make_s3tables_client(region_name="us-west-2")
+ west_arn = west.create_table_bucket(name=catalog["name"])["arn"]
+ west.create_namespace(tableBucketARN=west_arn, namespace=["ns"])
+ secret = west.create_table(tableBucketARN=west_arn, namespace="ns", name="secret", format="ICEBERG")["tableARN"]
+ reads = ["s3tables:GetTableMetadataLocation", "s3tables:GetTableData"]
+ user = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": "s3tables:ListNamespaces", "Resource": catalog["bucket"]},
+ {"Effect": "Allow", "Action": reads, "Resource": "*"},
+ {"Effect": "Deny", "Action": reads, "Resource": secret}]})
+ by_name = f"000000000000:s3tablescatalog/{catalog['name']}/namespaces"
+ table = f"{_arn_prefix(west_arn)}/namespaces/ns/tables/secret"
+ try:
+ with _iceberg_contract_signer(user) as signer:
+ calls = {(region, kind): _iceberg_signed(signer, "GET", path, region=region, presigned=True)
+ for region in ("us-east-1", "us-west-2") for kind, path in (("list", by_name), ("table", table))}
+ finally:
+ west.delete_table(tableBucketARN=west_arn, namespace="ns", name="secret")
+ west.delete_namespace(tableBucketARN=west_arn, namespace="ns")
+ west.delete_table_bucket(tableBucketARN=west_arn)
+ if not auth_enabled:
+ assert [status for status, _, _ in calls.values()] == [200, 200, 200, 200]
+ return
+ # signed for us-east-1, neither call reaches the us-west-2 bucket
+ assert calls["us-east-1", "list"][:2] == (200, {"namespaces": [["ns"]]})
+ assert calls["us-east-1", "table"][0] == 404
+ assert calls["us-west-2", "list"][:2] == (403, _iceberg_denial(user, "ListNamespaces", west_arn))
+ assert calls["us-west-2", "table"][:2] == (403, _iceberg_denial(
+ user, "GetTableMetadataLocation", secret, "with an explicit deny in an identity-based policy"))
+
+
[email protected]("warehouse", ["1:s3tablescatalog/{name}", "000000000000:s3tablescatalog/{name}/x"],
+ ids=["short-account", "trailing-segment"])
+def test_s3tables_iceberg_config_catalog_warehouse_contract(auth_enabled, iam_user, iceberg_auth_catalog, warehouse):
+ """getConfig serves any account:s3tablescatalog/ warehouse; one naming no bucket needs GetTableBucket on *."""
+ warehouse = _iceberg_fill(warehouse, iceberg_auth_catalog)
+ nobody = iam_user()
+ allowed = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": "s3tables:GetTableBucket", "Resource": "*"}]})
+ with _iceberg_contract_signer(allowed) as signer, _iceberg_contract_signer(nobody) as refused_signer:
+ served = _iceberg_signed(signer, "GET", "config", {"warehouse": warehouse})
+ refused = _iceberg_signed(refused_signer, "GET", "config", {"warehouse": warehouse})
+ assert served[0] == 200
+ if not auth_enabled:
+ assert refused[0] == 200
+ return
+ assert refused[:2] == (403, _iceberg_denial(nobody, "GetTableBucket", "*"))
+
+
+def test_s3tables_iceberg_table_named_rename_contract(auth_enabled, iam_user, iceberg_auth_catalog, s3tables):
+ """A commit to a table named rename is updateTable on that table, not renameTable."""
+ catalog = iceberg_auth_catalog
+ table = s3tables.create_table(tableBucketARN=catalog["bucket"], namespace="ns", name="rename",
+ format="ICEBERG")["tableARN"]
+ path = f"{catalog['prefix']}/namespaces/ns/tables/rename"
+ renamer = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": "s3tables:RenameTable", "Resource": "*"}]})
+ updater = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": f"s3tables:{action}", "Resource": table} for action, _ in _UPDATE_TABLE_T]})
+ commit = {"requirements": [], "updates": []}
+ with _iceberg_contract_signer(renamer) as refused_signer, _iceberg_contract_signer(updater) as signer:
+ assert _iceberg_signed(signer, "POST", path, body=commit)[0] == 200
+ refused = _iceberg_signed(refused_signer, "POST", path, body=commit)
+ if not auth_enabled:
+ assert refused[0] == 200
+ return
+ assert refused[:2] == (403, _iceberg_denial(renamer, "UpdateTableMetadataLocation", table))
+
+
[email protected]("body", ["t", "t-and-u", "no-changes", "no-object"])
+def test_s3tables_iceberg_transaction_commit_contract(auth_enabled, iam_user, iceberg_auth_catalog, body):
+ """transactions/commit needs updateTable's actions on each table it changes, on * when it names none."""
+ catalog = iceberg_auth_catalog
+ change = {"identifier": {"namespace": ["ns"], "name": "t"}, "requirements": [], "updates": []}
+ bodies = {"t": {"table-changes": [change]},
+ "t-and-u": {"table-changes": [change, {**change, "identifier": {"namespace": ["ns"], "name": "u"}}]},
+ "no-changes": {}, "no-object": "table-changes"}
+ allowed = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": f"s3tables:{action}", "Resource": catalog["t"]}
+ for action, _ in _UPDATE_TABLE_T]})
+ nobody = iam_user()
+ path = f"{catalog['prefix']}/transactions/commit"
+ with _iceberg_contract_signer(allowed) as signer, _iceberg_contract_signer(nobody) as refused_signer:
+ committed = _iceberg_signed(signer, "POST", path, body=bodies[body])
+ refused = _iceberg_signed(refused_signer, "POST", path, body=bodies[body])
+ if not auth_enabled:
+ assert (committed[0], refused[0]) == ((500, 500) if body == "no-object" else (200, 200))
+ return
+ first = "*" if body in ("no-changes", "no-object") else catalog["t"]
+ assert refused[:2] == (403, _iceberg_denial(nobody, "UpdateTableMetadataLocation", first))
+ if body == "t":
+ assert committed[0] == 200
+ elif body == "t-and-u":
+ assert committed[:2] == (403, {"error": {"message": "Access Denied", "type": "forbidden", "code": 403}})
+ else:
+ assert committed[:2] == (403, _iceberg_denial(allowed, "UpdateTableMetadataLocation", "*"))
+
+
def test_s3tables_bucket_subresources_defaults_and_updates(s3tables):
"""Documented defaults until set: SSE-S3, STANDARD, unreferenced file removal 3/10 days."""
arn = s3tables.create_table_bucket(name=f"sub-{_uuid_mod.uuid4().hex[:8]}",