Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
feat(auth): authorize S3 Tables and Glue Iceberg REST calls under AUTH=true (#2289)
iot-rockete2302e95ccde
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7729ea19..fd2269f5 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,10 +8,12 @@ Versioning follows [Semantic Versioning](https://semver.org/).## [Unreleased]### Added-- **Auth — WebSocket handshakes are verified for IAM user and STS session keys** — under `AUTH=true` an MQTT-over-WebSocket upgrade or an `AWS_IAM` WebSocket `$connect` signed with such a key was accepted with a wrong secret, a date more than 5 minutes off or a wrong session token; the signature in the query string or `Authorization` header is now checked like that of other requests, IoT answers 403 `Forbidden` and expects the session token after the signed part of the URL, and API Gateway names the canonical string it expected. Contributed by @iot-rocket.+- **S3 Tables, Glue — Iceberg REST calls are authorized under `AUTH=true`** — the Iceberg REST catalog at `/iceberg` served S3 Tables and Glue catalog calls without checking their credentials; each call is now authorized for the catalog that serves it, whatever service it is signed for: on a table bucket as the `s3tables` actions AWS lists for its operation (`loadTable` needs `GetTableMetadataLocation`, then `GetTableData`) on the bucket or table (`table/*` for one that does not exist), on the Glue catalog as its `glue` action on the catalog, then the database, then the table, and `transactions/commit` as `updateTable` on each table it changes. A denial answers 403 in the Iceberg error shape naming the action and resource ("Access Denied" for a later S3 Tables action, no body for `HEAD`), and a `{prefix}` that is neither a table bucket ARN nor `<account>:s3tablescatalog/<bucket>` answers 400 `bad_request`, as AWS does. Contributed by @iot-rocket.+- **RDS Data, S3 Control — calls are authorized under `AUTH=true`** — RDS Data calls and S3 Control tagging calls were served whatever the caller's policies allowed; each RDS Data call is now authorized as `rds-data:<operation>` on the cluster its `resourceArn` names, and `ListTagsForResource`, `TagResource` and `UntagResource` as `s3:<operation>` on the bucket, with a 403 denial (`AccessDeniedException` from RDS Data, `AccessDenied` from S3 Control) naming the action and resource, as on AWS. Contributed by @iot-rocket.+- **SES v2 — calls are authorized under `AUTH=true`** — SES v2 calls were served whatever the caller's policies allowed; each is now authorized as `ses:<operation>` on the configuration set, contact list, custom verification email template, dedicated IP pool, identity or template its path or create body names (`*` otherwise), and a denial answers 403 `AccessDeniedException` naming the action and resource, as on AWS. Contributed by @iot-rocket.### Fixed-- **Auth — SigV4 signatures that name no access key are refused** — under `AUTH=true` an `AWS4-HMAC-SHA256` `Authorization` header or a presigned query string without its credential, or with an empty access key, was served as the account root; it now answers `IncompleteSignature` naming every missing part, or the unknown-key error for an empty key, in each service's code and status, and Lambda answers a part that is left out as `MissingAuthenticationTokenException`. S3, API Gateway and Lambda function URLs, which take anonymous calls, are unchanged. Contributed by @iot-rocket.+- **Auth — requests that map to no IAM action are authenticated** — under `AUTH=true` a request MiniStack maps to no IAM action (SES v2, RDS Data, S3 Control, or a service MiniStack does not implement such as X-Ray) was served with a wrong secret, an unknown key, a session key without its token or a stale signature; it now answers the credential error AWS sends for the service, for example `InvalidSignatureException` 403 from SES v2 and RDS Data and `SignatureDoesNotMatch` 403 from S3 Control. Contributed by @iot-rocket.## [1.5.29] — 2026-10-17diff --git a/ministack/app.py b/ministack/app.pyindex 74c79f8d..f739eec6 100644--- a/ministack/app.py+++ b/ministack/app.py@@ -2428,6 +2428,63 @@ def _enforce_ses_v2(method: str, path: str, headers: dict, body: bytes, query_panamed_resource=resource_arn)+def _iceberg_json_failure(failure):+ """A credential failure as both Iceberg REST catalogs send it, in application/json."""+ status, resp_headers, resp_body = failure+ return status, {**resp_headers, "Content-Type": "application/json"}, resp_body+++def _enforce_iceberg(method: str, path: str, headers: dict, body: bytes, query_params: dict, request_id: str):+ """Authorize an Iceberg REST call for the catalog the router serves it from, whatever it is signed for."""+ from ministack.core.iam_actions import identity_denial_message+ from ministack.core.iam_evaluator import AuthError, enforce++ glue = _iceberg_targets_glue_catalog(path, query_params)+ service = "glue" if glue else "s3tables"+ catalog = _get_module(service)+ # A route without the table bucket prefix answers 404 before any credential check.+ if not glue and catalog.iceberg_lacks_prefix(method, path):+ return None+ checks = catalog.iceberg_iam_checks(method, path, body, query_params)+ if not checks:+ if failure := _authenticate_only(service, headers, query_params, request_id):+ return _iceberg_json_failure(failure)+ return None if glue else catalog.iceberg_prefix_error(method, path)+ access_key, region = extract_access_key_id(headers, query_params), extract_region(headers, query_params)+ for position, (iam_action, resource_arn) in enumerate(checks):+ denied = enforce(access_key, iam_action, service, region, resource_arn=resource_arn)+ if isinstance(denied, AuthError):+ return _iceberg_json_failure(_credential_failure(service, iam_action, request_id, denied, headers))+ if not denied:+ continue+ resp_headers = {"Content-Type": "application/json",+ "x-amzn-errortype": "ForbiddenResponse" if glue else "IrcForbiddenResponse"}+ if method == "HEAD":+ return 403, resp_headers, b""+ # S3 Tables names only the first action's denial; a later one answers "Access Denied".+ message = identity_denial_message(iam_action, denied.principal_arn, resource_arn,+ denied.decision == "Deny") if glue or position == 0 else "Access Denied"+ error = {"message": message, "type": "AccessDeniedException" if glue else "forbidden", "code": 403}+ return 403, resp_headers, json.dumps({"error": error}).encode()+ return None+++def _enforce_ses_v2(method: str, path: str, headers: dict, body: bytes, query_params: dict, request_id: str):+ """Authorize an SES v2 call as ses:<operation> on the resource it names, which its denial names too."""+ if not AUTH:+ return None+ from ministack.core.iam_actions import extract_iam_action, extract_resource_arn+ from ministack.core.responses import get_account_id++ iam_action = extract_iam_action("ses", method, path, headers, body, query_params)+ if iam_action is None:+ return _authenticate_only("sesv2", headers, query_params, request_id)+ resource_arn = extract_resource_arn("ses", method, path, headers, body, query_params,+ extract_region(headers, query_params), get_account_id())+ return _enforce_data_plane("sesv2", iam_action, headers, query_params, request_id, resource_arn=resource_arn,+ named_resource=resource_arn)++def _iceberg_targets_glue_catalog(path, query_params):"""Decide whether an ``/iceberg`` REST request belongs to the Glue DataCatalog (Glue jobs + Firehose lakehouse) or to S3 Tables.@@ -2477,6 +2534,8 @@ async def _handle_special_data_plane_request(# the other reads. Must not fall through to S3 (which reads "iceberg" as a# bucket name).if path.startswith("/iceberg"):+ if AUTH and (denied := _enforce_iceberg(method, path, headers, body, query_params, request_id)):+ return deniedtry:if _iceberg_targets_glue_catalog(path, query_params):result = _get_module("glue")._handle_iceberg_rest(method, path, query_params, body=body)diff --git a/ministack/services/glue.py b/ministack/services/glue.pyindex 33c3cfb7..c67dc312 100644--- a/ministack/services/glue.py+++ b/ministack/services/glue.py@@ -711,19 +711,91 @@ def _iceberg_rest_commit_table(namespace, table_name, body):def _iceberg_rest_commit_transaction(body):"""POST transactions/commit -- DuckDB's atomic multi-table commit."""for change in body.get("table-changes", []):- ident = change.get("identifier", {})- ns_field = ident.get("namespace", [""])- ns = ns_field[0] if isinstance(ns_field, list) and ns_field else (ns_field or "")- tbl = ident.get("name", "")+ ns, tbl = _iceberg_rest_change_identifier(change)result = _iceberg_rest_commit_table(ns, tbl, change)if result and result[0] not in (200, 204):return resultreturn _iceberg_json({})+# Iceberg REST calls below namespaces/ -> the glue action they need and how many of catalog, database, table it names.+_ICEBERG_IAM_ACTIONS = {+ (0, "GET"): ("GetDatabases", 1),+ (0, "POST"): ("CreateDatabase", 2),+ (1, "GET"): ("GetDatabase", 2),+ (1, "HEAD"): ("GetDatabase", 2),+ (1, "DELETE"): ("DeleteDatabase", 2),+ (2, "GET"): ("GetTables", 2),+ (2, "POST"): ("CreateTable", 3),+ (3, "GET"): ("GetTable", 3),+ (3, "HEAD"): ("GetTable", 3),+ (3, "POST"): ("UpdateTable", 3),+ (3, "DELETE"): ("DeleteTable", 3),+}+++def iceberg_iam_checks(method, path, body, query_params):+ """The glue actions an Iceberg REST call on the Glue catalog needs, with resources, in order; else None."""+ parts = _iceberg_rest_parts(path)[2:]+ catalog = f"arn:aws:glue:{get_region()}:{get_account_id()}:catalog"+ if parts == ["config"] and method == "GET":+ return [("glue:GetCatalog", catalog)]+ # Every {catalog} is served from the account's default catalog, so it is authorized as that one.+ if parts[:1] == ["catalogs"] and parts[2:3] == ["transactions"] and method == "POST":+ # Each table change is an updateTable; a body whose changes cannot be read needs it on every resource.+ try:+ tables = [_iceberg_rest_change_identifier(change)+ for change in json.loads(body or b"{}").get("table-changes", [])]+ except (ValueError, TypeError, AttributeError):+ tables = []+ if not tables:+ return [("glue:UpdateTable", "*")]+ return [("glue:UpdateTable", arn) for namespace, table in tables+ for arn in (catalog, _arn("database", namespace), _arn("table", f"{namespace}/{table}"))]+ if parts[:1] != ["catalogs"] or parts[2:3] != ["namespaces"]:+ return None+ rest = parts[3:]+ if rest[1:] == ["properties"] and method == "POST":+ resources = [catalog, _arn("database", rest[0])]+ return [(f"glue:{action}", arn) for action in ("GetDatabase", "UpdateDatabase") for arn in resources]+ entry = _ICEBERG_IAM_ACTIONS.get((len(rest), method))+ if not entry or rest[1:2] not in ([], ["tables"]):+ return None+ action, depth = entry+ namespace, _, table = rest + [""] * (3 - len(rest))+ if method == "POST" and len(rest) in (0, 2):+ named = _iceberg_body_name(body, "namespace" if not rest else "name")+ namespace, table = (named, table) if not rest else (namespace, named)+ resources = [catalog, _arn("database", namespace), _arn("table", f"{namespace}/{table}")]+ return [(f"glue:{action}", arn) for arn in resources[:depth]]+++def _iceberg_rest_parts(path):+ """The segments of an Iceberg REST path as the Glue router reads them, each decoded on its own."""+ return [unquote(p) for p in path.strip("/").split("/") if p]+++def _iceberg_rest_change_identifier(change):+ """The namespace and table name a transactions/commit table change names, read as the commit reads them."""+ ident = change.get("identifier", {})+ ns_field = ident.get("namespace", [""])+ return (ns_field[0] if isinstance(ns_field, list) and ns_field else (ns_field or "")), ident.get("name", "")+++def _iceberg_body_name(body, field):+ """The namespace or table name an Iceberg REST request body names, read as the handlers read it, or ''."""+ try:+ name = json.loads(body or b"{}").get(field, "")+ except (ValueError, TypeError, AttributeError):+ return ""+ if field == "namespace":+ name = name[0] if isinstance(name, list) and name else (name or "")+ return str(name)++def _handle_iceberg_rest(method, path, query_params, body=None):_iceberg_maybe_log_tls_hint()- parts = [unquote(p) for p in path.strip("/").split("/") if p]+ parts = _iceberg_rest_parts(path)if len(parts) < 3 or parts[0] != "iceberg" or parts[1] != "v1":return _iceberg_error(f"Unknown Iceberg REST path: {path}", "NotFoundException", 404)diff --git a/ministack/services/s3tables.py b/ministack/services/s3tables.pyindex 232187a9..4d0f6932 100644--- a/ministack/services/s3tables.py+++ b/ministack/services/s3tables.py@@ -689,7 +689,13 @@ def _in_bucket(bucket_arn: str, bucket_filter) -> bool:return bucket_arn.endswith("/" + bucket_filter)-def _iceberg_allows_cross_region(headers):+def _iceberg_allows_cross_region(headers, query_params=None):+ """Whether a call names no signing region; under AUTH a presigned call names it in its query, as checks read it."""+ if query_params and "X-Amz-Credential" in query_params:+ from ministack.app import AUTH++ if AUTH:+ return Falsereturn _SIGV4_CREDENTIAL_REGION_RE.search(headers.get("authorization", "")) is None@@ -1102,14 +1108,141 @@ def _unknown_operation():return 404, {"Content-Type": "application/xml"}, b"<UnknownOperationException/>"+# Iceberg REST calls on a namespace or table -> the s3tables actions they need, in order.+_ICEBERG_NAMESPACE_ACTIONS = {+ (0, "GET"): "ListNamespaces",+ (0, "POST"): "CreateNamespace",+ (1, "GET"): "GetNamespace",+ (1, "HEAD"): "GetNamespace",+ (1, "DELETE"): "DeleteNamespace",+}+_ICEBERG_TABLE_ACTIONS = {+ (2, "GET"): ("ListTables",),+ (2, "POST"): ("CreateTable", "PutTableData"),+ (3, "GET"): ("GetTableMetadataLocation", "GetTableData"),+ (3, "POST"): ("UpdateTableMetadataLocation", "PutTableData", "GetTableData"),+ (3, "DELETE"): ("DeleteTable",),+ (3, "HEAD"): ("GetTable",),+}+_ICEBERG_BUCKET_ACTIONS = frozenset({"ListTables", "CreateTable"})+_TABLE_BUCKET_CATALOG_RE = re.compile(r"\d{12}:s3tablescatalog/([a-z0-9-]+)")+++def _iceberg_route(method, path):+ """The route the Iceberg REST router reads from a path: (route, raw {prefix} segments, raw segments after it)."""+ parts = [p for p in path.strip("/").split("/") if p]+ if parts[1:2] == ["v1"] and parts[2:] == ["config"] and method == "GET":+ return "config", [], []+ if parts[1:2] == ["v1"] and method == "POST" and "transactions" in parts[2:]:+ index = parts.index("transactions", 2)+ return "transactions", parts[2:index], parts[index + 1:]+ if "namespaces" in parts[2:]:+ index = parts.index("namespaces", 2)+ return "namespaces", parts[2:index], parts[index + 1:]+ if method == "POST" and len(parts) > 3 and parts[-2:] == ["tables", "rename"]:+ return "rename", parts[2:-2], []+ return None+++def _iceberg_change_identifier(change):+ """The namespace and table name a transactions/commit table change names, read as the commit reads them."""+ ident = change.get("identifier", {})+ ns = ident.get("namespace", [""])+ return (ns[0] if isinstance(ns, list) else ns), ident.get("name", "")+++def iceberg_iam_checks(method, path, body, query_params):+ """The s3tables actions an Iceberg REST call on a table bucket needs, with resources, in AWS's order; else None."""+ route = _iceberg_route(method, path)+ if not route:+ return None+ kind, prefix, rest = route+ if kind == "config":+ warehouse = query_params.get("warehouse", "") if query_params else ""+ warehouse = (warehouse[0] if warehouse else "") if isinstance(warehouse, list) else warehouse+ bucket_arn = _iceberg_bucket_arn(warehouse)+ # getConfig serves any account:s3tablescatalog/ warehouse; one that names no bucket needs it on every bucket.+ if bucket_arn != "*" or ":s3tablescatalog/" in warehouse:+ return [("s3tables:GetTableBucket", bucket_arn)]+ return None+ bucket_arn = _iceberg_bucket_arn(unquote("/".join(prefix)))+ if bucket_arn == "*":+ return None+ if kind == "rename":+ try:+ source = json.loads(body or b"{}").get("source") or {}+ namespace, name = source.get("namespace") or [""], source.get("name", "")+ except (ValueError, AttributeError):+ namespace, name = [""], ""+ namespace = namespace[0] if isinstance(namespace, list) else namespace+ return [("s3tables:RenameTable", _iceberg_table_arn(bucket_arn, namespace, name))]+ if kind == "transactions":+ # Each table change is an updateTable; a body whose changes cannot be read needs them on every resource.+ try:+ data = json.loads(body) if body else {}+ tables = [_iceberg_change_identifier(change) for change in data.get("table-changes", [])]+ except (ValueError, TypeError, AttributeError, IndexError):+ tables = []+ arns = [_iceberg_table_arn(bucket_arn, ns, name) for ns, name in tables] or ["*"]+ return [(f"s3tables:{name}", arn) for arn in arns for name in _ICEBERG_TABLE_ACTIONS[3, "POST"]]+ if len(rest) < 2:+ action = _ICEBERG_NAMESPACE_ACTIONS.get((len(rest), method))+ return [(f"s3tables:{action}", bucket_arn)] if action else None+ if rest[1] != "tables" or len(rest) > 3:+ return None+ table_arn = _iceberg_table_arn(bucket_arn, rest[0], rest[2] if len(rest) == 3 else "")+ names = _ICEBERG_TABLE_ACTIONS.get((len(rest), method), ())+ return [(f"s3tables:{name}", bucket_arn if name in _ICEBERG_BUCKET_ACTIONS else table_arn)+ for name in names] or None+++def iceberg_lacks_prefix(method, path):+ """Whether an Iceberg REST route leaves out the table bucket {prefix}, which the router answers 404 unread."""+ route = _iceberg_route(method, path)+ return bool(route) and route[0] != "config" and not route[1]+++def iceberg_prefix_error(method, path):+ """The 400 for a route whose {prefix} names no table bucket, which AWS answers before any policy check."""+ route = _iceberg_route(method, path)+ if not route or route[0] == "config" or not route[1]:+ return None+ prefix = unquote("/".join(route[1]))+ if _iceberg_bucket_arn(prefix) != "*":+ return None+ if _TABLE_BUCKET_ARN_RE.match(prefix):+ message = "The specified bucket name isn't valid. Specify a different bucket name, and then try again."+ else:+ message = "The specified table bucket ARN is not valid."+ status, headers, body = _iceberg_error(message, "bad_request", 400)+ return status, headers, b"" if method == "HEAD" else body+++def _iceberg_bucket_arn(prefix):+ """The table bucket a {prefix} or warehouse names by its ARN or as account:s3tablescatalog/bucket, or '*'."""+ if _TABLE_BUCKET_ARN_RE.fullmatch(prefix):+ return prefix+ named = _TABLE_BUCKET_CATALOG_RE.fullmatch(prefix)+ return _bucket_arn(named.group(1)) if named else "*"+++def _iceberg_table_arn(bucket_arn, namespace, name):+ """A table's ARN; AWS names table/* for a table that does not exist."""+ if bucket_arn == "*":+ return "*"+ table = _tables.get(_table_key(bucket_arn, namespace, name))+ return table["tableARN"] if table else f"{bucket_arn}/table/*"++async def _handle_iceberg_request(method, path, headers, body, query_params):parts = [p for p in path.strip("/").split("/") if p]- allow_cross_region = _iceberg_allows_cross_region(headers)+ allow_cross_region = _iceberg_allows_cross_region(headers, query_params)# parts: ["iceberg", "v1", ...]if len(parts) < 2:return None- if parts[1] == "v1" and len(parts) == 3 and parts[2] == "config" and method == "GET":+ route = _iceberg_route(method, path)+ if route and route[0] == "config":return _iceberg_config(query_params, headers)# POST /iceberg/v1/transactions/commit — DuckDB atomic multi-table commit.@@ -1117,40 +1250,26 @@ async def _handle_iceberg_request(method, path, headers, body, query_params):# this arrives as /iceberg/v1/{prefix}/transactions/commit. Match on the# `transactions` segment wherever it lands, not a fixed position — otherwise# a prefixed commit is silently dropped and the delivery writes nothing.- if parts[1] == "v1" and method == "POST" and "transactions" in parts[2:]:+ if route and route[0] == "transactions":data = json.loads(body) if body else {}- tx_idx = parts.index("transactions", 2)- if tx_idx == 2:+ if not route[1]:return _unknown_operation()- tx_filter = _bucket_scope_from_prefix(unquote("/".join(parts[2:tx_idx])))+ tx_filter = _bucket_scope_from_prefix(unquote("/".join(route[1])))for change in data.get("table-changes", []):- ident = change.get("identifier", {})- ns = ident.get("namespace", [""])- ns = ns[0] if isinstance(ns, list) else ns- tbl = ident.get("name", "")+ ns, tbl = _iceberg_change_identifier(change)result = _iceberg_commit_table(ns, tbl, change, allow_cross_region, tx_filter)if result and result[0] not in (200, 204):return resultreturn json_response({})- if len(parts) < 3:- return None-# Every route but /v1/config takes the {prefix}, which "for S3 Tables is# always your url-encoded table bucket ARN"; it can span several segments.- bucket_filter = None- ns_idx = None- for i in range(2, len(parts)):- if parts[i] == "namespaces":- ns_idx = i- break- if ns_idx is None:+ if not route or route[0] != "namespaces":return None- if ns_idx == 2:+ prefix, rest = route[1], route[2] # rest: segments after "namespaces"+ if not prefix:return _unknown_operation()- bucket_filter = _bucket_scope_from_prefix(unquote("/".join(parts[2:ns_idx])))-- rest = parts[ns_idx + 1 :] # segments after "namespaces"+ bucket_filter = _bucket_scope_from_prefix(unquote("/".join(prefix)))if len(rest) == 0 and method == "GET":return _iceberg_list_namespaces(allow_cross_region, bucket_filter)diff --git a/tests/test_glue.py b/tests/test_glue.pyindex 45bc6036..34fdb4f8 100644--- a/tests/test_glue.py+++ b/tests/test_glue.py@@ -2445,6 +2445,228 @@ def test_iceberg_dispatch_routes_glue_vs_s3tables():assert targets_glue("/iceberg/v1/config", {}) is False+def test_iceberg_iam_checks_read_body_names_as_the_handlers_do():+ """The checks read createNamespace, createTable and transaction bodies as the handlers do."""+ from ministack.services.glue import iceberg_iam_checks++ arn = "arn:aws:glue:us-east-1:000000000000"+ catalog = "/iceberg/v1/catalogs/000000000000"+ assert iceberg_iam_checks("POST", f"{catalog}/namespaces", b'{"namespace": ["db"]}', {})[-1] == (+ "glue:CreateDatabase", f"{arn}:database/db")+ assert iceberg_iam_checks("POST", f"{catalog}/namespaces/db/tables", b'{"name": ["x"]}', {})[-1] == (+ "glue:CreateTable", f"{arn}:table/db/['x']")+ change = b'{"table-changes": [{"identifier": {"namespace": "db", "name": "t"}}]}'+ assert iceberg_iam_checks("POST", f"{catalog}/transactions/commit", change, {})[-1] == (+ "glue:UpdateTable", f"{arn}:table/db/t")+ for unread in (b"", b"[]", b'{"table-changes": [[]]}'):+ assert iceberg_iam_checks("POST", f"{catalog}/transactions/commit", unread, {}) == [("glue:UpdateTable", "*")]+++def _glue_iceberg_signed(credentials, method, path, body=None, query=None, service="glue"):+ """One Iceberg REST call signed for glue (or another service) over HTTP: status, body (JSON or None), headers."""+ import urllib.error+ import urllib.request++ from botocore.auth import S3SigV4Auth, SigV4Auth+ from botocore.awsrequest import AWSRequest+ from conftest import ENDPOINT++ request = AWSRequest(method=method, url=f"{ENDPOINT}/iceberg/v1/{path}", params=query or {},+ data=json.dumps(body).encode() if body is not None else None)+ (S3SigV4Auth if service == "s3" else SigV4Auth)(credentials, service, "us-east-1").add_auth(request)+ prepared = request.prepare()+ try:+ with urllib.request.urlopen(urllib.request.Request(+ prepared.url, data=prepared.body, method=method, headers=dict(prepared.headers))) as response:+ status, content, headers = response.status, response.read(), response.headers+ except urllib.error.HTTPError as error:+ status, content, headers = error.code, error.read(), error.headers+ return status, json.loads(content) if content else None, {k.lower(): v for k, v in headers.items()}+++def glue_iceberg_user(glue, iam_user):+ """A Glue database with Iceberg tables t and u, and a factory for users of it signing with a second access key."""+ import contextlib+ import uuid++ from botocore.credentials import Credentials+ from conftest import make_client++ database = f"irc_{uuid.uuid4().hex[:8]}"+ glue.create_database(DatabaseInput={"Name": database})+ root = Credentials("test", "test")+ for table in ("t", "u"):+ assert _glue_iceberg_signed(root, "POST", f"catalogs/000000000000/namespaces/{database}/tables", {+ "name": table, "schema": {"type": "struct", "schema-id": 0, "fields": []}})[0] == 200+ iam = make_client("iam")+ with contextlib.ExitStack() as cleanup:+ def drop_databases():+ for found in glue.get_databases()["DatabaseList"]:+ if found["Name"].startswith(database):+ glue.delete_database(Name=found["Name"])++ cleanup.callback(drop_databases)++ def create(statements=None):+ user = iam_user(policy={"Version": "2012-10-17", "Statement": statements} if statements else None)+ key = iam.create_access_key(UserName=user.name)["AccessKey"]+ cleanup.callback(iam.delete_access_key, UserName=user.name, AccessKeyId=key["AccessKeyId"])+ return user, Credentials(key["AccessKeyId"], key["SecretAccessKey"])++ yield database, create+++def _glue_iceberg_denial(user, action, resource, reason=None):+ """The Iceberg error body of a glue identity-policy denial."""+ reason = reason or f"because no identity-based policy allows the glue:{action} action"+ resource = resource if resource == "*" else f"arn:aws:glue:us-east-1:000000000000:{resource}"+ return {"error": {"message": f"User: {user.arn} is not authorized to perform: glue:{action} on resource: "+ f"{resource} {reason}", "type": "AccessDeniedException", "code": 403}}+++# Iceberg REST routes on the Glue catalog: method, path, body, the glue actions each needs with the resources it+# checks them on (catalog, then database, then table), and MiniStack's status once they are allowed.+_DATABASE = ["catalog", "database/{db}"]+_TABLE_T = ["catalog", "database/{db}", "table/{db}/t"]+_GLUE_ICEBERG_ROUTES = {+ "config": ("GET", "config", None, [("GetCatalog", ["catalog"])], 200),+ "list-namespaces": ("GET", "namespaces", None, [("GetDatabases", ["catalog"])], 200),+ "create-namespace": ("POST", "namespaces", {"namespace": ["{db}_{new}"]},+ [("CreateDatabase", ["catalog", "database/{db}_{new}"])], 200),+ "load-namespace": ("GET", "namespaces/{db}", None, [("GetDatabase", _DATABASE)], 200),+ "namespace-exists": ("HEAD", "namespaces/{db}", None, [("GetDatabase", _DATABASE)], 200),+ "drop-namespace": ("DELETE", "namespaces/{db}", None, [("DeleteDatabase", _DATABASE)], 501),+ "namespace-properties": ("POST", "namespaces/{db}/properties", {"updates": {"k": "v"}},+ [("GetDatabase", _DATABASE), ("UpdateDatabase", _DATABASE)], 501),+ "list-tables": ("GET", "namespaces/{db}/tables", None, [("GetTables", _DATABASE)], 200),+ "create-table": ("POST", "namespaces/{db}/tables", {"name": "{new}"},+ [("CreateTable", ["catalog", "database/{db}", "table/{db}/{new}"])], 200),+ "load-table": ("GET", "namespaces/{db}/tables/t", None, [("GetTable", _TABLE_T)], 200),+ "table-exists": ("HEAD", "namespaces/{db}/tables/t", None, [("GetTable", _TABLE_T)], 200),+ "update-table": ("POST", "namespaces/{db}/tables/t", {"requirements": [], "updates": []},+ [("UpdateTable", _TABLE_T)], 200),+ "drop-table": ("DELETE", "namespaces/{db}/tables/t", None, [("DeleteTable", _TABLE_T)], 501),+ "commit-transaction": ("POST", "transactions/commit", {"table-changes": [+ {"identifier": {"namespace": ["{db}"], "name": "t"}, "requirements": [], "updates": []}]},+ [("UpdateTable", _TABLE_T)], 200),+}+++def _glue_iceberg_fill(value, names):+ """A path, body or resource template with {name} placeholders filled from ``names``."""+ text = json.dumps(value)+ for key, name in names.items():+ text = text.replace("{" + key + "}", name)+ return json.loads(text)++[email protected]_contract[email protected]("service", ["glue", "s3tables", "s3", "sts"])[email protected]("route", list(_GLUE_ICEBERG_ROUTES))+def test_glue_iceberg_route_contract(auth_enabled, glue_iceberg_user, route, service):+ """Each Iceberg route on the Glue catalog needs exactly its glue actions, whatever service it is signed for."""+ database, create = glue_iceberg_user+ method, path, body, actions, served = _GLUE_ICEBERG_ROUTES[route]+ glue_arn = "arn:aws:glue:us-east-1:000000000000"+ allowed_names = {"db": database, "new": "a"}+ user, signer = create([+ {"Effect": "Allow", "Action": f"glue:{action}",+ "Resource": [f"{glue_arn}:{resource}" for resource in _glue_iceberg_fill(resources, allowed_names)]}+ for action, resources in actions])+ nobody, refused_signer = create()+ query = {"warehouse": "000000000000"} if path == "config" else None+ path = path if path == "config" else f"catalogs/000000000000/{path}"+ calls = [_glue_iceberg_signed(credentials, method, *_glue_iceberg_fill([path, body], names), query=query,+ service=service)+ for credentials, names in ((signer, allowed_names), (refused_signer, {"db": database, "new": "n"}))]+ assert calls[0][0] == served+ if not auth_enabled:+ assert calls[1][0] == served+ return+ status, error, headers = calls[1]+ assert (status, headers["content-type"], headers["x-amzn-errortype"]) == (403, "application/json",+ "ForbiddenResponse")+ assert error == (None if method == "HEAD" else _glue_iceberg_denial(nobody, actions[0][0], "catalog"))++[email protected]_contract[email protected]("method,path,body,action,resource,reason", [+ ("GET", "namespaces/{db}_new", None, "GetDatabase", "database/{db}_new", None),+ ("POST", "namespaces/{db}/properties", {"updates": {"k": "v"}}, "UpdateDatabase", "catalog", None),+ ("GET", "namespaces/{db}/tables/u", None, "GetTable", "table/{db}/u", None),+ ("POST", "namespaces/{db}/tables/t", {"requirements": [], "updates": []}, "UpdateTable", "table/{db}/t",+ "with an explicit deny in an identity-based policy"),+], ids=["absent-namespace", "namespace-properties", "other-table", "explicit-deny"])+def test_glue_iceberg_authorization_contract(auth_enabled, glue_iceberg_user, method, path, body, action, resource,+ reason):+ """A denial names the first resource the policy does not cover: the catalog, then the database, then the table."""+ database, create = glue_iceberg_user+ glue_arn = "arn:aws:glue:us-east-1:000000000000"+ scope = [f"{glue_arn}:catalog", f"{glue_arn}:database/{database}"]+ user, signer = create([+ {"Effect": "Allow", "Resource": scope,+ "Action": ["glue:GetCatalog", "glue:GetDatabases", "glue:GetDatabase", "glue:UpdateTable"]},+ {"Effect": "Allow", "Action": "glue:GetTable", "Resource": [*scope, f"{glue_arn}:table/{database}/t"]},+ {"Effect": "Deny", "Action": "glue:UpdateTable", "Resource": f"{glue_arn}:table/{database}/t"},+ ])+ assert _glue_iceberg_signed(signer, "GET", "config", query={"warehouse": "000000000000"})[0] == 200+ assert [database] in _glue_iceberg_signed(signer, "GET", "catalogs/:/namespaces")[1]["namespaces"]+ assert _glue_iceberg_signed(signer, "GET", f"catalogs/000000000000/namespaces/{database}/tables/t")[0] == 200+ status, error, headers = _glue_iceberg_signed(+ signer, method, "catalogs/000000000000/" + path.format(db=database), body)+ if not auth_enabled:+ assert status != 403+ return+ assert (status, headers["content-type"], headers["x-amzn-errortype"]) == (+ 403, "application/json", "ForbiddenResponse")+ assert error == _glue_iceberg_denial(user, action, resource.format(db=database), reason)++[email protected]_contract[email protected]("catalog", [":", "arn:aws:glue:us-east-1:000000000000:catalog"], ids=["colon", "arn"])+def test_glue_iceberg_catalog_forms_contract(auth_enabled, glue_iceberg_user, catalog):+ """MiniStack serves its default catalog for any {catalog}, so every form is authorized as that catalog."""+ database, create = glue_iceberg_user+ allowed, signer = create([{"Effect": "Allow", "Action": "glue:GetDatabases",+ "Resource": "arn:aws:glue:us-east-1:000000000000:catalog"}])+ nobody, refused_signer = create([{"Effect": "Allow", "Action": "glue:GetDatabase", "Resource": "*"}])+ assert [database] in _glue_iceberg_signed(signer, "GET", f"catalogs/{catalog}/namespaces")[1]["namespaces"]+ status, error, _ = _glue_iceberg_signed(refused_signer, "GET", f"catalogs/{catalog}/namespaces")+ if not auth_enabled:+ assert status == 200+ return+ assert (status, error) == (403, _glue_iceberg_denial(nobody, "GetDatabases", "catalog"))++[email protected]_contract[email protected]("body", ["t", "t-and-u", "no-changes", "no-object"])+def test_glue_iceberg_transaction_commit_contract(auth_enabled, glue_iceberg_user, body):+ """transactions/commit needs glue:UpdateTable on each table it changes, on * when it names none."""+ database, create = glue_iceberg_user+ glue_arn = "arn:aws:glue:us-east-1:000000000000"+ change = {"identifier": {"namespace": [database], "name": "t"}, "requirements": [], "updates": []}+ bodies = {"t": {"table-changes": [change]},+ "t-and-u": {"table-changes": [change, {**change, "identifier": {"namespace": [database], "name": "u"}}]},+ "no-changes": {}, "no-object": "table-changes"}+ allowed, signer = create([{"Effect": "Allow", "Action": "glue:UpdateTable", "Resource": [+ f"{glue_arn}:catalog", f"{glue_arn}:database/{database}", f"{glue_arn}:table/{database}/t"]}])+ nobody, refused_signer = create()+ path = "catalogs/000000000000/transactions/commit"+ committed = _glue_iceberg_signed(signer, "POST", path, bodies[body])+ refused = _glue_iceberg_signed(refused_signer, "POST", path, bodies[body])+ if not auth_enabled:+ assert (committed[0], refused[0]) == ((500, 500) if body == "no-object" else (200, 200))+ return+ unread = body in ("no-changes", "no-object")+ assert refused[:2] == (403, _glue_iceberg_denial(nobody, "UpdateTable", "*" if unread else "catalog"))+ if body == "t":+ assert committed[0] == 200+ else:+ assert committed[:2] == (403, _glue_iceberg_denial(allowed, "UpdateTable", "*" if unread else+ f"table/{database}/u"))++def test_non_catalogs_prefix_returns_501():"""Bare-warehouse prefixes (`/v1/{warehouse}/namespaces`) are theS3 Tables shape, not Glue's — reject rather than guess."""diff --git a/tests/test_s3tables.py b/tests/test_s3tables.pyindex 78b4f960..ffe6a00c 100644--- a/tests/test_s3tables.py+++ b/tests/test_s3tables.py@@ -1242,6 +1242,424 @@ def test_s3tables_iceberg_rest_sequence_numbers_and_requirements(s3tables):s3tables.delete_table_bucket(tableBucketARN=arn)+def test_iceberg_route_splits_paths_as_the_router():+ """The router and the IAM checks read one split: raw segments, the {prefix} decoded only afterwards."""+ from ministack.services.s3tables import _iceberg_route, iceberg_lacks_prefix++ bucket = "arn%3Aaws%3As3tables%3Aus-east-1%3A000000000000%3Abucket%2Fb"+ assert _iceberg_route("GET", "/iceberg/v1/config") == ("config", [], [])+ assert _iceberg_route("GET", f"/iceberg/v1/{bucket}/namespaces/ns") == ("namespaces", [bucket], ["ns"])+ # a decoded ARN spans two segments; an encoded keyword is a segment like any other+ assert _iceberg_route("GET", "/iceberg/v1/arn:aws:s3tables:us-east-1:000000000000:bucket/b/namespaces") == (+ "namespaces", ["arn:aws:s3tables:us-east-1:000000000000:bucket", "b"], [])+ assert _iceberg_route("GET", f"/iceberg/v1/%6Eamespaces/{bucket}/namespaces") == (+ "namespaces", ["%6Eamespaces", bucket], [])+ assert _iceberg_route("GET", f"/iceberg/v1/{bucket}/%6Eamespaces") is None+ # tables/rename after a namespace is updateTable on a table named rename+ assert _iceberg_route("POST", f"/iceberg/v1/{bucket}/tables/rename") == ("rename", [bucket], [])+ assert _iceberg_route("POST", f"/iceberg/v1/{bucket}/namespaces/ns/tables/rename") == (+ "namespaces", [bucket], ["ns", "tables", "rename"])+ assert _iceberg_route("POST", f"/iceberg/v1/{bucket}/transactions/commit") == (+ "transactions", [bucket], ["commit"])+ for method, path in (("GET", "/iceberg/v1/namespaces"), ("GET", "/iceberg/v1/namespaces/ns/tables/t"),+ ("POST", "/iceberg/v1/tables/rename"), ("POST", "/iceberg/v1/transactions/commit")):+ assert iceberg_lacks_prefix(method, path)+ # a first segment before "namespaces" is the {prefix}, whatever it reads+ for method, path in (("GET", "/iceberg/v1/config"), ("GET", f"/iceberg/v1/{bucket}/namespaces"),+ ("POST", f"/iceberg/v1/{bucket}/tables/rename"), ("GET", "/iceberg/v1/tables/namespaces"),+ ("GET", "/iceberg/v1/transactions/namespaces/ns"),+ ("POST", f"/iceberg/v1/{bucket}/transactions/commit")):+ assert not iceberg_lacks_prefix(method, path)+++def test_iceberg_iam_checks_name_the_bucket_the_prefix_names():+ """A bucket ARN or account:s3tablescatalog/bucket names a table bucket; any other {prefix} names none."""+ from ministack.services.s3tables import iceberg_iam_checks, iceberg_prefix_error++ bucket = "arn:aws:s3tables:us-east-1:000000000000:bucket/b"+ for prefix in (_arn_prefix(bucket), "000000000000:s3tablescatalog/b"):+ assert iceberg_iam_checks("GET", f"/iceberg/v1/{prefix}/namespaces", b"", {}) == [+ ("s3tables:ListNamespaces", bucket)]+ assert iceberg_prefix_error("GET", f"/iceberg/v1/{prefix}/namespaces") is None+ no_arn = "The specified table bucket ARN is not valid."+ for prefix, message in (("catalog", no_arn), ("%6Eamespaces/000000000000:s3tablescatalog/b", no_arn),+ (_arn_prefix(f"{bucket}/x"), "The specified bucket name isn't valid. Specify a different "+ "bucket name, and then try again.")):+ assert iceberg_iam_checks("POST", f"/iceberg/v1/{prefix}/namespaces", b"{}", {}) is None+ status, _, body = iceberg_prefix_error("POST", f"/iceberg/v1/{prefix}/namespaces")+ assert (status, json.loads(body)) == (400, {"error": {"message": message, "type": "bad_request", "code": 400}})+ # a rename body whose source is no object names no table+ assert iceberg_iam_checks("POST", f"/iceberg/v1/{_arn_prefix(bucket)}/tables/rename", b'{"source": "t"}', {}) == [+ ("s3tables:RenameTable", f"{bucket}/table/*")]+++def iceberg_auth_catalog(s3tables):+ """A table bucket with namespace ns and tables t and u, removed with whatever a test left in it."""+ name = f"irc-{_uuid_mod.uuid4().hex[:8]}"+ bucket = s3tables.create_table_bucket(name=name)["arn"]+ s3tables.create_namespace(tableBucketARN=bucket, namespace=["ns"])+ tables = {t: s3tables.create_table(tableBucketARN=bucket, namespace="ns", name=t, format="ICEBERG")["tableARN"]+ for t in ("t", "u")}+ yield {"bucket": bucket, "name": name, "prefix": _arn_prefix(bucket), **tables}+ for namespace in s3tables.list_namespaces(tableBucketARN=bucket)["namespaces"]:+ ns = namespace["namespace"][0]+ for table in s3tables.list_tables(tableBucketARN=bucket, namespace=ns)["tables"]:+ s3tables.delete_table(tableBucketARN=bucket, namespace=ns, name=table["name"])+ s3tables.delete_namespace(tableBucketARN=bucket, namespace=ns)+ s3tables.delete_table_bucket(tableBucketARN=bucket)+++def _iceberg_contract_signer(user):+ """A second access key of an ``iam_user`` for signed Iceberg REST calls, deleted before the fixture's cleanup."""+ from botocore.credentials import Credentials+ from conftest import make_client++ iam = make_client("iam")+ key = iam.create_access_key(UserName=user.name)["AccessKey"]+ try:+ yield Credentials(key["AccessKeyId"], key["SecretAccessKey"])+ finally:+ iam.delete_access_key(UserName=user.name, AccessKeyId=key["AccessKeyId"])+++def _iceberg_signed(credentials, method, path, query=None, body=None, service="s3tables", region="us-east-1",+ presigned=False):+ """One SigV4-signed Iceberg REST call over HTTP: status, body (parsed when JSON, None when empty) and headers."""+ from botocore.auth import S3SigV4Auth, SigV4Auth, SigV4QueryAuth+ from botocore.awsrequest import AWSRequest++ request = AWSRequest(method=method, url=f"{_ENDPOINT}/iceberg/v1/{path}", params=query or {},+ data=json.dumps(body).encode() if body is not None else None)+ if presigned:+ SigV4QueryAuth(credentials, service, region, expires=300).add_auth(request)+ else:+ (S3SigV4Auth if service == "s3" else SigV4Auth)(credentials, service, region).add_auth(request)+ prepared = request.prepare()+ try:+ with urllib.request.urlopen(urllib.request.Request(+ prepared.url, data=prepared.body, method=method, headers=dict(prepared.headers))) as response:+ status, content, headers = response.status, response.read(), response.headers+ except urllib.error.HTTPError as error:+ status, content, headers = error.code, error.read(), error.headers+ headers = {k.lower(): v for k, v in headers.items()}+ if content and "json" in headers.get("content-type", ""):+ content = json.loads(content)+ return status, content or None, headers+++def _iceberg_fill(value, names):+ """A path, query or body template with {name} placeholders filled from ``names``."""+ text = json.dumps(value)+ for key, name in names.items():+ text = text.replace("{" + key + "}", name)+ return json.loads(text)+++def _iceberg_denial(user, action, resource, reason=None):+ """The Iceberg error body of an s3tables identity-policy denial."""+ reason = reason or f"because no identity-based policy allows the s3tables:{action} action"+ return {"error": {"message": f"User: {user.arn} is not authorized to perform: s3tables:{action} "+ f"on resource: {resource} {reason}", "type": "forbidden", "code": 403}}+++# Iceberg REST routes on the S3 Tables catalog: method, path, query, body, the actions AWS lists for each (on the+# table bucket or table), and MiniStack's status once they are allowed.+_UPDATE_TABLE_T = [("UpdateTableMetadataLocation", "{t}"), ("PutTableData", "{t}"), ("GetTableData", "{t}")]+_S3TABLES_ICEBERG_ROUTES = {+ "config": ("GET", "config", {"warehouse": "{bucket}"}, None, [("GetTableBucket", "{bucket}")], 200),+ "list-namespaces": ("GET", "{prefix}/namespaces", None, None, [("ListNamespaces", "{bucket}")], 200),+ "create-namespace": ("POST", "{prefix}/namespaces", None, {"namespace": ["{new}"]},+ [("CreateNamespace", "{bucket}")], 200),+ "load-namespace": ("GET", "{prefix}/namespaces/ns", None, None, [("GetNamespace", "{bucket}")], 200),+ "drop-namespace": ("DELETE", "{prefix}/namespaces/ns", None, None, [("DeleteNamespace", "{bucket}")], 404),+ "list-tables": ("GET", "{prefix}/namespaces/ns/tables", None, None, [("ListTables", "{bucket}")], 200),+ "create-table": ("POST", "{prefix}/namespaces/ns/tables", None, {"name": "{new}"},+ [("CreateTable", "{bucket}"), ("PutTableData", "{bucket}/table/*")], 200),+ "load-table": ("GET", "{prefix}/namespaces/ns/tables/t", None, None,+ [("GetTableMetadataLocation", "{t}"), ("GetTableData", "{t}")], 200),+ "update-table": ("POST", "{prefix}/namespaces/ns/tables/t", None, {"requirements": [], "updates": []},+ _UPDATE_TABLE_T, 200),+ "drop-table": ("DELETE", "{prefix}/namespaces/ns/tables/t", None, None, [("DeleteTable", "{t}")], 404),+ "table-exists": ("HEAD", "{prefix}/namespaces/ns/tables/t", None, None, [("GetTable", "{t}")], 200),+ "rename-table": ("POST", "{prefix}/tables/rename", None, {"source": {"namespace": ["ns"], "name": "t"},+ "destination": {"namespace": ["ns"], "name": "{new}"}},+ [("RenameTable", "{t}")], 404),+ "commit-transaction": ("POST", "{prefix}/transactions/commit", None, {"table-changes": [+ {"identifier": {"namespace": ["ns"], "name": "t"}, "requirements": [], "updates": []}]}, _UPDATE_TABLE_T, 200),+}++[email protected]_contract[email protected]("service", ["s3tables", "glue", "s3", "sts"])[email protected]("route", list(_S3TABLES_ICEBERG_ROUTES))+def test_s3tables_iceberg_route_contract(auth_enabled, iam_user, iceberg_auth_catalog, route, service):+ """Each Iceberg route on a table bucket needs exactly its s3tables actions, whatever service it is signed for."""+ method, path, query, body, actions, served = _S3TABLES_ICEBERG_ROUTES[route]+ actions = _iceberg_fill(actions, iceberg_auth_catalog)+ allowed = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": f"s3tables:{action}", "Resource": resource} for action, resource in actions]})+ nobody = iam_user()+ with _iceberg_contract_signer(allowed) as signer, _iceberg_contract_signer(nobody) as refused_signer:+ calls = [_iceberg_signed(credentials, method, *_iceberg_fill([path, query, body], {+ **iceberg_auth_catalog, "new": f"by-{label}"}), service=service)+ for label, credentials in (("allowed", signer), ("nobody", refused_signer))]+ assert calls[0][0] == served+ if not auth_enabled:+ assert calls[1][0] == served+ return+ status, error, headers = calls[1]+ assert (status, headers["content-type"], headers["x-amzn-errortype"]) == (+ 403, "application/json", "IrcForbiddenResponse")+ assert error == (None if method == "HEAD" else _iceberg_denial(nobody, *actions[0]))++[email protected]_contract[email protected]("method,path,body,action,resource,reason", [+ ("GET", "config?warehouse={absent}", None, "GetTableBucket", "{absent}", None),+ ("POST", "{prefix}/namespaces", {"namespace": ["ns2"]}, "CreateNamespace", "{bucket}",+ "with an explicit deny in an identity-based policy"),+ ("GET", "{prefix}/namespaces/ns/tables/u", None, "GetTableMetadataLocation", "{u}", None),+ ("GET", "{prefix}/namespaces/ns/tables/absent", None, "GetTableMetadataLocation", "{bucket}/table/*", None),+], ids=["config-absent-bucket", "explicit-deny", "other-table", "absent-table"])+def test_s3tables_iceberg_authorization_contract(auth_enabled, iam_user, iceberg_auth_catalog,+ method, path, body, action, resource, reason):+ """A denial names the bucket or table the call names, table/* for a table that does not exist, and a Deny."""+ catalog = iceberg_auth_catalog+ user = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Resource": catalog["bucket"],+ "Action": ["s3tables:GetTableBucket", "s3tables:ListNamespaces", "s3tables:GetNamespace"]},+ {"Effect": "Allow", "Action": "s3tables:GetTableMetadataLocation", "Resource": catalog["t"]},+ {"Effect": "Deny", "Action": "s3tables:CreateNamespace", "Resource": catalog["bucket"]},+ ]})+ names = {**catalog, "absent": catalog["bucket"].rsplit("/", 1)[0] + "/absent"}+ path, query = _iceberg_fill(path, names), None+ if "?warehouse=" in path:+ path, warehouse = path.split("?warehouse=")+ query = {"warehouse": warehouse}+ with _iceberg_contract_signer(user) as signer:+ assert _iceberg_signed(signer, "GET", "config", {"warehouse": catalog["bucket"]})[0] == 200+ assert _iceberg_signed(signer, "GET", f"{catalog['prefix']}/namespaces")[1] == {"namespaces": [["ns"]]}+ status, error, headers = _iceberg_signed(signer, method, path, query, body)+ if not auth_enabled:+ assert status != 403+ return+ assert (status, headers["content-type"]) == (403, "application/json")+ assert error == _iceberg_denial(user, action, _iceberg_fill(resource, names), reason)++[email protected]_contract+def test_s3tables_iceberg_later_action_contract(auth_enabled, iam_user, iceberg_auth_catalog):+ """loadTable's second action answers Access Denied."""+ catalog = iceberg_auth_catalog+ user = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": "s3tables:GetTableMetadataLocation", "Resource": catalog["t"]}]})+ with _iceberg_contract_signer(user) as signer:+ loaded = _iceberg_signed(signer, "GET", f"{catalog['prefix']}/namespaces/ns/tables/t")+ if not auth_enabled:+ assert loaded[0] == 200+ return+ assert loaded[:2] == (403, {"error": {"message": "Access Denied", "type": "forbidden", "code": 403}})++[email protected]_contract[email protected]("service", ["s3tables", "glue"])+def test_s3tables_iceberg_wrong_secret_contract(auth_enabled, iam_user, iceberg_auth_catalog, service):+ """Iceberg REST calls check their signature, for S3 Tables and for the Glue catalog alike, in JSON."""+ from botocore.credentials import Credentials++ catalog = iceberg_auth_catalog+ warehouse = catalog["bucket"] if service == "s3tables" else "000000000000"+ action = "s3tables:GetTableBucket" if service == "s3tables" else "glue:GetCatalog"+ user = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": action, "Resource": "*"}]})+ with _iceberg_contract_signer(user) as signer:+ assert _iceberg_signed(signer, "GET", "config", {"warehouse": warehouse}, service=service)[0] == 200+ status, _, headers = _iceberg_signed(Credentials(signer.access_key, "wrong-secret"), "GET", "config",+ {"warehouse": warehouse}, service=service)+ if not auth_enabled:+ assert status == 200+ return+ assert (status, headers["x-amzn-errortype"], headers["content-type"]) == (+ 403, "InvalidSignatureException", "application/json")++[email protected]_contract+def test_s3tables_iceberg_calls_naming_no_table_bucket_contract(auth_enabled, iam_user):+ """An unprefixed route answers 404 whatever its signature; config without a table bucket answers 400."""+ from botocore.credentials import Credentials++ with _iceberg_contract_signer(iam_user()) as signer:+ assert _iceberg_signed(signer, "GET", "config")[0] == 400+ wrong = Credentials(signer.access_key, "wrong-secret")+ assert _iceberg_signed(wrong, "GET", "namespaces")[0] == 404+ assert _iceberg_signed(wrong, "HEAD", "namespaces/ns")[0] == 404+ status, _, headers = _iceberg_signed(wrong, "GET", "config")+ read_as_prefix = _iceberg_signed(wrong, "GET", "tables/namespaces", service="glue")+ assert (status, headers.get("x-amzn-errortype")) == (+ (403, "InvalidSignatureException") if auth_enabled else (400, None))+ # "tables" before "namespaces" is the {prefix}, so that call is authenticated however it is signed+ assert read_as_prefix[0] == (403 if auth_enabled else 200)++[email protected]_contract[email protected]("method,prefix,message,served", [+ ("GET", "somewhere", "The specified table bucket ARN is not valid.", [200, 200, 200]),+ ("POST", "%256Eamespaces/000000000000:s3tablescatalog/{name}", "The specified table bucket ARN is not valid.",+ [200, 409, 409]),+ ("POST", "{bucket}/x", "The specified bucket name isn't valid. Specify a different bucket name, and then try "+ "again.", [404, 404, 404]),+], ids=["list-namespaces", "encoded-namespaces-segment", "bucket-arn-and-more"])+def test_s3tables_iceberg_prefix_naming_no_table_bucket_contract(auth_enabled, iam_user, iceberg_auth_catalog,+ s3tables, method, prefix, message, served):+ """A {prefix} that is no table bucket answers 400 after the credential check, before any policy, as on AWS."""+ from botocore.credentials import Credentials++ catalog = iceberg_auth_catalog+ prefix = _iceberg_fill(prefix, catalog)+ prefix = _arn_prefix(prefix) if prefix.startswith("arn:") else prefix+ everything = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": "s3tables:*", "Resource": "*"}]})+ body = {"namespace": ["created"]} if method == "POST" else None+ with _iceberg_contract_signer(iam_user()) as nobody, _iceberg_contract_signer(everything) as signer:+ calls = [_iceberg_signed(credentials, method, f"{prefix}/namespaces", body=body)+ for credentials in (nobody, signer, Credentials(signer.access_key, "wrong-secret"))]+ if not auth_enabled:+ assert [status for status, _, _ in calls] == served+ return+ for status, error, _ in calls[:2]:+ assert (status, error) == (400, {"error": {"message": message, "type": "bad_request", "code": 400}})+ assert (calls[2][0], calls[2][2]["x-amzn-errortype"]) == (403, "InvalidSignatureException")+ assert [n["namespace"] for n in s3tables.list_namespaces(tableBucketARN=catalog["bucket"])["namespaces"]] == [+ ["ns"]]++[email protected]_contract[email protected]("prefix", ["{prefix}/%6Eamespaces", "000000000000:s3tablescatalog/{name}/namespaces"],+ ids=["encoded-namespaces-keyword", "account-catalog-prefix"])+def test_s3tables_iceberg_prefix_forms_served_are_authorized_contract(auth_enabled, iam_user, iceberg_auth_catalog,+ prefix):+ """A path MiniStack serves as listNamespaces is authorized as one, whichever form names the bucket."""+ catalog = iceberg_auth_catalog+ path = _iceberg_fill(prefix, catalog)+ allowed = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": "s3tables:ListNamespaces", "Resource": catalog["bucket"]}]})+ nobody = iam_user()+ with _iceberg_contract_signer(allowed) as signer, _iceberg_contract_signer(nobody) as refused_signer:+ listed = _iceberg_signed(signer, "GET", path)+ refused = _iceberg_signed(refused_signer, "GET", path)+ assert listed[:2] == (200, {"namespaces": [["ns"]]})+ if not auth_enabled:+ assert refused[:2] == listed[:2]+ return+ assert refused[:2] == (403, _iceberg_denial(nobody, "ListNamespaces", catalog["bucket"]))++[email protected]_contract+def test_s3tables_iceberg_presigned_region_contract(auth_enabled, iam_user, iceberg_auth_catalog):+ """Under AUTH a presigned call serves its credential region, whose bucket and table the checks read."""+ catalog = iceberg_auth_catalog+ west = _make_s3tables_client(region_name="us-west-2")+ west_arn = west.create_table_bucket(name=catalog["name"])["arn"]+ west.create_namespace(tableBucketARN=west_arn, namespace=["ns"])+ secret = west.create_table(tableBucketARN=west_arn, namespace="ns", name="secret", format="ICEBERG")["tableARN"]+ reads = ["s3tables:GetTableMetadataLocation", "s3tables:GetTableData"]+ user = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": "s3tables:ListNamespaces", "Resource": catalog["bucket"]},+ {"Effect": "Allow", "Action": reads, "Resource": "*"},+ {"Effect": "Deny", "Action": reads, "Resource": secret}]})+ by_name = f"000000000000:s3tablescatalog/{catalog['name']}/namespaces"+ table = f"{_arn_prefix(west_arn)}/namespaces/ns/tables/secret"+ try:+ with _iceberg_contract_signer(user) as signer:+ calls = {(region, kind): _iceberg_signed(signer, "GET", path, region=region, presigned=True)+ for region in ("us-east-1", "us-west-2") for kind, path in (("list", by_name), ("table", table))}+ finally:+ west.delete_table(tableBucketARN=west_arn, namespace="ns", name="secret")+ west.delete_namespace(tableBucketARN=west_arn, namespace="ns")+ west.delete_table_bucket(tableBucketARN=west_arn)+ if not auth_enabled:+ assert [status for status, _, _ in calls.values()] == [200, 200, 200, 200]+ return+ # signed for us-east-1, neither call reaches the us-west-2 bucket+ assert calls["us-east-1", "list"][:2] == (200, {"namespaces": [["ns"]]})+ assert calls["us-east-1", "table"][0] == 404+ assert calls["us-west-2", "list"][:2] == (403, _iceberg_denial(user, "ListNamespaces", west_arn))+ assert calls["us-west-2", "table"][:2] == (403, _iceberg_denial(+ user, "GetTableMetadataLocation", secret, "with an explicit deny in an identity-based policy"))++[email protected]_contract[email protected]("warehouse", ["1:s3tablescatalog/{name}", "000000000000:s3tablescatalog/{name}/x"],+ ids=["short-account", "trailing-segment"])+def test_s3tables_iceberg_config_catalog_warehouse_contract(auth_enabled, iam_user, iceberg_auth_catalog, warehouse):+ """getConfig serves any account:s3tablescatalog/ warehouse; one naming no bucket needs GetTableBucket on *."""+ warehouse = _iceberg_fill(warehouse, iceberg_auth_catalog)+ nobody = iam_user()+ allowed = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": "s3tables:GetTableBucket", "Resource": "*"}]})+ with _iceberg_contract_signer(allowed) as signer, _iceberg_contract_signer(nobody) as refused_signer:+ served = _iceberg_signed(signer, "GET", "config", {"warehouse": warehouse})+ refused = _iceberg_signed(refused_signer, "GET", "config", {"warehouse": warehouse})+ assert served[0] == 200+ if not auth_enabled:+ assert refused[0] == 200+ return+ assert refused[:2] == (403, _iceberg_denial(nobody, "GetTableBucket", "*"))++[email protected]_contract+def test_s3tables_iceberg_table_named_rename_contract(auth_enabled, iam_user, iceberg_auth_catalog, s3tables):+ """A commit to a table named rename is updateTable on that table, not renameTable."""+ catalog = iceberg_auth_catalog+ table = s3tables.create_table(tableBucketARN=catalog["bucket"], namespace="ns", name="rename",+ format="ICEBERG")["tableARN"]+ path = f"{catalog['prefix']}/namespaces/ns/tables/rename"+ renamer = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": "s3tables:RenameTable", "Resource": "*"}]})+ updater = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": f"s3tables:{action}", "Resource": table} for action, _ in _UPDATE_TABLE_T]})+ commit = {"requirements": [], "updates": []}+ with _iceberg_contract_signer(renamer) as refused_signer, _iceberg_contract_signer(updater) as signer:+ assert _iceberg_signed(signer, "POST", path, body=commit)[0] == 200+ refused = _iceberg_signed(refused_signer, "POST", path, body=commit)+ if not auth_enabled:+ assert refused[0] == 200+ return+ assert refused[:2] == (403, _iceberg_denial(renamer, "UpdateTableMetadataLocation", table))++[email protected]_contract[email protected]("body", ["t", "t-and-u", "no-changes", "no-object"])+def test_s3tables_iceberg_transaction_commit_contract(auth_enabled, iam_user, iceberg_auth_catalog, body):+ """transactions/commit needs updateTable's actions on each table it changes, on * when it names none."""+ catalog = iceberg_auth_catalog+ change = {"identifier": {"namespace": ["ns"], "name": "t"}, "requirements": [], "updates": []}+ bodies = {"t": {"table-changes": [change]},+ "t-and-u": {"table-changes": [change, {**change, "identifier": {"namespace": ["ns"], "name": "u"}}]},+ "no-changes": {}, "no-object": "table-changes"}+ allowed = iam_user(policy={"Version": "2012-10-17", "Statement": [+ {"Effect": "Allow", "Action": f"s3tables:{action}", "Resource": catalog["t"]}+ for action, _ in _UPDATE_TABLE_T]})+ nobody = iam_user()+ path = f"{catalog['prefix']}/transactions/commit"+ with _iceberg_contract_signer(allowed) as signer, _iceberg_contract_signer(nobody) as refused_signer:+ committed = _iceberg_signed(signer, "POST", path, body=bodies[body])+ refused = _iceberg_signed(refused_signer, "POST", path, body=bodies[body])+ if not auth_enabled:+ assert (committed[0], refused[0]) == ((500, 500) if body == "no-object" else (200, 200))+ return+ first = "*" if body in ("no-changes", "no-object") else catalog["t"]+ assert refused[:2] == (403, _iceberg_denial(nobody, "UpdateTableMetadataLocation", first))+ if body == "t":+ assert committed[0] == 200+ elif body == "t-and-u":+ assert committed[:2] == (403, {"error": {"message": "Access Denied", "type": "forbidden", "code": 403}})+ else:+ assert committed[:2] == (403, _iceberg_denial(allowed, "UpdateTableMetadataLocation", "*"))++def test_s3tables_bucket_subresources_defaults_and_updates(s3tables):"""Documented defaults until set: SSE-S3, STANDARD, unreferenced file removal 3/10 days."""arn = s3tables.create_table_bucket(name=f"sub-{_uuid_mod.uuid4().hex[:8]}",