Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(s3): stop verifying a presigned URL's query-string checksum (#1705)

Stephan Mielke491616467802
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b7772b8a..dabc6386 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,6 +17,8 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- **Lambda — warm local custom runtimes** — `provided.*` bootstraps now reuse the existing subprocess worker pool instead of restarting on every invocation. Each invocation receives its request metadata through the Lambda Runtime API, concurrent calls use separate workers, and failed environments are cleaned up before reuse. Durable `provided.*` invocations retain their one-shot executor, because a bootstrap process reads its environment once, at spawn; Docker, image, and proxy execution are unchanged.
### Fixed
+- **S3 — a presigned upload is no longer rejected for the checksum in its query string** — since v3.729.0 the JavaScript SDK signs `x-amz-checksum-crc32` into every presigned `PutObject` by default, computed over the *empty* body, because whoever holds the URL chooses the body later. MiniStack hoisted that parameter into the request headers alongside the user metadata a presigned URL legitimately carries there, and the header path then verified it against the real body, so a stock `getSignedUrl` upload failed with `400 BadDigest` where real S3 answers `200` and stores the object. The `x-amz-checksum-*` value parameters are now excluded from hoisting: they still take part in signature verification, so rewriting one in the URL is still `403`, but they are no longer read as a supplied integrity value. A checksum sent as a real request header is verified exactly as before. `x-amz-checksum-algorithm` and `x-amz-sdk-checksum-algorithm` keep hoisting — they name an algorithm for the server to compute rather than carrying a value, so they cannot disagree with a body, and a presigned `CreateMultipartUpload` uses either to choose the algorithm its parts are digested with.
+
- **CloudFormation — an `AWS::ApiGateway::Method`'s `MethodResponses` and `IntegrationResponses` are provisioned** — the provisioner read the method and integration properties and discarded both response lists, so every REST API deployed from a template lost its mapped response headers. The visible casualty was the CDK's `defaultCorsPreflightOptions`, whose generated `OPTIONS` method returns the `Access-Control-Allow-*` headers through a MOCK integration's `responseParameters`: the preflight answered without a single CORS header and the browser blocked the request. That became load-bearing once `COGNITO_USER_POOLS` authorizers were enforced in 1.5.10, because the preflight can no longer fall through to a proxy integration and be answered by the application's own CORS middleware. Both lists are now provisioned onto the method, where the existing request-time mapping picks them up, and a stack update reprovisions them from the template. Contributed by @ppettitau.
- **Transcribe — `StartTranscriptionJob` returns a job already `IN_PROGRESS`** — the start response reported `QUEUED` with no `StartTime`, and the job reached `IN_PROGRESS` only from the background worker, after the response had been sent. A caller that reads the status off the start response and then waits for the terminal `Transcribe Job State Change` event, rather than polling, therefore never saw an in-progress state at all: the job went straight from queued to complete. Jobs now start `IN_PROGRESS` with `StartTime` set, matching the response the AWS CLI walkthrough documents, and `TRANSCRIBE_JOB_RUN_SECONDS` covers the whole run rather than half of it. `QUEUED`, which AWS reaches only for a request that opted into job queueing through `JobExecutionSettings.AllowDeferredExecution` while the account is at its concurrent job limit, stays reachable through the new `TRANSCRIBE_JOB_QUEUE_SECONDS` (default `0`); both knobs are settable at runtime through `/_ministack/config`. Contributed by @ppettitau.
diff --git a/ministack/services/s3.py b/ministack/services/s3.py
index 205d5721..d4ab88ad 100644
--- a/ministack/services/s3.py
+++ b/ministack/services/s3.py
@@ -1619,6 +1619,24 @@ _PRESIGN_SIGNING_PARAMS = {
"x-amz-content-sha256",
}
+# A presigned URL's checksum *value* is signed but never read as a supplied
+# integrity value. Current SDKs (JS since v3.729.0) compute it over the *empty*
+# body at presign time, because whoever holds the URL picks the body later, so
+# the value cannot describe what gets uploaded. Real S3 signs the parameter —
+# rewriting it still yields 403 — and then ignores it, storing the body it was
+# sent. Hoisting it into the headers instead hands it to
+# `_resolve_object_checksums`, which rejects the upload with `BadDigest` for a
+# request AWS answers 200.
+#
+# Only the value parameters are excluded. `x-amz-checksum-algorithm` and
+# `x-amz-sdk-checksum-algorithm` name an algorithm for the server to compute
+# rather than carrying a value, so they cannot disagree with a body and are
+# never the reason a request is refused; CreateMultipartUpload is presigned
+# with either of them to choose the algorithm its parts are digested with.
+_PRESIGN_UNHOISTED_CHECKSUM_PARAMS = frozenset(
+ f"x-amz-checksum-{alg}" for alg in _S3_CHECKSUM_HEADERS
+)
+
def _merge_hoisted_amz_headers(headers: dict, query_params: dict) -> dict:
"""Fold a presigned URL's hoisted ``x-amz-*`` query params into headers.
@@ -1636,13 +1654,17 @@ def _merge_hoisted_amz_headers(headers: dict, query_params: dict) -> dict:
metadata in its query string stored the object without any: the PUT
succeeded and the metadata was silently dropped.
- An explicitly sent header always wins over its hoisted twin.
+ An explicitly sent header always wins over its hoisted twin. The checksum
+ values in ``_PRESIGN_UNHOISTED_CHECKSUM_PARAMS`` are the exception AWS
+ itself makes and stay out of the headers.
"""
hoisted = None
for name, values in query_params.items():
lname = name.lower()
if not lname.startswith("x-amz-") or lname in _PRESIGN_SIGNING_PARAMS:
continue
+ if lname in _PRESIGN_UNHOISTED_CHECKSUM_PARAMS:
+ continue
if lname in headers:
continue
value = values[0] if isinstance(values, list) else values
diff --git a/tests/test_s3.py b/tests/test_s3.py
index e3a4adf7..318339d5 100644
--- a/tests/test_s3.py
+++ b/tests/test_s3.py
@@ -4005,6 +4005,145 @@ def test_s3_presigned_put_metadata_hoisted_into_query(s3):
]
+def test_s3_presigned_put_query_checksum_is_not_verified(s3):
+ """A presigned URL's `x-amz-checksum-*` query parameter describes the empty
+ body the presigner had, not the body the holder of the URL later uploads,
+ so it cannot be an integrity value. Since v3.729.0 the JS SDK signs one into
+ every presigned PutObject by default, and AWS answers 200 for a body that
+ does not match it. Verifying it rejects a request real S3 accepts."""
+ import urllib.request
+
+ from botocore.auth import S3SigV4QueryAuth
+ from botocore.awsrequest import AWSRequest
+ from botocore.credentials import Credentials
+
+ bucket = "presign-qs-checksum-bkt"
+ key = "probe.txt"
+ s3.create_bucket(Bucket=bucket)
+
+ # `AAAAAA==` is CRC32 of the empty body — the literal value the JS SDK
+ # signs in, and the one AWS's own recorded snapshot carries.
+ signed = AWSRequest(
+ method="PUT",
+ url=f"{ENDPOINT}/{bucket}/{key}"
+ "?x-amz-checksum-crc32=AAAAAA%3D%3D"
+ "&x-amz-sdk-checksum-algorithm=CRC32",
+ )
+ S3SigV4QueryAuth(
+ Credentials("test", "test"), "s3", "us-east-1", expires=300
+ ).add_auth(signed)
+
+ resp = urllib.request.urlopen(
+ urllib.request.Request(signed.url, data=b"123456", method="PUT"))
+ assert resp.status == 200
+
+ # AWS stores the body it was sent, mismatching checksum parameter and all.
+ head = s3.head_object(Bucket=bucket, Key=key, ChecksumMode="ENABLED")
+ assert head["ContentLength"] == 6
+ assert s3.get_object(Bucket=bucket, Key=key)["Body"].read() == b"123456"
+
+ # `x-amz-sdk-checksum-algorithm` names an algorithm to compute rather than
+ # carrying a value, so it keeps hoisting: the stored checksum is CRC32 of
+ # the body that actually arrived, never the empty-body value off the URL.
+ assert head.get("ChecksumCRC32") == "CXLTYQ==", (
+ "the hoisted algorithm selector was not honoured")
+
+
+def test_s3_presigned_put_query_checksum_still_covered_by_signature(s3):
+ """The other half of the AWS contract: the parameter is ignored as an
+ integrity value but still signed, so rewriting it after the fact breaks the
+ signature. Ignoring it must not mean stripping it before verification."""
+ import urllib.error
+ import urllib.request
+
+ from botocore.auth import S3SigV4QueryAuth
+ from botocore.awsrequest import AWSRequest
+ from botocore.credentials import Credentials
+
+ bucket = "presign-qs-checksum-sig-bkt"
+ key = "probe.txt"
+ s3.create_bucket(Bucket=bucket)
+
+ signed = AWSRequest(
+ method="PUT",
+ url=f"{ENDPOINT}/{bucket}/{key}?x-amz-checksum-crc32=AAAAAA%3D%3D",
+ )
+ S3SigV4QueryAuth(
+ Credentials("test", "test"), "s3", "us-east-1", expires=300
+ ).add_auth(signed)
+
+ tampered = signed.url.replace("AAAAAA%3D%3D", "BBBBBB%3D%3D")
+ assert tampered != signed.url
+ try:
+ status = urllib.request.urlopen(
+ urllib.request.Request(tampered, data=b"123456", method="PUT")).status
+ except urllib.error.HTTPError as exc:
+ status = exc.code
+ assert status == 403, f"tampered checksum parameter accepted with {status}"
+
+
+def test_s3_put_header_checksum_is_still_verified(s3):
+ """Excluding the query parameter must not loosen the header case: a
+ `x-amz-checksum-*` sent as a real header is a value the client computed over
+ the body it is sending, and a mismatch is still `BadDigest`."""
+ import urllib.error
+ import urllib.request
+
+ bucket = "presign-hdr-checksum-bkt"
+ key = "probe.txt"
+ s3.create_bucket(Bucket=bucket)
+
+ url = s3.generate_presigned_url(
+ "put_object", Params={"Bucket": bucket, "Key": key}, ExpiresIn=300)
+ try:
+ status = urllib.request.urlopen(urllib.request.Request(
+ url, data=b"123456", method="PUT",
+ headers={"x-amz-checksum-crc32": "AAAAAA=="})).status
+ except urllib.error.HTTPError as exc:
+ status = exc.code
+ assert status == 400, f"mismatching checksum header accepted with {status}"
+
+
+def test_s3_presigned_multipart_hoisted_checksum_algorithm_is_honoured(s3):
+ """The exclusion covers checksum *values* only. `x-amz-checksum-algorithm`
+ and `x-amz-sdk-checksum-algorithm` name an algorithm for the server to
+ compute rather than carrying a value, so they cannot disagree with a body;
+ a presigned CreateMultipartUpload hoists one of them to choose the algorithm
+ its parts are digested with, and the completed object must carry it."""
+ import urllib.request
+ import xml.etree.ElementTree as ET
+
+ from botocore.auth import S3SigV4QueryAuth
+ from botocore.awsrequest import AWSRequest
+ from botocore.credentials import Credentials
+
+ bucket = "presign-mpu-algo-bkt"
+ s3.create_bucket(Bucket=bucket)
+
+ for param in ("x-amz-checksum-algorithm", "x-amz-sdk-checksum-algorithm"):
+ key = f"multipart/{param}.bin"
+ signed = AWSRequest(
+ method="POST", url=f"{ENDPOINT}/{bucket}/{key}?uploads&{param}=CRC32")
+ S3SigV4QueryAuth(
+ Credentials("test", "test"), "s3", "us-east-1", expires=300
+ ).add_auth(signed)
+
+ body = urllib.request.urlopen(urllib.request.Request(
+ signed.url, data=b"", method="POST")).read()
+ ns = "{http://s3.amazonaws.com/doc/2006-03-01/}"
+ upload_id = ET.fromstring(body).findtext(f"{ns}UploadId")
+ assert upload_id
+
+ part = s3.upload_part(Bucket=bucket, Key=key, UploadId=upload_id,
+ PartNumber=1, Body=b"x" * 16)
+ done = s3.complete_multipart_upload(
+ Bucket=bucket, Key=key, UploadId=upload_id,
+ MultipartUpload={"Parts": [
+ {"ETag": part["ETag"], "PartNumber": 1}]})
+ assert done.get("ChecksumCRC32"), (
+ f"{param} hoisted into the query string was not honoured")
+
+
def test_s3_presigned_put_metadata_sent_as_signed_headers(s3):
"""The other half of the contract: when the presigner leaves the metadata
in `X-Amz-SignedHeaders` instead of hoisting it, the uploader sends the