Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
fix(s3): honor If-Match and If-None-Match on CompleteMultipartUpload (#1435)
Andrew Gaul09cf8b313fa4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex fd3abf1a..6f00ca96 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).### Fixed- **CloudFormation — `AWS::IoT::Policy` updates apply instead of rolling the stack back** — the type had a create and a delete but no update handler, so a stack update fell through to create, hit `ResourceAlreadyExistsException` on the policy it had provisioned itself, and rolled back with the document unchanged: an IoT policy could be deployed once and never edited again. A changed `PolicyDocument` is now a no-interruption update stored as a new policy version and made the default, so `Ref` keeps naming the same policy and `GetPolicy` returns the new document; a changed `PolicyName` is a replacement, creating the policy under the new name and removing the old one. Because IoT caps a policy at five versions, the update prunes the oldest non-default versions before storing a new document, as the CloudFormation handler does on AWS.- **EC2 — instance public IP and DNS reach the SDKs, and generated addresses are addresses** — `DescribeInstances` and `RunInstances` emitted the public address under `publicIpAddress` / `publicDnsName`, which are not the tags the EC2 wire schema defines (`ipAddress` / `dnsName`, per botocore's `ec2-2016-11-15` model), so every SDK dropped both fields silently and `PublicIpAddress` came back absent on instances that had one. They now ride the real tags. Fixing that exposed a second one: `_random_ip` appended two octets whatever it was given, so a one-octet prefix produced `52.55.218` — `AllocateAddress` has been handing back that shape as `PublicIp` all along, and no address parser accepts it. The generator now completes any prefix to four octets. Contributed by @iot-rocket.+- **S3 — `CompleteMultipartUpload` honors `If-Match` / `If-None-Match`** — conditional writes landed on PutObject, but CompleteMultipartUpload accepted the same headers and ignored them, so a create-once (`If-None-Match: *`) or compare-and-swap (`If-Match: <etag>`) upload that took the multipart path silently overwrote whatever was there. The complete now evaluates the same preconditions PutObject does, against the current object at complete time — 412 on a violated condition, 404 `NoSuchKey` when If-Match addresses a key with no current object. That last rule also tightens PutObject: the `*` form previously answered the RFC 7232 412 for a missing key, but AWS documents a single If-Match error row — `NoSuchKey` — and ceph/s3-tests pins 404 for both forms, so both operations now answer it. The idempotent complete-retry replay still bypasses the check, since the original request passed its conditions when it committed. Contributed by @gaul.- **S3 — canned ACLs are stored, and object ACLs bind to versions** — SDKs send `put-bucket-acl --acl public-read` as an `x-amz-acl` header with an empty body, and `PutBucketAcl` read only the body, so the canned grant vanished and the bucket read back owner-only — a caller that gates anonymous access on the reported grants (S3Proxy's `aws-s3` backend among them) then treated every bucket as private. `CreateBucket`'s `x-amz-acl` was dropped the same way. Both now validate the canned value (`InvalidArgument` otherwise) and store the grants it implies, `log-delivery-write` included, exactly as the object side has since #1322; an XML body must be well-formed (`MalformedACLError`) and a request with neither header nor body answers `MissingSecurityHeader`, as on AWS. Object ACLs are now per-version like tags (#1058): `?versionId=` reads or writes that version's ACL, a fresh version starts from the default policy instead of inheriting its predecessor's, the canned value on `PutObject` sticks to the version that PUT created, and version and bucket deletes drop the stored entries. The browser-POST form names its field `acl` rather than `x-amz-acl`, so it never matched the header list and was discarded; it now applies to the uploaded object. Contributed by @gaul.- **S3 — `CRC64NVME` checksums are computed instead of refused** — CRC-64/NVME is the algorithm current AWS SDKs and the CLI checksum uploads with by default, and MiniStack answered it with `InvalidRequest` ("requires optional native dependencies"), so a stock `aws s3 cp` or `aws s3api put-object` failed before it began unless the caller knew to set `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`. The algorithm is plain arithmetic — the reflected form of polynomial `0xAD93D23594C93659` with all-ones init and xorout — so it is now computed from a byte-at-a-time table in the stdlib, adding no dependency: `PutObject` and `CopyObject` validate a client-supplied `x-amz-checksum-crc64nvme` (`BadDigest` on mismatch) and `GetObject` / `HeadObject` return it under `x-amz-checksum-mode: ENABLED`, typed `FULL_OBJECT`. The tests pin the implementation to the algorithm's published check value (`b"123456789"` → `0xAE8B14860A799888`) and to a bit-at-a-time reference rather than to itself. `CRC32C` still requires the native `google-crc32c` and is still refused.diff --git a/ministack/services/s3.py b/ministack/services/s3.pyindex 57c2b8e9..eefeef67 100644--- a/ministack/services/s3.py+++ b/ministack/services/s3.py@@ -436,9 +436,10 @@ def _validate_content_md5(headers: dict, body: bytes):def _check_put_preconditions(headers: dict, existing_obj: dict | None):- """Evaluate `If-Match` and `If-None-Match` on PUT.+ """Evaluate `If-Match` and `If-None-Match` on a conditional write.- AWS S3 added native conditional writes on PutObject in November 2024:+ AWS S3 added native conditional writes — on PutObject and+ CompleteMultipartUpload alike — in November 2024:- `If-None-Match: "*"` — succeed only when no object exists at the key.Used to implement create-once semantics (idempotent writes, distributedleader election via S3, two-file pair serialization).@@ -448,8 +449,9 @@ def _check_put_preconditions(headers: dict, existing_obj: dict | None):- `If-Match: "<etag>"` — succeed only when the existing object's ETagmatches.- Returns a 412 PreconditionFailed tuple when a condition is violated;- otherwise returns None and the caller proceeds with the PUT.+ Returns an error tuple when a condition is violated — 412+ PreconditionFailed, except that If-Match against a missing key is 404+ NoSuchKey — otherwise returns None and the caller proceeds with the write.ETag comparison strips surrounding quotes on both sides — S3 stores ETagswith quotes but client code is inconsistent about including them.@@ -482,27 +484,19 @@ def _check_put_preconditions(headers: dict, existing_obj: dict | None):)if if_match:- # "*" form: any existing object satisfies the condition; missing → 412 per RFC 7232.- # (AWS docs don't explicitly document If-Match:* for PutObject, but the inverse case- # is well-defined by RFC and real S3 follows it.)- if if_match == "*":- if existing_obj is None:- return _error(- "PreconditionFailed",- "At least one of the pre-conditions you specified did not hold",- 412,- )- elif existing_obj is None:- # AWS S3 specifically returns 404 (NoSuchKey) — not 412 — when If-Match: <etag>- # targets a key that doesn't exist (or whose current version is a delete marker).- # Documented under "Conditional write behavior" in the user guide:+ if existing_obj is None:+ # AWS S3 specifically returns 404 (NoSuchKey) — not the RFC 7232+ # 412 — when If-Match targets a key that doesn't exist (or whose+ # current version is a delete marker), for the "*" form as well as+ # the ETag form. Documented under "Conditional write behavior" in+ # the user guide:# https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html#conditional-error-responsereturn _error("NoSuchKey","The specified key does not exist.",404,)- elif if_match.strip('"') != existing_etag:+ if if_match != "*" and if_match.strip('"') != existing_etag:return _error("PreconditionFailed","At least one of the pre-conditions you specified did not hold",@@ -4861,6 +4855,15 @@ def _complete_multipart_upload(f"/{bucket_name}/{key}",)+ # CompleteMultipartUpload takes the same If-Match / If-None-Match+ # preconditions PutObject does, evaluated against the current object at+ # complete time. (The idempotent replay above deliberately skips them —+ # the original request already passed its conditions when it committed.)+ precondition_err = _check_put_preconditions(+ headers or {}, bucket.get("objects", {}).get(key))+ if precondition_err:+ return precondition_err+try:xml_root = fromstring(body)except ParseError:diff --git a/tests/test_s3.py b/tests/test_s3.pyindex c01b52e8..336385a8 100644--- a/tests/test_s3.py+++ b/tests/test_s3.py@@ -164,21 +164,28 @@ def test_s3_put_object_if_none_match_etag(s3):def test_s3_put_object_if_match_star_requires_existing(s3):- """If-Match: * succeeds when an object exists, fails when none does."""+ """If-Match: * succeeds when an object exists, 404s when none does.++ The missing-key answer is NoSuchKey — not the RFC 7232 412 — for the "*"+ form just like the ETag form: AWS documents one If-Match error row for a+ key that doesn't exist, and ceph/s3-tests pins 404 for both forms.+ https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html#conditional-error-response+ """bucket = "intg-s3-ifm-star"s3.create_bucket(Bucket=bucket)def _add_ifm_star(request, **_kwargs):request.headers["If-Match"] = "*"- # No existing object → 412.+ # No existing object → 404 NoSuchKey.s3.meta.events.register_first("before-send.s3.PutObject", _add_ifm_star)try:with pytest.raises(ClientError) as exc:s3.put_object(Bucket=bucket, Key="missing.txt", Body=b"x")finally:s3.meta.events.unregister("before-send.s3.PutObject", _add_ifm_star)- assert exc.value.response["Error"]["Code"] == "PreconditionFailed"+ assert exc.value.response["Error"]["Code"] == "NoSuchKey"+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 404# Now create it, then If-Match: * succeeds.s3.put_object(Bucket=bucket, Key="present.txt", Body=b"a")@@ -242,6 +249,65 @@ def test_s3_put_object_if_match_etag_missing_object_returns_404(s3):assert exc.value.response["Error"]["Code"] == "NoSuchKey"assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 404++# ─── Conditional CompleteMultipartUpload (If-Match / If-None-Match) ──────────+# Unlike PutObject, botocore models IfMatch/IfNoneMatch on+# CompleteMultipartUpload directly, so no header-injection hack is needed.++def _mpu_complete(s3, bucket, key, body=b"part-data", **conditions):+ """Run a full multipart upload for `key`, completing with `conditions`."""+ mp = s3.create_multipart_upload(Bucket=bucket, Key=key)+ part = s3.upload_part(Bucket=bucket, Key=key, UploadId=mp["UploadId"],+ PartNumber=1, Body=body)+ return s3.complete_multipart_upload(+ Bucket=bucket, Key=key, UploadId=mp["UploadId"],+ MultipartUpload={"Parts": [{"ETag": part["ETag"], "PartNumber": 1}]},+ **conditions)+++def test_s3_complete_multipart_if_none_match_star(s3):+ """If-None-Match: * on CompleteMultipartUpload is create-once: the first+ complete lands, a second over the existing key fails 412 and must not+ overwrite the object."""+ bucket = "intg-s3-mpu-ifnm-star"+ s3.create_bucket(Bucket=bucket)++ _mpu_complete(s3, bucket, "obj", body=b"first", IfNoneMatch="*")+ with pytest.raises(ClientError) as exc:+ _mpu_complete(s3, bucket, "obj", body=b"second", IfNoneMatch="*")+ assert exc.value.response["Error"]["Code"] == "PreconditionFailed"+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 412+ assert s3.get_object(Bucket=bucket, Key="obj")["Body"].read() == b"first"+++def test_s3_complete_multipart_if_match_etag(s3):+ """If-Match on CompleteMultipartUpload succeeds against the current ETag+ and fails 412 against a stale one."""+ bucket = "intg-s3-mpu-ifm-etag"+ s3.create_bucket(Bucket=bucket)+ etag = s3.put_object(Bucket=bucket, Key="obj", Body=b"v1")["ETag"]++ _mpu_complete(s3, bucket, "obj", body=b"v2", IfMatch=etag)+ assert s3.get_object(Bucket=bucket, Key="obj")["Body"].read() == b"v2"++ with pytest.raises(ClientError) as exc:+ _mpu_complete(s3, bucket, "obj", body=b"v3", IfMatch=etag) # now stale+ assert exc.value.response["Error"]["Code"] == "PreconditionFailed"+ assert s3.get_object(Bucket=bucket, Key="obj")["Body"].read() == b"v2"+++def test_s3_complete_multipart_if_match_missing_key_404(s3):+ """If-Match on CompleteMultipartUpload against a missing key is 404+ NoSuchKey — for the "*" form as well as the ETag form, like PutObject."""+ bucket = "intg-s3-mpu-ifm-missing"+ s3.create_bucket(Bucket=bucket)++ for cond in ("*", '"00000000000000000000000000000000"'):+ with pytest.raises(ClientError) as exc:+ _mpu_complete(s3, bucket, "absent", IfMatch=cond)+ assert exc.value.response["Error"]["Code"] == "NoSuchKey"+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 404+def test_s3_put_get_json_chunked(s3):"""AWS SDK v2 sends PutObject with chunked Transfer-Encoding — body must be decoded cleanly."""import json as _json