Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(s3): honor If-Match and If-None-Match on CompleteMultipartUpload (#1435)

Andrew Gaul09cf8b313fa4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index fd3abf1a..6f00ca96 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).
### Fixed
- **CloudFormation — `AWS::IoT::Policy` updates apply instead of rolling the stack back** — the type had a create and a delete but no update handler, so a stack update fell through to create, hit `ResourceAlreadyExistsException` on the policy it had provisioned itself, and rolled back with the document unchanged: an IoT policy could be deployed once and never edited again. A changed `PolicyDocument` is now a no-interruption update stored as a new policy version and made the default, so `Ref` keeps naming the same policy and `GetPolicy` returns the new document; a changed `PolicyName` is a replacement, creating the policy under the new name and removing the old one. Because IoT caps a policy at five versions, the update prunes the oldest non-default versions before storing a new document, as the CloudFormation handler does on AWS.
- **EC2 — instance public IP and DNS reach the SDKs, and generated addresses are addresses** — `DescribeInstances` and `RunInstances` emitted the public address under `publicIpAddress` / `publicDnsName`, which are not the tags the EC2 wire schema defines (`ipAddress` / `dnsName`, per botocore's `ec2-2016-11-15` model), so every SDK dropped both fields silently and `PublicIpAddress` came back absent on instances that had one. They now ride the real tags. Fixing that exposed a second one: `_random_ip` appended two octets whatever it was given, so a one-octet prefix produced `52.55.218` — `AllocateAddress` has been handing back that shape as `PublicIp` all along, and no address parser accepts it. The generator now completes any prefix to four octets. Contributed by @iot-rocket.
+- **S3 — `CompleteMultipartUpload` honors `If-Match` / `If-None-Match`** — conditional writes landed on PutObject, but CompleteMultipartUpload accepted the same headers and ignored them, so a create-once (`If-None-Match: *`) or compare-and-swap (`If-Match: <etag>`) upload that took the multipart path silently overwrote whatever was there. The complete now evaluates the same preconditions PutObject does, against the current object at complete time — 412 on a violated condition, 404 `NoSuchKey` when If-Match addresses a key with no current object. That last rule also tightens PutObject: the `*` form previously answered the RFC 7232 412 for a missing key, but AWS documents a single If-Match error row — `NoSuchKey` — and ceph/s3-tests pins 404 for both forms, so both operations now answer it. The idempotent complete-retry replay still bypasses the check, since the original request passed its conditions when it committed. Contributed by @gaul.
- **S3 — canned ACLs are stored, and object ACLs bind to versions** — SDKs send `put-bucket-acl --acl public-read` as an `x-amz-acl` header with an empty body, and `PutBucketAcl` read only the body, so the canned grant vanished and the bucket read back owner-only — a caller that gates anonymous access on the reported grants (S3Proxy's `aws-s3` backend among them) then treated every bucket as private. `CreateBucket`'s `x-amz-acl` was dropped the same way. Both now validate the canned value (`InvalidArgument` otherwise) and store the grants it implies, `log-delivery-write` included, exactly as the object side has since #1322; an XML body must be well-formed (`MalformedACLError`) and a request with neither header nor body answers `MissingSecurityHeader`, as on AWS. Object ACLs are now per-version like tags (#1058): `?versionId=` reads or writes that version's ACL, a fresh version starts from the default policy instead of inheriting its predecessor's, the canned value on `PutObject` sticks to the version that PUT created, and version and bucket deletes drop the stored entries. The browser-POST form names its field `acl` rather than `x-amz-acl`, so it never matched the header list and was discarded; it now applies to the uploaded object. Contributed by @gaul.
- **S3 — `CRC64NVME` checksums are computed instead of refused** — CRC-64/NVME is the algorithm current AWS SDKs and the CLI checksum uploads with by default, and MiniStack answered it with `InvalidRequest` ("requires optional native dependencies"), so a stock `aws s3 cp` or `aws s3api put-object` failed before it began unless the caller knew to set `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`. The algorithm is plain arithmetic — the reflected form of polynomial `0xAD93D23594C93659` with all-ones init and xorout — so it is now computed from a byte-at-a-time table in the stdlib, adding no dependency: `PutObject` and `CopyObject` validate a client-supplied `x-amz-checksum-crc64nvme` (`BadDigest` on mismatch) and `GetObject` / `HeadObject` return it under `x-amz-checksum-mode: ENABLED`, typed `FULL_OBJECT`. The tests pin the implementation to the algorithm's published check value (`b"123456789"` → `0xAE8B14860A799888`) and to a bit-at-a-time reference rather than to itself. `CRC32C` still requires the native `google-crc32c` and is still refused.
diff --git a/ministack/services/s3.py b/ministack/services/s3.py
index 57c2b8e9..eefeef67 100644
--- a/ministack/services/s3.py
+++ b/ministack/services/s3.py
@@ -436,9 +436,10 @@ def _validate_content_md5(headers: dict, body: bytes):
def _check_put_preconditions(headers: dict, existing_obj: dict | None):
- """Evaluate `If-Match` and `If-None-Match` on PUT.
+ """Evaluate `If-Match` and `If-None-Match` on a conditional write.
- AWS S3 added native conditional writes on PutObject in November 2024:
+ AWS S3 added native conditional writes — on PutObject and
+ CompleteMultipartUpload alike — in November 2024:
- `If-None-Match: "*"` — succeed only when no object exists at the key.
Used to implement create-once semantics (idempotent writes, distributed
leader election via S3, two-file pair serialization).
@@ -448,8 +449,9 @@ def _check_put_preconditions(headers: dict, existing_obj: dict | None):
- `If-Match: "<etag>"` — succeed only when the existing object's ETag
matches.
- Returns a 412 PreconditionFailed tuple when a condition is violated;
- otherwise returns None and the caller proceeds with the PUT.
+ Returns an error tuple when a condition is violated — 412
+ PreconditionFailed, except that If-Match against a missing key is 404
+ NoSuchKey — otherwise returns None and the caller proceeds with the write.
ETag comparison strips surrounding quotes on both sides — S3 stores ETags
with quotes but client code is inconsistent about including them.
@@ -482,27 +484,19 @@ def _check_put_preconditions(headers: dict, existing_obj: dict | None):
)
if if_match:
- # "*" form: any existing object satisfies the condition; missing → 412 per RFC 7232.
- # (AWS docs don't explicitly document If-Match:* for PutObject, but the inverse case
- # is well-defined by RFC and real S3 follows it.)
- if if_match == "*":
- if existing_obj is None:
- return _error(
- "PreconditionFailed",
- "At least one of the pre-conditions you specified did not hold",
- 412,
- )
- elif existing_obj is None:
- # AWS S3 specifically returns 404 (NoSuchKey) — not 412 — when If-Match: <etag>
- # targets a key that doesn't exist (or whose current version is a delete marker).
- # Documented under "Conditional write behavior" in the user guide:
+ if existing_obj is None:
+ # AWS S3 specifically returns 404 (NoSuchKey) — not the RFC 7232
+ # 412 — when If-Match targets a key that doesn't exist (or whose
+ # current version is a delete marker), for the "*" form as well as
+ # the ETag form. Documented under "Conditional write behavior" in
+ # the user guide:
# https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html#conditional-error-response
return _error(
"NoSuchKey",
"The specified key does not exist.",
404,
)
- elif if_match.strip('"') != existing_etag:
+ if if_match != "*" and if_match.strip('"') != existing_etag:
return _error(
"PreconditionFailed",
"At least one of the pre-conditions you specified did not hold",
@@ -4861,6 +4855,15 @@ def _complete_multipart_upload(
f"/{bucket_name}/{key}",
)
+ # CompleteMultipartUpload takes the same If-Match / If-None-Match
+ # preconditions PutObject does, evaluated against the current object at
+ # complete time. (The idempotent replay above deliberately skips them —
+ # the original request already passed its conditions when it committed.)
+ precondition_err = _check_put_preconditions(
+ headers or {}, bucket.get("objects", {}).get(key))
+ if precondition_err:
+ return precondition_err
+
try:
xml_root = fromstring(body)
except ParseError:
diff --git a/tests/test_s3.py b/tests/test_s3.py
index c01b52e8..336385a8 100644
--- a/tests/test_s3.py
+++ b/tests/test_s3.py
@@ -164,21 +164,28 @@ def test_s3_put_object_if_none_match_etag(s3):
def test_s3_put_object_if_match_star_requires_existing(s3):
- """If-Match: * succeeds when an object exists, fails when none does."""
+ """If-Match: * succeeds when an object exists, 404s when none does.
+
+ The missing-key answer is NoSuchKey — not the RFC 7232 412 — for the "*"
+ form just like the ETag form: AWS documents one If-Match error row for a
+ key that doesn't exist, and ceph/s3-tests pins 404 for both forms.
+ https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html#conditional-error-response
+ """
bucket = "intg-s3-ifm-star"
s3.create_bucket(Bucket=bucket)
def _add_ifm_star(request, **_kwargs):
request.headers["If-Match"] = "*"
- # No existing object → 412.
+ # No existing object → 404 NoSuchKey.
s3.meta.events.register_first("before-send.s3.PutObject", _add_ifm_star)
try:
with pytest.raises(ClientError) as exc:
s3.put_object(Bucket=bucket, Key="missing.txt", Body=b"x")
finally:
s3.meta.events.unregister("before-send.s3.PutObject", _add_ifm_star)
- assert exc.value.response["Error"]["Code"] == "PreconditionFailed"
+ assert exc.value.response["Error"]["Code"] == "NoSuchKey"
+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 404
# Now create it, then If-Match: * succeeds.
s3.put_object(Bucket=bucket, Key="present.txt", Body=b"a")
@@ -242,6 +249,65 @@ def test_s3_put_object_if_match_etag_missing_object_returns_404(s3):
assert exc.value.response["Error"]["Code"] == "NoSuchKey"
assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 404
+
+# ─── Conditional CompleteMultipartUpload (If-Match / If-None-Match) ──────────
+# Unlike PutObject, botocore models IfMatch/IfNoneMatch on
+# CompleteMultipartUpload directly, so no header-injection hack is needed.
+
+def _mpu_complete(s3, bucket, key, body=b"part-data", **conditions):
+ """Run a full multipart upload for `key`, completing with `conditions`."""
+ mp = s3.create_multipart_upload(Bucket=bucket, Key=key)
+ part = s3.upload_part(Bucket=bucket, Key=key, UploadId=mp["UploadId"],
+ PartNumber=1, Body=body)
+ return s3.complete_multipart_upload(
+ Bucket=bucket, Key=key, UploadId=mp["UploadId"],
+ MultipartUpload={"Parts": [{"ETag": part["ETag"], "PartNumber": 1}]},
+ **conditions)
+
+
+def test_s3_complete_multipart_if_none_match_star(s3):
+ """If-None-Match: * on CompleteMultipartUpload is create-once: the first
+ complete lands, a second over the existing key fails 412 and must not
+ overwrite the object."""
+ bucket = "intg-s3-mpu-ifnm-star"
+ s3.create_bucket(Bucket=bucket)
+
+ _mpu_complete(s3, bucket, "obj", body=b"first", IfNoneMatch="*")
+ with pytest.raises(ClientError) as exc:
+ _mpu_complete(s3, bucket, "obj", body=b"second", IfNoneMatch="*")
+ assert exc.value.response["Error"]["Code"] == "PreconditionFailed"
+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 412
+ assert s3.get_object(Bucket=bucket, Key="obj")["Body"].read() == b"first"
+
+
+def test_s3_complete_multipart_if_match_etag(s3):
+ """If-Match on CompleteMultipartUpload succeeds against the current ETag
+ and fails 412 against a stale one."""
+ bucket = "intg-s3-mpu-ifm-etag"
+ s3.create_bucket(Bucket=bucket)
+ etag = s3.put_object(Bucket=bucket, Key="obj", Body=b"v1")["ETag"]
+
+ _mpu_complete(s3, bucket, "obj", body=b"v2", IfMatch=etag)
+ assert s3.get_object(Bucket=bucket, Key="obj")["Body"].read() == b"v2"
+
+ with pytest.raises(ClientError) as exc:
+ _mpu_complete(s3, bucket, "obj", body=b"v3", IfMatch=etag) # now stale
+ assert exc.value.response["Error"]["Code"] == "PreconditionFailed"
+ assert s3.get_object(Bucket=bucket, Key="obj")["Body"].read() == b"v2"
+
+
+def test_s3_complete_multipart_if_match_missing_key_404(s3):
+ """If-Match on CompleteMultipartUpload against a missing key is 404
+ NoSuchKey — for the "*" form as well as the ETag form, like PutObject."""
+ bucket = "intg-s3-mpu-ifm-missing"
+ s3.create_bucket(Bucket=bucket)
+
+ for cond in ("*", '"00000000000000000000000000000000"'):
+ with pytest.raises(ClientError) as exc:
+ _mpu_complete(s3, bucket, "absent", IfMatch=cond)
+ assert exc.value.response["Error"]["Code"] == "NoSuchKey"
+ assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 404
+
def test_s3_put_get_json_chunked(s3):
"""AWS SDK v2 sends PutObject with chunked Transfer-Encoding — body must be decoded cleanly."""
import json as _json