Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

prerelease (#1840)

Nahuel Nuceraff9015207854
diff --git a/CHANGELOG.md b/CHANGELOG.md
index acac654f..3e24dd58 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,22 +9,25 @@ Versioning follows [Semantic Versioning](https://semver.org/).
### Added
-- **SNS — `AddPermission` and `RemovePermission`** — they append to and prune the topic's `Policy` attribute with the AWS statement shape (bare account ids, lowercase `sns:` actions), mirroring the long-standing SQS pair, and a duplicate label is `InvalidParameterException`.
+- **Cognito — OIDC discovery through the real issuer host (`USE_SSL=1`)** — the generated certificate now names `cognito-idp.<region>.amazonaws.com` for every region, the gateway also listens on 443 when it can, and Lambda containers resolve those hosts to MiniStack and trust its certificate. With an `/etc/hosts` entry and the certificate trusted, an unmodified client follows the token's `iss` to MiniStack while `iss` stays exactly as AWS issues it. Reported by @epcap90.
+- **Bedrock AgentCore — `InvokeAgentRuntime` runs the runtime's container** — with Docker available (full image), the runtime's `containerConfiguration.containerUri` image is started once, checked on `/ping`, and invocations are forwarded to `/invocations` on port 8080; worker and container failures answer `RuntimeClientError` (424). The container is removed on update, delete or reset. Without Docker, or for a code artifact, the deterministic echo remains. Contributed by @pingedbrain.
+- **SNS — `AddPermission` and `RemovePermission`** — they add and remove labelled statements in the topic's `Policy`; a duplicate label answers `InvalidParameter`. Contributed by @AdrianAcala.
### Fixed
-- **SQS/SNS — queue and topic policies are enforced under `AUTH=true`** — the `Policy` attributes were stored but never evaluated, so S3→SQS and SNS→SQS deliveries landed and cross-account send/publish failed regardless of the policy. Same-account calls now pass unless the policy carries an explicit Deny; cross-account calls need an explicit Allow (`AccessDenied` for SQS, `AuthorizationError` for SNS); S3 and SNS deliveries are dropped unless the destination policy Allows the calling service with its `aws:SourceArn`/`aws:SourceAccount` conditions, exactly like real AWS — including queues and topics with no policy at all. With auth disabled every path stays permissive. `GetQueueUrl` accepts `QueueOwnerAWSAccountId`.
-- **S3 — notification destinations are validated at PUT time** — configuring an SQS/SNS target whose policy does not let S3 publish now fails the whole `PutBucketNotificationConfiguration` with `InvalidArgument` ("Unable to validate the following destination configurations"), like real S3, when `AUTH=true`; with auth disabled the probe checks existence only. SQS/SNS targets may live in another account when their policy allows the delivery. The `SkipDestinationValidation` option skips the probe (and the test event).
+- **SQS/SNS — queue and topic policies are enforced under `AUTH=true`** — the `Policy` attributes were stored but never evaluated. Same-account calls pass unless the policy explicitly denies them; cross-account calls need an explicit Allow (`AccessDenied` for SQS, `AuthorizationError` for SNS). S3→SQS, S3→SNS and SNS→SQS deliveries need the destination policy to allow the calling service. With `AUTH=false` every path stays permissive. `GetQueueUrl` accepts `QueueOwnerAWSAccountId`. Contributed by @AdrianAcala.
+- **S3 — notification destinations are validated at PUT time** — under `AUTH=true`, `PutBucketNotificationConfiguration` fails with `InvalidArgument` ("Unable to validate the following destination configurations") when an SQS/SNS destination's policy does not let S3 publish; with `AUTH=false` only existence is checked. Cross-account SQS/SNS destinations are accepted, and `SkipDestinationValidation` skips the check. Contributed by @AdrianAcala.
- **Bedrock AgentCore — opt-in Docker runtime** — set `MINISTACK_AGENTCORE_DOCKER=1` to start the declared `containerConfiguration.containerUri` image on first invocation, connect via a localhost port when MiniStack runs on the host or via MiniStack's Docker network when containerized, call `/ping`, and forward `InvokeAgentRuntime` to `/invocations` on port 8080. The container is reused and removed on update, delete, or reset. Container and worker HTTP errors become `RuntimeClientError` (424); caller credentials are not forwarded. Without Docker mode, the existing deterministic echo remains for SDK tests. This covers HTTP only and does not emulate AWS IAM/VPC isolation or model inference.
- **CloudWatch Logs — delivery destination policy** — `PutDeliveryDestinationPolicy`, `GetDeliveryDestinationPolicy`, `DeleteDeliveryDestinationPolicy`. Required by an operator that authorizes `delivery.logs.amazonaws.com` to create deliveries to a `DeliveryDestination` (e.g. CloudFront access-log delivery to a log group) — `PutDeliveryDestinationPolicy` previously had no route, blocking the delivery from ever being created. Contributed by @fabio-andre-rodrigues.
-- **S3 — object annotations** — `PutObjectAnnotation`, `GetObjectAnnotation`, `ListObjectAnnotations` (prefix, `max-annotation-results`, continuation tokens) and `DeleteObjectAnnotation`: named UTF-8 payloads of 1 byte to 1 MiB, up to 1,000 per object version, with S3's naming rules (`InvalidAnnotationName`, `AnnotationNameTooLong`), `AnnotationLimitExceeded`, `UnsupportedMediaType` and `x-amz-object-if-match`. Each belongs to one object version: a new version starts with none, deleting a version deletes its annotations, a delete marker keeps them, and an unversioned overwrite or delete drops them; writing one changes neither the object's ETag nor its version. `CopyObject` carries them unless `x-amz-object-annotation-directive: EXCLUDE`, and recomputes their checksums when the copy names a new algorithm. Payload checksums are verified (`BadDigest`) and default to CRC64NVME; CRC32, CRC32C, CRC64NVME, SHA1, SHA256, SHA512, MD5 and XXHASH64 are computed, while XXHASH3 and XXHASH128 are refused rather than stored unverified. Annotations take the object's encryption and are refused on SSE-C objects, replicate with the object (`COMPLETED` / `REPLICA`), respect Object Lock on every write (governance retention needs `x-amz-bypass-governance-retention` to put, overwrite or delete one; compliance retention and a legal hold refuse all three, bypass or not, each as 403 `AccessDenied` naming retention or the legal hold, as observed on AWS), persist per version with `S3_PERSIST=1`, and send `s3:ObjectAnnotation:Put` / `s3:ObjectAnnotation:Delete` events with the record's `objectAnnotation` block; a `CopyObject` event reports `hasObjectAnnotation`. Previously a `PUT ?annotation` fell through to `PutObject` and replaced the object with the payload. The S3 Metadata annotation table (`UpdateBucketMetadataAnnotationTableConfiguration`) is not included: MiniStack has no S3 Metadata configuration for it to belong to. Checked against the S3 API Reference, the S3 User Guide's "Annotating your objects" page and the botocore model; the Object Lock behaviour was validated against AWS (ap-southeast-2).
+- **S3 — object annotations** — `PutObjectAnnotation`, `GetObjectAnnotation`, `ListObjectAnnotations` and `DeleteObjectAnnotation`: up to 1,000 named UTF-8 payloads (1 byte to 1 MiB) per object version, with S3's naming rules and errors. `CopyObject` carries them unless `x-amz-object-annotation-directive: EXCLUDE`; checksums default to CRC64NVME (XXHASH3 and XXHASH128 are refused); annotations take the object's encryption (refused on SSE-C), replicate, persist with `S3_PERSIST=1`, and send `s3:ObjectAnnotation:Put` / `s3:ObjectAnnotation:Delete` events. Under `AUTH=true`, Object Lock refuses annotation writes with 403 `AccessDenied` (governance retention unless `x-amz-bypass-governance-retention: true`, which also needs `s3:BypassGovernanceRetention`; compliance retention and a legal hold always), and the operations authorize as `s3:PutObjectAnnotation`, `s3:GetObjectAnnotation`, `s3:ListObjectAnnotations` and `s3:DeleteObjectAnnotation`. Previously a `PUT ?annotation` replaced the object with the payload. Contributed by @BoLaMN.
### Fixed
-- **API Gateway v1 — REQUEST authorizer events carry `requestContext.identity`** — the event sent to a REQUEST Lambda authorizer had no `identity` block, so an authorizer reading `event.requestContext.identity.sourceIp` or `.userAgent` crashed and every request answered 500 or its own fallback Deny. It now reports `sourceIp` (first `X-Forwarded-For` hop) and `userAgent`, the same values the `AWS_PROXY` integration event already carries. Checked against the API Gateway Developer Guide ("Input to an API Gateway Lambda authorizer", whose REQUEST example includes `requestContext.identity.sourceIp`, and the `$context.identity.userAgent` context variable); a production REQUEST authorizer that reads `identity.userAgent` works on AWS. Not validated against a captured AWS event.
+- **API Gateway v1 — REQUEST authorizer events carry `requestContext.identity`** — the event had no `identity` block, so an authorizer reading `sourceIp` or `userAgent` crashed. It now carries both, as the `AWS_PROXY` integration event does. Contributed by @yosriady.
- **DynamoDB — `DescribeTable` accepts a table ARN** — `TableName` given as the table's ARN answered `ResourceNotFoundException`, where AWS accepts either form. It now resolves through the same ARN normalization the other table operations use, and an ARN from another account or region still answers `ResourceNotFoundException`. This unblocks OpenSearch Data Prepper's DynamoDB source, which always describes the table by ARN.
-- **S3 — event notification records match S3's** — the object block carries the key URL-encoded (`red flower.jpg` → `red+flower.jpg`), the `versionId` the write made or removed, and a `sequencer` that grows with every create and delete (it was always `"0"`); records are `eventVersion` `2.6`. A delete that leaves a delete marker is `ObjectRemoved:DeleteMarkerCreated` (it was `ObjectRemoved:Delete`, so a subscription to permanent deletes received markers), and `DeleteObjects` sends an event for each object it removes (it sent none). Checked against the S3 User Guide's "Event message structure" and "Event notification types and destinations" pages; not validated against a real AWS account.
-- **STS — `GetCallerIdentity` resolves IAM-user callers** — keys created with `CreateAccessKey` returned the `root` ARN; they now return the user's ARN and ID in both XML and JSON protocols. Under `AUTH=true`, unknown and inactive keys are rejected instead of reported as root.
+- **S3 — event notification records match S3's** — the key is URL-encoded (`red flower.jpg` → `red+flower.jpg`), `versionId` names the version the write made or removed, `sequencer` grows with every create and delete (it was always `"0"`), and `eventVersion` is `2.6`. A delete that leaves a delete marker is `ObjectRemoved:DeleteMarkerCreated` (it was `ObjectRemoved:Delete`), and `DeleteObjects` sends an event for each object it removes (it sent none). Contributed by @BoLaMN.
+- **STS — `GetCallerIdentity` resolves IAM-user callers** — keys created with `CreateAccessKey` returned the `root` ARN; they now return the user's ARN and ID. Under `AUTH=true`, unknown and inactive keys are refused with 403 `InvalidClientTokenId` instead of reported as root. Contributed by @AdrianAcala.
+- **STS — an unknown access key answers `InvalidClientTokenId`** — under `AUTH=true`, STS operations refused an unknown key with `UnrecognizedClientException`; real STS answers `InvalidClientTokenId`.
## [1.5.17] — 2026-09-25
diff --git a/README.md b/README.md
index af9dc11c..8dbfdd73 100644
--- a/README.md
+++ b/README.md
@@ -432,7 +432,7 @@ subnet = ec2.create_subnet(
| **Bedrock Runtime** | Converse, ConverseStream, InvokeModel, InvokeModelWithResponseStream, ApplyGuardrail, StartAsyncInvoke, GetAsyncInvoke, ListAsyncInvokes | Real eventstream wire format on streaming operations; **guardrails enforce their deterministic policies** — word policy, sensitive-information regexes, and the pattern-matchable PII types with `BLOCK`/`ANONYMIZE` per input/output stage — on `ApplyGuardrail` and on `Converse`/`ConverseStream` via `guardrailConfig` (`GUARDRAIL_INTERVENED`, blocked messaging, per-policy assessments, `ResourceNotFoundException` for an unknown id); the model-graded policies (topic, content, contextual grounding, `PROFANITY`) are not evaluated; deterministic family-aware mock responses selected by model ID prefix (`anthropic.*`, `amazon.titan*`, `amazon.nova*`, `meta.llama*`, `mistral.*`, `cohere.*`, `ai21.*`); verified against botocore `bedrock-runtime-2023-09-30` |
| **Bedrock Agent** | 72 operations — agents (versions, aliases, action groups, collaborators, agent knowledge bases, PrepareAgent), knowledge bases (data sources, ingestion jobs — `StartIngestionJob` reads the S3 data source and reports real statistics, `FAILED` with `failureReasons` when the source is unreadable — documents), flows (aliases, versions, ValidateFlowDefinition, PrepareFlow), prompts with versions, tagging | All 72 operations verified against botocore `bedrock-agent-2023-06-05` |
| **Bedrock Agent Runtime** | InvokeAgent, InvokeInlineAgent, GetAgentMemory, DeleteAgentMemory, Retrieve, RetrieveAndGenerate, RetrieveAndGenerateStream, Rerank, CreateSession, GetSession, UpdateSession, DeleteSession, EndSession, ListSessions, CreateInvocation, ListInvocations, PutInvocationStep, GetInvocationStep, ListInvocationSteps, InvokeFlow, StartFlowExecution, StopFlowExecution, GetFlowExecution, ListFlowExecutions, ListFlowExecutionEvents, GetExecutionFlowSnapshot, OptimizePrompt | All 31 operations verified against botocore `bedrock-agent-runtime-2023-07-26`; eventstream responses on streaming operations; `Retrieve` searches the documents Knowledge Base ingestion stored (lexical scoring — no embeddings) and returns content, `s3Location` and score; `InvokeAgent` answers a canned reply without orchestration and emits no trace events — it never fabricates an `ORCHESTRATE` trace |
-| **Bedrock AgentCore** | Control plane (`bedrock-agentcore-control`): CreateAgentRuntime, GetAgentRuntime, ListAgentRuntimes, UpdateAgentRuntime, DeleteAgentRuntime, ListAgentRuntimeVersions, CreateAgentRuntimeEndpoint, GetAgentRuntimeEndpoint, ListAgentRuntimeEndpoints, UpdateAgentRuntimeEndpoint, DeleteAgentRuntimeEndpoint. Data plane (`bedrock-agentcore`): InvokeAgentRuntime | Agent-runtime lifecycle with `CREATING`/`UPDATING`→`READY` transitions, versioned ARNs (`agent/{uuid}:{version}`, `agentEndpoint/{uuid}`); request/response shapes verified against botocore `bedrock-agentcore-control` / `bedrock-agentcore`; account- and region-scoped state; `InvokeAgentRuntime` echoes by default or, with `MINISTACK_AGENTCORE_DOCKER=1`, starts the declared container image and forwards HTTP invocations to its port 8080 (no model inference in MiniStack) |
+| **Bedrock AgentCore** | Control plane (`bedrock-agentcore-control`): CreateAgentRuntime, GetAgentRuntime, ListAgentRuntimes, UpdateAgentRuntime, DeleteAgentRuntime, ListAgentRuntimeVersions, CreateAgentRuntimeEndpoint, GetAgentRuntimeEndpoint, ListAgentRuntimeEndpoints, UpdateAgentRuntimeEndpoint, DeleteAgentRuntimeEndpoint. Data plane (`bedrock-agentcore`): InvokeAgentRuntime | Agent-runtime lifecycle with `CREATING`/`UPDATING`→`READY` transitions, versioned ARNs (`agent/{uuid}:{version}`, `agentEndpoint/{uuid}`); request/response shapes verified against botocore `bedrock-agentcore-control` / `bedrock-agentcore`; account- and region-scoped state; `InvokeAgentRuntime` runs the runtime's `containerUri` image in Docker (full image), waits for `/ping` and forwards to `/invocations` on port 8080, with container errors as `RuntimeClientError` (424); without Docker (slim image) or for a code artifact it answers a deterministic echo |
| **Transcribe** | StartTranscriptionJob, GetTranscriptionJob, ListTranscriptionJobs, DeleteTranscriptionJob | Batch transcription jobs only; streaming is a separate endpoint (`transcribestreaming`). All four operations verified against botocore `transcribe-2017-10-26`. A job starts `IN_PROGRESS` with `StartTime` set, as on AWS, and reaches `COMPLETED` over two seconds; `QUEUED`, which AWS reaches only for a request that opted into job queueing (`JobExecutionSettings.AllowDeferredExecution`) while the account is at its concurrent job limit, is not modelled; the media object is read from MiniStack S3 via an `s3://`, path-style or virtual-hosted HTTPS URI, and an unreadable URI fails the job with a `FailureReason` rather than erroring at `Start`, as on AWS; the transcript is written back to S3 in the real result format (`results.transcripts`, word-level `results.items`, `results.audio_segments`, plus `results.segments` and `results.speaker_labels` when the matching `Settings` are set) at `OutputBucketName`/`OutputKey`, or to an auto-created `ministack-transcribe-output` bucket. `IdentifyLanguage` and `IdentifyMultipleLanguages` resolve a language on completion, `ContentRedaction` controls whether a `redacted-` copy replaces or accompanies the transcript, and `Subtitles` writes WebVTT and SubRip files, redacted alongside the transcript when redaction is on. Reaching a terminal state publishes a `Transcribe Job State Change` event, so EventBridge rules downstream of a job fire. There is no speech recognition, so the transcript is deterministic canned text keyed on a digest of the media URI |
| **Translate** | StartTextTranslationJob, DescribeTextTranslationJob, ListTextTranslationJobs, StopTextTranslationJob | Batch text translation jobs only; the synchronous `TranslateText` / `TranslateDocument` pair and the custom terminology / parallel data families are not implemented. All four operations verified against botocore `translate-2017-07-01`. Jobs walk `SUBMITTED` → `IN_PROGRESS` → `COMPLETED` over `TRANSLATE_JOB_RUN_SECONDS` (default 2, `0` completes immediately); every document under `InputDataConfig.S3Uri` is read from MiniStack S3 and written back to the rewritten output location AWS uses — `OutputDataConfig.S3Uri` is reported as `s3://<bucket>/<prefix><account-id>-TranslateText-<JobId>/` — as `<target-code>.<input name>`, with a `details/<code>.auxiliary-translation-details.json` summary per target language. `text/plain` is translated line by line, `text/html` keeps its markup and translates only text nodes, `application/x-xliff+xml` gets a populated `<target>` in every `<trans-unit>` plus `target-language` on every `<file>`, and the three Office ZIP types are copied through. A document that can't be read as its content type is counted in `JobDetails` and the job ends `COMPLETED_WITH_ERROR`; an empty or unreadable input location fails the job with a `Message` rather than erroring at `Start`. `ClientToken` is idempotent, `StopTextTranslationJob` reaches a running worker, and reaching a terminal state publishes a `Translate TextTranslationJob State Change` event, so EventBridge rules downstream of a job fire. There is no machine translation, so the output is deterministic canned text derived from the source |
| **MSK** | CreateCluster, ListClusters, DescribeCluster, DeleteCluster, GetBootstrapBrokers, ListNodes, CreateConfiguration, ListConfigurations, DescribeConfiguration, ListConfigurationRevisions, DescribeConfigurationRevision, BatchAssociateScramSecret, BatchDisassociateScramSecret, ListScramSecrets, TagResource, UntagResource, ListTagsForResource | Kafka control plane; `GetBootstrapBrokers` honors `MINISTACK_MSK_BOOTSTRAP` so clients route to a real broker you bring (Redpanda, Kafka, KRaft) — the Kafka wire protocol itself is not emulated |
@@ -865,7 +865,6 @@ end-to-end without any client config.
| `MINISTACK_IMDS_V2_REQUIRED` | `0` | Reject token-less GETs on `/latest/meta-data/...`. When set, callers must first `PUT /latest/api/token` and pass the token as `X-aws-ec2-metadata-token`, matching real-AWS hop-limit-1 IMDSv2-only instances |
| `MINISTACK_BEDROCK_PROXY_URL` | _(unset)_ | OpenAI-compatible `/chat/completions` endpoint (Ollama, llama.cpp, vLLM). When set, Bedrock `Converse`/`InvokeModel` prompts are translated to OpenAI shape, forwarded, and translated back to Bedrock shape — real completions through the Bedrock API. Falls back to the deterministic mock on connection error |
| `MINISTACK_BEDROCK_PROXY_TIMEOUT_SECONDS` | `30` | Upstream request timeout for the Bedrock proxy |
-| `MINISTACK_AGENTCORE_DOCKER` | _(unset)_ | Set to `1` to run the image in `agentRuntimeArtifact.containerConfiguration.containerUri` on first `InvokeAgentRuntime`. When MiniStack runs on the host, it publishes the worker's port 8080 on localhost; when MiniStack runs in Docker, it joins the worker to MiniStack's Docker network. It calls `/ping`, then forwards to `/invocations`. The container is reused until the runtime is updated, deleted, or MiniStack resets. Requires a reachable Docker daemon and an image Docker can pull or find locally. Container and HTTP errors return `RuntimeClientError` (424). Without this opt-in, the existing deterministic echo remains available for control-plane and SDK tests. This covers HTTP; IAM, VPC isolation, MCP, A2A and WebSocket are not emulated. **Keep the MiniStack API private when this is enabled:** anyone who can create and invoke a runtime can cause Docker to start an image of their choosing. Use only trusted images. |
| `MINISTACK_MSK_BOOTSTRAP` | _(unset)_ | Plaintext bootstrap broker string returned by MSK `GetBootstrapBrokers` — point it at a real broker you bring (Redpanda, Kafka, KRaft). Unset → placeholder endpoint (control-plane only) |
| `MINISTACK_MSK_BOOTSTRAP_TLS` | _(unset)_ | TLS bootstrap string (`BootstrapBrokerStringTls`) |
| `MINISTACK_MSK_BOOTSTRAP_SASL_SCRAM` | _(unset)_ | SASL/SCRAM bootstrap string (`BootstrapBrokerStringSaslScram`) |
@@ -876,6 +875,25 @@ end-to-end without any client config.
API Gateway v1/v2 HTTP proxy forwarding uses non-blocking event-loop semantics by offloading the upstream socket call to a worker thread. This preserves AWS-compatible response behavior while preventing long-running proxy calls from stalling unrelated requests (for example, parallel DynamoDB operations). The same non-blocking path is used for JWT **JWKS** fetches. Tune wall-clock limits with `MINISTACK_APIGW_PROXY_TIMEOUT_SECONDS` and `MINISTACK_APIGW_JWKS_TIMEOUT_SECONDS` at the bottom of the [Configuration](#configuration) table above.
+### Cognito OIDC discovery (`USE_SSL=1`)
+
+A Cognito token's `iss` is `https://cognito-idp.<region>.amazonaws.com/<poolId>`, exactly as on AWS. To let an unmodified client (Spring, Vault, `aws-jwt-verify`) follow it to MiniStack:
+
+```bash
+docker run -d --name ministack -p 443:4566 -p 4566:4566 -e USE_SSL=1 ministackorg/ministack
+docker cp ministack:/tmp/ministack-tls/server.crt ./ministack-ca.pem
+```
+
+Then point the issuer host at MiniStack in `/etc/hosts` (`127.0.0.1 cognito-idp.us-east-1.amazonaws.com`) and trust `ministack-ca.pem` in your client's runtime. The OS keychain is not enough for these:
+
+| Runtime | Trust step |
+|---|---|
+| Java | `keytool -importcert -file ministack-ca.pem -alias ministack -cacerts -storepass changeit` |
+| Node | `NODE_EXTRA_CA_CERTS=./ministack-ca.pem` |
+| Python (`requests`, boto3) | `REQUESTS_CA_BUNDLE` / `AWS_CA_BUNDLE` pointing at a bundle of your system roots plus `ministack-ca.pem` |
+
+The generated certificate covers every region's issuer host. Lambda containers MiniStack starts get the hosts entries and the trust settings automatically. Remove the `/etc/hosts` line before calling the real Cognito.
+
### Startup Scripts
MiniStack supports two types of init scripts, with LocalStack-compatible paths:
diff --git a/ministack/app.py b/ministack/app.py
index 2ea6624b..41893cdc 100644
--- a/ministack/app.py
+++ b/ministack/app.py
@@ -3239,6 +3239,16 @@ def main():
logging.getLogger("hypercorn.access").addFilter(_HealthLogFilter())
+ # Raise the soft fd limit to the hard one so connection bursts don't hit 1024.
+ try:
+ import resource
+
+ _soft, _hard = resource.getrlimit(resource.RLIMIT_NOFILE)
+ if _soft < _hard:
+ resource.setrlimit(resource.RLIMIT_NOFILE, (_hard, _hard))
+ except (ImportError, ValueError, OSError):
+ pass
+
config = HypercornConfig()
config.bind = [f"{bind_host}:{port}"]
config.keep_alive_timeout = 75
diff --git a/ministack/core/iam_actions.py b/ministack/core/iam_actions.py
index cd6c9301..e298764e 100644
--- a/ministack/core/iam_actions.py
+++ b/ministack/core/iam_actions.py
@@ -244,6 +244,8 @@ _S3_OBJECT_QUERY_ACTIONS: dict[str, dict[str, str]] = {
"retention": {"GET": "GetObjectRetention", "PUT": "PutObjectRetention"},
"legal-hold": {"GET": "GetObjectLegalHold", "PUT": "PutObjectLegalHold"},
"attributes": {"GET": "GetObjectAttributes"},
+ "annotation": {"GET": "GetObjectAnnotation", "PUT": "PutObjectAnnotation",
+ "DELETE": "DeleteObjectAnnotation"},
"select": {"POST": "GetObject"}, # SelectObjectContent reads the object
"torrent": {"GET": "GetObject"},
}
@@ -282,6 +284,9 @@ def _s3_action(method: str, path: str, query_params: dict) -> str | None:
break
if action is None:
action = _S3_ACTIONS.get((method, depth))
+ # GET ?annotation without annotationName is ListObjectAnnotations.
+ if action == "GetObjectAnnotation" and "annotationName" not in query_params:
+ action = "ListObjectAnnotations"
if depth == 2 and action and _query_param(query_params, "versionId"):
action = _S3_VERSIONED_ACTIONS.get(action, action)
@@ -290,7 +295,8 @@ def _s3_action(method: str, path: str, query_params: dict) -> str | None:
# Operations that take x-amz-bypass-governance-retention; when the header says
# true they also need s3:BypassGovernanceRetention on the object.
-_S3_GOVERNANCE_BYPASS_ACTIONS = frozenset({"DeleteObject", "DeleteObjectVersion", "PutObjectRetention"})
+_S3_GOVERNANCE_BYPASS_ACTIONS = frozenset({"DeleteObject", "DeleteObjectVersion", "PutObjectRetention",
+ "PutObjectAnnotation", "DeleteObjectAnnotation"})
def _s3_source_object(headers: dict) -> tuple[str, str] | None:
diff --git a/ministack/core/iam_evaluator.py b/ministack/core/iam_evaluator.py
index 0994c7a3..98cb4b4c 100644
--- a/ministack/core/iam_evaluator.py
+++ b/ministack/core/iam_evaluator.py
@@ -606,6 +606,9 @@ _request_caller_arn: contextvars.ContextVar[str] = contextvars.ContextVar(
def pin_request_caller(headers: dict, query_params: dict | None = None) -> None:
"""Remember the caller's principal ARN for this request's policy checks."""
+ from ministack.app import AUTH
+ if not AUTH:
+ return
_request_caller_arn.set(request_caller_arn(headers, query_params))
diff --git a/ministack/core/tls.py b/ministack/core/tls.py
index ca4abbc3..f193477c 100644
--- a/ministack/core/tls.py
+++ b/ministack/core/tls.py
@@ -15,11 +15,7 @@ import tempfile
def _write_atomic(path: str, data: bytes) -> "str | None":
- """Write `data` to `path` via a rename, or None on failure.
-
- Containers spawn concurrently and mount these files, so a reader must never
- see a half-written one.
- """
+ """Write `data` to `path` via an atomic rename; None on failure."""
try:
handle, staging = tempfile.mkstemp(dir=os.path.dirname(path))
with os.fdopen(handle, "wb") as out:
@@ -40,12 +36,16 @@ def cognito_idp_host() -> str:
return f"cognito-idp.{region}.amazonaws.com"
-def ca_bundle_path(cert_path: str) -> "str | None":
- """Public roots plus `cert_path`, or None.
+def cognito_idp_hosts() -> "list[str]":
+ """Every region's Cognito issuer host, so a pool in any region validates."""
+ from ministack.services.account import _REGIONS_LIST
- AWS_CA_BUNDLE and REQUESTS_CA_BUNDLE replace the trust store rather than
- adding to it, so ours alone would cost the caller every other endpoint.
- """
+ hosts = [cognito_idp_host()] + [f"cognito-idp.{r}.amazonaws.com" for r in _REGIONS_LIST]
+ return list(dict.fromkeys(hosts))
+
+
+def ca_bundle_path(cert_path: str) -> "str | None":
+ """System roots plus `cert_path` (the *_CA_BUNDLE vars replace the store), or None."""
import ssl
paths = ssl.get_default_verify_paths()
@@ -63,10 +63,7 @@ JAVA_TRUSTSTORE_PASSWORD = "changeit"
def java_truststore_path(cert_path: str) -> "str | None":
- """The same trust as `ca_bundle_path` in the PKCS12 a JVM reads, or None.
-
- Written here rather than with `keytool` so no JDK is needed on this side.
- """
+ """The same trust as a PKCS12 truststore for the JVM, or None."""
try:
from cryptography import x509
from cryptography.hazmat.primitives.serialization import (
@@ -98,8 +95,8 @@ def java_truststore_path(cert_path: str) -> "str | None":
return _write_atomic(os.path.join(os.path.dirname(cert_path), "truststore.p12"), blob)
-def _cert_names(cert_path: str, name: str) -> bool:
- """Whether the certificate at `cert_path` carries `name` as a SAN."""
+def _cert_names(cert_path: str, names: "list[str]") -> bool:
+ """Whether the certificate at `cert_path` carries every name in `names` as a SAN."""
try:
out = subprocess.run(
["openssl", "x509", "-in", cert_path, "-noout", "-ext", "subjectAltName"],
@@ -107,7 +104,7 @@ def _cert_names(cert_path: str, name: str) -> bool:
)
except OSError:
return True # No openssl to check with; leave the cached cert alone.
- return name in (out.stdout or "")
+ return all(f"DNS:{name}" in (out.stdout or "") for name in names)
def resolve_tls_material() -> "tuple[str, str]":
@@ -135,8 +132,8 @@ def resolve_tls_material() -> "tuple[str, str]":
cert_path = os.path.join(tls_dir, "server.crt")
key_path = os.path.join(tls_dir, "server.key")
if (os.path.exists(cert_path) and os.path.exists(key_path)
- and not _cert_names(cert_path, cognito_idp_host())):
- # Cached by an older build or another region: cannot serve that name.
+ and not _cert_names(cert_path, cognito_idp_hosts())):
+ # Cached by an older build: regenerate so every issuer host is covered.
os.remove(cert_path)
os.remove(key_path)
if not (os.path.exists(cert_path) and os.path.exists(key_path)):
@@ -147,7 +144,8 @@ def resolve_tls_material() -> "tuple[str, str]":
"-subj", "/CN=ministack-local/O=MiniStack",
"-addext",
"subjectAltName=DNS:localhost,DNS:ministack,"
- f"DNS:{cognito_idp_host()},IP:127.0.0.1,IP:0:0:0:0:0:0:0:1",
+ + "".join(f"DNS:{host}," for host in cognito_idp_hosts())
+ + "IP:127.0.0.1,IP:0:0:0:0:0:0:0:1",
], check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
os.chmod(cert_path, 0o600)
os.chmod(key_path, 0o600)
diff --git a/ministack/services/bedrock_agentcore.py b/ministack/services/bedrock_agentcore.py
index 89f375b8..3f0d6516 100644
--- a/ministack/services/bedrock_agentcore.py
+++ b/ministack/services/bedrock_agentcore.py
@@ -426,7 +426,8 @@ def _invoke_agent_runtime(runtime_arn, headers, body):
session_id = (headers.get("x-amzn-bedrock-agentcore-runtime-session-id")
or new_uuid())
content_type = headers.get("content-type", "application/json")
- if os.environ.get("MINISTACK_AGENTCORE_DOCKER") == "1":
+ artifact = runtime.get("agentRuntimeArtifact", {}).get("containerConfiguration", {})
+ if artifact.get("containerUri") and _docker_client() is not None:
try:
url = _container_invocations_url(runtime)
except (RuntimeError, ValueError) as error:
@@ -468,6 +469,23 @@ def _local_open(request, *, timeout):
return _LOCAL_HTTP.open(request, timeout=timeout)
+_docker = None
+
+
+def _docker_client():
+ """A Docker client when the SDK is installed and the daemon answers, else None."""
+ global _docker
+ if _docker is None:
+ try:
+ import docker
+ client = docker.from_env(timeout=60)
+ client.ping()
+ _docker = client
+ except Exception:
+ return None
+ return _docker
+
+
def _container_key(runtime_id):
return get_account_id(), get_region(), runtime_id
@@ -519,13 +537,15 @@ def _container_invocations_url(runtime):
raise RuntimeError(f"Could not inspect runtime container: {error}") from error
if container is None:
client = None
- labels = {"ministack": "agentcore",
- "ministack.agentcore.runtime": runtime["agentRuntimeId"],
- "ministack.agentcore.account": key[0],
- "ministack.agentcore.region": key[1]}
+ from ministack.core.container_reaper import own_labels
+
+ labels = own_labels("agentcore", **{
+ "ministack.agentcore.runtime": runtime["agentRuntimeId"],
+ "ministack.agentcore.account": key[0],
+ "ministack.agentcore.region": key[1]})
try:
import docker
- client = docker.from_env(timeout=60)
+ client = _docker_client() or docker.from_env(timeout=60)
run_kwargs = {
"environment": runtime.get("environmentVariables", {}),
"labels": labels,
diff --git a/ministack/services/lambda_svc.py b/ministack/services/lambda_svc.py
index 800db387..b55cdb5c 100644
--- a/ministack/services/lambda_svc.py
+++ b/ministack/services/lambda_svc.py
@@ -4089,11 +4089,7 @@ _CONTAINER_TRUSTSTORE_PATH = "/var/ministack/truststore.p12"
def _wire_cognito_issuer_host(run_kwargs, container_env, mounts, runtime=""):
- """Resolve the host a Cognito token's `iss` names to the gateway, and trust it.
-
- A verifier built from the pool id alone has no other way to reach us. Only
- under USE_SSL=1: `iss` is https, so plain HTTP has nothing to resolve to.
- """
+ """Under USE_SSL=1, resolve the Cognito issuer hosts to the gateway and trust its cert."""
from ministack.core import tls as _tls
if not _tls.use_ssl_enabled():
@@ -4102,8 +4098,9 @@ def _wire_cognito_issuer_host(run_kwargs, container_env, mounts, runtime=""):
cert_path, _key_path = _tls.resolve_tls_material()
except SystemExit:
return
- run_kwargs.setdefault("extra_hosts", {}).setdefault(
- _tls.cognito_idp_host(), "host-gateway")
+ extra_hosts = run_kwargs.setdefault("extra_hosts", {})
+ for host in _tls.cognito_idp_hosts():
+ extra_hosts.setdefault(host, "host-gateway")
if not os.path.exists(cert_path):
return
# NODE_EXTRA_CA_CERTS adds to node's roots; the other two replace the store.
@@ -4904,8 +4901,7 @@ def _inflight_key(config: dict) -> str:
account, region = _account_region_from_function_config(config)
except Exception:
account, region = get_account_id(), get_region()
- # Reserved concurrency belongs to the function, not an individual
- # published version or alias. All versions therefore share one counter.
+ # Reserved concurrency is shared across all versions and aliases.
return f"{account}:{region}:{config.get('FunctionName', '?')}"
diff --git a/ministack/services/s3.py b/ministack/services/s3.py
index bed909ae..09c1fc4b 100644
--- a/ministack/services/s3.py
+++ b/ministack/services/s3.py
@@ -144,8 +144,6 @@ _bucket_request_payment_config = AccountScopedDict()
_object_tags = AccountScopedDict()
_object_acl = AccountScopedDict() # (bucket, key, version_id) -> stored ACL XML string
-# (bucket, key, version_id) -> {annotation name: {payload, size, etag, last_modified, checksums,
-# replication_status}}; version_id is None for the object without one, as for tags.
_object_annotations = AccountScopedDict()
_object_versions = AccountScopedDict() # (bucket, key) -> [{version_id, obj_record}, ...]
@@ -3433,30 +3431,20 @@ def _s3_event_to_eventbridge(event_name: str) -> tuple[str, str | None]:
return detail_type, reason
-# The notification record's `sequencer`: a hexadecimal string that grows with
-# every create and delete, so "the event notification with the greater
-# `sequencer` hexadecimal value is the event that occurred later" for one key
-# (S3 User Guide, "Event message structure"). Taken when the write happens,
-# not when the background thread builds the record, so it orders the writes.
-# Seeded from the clock so values keep growing across a restart.
+# Clock-seeded so sequencers keep growing across restarts.
_event_sequence = itertools.count(time.time_ns() // 1000)
_EVENT_SEQUENCED_FAMILIES = ("s3:ObjectCreated:", "s3:ObjectRemoved:")
def _next_event_sequencer(event_name: str) -> str | None:
- """A sequencer for an object create or delete event; None for the others,
- which the record carries no sequencer for ("only used with PUT and DELETE
- requests")."""
+ """Sequencer for create/delete events only; None otherwise."""
if not event_name.startswith(_EVENT_SEQUENCED_FAMILIES):
return None
return f"{next(_event_sequence):018X}"
def _event_object_key(key: str) -> str:
- """The key as a notification record carries it: "The object key name value
- is URL encoded. For example, `red flower.jpg` becomes `red+flower.jpg`"
- (S3 User Guide, "Event message structure"); slashes are left as they are.
- EventBridge details carry the key as stored."""
+ """URL-encoded key for notification records (EventBridge keeps it raw)."""
return _url_quote_plus(key, safe="/")
@@ -3473,13 +3461,7 @@ def _fire_s3_event(
object_extra: dict | None = None,
record_extra: dict | None = None,
) -> None:
- """Build and deliver an S3 event notification. Best-effort — errors are logged.
-
- `version_id` is the version the write made or removed ("null or not present
- if the bucket isn't versioning-enabled"), `sequencer` the one taken when it
- happened, `object_extra` fields the record's `s3.object` gains for this
- event (`hasObjectAnnotation` on a copy), and `record_extra` fields the record
- itself gains (`objectAnnotation` on an annotation event)."""
+ """Build and deliver an S3 event notification. Best-effort — errors are logged."""
try:
configs = _parse_notification_config(bucket_name)
raw_xml = _bucket_notifications.get(bucket_name, "")
@@ -4634,11 +4616,7 @@ def _record_object_version(bucket_name: str, key: str, prior_obj: dict | None, o
overwrite), preserved as the null version when it predates versioning."""
versioning = _bucket_versioning.get(bucket_name)
if versioning != "Enabled":
- # An unversioned or suspended write replaces the object that has no
- # version id, and "Overwriting an object replaces its annotations with
- # whatever annotations the new version has": none, until CopyObject or
- # PutObjectAnnotation gives it some. An Enabled write mints a new
- # version, which has none of its own.
+ # Overwriting the null version drops its annotations.
_object_annotations.pop((bucket_name, key, None), None)
if versioning not in ("Enabled", "Suspended"):
return None
@@ -4756,8 +4734,6 @@ def _delete_object_version(bucket: dict, bucket_name: str, key: str, version_id:
# Per-version tags and ACLs travel with the version being removed.
_object_tags.pop((bucket_name, key, version_id), None)
_object_acl.pop((bucket_name, key, version_id), None)
- # "Deleting a specific version ID deletes that version and all associated
- # annotations." The null version's are keyed None, as its tags are read.
_object_annotations.pop((bucket_name, key, None if version_id == "null" else version_id), None)
if not versions:
@@ -4857,8 +4833,6 @@ def _delete_object(bucket_name: str, key: str, headers: dict | None = None, quer
existed = key in bucket["objects"]
bucket["objects"].pop(key, None)
if existed:
- # "s3:ObjectRemoved:DeleteMarkerCreated" is its own event type:
- # s3:ObjectRemoved:Delete is for an object or version that is gone.
_fire_s3_event_async(
bucket_name,
key,
@@ -5014,10 +4988,7 @@ def _copy_object(bucket_name: str, dest_key: str, headers: dict):
if canned_acl and canned_acl not in _CANNED_OBJECT_ACLS:
return _error("InvalidArgument", f"Invalid x-amz-acl value: {canned_acl}", 400)
- # "CopyObject – Copies annotations from the source object by default",
- # COPY or EXCLUDE (x-amz-object-annotation-directive, the botocore model's
- # AnnotationDirective). A new checksum algorithm on the copy "applies to
- # both the object and its annotations", so it must be one they can take.
+ # A new checksum algorithm on the copy also applies to the annotations.
annotation_directive = headers.get("x-amz-object-annotation-directive", "COPY").upper()
if annotation_directive not in ("COPY", "EXCLUDE"):
return _error("InvalidArgument", f"Invalid annotation directive: {annotation_directive}", 400)
@@ -5152,8 +5123,6 @@ def _copy_object(bucket_name: str, dest_key: str, headers: dict):
if src_tags:
pending_dest_tags = dict(src_tags)
- # Annotations of the version copied, for the version the copy makes. Their
- # checksums are the source's unless the copy names another algorithm.
pending_dest_annotations: dict | None = None
if annotation_directive == "COPY":
src_annotations = _object_annotations.get((src_bucket_name, src_key, src_obj.get("version_id")))
@@ -5197,8 +5166,6 @@ def _copy_object(bucket_name: str, dest_key: str, headers: dict):
size=dest_obj["size"],
etag=new_etag,
version_id=version_id,
- # "If the copied object has annotations, the event includes a
- # hasObjectAnnotation field set to true."
object_extra={"hasObjectAnnotation": bool(pending_dest_annotations)},
)
@@ -5371,26 +5338,12 @@ def _delete_object_tagging(bucket_name: str, key: str, query_params: dict | None
# ---------------------------------------------------------------------------
# Object annotations
# ---------------------------------------------------------------------------
-# PutObjectAnnotation, GetObjectAnnotation, ListObjectAnnotations and
-# DeleteObjectAnnotation. Evidence: the S3 API Reference pages for the four
-# operations, the S3 User Guide's "Annotating your objects", "Event
-# notification types and destinations" and "Event message structure" pages,
-# and the botocore s3 model (1.43.63). Not validated against a real AWS
-# account. Where the documentation is silent the choice is marked inference.
-#
-# An annotation belongs to one object version: a new version starts with none,
-# deleting a version deletes its annotations, and a delete marker leaves the
-# annotations of the version beneath it alone. Writing one changes nothing
-# about the object (not its ETag, not its version) and sends only the
-# s3:ObjectAnnotation:* events.
_ANNOTATION_MAX_PER_VERSION = 1000
_ANNOTATION_MAX_NAME_BYTES = 512
_ANNOTATION_MAX_PAYLOAD = 1024 * 1024
_ANNOTATION_NAME_PUNCTUATION = frozenset("0123456789_.-")
-# The algorithms an annotation takes. MiniStack computes all but the two
-# XXHASH3 variants, which have no stdlib or small-table implementation; like
-# PutObject's unverifiable algorithms they are refused, never stored unchecked.
+# XXHASH3/XXHASH128 have no stdlib implementation and are refused.
_ANNOTATION_CHECKSUM_ALGORITHMS = (
"CRC32",
"CRC32C",
@@ -5404,9 +5357,7 @@ _ANNOTATION_CHECKSUM_ALGORITHMS = (
"XXHASH128",
)
-# XXH64 (the xxHash specification, "XXH64 algorithm description"): plain
-# 64-bit arithmetic, so it stays in the stdlib. Check values in the tests:
-# xxh64(b"") == 0xEF46DB3751D8E999, xxh64(b"abc") == 0x44BC2CF5AD770999.
+# XXH64 per the xxHash spec.
_XXH64_P1 = 0x9E3779B185EBCA87
_XXH64_P2 = 0xC2B2AE3D27D4EB4F
_XXH64_P3 = 0x165667B19E3779F9
@@ -5463,8 +5414,7 @@ def _xxh64(data: bytes, seed: int = 0) -> int:
def _annotation_checksum(algorithm: str, data: bytes) -> str | None:
- """Base64 checksum of an annotation payload, or None for an algorithm that
- is not an annotation's or that MiniStack cannot compute."""
+ """Base64 checksum of a payload, or None if the algorithm is unsupported."""
algo = (algorithm or "").upper().replace("_", "")
if algo not in _ANNOTATION_CHECKSUM_ALGORITHMS:
return None
@@ -5487,14 +5437,7 @@ def _annotation_checksum_unsupported(algorithm: str) -> tuple:
def _resolve_annotation_checksums(payload: bytes, headers: dict):
- """The checksum stored with an annotation, and the refusal for a bad one.
-
- One algorithm per payload: the one the client sent a value for, or named
- in x-amz-sdk-checksum-algorithm, or else CRC64NVME ("If the annotation
- doesn't have a specified checksum algorithm or checksum value, Amazon S3
- uses the CRC-64/NVME algorithm"). A value sent is checked against the
- payload (BadDigest), as PutObject checks its body; so is Content-MD5.
- Returns ``(checksums, error_or_None)``."""
+ """Return ``(checksums, error_or_None)``; CRC64NVME when none is named."""
provided = {
alg: headers[f"x-amz-checksum-{alg.lower()}"]
for alg in _ANNOTATION_CHECKSUM_ALGORITHMS
@@ -5529,9 +5472,7 @@ def _resolve_annotation_checksums(payload: bytes, headers: dict):
def _annotation_name_error(name: str) -> tuple | None:
- """The annotation naming rules: 1 to 512 bytes of letters (any language),
- digits, underscore, period and hyphen, not all whitespace, not starting
- with "aws" or "s3" in any case (S3 User Guide, "Annotation naming rules")."""
+ """Validate an annotation name against S3's naming rules."""
if not name or not name.strip():
return _error("InvalidAnnotationName", "The annotation name you provided is invalid.", 400)
if len(name.encode("utf-8")) > _ANNOTATION_MAX_NAME_BYTES:
@@ -5544,15 +5485,7 @@ def _annotation_name_error(name: str) -> tuple | None:
def _annotation_target(bucket_name: str, key: str, query_params: dict):
- """The object version an annotation operation acts on.
-
- Returns ``((bucket, version_key, record), None)`` or ``(None, error)``:
- `version_key` keys the version's annotations (None for the object without
- a version id, as for tags) and `record` is the current-object record or
- version entry, for its ETag and encryption. With no ``versionId`` it is
- the current object; a key whose current version is a delete marker is
- NoSuchKey, as a GetObject of it is. A ``versionId`` naming a delete marker
- is refused as a read of one is (inference: the documentation is silent)."""
+ """Resolve the object version an annotation call acts on."""
bucket = _ensure_bucket(bucket_name)
if bucket is None:
return None, _no_such_bucket(bucket_name)
@@ -5578,9 +5511,7 @@ def _annotation_target(bucket_name: str, key: str, query_params: dict):
def _annotation_version_header(bucket_name: str, version_key: str | None) -> dict:
- """x-amz-object-version-id: the version annotated. The object without a
- version id is "null" in a bucket that has had versioning; in one that
- never has, the header is left off, as x-amz-version-id is."""
+ """x-amz-object-version-id for the annotated version."""
if version_key:
return {"x-amz-object-version-id": version_key}
if _bucket_versioning.get(bucket_name) in ("Enabled", "Suspended"):
@@ -5589,9 +5520,7 @@ def _annotation_version_header(bucket_name: str, version_key: str | None) -> dic
def _annotation_sse_headers(record: dict) -> dict:
- """Annotations take the parent object's encryption: SSE-KMS and DSSE-KMS
- with the object's key, and SSE-S3 otherwise, including for an object with
- no server-side encryption (S3 User Guide, "Encryption")."""
+ """Annotations inherit the object's encryption; SSE-S3 when it has none."""
sse = _stored_sse_headers(record)
out = {"x-amz-server-side-encryption": sse.get("x-amz-server-side-encryption") or "AES256"}
if sse.get("x-amz-server-side-encryption-aws-kms-key-id"):
@@ -5600,8 +5529,7 @@ def _annotation_sse_headers(record: dict) -> dict:
def _annotation_if_match(headers: dict, record: dict) -> tuple | None:
- """x-amz-object-if-match: the operation "only succeeds if the object's ETag
- matches the provided value"; it checks the object, never the annotation."""
+ """x-amz-object-if-match checks the object's ETag."""
expected = headers.get("x-amz-object-if-match")
if expected and expected != "*" and expected.strip('"') != (record.get("etag") or "").strip('"'):
return _error("PreconditionFailed", "At least one of the pre-conditions you specified did not hold", 412)
@@ -5616,13 +5544,10 @@ def _annotation_checksum_headers(entry: dict) -> dict:
def _annotation_lock_error(bucket_name: str, key: str, headers: dict) -> tuple | None:
- """Object Lock over an annotation write (put, overwrite or delete), as AWS answers it.
-
- Observed on AWS (ap-southeast-2, 2026-09-28): governance retention refuses
- both unless x-amz-bypass-governance-retention is true; compliance retention
- and a legal hold refuse both whatever the header says. Each refusal is 403
- AccessDenied, naming retention or the legal hold.
- """
+ """Object Lock refusal for an annotation write (observed on AWS)."""
+ from ministack.app import AUTH
+ if not AUTH:
+ return None
if _object_legal_hold.get((bucket_name, key)) == "ON":
return _error("AccessDenied", "Access Denied because object protected by object lock legal hold.", 403)
retention = _object_retention.get((bucket_name, key))
@@ -5681,9 +5606,7 @@ def _put_object_annotation(bucket_name: str, key: str, body: bytes, headers: dic
"last_modified": now_iso(),
"checksums": checksums,
}
- # "If you have S3 Replication configured on your bucket, Amazon S3
- # replicates annotations automatically": onto the replica of the version,
- # when it is the one this bucket replicated.
+ # Annotations replicate onto the version's replica.
current = bucket["objects"].get(key)
if current is not None and current.get("version_id") == version_key and current.get("_replica"):
replica_bucket, replica_version = current["_replica"]
@@ -5772,8 +5695,7 @@ def _list_object_annotations(bucket_name: str, key: str, query_params: dict):
except (ValueError, UnicodeDecodeError):
return _error("InvalidArgument", "The continuation token provided is incorrect.", 400)
- # Names in UTF-8 binary order, as S3 lists keys (inference: the operation
- # documents no order); the token is the last name returned.
+ # UTF-8 binary order (undocumented); the token is the last name returned.
annotations = _object_annotations.get((bucket_name, key, version_key)) or {}
names = sorted(
(n for n in annotations if n.startswith(prefix) and (not after or n.encode() > after.encode())),
@@ -5827,8 +5749,7 @@ def _delete_object_annotation(bucket_name: str, key: str, headers: dict, query_p
store_key = (bucket_name, key, version_key)
annotations = dict(_object_annotations.get(store_key) or {})
version_headers = _annotation_version_header(bucket_name, version_key)
- # Deleting an annotation that is not there succeeds, as the operation
- # documents no NoSuchAnnotation; only a real delete sends the event.
+ # A missing annotation still answers 204; only a real delete fires the event.
if annotations.pop(name, None) is not None:
if annotations:
_object_annotations[store_key] = annotations
@@ -6635,8 +6556,7 @@ def _delete_objects(bucket_name: str, body: bytes, headers: dict = None):
# S3 reports the delete as successful even if the version was absent.
_found, was_marker = _delete_object_version(bucket, bucket_name, k, version_id)
deleted.append({"key": k, "version_id": version_id, "was_marker": was_marker})
- # "notification when an object or a batch of objects is removed":
- # each entry sends what the single-object DELETE would.
+ # Each entry sends the single-object DELETE's event.
if _found:
_fire_s3_event_async(bucket_name, k, "s3:ObjectRemoved:Delete", version_id=version_id)
elif _bucket_versioning.get(bucket_name) in ("Enabled", "Suspended"):
diff --git a/ministack/services/sns.py b/ministack/services/sns.py
index 031834f0..b4ec96e2 100644
--- a/ministack/services/sns.py
+++ b/ministack/services/sns.py
@@ -531,15 +531,15 @@ def _add_permission(params):
topic_arn = topic["arn"]
label = _p(params, "Label")
if not label:
- return _error("InvalidParameterException",
+ return _error("InvalidParameter",
"Label is required", 400)
account_ids = _member_list(params, "AWSAccountId")
actions = _member_list(params, "ActionName")
if not account_ids:
- return _error("InvalidParameterException",
+ return _error("InvalidParameter",
"AWSAccountId is required", 400)
if not actions:
- return _error("InvalidParameterException",
+ return _error("InvalidParameter",
"ActionName is required", 400)
raw = topic["attributes"].get("Policy") or ""
@@ -555,7 +555,7 @@ def _add_permission(params):
policy["Statement"] = statements
if any(s.get("Sid") == label for s in statements):
- return _error("InvalidParameterException",
+ return _error("InvalidParameter",
f"Value {label} for parameter Label is invalid. "
f"Reason: Already exists.", 400)
@@ -576,7 +576,7 @@ def _remove_permission(params):
return err
label = _p(params, "Label")
if not label:
- return _error("InvalidParameterException",
+ return _error("InvalidParameter",
"Label is required", 400)
raw = topic["attributes"].get("Policy") or ""
diff --git a/ministack/services/sqs.py b/ministack/services/sqs.py
index 653eeb8c..3acfc3c2 100644
--- a/ministack/services/sqs.py
+++ b/ministack/services/sqs.py
@@ -146,8 +146,7 @@ def _enforce_queue_policy(q: dict, iam_action: str) -> None:
if not resource_policy_allows(q["attributes"].get("Policy") or "", ctx,
owner is None or owner == caller):
raise _Err("AccessDenied",
- f"User: {ctx.principal_arn} is not authorized to perform: "
- f"{iam_action} on resource: {queue_arn}", 403)
+ f"Access to the resource https://sqs.{get_region()}.amazonaws.com/ is denied.", 403)
def queue_policy_allows(queue_arn: str, service: str,
diff --git a/ministack/services/sts.py b/ministack/services/sts.py
index 613368ce..7d091320 100644
--- a/ministack/services/sts.py
+++ b/ministack/services/sts.py
@@ -92,7 +92,8 @@ def _caller_identity(
def _credential_error_response(error: CredentialResolutionError):
- code = "ExpiredToken" if error.code == "ExpiredTokenException" else error.code
+ code = {"ExpiredTokenException": "ExpiredToken",
+ "UnrecognizedClientException": "InvalidClientTokenId"}.get(error.code, error.code)
return _error(403, code, error.message, ns="sts")
diff --git a/tests/test_bedrock_agentcore.py b/tests/test_bedrock_agentcore.py
index 6f3e84fd..6c0c3eb1 100644
--- a/tests/test_bedrock_agentcore.py
+++ b/tests/test_bedrock_agentcore.py
@@ -3,10 +3,15 @@
Covers the v1 surface: agent-runtime CRUD, runtime-endpoint CRUD,
InvokeAgentRuntime (deterministic echo), region isolation, and validation.
"""
+import asyncio
import json
import os
+import sys
+import threading
+import types
import urllib.request
import uuid as _uuid_mod
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import boto3
import pytest
@@ -14,6 +19,9 @@ from botocore.config import Config
from botocore.exceptions import ClientError
from botocore.utils import parse_timestamp
+from ministack.core.responses import StreamingResponse
+from ministack.services import bedrock_agentcore as agentcore
+
ENDPOINT = os.environ.get("MINISTACK_ENDPOINT", "http://localhost:4566")
_ARTIFACT = {"containerConfiguration": {"containerUri": "0.dkr.ecr.us-east-1.amazonaws.com/agent:latest"}}
@@ -32,9 +40,14 @@ def _client(service, region="us-east-1"):
)
-def _create(ctl, name):
+_CODE_ARTIFACT = {"codeConfiguration": {
+ "code": {"s3": {"bucket": "agent-code", "prefix": "agent.zip"}},
+ "runtime": "PYTHON_3_12", "entryPoint": ["main.py"]}}
+
+
+def _create(ctl, name, artifact=_ARTIFACT):
return ctl.create_agent_runtime(
- agentRuntimeName=name, agentRuntimeArtifact=_ARTIFACT,
+ agentRuntimeName=name, agentRuntimeArtifact=artifact,
roleArn=_ROLE, networkConfiguration=_NET,
)
@@ -111,7 +124,8 @@ def test_agentcore_endpoint_lifecycle():
def test_agentcore_invoke_returns_deterministic_echo():
ctl = _client("bedrock-agentcore-control")
rt = _client("bedrock-agentcore")
- rid_resp = _create(ctl, f"rt_{_uuid_mod.uuid4().hex[:8]}")
+ # A code artifact never starts a container, so this echoes with or without Docker.
+ rid_resp = _create(ctl, f"rt_{_uuid_mod.uuid4().hex[:8]}", _CODE_ARTIFACT)
arn = rid_resp["agentRuntimeArn"]
try:
resp = rt.invoke_agent_runtime(
@@ -217,3 +231,330 @@ def test_agentcore_timestamps_are_iso8601_strings_on_the_wire():
where="ListAgentRuntimeEndpoints.")
finally:
ctl.delete_agent_runtime(agentRuntimeId=rid)
+
+
+# --- InvokeAgentRuntime against the runtime container (in-process) ---
+
+def _runtime(name="container_test", image="example.local/worker:1"):
+ status, _, body = agentcore._create_agent_runtime(json.dumps({
+ "agentRuntimeName": name,
+ "agentRuntimeArtifact": {"containerConfiguration": {"containerUri": image}},
+ "roleArn": "arn:aws:iam::000000000000:role/agentcore",
+ "networkConfiguration": {"networkMode": "PUBLIC"},
+ "environmentVariables": {"WORKER_MODE": "test"},
+ }).encode())
+ assert status == 200
+ return json.loads(body)
+
+
+def _invoke(arn, headers=None, body=b"{}"):
+ return asyncio.run(agentcore.handle_request(
+ "POST", f"/runtimes/{arn}/invocations",
+ headers or {"content-type": "application/json"}, body, {},
+ ))
+
+
+def _fake_docker(monkeypatch, port, *, fail=False, network=None,
+ fail_on_start=False, fail_after_create=False, missing_image=False):
+ started = []
+ removed = []
+ created = []
+ pulled = []
+
+ class ImageNotFound(Exception):
+ pass
+
+ class Container:
+ status = "running"
+ attrs = {"NetworkSettings": {
+ "Ports": {"8080/tcp": [{"HostPort": str(port)}]},
+ "Networks": {network: {"IPAddress": "127.0.0.1"}} if network else {},
+ }}
+
+ def reload(self):
+ pass
+
+ def start(self):
+ if fail_on_start:
+ raise TimeoutError("Docker start timed out")
+
+ def remove(self, force=False):
+ removed.append(force)
+
+ class Containers:
+ def get(self, _name):
+ if not network:
+ raise RuntimeError("MiniStack is not in Docker")
+ return Container()
+
+ def create(self, image, **kwargs):
+ if fail:
+ raise RuntimeError("image unavailable")
+ if missing_image and not pulled:
+ raise ImageNotFound(image)
+ started.append((image, kwargs))
+ container = Container()
+ created.append(container)
+ if fail_after_create:
+ raise TimeoutError("Docker create timed out after allocation")
+ return container
+
+ def list(self, **_kwargs):
+ return created
+
+ fake_docker = types.SimpleNamespace(
+ errors=types.SimpleNamespace(ImageNotFound=ImageNotFound),
+ from_env=lambda **_kwargs: types.SimpleNamespace(
+ containers=Containers(),
+ images=types.SimpleNamespace(pull=lambda image: pulled.append(image)),
+ ping=lambda: True,
+ ),
+ )
+ monkeypatch.setitem(sys.modules, "docker", fake_docker)
+ from ministack.services import bedrock_agentcore
+ monkeypatch.setattr(bedrock_agentcore, "_docker", None)
+ return started, removed
+
+
+def _worker(response_status=200):
+ requests = []
+
+ class Worker(BaseHTTPRequestHandler):
+ def do_GET(self):
+ self.send_response(200 if self.path == "/ping" else 404)
+ self.end_headers()
+
+ def do_POST(self):
+ requests.append((self.path, dict(self.headers),
+ self.rfile.read(int(self.headers["Content-Length"]))))
+ payload = b'{"evidence":[{"id":"deployment-1"}]}'
+ self.send_response(response_status)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(payload)))
+ self.end_headers()
+ self.wfile.write(payload)
+
+ def log_message(self, *_args):
+ pass
+
+ server = ThreadingHTTPServer(("127.0.0.1", 0), Worker)
+ thread = threading.Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ return server, thread, requests
+
+
+async def _discard(_message):
+ pass
+
+
+def test_container_image_invoked_and_removed(monkeypatch):
+ server, thread, requests = _worker()
+ started, removed = _fake_docker(monkeypatch, server.server_port)
+ runtime = _runtime()
+ try:
+ headers = {"content-type": "application/json", "authorization": "secret",
+ "x-amzn-bedrock-agentcore-runtime-session-id": "session-1"}
+ status, response_headers, body = _invoke(runtime["agentRuntimeArn"], headers,
+ b'{"prompt":"why?"}')
+ assert status == 200
+ assert isinstance(body, StreamingResponse)
+ messages = []
+
+ async def send(message):
+ messages.append(message)
+
+ asyncio.run(body.runner(send, None))
+ assert json.loads(b"".join(m["body"] for m in messages)) == {
+ "evidence": [{"id": "deployment-1"}]}
+ assert response_headers["x-amzn-bedrock-agentcore-runtime-session-id"] == "session-1"
+ assert started[0][0] == "example.local/worker:1"
+ assert started[0][1]["ports"] == {"8080/tcp": ("127.0.0.1", None)}
+ assert started[0][1]["environment"] == {"WORKER_MODE": "test"}
+ from ministack.core.container_reaper import INSTANCE_LABEL
+ assert started[0][1]["labels"]["ministack"] == "agentcore"
+ assert INSTANCE_LABEL in started[0][1]["labels"]
+ path, forwarded, payload = requests[0]
+ assert path == "/invocations" and payload == b'{"prompt":"why?"}'
+ assert "authorization" not in {key.lower() for key in forwarded}
+ _, _, second_body = _invoke(runtime["agentRuntimeArn"])
+ asyncio.run(second_body.runner(_discard, None))
+ assert len(started) == 1
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
+ server.shutdown()
+ server.server_close()
+ thread.join()
+ assert removed == [True]
+
+
+def test_container_joins_ministack_network(monkeypatch):
+ started, _ = _fake_docker(monkeypatch, 8080, network="ministack-net")
+ class Healthy:
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *_args):
+ pass
+
+ monkeypatch.setattr(agentcore, "_local_open", lambda *_args, **_kwargs: Healthy())
+ runtime = _runtime("network_test")
+ try:
+ url = agentcore._container_invocations_url(agentcore._runtimes[runtime["agentRuntimeId"]])
+ assert url == "http://127.0.0.1:8080/invocations"
+ assert started[0][1]["network"] == "ministack-net"
+ assert "ports" not in started[0][1]
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
+
+
+def test_worker_error_is_runtime_client_error(monkeypatch):
+ server, thread, _ = _worker(400)
+ _fake_docker(monkeypatch, server.server_port)
+ runtime = _runtime("worker_error")
+ try:
+ status, headers, body = _invoke(runtime["agentRuntimeArn"])
+ assert status == 424
+ assert headers["x-amzn-errortype"] == "RuntimeClientError"
+ assert json.loads(body)["__type"] == "RuntimeClientError"
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
+ server.shutdown()
+ server.server_close()
+ thread.join()
+
+
+def test_container_start_failure_is_not_echo(monkeypatch):
+ _fake_docker(monkeypatch, 1, fail=True)
+ runtime = _runtime("image_failure")
+ try:
+ status, headers, body = _invoke(runtime["agentRuntimeArn"])
+ assert status == 424
+ assert headers["x-amzn-errortype"] == "RuntimeClientError"
+ assert b"image unavailable" in body
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
+
+
+def test_container_is_removed_if_start_times_out(monkeypatch):
+ _, removed = _fake_docker(monkeypatch, 1, fail_on_start=True)
+ runtime = _runtime("start_timeout")
+ try:
+ status, _, _ = _invoke(runtime["agentRuntimeArn"])
+ assert status == 424
+ assert removed == [True]
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
+
+
+def test_missing_image_is_pulled_then_started(monkeypatch):
+ server, thread, _ = _worker()
+ started, _ = _fake_docker(monkeypatch, server.server_port, missing_image=True)
+ runtime = _runtime("pull_image")
+ try:
+ status, _, response = _invoke(runtime["agentRuntimeArn"])
+ assert status == 200
+ asyncio.run(response.runner(_discard, None))
+ assert [image for image, _ in started] == ["example.local/worker:1"]
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
+ server.shutdown()
+ server.server_close()
+ thread.join()
+
+
+def test_orphan_is_removed_if_create_times_out(monkeypatch):
+ _, removed = _fake_docker(monkeypatch, 1, fail_after_create=True)
+ runtime = _runtime("create_timeout")
+ try:
+ status, _, _ = _invoke(runtime["agentRuntimeArn"])
+ assert status == 424
+ assert removed == [True]
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
+
+
+def test_connection_reset_maps_to_runtime_client_error(monkeypatch):
+ server, thread, _ = _worker()
+ _fake_docker(monkeypatch, server.server_port)
+ runtime = _runtime("connection_reset")
+ try:
+ agentcore._container_invocations_url(agentcore._runtimes[runtime["agentRuntimeId"]])
+ monkeypatch.setattr(agentcore, "_local_open", lambda *_args, **_kwargs:
+ (_ for _ in ()).throw(ConnectionResetError("peer reset")))
+ status, headers, _ = _invoke(runtime["agentRuntimeArn"])
+ assert status == 424
+ assert headers["x-amzn-errortype"] == "RuntimeClientError"
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
+ server.shutdown()
+ server.server_close()
+ thread.join()
+
+
+def test_stream_failure_leaves_response_truncated(monkeypatch):
+ class BrokenResponse:
+ status = 200
+ headers = {"Content-Type": "text/event-stream"}
+ closed = False
+ reads = 0
+
+ def read1(self, _size):
+ self.reads += 1
+ if self.reads == 1:
+ return b"first\n"
+ raise ConnectionResetError("stream interrupted")
+
+ def close(self):
+ self.closed = True
+
+ response = BrokenResponse()
+ monkeypatch.setattr(agentcore, "_local_open", lambda *_args, **_kwargs: response)
+ status, _, stream = agentcore._invoke_container(
+ "http://127.0.0.1:8080/invocations", b"{}", {}, "application/json", "session-1")
+ assert status == 200
+ frames = []
+
+ async def send(frame):
+ frames.append(frame)
+
+ asyncio.run(stream.runner(send, None))
+ assert frames == [{"type": "http.response.body", "body": b"first\n", "more_body": True}]
+ assert response.closed
+
+
+def test_update_restarts_image(monkeypatch):
+ server, thread, _ = _worker()
+ started, removed = _fake_docker(monkeypatch, server.server_port)
+ runtime = _runtime("update_image")
+ try:
+ status, _, response = _invoke(runtime["agentRuntimeArn"])
+ assert status == 200
+ asyncio.run(response.runner(_discard, None))
+ agentcore._update_agent_runtime(runtime["agentRuntimeId"], json.dumps({
+ "agentRuntimeArtifact": {"containerConfiguration": {"containerUri": "example.local/worker:2"}},
+ "roleArn": "arn:aws:iam::000000000000:role/agentcore",
+ "networkConfiguration": {"networkMode": "PUBLIC"},
+ }).encode())
+ assert removed == [True]
+ current = agentcore._runtimes[runtime["agentRuntimeId"]]
+ status, _, response = _invoke(current["agentRuntimeArn"])
+ assert status == 200
+ assert [image for image, _ in started] == ["example.local/worker:1", "example.local/worker:2"]
+ asyncio.run(response.runner(_discard, None))
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
+ server.shutdown()
+ server.server_close()
+ thread.join()
+
+
+def test_without_docker_the_invocation_echoes(monkeypatch):
+ monkeypatch.setitem(sys.modules, "docker", None)
+ monkeypatch.setattr(agentcore, "_docker", None)
+ runtime = _runtime(name="no_docker_echo")
+ try:
+ status, _, body = _invoke(runtime["agentRuntimeArn"], body=b'{"prompt":"hi"}')
+ assert status == 200
+ assert json.loads(body)["input"] == {"prompt": "hi"}
+ finally:
+ agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
diff --git a/tests/test_bedrock_agentcore_container.py b/tests/test_bedrock_agentcore_container.py
deleted file mode 100644
index 5cba2cc4..00000000
--- a/tests/test_bedrock_agentcore_container.py
+++ /dev/null
@@ -1,319 +0,0 @@
-"""AgentCore container invocation contract, without a live AWS account."""
-
-import asyncio
-import json
-import sys
-import threading
-import types
-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
-
-from ministack.core.responses import StreamingResponse
-from ministack.services import bedrock_agentcore as agentcore
-
-
-def _runtime(name="container_test", image="example.local/worker:1"):
- status, _, body = agentcore._create_agent_runtime(json.dumps({
- "agentRuntimeName": name,
- "agentRuntimeArtifact": {"containerConfiguration": {"containerUri": image}},
- "roleArn": "arn:aws:iam::000000000000:role/agentcore",
- "networkConfiguration": {"networkMode": "PUBLIC"},
- "environmentVariables": {"WORKER_MODE": "test"},
- }).encode())
- assert status == 200
- return json.loads(body)
-
-
-def _invoke(arn, headers=None, body=b"{}"):
- return asyncio.run(agentcore.handle_request(
- "POST", f"/runtimes/{arn}/invocations",
- headers or {"content-type": "application/json"}, body, {},
- ))
-
-
-def _fake_docker(monkeypatch, port, *, fail=False, network=None,
- fail_on_start=False, fail_after_create=False, missing_image=False):
- started = []
- removed = []
- created = []
- pulled = []
-
- class ImageNotFound(Exception):
- pass
-
- class Container:
- status = "running"
- attrs = {"NetworkSettings": {
- "Ports": {"8080/tcp": [{"HostPort": str(port)}]},
- "Networks": {network: {"IPAddress": "127.0.0.1"}} if network else {},
- }}
-
- def reload(self):
- pass
-
- def start(self):
- if fail_on_start:
- raise TimeoutError("Docker start timed out")
-
- def remove(self, force=False):
- removed.append(force)
-
- class Containers:
- def get(self, _name):
- if not network:
- raise RuntimeError("MiniStack is not in Docker")
- return Container()
-
- def create(self, image, **kwargs):
- if fail:
- raise RuntimeError("image unavailable")
- if missing_image and not pulled:
- raise ImageNotFound(image)
- started.append((image, kwargs))
- container = Container()
- created.append(container)
- if fail_after_create:
- raise TimeoutError("Docker create timed out after allocation")
- return container
-
- def list(self, **_kwargs):
- return created
-
- fake_docker = types.SimpleNamespace(
- errors=types.SimpleNamespace(ImageNotFound=ImageNotFound),
- from_env=lambda **_kwargs: types.SimpleNamespace(
- containers=Containers(),
- images=types.SimpleNamespace(pull=lambda image: pulled.append(image)),
- ),
- )
- monkeypatch.setitem(sys.modules, "docker", fake_docker)
- monkeypatch.setenv("MINISTACK_AGENTCORE_DOCKER", "1")
- return started, removed
-
-
-def _worker(response_status=200):
- requests = []
-
- class Worker(BaseHTTPRequestHandler):
- def do_GET(self):
- self.send_response(200 if self.path == "/ping" else 404)
- self.end_headers()
-
- def do_POST(self):
- requests.append((self.path, dict(self.headers),
- self.rfile.read(int(self.headers["Content-Length"]))))
- payload = b'{"evidence":[{"id":"deployment-1"}]}'
- self.send_response(response_status)
- self.send_header("Content-Type", "application/json")
- self.send_header("Content-Length", str(len(payload)))
- self.end_headers()
- self.wfile.write(payload)
-
- def log_message(self, *_args):
- pass
-
- server = ThreadingHTTPServer(("127.0.0.1", 0), Worker)
- thread = threading.Thread(target=server.serve_forever, daemon=True)
- thread.start()
- return server, thread, requests
-
-
-async def _discard(_message):
- pass
-
-
-def test_container_image_invoked_and_removed(monkeypatch):
- server, thread, requests = _worker()
- started, removed = _fake_docker(monkeypatch, server.server_port)
- runtime = _runtime()
- try:
- headers = {"content-type": "application/json", "authorization": "secret",
- "x-amzn-bedrock-agentcore-runtime-session-id": "session-1"}
- status, response_headers, body = _invoke(runtime["agentRuntimeArn"], headers,
- b'{"prompt":"why?"}')
- assert status == 200
- assert isinstance(body, StreamingResponse)
- messages = []
-
- async def send(message):
- messages.append(message)
-
- asyncio.run(body.runner(send, None))
- assert json.loads(b"".join(m["body"] for m in messages)) == {
- "evidence": [{"id": "deployment-1"}]}
- assert response_headers["x-amzn-bedrock-agentcore-runtime-session-id"] == "session-1"
- assert started[0][0] == "example.local/worker:1"
- assert started[0][1]["ports"] == {"8080/tcp": ("127.0.0.1", None)}
- assert started[0][1]["environment"] == {"WORKER_MODE": "test"}
- path, forwarded, payload = requests[0]
- assert path == "/invocations" and payload == b'{"prompt":"why?"}'
- assert "authorization" not in {key.lower() for key in forwarded}
- _, _, second_body = _invoke(runtime["agentRuntimeArn"])
- asyncio.run(second_body.runner(_discard, None))
- assert len(started) == 1
- finally:
- agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
- server.shutdown()
- server.server_close()
- thread.join()
- assert removed == [True]
-
-
-def test_container_joins_ministack_network(monkeypatch):
- started, _ = _fake_docker(monkeypatch, 8080, network="ministack-net")
- class Healthy:
- def __enter__(self):
- return self
-
- def __exit__(self, *_args):
- pass
-
- monkeypatch.setattr(agentcore, "_local_open", lambda *_args, **_kwargs: Healthy())
- runtime = _runtime("network_test")
- try:
- url = agentcore._container_invocations_url(agentcore._runtimes[runtime["agentRuntimeId"]])
- assert url == "http://127.0.0.1:8080/invocations"
- assert started[0][1]["network"] == "ministack-net"
- assert "ports" not in started[0][1]
- finally:
- agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
-
-
-def test_worker_error_is_runtime_client_error(monkeypatch):
- server, thread, _ = _worker(400)
- _fake_docker(monkeypatch, server.server_port)
- runtime = _runtime("worker_error")
- try:
- status, headers, body = _invoke(runtime["agentRuntimeArn"])
- assert status == 424
- assert headers["x-amzn-errortype"] == "RuntimeClientError"
- assert json.loads(body)["__type"] == "RuntimeClientError"
- finally:
- agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
- server.shutdown()
- server.server_close()
- thread.join()
-
-
-def test_container_start_failure_is_not_echo(monkeypatch):
- _fake_docker(monkeypatch, 1, fail=True)
- runtime = _runtime("image_failure")
- try:
- status, headers, body = _invoke(runtime["agentRuntimeArn"])
- assert status == 424
- assert headers["x-amzn-errortype"] == "RuntimeClientError"
- assert b"image unavailable" in body
- finally:
- agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
-
-
-def test_container_is_removed_if_start_times_out(monkeypatch):
- _, removed = _fake_docker(monkeypatch, 1, fail_on_start=True)
- runtime = _runtime("start_timeout")
- try:
- status, _, _ = _invoke(runtime["agentRuntimeArn"])
- assert status == 424
- assert removed == [True]
- finally:
- agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
-
-
-def test_missing_image_is_pulled_then_started(monkeypatch):
- server, thread, _ = _worker()
- started, _ = _fake_docker(monkeypatch, server.server_port, missing_image=True)
- runtime = _runtime("pull_image")
- try:
- status, _, response = _invoke(runtime["agentRuntimeArn"])
- assert status == 200
- asyncio.run(response.runner(_discard, None))
- assert [image for image, _ in started] == ["example.local/worker:1"]
- finally:
- agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
- server.shutdown()
- server.server_close()
- thread.join()
-
-
-def test_orphan_is_removed_if_create_times_out(monkeypatch):
- _, removed = _fake_docker(monkeypatch, 1, fail_after_create=True)
- runtime = _runtime("create_timeout")
- try:
- status, _, _ = _invoke(runtime["agentRuntimeArn"])
- assert status == 424
- assert removed == [True]
- finally:
- agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
-
-
-def test_connection_reset_maps_to_runtime_client_error(monkeypatch):
- server, thread, _ = _worker()
- _fake_docker(monkeypatch, server.server_port)
- runtime = _runtime("connection_reset")
- try:
- agentcore._container_invocations_url(agentcore._runtimes[runtime["agentRuntimeId"]])
- monkeypatch.setattr(agentcore, "_local_open", lambda *_args, **_kwargs:
- (_ for _ in ()).throw(ConnectionResetError("peer reset")))
- status, headers, _ = _invoke(runtime["agentRuntimeArn"])
- assert status == 424
- assert headers["x-amzn-errortype"] == "RuntimeClientError"
- finally:
- agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
- server.shutdown()
- server.server_close()
- thread.join()
-
-
-def test_stream_failure_leaves_response_truncated(monkeypatch):
- class BrokenResponse:
- status = 200
- headers = {"Content-Type": "text/event-stream"}
- closed = False
- reads = 0
-
- def read1(self, _size):
- self.reads += 1
- if self.reads == 1:
- return b"first\n"
- raise ConnectionResetError("stream interrupted")
-
- def close(self):
- self.closed = True
-
- response = BrokenResponse()
- monkeypatch.setattr(agentcore, "_local_open", lambda *_args, **_kwargs: response)
- status, _, stream = agentcore._invoke_container(
- "http://127.0.0.1:8080/invocations", b"{}", {}, "application/json", "session-1")
- assert status == 200
- frames = []
-
- async def send(frame):
- frames.append(frame)
-
- asyncio.run(stream.runner(send, None))
- assert frames == [{"type": "http.response.body", "body": b"first\n", "more_body": True}]
- assert response.closed
-
-
-def test_update_restarts_image(monkeypatch):
- server, thread, _ = _worker()
- started, removed = _fake_docker(monkeypatch, server.server_port)
- runtime = _runtime("update_image")
- try:
- status, _, response = _invoke(runtime["agentRuntimeArn"])
- assert status == 200
- asyncio.run(response.runner(_discard, None))
- agentcore._update_agent_runtime(runtime["agentRuntimeId"], json.dumps({
- "agentRuntimeArtifact": {"containerConfiguration": {"containerUri": "example.local/worker:2"}},
- "roleArn": "arn:aws:iam::000000000000:role/agentcore",
- "networkConfiguration": {"networkMode": "PUBLIC"},
- }).encode())
- assert removed == [True]
- current = agentcore._runtimes[runtime["agentRuntimeId"]]
- status, _, response = _invoke(current["agentRuntimeArn"])
- assert status == 200
- assert [image for image, _ in started] == ["example.local/worker:1", "example.local/worker:2"]
- asyncio.run(response.runner(_discard, None))
- finally:
- agentcore._delete_agent_runtime(runtime["agentRuntimeId"])
- server.shutdown()
- server.server_close()
- thread.join()
diff --git a/tests/test_lambda.py b/tests/test_lambda.py
index 9621da90..aa41337d 100644
--- a/tests/test_lambda.py
+++ b/tests/test_lambda.py
@@ -13708,8 +13708,8 @@ def test_cognito_issuer_host_resolves_to_the_gateway_under_use_ssl(monkeypatch,
run_kwargs, container_env, mounts = _wire(
monkeypatch, USE_SSL="1", TMPDIR=str(tmp_path), MINISTACK_REGION="us-east-1")
- assert run_kwargs["extra_hosts"] == {
- "cognito-idp.us-east-1.amazonaws.com": "host-gateway"}
+ assert run_kwargs["extra_hosts"]["cognito-idp.us-east-1.amazonaws.com"] == "host-gateway"
+ assert run_kwargs["extra_hosts"]["cognito-idp.eu-west-2.amazonaws.com"] == "host-gateway"
# NODE_EXTRA_CA_CERTS adds to node's roots, so it takes the bare certificate.
assert container_env["NODE_EXTRA_CA_CERTS"] == lsvc._CONTAINER_CA_PATH
# The other two replace the trust store, so they take the public roots with
diff --git a/tests/test_messaging_policies.py b/tests/test_messaging_policies.py
deleted file mode 100644
index 6c5efbae..00000000
--- a/tests/test_messaging_policies.py
+++ /dev/null
@@ -1,654 +0,0 @@
-"""SQS queue-policy and SNS topic-policy enforcement.
-
-The Policy attributes were stored (CRUD) but never evaluated. Under AUTH=true
-they are now enforced:
-
-- Same-account API calls pass unless the policy carries an explicit Deny.
-- Cross-account API calls need an explicit Allow (AccessDenied /
- AuthorizationError otherwise).
-- S3→SQS, S3→SNS and SNS→SQS deliveries need an explicit Allow for the
- calling service in the destination policy (with its SourceArn/SourceAccount
- conditions); without one the delivery is dropped, like real AWS.
-- S3 refuses a notification config whose SQS/SNS destination denies it
- (InvalidArgument), unless SkipDestinationValidation skips the probe.
-
-With AUTH=false every path stays permissive, like the rest of the identity
-layer. The shared CI server runs with auth disabled, so these tests drive the
-service modules in-process with AUTH monkeypatched (the test_cfn capabilities
-pattern) instead of going over HTTP.
-"""
-
-import json
-import re
-import uuid
-
-import pytest
-from conftest import (
- REGION,
- sqs_policy_allow_s3,
- sqs_policy_allow_sns,
-)
-
-from ministack.core.iam_evaluator import (
- EvalContext,
- evaluate_resource_policy,
-)
-from ministack.core.responses import request_scope
-from ministack.services import s3 as s3_svc
-from ministack.services import sns as sns_svc
-from ministack.services import sqs as sqs_svc
-
-ACCT_A = "111111111111"
-ACCT_B = "222222222222"
-
-
-def _reset_messaging_state():
- from ministack.core.iam_evaluator import _request_caller_arn
-
- sqs_svc.reset()
- sns_svc.reset()
- s3_svc.reset()
- _request_caller_arn.set("")
-
-
-def auth(monkeypatch):
- """AUTH=true with clean in-process SQS/SNS/S3 state."""
- import ministack.app as app_mod
-
- monkeypatch.setattr(app_mod, "AUTH", True)
- _reset_messaging_state()
- yield
- _reset_messaging_state()
-
-
-def noauth(monkeypatch):
- """AUTH=false with clean in-process SQS/SNS/S3 state."""
- import ministack.app as app_mod
-
- monkeypatch.setattr(app_mod, "AUTH", False)
- _reset_messaging_state()
- yield
- _reset_messaging_state()
-
-
-def _uniq(prefix):
- return f"{prefix}-{uuid.uuid4().hex[:8]}"
-
-
-def _as(account, fn, *args, **kwargs):
- """Call an in-process service function as *account* in the test region."""
- with request_scope(account, REGION):
- return fn(*args, **kwargs)
-
-
-def _resp_detail(resp):
- """``(status, code, message)`` of an in-process SNS/S3 response tuple."""
- status, _headers, body = resp
- text = body.decode("utf-8", errors="replace") if isinstance(body, bytes) else str(body)
- code = re.search(r"<Code>([^<]*)</Code>", text)
- msg = re.search(r"<Message>([^<]*)</Message>", text)
- return (status, code.group(1) if code else "",
- msg.group(1) if msg else "")
-
-
-def _resp_code(resp):
- """``(status, code)`` of an in-process SNS/S3 response tuple."""
- status, code, _ = _resp_detail(resp)
- return status, code
-
-
-def _qarn(account, name):
- return f"arn:aws:sqs:{REGION}:{account}:{name}"
-
-
-def _tarn(account, name):
- return f"arn:aws:sns:{REGION}:{account}:{name}"
-
-
-def _make_queue(account, name, policy=None):
- url = _as(account, sqs_svc._act_create_queue,
- {"QueueName": name}, "")["QueueUrl"]
- if policy is not None:
- _as(account, sqs_svc._act_set_queue_attributes,
- {"QueueUrl": url,
- "Attributes": {"Policy": json.dumps(policy)}}, url)
- return url
-
-
-def _queue_bodies(arn):
- q = sqs_svc._queue_by_arn(arn)
- assert q is not None
- return [m["body"] for m in q["messages"]]
-
-
-def _deny_all(arn, action):
- return {"Version": "2012-10-17", "Statement": [{
- "Sid": "deny", "Effect": "Deny", "Principal": {"AWS": "*"},
- "Action": action, "Resource": arn}]}
-
-
-def _allow_account(arn, account, action):
- return {"Version": "2012-10-17", "Statement": [{
- "Sid": "x", "Effect": "Allow", "Principal": {"AWS": account},
- "Action": action, "Resource": arn}]}
-
-
-def _make_topic(account, name):
- assert _resp_code(_as(account, sns_svc._create_topic,
- {"Name": name})) == (200, "")
- return _tarn(account, name)
-
-
-def _set_topic_policy(account, arn, policy):
- assert _resp_code(_as(account, sns_svc._set_topic_attributes, {
- "TopicArn": arn, "AttributeName": "Policy",
- "AttributeValue": json.dumps(policy)})) == (200, "")
-
-
-def _topic_policy(arn):
- topic = sns_svc._topic_by_arn_any_scope(arn)
- assert topic is not None
- return json.loads(topic["attributes"]["Policy"])
-
-
-def _make_bucket(account, name):
- assert _resp_code(_as(account, s3_svc._create_bucket,
- name, b"", {})) == (200, "")
-
-
-def _notif_xml(kind, arn):
- tag = "Queue" if kind == "sqs" else "Topic"
- return (
- "<NotificationConfiguration>"
- f"<{tag}Configuration><{tag}>{arn}</{tag}>"
- "<Event>s3:ObjectCreated:*</Event>"
- f"</{tag}Configuration>"
- "</NotificationConfiguration>"
- ).encode()
-
-
-def _put_notif(account, bucket, kind, arn, skip=False):
- headers = {"x-amz-skip-destination-validation": "true"} if skip else {}
- return _as(account, s3_svc._put_bucket_notification,
- bucket, _notif_xml(kind, arn), headers)
-
-
-# ---------------------------------------------------------------------------
-# Unit: evaluate_resource_policy Service principals
-# ---------------------------------------------------------------------------
-
-def _ctx(action="sqs:SendMessage", resource="arn:aws:sqs:us-east-1:1:q",
- source_arn="", source_account="", principal="arn:aws:iam::1:root"):
- return EvalContext(
- principal_arn=principal,
- principal_type="Service",
- principal_account="1",
- action=action,
- resource_arn=resource,
- region="us-east-1",
- service_context={"aws:sourcearn": source_arn,
- "aws:sourceaccount": source_account,
- "aws:sourceowner": source_account},
- )
-
-
-def _svc_policy():
- return {
- "Version": "2012-10-17",
- "Statement": [{
- "Effect": "Allow",
- "Principal": {"Service": "s3.amazonaws.com"},
- "Action": "sqs:SendMessage",
- "Resource": "arn:aws:sqs:us-east-1:1:q",
- "Condition": {"ArnLike": {"aws:SourceArn": "arn:aws:s3:*:*:bkt"}},
- }],
- }
-
-
-class TestEvaluateResourcePolicyService:
- def test_service_principal_allows_matching_service(self):
- assert evaluate_resource_policy(
- _svc_policy(), _ctx(source_arn="arn:aws:s3:::bkt"),
- service="s3.amazonaws.com").decision == "Allow"
-
- def test_service_principal_ignores_other_service(self):
- assert evaluate_resource_policy(
- _svc_policy(), _ctx(source_arn="arn:aws:s3:::bkt"),
- service="sns.amazonaws.com").decision == "ImplicitDeny"
-
- def test_service_principal_ignored_without_service_arg(self):
- # Legacy callers (Lambda layer policies, S3 bucket policies) keep the
- # old behaviour: Service entries never match.
- assert evaluate_resource_policy(
- _svc_policy(), _ctx(source_arn="arn:aws:s3:::bkt")).decision == "ImplicitDeny"
-
- def test_wrong_source_arn_denies(self):
- assert evaluate_resource_policy(
- _svc_policy(), _ctx(source_arn="arn:aws:s3:::other"),
- service="s3.amazonaws.com").decision == "ImplicitDeny"
-
- def test_aws_principal_still_matches(self):
- doc = {"Statement": [{
- "Effect": "Allow", "Principal": {"AWS": "222222222222"},
- "Action": "sqs:SendMessage", "Resource": "*"}]}
- ctx = _ctx(principal="arn:aws:iam::222222222222:root")
- assert evaluate_resource_policy(doc, ctx).decision == "Allow"
-
- def test_star_principal_matches(self):
- doc = {"Statement": [{
- "Effect": "Allow", "Principal": "*",
- "Action": "sqs:SendMessage", "Resource": "*"}]}
- assert evaluate_resource_policy(doc, _ctx()).decision == "Allow"
-
-
-# ---------------------------------------------------------------------------
-# Cross-account SQS
-# ---------------------------------------------------------------------------
-
-class TestCrossAccountSqs:
- def test_send_denied_without_policy(self, auth):
- url = _make_queue(ACCT_A, _uniq("xpol-q"))
- with pytest.raises(sqs_svc._Err) as exc:
- _as(ACCT_B, sqs_svc._act_send_message,
- {"QueueUrl": url, "MessageBody": "no-policy"}, url)
- assert exc.value.code == "AccessDenied"
- assert exc.value.status == 403
-
- def test_send_allowed_with_policy(self, auth):
- name = _uniq("xpol-q")
- url = _make_queue(ACCT_A, name, _allow_account(
- _qarn(ACCT_A, name), ACCT_B, "sqs:SendMessage"))
- _as(ACCT_B, sqs_svc._act_send_message,
- {"QueueUrl": url, "MessageBody": "with-policy"}, url)
- assert _queue_bodies(_qarn(ACCT_A, name)) == ["with-policy"]
-
- def test_send_allowed_via_add_permission(self, auth):
- name = _uniq("xpol-q")
- url = _make_queue(ACCT_A, name)
- _as(ACCT_A, sqs_svc._act_add_permission,
- {"QueueUrl": url, "Label": "xacct",
- "AWSAccountIds": [ACCT_B], "Actions": ["SendMessage"]}, url)
- _as(ACCT_B, sqs_svc._act_send_message,
- {"QueueUrl": url, "MessageBody": "via-add-permission"}, url)
- assert _queue_bodies(_qarn(ACCT_A, name)) == ["via-add-permission"]
-
- def test_explicit_deny_blocks_same_account(self, auth):
- name = _uniq("xpol-q")
- url = _make_queue(ACCT_A, name, _deny_all(
- _qarn(ACCT_A, name), "sqs:SendMessage"))
- with pytest.raises(sqs_svc._Err) as exc:
- _as(ACCT_A, sqs_svc._act_send_message,
- {"QueueUrl": url, "MessageBody": "deny-me"}, url)
- assert exc.value.code == "AccessDenied"
-
- def test_get_queue_url_with_owner_account(self, auth):
- name = _uniq("xpol-q")
- url = _make_queue(ACCT_A, name, _allow_account(
- _qarn(ACCT_A, name), ACCT_B, "sqs:GetQueueUrl"))
- found = _as(ACCT_B, sqs_svc._act_get_queue_url,
- {"QueueName": name,
- "QueueOwnerAWSAccountId": ACCT_A}, "")["QueueUrl"]
- assert found == url
-
- def test_get_queue_url_cross_account_denied_without_policy(self, auth):
- name = _uniq("xpol-q")
- _make_queue(ACCT_A, name)
- with pytest.raises(sqs_svc._Err) as exc:
- _as(ACCT_B, sqs_svc._act_get_queue_url,
- {"QueueName": name,
- "QueueOwnerAWSAccountId": ACCT_A}, "")
- assert exc.value.code == "AccessDenied"
- assert exc.value.status == 403
-
- def test_get_queue_url_same_account_denied_by_explicit_deny(self, auth):
- name = _uniq("xpol-q")
- _make_queue(ACCT_A, name, _deny_all(
- _qarn(ACCT_A, name), "sqs:GetQueueUrl"))
- with pytest.raises(sqs_svc._Err) as exc:
- _as(ACCT_A, sqs_svc._act_get_queue_url,
- {"QueueName": name}, "")
- assert exc.value.code == "AccessDenied"
- assert exc.value.status == 403
-
- def test_get_queue_url_same_account_passes_without_policy(self, auth):
- name = _uniq("xpol-q")
- url = _make_queue(ACCT_A, name)
- assert _as(ACCT_A, sqs_svc._act_get_queue_url,
- {"QueueName": name}, "") == {"QueueUrl": url}
-
-
-# ---------------------------------------------------------------------------
-# Cross-account SNS (+ AddPermission/RemovePermission)
-# ---------------------------------------------------------------------------
-
-class TestCrossAccountSns:
- def test_publish_denied_by_default(self, auth):
- arn = _make_topic(ACCT_A, _uniq("xpol-t"))
- assert _resp_code(_as(ACCT_B, sns_svc._publish, {
- "TopicArn": arn, "Message": "no-grant"})) == (403, "AuthorizationError")
-
- def test_publish_allowed_with_policy(self, auth):
- name = _uniq("xpol-t")
- arn = _make_topic(ACCT_A, name)
- _set_topic_policy(ACCT_A, arn, _allow_account(
- arn, ACCT_B, "sns:Publish"))
- assert _resp_code(_as(ACCT_B, sns_svc._publish, {
- "TopicArn": arn, "Message": "granted"})) == (200, "")
- topic = sns_svc._topic_by_arn_any_scope(arn)
- assert [m["message"] for m in topic["messages"]] == ["granted"]
-
- def test_explicit_deny_blocks_same_account(self, auth):
- name = _uniq("xpol-t")
- arn = _make_topic(ACCT_A, name)
- _set_topic_policy(ACCT_A, arn, _deny_all(arn, "sns:Publish"))
- assert _resp_code(_as(ACCT_A, sns_svc._publish, {
- "TopicArn": arn, "Message": "deny-me"})) == (403, "AuthorizationError")
-
- def test_add_permission_grants_cross_account_publish(self, auth):
- arn = _make_topic(ACCT_A, _uniq("xpol-t"))
- assert _resp_code(_as(ACCT_A, sns_svc._add_permission, {
- "TopicArn": arn, "Label": "perm-1",
- "AWSAccountId.member.1": ACCT_B,
- "ActionName.member.1": "Publish"})) == (200, "")
- stmt = next(s for s in _topic_policy(arn)["Statement"]
- if s.get("Sid") == "perm-1")
- assert ACCT_B in stmt["Principal"]["AWS"]
- assert "sns:Publish" in stmt["Action"]
- assert _resp_code(_as(ACCT_B, sns_svc._publish, {
- "TopicArn": arn,
- "Message": "via-add-permission"})) == (200, "")
-
- def test_add_permission_rejects_duplicate_label(self, auth):
- arn = _make_topic(ACCT_A, _uniq("xpol-t"))
- params = {"TopicArn": arn, "Label": "dup",
- "AWSAccountId.member.1": ACCT_B,
- "ActionName.member.1": "Publish"}
- assert _resp_code(_as(ACCT_A, sns_svc._add_permission, params)) == (200, "")
- assert _resp_code(
- _as(ACCT_A, sns_svc._add_permission, params)) == (
- 400, "InvalidParameterException")
-
- def test_cross_account_tag_access_needs_grant(self, auth):
- name = _uniq("xpol-t")
- arn = _make_topic(ACCT_A, name)
- assert _resp_code(_as(ACCT_B, sns_svc._list_tags_for_resource, {
- "ResourceArn": arn})) == (403, "AuthorizationError")
- _set_topic_policy(ACCT_A, arn, _allow_account(
- arn, ACCT_B, ["sns:TagResource", "sns:ListTagsForResource"]))
- assert _resp_code(_as(ACCT_B, sns_svc._tag_resource, {
- "ResourceArn": arn, "Tags.member.1.Key": "k",
- "Tags.member.1.Value": "v"})) == (200, "")
- assert _resp_code(_as(ACCT_B, sns_svc._list_tags_for_resource, {
- "ResourceArn": arn})) == (200, "")
- assert sns_svc._topic_by_arn_any_scope(arn)["tags"] == {"k": "v"}
-
- def test_remove_permission_drops_statement(self, auth):
- arn = _make_topic(ACCT_A, _uniq("xpol-t"))
- _as(ACCT_A, sns_svc._add_permission, {
- "TopicArn": arn, "Label": "drop",
- "AWSAccountId.member.1": ACCT_B,
- "ActionName.member.1": "Publish"})
- assert _resp_code(_as(ACCT_B, sns_svc._publish, {
- "TopicArn": arn, "Message": "before"})) == (200, "")
- assert _resp_code(_as(ACCT_A, sns_svc._remove_permission, {
- "TopicArn": arn, "Label": "drop"})) == (200, "")
- assert all(s.get("Sid") != "drop"
- for s in _topic_policy(arn)["Statement"])
- assert _resp_code(_as(ACCT_B, sns_svc._publish, {
- "TopicArn": arn, "Message": "after"})) == (403, "AuthorizationError")
-
-
-# ---------------------------------------------------------------------------
-# S3 notifications gated on the destination policy
-# ---------------------------------------------------------------------------
-
-def _bucket_and_queue(tag):
- bucket, qname = _uniq(f"{tag}-bkt"), _uniq(f"{tag}-q")
- _make_bucket(ACCT_A, bucket)
- url = _make_queue(ACCT_A, qname)
- return bucket, url, _qarn(ACCT_A, qname)
-
-
-def _assert_put_rejected(bucket, kind, arn):
- status, code, msg = _resp_detail(_put_notif(ACCT_A, bucket, kind, arn))
- assert status == 400
- assert code == "InvalidArgument"
- assert "Unable to validate the following destination configurations" in msg
-
-
-def _bucket_topic_queue(tag):
- bucket = _uniq(f"{tag}-bkt")
- _make_bucket(ACCT_A, bucket)
- topic = _make_topic(ACCT_A, _uniq(f"{tag}-t"))
- qname = _uniq(f"{tag}-q")
- qurl = _make_queue(ACCT_A, qname,
- sqs_policy_allow_sns(_qarn(ACCT_A, qname), topic))
- assert _resp_code(_as(ACCT_A, sns_svc._subscribe, {
- "TopicArn": topic, "Protocol": "sqs",
- "Endpoint": _qarn(ACCT_A, qname)})) == (200, "")
- return bucket, topic, qurl, _qarn(ACCT_A, qname)
-
-
-class TestS3NotificationPolicies:
- def test_s3_put_rejected_without_queue_policy(self, auth):
- bucket, _url, arn = _bucket_and_queue("xpol")
- _assert_put_rejected(bucket, "sqs", arn)
-
- def test_s3_put_rejected_by_unrelated_policy(self, auth):
- bucket, url, arn = _bucket_and_queue("xpol")
- _as(ACCT_A, sqs_svc._act_set_queue_attributes,
- {"QueueUrl": url, "Attributes": {"Policy": json.dumps(
- _allow_account(arn, ACCT_B, "sqs:SendMessage"))}}, url)
- _assert_put_rejected(bucket, "sqs", arn)
-
- def test_s3_put_rejected_by_wrong_source_arn(self, auth):
- bucket, url, arn = _bucket_and_queue("xpol")
- _as(ACCT_A, sqs_svc._act_set_queue_attributes,
- {"QueueUrl": url, "Attributes": {"Policy": json.dumps(
- sqs_policy_allow_s3(arn, "no-such-bucket", ACCT_A))}}, url)
- _assert_put_rejected(bucket, "sqs", arn)
-
- def test_s3_to_sqs_delivered_with_policy(self, auth):
- bucket, url, arn = _bucket_and_queue("xpol")
- _as(ACCT_A, sqs_svc._act_set_queue_attributes,
- {"QueueUrl": url, "Attributes": {"Policy": json.dumps(
- sqs_policy_allow_s3(arn, bucket, ACCT_A))}}, url)
- assert _resp_code(_put_notif(ACCT_A, bucket, "sqs", arn)) == (200, "")
- bodies = _queue_bodies(arn)
- assert len(bodies) == 1
- assert "s3:TestEvent" in bodies[0]
-
- def test_s3_skip_validation_stores_but_delivery_blocked(self, auth):
- bucket, _url, arn = _bucket_and_queue("xpol")
- assert _resp_code(
- _put_notif(ACCT_A, bucket, "sqs", arn, skip=True)) == (200, "")
- assert _queue_bodies(arn) == []
- _as(ACCT_A, s3_svc._deliver_event_to_sqs,
- arn, {"Records": []}, REGION, bucket_name=bucket)
- assert _queue_bodies(arn) == []
-
- def test_s3_to_sns_default_policy_delivers(self, auth):
- bucket, topic, _qurl, qarn = _bucket_topic_queue("xpol")
- assert _resp_code(_put_notif(ACCT_A, bucket, "sns", topic)) == (200, "")
- bodies = _queue_bodies(qarn)
- assert len(bodies) == 1
- assert "s3:TestEvent" in bodies[0]
-
- def test_s3_put_rejected_by_topic_policy_without_s3(self, auth):
- bucket, topic, _qurl, _qarn = _bucket_topic_queue("xpol")
- _set_topic_policy(ACCT_A, topic, _allow_account(
- topic, ACCT_A, "sns:Publish"))
- _assert_put_rejected(bucket, "sns", topic)
-
- def test_s3_to_sns_skip_validation_delivery_blocked(self, auth):
- bucket, topic, _qurl, qarn = _bucket_topic_queue("xpol")
- _set_topic_policy(ACCT_A, topic, _allow_account(
- topic, ACCT_A, "sns:Publish"))
- assert _resp_code(
- _put_notif(ACCT_A, bucket, "sns", topic, skip=True)) == (200, "")
- _as(ACCT_A, s3_svc._deliver_event_to_sns,
- topic, {"Records": []}, REGION, bucket_name=bucket)
- assert _queue_bodies(qarn) == []
-
- def test_s3_to_cross_account_sns_topic_delivered_with_policy(self, auth):
- bucket = _uniq("xpol-xbkt")
- _make_bucket(ACCT_A, bucket)
- topic = _make_topic(ACCT_B, _uniq("xpol-xt"))
- _set_topic_policy(ACCT_B, topic, {
- "Version": "2012-10-17", "Statement": [{
- "Sid": "s3-xacct", "Effect": "Allow",
- "Principal": {"Service": "s3.amazonaws.com"},
- "Action": "sns:Publish", "Resource": topic,
- "Condition": {
- "ArnLike": {"aws:SourceArn": f"arn:aws:s3:::{bucket}"},
- "StringEquals": {"aws:SourceAccount": ACCT_A}}}]})
- qname = _uniq("xpol-xq")
- _make_queue(ACCT_B, qname, sqs_policy_allow_sns(
- _qarn(ACCT_B, qname), topic))
- assert _resp_code(_as(ACCT_B, sns_svc._subscribe, {
- "TopicArn": topic, "Protocol": "sqs",
- "Endpoint": _qarn(ACCT_B, qname)})) == (200, "")
- # The topic lives in another account, but its policy allows this
- # bucket's deliveries, so the PUT validates instead of failing.
- assert _resp_code(_put_notif(ACCT_A, bucket, "sns", topic)) == (200, "")
- bodies = _queue_bodies(_qarn(ACCT_B, qname))
- assert len(bodies) == 1
- assert "s3:TestEvent" in bodies[0]
-
- def test_s3_to_cross_account_sqs_queue_delivered_with_policy(self, auth):
- bucket = _uniq("xpol-xbkt")
- _make_bucket(ACCT_A, bucket)
- qname = _uniq("xpol-xq")
- _make_queue(ACCT_B, qname, sqs_policy_allow_s3(
- _qarn(ACCT_B, qname), bucket, ACCT_A))
- assert _resp_code(
- _put_notif(ACCT_A, bucket, "sqs", _qarn(ACCT_B, qname))) == (200, "")
- bodies = _queue_bodies(_qarn(ACCT_B, qname))
- assert len(bodies) == 1
- assert "s3:TestEvent" in bodies[0]
-
- def test_s3_to_cross_account_topic_rejected_without_policy(self, auth):
- bucket = _uniq("xpol-xbkt")
- _make_bucket(ACCT_A, bucket)
- topic = _make_topic(ACCT_B, _uniq("xpol-xt"))
- _assert_put_rejected(bucket, "sns", topic)
-
-
-# ---------------------------------------------------------------------------
-# SNS→SQS fanout gated on the queue policy
-# ---------------------------------------------------------------------------
-
-def _topic_and_queue(tag):
- topic = _make_topic(ACCT_A, _uniq(f"{tag}-t"))
- qname = _uniq(f"{tag}-q")
- qurl = _make_queue(ACCT_A, qname)
- assert _resp_code(_as(ACCT_A, sns_svc._subscribe, {
- "TopicArn": topic, "Protocol": "sqs",
- "Endpoint": _qarn(ACCT_A, qname)})) == (200, "")
- return topic, qurl, _qarn(ACCT_A, qname)
-
-
-class TestSnsFanoutPolicies:
- def test_fanout_blocked_without_policy(self, auth):
- topic, _qurl, qarn = _topic_and_queue("xpol-fan")
- assert _resp_code(_as(ACCT_A, sns_svc._publish, {
- "TopicArn": topic, "Message": "no-policy"})) == (200, "")
- assert _queue_bodies(qarn) == []
-
- def test_fanout_blocked_by_wrong_source_arn(self, auth):
- topic, qurl, qarn = _topic_and_queue("xpol-fan")
- _as(ACCT_A, sqs_svc._act_set_queue_attributes,
- {"QueueUrl": qurl, "Attributes": {"Policy": json.dumps(
- sqs_policy_allow_sns(
- qarn, f"arn:aws:sns:{REGION}:{ACCT_A}:wrong-topic"))}},
- qurl)
- assert _resp_code(_as(ACCT_A, sns_svc._publish, {
- "TopicArn": topic, "Message": "wrong-source"})) == (200, "")
- assert _queue_bodies(qarn) == []
-
- def test_fanout_delivered_with_policy(self, auth):
- topic, qurl, qarn = _topic_and_queue("xpol-fan")
- _as(ACCT_A, sqs_svc._act_set_queue_attributes,
- {"QueueUrl": qurl, "Attributes": {"Policy": json.dumps(
- sqs_policy_allow_sns(qarn, topic))}}, qurl)
- assert _resp_code(_as(ACCT_A, sns_svc._publish, {
- "TopicArn": topic, "Message": "with-policy"})) == (200, "")
- bodies = _queue_bodies(qarn)
- assert len(bodies) == 1
- assert "with-policy" in bodies[0]
-
- def test_cross_account_fanout_needs_queue_policy(self, auth):
- topic = _make_topic(ACCT_A, _uniq("xpol-xfan"))
- _set_topic_policy(ACCT_A, topic, _allow_account(
- topic, ACCT_B, "sns:Subscribe"))
- qname = _uniq("xpol-xfanq")
- qurl = _make_queue(ACCT_B, qname)
- assert _resp_code(_as(ACCT_B, sns_svc._subscribe, {
- "TopicArn": topic, "Protocol": "sqs",
- "Endpoint": _qarn(ACCT_B, qname)})) == (200, "")
- assert _resp_code(_as(ACCT_A, sns_svc._publish, {
- "TopicArn": topic, "Message": "no-queue-policy"})) == (200, "")
- assert _queue_bodies(_qarn(ACCT_B, qname)) == []
-
- _as(ACCT_B, sqs_svc._act_set_queue_attributes,
- {"QueueUrl": qurl, "Attributes": {"Policy": json.dumps(
- sqs_policy_allow_sns(_qarn(ACCT_B, qname), topic))}}, qurl)
- assert _resp_code(_as(ACCT_A, sns_svc._publish, {
- "TopicArn": topic, "Message": "with-queue-policy"})) == (200, "")
- bodies = _queue_bodies(_qarn(ACCT_B, qname))
- assert len(bodies) == 1
- assert "with-queue-policy" in bodies[0]
-
- def test_cross_account_subscribe_denied_without_policy(self, auth):
- topic = _make_topic(ACCT_A, _uniq("xpol-xfan"))
- assert _resp_code(_as(ACCT_B, sns_svc._subscribe, {
- "TopicArn": topic, "Protocol": "email",
- "Endpoint": "[email protected]"})) == (403, "AuthorizationError")
-
-
-# ---------------------------------------------------------------------------
-# AUTH=false stays permissive
-# ---------------------------------------------------------------------------
-
-class TestAuthDisabledIsPermissive:
- def test_cross_account_send_passes_without_policy(self, noauth):
- name = _uniq("xpol-q")
- url = _make_queue(ACCT_A, name)
- _as(ACCT_B, sqs_svc._act_send_message,
- {"QueueUrl": url, "MessageBody": "open"}, url)
- assert _queue_bodies(_qarn(ACCT_A, name)) == ["open"]
-
- def test_cross_account_publish_passes_without_policy(self, noauth):
- arn = _make_topic(ACCT_A, _uniq("xpol-t"))
- assert _resp_code(_as(ACCT_B, sns_svc._publish, {
- "TopicArn": arn, "Message": "open"})) == (200, "")
-
- def test_explicit_deny_ignored(self, noauth):
- name = _uniq("xpol-q")
- url = _make_queue(ACCT_A, name, _deny_all(
- _qarn(ACCT_A, name), "sqs:SendMessage"))
- _as(ACCT_A, sqs_svc._act_send_message,
- {"QueueUrl": url, "MessageBody": "deny-ignored"}, url)
- assert _queue_bodies(_qarn(ACCT_A, name)) == ["deny-ignored"]
-
- def test_get_queue_url_ignores_deny(self, noauth):
- name = _uniq("xpol-q")
- url = _make_queue(ACCT_A, name, _deny_all(
- _qarn(ACCT_A, name), "sqs:GetQueueUrl"))
- assert _as(ACCT_A, sqs_svc._act_get_queue_url,
- {"QueueName": name}, "") == {"QueueUrl": url}
-
- def test_s3_put_and_delivery_without_policy(self, noauth):
- bucket, _url, arn = _bucket_and_queue("xpol")
- assert _resp_code(_put_notif(ACCT_A, bucket, "sqs", arn)) == (200, "")
- bodies = _queue_bodies(arn)
- assert len(bodies) == 1
- assert "s3:TestEvent" in bodies[0]
-
- def test_fanout_without_policy(self, noauth):
- topic, _qurl, qarn = _topic_and_queue("xpol-fan")
- assert _resp_code(_as(ACCT_A, sns_svc._publish, {
- "TopicArn": topic, "Message": "open"})) == (200, "")
- assert len(_queue_bodies(qarn)) == 1
diff --git a/tests/test_s3.py b/tests/test_s3.py
index b161593e..150de7b9 100644
--- a/tests/test_s3.py
+++ b/tests/test_s3.py
@@ -1,7 +1,11 @@
+import base64
+import hashlib
import json
import os
import re
+import struct
import time
+import uuid
import uuid as _uuid_mod
from datetime import datetime
from urllib.parse import urlparse
@@ -3614,11 +3618,13 @@ def s3_persist(tmp_path, monkeypatch):
buckets_before = set(s3mod._buckets._data)
versions_before = set(s3mod._object_versions._data)
versioning_before = set(s3mod._bucket_versioning._data)
+ annotations_before = set(s3mod._object_annotations._data)
yield s3mod
for store, before in (
(s3mod._buckets._data, buckets_before),
(s3mod._object_versions._data, versions_before),
(s3mod._bucket_versioning._data, versioning_before),
+ (s3mod._object_annotations._data, annotations_before),
):
for key in set(store) - before:
store.pop(key, None)
@@ -6998,3 +7004,785 @@ def test_s3_notification_keeps_an_explicit_id(s3, sqs):
)
assert s3.get_bucket_notification_configuration(
Bucket=bucket)["QueueConfigurations"][0]["Id"] == "mine"
+
+
+# --- Object annotations ---
+
+def _bucket(s3, versioned=False, **create):
+ name = f"annot-{uuid.uuid4().hex[:12]}"
+ s3.create_bucket(Bucket=name, **create)
+ if versioned:
+ s3.put_bucket_versioning(Bucket=name, VersioningConfiguration={"Status": "Enabled"})
+ return name
+
+
+def _code(exc):
+ return exc.value.response["Error"]["Code"]
+
+
+def _status(exc):
+ return exc.value.response["ResponseMetadata"]["HTTPStatusCode"]
+
+
+# ── Put / Get ──────────────────────────────────────────────────────────
+
+
+def test_put_and_get_an_annotation(s3):
+ bucket = _bucket(s3, versioned=True)
+ payload = b'{"archive":"Images"}'
+ put = s3.put_object(Bucket=bucket, Key="photo.jpg", Body=b"jpeg")
+ out = s3.put_object_annotation(Bucket=bucket, Key="photo.jpg", AnnotationName="nexus.system", AnnotationPayload=payload)
+ assert out["ObjectVersionId"] == put["VersionId"]
+ assert out["ETag"] == f'"{hashlib.md5(payload).hexdigest()}"'
+ got = s3.get_object_annotation(Bucket=bucket, Key="photo.jpg", AnnotationName="nexus.system")
+ assert got["AnnotationPayload"].read() == payload
+ assert got["ObjectVersionId"] == put["VersionId"]
+ assert got["ContentLength"] == len(payload)
+ assert got["ServerSideEncryption"] == "AES256"
+
+
+def test_an_annotation_leaves_the_object_alone(s3):
+ """"Adding, updating, or removing an annotation does not modify the parent
+ object's ETag", and makes no new version."""
+ bucket = _bucket(s3, versioned=True)
+ put = s3.put_object(Bucket=bucket, Key="k", Body=b"body")
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ head = s3.head_object(Bucket=bucket, Key="k")
+ assert head["ETag"] == put["ETag"] and head["VersionId"] == put["VersionId"]
+ assert len(s3.list_object_versions(Bucket=bucket)["Versions"]) == 1
+ assert s3.get_object(Bucket=bucket, Key="k")["Body"].read() == b"body"
+
+
+def test_putting_again_replaces_the_payload(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"first")
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"second")
+ got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+ assert got["AnnotationPayload"].read() == b"second"
+
+
+def test_get_of_a_missing_annotation(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ with pytest.raises(ClientError) as exc:
+ s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="absent")
+ assert (_code(exc), _status(exc)) == ("NoSuchAnnotation", 404)
+
+
+def test_annotation_on_a_missing_key_or_bucket(s3):
+ bucket = _bucket(s3)
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(Bucket=bucket, Key="nope", AnnotationName="a", AnnotationPayload=b"1")
+ assert _code(exc) == "NoSuchKey"
+ with pytest.raises(ClientError) as exc:
+ s3.get_object_annotation(Bucket=f"missing-{uuid.uuid4().hex[:8]}", Key="k", AnnotationName="a")
+ assert _code(exc) == "NoSuchBucket"
+
+
+ "name, code",
+ [
+ ("aws.reserved", "InvalidAnnotationName"),
+ ("AWSthing", "InvalidAnnotationName"),
+ ("s3-thing", "InvalidAnnotationName"),
+ ("S3", "InvalidAnnotationName"),
+ ("has space", "InvalidAnnotationName"),
+ ("slash/name", "InvalidAnnotationName"),
+ (" ", "InvalidAnnotationName"),
+ ("n" * 513, "AnnotationNameTooLong"),
+ ],
+)
+def test_annotation_naming_rules(s3, name, code):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=name, AnnotationPayload=b"1")
+ assert (_code(exc), _status(exc)) == (code, 400)
+
+
+def test_names_may_use_letters_of_any_language(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ for name in ("données_1.2-3", "名前", "n" * 512):
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=name, AnnotationPayload=b"1")
+
+
+def test_payload_limits(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="max", AnnotationPayload=b"a" * (1024 * 1024))
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="big", AnnotationPayload=b"a" * (1024 * 1024 + 1))
+ assert _code(exc) == "InvalidRequest"
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="empty", AnnotationPayload=b"")
+ assert _code(exc) == "InvalidRequest"
+
+
+def test_payload_must_be_utf8(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="bin", AnnotationPayload=b"\xff\xfe")
+ assert (_code(exc), _status(exc)) == ("UnsupportedMediaType", 415)
+
+
+def test_a_version_holds_at_most_1000_annotations(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ for i in range(1000):
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=f"n{i}", AnnotationPayload=b"1")
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="n1000", AnnotationPayload=b"1")
+ assert _code(exc) == "AnnotationLimitExceeded"
+ # Replacing one of the thousand is not a new one.
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="n0", AnnotationPayload=b"2")
+
+
+def test_object_if_match(s3):
+ bucket = _bucket(s3)
+ etag = s3.put_object(Bucket=bucket, Key="k", Body=b"x")["ETag"]
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1", ObjectIfMatch=etag)
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(
+ Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"2", ObjectIfMatch='"deadbeef"'
+ )
+ assert _status(exc) == 412
+ with pytest.raises(ClientError) as exc:
+ s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", ObjectIfMatch='"deadbeef"')
+ assert _status(exc) == 412
+ s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", ObjectIfMatch=etag)
+
+
+# ── Checksums ──────────────────────────────────────────────────────────
+
+
+def test_default_checksum_is_crc64nvme(s3):
+ """"If the annotation doesn't have a specified checksum algorithm or
+ checksum value, Amazon S3 uses the CRC-64/NVME algorithm"."""
+ plain = make_client("s3", {"request_checksum_calculation": "when_required"})
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ out = plain.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"123456789")
+ expected = base64.b64encode(struct.pack(">Q", _crc64nvme_bitwise(b"123456789"))).decode()
+ assert out["ChecksumCRC64NVME"] == expected
+ got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", ChecksumMode="ENABLED")
+ assert got["ChecksumCRC64NVME"] == expected
+ # Only when asked: a client that does not send x-amz-checksum-mode gets none back.
+ unasked = make_client("s3", {"response_checksum_validation": "when_required"})
+ assert "ChecksumCRC64NVME" not in unasked.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+
+
+ "member, digest",
+ [
+ ("ChecksumSHA256", lambda b: hashlib.sha256(b).digest()),
+ ("ChecksumSHA512", lambda b: hashlib.sha512(b).digest()),
+ ("ChecksumMD5", lambda b: hashlib.md5(b).digest()),
+ # XXH64's published check value for b"abc" (the xxHash reference).
+ ("ChecksumXXHASH64", lambda b: struct.pack(">Q", 0x44BC2CF5AD770999)),
+ ],
+)
+def test_a_checksum_sent_is_verified(s3, member, digest):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ good = base64.b64encode(digest(b"abc")).decode()
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"abc", **{member: good})
+ got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", ChecksumMode="ENABLED")
+ assert got[member] == good and got["ChecksumType"] == "FULL_OBJECT"
+ bad = base64.b64encode(bytes(x ^ 0xFF for x in digest(b"abc"))).decode()
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"abc", **{member: bad})
+ assert _code(exc) == "BadDigest"
+
+
+def test_xxh64_check_values():
+ from ministack.services import s3 as s3mod
+
+ assert s3mod._xxh64(b"") == 0xEF46DB3751D8E999
+ assert s3mod._xxh64(b"abc") == 0x44BC2CF5AD770999
+ # Past the 32-byte stripe loop, against the reference implementation's output.
+ assert s3mod._xxh64(b"Nobody inspects the spammish repetition") == 0xFBCEA83C8A378BF1
+
+
+def test_an_algorithm_ministack_cannot_compute_is_refused(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(
+ Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"abc", ChecksumXXHASH3="AAAAAAAAAAA="
+ )
+ assert _code(exc) == "InvalidRequest"
+
+
+# ── Encryption ─────────────────────────────────────────────────────────
+
+
+def test_annotations_take_the_objects_encryption(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x", ServerSideEncryption="aws:kms", SSEKMSKeyId="alias/annot")
+ out = s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ assert out["ServerSideEncryption"] == "aws:kms"
+ got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+ assert got["ServerSideEncryption"] == "aws:kms"
+
+
+def test_sse_c_objects_cannot_have_annotations(s3):
+ bucket = _bucket(s3)
+ key = os.urandom(32)
+ s3.put_object(
+ Bucket=bucket,
+ Key="k",
+ Body=b"x",
+ SSECustomerAlgorithm="AES256",
+ SSECustomerKey=key,
+ )
+ with pytest.raises(ClientError) as exc:
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ assert _code(exc) == "InvalidRequest"
+
+
+# ── Versions ───────────────────────────────────────────────────────────
+
+
+def test_annotations_belong_to_one_version(s3):
+ bucket = _bucket(s3, versioned=True)
+ v1 = s3.put_object(Bucket=bucket, Key="k", Body=b"one")["VersionId"]
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"on v1")
+ v2 = s3.put_object(Bucket=bucket, Key="k", Body=b"two")["VersionId"]
+ # "Creating a new version does not copy annotations from the previous version."
+ with pytest.raises(ClientError) as exc:
+ s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+ assert _code(exc) == "NoSuchAnnotation"
+ got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v1)
+ assert got["AnnotationPayload"].read() == b"on v1" and got["ObjectVersionId"] == v1
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"on v2")
+ assert s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v2)["AnnotationPayload"].read() == b"on v2"
+ assert s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v1)["AnnotationPayload"].read() == b"on v1"
+
+
+def test_a_delete_marker_keeps_the_versions_annotations(s3):
+ bucket = _bucket(s3, versioned=True)
+ v1 = s3.put_object(Bucket=bucket, Key="k", Body=b"one")["VersionId"]
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"kept")
+ marker = s3.delete_object(Bucket=bucket, Key="k")["VersionId"]
+ with pytest.raises(ClientError) as exc:
+ s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+ assert _code(exc) == "NoSuchKey"
+ with pytest.raises(ClientError) as exc:
+ s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=marker)
+ assert _status(exc) == 405
+ assert s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v1)["AnnotationPayload"].read() == b"kept"
+ # Removing the marker brings the version back with its annotations.
+ s3.delete_object(Bucket=bucket, Key="k", VersionId=marker)
+ assert s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")["AnnotationPayload"].read() == b"kept"
+
+
+def test_deleting_a_version_deletes_its_annotations(s3):
+ bucket = _bucket(s3, versioned=True)
+ v1 = s3.put_object(Bucket=bucket, Key="k", Body=b"one")["VersionId"]
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ s3.put_object(Bucket=bucket, Key="k", Body=b"two")
+ s3.delete_object(Bucket=bucket, Key="k", VersionId=v1)
+ with pytest.raises(ClientError) as exc:
+ s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v1)
+ assert _code(exc) == "NoSuchVersion"
+
+
+def test_unversioned_overwrite_and_delete_drop_annotations(s3):
+ """"In a non-versioned bucket, if you delete or overwrite the object, the
+ annotations are deleted with it.\""""
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"one")
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ s3.put_object(Bucket=bucket, Key="k", Body=b"two")
+ with pytest.raises(ClientError) as exc:
+ s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+ assert _code(exc) == "NoSuchAnnotation"
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="b", AnnotationPayload=b"1")
+ s3.delete_object(Bucket=bucket, Key="k")
+ s3.put_object(Bucket=bucket, Key="k", Body=b"three")
+ assert s3.list_object_annotations(Bucket=bucket, Key="k")["Annotations"] == []
+
+
+def test_batch_delete_drops_annotations(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"one")
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ s3.delete_objects(Bucket=bucket, Delete={"Objects": [{"Key": "k"}]})
+ s3.put_object(Bucket=bucket, Key="k", Body=b"two")
+ assert s3.list_object_annotations(Bucket=bucket, Key="k")["Annotations"] == []
+
+
+def test_annotations_need_no_restore_for_archived_objects(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="cold", Body=b"x", StorageClass="GLACIER")
+ s3.put_object_annotation(Bucket=bucket, Key="cold", AnnotationName="a", AnnotationPayload=b"1")
+ assert s3.get_object_annotation(Bucket=bucket, Key="cold", AnnotationName="a")["AnnotationPayload"].read() == b"1"
+
+
+# ── List ───────────────────────────────────────────────────────────────
+
+
+def test_list_annotations(s3):
+ bucket = _bucket(s3, versioned=True)
+ version = s3.put_object(Bucket=bucket, Key="k", Body=b"x")["VersionId"]
+ for name, payload in (("ml.label", b"cat"), ("etl.status", b"done"), ("ml.score", b"0.97")):
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=name, AnnotationPayload=payload, ChecksumAlgorithm="SHA256")
+ out = s3.list_object_annotations(Bucket=bucket, Key="k")
+ assert [a["AnnotationName"] for a in out["Annotations"]] == ["etl.status", "ml.label", "ml.score"]
+ assert out["AnnotationCount"] == 3 and out["ObjectVersionId"] == version
+ assert out["Bucket"] == bucket and out["Key"] == "k"
+ entry = out["Annotations"][1]
+ assert entry["Size"] == 3 and entry["ETag"] == f'"{hashlib.md5(b"cat").hexdigest()}"'
+ assert entry["ChecksumAlgorithm"] == ["SHA256"] and "LastModified" in entry
+ assert "NextContinuationToken" not in out
+
+
+def test_list_annotations_by_prefix_and_page(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ for i in range(5):
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=f"page.{i}", AnnotationPayload=b"1")
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="other", AnnotationPayload=b"1")
+ first = s3.list_object_annotations(Bucket=bucket, Key="k", AnnotationPrefix="page.", MaxAnnotationResults=2)
+ assert [a["AnnotationName"] for a in first["Annotations"]] == ["page.0", "page.1"]
+ assert first["AnnotationPrefix"] == "page." and first["MaxAnnotationResults"] == 2
+ token = first["NextContinuationToken"]
+ rest = s3.list_object_annotations(Bucket=bucket, Key="k", AnnotationPrefix="page.", ContinuationToken=token)
+ assert [a["AnnotationName"] for a in rest["Annotations"]] == ["page.2", "page.3", "page.4"]
+ assert rest["ContinuationToken"] == token and "NextContinuationToken" not in rest
+
+
+def test_list_rejects_a_bad_page_size_or_prefix(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ plain = make_client("s3", {"parameter_validation": False})
+ with pytest.raises(ClientError) as exc:
+ plain.list_object_annotations(Bucket=bucket, Key="k", MaxAnnotationResults=1001)
+ assert _code(exc) == "InvalidArgument"
+ with pytest.raises(ClientError) as exc:
+ s3.list_object_annotations(Bucket=bucket, Key="k", AnnotationPrefix="p" * 513)
+ assert _code(exc) == "InvalidPrefix"
+
+
+# ── Delete ─────────────────────────────────────────────────────────────
+
+
+def test_delete_an_annotation(s3):
+ bucket = _bucket(s3, versioned=True)
+ version = s3.put_object(Bucket=bucket, Key="k", Body=b"x")["VersionId"]
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ out = s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+ assert out["ResponseMetadata"]["HTTPStatusCode"] == 204 and out["ObjectVersionId"] == version
+ with pytest.raises(ClientError) as exc:
+ s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+ assert _code(exc) == "NoSuchAnnotation"
+ # Deleting one that is not there is not an error.
+ s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+
+
+# Observed on AWS (ap-southeast-2, 2026-09-28): annotations are writes to the object version, so Object Lock guards
+# PutObjectAnnotation and DeleteObjectAnnotation alike. Governance retention yields to
+# x-amz-bypass-governance-retention; compliance retention and a legal hold refuse both, bypass or not.
+
+
+RETENTION = "Access Denied because object protected by object lock retention."
+LEGAL_HOLD = "Access Denied because object protected by object lock legal hold."
+
+
+def _lock_call(method, path, headers=None, body=b"", query=None):
+ import asyncio
+
+ from ministack.core.responses import request_scope
+ from ministack.services import s3 as s3mod
+
+ with request_scope("000000000000", "us-east-1"):
+ return asyncio.run(s3mod.handle_request(method, path, headers or {}, body, query or {}))
+
+
+def _locked_in_process(**lock_headers):
+ bucket = f"ann-lock-{uuid.uuid4().hex[:10]}"
+ assert _lock_call("PUT", f"/{bucket}", {"x-amz-bucket-object-lock-enabled": "true"})[0] == 200
+ assert _lock_call("PUT", f"/{bucket}/k", lock_headers, b"x")[0] == 200
+ return bucket
+
+
+def _annotation_write(bucket, method, bypass=False):
+ headers = {"x-amz-bypass-governance-retention": "true"} if bypass else {}
+ query = {"annotation": [""], "annotationName": ["a"]}
+ return _lock_call(method, f"/{bucket}/k", headers, b"1" if method == "PUT" else b"", query)
+
+
+def _assert_refused(response, message):
+ status, _, body = response
+ assert status == 403
+ assert b"<Code>AccessDenied</Code>" in body and message.encode() in body
+
+
+def _until():
+ return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() + 3600))
+
+
+def test_governance_retention_needs_the_bypass_to_put_or_delete_an_annotation(monkeypatch):
+ import ministack.app as app_mod
+
+ monkeypatch.setattr(app_mod, "AUTH", True)
+ bucket = _locked_in_process(**{"x-amz-object-lock-mode": "GOVERNANCE",
+ "x-amz-object-lock-retain-until-date": _until()})
+ _assert_refused(_annotation_write(bucket, "PUT"), RETENTION)
+ assert _annotation_write(bucket, "PUT", bypass=True)[0] == 200
+ _assert_refused(_annotation_write(bucket, "DELETE"), RETENTION)
+ assert _annotation_write(bucket, "DELETE", bypass=True)[0] == 204
+
+
[email protected]("lock_headers,message", [
+ ({"x-amz-object-lock-mode": "COMPLIANCE"}, RETENTION),
+ ({"x-amz-object-lock-legal-hold": "ON"}, LEGAL_HOLD),
+])
+def test_compliance_and_legal_hold_refuse_annotation_writes_even_with_the_bypass(monkeypatch, lock_headers, message):
+ import ministack.app as app_mod
+
+ monkeypatch.setattr(app_mod, "AUTH", True)
+ if "x-amz-object-lock-mode" in lock_headers:
+ lock_headers = dict(lock_headers, **{"x-amz-object-lock-retain-until-date": _until()})
+ bucket = _locked_in_process(**lock_headers)
+ for method in ("PUT", "DELETE"):
+ for bypass in (False, True):
+ _assert_refused(_annotation_write(bucket, method, bypass), message)
+
+
+def test_object_lock_does_not_refuse_annotation_writes_without_auth(monkeypatch):
+ import ministack.app as app_mod
+
+ monkeypatch.setattr(app_mod, "AUTH", False)
+ bucket = _locked_in_process(**{"x-amz-object-lock-legal-hold": "ON"})
+ assert _annotation_write(bucket, "PUT")[0] == 200
+ assert _annotation_write(bucket, "DELETE")[0] == 204
+
+
+def _locked(s3, **lock):
+ bucket = _bucket(s3, ObjectLockEnabledForBucket=True)
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x", **lock)
+ return bucket
+
+
+def test_an_unlocked_object_in_a_lock_enabled_bucket_takes_annotations_freely(s3):
+ bucket = _locked(s3)
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"2")
+ s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+
+
+# ── CopyObject ─────────────────────────────────────────────────────────
+
+
+def test_copy_carries_annotations_by_default(s3):
+ src = _bucket(s3, versioned=True)
+ dst = _bucket(s3, versioned=True)
+ s3.put_object(Bucket=src, Key="landing/x.jpg", Body=b"x")
+ s3.put_object_annotation(Bucket=src, Key="landing/x.jpg", AnnotationName="nexus.system", AnnotationPayload=b"{}")
+ copied = s3.copy_object(Bucket=dst, Key="assets/x.jpg", CopySource={"Bucket": src, "Key": "landing/x.jpg"})
+ got = s3.get_object_annotation(Bucket=dst, Key="assets/x.jpg", AnnotationName="nexus.system")
+ assert got["AnnotationPayload"].read() == b"{}" and got["ObjectVersionId"] == copied["VersionId"]
+
+
+def test_copy_of_a_named_version_carries_that_versions_annotations(s3):
+ bucket = _bucket(s3, versioned=True)
+ v1 = s3.put_object(Bucket=bucket, Key="k", Body=b"one")["VersionId"]
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"v1")
+ s3.put_object(Bucket=bucket, Key="k", Body=b"two")
+ s3.copy_object(Bucket=bucket, Key="copy", CopySource={"Bucket": bucket, "Key": "k", "VersionId": v1})
+ assert s3.get_object_annotation(Bucket=bucket, Key="copy", AnnotationName="a")["AnnotationPayload"].read() == b"v1"
+
+
+def test_copy_with_exclude_leaves_them_behind(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="src", Body=b"x")
+ s3.put_object_annotation(Bucket=bucket, Key="src", AnnotationName="a", AnnotationPayload=b"1")
+ s3.copy_object(Bucket=bucket, Key="dst", CopySource={"Bucket": bucket, "Key": "src"}, AnnotationDirective="EXCLUDE")
+ assert s3.list_object_annotations(Bucket=bucket, Key="dst")["Annotations"] == []
+
+
+def test_copy_rejects_an_unknown_directive(s3):
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="src", Body=b"x")
+ plain = make_client("s3", {"parameter_validation": False})
+ with pytest.raises(ClientError) as exc:
+ plain.copy_object(Bucket=bucket, Key="dst", CopySource={"Bucket": bucket, "Key": "src"}, AnnotationDirective="REPLACE")
+ assert _code(exc) == "InvalidArgument"
+
+
+def test_copy_with_a_new_algorithm_rechecksums_the_annotations(s3):
+ """"If you specify a different checksum algorithm in the copy request, the
+ new algorithm applies to both the object and its annotations.\""""
+ bucket = _bucket(s3)
+ s3.put_object(Bucket=bucket, Key="src", Body=b"x")
+ s3.put_object_annotation(Bucket=bucket, Key="src", AnnotationName="a", AnnotationPayload=b"abc")
+ s3.copy_object(Bucket=bucket, Key="dst", CopySource={"Bucket": bucket, "Key": "src"}, ChecksumAlgorithm="SHA256")
+ got = s3.get_object_annotation(Bucket=bucket, Key="dst", AnnotationName="a", ChecksumMode="ENABLED")
+ assert got["ChecksumSHA256"] == base64.b64encode(hashlib.sha256(b"abc").digest()).decode()
+
+
+# ── Replication ────────────────────────────────────────────────────────
+
+
+def test_annotations_replicate(s3):
+ src = _bucket(s3, versioned=True)
+ dst = _bucket(s3, versioned=True)
+ s3.put_bucket_replication(
+ Bucket=src,
+ ReplicationConfiguration={
+ "Role": "arn:aws:iam::000000000000:role/repl",
+ "Rules": [{"ID": "r1", "Status": "Enabled", "Prefix": "", "Destination": {"Bucket": f"arn:aws:s3:::{dst}"}}],
+ },
+ )
+ s3.put_object(Bucket=src, Key="k", Body=b"x")
+ s3.put_object_annotation(Bucket=src, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ assert s3.get_object_annotation(Bucket=src, Key="k", AnnotationName="a")["ReplicationStatus"] == "COMPLETED"
+ replica = s3.get_object_annotation(Bucket=dst, Key="k", AnnotationName="a")
+ assert replica["AnnotationPayload"].read() == b"1" and replica["ReplicationStatus"] == "REPLICA"
+ listed = s3.list_object_annotations(Bucket=dst, Key="k")["Annotations"]
+ assert [(a["AnnotationName"], a["ReplicationStatus"]) for a in listed] == [("a", "REPLICA")]
+
+
+# ── Event notifications ────────────────────────────────────────────────
+
+
+def _notified(s3, sqs, events, versioned=True):
+ bucket = _bucket(s3, versioned=versioned)
+ url = sqs.create_queue(QueueName=bucket)["QueueUrl"]
+ arn = sqs.get_queue_attributes(QueueUrl=url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"]
+ s3.put_bucket_notification_configuration(
+ Bucket=bucket, NotificationConfiguration={"QueueConfigurations": [{"QueueArn": arn, "Events": events}]}
+ )
+ return bucket, url
+
+
+def _records(sqs, url, expected, timeout=10):
+ records = []
+ deadline = time.time() + timeout
+ while len(records) < expected and time.time() < deadline:
+ for m in sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=10, WaitTimeSeconds=1).get("Messages", []):
+ records.extend(json.loads(m["Body"]).get("Records", []))
+ sqs.delete_message(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"])
+ return records
+
+
+def test_annotation_events(s3, sqs):
+ bucket, url = _notified(s3, sqs, ["s3:ObjectAnnotation:*", "s3:ObjectCreated:*"])
+ version = s3.put_object(Bucket=bucket, Key="k", Body=b"x")["VersionId"]
+ assert [r["eventName"] for r in _records(sqs, url, 1)] == ["ObjectCreated:Put"]
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"abc")
+ s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+ records = sorted(_records(sqs, url, 2), key=lambda r: r["eventName"], reverse=True)
+ # "PutObjectAnnotation and DeleteObjectAnnotation send only annotation-specific events."
+ assert [r["eventName"] for r in records] == ["ObjectAnnotation:Put", "ObjectAnnotation:Delete"]
+ put, delete = records
+ assert put["objectAnnotation"] == [{"name": "a", "size": 3, "eTag": hashlib.md5(b"abc").hexdigest()}]
+ assert delete["objectAnnotation"] == [{"name": "a"}]
+ assert put["s3"]["object"]["versionId"] == version
+ assert "sequencer" not in put["s3"]["object"]
+
+
+def test_get_and_list_send_no_events(s3, sqs):
+ bucket, url = _notified(s3, sqs, ["s3:ObjectAnnotation:*"])
+ s3.put_object(Bucket=bucket, Key="k", Body=b"x")
+ s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
+ assert len(_records(sqs, url, 1)) == 1
+ s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
+ s3.list_object_annotations(Bucket=bucket, Key="k")
+ assert _records(sqs, url, 1, timeout=2) == []
+
+
+def test_copy_event_says_whether_annotations_came(s3, sqs):
+ bucket, url = _notified(s3, sqs, ["s3:ObjectCreated:Copy"])
+ s3.put_object(Bucket=bucket, Key="src", Body=b"x")
+ s3.put_object_annotation(Bucket=bucket, Key="src", AnnotationName="a", AnnotationPayload=b"1")
+ s3.copy_object(Bucket=bucket, Key="with", CopySource={"Bucket": bucket, "Key": "src"})
+ s3.copy_object(Bucket=bucket, Key="without", CopySource={"Bucket": bucket, "Key": "src"}, AnnotationDirective="EXCLUDE")
+ flags = {r["s3"]["object"]["key"]: r["s3"]["object"]["hasObjectAnnotation"] for r in _records(sqs, url, 2)}
+ assert flags == {"with": True, "without": False}
+
+
+# ── Persistence ────────────────────────────────────────────────────────
+
+
+def test_annotations_survive_a_restart(s3_persist, tmp_path):
+ s3mod = s3_persist
+ bucket = "qa-annot-persist"
+ s3mod._create_bucket(bucket, b"")
+ s3mod._bucket_versioning[bucket] = "Enabled"
+ v1 = s3mod._put_object(bucket, "k", b"one", {})[1]["x-amz-version-id"]
+ s3mod._put_object_annotation(bucket, "k", b"on v1", {}, {"annotationName": ["a"]})
+ v2 = s3mod._put_object(bucket, "k", b"two", {})[1]["x-amz-version-id"]
+ s3mod._put_object_annotation(bucket, "k", b"on v2", {}, {"annotationName": ["a"]})
+
+ s3mod._buckets._data.pop(("000000000000", bucket), None)
+ for store in (s3mod._object_versions._data, s3mod._object_annotations._data):
+ for k in [k for k in store if k[0] == "000000000000" and k[1][0] == bucket]:
+ store.pop(k, None)
+ s3mod._load_persisted_bucket("000000000000", bucket, os.path.join(str(tmp_path), "000000000000", bucket))
+
+ assert s3mod._object_annotations[(bucket, "k", v1)]["a"]["payload"] == "on v1"
+ assert s3mod._object_annotations[(bucket, "k", v2)]["a"]["payload"] == "on v2"
+
+
+# --- Event notification records ---
+
+def _queue(sqs, name):
+ url = sqs.create_queue(QueueName=name)["QueueUrl"]
+ arn = sqs.get_queue_attributes(QueueUrl=url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"]
+ return url, arn
+
+
+def _event_records(sqs, url, expected, timeout=10):
+ """Every S3 record delivered to the queue, waiting until `expected` have arrived."""
+ records = []
+ deadline = time.time() + timeout
+ while len(records) < expected and time.time() < deadline:
+ resp = sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=10, WaitTimeSeconds=1)
+ for m in resp.get("Messages", []):
+ body = json.loads(m["Body"])
+ records.extend(body.get("Records", []))
+ sqs.delete_message(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"])
+ return records
+
+
+def notified(s3, sqs):
+ """A bucket whose object events go to a fresh queue: (bucket, queue url, configure)."""
+ suffix = uuid.uuid4().hex[:8]
+ bucket = f"evt-records-{suffix}"
+ s3.create_bucket(Bucket=bucket)
+ url, arn = _queue(sqs, f"evt-records-{suffix}")
+
+ def configure(*events, versioned=False):
+ if versioned:
+ s3.put_bucket_versioning(Bucket=bucket, VersioningConfiguration={"Status": "Enabled"})
+ s3.put_bucket_notification_configuration(
+ Bucket=bucket,
+ NotificationConfiguration={"QueueConfigurations": [{"QueueArn": arn, "Events": list(events)}]},
+ )
+ # The configuration's own s3:TestEvent carries no Records; drain it.
+ _event_records(sqs, url, 0, timeout=1)
+
+ return bucket, url, configure
+
+
+def test_record_key_is_url_encoded(s3, sqs, notified):
+ bucket, url, configure = notified
+ configure("s3:ObjectCreated:*")
+ s3.put_object(Bucket=bucket, Key="photos/red flower+(1).jpg", Body=b"x")
+ [record] = _event_records(sqs, url, 1)
+ # "`red flower.jpg` becomes `red+flower.jpg`"; slashes stay, a literal + is escaped.
+ assert record["s3"]["object"]["key"] == "photos/red+flower%2B%281%29.jpg"
+ assert record["eventVersion"] == "2.6"
+
+
+def test_record_carries_the_version_the_put_made(s3, sqs, notified):
+ bucket, url, configure = notified
+ configure("s3:ObjectCreated:*", versioned=True)
+ first = s3.put_object(Bucket=bucket, Key="doc.txt", Body=b"one")["VersionId"]
+ second = s3.put_object(Bucket=bucket, Key="doc.txt", Body=b"two")["VersionId"]
+ records = _event_records(sqs, url, 2)
+ assert sorted(r["s3"]["object"]["versionId"] for r in records) == sorted([first, second])
+
+
+def test_record_has_no_version_in_an_unversioned_bucket(s3, sqs, notified):
+ bucket, url, configure = notified
+ configure("s3:ObjectCreated:*")
+ s3.put_object(Bucket=bucket, Key="plain.txt", Body=b"x")
+ [record] = _event_records(sqs, url, 1)
+ assert "versionId" not in record["s3"]["object"]
+
+
+def test_sequencer_grows_with_each_write(s3, sqs, notified):
+ bucket, url, configure = notified
+ configure("s3:ObjectCreated:*", "s3:ObjectRemoved:*", versioned=True)
+ v1 = s3.put_object(Bucket=bucket, Key="seq.txt", Body=b"1")["VersionId"]
+ v2 = s3.put_object(Bucket=bucket, Key="seq.txt", Body=b"2")["VersionId"]
+ marker = s3.delete_object(Bucket=bucket, Key="seq.txt")["VersionId"]
+ by_version = {r["s3"]["object"]["versionId"]: r["s3"]["object"]["sequencer"] for r in _event_records(sqs, url, 3)}
+ # Compared as S3 says to: left-pad the shorter with zeros, then compare.
+ width = max(len(v) for v in by_version.values())
+ ordered = [by_version[v].rjust(width, "0") for v in (v1, v2, marker)]
+ assert ordered == sorted(ordered) and len(set(ordered)) == 3
+ assert all(int(s, 16) >= 0 for s in ordered)
+
+
+def test_delete_without_version_reports_the_marker(s3, sqs, notified):
+ bucket, url, configure = notified
+ configure("s3:ObjectRemoved:*", versioned=True)
+ s3.put_object(Bucket=bucket, Key="gone.txt", Body=b"x")
+ marker = s3.delete_object(Bucket=bucket, Key="gone.txt")["VersionId"]
+ [record] = _event_records(sqs, url, 1)
+ assert record["eventName"] == "ObjectRemoved:DeleteMarkerCreated"
+ assert record["s3"]["object"]["versionId"] == marker
+
+
+def test_marker_is_not_an_object_removed_delete(s3, sqs, notified):
+ bucket, url, configure = notified
+ # Subscribed to permanent deletes only: a marker is its own event type.
+ configure("s3:ObjectRemoved:Delete", versioned=True)
+ version = s3.put_object(Bucket=bucket, Key="kept.txt", Body=b"x")["VersionId"]
+ s3.delete_object(Bucket=bucket, Key="kept.txt")
+ s3.delete_object(Bucket=bucket, Key="kept.txt", VersionId=version)
+ [record] = _event_records(sqs, url, 1)
+ assert record["eventName"] == "ObjectRemoved:Delete"
+ assert record["s3"]["object"]["versionId"] == version
+ assert _event_records(sqs, url, 1, timeout=2) == []
+
+
+def test_copy_record_names_the_new_version(s3, sqs, notified):
+ bucket, url, configure = notified
+ configure("s3:ObjectCreated:Copy", versioned=True)
+ s3.put_object(Bucket=bucket, Key="src.txt", Body=b"x")
+ copied = s3.copy_object(Bucket=bucket, Key="dst.txt", CopySource={"Bucket": bucket, "Key": "src.txt"})
+ [record] = _event_records(sqs, url, 1)
+ assert record["eventName"] == "ObjectCreated:Copy"
+ assert record["s3"]["object"]["versionId"] == copied["VersionId"]
+
+
+def test_multipart_record_names_the_new_version(s3, sqs, notified):
+ bucket, url, configure = notified
+ configure("s3:ObjectCreated:CompleteMultipartUpload", versioned=True)
+ upload = s3.create_multipart_upload(Bucket=bucket, Key="big.bin")["UploadId"]
+ part = s3.upload_part(Bucket=bucket, Key="big.bin", UploadId=upload, PartNumber=1, Body=b"x" * 16)
+ done = s3.complete_multipart_upload(
+ Bucket=bucket,
+ Key="big.bin",
+ UploadId=upload,
+ MultipartUpload={"Parts": [{"PartNumber": 1, "ETag": part["ETag"]}]},
+ )
+ [record] = _event_records(sqs, url, 1)
+ assert record["s3"]["object"]["versionId"] == done["VersionId"]
+
+
+def test_delete_objects_sends_an_event_per_object(s3, sqs, notified):
+ bucket, url, configure = notified
+ configure("s3:ObjectRemoved:*", versioned=True)
+ for k in ("a.txt", "b.txt"):
+ s3.put_object(Bucket=bucket, Key=k, Body=b"x")
+ version = s3.put_object(Bucket=bucket, Key="c.txt", Body=b"x")["VersionId"]
+ s3.delete_objects(
+ Bucket=bucket,
+ Delete={"Objects": [{"Key": "a.txt"}, {"Key": "b.txt"}, {"Key": "c.txt", "VersionId": version}]},
+ )
+ records = _event_records(sqs, url, 3)
+ got = sorted((r["s3"]["object"]["key"], r["eventName"]) for r in records)
+ assert got == [
+ ("a.txt", "ObjectRemoved:DeleteMarkerCreated"),
+ ("b.txt", "ObjectRemoved:DeleteMarkerCreated"),
+ ("c.txt", "ObjectRemoved:Delete"),
+ ]
+
+
+def test_delete_objects_in_an_unversioned_bucket(s3, sqs, notified):
+ bucket, url, configure = notified
+ configure("s3:ObjectRemoved:*")
+ s3.put_object(Bucket=bucket, Key="u.txt", Body=b"x")
+ s3.delete_objects(Bucket=bucket, Delete={"Objects": [{"Key": "u.txt"}, {"Key": "never-there.txt"}]})
+ [record] = _event_records(sqs, url, 1)
+ assert (record["s3"]["object"]["key"], record["eventName"]) == ("u.txt", "ObjectRemoved:Delete")
diff --git a/tests/test_s3_annotations.py b/tests/test_s3_annotations.py
deleted file mode 100644
index 943324e4..00000000
--- a/tests/test_s3_annotations.py
+++ /dev/null
@@ -1,659 +0,0 @@
-"""S3 object annotations: PutObjectAnnotation, GetObjectAnnotation,
-ListObjectAnnotations and DeleteObjectAnnotation, and how annotations follow
-object versions, CopyObject, deletes, replication, encryption, Object Lock,
-restarts and event notifications.
-
-Evidence: the S3 API Reference pages for the four operations, the S3 User
-Guide's "Annotating your objects", "Event notification types and
-destinations" and "Event message structure" pages, and the botocore s3 model.
-Not validated against a real AWS account."""
-
-import base64
-import hashlib
-import json
-import os
-import struct
-import time
-import uuid
-
-import pytest
-from botocore.exceptions import ClientError
-from conftest import make_client
-
-
-def _bucket(s3, versioned=False, **create):
- name = f"annot-{uuid.uuid4().hex[:12]}"
- s3.create_bucket(Bucket=name, **create)
- if versioned:
- s3.put_bucket_versioning(Bucket=name, VersioningConfiguration={"Status": "Enabled"})
- return name
-
-
-def _code(exc):
- return exc.value.response["Error"]["Code"]
-
-
-def _status(exc):
- return exc.value.response["ResponseMetadata"]["HTTPStatusCode"]
-
-
-def _crc64nvme_bitwise(data: bytes) -> int:
- """Bit-at-a-time CRC-64/NVME, independent of the server's table."""
- crc = 0xFFFFFFFFFFFFFFFF
- for byte in data:
- crc ^= byte
- for _ in range(8):
- crc = (crc >> 1) ^ 0x9A6C9329AC4BC9B5 if crc & 1 else crc >> 1
- return crc ^ 0xFFFFFFFFFFFFFFFF
-
-
-# ── Put / Get ──────────────────────────────────────────────────────────
-
-
-def test_put_and_get_an_annotation(s3):
- bucket = _bucket(s3, versioned=True)
- payload = b'{"archive":"Images"}'
- put = s3.put_object(Bucket=bucket, Key="photo.jpg", Body=b"jpeg")
- out = s3.put_object_annotation(Bucket=bucket, Key="photo.jpg", AnnotationName="nexus.system", AnnotationPayload=payload)
- assert out["ObjectVersionId"] == put["VersionId"]
- assert out["ETag"] == f'"{hashlib.md5(payload).hexdigest()}"'
- got = s3.get_object_annotation(Bucket=bucket, Key="photo.jpg", AnnotationName="nexus.system")
- assert got["AnnotationPayload"].read() == payload
- assert got["ObjectVersionId"] == put["VersionId"]
- assert got["ContentLength"] == len(payload)
- assert got["ServerSideEncryption"] == "AES256"
-
-
-def test_an_annotation_leaves_the_object_alone(s3):
- """"Adding, updating, or removing an annotation does not modify the parent
- object's ETag", and makes no new version."""
- bucket = _bucket(s3, versioned=True)
- put = s3.put_object(Bucket=bucket, Key="k", Body=b"body")
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- head = s3.head_object(Bucket=bucket, Key="k")
- assert head["ETag"] == put["ETag"] and head["VersionId"] == put["VersionId"]
- assert len(s3.list_object_versions(Bucket=bucket)["Versions"]) == 1
- assert s3.get_object(Bucket=bucket, Key="k")["Body"].read() == b"body"
-
-
-def test_putting_again_replaces_the_payload(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"first")
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"second")
- got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
- assert got["AnnotationPayload"].read() == b"second"
-
-
-def test_get_of_a_missing_annotation(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- with pytest.raises(ClientError) as exc:
- s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="absent")
- assert (_code(exc), _status(exc)) == ("NoSuchAnnotation", 404)
-
-
-def test_annotation_on_a_missing_key_or_bucket(s3):
- bucket = _bucket(s3)
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(Bucket=bucket, Key="nope", AnnotationName="a", AnnotationPayload=b"1")
- assert _code(exc) == "NoSuchKey"
- with pytest.raises(ClientError) as exc:
- s3.get_object_annotation(Bucket=f"missing-{uuid.uuid4().hex[:8]}", Key="k", AnnotationName="a")
- assert _code(exc) == "NoSuchBucket"
-
-
- "name, code",
- [
- ("aws.reserved", "InvalidAnnotationName"),
- ("AWSthing", "InvalidAnnotationName"),
- ("s3-thing", "InvalidAnnotationName"),
- ("S3", "InvalidAnnotationName"),
- ("has space", "InvalidAnnotationName"),
- ("slash/name", "InvalidAnnotationName"),
- (" ", "InvalidAnnotationName"),
- ("n" * 513, "AnnotationNameTooLong"),
- ],
-)
-def test_annotation_naming_rules(s3, name, code):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=name, AnnotationPayload=b"1")
- assert (_code(exc), _status(exc)) == (code, 400)
-
-
-def test_names_may_use_letters_of_any_language(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- for name in ("données_1.2-3", "名前", "n" * 512):
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=name, AnnotationPayload=b"1")
-
-
-def test_payload_limits(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="max", AnnotationPayload=b"a" * (1024 * 1024))
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="big", AnnotationPayload=b"a" * (1024 * 1024 + 1))
- assert _code(exc) == "InvalidRequest"
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="empty", AnnotationPayload=b"")
- assert _code(exc) == "InvalidRequest"
-
-
-def test_payload_must_be_utf8(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="bin", AnnotationPayload=b"\xff\xfe")
- assert (_code(exc), _status(exc)) == ("UnsupportedMediaType", 415)
-
-
-def test_a_version_holds_at_most_1000_annotations(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- for i in range(1000):
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=f"n{i}", AnnotationPayload=b"1")
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="n1000", AnnotationPayload=b"1")
- assert _code(exc) == "AnnotationLimitExceeded"
- # Replacing one of the thousand is not a new one.
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="n0", AnnotationPayload=b"2")
-
-
-def test_object_if_match(s3):
- bucket = _bucket(s3)
- etag = s3.put_object(Bucket=bucket, Key="k", Body=b"x")["ETag"]
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1", ObjectIfMatch=etag)
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(
- Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"2", ObjectIfMatch='"deadbeef"'
- )
- assert _status(exc) == 412
- with pytest.raises(ClientError) as exc:
- s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", ObjectIfMatch='"deadbeef"')
- assert _status(exc) == 412
- s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", ObjectIfMatch=etag)
-
-
-# ── Checksums ──────────────────────────────────────────────────────────
-
-
-def test_default_checksum_is_crc64nvme(s3):
- """"If the annotation doesn't have a specified checksum algorithm or
- checksum value, Amazon S3 uses the CRC-64/NVME algorithm"."""
- plain = make_client("s3", {"request_checksum_calculation": "when_required"})
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- out = plain.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"123456789")
- expected = base64.b64encode(struct.pack(">Q", _crc64nvme_bitwise(b"123456789"))).decode()
- assert out["ChecksumCRC64NVME"] == expected
- got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", ChecksumMode="ENABLED")
- assert got["ChecksumCRC64NVME"] == expected
- # Only when asked: a client that does not send x-amz-checksum-mode gets none back.
- unasked = make_client("s3", {"response_checksum_validation": "when_required"})
- assert "ChecksumCRC64NVME" not in unasked.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
-
-
- "member, digest",
- [
- ("ChecksumSHA256", lambda b: hashlib.sha256(b).digest()),
- ("ChecksumSHA512", lambda b: hashlib.sha512(b).digest()),
- ("ChecksumMD5", lambda b: hashlib.md5(b).digest()),
- # XXH64's published check value for b"abc" (the xxHash reference).
- ("ChecksumXXHASH64", lambda b: struct.pack(">Q", 0x44BC2CF5AD770999)),
- ],
-)
-def test_a_checksum_sent_is_verified(s3, member, digest):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- good = base64.b64encode(digest(b"abc")).decode()
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"abc", **{member: good})
- got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", ChecksumMode="ENABLED")
- assert got[member] == good and got["ChecksumType"] == "FULL_OBJECT"
- bad = base64.b64encode(bytes(x ^ 0xFF for x in digest(b"abc"))).decode()
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"abc", **{member: bad})
- assert _code(exc) == "BadDigest"
-
-
-def test_xxh64_check_values():
- from ministack.services import s3 as s3mod
-
- assert s3mod._xxh64(b"") == 0xEF46DB3751D8E999
- assert s3mod._xxh64(b"abc") == 0x44BC2CF5AD770999
- # Past the 32-byte stripe loop, against the reference implementation's output.
- assert s3mod._xxh64(b"Nobody inspects the spammish repetition") == 0xFBCEA83C8A378BF1
-
-
-def test_an_algorithm_ministack_cannot_compute_is_refused(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(
- Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"abc", ChecksumXXHASH3="AAAAAAAAAAA="
- )
- assert _code(exc) == "InvalidRequest"
-
-
-# ── Encryption ─────────────────────────────────────────────────────────
-
-
-def test_annotations_take_the_objects_encryption(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x", ServerSideEncryption="aws:kms", SSEKMSKeyId="alias/annot")
- out = s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- assert out["ServerSideEncryption"] == "aws:kms"
- got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
- assert got["ServerSideEncryption"] == "aws:kms"
-
-
-def test_sse_c_objects_cannot_have_annotations(s3):
- bucket = _bucket(s3)
- key = os.urandom(32)
- s3.put_object(
- Bucket=bucket,
- Key="k",
- Body=b"x",
- SSECustomerAlgorithm="AES256",
- SSECustomerKey=key,
- )
- with pytest.raises(ClientError) as exc:
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- assert _code(exc) == "InvalidRequest"
-
-
-# ── Versions ───────────────────────────────────────────────────────────
-
-
-def test_annotations_belong_to_one_version(s3):
- bucket = _bucket(s3, versioned=True)
- v1 = s3.put_object(Bucket=bucket, Key="k", Body=b"one")["VersionId"]
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"on v1")
- v2 = s3.put_object(Bucket=bucket, Key="k", Body=b"two")["VersionId"]
- # "Creating a new version does not copy annotations from the previous version."
- with pytest.raises(ClientError) as exc:
- s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
- assert _code(exc) == "NoSuchAnnotation"
- got = s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v1)
- assert got["AnnotationPayload"].read() == b"on v1" and got["ObjectVersionId"] == v1
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"on v2")
- assert s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v2)["AnnotationPayload"].read() == b"on v2"
- assert s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v1)["AnnotationPayload"].read() == b"on v1"
-
-
-def test_a_delete_marker_keeps_the_versions_annotations(s3):
- bucket = _bucket(s3, versioned=True)
- v1 = s3.put_object(Bucket=bucket, Key="k", Body=b"one")["VersionId"]
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"kept")
- marker = s3.delete_object(Bucket=bucket, Key="k")["VersionId"]
- with pytest.raises(ClientError) as exc:
- s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
- assert _code(exc) == "NoSuchKey"
- with pytest.raises(ClientError) as exc:
- s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=marker)
- assert _status(exc) == 405
- assert s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v1)["AnnotationPayload"].read() == b"kept"
- # Removing the marker brings the version back with its annotations.
- s3.delete_object(Bucket=bucket, Key="k", VersionId=marker)
- assert s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")["AnnotationPayload"].read() == b"kept"
-
-
-def test_deleting_a_version_deletes_its_annotations(s3):
- bucket = _bucket(s3, versioned=True)
- v1 = s3.put_object(Bucket=bucket, Key="k", Body=b"one")["VersionId"]
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- s3.put_object(Bucket=bucket, Key="k", Body=b"two")
- s3.delete_object(Bucket=bucket, Key="k", VersionId=v1)
- with pytest.raises(ClientError) as exc:
- s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", VersionId=v1)
- assert _code(exc) == "NoSuchVersion"
-
-
-def test_unversioned_overwrite_and_delete_drop_annotations(s3):
- """"In a non-versioned bucket, if you delete or overwrite the object, the
- annotations are deleted with it.\""""
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"one")
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- s3.put_object(Bucket=bucket, Key="k", Body=b"two")
- with pytest.raises(ClientError) as exc:
- s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
- assert _code(exc) == "NoSuchAnnotation"
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="b", AnnotationPayload=b"1")
- s3.delete_object(Bucket=bucket, Key="k")
- s3.put_object(Bucket=bucket, Key="k", Body=b"three")
- assert s3.list_object_annotations(Bucket=bucket, Key="k")["Annotations"] == []
-
-
-def test_batch_delete_drops_annotations(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"one")
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- s3.delete_objects(Bucket=bucket, Delete={"Objects": [{"Key": "k"}]})
- s3.put_object(Bucket=bucket, Key="k", Body=b"two")
- assert s3.list_object_annotations(Bucket=bucket, Key="k")["Annotations"] == []
-
-
-def test_annotations_need_no_restore_for_archived_objects(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="cold", Body=b"x", StorageClass="GLACIER")
- s3.put_object_annotation(Bucket=bucket, Key="cold", AnnotationName="a", AnnotationPayload=b"1")
- assert s3.get_object_annotation(Bucket=bucket, Key="cold", AnnotationName="a")["AnnotationPayload"].read() == b"1"
-
-
-# ── List ───────────────────────────────────────────────────────────────
-
-
-def test_list_annotations(s3):
- bucket = _bucket(s3, versioned=True)
- version = s3.put_object(Bucket=bucket, Key="k", Body=b"x")["VersionId"]
- for name, payload in (("ml.label", b"cat"), ("etl.status", b"done"), ("ml.score", b"0.97")):
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=name, AnnotationPayload=payload, ChecksumAlgorithm="SHA256")
- out = s3.list_object_annotations(Bucket=bucket, Key="k")
- assert [a["AnnotationName"] for a in out["Annotations"]] == ["etl.status", "ml.label", "ml.score"]
- assert out["AnnotationCount"] == 3 and out["ObjectVersionId"] == version
- assert out["Bucket"] == bucket and out["Key"] == "k"
- entry = out["Annotations"][1]
- assert entry["Size"] == 3 and entry["ETag"] == f'"{hashlib.md5(b"cat").hexdigest()}"'
- assert entry["ChecksumAlgorithm"] == ["SHA256"] and "LastModified" in entry
- assert "NextContinuationToken" not in out
-
-
-def test_list_annotations_by_prefix_and_page(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- for i in range(5):
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName=f"page.{i}", AnnotationPayload=b"1")
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="other", AnnotationPayload=b"1")
- first = s3.list_object_annotations(Bucket=bucket, Key="k", AnnotationPrefix="page.", MaxAnnotationResults=2)
- assert [a["AnnotationName"] for a in first["Annotations"]] == ["page.0", "page.1"]
- assert first["AnnotationPrefix"] == "page." and first["MaxAnnotationResults"] == 2
- token = first["NextContinuationToken"]
- rest = s3.list_object_annotations(Bucket=bucket, Key="k", AnnotationPrefix="page.", ContinuationToken=token)
- assert [a["AnnotationName"] for a in rest["Annotations"]] == ["page.2", "page.3", "page.4"]
- assert rest["ContinuationToken"] == token and "NextContinuationToken" not in rest
-
-
-def test_list_rejects_a_bad_page_size_or_prefix(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- plain = make_client("s3", {"parameter_validation": False})
- with pytest.raises(ClientError) as exc:
- plain.list_object_annotations(Bucket=bucket, Key="k", MaxAnnotationResults=1001)
- assert _code(exc) == "InvalidArgument"
- with pytest.raises(ClientError) as exc:
- s3.list_object_annotations(Bucket=bucket, Key="k", AnnotationPrefix="p" * 513)
- assert _code(exc) == "InvalidPrefix"
-
-
-# ── Delete ─────────────────────────────────────────────────────────────
-
-
-def test_delete_an_annotation(s3):
- bucket = _bucket(s3, versioned=True)
- version = s3.put_object(Bucket=bucket, Key="k", Body=b"x")["VersionId"]
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- out = s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
- assert out["ResponseMetadata"]["HTTPStatusCode"] == 204 and out["ObjectVersionId"] == version
- with pytest.raises(ClientError) as exc:
- s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
- assert _code(exc) == "NoSuchAnnotation"
- # Deleting one that is not there is not an error.
- s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
-
-
-# Observed on AWS (ap-southeast-2, 2026-09-28): annotations are writes to the object version, so Object Lock guards
-# PutObjectAnnotation and DeleteObjectAnnotation alike. Governance retention yields to
-# x-amz-bypass-governance-retention; compliance retention and a legal hold refuse both, bypass or not.
-
-
-def _bypassing(operation):
- raw = make_client("s3")
-
- def bypass(request, **_):
- request.headers["x-amz-bypass-governance-retention"] = "true"
-
- raw.meta.events.register(f"before-sign.s3.{operation}", bypass)
- return raw
-
-
-def _refused(call, message):
- with pytest.raises(ClientError) as exc:
- call()
- assert _code(exc) == "AccessDenied"
- assert exc.value.response["ResponseMetadata"]["HTTPStatusCode"] == 403
- assert exc.value.response["Error"]["Message"] == message
-
-
-def _locked(s3, **lock):
- bucket = _bucket(s3, ObjectLockEnabledForBucket=True)
- s3.put_object(Bucket=bucket, Key="k", Body=b"x", **lock)
- return bucket
-
-
-def _until():
- return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() + 3600))
-
-
-RETENTION = "Access Denied because object protected by object lock retention."
-LEGAL_HOLD = "Access Denied because object protected by object lock legal hold."
-
-
-def test_governance_retention_needs_the_bypass_to_put_or_delete_an_annotation(s3):
- bucket = _locked(s3, ObjectLockMode="GOVERNANCE", ObjectLockRetainUntilDate=_until())
- _refused(lambda: s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1"), RETENTION)
- _bypassing("PutObjectAnnotation").put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- _refused(lambda: s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"2"), RETENTION)
- _refused(lambda: s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a"), RETENTION)
- _bypassing("DeleteObjectAnnotation").delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
-
-
-def test_compliance_retention_refuses_annotation_writes_even_with_the_bypass(s3):
- bucket = _locked(s3, ObjectLockMode="COMPLIANCE", ObjectLockRetainUntilDate=_until())
- def put(c):
- return c.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
-
- def delete(c):
- return c.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
-
- _refused(lambda: put(s3), RETENTION)
- _refused(lambda: put(_bypassing("PutObjectAnnotation")), RETENTION)
- _refused(lambda: delete(s3), RETENTION)
- _refused(lambda: delete(_bypassing("DeleteObjectAnnotation")), RETENTION)
-
-
-def test_a_legal_hold_refuses_annotation_writes_even_with_the_bypass(s3):
- bucket = _locked(s3, ObjectLockLegalHoldStatus="ON")
- def put(c):
- return c.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
-
- def delete(c):
- return c.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
-
- _refused(lambda: put(s3), LEGAL_HOLD)
- _refused(lambda: put(_bypassing("PutObjectAnnotation")), LEGAL_HOLD)
- _refused(lambda: delete(s3), LEGAL_HOLD)
- _refused(lambda: delete(_bypassing("DeleteObjectAnnotation")), LEGAL_HOLD)
-
-
-def test_an_unlocked_object_in_a_lock_enabled_bucket_takes_annotations_freely(s3):
- bucket = _locked(s3)
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"2")
- s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
-
-
-# ── CopyObject ─────────────────────────────────────────────────────────
-
-
-def test_copy_carries_annotations_by_default(s3):
- src = _bucket(s3, versioned=True)
- dst = _bucket(s3, versioned=True)
- s3.put_object(Bucket=src, Key="landing/x.jpg", Body=b"x")
- s3.put_object_annotation(Bucket=src, Key="landing/x.jpg", AnnotationName="nexus.system", AnnotationPayload=b"{}")
- copied = s3.copy_object(Bucket=dst, Key="assets/x.jpg", CopySource={"Bucket": src, "Key": "landing/x.jpg"})
- got = s3.get_object_annotation(Bucket=dst, Key="assets/x.jpg", AnnotationName="nexus.system")
- assert got["AnnotationPayload"].read() == b"{}" and got["ObjectVersionId"] == copied["VersionId"]
-
-
-def test_copy_of_a_named_version_carries_that_versions_annotations(s3):
- bucket = _bucket(s3, versioned=True)
- v1 = s3.put_object(Bucket=bucket, Key="k", Body=b"one")["VersionId"]
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"v1")
- s3.put_object(Bucket=bucket, Key="k", Body=b"two")
- s3.copy_object(Bucket=bucket, Key="copy", CopySource={"Bucket": bucket, "Key": "k", "VersionId": v1})
- assert s3.get_object_annotation(Bucket=bucket, Key="copy", AnnotationName="a")["AnnotationPayload"].read() == b"v1"
-
-
-def test_copy_with_exclude_leaves_them_behind(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="src", Body=b"x")
- s3.put_object_annotation(Bucket=bucket, Key="src", AnnotationName="a", AnnotationPayload=b"1")
- s3.copy_object(Bucket=bucket, Key="dst", CopySource={"Bucket": bucket, "Key": "src"}, AnnotationDirective="EXCLUDE")
- assert s3.list_object_annotations(Bucket=bucket, Key="dst")["Annotations"] == []
-
-
-def test_copy_rejects_an_unknown_directive(s3):
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="src", Body=b"x")
- plain = make_client("s3", {"parameter_validation": False})
- with pytest.raises(ClientError) as exc:
- plain.copy_object(Bucket=bucket, Key="dst", CopySource={"Bucket": bucket, "Key": "src"}, AnnotationDirective="REPLACE")
- assert _code(exc) == "InvalidArgument"
-
-
-def test_copy_with_a_new_algorithm_rechecksums_the_annotations(s3):
- """"If you specify a different checksum algorithm in the copy request, the
- new algorithm applies to both the object and its annotations.\""""
- bucket = _bucket(s3)
- s3.put_object(Bucket=bucket, Key="src", Body=b"x")
- s3.put_object_annotation(Bucket=bucket, Key="src", AnnotationName="a", AnnotationPayload=b"abc")
- s3.copy_object(Bucket=bucket, Key="dst", CopySource={"Bucket": bucket, "Key": "src"}, ChecksumAlgorithm="SHA256")
- got = s3.get_object_annotation(Bucket=bucket, Key="dst", AnnotationName="a", ChecksumMode="ENABLED")
- assert got["ChecksumSHA256"] == base64.b64encode(hashlib.sha256(b"abc").digest()).decode()
-
-
-# ── Replication ────────────────────────────────────────────────────────
-
-
-def test_annotations_replicate(s3):
- src = _bucket(s3, versioned=True)
- dst = _bucket(s3, versioned=True)
- s3.put_bucket_replication(
- Bucket=src,
- ReplicationConfiguration={
- "Role": "arn:aws:iam::000000000000:role/repl",
- "Rules": [{"ID": "r1", "Status": "Enabled", "Prefix": "", "Destination": {"Bucket": f"arn:aws:s3:::{dst}"}}],
- },
- )
- s3.put_object(Bucket=src, Key="k", Body=b"x")
- s3.put_object_annotation(Bucket=src, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- assert s3.get_object_annotation(Bucket=src, Key="k", AnnotationName="a")["ReplicationStatus"] == "COMPLETED"
- replica = s3.get_object_annotation(Bucket=dst, Key="k", AnnotationName="a")
- assert replica["AnnotationPayload"].read() == b"1" and replica["ReplicationStatus"] == "REPLICA"
- listed = s3.list_object_annotations(Bucket=dst, Key="k")["Annotations"]
- assert [(a["AnnotationName"], a["ReplicationStatus"]) for a in listed] == [("a", "REPLICA")]
-
-
-# ── Event notifications ────────────────────────────────────────────────
-
-
-def _notified(s3, sqs, events, versioned=True):
- bucket = _bucket(s3, versioned=versioned)
- url = sqs.create_queue(QueueName=bucket)["QueueUrl"]
- arn = sqs.get_queue_attributes(QueueUrl=url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"]
- s3.put_bucket_notification_configuration(
- Bucket=bucket, NotificationConfiguration={"QueueConfigurations": [{"QueueArn": arn, "Events": events}]}
- )
- return bucket, url
-
-
-def _records(sqs, url, expected, timeout=10):
- records = []
- deadline = time.time() + timeout
- while len(records) < expected and time.time() < deadline:
- for m in sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=10, WaitTimeSeconds=1).get("Messages", []):
- records.extend(json.loads(m["Body"]).get("Records", []))
- sqs.delete_message(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"])
- return records
-
-
-def test_annotation_events(s3, sqs):
- bucket, url = _notified(s3, sqs, ["s3:ObjectAnnotation:*", "s3:ObjectCreated:*"])
- version = s3.put_object(Bucket=bucket, Key="k", Body=b"x")["VersionId"]
- assert [r["eventName"] for r in _records(sqs, url, 1)] == ["ObjectCreated:Put"]
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"abc")
- s3.delete_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
- records = sorted(_records(sqs, url, 2), key=lambda r: r["eventName"], reverse=True)
- # "PutObjectAnnotation and DeleteObjectAnnotation send only annotation-specific events."
- assert [r["eventName"] for r in records] == ["ObjectAnnotation:Put", "ObjectAnnotation:Delete"]
- put, delete = records
- assert put["objectAnnotation"] == [{"name": "a", "size": 3, "eTag": hashlib.md5(b"abc").hexdigest()}]
- assert delete["objectAnnotation"] == [{"name": "a"}]
- assert put["s3"]["object"]["versionId"] == version
- assert "sequencer" not in put["s3"]["object"]
-
-
-def test_get_and_list_send_no_events(s3, sqs):
- bucket, url = _notified(s3, sqs, ["s3:ObjectAnnotation:*"])
- s3.put_object(Bucket=bucket, Key="k", Body=b"x")
- s3.put_object_annotation(Bucket=bucket, Key="k", AnnotationName="a", AnnotationPayload=b"1")
- assert len(_records(sqs, url, 1)) == 1
- s3.get_object_annotation(Bucket=bucket, Key="k", AnnotationName="a")
- s3.list_object_annotations(Bucket=bucket, Key="k")
- assert _records(sqs, url, 1, timeout=2) == []
-
-
-def test_copy_event_says_whether_annotations_came(s3, sqs):
- bucket, url = _notified(s3, sqs, ["s3:ObjectCreated:Copy"])
- s3.put_object(Bucket=bucket, Key="src", Body=b"x")
- s3.put_object_annotation(Bucket=bucket, Key="src", AnnotationName="a", AnnotationPayload=b"1")
- s3.copy_object(Bucket=bucket, Key="with", CopySource={"Bucket": bucket, "Key": "src"})
- s3.copy_object(Bucket=bucket, Key="without", CopySource={"Bucket": bucket, "Key": "src"}, AnnotationDirective="EXCLUDE")
- flags = {r["s3"]["object"]["key"]: r["s3"]["object"]["hasObjectAnnotation"] for r in _records(sqs, url, 2)}
- assert flags == {"with": True, "without": False}
-
-
-# ── Persistence ────────────────────────────────────────────────────────
-
-
-def s3_persist(tmp_path, monkeypatch):
- """S3 in this process with persistence on, over a throwaway DATA_DIR."""
- from ministack.core import responses as respmod
- from ministack.services import s3 as s3mod
-
- monkeypatch.setattr(s3mod, "DATA_DIR", str(tmp_path))
- monkeypatch.setattr(s3mod, "S3_PERSIST", True)
- monkeypatch.setattr(s3mod, "get_account_id", lambda: "000000000000")
- monkeypatch.setattr(respmod, "get_account_id", lambda: "000000000000")
- stores = (s3mod._buckets._data, s3mod._object_versions._data, s3mod._bucket_versioning._data, s3mod._object_annotations._data)
- before = [set(store) for store in stores]
- yield s3mod
- for store, keys in zip(stores, before):
- for key in set(store) - keys:
- store.pop(key, None)
-
-
-def test_annotations_survive_a_restart(s3_persist, tmp_path):
- s3mod = s3_persist
- bucket = "qa-annot-persist"
- s3mod._create_bucket(bucket, b"")
- s3mod._bucket_versioning[bucket] = "Enabled"
- v1 = s3mod._put_object(bucket, "k", b"one", {})[1]["x-amz-version-id"]
- s3mod._put_object_annotation(bucket, "k", b"on v1", {}, {"annotationName": ["a"]})
- v2 = s3mod._put_object(bucket, "k", b"two", {})[1]["x-amz-version-id"]
- s3mod._put_object_annotation(bucket, "k", b"on v2", {}, {"annotationName": ["a"]})
-
- s3mod._buckets._data.pop(("000000000000", bucket), None)
- for store in (s3mod._object_versions._data, s3mod._object_annotations._data):
- for k in [k for k in store if k[0] == "000000000000" and k[1][0] == bucket]:
- store.pop(k, None)
- s3mod._load_persisted_bucket("000000000000", bucket, os.path.join(str(tmp_path), "000000000000", bucket))
-
- assert s3mod._object_annotations[(bucket, "k", v1)]["a"]["payload"] == "on v1"
- assert s3mod._object_annotations[(bucket, "k", v2)]["a"]["payload"] == "on v2"
diff --git a/tests/test_s3_event_records.py b/tests/test_s3_event_records.py
deleted file mode 100644
index 16faa583..00000000
--- a/tests/test_s3_event_records.py
+++ /dev/null
@@ -1,170 +0,0 @@
-"""The S3 notification record's object block, per the S3 User Guide's
-"Event message structure" and "Event notification types and destinations"
-pages: a URL-encoded key, the version the write made or removed, a sequencer
-that grows with each create and delete, eventVersion 2.6, a delete marker
-reported as ObjectRemoved:DeleteMarkerCreated, and one event per object a
-DeleteObjects removes. Evidence: official AWS documentation, not a run against
-a real AWS account."""
-
-import json
-import time
-import uuid
-
-import pytest
-
-
-def _queue(sqs, name):
- url = sqs.create_queue(QueueName=name)["QueueUrl"]
- arn = sqs.get_queue_attributes(QueueUrl=url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"]
- return url, arn
-
-
-def _records(sqs, url, expected, timeout=10):
- """Every S3 record delivered to the queue, waiting until `expected` have arrived."""
- records = []
- deadline = time.time() + timeout
- while len(records) < expected and time.time() < deadline:
- resp = sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=10, WaitTimeSeconds=1)
- for m in resp.get("Messages", []):
- body = json.loads(m["Body"])
- records.extend(body.get("Records", []))
- sqs.delete_message(QueueUrl=url, ReceiptHandle=m["ReceiptHandle"])
- return records
-
-
-def notified(s3, sqs):
- """A bucket whose object events go to a fresh queue: (bucket, queue url, configure)."""
- suffix = uuid.uuid4().hex[:8]
- bucket = f"evt-records-{suffix}"
- s3.create_bucket(Bucket=bucket)
- url, arn = _queue(sqs, f"evt-records-{suffix}")
-
- def configure(*events, versioned=False):
- if versioned:
- s3.put_bucket_versioning(Bucket=bucket, VersioningConfiguration={"Status": "Enabled"})
- s3.put_bucket_notification_configuration(
- Bucket=bucket,
- NotificationConfiguration={"QueueConfigurations": [{"QueueArn": arn, "Events": list(events)}]},
- )
- # The configuration's own s3:TestEvent carries no Records; drain it.
- _records(sqs, url, 0, timeout=1)
-
- return bucket, url, configure
-
-
-def test_record_key_is_url_encoded(s3, sqs, notified):
- bucket, url, configure = notified
- configure("s3:ObjectCreated:*")
- s3.put_object(Bucket=bucket, Key="photos/red flower+(1).jpg", Body=b"x")
- [record] = _records(sqs, url, 1)
- # "`red flower.jpg` becomes `red+flower.jpg`"; slashes stay, a literal + is escaped.
- assert record["s3"]["object"]["key"] == "photos/red+flower%2B%281%29.jpg"
- assert record["eventVersion"] == "2.6"
-
-
-def test_record_carries_the_version_the_put_made(s3, sqs, notified):
- bucket, url, configure = notified
- configure("s3:ObjectCreated:*", versioned=True)
- first = s3.put_object(Bucket=bucket, Key="doc.txt", Body=b"one")["VersionId"]
- second = s3.put_object(Bucket=bucket, Key="doc.txt", Body=b"two")["VersionId"]
- records = _records(sqs, url, 2)
- assert sorted(r["s3"]["object"]["versionId"] for r in records) == sorted([first, second])
-
-
-def test_record_has_no_version_in_an_unversioned_bucket(s3, sqs, notified):
- bucket, url, configure = notified
- configure("s3:ObjectCreated:*")
- s3.put_object(Bucket=bucket, Key="plain.txt", Body=b"x")
- [record] = _records(sqs, url, 1)
- assert "versionId" not in record["s3"]["object"]
-
-
-def test_sequencer_grows_with_each_write(s3, sqs, notified):
- bucket, url, configure = notified
- configure("s3:ObjectCreated:*", "s3:ObjectRemoved:*", versioned=True)
- v1 = s3.put_object(Bucket=bucket, Key="seq.txt", Body=b"1")["VersionId"]
- v2 = s3.put_object(Bucket=bucket, Key="seq.txt", Body=b"2")["VersionId"]
- marker = s3.delete_object(Bucket=bucket, Key="seq.txt")["VersionId"]
- by_version = {r["s3"]["object"]["versionId"]: r["s3"]["object"]["sequencer"] for r in _records(sqs, url, 3)}
- # Compared as S3 says to: left-pad the shorter with zeros, then compare.
- width = max(len(v) for v in by_version.values())
- ordered = [by_version[v].rjust(width, "0") for v in (v1, v2, marker)]
- assert ordered == sorted(ordered) and len(set(ordered)) == 3
- assert all(int(s, 16) >= 0 for s in ordered)
-
-
-def test_delete_without_version_reports_the_marker(s3, sqs, notified):
- bucket, url, configure = notified
- configure("s3:ObjectRemoved:*", versioned=True)
- s3.put_object(Bucket=bucket, Key="gone.txt", Body=b"x")
- marker = s3.delete_object(Bucket=bucket, Key="gone.txt")["VersionId"]
- [record] = _records(sqs, url, 1)
- assert record["eventName"] == "ObjectRemoved:DeleteMarkerCreated"
- assert record["s3"]["object"]["versionId"] == marker
-
-
-def test_marker_is_not_an_object_removed_delete(s3, sqs, notified):
- bucket, url, configure = notified
- # Subscribed to permanent deletes only: a marker is its own event type.
- configure("s3:ObjectRemoved:Delete", versioned=True)
- version = s3.put_object(Bucket=bucket, Key="kept.txt", Body=b"x")["VersionId"]
- s3.delete_object(Bucket=bucket, Key="kept.txt")
- s3.delete_object(Bucket=bucket, Key="kept.txt", VersionId=version)
- [record] = _records(sqs, url, 1)
- assert record["eventName"] == "ObjectRemoved:Delete"
- assert record["s3"]["object"]["versionId"] == version
- assert _records(sqs, url, 1, timeout=2) == []
-
-
-def test_copy_record_names_the_new_version(s3, sqs, notified):
- bucket, url, configure = notified
- configure("s3:ObjectCreated:Copy", versioned=True)
- s3.put_object(Bucket=bucket, Key="src.txt", Body=b"x")
- copied = s3.copy_object(Bucket=bucket, Key="dst.txt", CopySource={"Bucket": bucket, "Key": "src.txt"})
- [record] = _records(sqs, url, 1)
- assert record["eventName"] == "ObjectCreated:Copy"
- assert record["s3"]["object"]["versionId"] == copied["VersionId"]
-
-
-def test_multipart_record_names_the_new_version(s3, sqs, notified):
- bucket, url, configure = notified
- configure("s3:ObjectCreated:CompleteMultipartUpload", versioned=True)
- upload = s3.create_multipart_upload(Bucket=bucket, Key="big.bin")["UploadId"]
- part = s3.upload_part(Bucket=bucket, Key="big.bin", UploadId=upload, PartNumber=1, Body=b"x" * 16)
- done = s3.complete_multipart_upload(
- Bucket=bucket,
- Key="big.bin",
- UploadId=upload,
- MultipartUpload={"Parts": [{"PartNumber": 1, "ETag": part["ETag"]}]},
- )
- [record] = _records(sqs, url, 1)
- assert record["s3"]["object"]["versionId"] == done["VersionId"]
-
-
-def test_delete_objects_sends_an_event_per_object(s3, sqs, notified):
- bucket, url, configure = notified
- configure("s3:ObjectRemoved:*", versioned=True)
- for k in ("a.txt", "b.txt"):
- s3.put_object(Bucket=bucket, Key=k, Body=b"x")
- version = s3.put_object(Bucket=bucket, Key="c.txt", Body=b"x")["VersionId"]
- s3.delete_objects(
- Bucket=bucket,
- Delete={"Objects": [{"Key": "a.txt"}, {"Key": "b.txt"}, {"Key": "c.txt", "VersionId": version}]},
- )
- records = _records(sqs, url, 3)
- got = sorted((r["s3"]["object"]["key"], r["eventName"]) for r in records)
- assert got == [
- ("a.txt", "ObjectRemoved:DeleteMarkerCreated"),
- ("b.txt", "ObjectRemoved:DeleteMarkerCreated"),
- ("c.txt", "ObjectRemoved:Delete"),
- ]
-
-
-def test_delete_objects_in_an_unversioned_bucket(s3, sqs, notified):
- bucket, url, configure = notified
- configure("s3:ObjectRemoved:*")
- s3.put_object(Bucket=bucket, Key="u.txt", Body=b"x")
- s3.delete_objects(Bucket=bucket, Delete={"Objects": [{"Key": "u.txt"}, {"Key": "never-there.txt"}]})
- [record] = _records(sqs, url, 1)
- assert (record["s3"]["object"]["key"], record["eventName"]) == ("u.txt", "ObjectRemoved:Delete")
diff --git a/tests/test_sqs.py b/tests/test_sqs.py
index 712855c7..0624d149 100644
--- a/tests/test_sqs.py
+++ b/tests/test_sqs.py
@@ -1,7 +1,9 @@
import io
import json
import os
+import re
import time
+import uuid
import uuid as _uuid_mod
import zipfile
from urllib.parse import urlparse
@@ -10,6 +12,13 @@ import boto3
import pytest
from botocore.config import Config
from botocore.exceptions import ClientError
+from conftest import REGION, sqs_policy_allow_s3, sqs_policy_allow_sns
+
+from ministack.core.iam_evaluator import EvalContext, evaluate_resource_policy
+from ministack.core.responses import request_scope
+from ministack.services import s3 as s3_svc
+from ministack.services import sns as sns_svc
+from ministack.services import sqs as sqs_svc
def _make_zip(code: str) -> bytes:
@@ -1627,3 +1636,621 @@ def test_unicode_sqs_message(sqs):
sqs.send_message(QueueUrl=url, MessageBody=msg)
resp = sqs.receive_message(QueueUrl=url, MaxNumberOfMessages=1)
assert resp["Messages"][0]["Body"] == msg
+
+
+# --- Queue and topic resource policies (in-process, AUTH toggled) ---
+
+ACCT_A = "111111111111"
+ACCT_B = "222222222222"
+
+
+def _reset_messaging_state():
+ from ministack.core.iam_evaluator import _request_caller_arn
+
+ sqs_svc.reset()
+ sns_svc.reset()
+ s3_svc.reset()
+ _request_caller_arn.set("")
+
+
+def auth(monkeypatch):
+ """AUTH=true with clean in-process SQS/SNS/S3 state."""
+ import ministack.app as app_mod
+
+ monkeypatch.setattr(app_mod, "AUTH", True)
+ _reset_messaging_state()
+ yield
+ _reset_messaging_state()
+
+
+def noauth(monkeypatch):
+ """AUTH=false with clean in-process SQS/SNS/S3 state."""
+ import ministack.app as app_mod
+
+ monkeypatch.setattr(app_mod, "AUTH", False)
+ _reset_messaging_state()
+ yield
+ _reset_messaging_state()
+
+
+def _uniq(prefix):
+ return f"{prefix}-{uuid.uuid4().hex[:8]}"
+
+
+def _as(account, fn, *args, **kwargs):
+ """Call an in-process service function as *account* in the test region."""
+ with request_scope(account, REGION):
+ return fn(*args, **kwargs)
+
+
+def _resp_detail(resp):
+ """``(status, code, message)`` of an in-process SNS/S3 response tuple."""
+ status, _headers, body = resp
+ text = body.decode("utf-8", errors="replace") if isinstance(body, bytes) else str(body)
+ code = re.search(r"<Code>([^<]*)</Code>", text)
+ msg = re.search(r"<Message>([^<]*)</Message>", text)
+ return (status, code.group(1) if code else "",
+ msg.group(1) if msg else "")
+
+
+def _resp_code(resp):
+ """``(status, code)`` of an in-process SNS/S3 response tuple."""
+ status, code, _ = _resp_detail(resp)
+ return status, code
+
+
+def _qarn(account, name):
+ return f"arn:aws:sqs:{REGION}:{account}:{name}"
+
+
+def _tarn(account, name):
+ return f"arn:aws:sns:{REGION}:{account}:{name}"
+
+
+def _make_queue(account, name, policy=None):
+ url = _as(account, sqs_svc._act_create_queue,
+ {"QueueName": name}, "")["QueueUrl"]
+ if policy is not None:
+ _as(account, sqs_svc._act_set_queue_attributes,
+ {"QueueUrl": url,
+ "Attributes": {"Policy": json.dumps(policy)}}, url)
+ return url
+
+
+def _queue_bodies(arn):
+ q = sqs_svc._queue_by_arn(arn)
+ assert q is not None
+ return [m["body"] for m in q["messages"]]
+
+
+def _deny_all(arn, action):
+ return {"Version": "2012-10-17", "Statement": [{
+ "Sid": "deny", "Effect": "Deny", "Principal": {"AWS": "*"},
+ "Action": action, "Resource": arn}]}
+
+
+def _allow_account(arn, account, action):
+ return {"Version": "2012-10-17", "Statement": [{
+ "Sid": "x", "Effect": "Allow", "Principal": {"AWS": account},
+ "Action": action, "Resource": arn}]}
+
+
+def _make_topic(account, name):
+ assert _resp_code(_as(account, sns_svc._create_topic,
+ {"Name": name})) == (200, "")
+ return _tarn(account, name)
+
+
+def _set_topic_policy(account, arn, policy):
+ assert _resp_code(_as(account, sns_svc._set_topic_attributes, {
+ "TopicArn": arn, "AttributeName": "Policy",
+ "AttributeValue": json.dumps(policy)})) == (200, "")
+
+
+def _topic_policy(arn):
+ topic = sns_svc._topic_by_arn_any_scope(arn)
+ assert topic is not None
+ return json.loads(topic["attributes"]["Policy"])
+
+
+def _make_bucket(account, name):
+ assert _resp_code(_as(account, s3_svc._create_bucket,
+ name, b"", {})) == (200, "")
+
+
+def _notif_xml(kind, arn):
+ tag = "Queue" if kind == "sqs" else "Topic"
+ return (
+ "<NotificationConfiguration>"
+ f"<{tag}Configuration><{tag}>{arn}</{tag}>"
+ "<Event>s3:ObjectCreated:*</Event>"
+ f"</{tag}Configuration>"
+ "</NotificationConfiguration>"
+ ).encode()
+
+
+def _put_notif(account, bucket, kind, arn, skip=False):
+ headers = {"x-amz-skip-destination-validation": "true"} if skip else {}
+ return _as(account, s3_svc._put_bucket_notification,
+ bucket, _notif_xml(kind, arn), headers)
+
+
+# ---------------------------------------------------------------------------
+# Unit: evaluate_resource_policy Service principals
+# ---------------------------------------------------------------------------
+
+def _ctx(action="sqs:SendMessage", resource="arn:aws:sqs:us-east-1:1:q",
+ source_arn="", source_account="", principal="arn:aws:iam::1:root"):
+ return EvalContext(
+ principal_arn=principal,
+ principal_type="Service",
+ principal_account="1",
+ action=action,
+ resource_arn=resource,
+ region="us-east-1",
+ service_context={"aws:sourcearn": source_arn,
+ "aws:sourceaccount": source_account,
+ "aws:sourceowner": source_account},
+ )
+
+
+def _svc_policy():
+ return {
+ "Version": "2012-10-17",
+ "Statement": [{
+ "Effect": "Allow",
+ "Principal": {"Service": "s3.amazonaws.com"},
+ "Action": "sqs:SendMessage",
+ "Resource": "arn:aws:sqs:us-east-1:1:q",
+ "Condition": {"ArnLike": {"aws:SourceArn": "arn:aws:s3:*:*:bkt"}},
+ }],
+ }
+
+
+class TestEvaluateResourcePolicyService:
+ def test_service_principal_allows_matching_service(self):
+ assert evaluate_resource_policy(
+ _svc_policy(), _ctx(source_arn="arn:aws:s3:::bkt"),
+ service="s3.amazonaws.com").decision == "Allow"
+
+ def test_service_principal_ignores_other_service(self):
+ assert evaluate_resource_policy(
+ _svc_policy(), _ctx(source_arn="arn:aws:s3:::bkt"),
+ service="sns.amazonaws.com").decision == "ImplicitDeny"
+
+ def test_service_principal_ignored_without_service_arg(self):
+ # Legacy callers (Lambda layer policies, S3 bucket policies) keep the
+ # old behaviour: Service entries never match.
+ assert evaluate_resource_policy(
+ _svc_policy(), _ctx(source_arn="arn:aws:s3:::bkt")).decision == "ImplicitDeny"
+
+ def test_wrong_source_arn_denies(self):
+ assert evaluate_resource_policy(
+ _svc_policy(), _ctx(source_arn="arn:aws:s3:::other"),
+ service="s3.amazonaws.com").decision == "ImplicitDeny"
+
+ def test_aws_principal_still_matches(self):
+ doc = {"Statement": [{
+ "Effect": "Allow", "Principal": {"AWS": "222222222222"},
+ "Action": "sqs:SendMessage", "Resource": "*"}]}
+ ctx = _ctx(principal="arn:aws:iam::222222222222:root")
+ assert evaluate_resource_policy(doc, ctx).decision == "Allow"
+
+ def test_star_principal_matches(self):
+ doc = {"Statement": [{
+ "Effect": "Allow", "Principal": "*",
+ "Action": "sqs:SendMessage", "Resource": "*"}]}
+ assert evaluate_resource_policy(doc, _ctx()).decision == "Allow"
+
+
+# ---------------------------------------------------------------------------
+# Cross-account SQS
+# ---------------------------------------------------------------------------
+
+class TestCrossAccountSqs:
+ def test_send_denied_without_policy(self, auth):
+ url = _make_queue(ACCT_A, _uniq("xpol-q"))
+ with pytest.raises(sqs_svc._Err) as exc:
+ _as(ACCT_B, sqs_svc._act_send_message,
+ {"QueueUrl": url, "MessageBody": "no-policy"}, url)
+ assert exc.value.code == "AccessDenied"
+ assert exc.value.status == 403
+
+ def test_send_allowed_with_policy(self, auth):
+ name = _uniq("xpol-q")
+ url = _make_queue(ACCT_A, name, _allow_account(
+ _qarn(ACCT_A, name), ACCT_B, "sqs:SendMessage"))
+ _as(ACCT_B, sqs_svc._act_send_message,
+ {"QueueUrl": url, "MessageBody": "with-policy"}, url)
+ assert _queue_bodies(_qarn(ACCT_A, name)) == ["with-policy"]
+
+ def test_send_allowed_via_add_permission(self, auth):
+ name = _uniq("xpol-q")
+ url = _make_queue(ACCT_A, name)
+ _as(ACCT_A, sqs_svc._act_add_permission,
+ {"QueueUrl": url, "Label": "xacct",
+ "AWSAccountIds": [ACCT_B], "Actions": ["SendMessage"]}, url)
+ _as(ACCT_B, sqs_svc._act_send_message,
+ {"QueueUrl": url, "MessageBody": "via-add-permission"}, url)
+ assert _queue_bodies(_qarn(ACCT_A, name)) == ["via-add-permission"]
+
+ def test_explicit_deny_blocks_same_account(self, auth):
+ name = _uniq("xpol-q")
+ url = _make_queue(ACCT_A, name, _deny_all(
+ _qarn(ACCT_A, name), "sqs:SendMessage"))
+ with pytest.raises(sqs_svc._Err) as exc:
+ _as(ACCT_A, sqs_svc._act_send_message,
+ {"QueueUrl": url, "MessageBody": "deny-me"}, url)
+ assert exc.value.code == "AccessDenied"
+
+ def test_get_queue_url_with_owner_account(self, auth):
+ name = _uniq("xpol-q")
+ url = _make_queue(ACCT_A, name, _allow_account(
+ _qarn(ACCT_A, name), ACCT_B, "sqs:GetQueueUrl"))
+ found = _as(ACCT_B, sqs_svc._act_get_queue_url,
+ {"QueueName": name,
+ "QueueOwnerAWSAccountId": ACCT_A}, "")["QueueUrl"]
+ assert found == url
+
+ def test_get_queue_url_cross_account_denied_without_policy(self, auth):
+ name = _uniq("xpol-q")
+ _make_queue(ACCT_A, name)
+ with pytest.raises(sqs_svc._Err) as exc:
+ _as(ACCT_B, sqs_svc._act_get_queue_url,
+ {"QueueName": name,
+ "QueueOwnerAWSAccountId": ACCT_A}, "")
+ assert exc.value.code == "AccessDenied"
+ assert exc.value.status == 403
+
+ def test_get_queue_url_same_account_denied_by_explicit_deny(self, auth):
+ name = _uniq("xpol-q")
+ _make_queue(ACCT_A, name, _deny_all(
+ _qarn(ACCT_A, name), "sqs:GetQueueUrl"))
+ with pytest.raises(sqs_svc._Err) as exc:
+ _as(ACCT_A, sqs_svc._act_get_queue_url,
+ {"QueueName": name}, "")
+ assert exc.value.code == "AccessDenied"
+ assert exc.value.status == 403
+
+ def test_get_queue_url_same_account_passes_without_policy(self, auth):
+ name = _uniq("xpol-q")
+ url = _make_queue(ACCT_A, name)
+ assert _as(ACCT_A, sqs_svc._act_get_queue_url,
+ {"QueueName": name}, "") == {"QueueUrl": url}
+
+
+# ---------------------------------------------------------------------------
+# Cross-account SNS (+ AddPermission/RemovePermission)
+# ---------------------------------------------------------------------------
+
+class TestCrossAccountSns:
+ def test_publish_denied_by_default(self, auth):
+ arn = _make_topic(ACCT_A, _uniq("xpol-t"))
+ assert _resp_code(_as(ACCT_B, sns_svc._publish, {
+ "TopicArn": arn, "Message": "no-grant"})) == (403, "AuthorizationError")
+
+ def test_publish_allowed_with_policy(self, auth):
+ name = _uniq("xpol-t")
+ arn = _make_topic(ACCT_A, name)
+ _set_topic_policy(ACCT_A, arn, _allow_account(
+ arn, ACCT_B, "sns:Publish"))
+ assert _resp_code(_as(ACCT_B, sns_svc._publish, {
+ "TopicArn": arn, "Message": "granted"})) == (200, "")
+ topic = sns_svc._topic_by_arn_any_scope(arn)
+ assert [m["message"] for m in topic["messages"]] == ["granted"]
+
+ def test_explicit_deny_blocks_same_account(self, auth):
+ name = _uniq("xpol-t")
+ arn = _make_topic(ACCT_A, name)
+ _set_topic_policy(ACCT_A, arn, _deny_all(arn, "sns:Publish"))
+ assert _resp_code(_as(ACCT_A, sns_svc._publish, {
+ "TopicArn": arn, "Message": "deny-me"})) == (403, "AuthorizationError")
+
+ def test_add_permission_grants_cross_account_publish(self, auth):
+ arn = _make_topic(ACCT_A, _uniq("xpol-t"))
+ assert _resp_code(_as(ACCT_A, sns_svc._add_permission, {
+ "TopicArn": arn, "Label": "perm-1",
+ "AWSAccountId.member.1": ACCT_B,
+ "ActionName.member.1": "Publish"})) == (200, "")
+ stmt = next(s for s in _topic_policy(arn)["Statement"]
+ if s.get("Sid") == "perm-1")
+ assert ACCT_B in stmt["Principal"]["AWS"]
+ assert "sns:Publish" in stmt["Action"]
+ assert _resp_code(_as(ACCT_B, sns_svc._publish, {
+ "TopicArn": arn,
+ "Message": "via-add-permission"})) == (200, "")
+
+ def test_add_permission_rejects_duplicate_label(self, auth):
+ arn = _make_topic(ACCT_A, _uniq("xpol-t"))
+ params = {"TopicArn": arn, "Label": "dup",
+ "AWSAccountId.member.1": ACCT_B,
+ "ActionName.member.1": "Publish"}
+ assert _resp_code(_as(ACCT_A, sns_svc._add_permission, params)) == (200, "")
+ assert _resp_code(
+ _as(ACCT_A, sns_svc._add_permission, params)) == (
+ 400, "InvalidParameter")
+
+ def test_cross_account_tag_access_needs_grant(self, auth):
+ name = _uniq("xpol-t")
+ arn = _make_topic(ACCT_A, name)
+ assert _resp_code(_as(ACCT_B, sns_svc._list_tags_for_resource, {
+ "ResourceArn": arn})) == (403, "AuthorizationError")
+ _set_topic_policy(ACCT_A, arn, _allow_account(
+ arn, ACCT_B, ["sns:TagResource", "sns:ListTagsForResource"]))
+ assert _resp_code(_as(ACCT_B, sns_svc._tag_resource, {
+ "ResourceArn": arn, "Tags.member.1.Key": "k",
+ "Tags.member.1.Value": "v"})) == (200, "")
+ assert _resp_code(_as(ACCT_B, sns_svc._list_tags_for_resource, {
+ "ResourceArn": arn})) == (200, "")
+ assert sns_svc._topic_by_arn_any_scope(arn)["tags"] == {"k": "v"}
+
+ def test_remove_permission_drops_statement(self, auth):
+ arn = _make_topic(ACCT_A, _uniq("xpol-t"))
+ _as(ACCT_A, sns_svc._add_permission, {
+ "TopicArn": arn, "Label": "drop",
+ "AWSAccountId.member.1": ACCT_B,
+ "ActionName.member.1": "Publish"})
+ assert _resp_code(_as(ACCT_B, sns_svc._publish, {
+ "TopicArn": arn, "Message": "before"})) == (200, "")
+ assert _resp_code(_as(ACCT_A, sns_svc._remove_permission, {
+ "TopicArn": arn, "Label": "drop"})) == (200, "")
+ assert all(s.get("Sid") != "drop"
+ for s in _topic_policy(arn)["Statement"])
+ assert _resp_code(_as(ACCT_B, sns_svc._publish, {
+ "TopicArn": arn, "Message": "after"})) == (403, "AuthorizationError")
+
+
+# ---------------------------------------------------------------------------
+# S3 notifications gated on the destination policy
+# ---------------------------------------------------------------------------
+
+def _bucket_and_queue(tag):
+ bucket, qname = _uniq(f"{tag}-bkt"), _uniq(f"{tag}-q")
+ _make_bucket(ACCT_A, bucket)
+ url = _make_queue(ACCT_A, qname)
+ return bucket, url, _qarn(ACCT_A, qname)
+
+
+def _assert_put_rejected(bucket, kind, arn):
+ status, code, msg = _resp_detail(_put_notif(ACCT_A, bucket, kind, arn))
+ assert status == 400
+ assert code == "InvalidArgument"
+ assert "Unable to validate the following destination configurations" in msg
+
+
+def _bucket_topic_queue(tag):
+ bucket = _uniq(f"{tag}-bkt")
+ _make_bucket(ACCT_A, bucket)
+ topic = _make_topic(ACCT_A, _uniq(f"{tag}-t"))
+ qname = _uniq(f"{tag}-q")
+ qurl = _make_queue(ACCT_A, qname,
+ sqs_policy_allow_sns(_qarn(ACCT_A, qname), topic))
+ assert _resp_code(_as(ACCT_A, sns_svc._subscribe, {
+ "TopicArn": topic, "Protocol": "sqs",
+ "Endpoint": _qarn(ACCT_A, qname)})) == (200, "")
+ return bucket, topic, qurl, _qarn(ACCT_A, qname)
+
+
+class TestS3NotificationPolicies:
+ def test_s3_put_rejected_without_queue_policy(self, auth):
+ bucket, _url, arn = _bucket_and_queue("xpol")
+ _assert_put_rejected(bucket, "sqs", arn)
+
+ def test_s3_put_rejected_by_unrelated_policy(self, auth):
+ bucket, url, arn = _bucket_and_queue("xpol")
+ _as(ACCT_A, sqs_svc._act_set_queue_attributes,
+ {"QueueUrl": url, "Attributes": {"Policy": json.dumps(
+ _allow_account(arn, ACCT_B, "sqs:SendMessage"))}}, url)
+ _assert_put_rejected(bucket, "sqs", arn)
+
+ def test_s3_put_rejected_by_wrong_source_arn(self, auth):
+ bucket, url, arn = _bucket_and_queue("xpol")
+ _as(ACCT_A, sqs_svc._act_set_queue_attributes,
+ {"QueueUrl": url, "Attributes": {"Policy": json.dumps(
+ sqs_policy_allow_s3(arn, "no-such-bucket", ACCT_A))}}, url)
+ _assert_put_rejected(bucket, "sqs", arn)
+
+ def test_s3_to_sqs_delivered_with_policy(self, auth):
+ bucket, url, arn = _bucket_and_queue("xpol")
+ _as(ACCT_A, sqs_svc._act_set_queue_attributes,
+ {"QueueUrl": url, "Attributes": {"Policy": json.dumps(
+ sqs_policy_allow_s3(arn, bucket, ACCT_A))}}, url)
+ assert _resp_code(_put_notif(ACCT_A, bucket, "sqs", arn)) == (200, "")
+ bodies = _queue_bodies(arn)
+ assert len(bodies) == 1
+ assert "s3:TestEvent" in bodies[0]
+
+ def test_s3_skip_validation_stores_but_delivery_blocked(self, auth):
+ bucket, _url, arn = _bucket_and_queue("xpol")
+ assert _resp_code(
+ _put_notif(ACCT_A, bucket, "sqs", arn, skip=True)) == (200, "")
+ assert _queue_bodies(arn) == []
+ _as(ACCT_A, s3_svc._deliver_event_to_sqs,
+ arn, {"Records": []}, REGION, bucket_name=bucket)
+ assert _queue_bodies(arn) == []
+
+ def test_s3_to_sns_default_policy_delivers(self, auth):
+ bucket, topic, _qurl, qarn = _bucket_topic_queue("xpol")
+ assert _resp_code(_put_notif(ACCT_A, bucket, "sns", topic)) == (200, "")
+ bodies = _queue_bodies(qarn)
+ assert len(bodies) == 1
+ assert "s3:TestEvent" in bodies[0]
+
+ def test_s3_put_rejected_by_topic_policy_without_s3(self, auth):
+ bucket, topic, _qurl, _qarn = _bucket_topic_queue("xpol")
+ _set_topic_policy(ACCT_A, topic, _allow_account(
+ topic, ACCT_A, "sns:Publish"))
+ _assert_put_rejected(bucket, "sns", topic)
+
+ def test_s3_to_sns_skip_validation_delivery_blocked(self, auth):
+ bucket, topic, _qurl, qarn = _bucket_topic_queue("xpol")
+ _set_topic_policy(ACCT_A, topic, _allow_account(
+ topic, ACCT_A, "sns:Publish"))
+ assert _resp_code(
+ _put_notif(ACCT_A, bucket, "sns", topic, skip=True)) == (200, "")
+ _as(ACCT_A, s3_svc._deliver_event_to_sns,
+ topic, {"Records": []}, REGION, bucket_name=bucket)
+ assert _queue_bodies(qarn) == []
+
+ def test_s3_to_cross_account_sns_topic_delivered_with_policy(self, auth):
+ bucket = _uniq("xpol-xbkt")
+ _make_bucket(ACCT_A, bucket)
+ topic = _make_topic(ACCT_B, _uniq("xpol-xt"))
+ _set_topic_policy(ACCT_B, topic, {
+ "Version": "2012-10-17", "Statement": [{
+ "Sid": "s3-xacct", "Effect": "Allow",
+ "Principal": {"Service": "s3.amazonaws.com"},
+ "Action": "sns:Publish", "Resource": topic,
+ "Condition": {
+ "ArnLike": {"aws:SourceArn": f"arn:aws:s3:::{bucket}"},
+ "StringEquals": {"aws:SourceAccount": ACCT_A}}}]})
+ qname = _uniq("xpol-xq")
+ _make_queue(ACCT_B, qname, sqs_policy_allow_sns(
+ _qarn(ACCT_B, qname), topic))
+ assert _resp_code(_as(ACCT_B, sns_svc._subscribe, {
+ "TopicArn": topic, "Protocol": "sqs",
+ "Endpoint": _qarn(ACCT_B, qname)})) == (200, "")
+ # The topic lives in another account, but its policy allows this
+ # bucket's deliveries, so the PUT validates instead of failing.
+ assert _resp_code(_put_notif(ACCT_A, bucket, "sns", topic)) == (200, "")
+ bodies = _queue_bodies(_qarn(ACCT_B, qname))
+ assert len(bodies) == 1
+ assert "s3:TestEvent" in bodies[0]
+
+ def test_s3_to_cross_account_sqs_queue_delivered_with_policy(self, auth):
+ bucket = _uniq("xpol-xbkt")
+ _make_bucket(ACCT_A, bucket)
+ qname = _uniq("xpol-xq")
+ _make_queue(ACCT_B, qname, sqs_policy_allow_s3(
+ _qarn(ACCT_B, qname), bucket, ACCT_A))
+ assert _resp_code(
+ _put_notif(ACCT_A, bucket, "sqs", _qarn(ACCT_B, qname))) == (200, "")
+ bodies = _queue_bodies(_qarn(ACCT_B, qname))
+ assert len(bodies) == 1
+ assert "s3:TestEvent" in bodies[0]
+
+ def test_s3_to_cross_account_topic_rejected_without_policy(self, auth):
+ bucket = _uniq("xpol-xbkt")
+ _make_bucket(ACCT_A, bucket)
+ topic = _make_topic(ACCT_B, _uniq("xpol-xt"))
+ _assert_put_rejected(bucket, "sns", topic)
+
+
+# ---------------------------------------------------------------------------
+# SNS→SQS fanout gated on the queue policy
+# ---------------------------------------------------------------------------
+
+def _topic_and_queue(tag):
+ topic = _make_topic(ACCT_A, _uniq(f"{tag}-t"))
+ qname = _uniq(f"{tag}-q")
+ qurl = _make_queue(ACCT_A, qname)
+ assert _resp_code(_as(ACCT_A, sns_svc._subscribe, {
+ "TopicArn": topic, "Protocol": "sqs",
+ "Endpoint": _qarn(ACCT_A, qname)})) == (200, "")
+ return topic, qurl, _qarn(ACCT_A, qname)
+
+
+class TestSnsFanoutPolicies:
+ def test_fanout_blocked_without_policy(self, auth):
+ topic, _qurl, qarn = _topic_and_queue("xpol-fan")
+ assert _resp_code(_as(ACCT_A, sns_svc._publish, {
+ "TopicArn": topic, "Message": "no-policy"})) == (200, "")
+ assert _queue_bodies(qarn) == []
+
+ def test_fanout_blocked_by_wrong_source_arn(self, auth):
+ topic, qurl, qarn = _topic_and_queue("xpol-fan")
+ _as(ACCT_A, sqs_svc._act_set_queue_attributes,
+ {"QueueUrl": qurl, "Attributes": {"Policy": json.dumps(
+ sqs_policy_allow_sns(
+ qarn, f"arn:aws:sns:{REGION}:{ACCT_A}:wrong-topic"))}},
+ qurl)
+ assert _resp_code(_as(ACCT_A, sns_svc._publish, {
+ "TopicArn": topic, "Message": "wrong-source"})) == (200, "")
+ assert _queue_bodies(qarn) == []
+
+ def test_fanout_delivered_with_policy(self, auth):
+ topic, qurl, qarn = _topic_and_queue("xpol-fan")
+ _as(ACCT_A, sqs_svc._act_set_queue_attributes,
+ {"QueueUrl": qurl, "Attributes": {"Policy": json.dumps(
+ sqs_policy_allow_sns(qarn, topic))}}, qurl)
+ assert _resp_code(_as(ACCT_A, sns_svc._publish, {
+ "TopicArn": topic, "Message": "with-policy"})) == (200, "")
+ bodies = _queue_bodies(qarn)
+ assert len(bodies) == 1
+ assert "with-policy" in bodies[0]
+
+ def test_cross_account_fanout_needs_queue_policy(self, auth):
+ topic = _make_topic(ACCT_A, _uniq("xpol-xfan"))
+ _set_topic_policy(ACCT_A, topic, _allow_account(
+ topic, ACCT_B, "sns:Subscribe"))
+ qname = _uniq("xpol-xfanq")
+ qurl = _make_queue(ACCT_B, qname)
+ assert _resp_code(_as(ACCT_B, sns_svc._subscribe, {
+ "TopicArn": topic, "Protocol": "sqs",
+ "Endpoint": _qarn(ACCT_B, qname)})) == (200, "")
+ assert _resp_code(_as(ACCT_A, sns_svc._publish, {
+ "TopicArn": topic, "Message": "no-queue-policy"})) == (200, "")
+ assert _queue_bodies(_qarn(ACCT_B, qname)) == []
+
+ _as(ACCT_B, sqs_svc._act_set_queue_attributes,
+ {"QueueUrl": qurl, "Attributes": {"Policy": json.dumps(
+ sqs_policy_allow_sns(_qarn(ACCT_B, qname), topic))}}, qurl)
+ assert _resp_code(_as(ACCT_A, sns_svc._publish, {
+ "TopicArn": topic, "Message": "with-queue-policy"})) == (200, "")
+ bodies = _queue_bodies(_qarn(ACCT_B, qname))
+ assert len(bodies) == 1
+ assert "with-queue-policy" in bodies[0]
+
+ def test_cross_account_subscribe_denied_without_policy(self, auth):
+ topic = _make_topic(ACCT_A, _uniq("xpol-xfan"))
+ assert _resp_code(_as(ACCT_B, sns_svc._subscribe, {
+ "TopicArn": topic, "Protocol": "email",
+ "Endpoint": "[email protected]"})) == (403, "AuthorizationError")
+
+
+# ---------------------------------------------------------------------------
+# AUTH=false stays permissive
+# ---------------------------------------------------------------------------
+
+class TestAuthDisabledIsPermissive:
+ def test_cross_account_send_passes_without_policy(self, noauth):
+ name = _uniq("xpol-q")
+ url = _make_queue(ACCT_A, name)
+ _as(ACCT_B, sqs_svc._act_send_message,
+ {"QueueUrl": url, "MessageBody": "open"}, url)
+ assert _queue_bodies(_qarn(ACCT_A, name)) == ["open"]
+
+ def test_cross_account_publish_passes_without_policy(self, noauth):
+ arn = _make_topic(ACCT_A, _uniq("xpol-t"))
+ assert _resp_code(_as(ACCT_B, sns_svc._publish, {
+ "TopicArn": arn, "Message": "open"})) == (200, "")
+
+ def test_explicit_deny_ignored(self, noauth):
+ name = _uniq("xpol-q")
+ url = _make_queue(ACCT_A, name, _deny_all(
+ _qarn(ACCT_A, name), "sqs:SendMessage"))
+ _as(ACCT_A, sqs_svc._act_send_message,
+ {"QueueUrl": url, "MessageBody": "deny-ignored"}, url)
+ assert _queue_bodies(_qarn(ACCT_A, name)) == ["deny-ignored"]
+
+ def test_get_queue_url_ignores_deny(self, noauth):
+ name = _uniq("xpol-q")
+ url = _make_queue(ACCT_A, name, _deny_all(
+ _qarn(ACCT_A, name), "sqs:GetQueueUrl"))
+ assert _as(ACCT_A, sqs_svc._act_get_queue_url,
+ {"QueueName": name}, "") == {"QueueUrl": url}
+
+ def test_s3_put_and_delivery_without_policy(self, noauth):
+ bucket, _url, arn = _bucket_and_queue("xpol")
+ assert _resp_code(_put_notif(ACCT_A, bucket, "sqs", arn)) == (200, "")
+ bodies = _queue_bodies(arn)
+ assert len(bodies) == 1
+ assert "s3:TestEvent" in bodies[0]
+
+ def test_fanout_without_policy(self, noauth):
+ topic, _qurl, qarn = _topic_and_queue("xpol-fan")
+ assert _resp_code(_as(ACCT_A, sns_svc._publish, {
+ "TopicArn": topic, "Message": "open"})) == (200, "")
+ assert len(_queue_bodies(qarn)) == 1
diff --git a/tests/test_sts.py b/tests/test_sts.py
index 77ee1e3e..d1212943 100644
--- a/tests/test_sts.py
+++ b/tests/test_sts.py
@@ -190,7 +190,7 @@ def test_sts_get_session_token_rejects_unknown_caller_when_auth_enabled(monkeypa
)
assert status == 403
- assert b"UnrecognizedClientException" in payload
+ assert b"InvalidClientTokenId" in payload
assert issued_key not in sts_mod._sessions
@@ -708,12 +708,7 @@ def test_sts_get_caller_identity_credential_rejections_require_auth(
if not auth_enabled and credential_kind == "unknown":
assert b":root</Arn>" in payload
if auth_enabled:
- expected = (
- b"UnrecognizedClientException"
- if credential_kind == "unknown"
- else b"InvalidClientTokenId"
- )
- assert expected in payload
+ assert b"InvalidClientTokenId" in payload
finally:
iam_svc._access_keys.pop_scoped(account, None, key, None)
diff --git a/tests/test_tls.py b/tests/test_tls.py
index 236ff7ff..e0738c7e 100644
--- a/tests/test_tls.py
+++ b/tests/test_tls.py
@@ -167,13 +167,31 @@ def test_generated_cert_names_the_cognito_issuer_host(tmp_path, monkeypatch):
monkeypatch.delenv("MINISTACK_SSL_CERT", raising=False)
monkeypatch.delenv("MINISTACK_SSL_KEY", raising=False)
cert_path, _key = tls.resolve_tls_material()
- assert "DNS:cognito-idp.eu-west-2.amazonaws.com" in _san_of(cert_path)
+ san = _san_of(cert_path)
+ assert "DNS:cognito-idp.eu-west-2.amazonaws.com" in san
+ assert "DNS:cognito-idp.ap-southeast-2.amazonaws.com" in san
assert tls.cognito_idp_host() == "cognito-idp.eu-west-2.amazonaws.com"
-def test_cached_cert_without_the_issuer_host_is_regenerated(tmp_path, monkeypatch):
- """A cert cached by an older build, or for another region, cannot serve the
- issuer host, so reusing it would fail the handshake."""
+def test_cached_cert_without_the_issuer_hosts_is_regenerated(tmp_path, monkeypatch):
+ """A cert cached by an older build lacks the issuer hosts and is replaced."""
+ from ministack.core import tls
+
+ monkeypatch.setenv("TMPDIR", str(tmp_path))
+ monkeypatch.delenv("MINISTACK_SSL_CERT", raising=False)
+ monkeypatch.delenv("MINISTACK_SSL_KEY", raising=False)
+ tls_dir = tmp_path / "ministack-tls"
+ tls_dir.mkdir()
+ subprocess.run([
+ "openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes",
+ "-keyout", str(tls_dir / "server.key"), "-out", str(tls_dir / "server.crt"),
+ "-days", "1", "-subj", "/CN=old", "-addext", "subjectAltName=DNS:localhost",
+ ], check=True, capture_output=True)
+ cert, _ = tls.resolve_tls_material()
+ assert "DNS:cognito-idp.us-east-1.amazonaws.com" in _san_of(cert)
+
+
+def test_cached_cert_is_reused_across_regions(tmp_path, monkeypatch):
from ministack.core import tls
monkeypatch.setenv("TMPDIR", str(tmp_path))
@@ -182,11 +200,9 @@ def test_cached_cert_without_the_issuer_host_is_regenerated(tmp_path, monkeypatc
monkeypatch.setenv("MINISTACK_REGION", "eu-west-2")
first, _ = tls.resolve_tls_material()
first_bytes = open(first, "rb").read()
-
monkeypatch.setenv("MINISTACK_REGION", "us-east-1")
second, _ = tls.resolve_tls_material()
- assert "DNS:cognito-idp.us-east-1.amazonaws.com" in _san_of(second)
- assert open(second, "rb").read() != first_bytes
+ assert open(second, "rb").read() == first_bytes
def test_byo_certificate_is_never_regenerated(tmp_path, monkeypatch):