Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

prerelease (#2396)

Nahuel Nucera3499c445cdcc
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4d4fef37..fc94db1b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,49 +7,38 @@ Versioning follows [Semantic Versioning](https://semver.org/).
## [Unreleased]
-### Fixed
-- **Cognito — OIDC identity providers with only `oidc_issuer` use its discovered endpoints** — federated sign-in sent users to `{oidc_issuer}/authorize` and exchanged codes at `{oidc_issuer}/token` unless `authorize_url` and `token_url` were set, so an issuer whose endpoints live elsewhere, such as Google's, failed. Without those keys the endpoints now come from the issuer's `/.well-known/openid-configuration`, as on AWS, and a document naming another issuer is refused with `InvalidParameterException`. Contributed by @kinoh.
-- **S3 — Block Public Access on new buckets** — `GetPublicAccessBlock` on a new bucket, from `CreateBucket` or from a template without `PublicAccessBlockConfiguration`, answered 404 and nothing refused a public bucket policy or ACL; under `AUTH=true` a new bucket now has all four settings on, as on AWS, so a public `PutBucketPolicy`, `PutBucketAcl` or object ACL answers 403 `AccessDenied` and `CreateBucket` with a public ACL 400 `InvalidBucketAclWithBlockPublicAccessError` until `PutPublicAccessBlock` or `DeletePublicAccessBlock` changes them, and removing the property from a template turns all four on again. With `AUTH` unset nothing changes. Contributed by @iot-rocket.
-- **S3 — CORS under `AUTH=true`** — every S3 preflight answered 200 with `Access-Control-Allow-Origin: *` and every response carried the same wildcard headers, whatever the bucket's CORS configuration; under `AUTH=true` a preflight to an existing bucket now has to match one of its CORS rules (403 `AccessForbidden` without a configuration or a matching rule, 400 `BadRequest` without `Origin`) and gets that rule's headers, and a request with an `Origin` carries CORS headers only when a rule allows its origin and method. Contributed by @iot-rocket.
-- **IoT — MQTT 5 No Local follows AWS IoT** — a subscription with No Local kept the client's own messages from it; AWS IoT Core ignores that option and delivers them, and so does the broker now. Contributed by @iot-rocket.
-- **CloudFront — a viewer-request function that leaves the query string alone no longer re-encodes it** — any function that returned the request it was given counted as a query-string rewrite, so the origin got the query rebuilt from the parsed parameters instead of as the viewer sent it: a bare key gained an `=` (`?flag` became `?flag=`). The query string is now rebuilt only when the function's returned `querystring` differs from the one in its event. Contributed by @jeffreyheller.
-- **API Gateway — an HTTP API's responses carry `apigw-requestid`, not a REST API's request ids** — a response from an HTTP API's route got `x-amzn-requestid` and `x-amz-request-id`; an HTTP API names its request id `apigw-requestid`, 16 base64 characters, which it now sends instead. REST APIs keep `x-amzn-requestid`. Contributed by @jeffreyheller.
-- **Lambda — publishing on create or code update leaves `$LATEST` unpublished** — `CreateFunction` and `UpdateFunctionCode` with `Publish: true` overwrote `$LATEST`'s `Version` with the new version number. `ListVersionsByFunction` then listed that number twice and no `$LATEST`, `GetFunctionConfiguration` without a qualifier reported it, and invoking `$LATEST` set `AWS_LAMBDA_FUNCTION_VERSION` to it. `$LATEST` now stays `$LATEST`, as on AWS; the create and update responses are unchanged.
### Added
+- **EC2 — Verified Access control plane** — instances, trust providers (`oidc`, and `jamf`, `crowdstrike` and `jumpcloud` devices), groups, `load-balancer` and `network-interface` endpoints, trust provider attachments, instance logging configuration, and group and endpoint policies, as the `aws_verifiedaccess_*` Terraform resources call them, with AWS's codes, messages and check order. Under `AUTH=true` each call is authorized on every resource AWS checks it against. No access evaluation or Cedar parsing; `rds` and `cidr` endpoints and `NativeApplicationOidcOptions` answer `UnsupportedOperation`. Contributed by @kharabasz.
+- **Step Functions — Distributed Map** — a Map whose `ProcessorConfig` mode is `DISTRIBUTED` ran inline, with no map run and its `ItemReader` ignored; it now creates a Map Run (`ListMapRuns`, `DescribeMapRun`, `UpdateMapRun`, `ListExecutions` with `mapRunArn`) whose items run as child executions under `<state machine>/<label>`, reads its items from S3 (`s3:getObject` as JSON, JSONL or CSV, `s3:listObjectsV2` object metadata, `MaxItems`), and fails with `States.ExceedToleratedFailureThreshold` past its tolerated failures. Reported by @cupidchan.
- **SSM — service settings** — `GetServiceSetting`, `UpdateServiceSetting` and `ResetServiceSetting`, per account and region, persisted. `SettingValue` is validated for the ids whose accepted values AWS documents, and `/ssm/parameter-store/default-parameter-tier` sets the `Tier` of a `PutParameter` that names none. Unblocks Terraform's `aws_ssm_service_setting`. Contributed by @skialpine.
- **Auth — SigV4a signatures are verified under `AUTH=true`** — requests signed with `AWS4-ECDSA-P256-SHA256`, which SDKs use for Multi-Region Access Points, were refused for every IAM user. Header and presigned S3 signatures are now checked with the P-256 key SigV4a derives from the secret, with their date or expiry, and an `X-Amz-Region-Set` that leaves out the serving region is refused (`RegionSetMismatch` on S3, the unknown-key error elsewhere). Contributed by @iot-rocket.
-- **EC2 — Verified Access control plane** — instances, trust providers (`oidc`, and `jamf`, `crowdstrike` and `jumpcloud` devices), groups, `load-balancer` and `network-interface` endpoints, trust provider attach and detach, instance logging configuration, and group and endpoint policies, per account and region, as the `aws_verifiedaccess_*` Terraform resources call them. Codes, messages and check order follow AWS: an instance takes one user trust provider, a provider attaches to one instance, a group needs a user provider and locks it, a group or instance with endpoints can't be deleted or moved, and endpoints need an existing load balancer or ENI, subnets, security groups and certificate. An endpoint answers `pending` on create and `updating` on modify, and is `active` on the next read; OIDC client secrets read back as `REDACTED`. No access evaluation or Cedar parsing; `rds` and `cidr` endpoints, `NativeApplicationOidcOptions` and `CidrEndpointsCustomSubDomain` answer `UnsupportedOperation`. Contributed by @kharabasz.
-
-### Fixed
-- **Lambda — publishing on create or code update leaves `$LATEST` unpublished** — `CreateFunction` and `UpdateFunctionCode` with `Publish: true` overwrote `$LATEST`'s `Version` with the new version number. `ListVersionsByFunction` then listed that number twice and no `$LATEST`, `GetFunctionConfiguration` without a qualifier reported it, and invoking `$LATEST` set `AWS_LAMBDA_FUNCTION_VERSION` to it. `$LATEST` now stays `$LATEST`, as on AWS; the create and update responses are unchanged. Contributed by @skialpine.
-- **S3 — presigned URL `X-Amz-Expires`** — under `AUTH=true` a presigned URL without `X-Amz-Expires` never expired; it now answers 400 `AuthorizationQueryParametersError` "Query-string authentication version 4 requires the X-Amz-Algorithm, X-Amz-Credential, X-Amz-Signature, X-Amz-Date, X-Amz-SignedHeaders, and X-Amz-Expires parameters.", a negative value "X-Amz-Expires must be non-negative", and SigV4a presigned URLs follow the same limits.
-- **S3 — `x-amz-mp-object-size` on `CompleteMultipartUpload`** — the header (`MpuObjectSize`) was ignored; a size that differs from the parts' total now answers 400 `InvalidRequest` "The provided 'x-amz-mp-object-size' header value 11 does not match what was computed: 10", a non-numeric one "Value for x-amz-mp-object-size header is invalid: 'abc'", and neither writes the object or closes the upload. Reported by @adzcodemi.
-- **IoT — QoS 1 redelivery follows AWS IoT** — the broker resent an unacknowledged QoS 1 message every 10 seconds with the DUP flag set; it now resends it 30 seconds after the last delivery (`IOT_RETRANSMIT_SECONDS` keeps changing that), and with DUP 0, as AWS IoT Core does, on the open connection and after a persistent session resumes. Contributed by @iot-rocket.
-- **Auth — requests without credentials are anonymous in every service** — under `AUTH=true` a request with no `Authorization` header and no query signature ran as the account root everywhere but S3, so unsigned STS, SQS, DynamoDB or Lambda calls were served. It is now refused in the service's own error, as on AWS: `MissingAuthenticationToken` 403 from Query and REST-XML services, `MissingAuthenticationTokenException` from JSON services, `AccessDenied` 403 from SQS, `UnrecognizedClientException` from Secrets Manager. Query credentials without `X-Amz-Algorithm` count as none. Operations the SDKs send unsigned (Cognito sign-in, `GetId`, `AssumeRoleWithWebIdentity`), SNS `ConfirmSubscription` and `Unsubscribe`, API Gateway and Function URL invokes and S3 keep their behaviour. Contributed by @iot-rocket.
-- **AppSync Events — `AUTH=true` checks the Event API's authorization** — under `AUTH=true` an HTTP publish, a WebSocket connection or a subscription was accepted with any API key or none unless `APPSYNC_EVENTS_ENFORCE_AUTH` was set. It now needs what the API or the channel namespace accepts, as on AWS: a key registered on the API for `API_KEY`, a SigV4 signature for `AWS_IAM`, a bearer token for Cognito and OIDC. A refused publish answers 401 `UnauthorizedException`, with AWS's "You are not authorized to make this call." for a key the API does not know. `APPSYNC_EVENTS_ENFORCE_AUTH` works as before with `AUTH` unset. Contributed by @iot-rocket.
-- **Cognito — pre token generation for `client_credentials`** — a `client_credentials` grant invoked a `V2_0` pre token generation trigger; it now invokes the trigger only when `LambdaVersion` is `V3_0`, as AWS documents and does. Contributed by @iot-rocket.
-- **Cognito — `client_credentials` needs the flow** — the `/oauth2/token` endpoint gave a `client_credentials` token to an app client without `client_credentials` in `AllowedOAuthFlows`; such a client now gets `invalid_grant`, or `invalid_scope` first when a requested scope is unknown or none is left, as on AWS. Contributed by @iot-rocket.
-- **S3 — ACLs on buckets with `BucketOwnerEnforced`** — on a bucket with ACLs disabled, the default for new buckets, ACL writes answered 200 and a stored bucket grant still opened the bucket to other accounts and anonymous callers; `PutBucketAcl` and `PutObjectAcl` now answer 400 `AccessControlListNotSupported`, `PutObject`, `CopyObject`, `CreateMultipartUpload` and POST uploads refuse any ACL but an owner-only one with the same error, `CreateBucket` and a switch to `BucketOwnerEnforced` refuse an ACL that grants others with 400 `InvalidBucketAclWithObjectOwnership`, and ACLs stored earlier grant nothing and read as the owner's until ACLs are enabled again. Contributed by @iot-rocket.
-- **S3 — `RestrictPublicBuckets` and policies that are not public** — the setting dropped the bucket policy for every caller from another account, so a policy that names one account stopped granting it; it now drops the policy only when the policy is public, as the S3 user guide describes. Contributed by @iot-rocket.
-- **IoT — thing-group membership by ARN** — `AddThingToThingGroup` and `RemoveThingFromThingGroup` ignored `thingGroupArn` and `thingArn`, so a body with ARNs answered 400 or 404; both calls now act on the thing group and thing the ARNs name (in the request's region, as on AWS), refuse a name and ARN that differ, an ARN of another type or account and a body without the group or the thing with 400 `InvalidRequestException` and AWS's messages, and removing a thing from a group that does not exist answers 200 also when the group is given by ARN. Under `AUTH=true` such an ARN is authorized as that resource. Contributed by @iot-rocket.
-- **Reset — `/_ministack/reset` no longer removes another MiniStack's containers** — the OpenSearch, OpenSearch Serverless, DSQL, ECS and EKS resets removed every container carrying the service's label on the Docker daemon, including those of other MiniStack instances sharing it. They now remove only this instance's (`ministack.instance`), as the EC2, Lambda microVM and Flink resets and the boot and shutdown sweeps already do. Contributed by @jeffreyheller.
-- **S3 — Multi-Region Access Point requests are authorized** — under `AUTH=true`, requests through an MRAP hostname were served without an IAM check. They are now authorized on the member bucket and then on the MRAP, whose policy is evaluated like a bucket policy, and answer `AccessDenied` naming the resource that refused them. Contributed by @iot-rocket.
-- **Cognito — signing keys, managed login and federated users** — every user pool signed with one shared key (`kid` `ministack-key-1`); each pool now has two keys of its own, one for access tokens and one for ID tokens, both published at its JWKS URI, and a token verifies only against its own pool. Managed login refuses PKCE `plain` and a `code_challenge` without `S256` (`error=invalid_request`), any `redirect_uri` the client didn't register, also for a client with no callback URLs, and a `grant_type` other than `authorization_code`, `refresh_token` or `client_credentials` (400 `unsupported_grant_type`). Adding an identity provider creates the group `<user pool id>_<provider name>`, and a federated user's new profile joins it and carries its `identities`, so the ID token has both claims. Reported by @cupidchan.
-- **S3 — bucket policy writes through an access point** — under `AUTH=true` `PutBucketPolicy` and `DeleteBucketPolicy` sent through an access point alias or ARN changed the bucket's policy; they now get 403 `AccessDenied` with an explicit deny on the access point, while `GetBucketPolicy` through it and both writes on the bucket itself work as before. Contributed by @iot-rocket.
-- **S3 — SigV2 requests through an access point** — under `AUTH=true` a SigV2 presigned URL for an access point ARN failed its signature check, because the check signed the access point host as a bucket, and a SigV2 `Authorization` header through an access point alias or ARN was served. The presigned URL is now checked on its path as the SDK signs it, and the header request gets 400 `InvalidRequest`. Contributed by @iot-rocket.
-- **IoT — thing-group membership authorized on the group and the thing** — with `AUTH=true`, `AddThingToThingGroup` and `RemoveThingFromThingGroup` were authorized on `thinggroup/addThingToThingGroup` (the request path), so a policy granting exactly the thing group and the thing was denied; both calls are now checked on the thing group and then the thing named in the body (by name or ARN), and a denial names the resource that failed. Contributed by @iot-rocket.
-- **API Gateway — `AWS_IAM` methods refuse an `Authorization` header that is not SigV4** — under `AUTH=true` a REST API method with `AWS_IAM` authorization served a bearer token or an algorithm in lower case, and answered the SigV4a algorithm name without a signed `x-amz-region-set` or an unknown `AWS4-` algorithm with `UnrecognizedClientException`; it now answers 403 `IncompleteSignatureException` with API Gateway's message for each, while a method without `AWS_IAM` still serves such calls. Contributed by @iot-rocket.
-- **S3 — access point policies and unsigned requests** — an access point policy was stored but never applied, and an unsigned request through an access point was served. With `AUTH=true` a request through an access point alias or ARN is allowed on the access point by an identity policy or the access point policy (a Deny in either refuses it), and an unsigned one gets 400 `InvalidRequest`; S3 Control calls on the access point are unaffected. Contributed by @iot-rocket.
-- **API Gateway v2 — WebSocket connections, image functions and dropped fields** — `DeleteConnection` answered 204 but left the socket open; it now sends the client a close frame, `GetConnection` and `PostToConnection` answer 410 `GoneException`, and `$disconnect` runs after the close. A WebSocket route whose Lambda is a container image or a runtime other than Python or Node.js answered an empty 200 without running it; the function now runs. Routes keep `RouteResponseSelectionExpression` and `ModelSelectionExpression`, and stages keep `AccessLogSettings`, `ClientCertificateId` and `DeploymentId`, also from CloudFormation. Reported by @cupidchan.
-- **Lambda — layers from other accounts** — attaching a layer MiniStack doesn't hold, such as Datadog's, was refused with `AccessDeniedException`; it now resolves as an empty layer (`CodeSize` 0, a warning in the log, never run), for `CreateFunction`, `UpdateFunctionConfiguration`, `GetLayerVersionByArn` and `GetLayerVersion`. A layer MiniStack holds still needs its grant, another region is still refused, and `GetLayerVersion` refuses a `LayerName` over 140 characters with `ValidationException`. Contributed by @kharabasz.
-- **CloudFront — `CreateFunction` tags** — the request's `Tags` were dropped, so `ListTagsForResource` on the new function's ARN was empty and Terraform re-planned `tags_all` on `aws_cloudfront_function`; they are now stored, a tag without `Value` is listed with an empty one, a refused create stores none, and `DeleteFunction` removes them so a function created again under the same name starts without them. `CreateFunction` and `UpdateFunction` answer invalid base64 code with `MalformedInput` "Invalid Base64 encoding", and `CreateFunction` now refuses what AWS refuses, before the name lookup: a missing or empty `FunctionCode`, a missing `FunctionConfig` or `Runtime`, an unknown runtime, a missing or empty `Name`, and a tag key that is missing, empty, over 128 characters or outside letters, numbers, spaces and `_ . : / = + - @` (values over 256 characters or outside the same set), listed together (at most ten) in one `MalformedXML`; and after it, `InvalidTagging` for a repeated key, an `aws:` key or more than 50 tags. Contributed by @kharabasz.
-- **CloudWatch — `PutMetricAlarm` and `PutCompositeAlarm` tags on create** — `Tags` were dropped when either call created an alarm, so Terraform re-planned `tags_all` on `aws_cloudwatch_metric_alarm` and `aws_cloudwatch_composite_alarm`; they are now stored over CBOR, JSON 1.0 and Query, keeping empty values and the last value of a repeated key, and they are still ignored when the call updates an existing alarm, as on AWS (use `TagResource` / `UntagResource`). Both calls now refuse what AWS refuses, before anything is stored (`PutCompositeAlarm` before its `AlarmRule` checks): a wrong JSON type in `Tags` (`MalformedInput`), a missing or empty `Key`, a missing `Value`, a key over 128 or a value over 256 characters, or a `DatapointsToAlarm` below 1, listed together (at most ten) in one `ValidationError`, on create a key or value with characters outside letters, numbers, spaces and `_ . : / = + - @` or too long in UTF-16 units, an `aws:` key on create (`PutMetricAlarm` 403 `AccessDenied`, `PutCompositeAlarm` `ValidationError` "Invalid System Tag"), and more than 50 tags on create. Query error messages are now XML-escaped. Contributed by @kharabasz.
-- **CloudWatch — `DatapointsToAlarm` only when set** — an alarm created by `PutMetricAlarm` or `AWS::CloudWatch::Alarm` without `DatapointsToAlarm` reported it as `EvaluationPeriods`, so Terraform planned `datapoints_to_alarm = 3 -> null`; `DescribeAlarms` now leaves it out unless the request or template sets it, the Query response includes it when set, and an update that omits it clears it. Contributed by @kharabasz.
-- **API Gateway v2 — `PostToConnection` sends a text frame** — data posted to a WebSocket connection reached the client as a binary frame, so a browser `WebSocket` handed `onmessage` a `Blob` where AWS gives a string; it now arrives as a text frame, as on AWS, with bytes that aren't UTF-8 replaced by U+FFFD. Contributed by @toby6578.
-- **DynamoDB — PartiQL statements are parsed as DynamoDB parses them** — the regular-expression reader is replaced by a parser that answers malformed statements with AWS's messages (`Statement wasn't well formed, can't be processed: Unexpected term`, `Unexpected token after expression`, `Missing right-hand side expression of infix operator`, `Aliasing is not supported`, positions as `at <line>:<column>:<length>`), skips `--` and `/* */` comments, reads exponent and leading-dot number literals and refuses out-of-range ones `under root` with their path, and supports `ORDER BY`, `attribute_exists`, `attribute_not_exists`, `size`, `attribute_type` and `IS` in a WHERE (a false one on UPDATE or DELETE is `ConditionalCheckFailedException`), `set_add` and `set_delete`, the `FROM ... WHERE ... SET` form of UPDATE, and `EXISTS(SELECT ...)` in `ExecuteTransaction`. Number parameters are read as Java's `BigDecimal` reads them, a SELECT on a multi-attribute index is refused, overlapping OR branches on the sort key are refused, an INSERT returns an empty `Items` list and a write transaction an empty `Responses` list. Reported by @hicksy (paritysuite.org).
-- **DynamoDB — numbers and duplicate set members** — a number's exponent is now read in full before the 1E-130 to 9.99E+125 limits, so `1e99999999999` is a `Number overflow` rather than unconvertible, zero written with an exponent is range-checked, and a duplicate set member is reported with its members in byte order; PutItem, UpdateItem and DeleteItem give these inside `1 validation error detected: `, the other operations bare, wherever the value sits (item, key, expression value, `Expected`, `AttributeUpdates`, `QueryFilter`). Reported by @hicksy (paritysuite.org).
-- **DynamoDB — which error comes first** — TransactWriteItems, TransactGetItems, BatchWriteItem and BatchGetItem report their member errors (enums, table names, `ClientRequestToken`, empty lists) together before any table is looked up, and TransactWriteItems then checks each action in request order; PutItem, UpdateItem and DeleteItem read a request without a `Key` or with a value nested past about 62 levels in the older per-value form; legacy `Expected` conditions are validated; a KeyConditionExpression reports an operand of the wrong type in the order written and names a repeated or extra key condition; a missing RANGE equality on a multi-attribute key is named; CreateTable reports short index names together, GSIs first; SearchVectors answers a `null` member with `SerializationException`. Reported by @hicksy (paritysuite.org).
-- **DynamoDB — sizes and limits** — UpdateItem's 400KB check now charges the request clause by clause (19 per SET or ADD plus its target and operands, 1 per REMOVE or DELETE plus its target) before the condition is evaluated; the expression-size and nesting-depth messages take each operation's form; `SET l[n]` past the end of a list appends instead of padding with `NULL`; a write bills an attribute a vector index reads at its name plus four bytes per dimension; BatchExecuteStatement passes `ReturnValuesOnConditionCheckFailure` to its members. Reported by @hicksy (paritysuite.org).
+
+### Fixed
+- **Auth — requests without credentials are anonymous in every service** — under `AUTH=true` a request with no `Authorization` header and no query signature ran as the account root everywhere but S3; it is now refused in the service's own error, as on AWS: `MissingAuthenticationToken` 403 from Query and REST-XML services, `MissingAuthenticationTokenException` from JSON services, `AccessDenied` 403 from SQS, `UnrecognizedClientException` from Secrets Manager. Operations the SDKs send unsigned (Cognito sign-in, `GetId`, `AssumeRoleWithWebIdentity`), SNS `ConfirmSubscription` and `Unsubscribe`, API Gateway and Function URL invokes and S3 keep their behaviour. Contributed by @iot-rocket.
+- **Auth — a query signed for `s3` is refused by other services** — under `AUTH=true` a SigV4 presigned query with an `s3` credential scope skipped the signature check when another service served it; it is now refused with `SignatureDoesNotMatch` "Credential should be scoped to correct service: '<service>'. ", as AWS answers, while S3 keeps verifying its own presigned URLs. Reported by @iot-rocket.
+- **S3 — access points and Multi-Region Access Points under `AUTH=true`** — an access point policy was stored but never applied, and unsigned requests, SigV2 `Authorization` headers and `PutBucketPolicy`/`DeleteBucketPolicy` through an access point were served; a request through an access point alias or ARN is now allowed on the access point by an identity policy or its policy, an unsigned or SigV2-header one gets 400 `InvalidRequest`, bucket policy writes get 403 `AccessDenied`, and SigV2 presigned URLs for an access point ARN verify. MRAP requests are authorized on the member bucket and then on the MRAP. Contributed by @iot-rocket.
+- **S3 — ACLs and Block Public Access** — on a `BucketOwnerEnforced` bucket ACL writes answered 200 and a stored grant still opened the bucket; `PutBucketAcl` and `PutObjectAcl` now answer 400 `AccessControlListNotSupported`, uploads refuse any ACL but an owner-only one, and `CreateBucket` or a switch to `BucketOwnerEnforced` refuses a granting ACL with `InvalidBucketAclWithObjectOwnership`. Under `AUTH=true` a new bucket has all four Block Public Access settings on, as on AWS, so a public policy or ACL answers 403 `AccessDenied` and `CreateBucket` with a public ACL `InvalidBucketAclWithBlockPublicAccessError`; `RestrictPublicBuckets` drops a bucket policy only when it is public. Contributed by @iot-rocket.
+- **S3 — CORS under `AUTH=true`** — every preflight answered 200 with `Access-Control-Allow-Origin: *`, whatever the bucket's CORS configuration; a preflight now has to match one of the bucket's rules (403 `AccessForbidden` without a configuration or a match, 400 `BadRequest` without `Origin`) and gets that rule's headers. Contributed by @iot-rocket.
+- **S3 — presigned expiry and multipart object size** — under `AUTH=true` a presigned URL without `X-Amz-Expires` never expired; it now answers 400 `AuthorizationQueryParametersError` "Query-string authentication version 4 requires the X-Amz-Algorithm, X-Amz-Credential, X-Amz-Signature, X-Amz-Date, X-Amz-SignedHeaders, and X-Amz-Expires parameters.", and a negative value "X-Amz-Expires must be non-negative". `CompleteMultipartUpload` checks `x-amz-mp-object-size`: a size other than the parts' total answers 400 `InvalidRequest` "The provided 'x-amz-mp-object-size' header value 11 does not match what was computed: 10", a non-numeric one "Value for x-amz-mp-object-size header is invalid: 'abc'". Reported by @adzcodemi.
+- **CloudFront — origins that name a load balancer or a Function URL in this MiniStack** — a custom origin at a load balancer's `DNSName` or a Function URL's `<url-id>.lambda-url.<region>.on.aws` answered 502, as for any AWS hostname. A load balancer is now reached on its HTTP listener at the origin's `HTTPPort` and a Function URL over HTTPS, with the origin's custom headers, so a target that checks a secret header works; `OriginReadTimeout` applies (504). A Function URL refuses a forwarded viewer `Host` with 403 `{"Message":null}` and an unsigned `AWS_IAM` URL answers 403 `{"Message":"Forbidden"}`, as on AWS. Reported by @kinoh and @smiley-uriux.
+- **CloudFront — functions** — a viewer-request function that returned the request it was given rebuilt the query string, so `?flag` reached the origin as `?flag=`; it is now rebuilt only when the returned `querystring` differs. `CreateFunction` keeps its `Tags` (deleted with the function), answers invalid base64 code with `MalformedInput` "Invalid Base64 encoding", and refuses a missing code, configuration, runtime or name and invalid tags in one `MalformedXML` before the name lookup, then `InvalidTagging` for repeated, `aws:` or more than 50 keys. Contributed by @jeffreyheller and @kharabasz.
+- **ALB — `X-Forwarded-*` and `X-Amzn-Trace-Id` as the load balancer sets them** — targets got an incoming `X-Forwarded-For` unchanged and a client-sent `X-Forwarded-Proto`/`X-Forwarded-Port`, and Lambda targets none of them; all targets now get `X-Forwarded-For` with the client address appended (kept or dropped under `routing.http.xff_header_processing.mode` `preserve`/`remove`), `X-Forwarded-Proto` and `X-Forwarded-Port` from the listener, and an `X-Amzn-Trace-Id` with a new `Root`, or a `Self` added to or updated in the one sent. Reported by @kinoh.
+- **Lambda — Function URLs as AWS serves them** — URL ids are 32 lowercase letters and digits; the event's headers carry the URL's own `host`, `x-forwarded-for` (the first address sent, or the caller), `x-forwarded-port` `443`, `x-forwarded-proto` `https`, `x-amzn-trace-id` and the TLS fields; `requestContext.accountId` is `anonymous` for an `AuthType` `NONE` URL; and responses carry `x-amzn-RequestId` and `X-Amzn-Trace-Id`, with `x-amzn-ErrorType` `AccessDeniedException` on a 403. Reported by @smiley-uriux.
+- **Lambda — versions, layers and containers** — `Publish: true` on create or code update overwrote `$LATEST`'s `Version`, so `$LATEST` disappeared from `ListVersionsByFunction`; it now stays `$LATEST`. A layer MiniStack doesn't hold from another account, such as Datadog's, resolves as an empty layer instead of `AccessDeniedException`. Function containers no longer publish port 8080 on every host address: none with `DOCKER_NETWORK`, `127.0.0.1` otherwise. Contributed by @skialpine and @kharabasz. Reported by @cupidchan.
+- **API Gateway — HTTP API request ids and `AWS_IAM` headers** — an HTTP API's responses carried `x-amzn-requestid` and `x-amz-request-id`; they now carry `apigw-requestid`, 16 base64 characters, as AWS does. Under `AUTH=true` a REST `AWS_IAM` method served a bearer token or a lower-case algorithm; it now answers 403 `IncompleteSignatureException` with API Gateway's message. Contributed by @jeffreyheller and @iot-rocket.
+- **API Gateway v2 — WebSocket connections** — `DeleteConnection` answered 204 but left the socket open; it now closes it, `GetConnection` and `PostToConnection` answer 410 `GoneException`, and `$disconnect` runs after the close. A route whose Lambda is a container image or another runtime now runs it, `PostToConnection` sends a text frame, and routes and stages keep `RouteResponseSelectionExpression`, `ModelSelectionExpression`, `AccessLogSettings`, `ClientCertificateId` and `DeploymentId`. Contributed by @toby6578. Reported by @cupidchan.
+- **AppSync Events — `AUTH=true` checks the Event API's authorization** — an HTTP publish, a WebSocket connection or a subscription was accepted with any API key or none unless `APPSYNC_EVENTS_ENFORCE_AUTH` was set; it now needs what the API or channel namespace accepts (an API key, a SigV4 signature, a Cognito or OIDC token), and a refused publish answers 401 `UnauthorizedException` "You are not authorized to make this call.". Contributed by @iot-rocket.
+- **Cognito — signing keys, managed login and federation** — every pool signed with one shared key (`kid` `ministack-key-1`); each pool now has its own access and ID token keys at its JWKS URI. Managed login refuses PKCE `plain`, an unregistered `redirect_uri` and an unknown `grant_type` (400 `unsupported_grant_type`); a federated user joins `<user pool id>_<provider name>` and carries `identities`; an OIDC provider with only `oidc_issuer` takes its endpoints from the issuer's `/.well-known/openid-configuration`. Contributed by @kinoh. Reported by @cupidchan.
+- **Cognito — `client_credentials`** — `/oauth2/token` issued a token to a client without `client_credentials` in `AllowedOAuthFlows`; it now answers `invalid_grant` (`invalid_scope` first for an unknown scope), and the pre token generation trigger runs for this grant only when `LambdaVersion` is `V3_0`. Contributed by @iot-rocket.
+- **IoT — MQTT delivery as AWS IoT does** — an unacknowledged QoS 1 message was resent every 10 seconds with DUP set; it is now resent after 30 seconds with DUP 0, on the open connection and after a persistent session resumes, and a No Local subscription receives the client's own messages, as AWS IoT Core ignores that option. Contributed by @iot-rocket.
+- **IoT — thing-group membership by ARN** — `AddThingToThingGroup` and `RemoveThingFromThingGroup` ignored `thingGroupArn` and `thingArn` and were authorized on the request path; they now act on the group and thing the ARNs name, refuse a mismatched or foreign ARN with 400 `InvalidRequestException`, and under `AUTH=true` are checked on the group and then the thing. Contributed by @iot-rocket.
+- **CloudWatch — alarm tags and `DatapointsToAlarm`** — `PutMetricAlarm` and `PutCompositeAlarm` dropped `Tags` on create; they are now stored over CBOR, JSON 1.0 and Query and ignored on update, as on AWS, with AWS's `ValidationError`, `MalformedInput` and `aws:`-key refusals. An alarm without `DatapointsToAlarm` no longer reports it as `EvaluationPeriods`. Contributed by @kharabasz.
+- **DynamoDB — PartiQL statements are parsed as DynamoDB parses them** — a parser replaces the regular-expression reader and answers malformed statements with AWS's messages (`Statement wasn't well formed, can't be processed: Unexpected term`, `Missing right-hand side expression of infix operator`, `Aliasing is not supported`, positions as `at <line>:<column>:<length>`); it skips comments, reads every number literal form, and supports `ORDER BY`, `attribute_exists`, `attribute_not_exists`, `size`, `attribute_type`, `IS`, `set_add`, `set_delete`, `FROM ... WHERE ... SET` and `EXISTS(SELECT ...)` in `ExecuteTransaction`. Reported by @hicksy (paritysuite.org).
+- **DynamoDB — validation order, numbers and limits** — transactions and batches report their member errors together before any table lookup; numbers are range-checked with their full exponent (`Number overflow`) and duplicate set members reported in byte order; legacy `Expected` conditions are validated; UpdateItem's 400KB check charges the request clause by clause before the condition; `SET l[n]` past the end appends instead of padding with `NULL`. Reported by @hicksy (paritysuite.org).
+- **ElastiCache — transit encryption, AUTH tokens and subnet groups** — a group with `TransitEncryptionEnabled` and an `AuthToken` served plaintext without a password; it now serves TLS only, signed by the CA at `/_ministack/elasticache/ca.pem`, requires the token, and `ModifyReplicationGroup` applies `ROTATE`, `SET` and `DELETE`. `CreateCacheSubnetGroup` now keeps its subnets, VPC and tags, and answers `InvalidSubnet`, `CacheSubnetGroupAlreadyExists` and `CacheSubnetGroupInUse`. Reported by @aditya-drizz.
+- **ECS — capacity provider strategies** — a service or task with `capacityProviderStrategy` dropped it and reported `launchType` `EC2`; the strategy is now kept without a `launchType`, the cluster's default applies when neither is given, both together are refused with `InvalidParameterException` "Specifying both a launch type and capacity provider strategy is not supported. Remove one and try again.", and `AWS::ECS::Service` passes it through. Reported by @aditya-drizz.
+- **RDS — `MaxAllocatedStorage` only with storage autoscaling** — every DB instance reported `MaxAllocatedStorage` equal to its allocated storage; it is now present only when storage autoscaling was turned on. Reported by @aditya-drizz.
+- **Reset — `/_ministack/reset` no longer removes another MiniStack's containers** — the OpenSearch, OpenSearch Serverless, DSQL, ECS and EKS resets removed every container with the service's label on the Docker daemon; they now remove only this instance's. Contributed by @jeffreyheller.
## [1.5.31] — 2026-10-21
diff --git a/ministack/app.py b/ministack/app.py
index 666f6ef4..fb4d95fd 100644
--- a/ministack/app.py
+++ b/ministack/app.py
@@ -55,7 +55,7 @@ _CLOUDFRONT_HOST_RE = re.compile(
# FunctionUrl carries AWS's own `.on.aws` suffix, so we match any suffix rather
# than only _MINISTACK_HOST — pointing a proxy or an /etc/hosts entry at the
# AWS-shaped hostname is the whole point of addressing a function this way.
-_LAMBDA_URL_RE = re.compile(r"^([0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\.lambda-url\.[a-z0-9-]+\.")
+_LAMBDA_URL_RE = re.compile(r"^([a-z0-9]{32}|[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})\.lambda-url\.[a-z0-9-]+\.")
# OpenSearch Serverless collection endpoint: {collectionId}.{region}.aoss.<anything>[:port].
_AOSS_COLLECTION_HOST_RE = re.compile(r"^[a-z0-9]{3,40}\.[a-z0-9-]+\.aoss\.", re.IGNORECASE)
# AppSync Events realtime WebSocket: {apiId}.appsync-realtime-api.<anything>[:port].
@@ -1981,7 +1981,7 @@ async def _handle_cloudfront_dataplane_request(
if dist is None:
return None
try:
- return await _get_module("cloudfront_dataplane").handle_request(
+ return await _get_module("cloudfront").handle_viewer_request(
dist, method, path, raw_path, raw_query_string, headers, body, query_params, client_ip,
)
except Exception as e:
@@ -2136,7 +2136,8 @@ def _function_url_auth_target(url_id: str) -> tuple[str, dict, tuple | None]:
return function_arn, {"lambda:FunctionUrlAuthType": cfg.get("AuthType", "AWS_IAM")}, resolved
-async def _handle_lambda_url_request(host: str, path: str, method: str, headers: dict, body: bytes, query_params: dict):
+async def _handle_lambda_url_request(host: str, path: str, method: str, headers: dict, body: bytes, query_params: dict,
+ source_ip: str | None = None):
"""Handle Lambda Function URL data plane requests (Host-based + path-based)."""
parsed = _parse_lambda_url(host, path)
if parsed is None:
@@ -2154,7 +2155,7 @@ async def _handle_lambda_url_request(host: str, path: str, method: str, headers:
try:
# The handler reuses the lookup above; None makes it resolve again.
return await _get_module("lambda_svc").handle_function_url_request(
- url_id, method, function_path, headers, body, query_params, resolved=resolved,
+ url_id, method, function_path, headers, body, query_params, resolved=resolved, source_ip=source_ip,
)
except Exception as e:
logger.exception("Error in Lambda Function URL dispatch: %s", e)
@@ -2453,7 +2454,7 @@ def _authenticate_only(service: str, headers: dict, query_params: dict, request_
return None
from ministack.core.iam_evaluator import check_credentials
- failure = check_credentials(extract_access_key_id(headers, query_params))
+ failure = check_credentials(extract_access_key_id(headers, query_params), service)
return _credential_failure(service, "", request_id, failure, headers) if failure else None
@@ -2946,6 +2947,7 @@ async def _dispatch_service_request(
request_tag_context,
s3_access_point_resources,
s3_service_context,
+ verified_access_resource_arns,
)
from ministack.core.iam_evaluator import AuthError, enforce, pin_request_caller
from ministack.core.responses import get_account_id
@@ -3076,6 +3078,16 @@ async def _dispatch_service_request(
if denied:
resource_arn = extra_arn # named in the AccessDenied message
break
+ # A Verified Access call is also checked on every other resource it names.
+ if service == "ec2" and not denied:
+ for extra_arn in verified_access_resource_arns(routing_params, region, get_account_id())[1:]:
+ denied = enforce(
+ access_key, iam_action, service, region,
+ resource_arn=extra_arn, service_context=service_context,
+ )
+ if denied:
+ resource_arn = extra_arn
+ break
if (
denied
and service == "bedrock-agentcore"
diff --git a/ministack/core/iam_actions.py b/ministack/core/iam_actions.py
index f9e8515e..5f7452c3 100644
--- a/ministack/core/iam_actions.py
+++ b/ministack/core/iam_actions.py
@@ -964,20 +964,22 @@ _VERIFIED_ACCESS_ID_FIELDS = (
)
+def _va_arn(kind, rtype, value, region, account_id):
+ """The id's ARN when it exists, ``<type>/*`` when it doesn't or is a deleting endpoint (which answers NotFound)."""
+ from ministack.services import ec2
+
+ record = ec2._VA_STORES[kind].get_scoped(account_id, region, value)
+ if record is not None and record.get("Status") != "deleting":
+ return f"arn:aws:ec2:{region}:{account_id}:{rtype}/{value}"
+ return f"arn:aws:ec2:{region}:{account_id}:{rtype}/*"
+
+
def _verified_access_resource(action, query_params, region, account_id):
- """The resource AWS names in a Verified Access denial: the id's ARN when
- the resource exists, ``<type>/*`` when it does not. Attach and Detach move
- on to the trust provider when the instance id is malformed, and
- CreateVerifiedAccessEndpoint names the first subnet it knows, then the first
- security group it knows, then a known network interface, then the group, then ``verified-access-endpoint/*``."""
+ """The resource AWS names in a Verified Access denial (see verified_access_resource_arns for the rest)."""
from ministack.services import ec2
def name(kind, rtype, value):
- record = ec2._VA_STORES[kind].get_scoped(account_id, region, value)
- # A deleting endpoint answers NotFound, so it is named like a missing one.
- if record is not None and record.get("Status") != "deleting":
- return f"arn:aws:ec2:{region}:{account_id}:{rtype}/{value}"
- return f"arn:aws:ec2:{region}:{account_id}:{rtype}/*"
+ return _va_arn(kind, rtype, value, region, account_id)
new = _VERIFIED_ACCESS_CREATE_RESOURCES.get(action)
if new:
@@ -1013,6 +1015,53 @@ def _verified_access_resource(action, query_params, region, account_id):
return "*"
+# Every resource type AWS authorizes each call on (Service Authorization Reference).
+_VERIFIED_ACCESS_AUTHORIZED = {
+ "AttachVerifiedAccessTrustProvider": ("instance", "trust-provider"),
+ "DetachVerifiedAccessTrustProvider": ("instance", "trust-provider"),
+ "CreateVerifiedAccessGroup": ("new-group", "instance"),
+ "ModifyVerifiedAccessGroup": ("group", "instance"),
+ "CreateVerifiedAccessEndpoint": ("network-interface", "security-group", "subnet", "new-endpoint", "group"),
+ "ModifyVerifiedAccessEndpoint": ("subnet", "endpoint", "group"),
+}
+
+
+def verified_access_resource_arns(query_params: dict, region: str, account_id: str) -> list[str]:
+ """The resources a Verified Access call is authorized on: the one AWS names in a denial, then the others."""
+ action = _query_param(query_params, "Action")
+ if "VerifiedAccess" not in action or action.startswith("DescribeVerifiedAccess"):
+ return []
+
+ def listed(member):
+ lead = member + "."
+ return [v for k in query_params if k.startswith(lead) and (v := _query_param(query_params, k))]
+
+ def arns(kind):
+ if kind == "new-group":
+ return [f"arn:aws:ec2:{region}:{account_id}:verified-access-group/*"]
+ if kind == "new-endpoint":
+ return [f"arn:aws:ec2:{region}:{account_id}:verified-access-endpoint/*"]
+ if kind in ("instance", "trust-provider", "group", "endpoint"):
+ member, store = {"instance": ("VerifiedAccessInstanceId", "Instance"),
+ "trust-provider": ("VerifiedAccessTrustProviderId", "TrustProvider"),
+ "group": ("VerifiedAccessGroupId", "Group"),
+ "endpoint": ("VerifiedAccessEndpointId", "Endpoint")}[kind]
+ value = _query_param(query_params, member)
+ return [_va_arn(store, f"verified-access-{kind}", value, region, account_id)] if value else []
+ if kind == "subnet":
+ ids = listed("LoadBalancerOptions.SubnetId")
+ elif kind == "security-group":
+ ids = listed("SecurityGroupId")
+ else:
+ ids = [v for v in (_query_param(query_params, "NetworkInterfaceOptions.NetworkInterfaceId"),) if v]
+ return [f"arn:aws:ec2:{region}:{account_id}:{kind}/{v}" for v in ids]
+
+ resources = [_verified_access_resource(action, query_params, region, account_id)]
+ for kind in _VERIFIED_ACCESS_AUTHORIZED.get(action, ()):
+ resources += [arn for arn in arns(kind) if arn not in resources]
+ return resources
+
+
def extract_resource_arn(service: str, method: str, path: str,
headers: dict, body: bytes,
query_params: dict, region: str,
diff --git a/ministack/core/iam_evaluator.py b/ministack/core/iam_evaluator.py
index 2c7f04c3..eaf5fdc4 100644
--- a/ministack/core/iam_evaluator.py
+++ b/ministack/core/iam_evaluator.py
@@ -1255,6 +1255,8 @@ class SignedRequest:
# A SigV2 (HmacV1) header or query signature, checked against the path S3 signs.
sigv2: bool = False
auth_path: str = ""
+ # A SigV4 presigned query scoped to s3: S3 checks its own, any other service refuses its scope.
+ s3_scoped: bool = False
_signed_request: contextvars.ContextVar[SignedRequest | None] = contextvars.ContextVar(
@@ -1309,6 +1311,9 @@ def pin_signed_request(
_signed_request.set(SignedRequest(claimed, method, raw_path, query_params, headers, sigv2=True,
auth_path=f"/{bucket}{raw_path}" if bucket else raw_path))
return
+ if not authorization and claimed and not _is_root_key(claimed) and _s3_presigned_query(query_params):
+ _signed_request.set(SignedRequest(claimed, method, raw_path, query_params, {}, presigned=True, s3_scoped=True))
+ return
# A SigV4 signature that names no key is checked too: it is incomplete or names an unknown key.
names_no_key = not claimed and (authorization.startswith("AWS4-HMAC-SHA256") or presigned)
if not verifiable and not names_no_key and _signature_attempted(authorization, query_params) and not (
@@ -1430,7 +1435,21 @@ def pin_signed_body(body: bytes) -> None:
_signed_request.set(replace(request, body=body))
-def authenticate_handshake(scope: dict) -> AuthError | ResolvedCredential | None:
+def _s3_scope_failure(service: str | None) -> AuthError | None:
+ """An s3-scoped presigned query: S3 verifies it itself, any other service refuses the scope."""
+ request = _signed_request.get()
+ if request is None or not request.s3_scoped:
+ return None
+ _signed_request.set(None)
+ if service == "s3":
+ return None
+ from ministack.core.iam_actions import SERVICE_TO_IAM_NAMESPACE
+
+ name = SERVICE_TO_IAM_NAMESPACE.get(service or "", service or "")
+ return AuthError("SignatureDoesNotMatch", f"Credential should be scoped to correct service: '{name}'. ")
+
+
+def authenticate_handshake(scope: dict, service: str | None = None) -> AuthError | ResolvedCredential | None:
"""Check a WebSocket upgrade signed in its Authorization header or query string, as ``authenticate`` does."""
from urllib.parse import parse_qs, quote
@@ -1439,6 +1458,9 @@ def authenticate_handshake(scope: dict) -> AuthError | ResolvedCredential | None
raw_path = scope.get("raw_path") or quote(scope.get("path", "")).encode()
query_params = parse_qs(scope.get("query_string", b"").decode("utf-8", "replace"), keep_blank_values=True)
pin_signed_request("GET", raw_path.decode("latin-1"), query_params, scope.get("headers", []), handshake=True)
+ failure = _s3_scope_failure(service)
+ if failure:
+ return failure
request = _signed_request.get()
if request is None:
return None
@@ -1534,10 +1556,13 @@ def authenticate(access_key_id: str, account_id: str) -> AuthError | ResolvedCre
return credential
-def check_credentials(access_key_id: str) -> AuthError | None:
+def check_credentials(access_key_id: str, service: str | None = None) -> AuthError | None:
"""Authenticate a request no IAM action covers, with the checks enforce() makes before any policy."""
from ministack.core.responses import get_account_id
+ failure = _s3_scope_failure(service)
+ if failure:
+ return failure
account_id = get_account_id()
checked = authenticate(access_key_id or "", account_id)
if checked is None:
@@ -1637,6 +1662,9 @@ def enforce(access_key_id: str, iam_action: str, service: str,
from ministack.core.responses import get_account_id
account_id = get_account_id()
+ failure = _s3_scope_failure(service)
+ if failure:
+ return failure
# S3, API Gateway methods and function URLs can be called anonymously: an unsigned request is not checked.
anonymous_ok = service == "s3" or iam_action in ("execute-api:Invoke", "lambda:InvokeFunctionUrl")
if _anonymous.get() and not anonymous_ok and not _served_unsigned(service, iam_action):
diff --git a/ministack/services/alb.py b/ministack/services/alb.py
index 74b18d3c..37663c6f 100644
--- a/ministack/services/alb.py
+++ b/ministack/services/alb.py
@@ -37,7 +37,14 @@ from xml.sax.saxutils import escape as _esc
from ministack.core.arn import ArnParseError, parse_arn
from ministack.core.concurrency import run_reentrant
-from ministack.core.responses import AccountRegionScopedDict, AccountScopedDict, get_account_id, get_region, new_uuid
+from ministack.core.responses import (
+ AccountRegionScopedDict,
+ AccountScopedDict,
+ get_account_id,
+ get_region,
+ get_request_client_ip,
+ new_uuid,
+)
logger = logging.getLogger("alb")
@@ -2186,8 +2193,8 @@ async def _proxy_http_target(target, tg, method, path, headers, body, query_para
ip target groups, or any resolvable hostname for instance target
groups (an emulator has no EC2 metadata to resolve instance ids
against, so a hostname is the useful interpretation). Mirrors ALB
- behavior: hop-by-hop headers are stripped, X-Forwarded-* and
- X-Amzn-Trace-Id are injected, target HTTP errors pass through, and
+ behavior: hop-by-hop headers are stripped (dispatch_request has already
+ set X-Forwarded-* and X-Amzn-Trace-Id), target HTTP errors pass through, and
connection failures surface as 502.
"""
import http.client
@@ -2200,14 +2207,7 @@ async def _proxy_http_target(target, tg, method, path, headers, body, query_para
hop_by_hop = {"host", "content-length", "connection", "transfer-encoding",
"accept-encoding"}
- fwd = {k: v for k, v in (headers or {}).items() if k.lower() not in hop_by_hop}
- fwd["X-Forwarded-For"] = (headers or {}).get("x-forwarded-for") or "127.0.0.1"
- fwd.setdefault("X-Forwarded-Proto", "http")
- fwd.setdefault("X-Forwarded-Port", "80")
- fwd["X-Amzn-Trace-Id"] = "Root=1-%08x-%s" % (
- int(time.time()),
- "".join(random.choices("0123456789abcdef", k=24)),
- )
+ fwd = {k.lower(): v for k, v in (headers or {}).items() if k.lower() not in hop_by_hop}
data = body if isinstance(body, (bytes, bytearray)) else (body.encode() if body else None)
@@ -2215,8 +2215,9 @@ async def _proxy_http_target(target, tg, method, path, headers, body, query_para
# https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#http-504-issues
from ministack.core.responses import StreamingResponse
+ conn = http.client.HTTPConnection(host, port, timeout=TARGET_CONNECT_TIMEOUT)
+
def _open():
- conn = http.client.HTTPConnection(host, port, timeout=TARGET_CONNECT_TIMEOUT)
try:
conn.connect()
sock = conn.sock
@@ -2226,10 +2227,18 @@ async def _proxy_http_target(target, tg, method, path, headers, body, query_para
except Exception as e:
with contextlib.suppress(Exception):
conn.close()
- return None, None, None, e
- return conn, resp, sock, None
+ return None, None, e
+ return resp, sock, None
- conn, resp, sock, err = await asyncio.to_thread(_open)
+ opening = asyncio.ensure_future(asyncio.to_thread(_open))
+ try:
+ resp, sock, err = await asyncio.shield(opening)
+ except asyncio.CancelledError:
+ # The caller gave up: unblock the read and close once the thread returns.
+ with contextlib.suppress(Exception):
+ conn.sock.shutdown(socket.SHUT_RDWR)
+ opening.add_done_callback(lambda _: conn.close())
+ raise
if resp is None:
return (502, {"Content-Type": "application/json"},
json.dumps({"message": f"Target {host}:{port} connect error: {err}"}).encode())
@@ -2289,7 +2298,43 @@ async def _proxy_http_target(target, tg, method, path, headers, body, query_para
# Data-plane: main dispatcher
# ---------------------------------------------------------------------------
-async def dispatch_request(lb, method, path, headers, body, query_params, port=80):
+def _trace_id():
+ return "1-%08x-%s" % (int(time.time()), "".join(random.choices("0123456789abcdef", k=24)))
+
+
+def _lb_attr(lb_arn, key, default):
+ return next((a["Value"] for a in _lb_attrs.get(lb_arn, []) if a.get("Key") == key), default)
+
+
+def _target_headers(lb_arn, listener, headers, client_ip):
+ """The request fields a target receives: X-Forwarded-* and X-Amzn-Trace-Id set by the load balancer.
+
+ https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html
+ https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-request-tracing.html
+ """
+ out = {k.lower(): v for k, v in (headers or {}).items()}
+ mode = _lb_attr(lb_arn, "routing.http.xff_header_processing.mode", "append")
+ xff = out.pop("x-forwarded-for", None)
+ if mode == "preserve" and xff:
+ out["x-forwarded-for"] = xff
+ elif mode == "append":
+ # An IPv6 address appended to an existing header is bracketed.
+ out["x-forwarded-for"] = f"{xff}, [{client_ip}]" if xff and ":" in client_ip else (
+ f"{xff}, {client_ip}" if xff else client_ip)
+ out["x-forwarded-proto"] = (listener.get("Protocol") or "HTTP").lower()
+ out["x-forwarded-port"] = str(listener.get("Port", 80))
+ fields = [f.strip() for f in (out.get("x-amzn-trace-id") or "").split(";") if f.strip()]
+ if any(f.startswith("Self=") for f in fields):
+ fields = [f"Self={_trace_id()}" if f.startswith("Self=") else f for f in fields]
+ elif any(f.startswith("Root=") for f in fields):
+ fields.insert(0, f"Self={_trace_id()}")
+ else:
+ fields.insert(0, f"Root={_trace_id()}")
+ out["x-amzn-trace-id"] = ";".join(fields)
+ return out
+
+
+async def dispatch_request(lb, method, path, headers, body, query_params, port=80, client_ip=None):
lb_arn = lb["LoadBalancerArn"]
candidates = [l for l in _listeners.values() if l["LoadBalancerArn"] == lb_arn]
@@ -2317,8 +2362,10 @@ async def dispatch_request(lb, method, path, headers, body, query_params, port=8
if matched:
actions = rule.get("Actions") or listener.get("DefaultActions", [])
if actions:
+ target_headers = _target_headers(lb_arn, listener, headers,
+ client_ip or get_request_client_ip())
return await _execute_actions(actions, method, path,
- headers, body, query_params, port,
+ target_headers, body, query_params, port,
lb_arn=lb.get("LoadBalancerArn", ""))
return (502, {"Content-Type": "application/json"},
diff --git a/ministack/services/apigateway.py b/ministack/services/apigateway.py
index 969eb1ec..cd8d9e98 100644
--- a/ministack/services/apigateway.py
+++ b/ministack/services/apigateway.py
@@ -3088,7 +3088,7 @@ def _authorize_ws_iam(scope, connect_arn, headers, query_params, region):
access_key = extract_access_key_id(headers, query_params)
if not access_key:
return 403, "Missing Authentication Token"
- denied = AUTH and authenticate_handshake(scope)
+ denied = AUTH and authenticate_handshake(scope, "execute-api")
if not isinstance(denied, AuthError):
denied = AUTH and enforce(access_key, "execute-api:Invoke", "apigateway", region, resource_arn=connect_arn)
if isinstance(denied, AuthError):
diff --git a/ministack/services/cloudformation/provisioners.py b/ministack/services/cloudformation/provisioners.py
index 2781d2b8..6230740c 100644
--- a/ministack/services/cloudformation/provisioners.py
+++ b/ministack/services/cloudformation/provisioners.py
@@ -8665,7 +8665,8 @@ def _ecs_service_create(logical_id, props, stack_name):
"cluster": cluster,
"taskDefinition": props.get("TaskDefinition", ""),
"desiredCount": props.get("DesiredCount", 1),
- "launchType": props.get("LaunchType", "EC2"),
+ "launchType": props.get("LaunchType"),
+ "capacityProviderStrategy": _pascal_to_camel(props.get("CapacityProviderStrategy") or []),
"loadBalancers": _pascal_to_camel(props.get("LoadBalancers") or []),
"networkConfiguration": _pascal_to_camel(props.get("NetworkConfiguration") or {}),
"tags": [{"key": t["Key"], "value": t["Value"]} for t in props.get("Tags", [])],
@@ -8673,7 +8674,9 @@ def _ecs_service_create(logical_id, props, stack_name):
deployment_configuration = _ecs_deployment_configuration(props)
if deployment_configuration is not None:
request["deploymentConfiguration"] = deployment_configuration
- _ecs._create_service(request)
+ response = _ecs._create_service(request)
+ if response[0] >= 400:
+ raise ValueError(f"AWS::ECS::Service create failed: {response[2]!r}")
arn = f"arn:aws:ecs:{get_region()}:{get_account_id()}:service/{cluster}/{name}"
return arn, {"ServiceArn": arn, "Name": name}
@@ -8691,6 +8694,7 @@ def _ecs_service_update(physical_id, old_props, new_props, stack_name):
"HealthCheckGracePeriodSeconds": "healthCheckGracePeriodSeconds",
"EnableExecuteCommand": "enableExecuteCommand",
"PlatformVersion": "platformVersion",
+ "CapacityProviderStrategy": "capacityProviderStrategy",
}
for cf_property, ecs_property in property_map.items():
if new_props.get(cf_property) != old_props.get(cf_property):
diff --git a/ministack/services/cloudfront.py b/ministack/services/cloudfront.py
index 24103be4..48b13584 100644
--- a/ministack/services/cloudfront.py
+++ b/ministack/services/cloudfront.py
@@ -54,24 +54,39 @@ Supports:
ListDomainConflicts, UpdateDomainAssociation,
ListDistributionsByConnectionMode
Web ACL: AssociateDistributionWebACL, DisassociateDistributionWebACL
+
+Data plane (handle_viewer_request): serves viewer requests to <label>.cloudfront.net /
+<label>.cloudfront.<MINISTACK_HOST>. Not modelled: caching, WAF, logging, geo restrictions,
+signed URLs and cookies, Lambda@Edge, CORS preflight, synthetic CloudFront-Viewer-* headers,
+origins at AWS hostnames (502) other than a load balancer or Function URL in this MiniStack, which is
+served in-process.
"""
+import asyncio
import base64
import copy
+import fnmatch
+import http.client
import logging
+import os
import random
import re
+import ssl
import string
import unicodedata
import uuid
from datetime import datetime, timezone
-from urllib.parse import unquote
+from re import compile as _re_compile
+from re import escape as _re_escape
+from urllib.parse import parse_qs, quote, unquote, urlencode
from xml.etree.ElementTree import Element, SubElement, tostring
from defusedxml.ElementTree import fromstring
+from ministack.core import cloudfront_js
from ministack.core.arn import ArnParseError, parse_arn
-from ministack.core.responses import AccountScopedDict, get_account_id, new_uuid
+from ministack.core.concurrency import run_offloop, run_reentrant
+from ministack.core.responses import AccountScopedDict, StreamingResponse, get_account_id, new_uuid
logger = logging.getLogger("cloudfront")
@@ -206,7 +221,6 @@ def reset():
_vpc_origin_etags_issued.clear()
# Drop the JS workers so their compiled-function cache does not outlive a
# reset (mirrors appsync.reset()'s appsync_js.reset()).
- from ministack.core import cloudfront_js
cloudfront_js.reset()
@@ -5255,8 +5269,8 @@ def _list_distributions_by_connection_mode(mode):
# ---------------------------------------------------------------------------
-# Data-plane support: cloudfront_dataplane serves traffic through this
-# accessor surface rather than the module-private stores directly.
+# Data-plane support: handle_viewer_request reads distributions through
+# this accessor surface rather than the module-private stores directly.
# ---------------------------------------------------------------------------
@@ -5345,7 +5359,7 @@ def _dataplane_forwarded_values(behavior_el):
def _dataplane_allowed_methods(behavior_el):
"""The behavior's AllowedMethods.Items, or None when the block is absent
- (nothing to enforce — see cloudfront_dataplane.py's method check)."""
+ (nothing to enforce — see handle_viewer_request's method check)."""
am_el = _find(behavior_el, "AllowedMethods")
if am_el is None:
return None
@@ -5444,3 +5458,996 @@ def live_function_code(function_arn: str):
if not fn or not fn.get("live_etag"):
return None
return _function_view(fn, "LIVE").get("code", fn["code"])
+
+
+# ---------------------------------------------------------------------------
+# Data plane: viewer requests to a distribution
+# ---------------------------------------------------------------------------
+
+_HOP_BY_HOP_REQUEST_HEADERS = {"host", "cookie", "content-length", "connection", "transfer-encoding"}
+# Hop-by-hop headers dropped at the socket boundary (_forward_to_origin).
+_CONNECTION_MANAGEMENT_HEADERS = {"content-length", "connection", "transfer-encoding"}
+
+# Always forwarded, whatever the policies say (Developer Guide, request headers table).
+_ALWAYS_FORWARD_REQUEST_HEADERS = {
+ "range", "if-match", "if-modified-since", "if-none-match", "if-range", "if-unmodified-since",
+}
+
+# Connection-management fields dropped from an origin's response.
+_HOP_BY_HOP_RESPONSE_HEADERS = {"connection", "keep-alive", "transfer-encoding", "upgrade", "trailer"}
+
+
+class CloudFrontFunctionError(Exception):
+ """A CloudFront Function threw, timed out, or its code didn't define a
+ handler — CloudFront answers every one of these with a 503."""
+
+
+# ---------------------------------------------------------------------------
+# CloudFront Functions event <-> HTTP conversion (functions-event-structure.html)
+# ---------------------------------------------------------------------------
+
+
+def _title_case_header_name(name: str) -> str:
+ """"example-header-name" -> "Example-Header-Name", as CloudFront Functions title-case header names."""
+ return "-".join(part[:1].upper() + part[1:] if part[:1].isascii() else part for part in name.split("-"))
+
+
+def _event_items(values: dict) -> dict:
+ """A querystring/headers object: one field per name, carrying a
+ ``multiValue`` array when the same name repeated (functions-event-
+ structure.html, "Duplicate query strings, headers, and cookies")."""
+ out = {}
+ for name, value in values.items():
+ items = value if isinstance(value, list) else [value]
+ out[name] = {"value": items[0]}
+ if len(items) > 1:
+ out[name]["multiValue"] = [{"value": v} for v in items]
+ return out
+
+
+def _event_cookies_from_header(cookie_header: str) -> dict:
+ cookies = {}
+ for part in (cookie_header or "").split(";"):
+ name, sep, value = part.strip().partition("=")
+ if sep and name:
+ cookies[name] = {"value": value.strip()}
+ return cookies
+
+
+def _event_cookies_from_set_cookie(set_cookie) -> dict:
+ """The response.cookies event object built from raw Set-Cookie header values."""
+ values = set_cookie if isinstance(set_cookie, list) else ([set_cookie] if set_cookie else [])
+ cookies = {}
+ for raw in values:
+ name_value, _, attrs = raw.partition(";")
+ name, _, value = name_value.partition("=")
+ name = name.strip()
+ if not name:
+ continue
+ entry = {"value": value.strip()}
+ if attrs.strip():
+ entry["attributes"] = attrs.strip()
+ if name in cookies:
+ existing = cookies[name]
+ existing.setdefault("multiValue", [{k: v for k, v in existing.items() if k != "multiValue"}])
+ existing["multiValue"].append(entry)
+ else:
+ cookies[name] = entry
+ return cookies
+
+
+def _set_cookie_values_from_event(cookies: dict) -> list:
+ out = []
+ for name, field in (cookies or {}).items():
+ if not isinstance(field, dict):
+ continue
+ entries = field.get("multiValue") or [field]
+ for entry in entries:
+ line = f"{name}={entry.get('value', '')}"
+ if entry.get("attributes"):
+ line += f"; {entry['attributes']}"
+ out.append(line)
+ return out
+
+
+def _event_querystring(query_params: dict) -> dict:
+ return _event_items({name: values for name, values in (query_params or {}).items()})
+
+
+def _merge_function_headers(original_headers_obj: dict, returned_headers_obj: dict) -> dict:
+ """A name(lowercase)->value dict built from a function's returned headers object."""
+ original = original_headers_obj or {}
+ out = {}
+ for name, field in (returned_headers_obj or {}).items():
+ if not isinstance(field, dict):
+ continue
+ orig_field = original.get(name)
+ orig_multi = orig_field.get("multiValue") if isinstance(orig_field, dict) else None
+ new_multi = field.get("multiValue")
+ if new_multi and new_multi != orig_multi:
+ out[name.lower()] = [str(v.get("value", "")) for v in new_multi]
+ elif orig_multi:
+ out[name.lower()] = [str(field.get("value", ""))] + [str(v.get("value", "")) for v in orig_multi[1:]]
+ elif "value" in field:
+ out[name.lower()] = str(field["value"])
+ return out
+
+
+def _build_function_event(event_type: str, dist_id: str, dist_domain: str, request_id: str,
+ method: str, uri: str, query_params: dict, headers: dict, client_ip: str) -> dict:
+ request_headers = {k: v for k, v in headers.items() if k != "cookie"}
+ return {
+ "version": "1.0",
+ "context": {
+ "distributionDomainName": dist_domain,
+ "distributionId": dist_id,
+ "eventType": event_type,
+ "requestId": request_id,
+ },
+ # functions-event-structure.html: the TCP peer, not a header value.
+ "viewer": {"ip": client_ip},
+ "request": {
+ "method": method,
+ "uri": uri,
+ "querystring": _event_querystring(query_params),
+ "headers": _event_items(request_headers),
+ "cookies": _event_cookies_from_header(headers.get("cookie", "")),
+ },
+ }
+
+
+def _response_event_fields(status: int, reason: str, headers: dict) -> dict:
+ response_headers = {k: v for k, v in headers.items() if k != "set-cookie"}
+ return {
+ "statusCode": status,
+ "statusDescription": reason or "",
+ "headers": _event_items(response_headers),
+ "cookies": _event_cookies_from_set_cookie(headers.get("set-cookie")),
+ }
+
+
+def _body_from_response_event(response: dict, fallback: bytes) -> bytes:
+ body = response.get("body")
+ if body is None:
+ return fallback
+ if isinstance(body, str):
+ return body.encode("utf-8")
+ if isinstance(body, dict):
+ data = body.get("data", "")
+ if body.get("encoding") == "base64":
+ return base64.b64decode(data)
+ return str(data).encode("utf-8")
+ return fallback
+
+
+async def _run_function(function_arn: str, event: dict) -> dict:
+ code = live_function_code(function_arn)
+ if code is None:
+ raise CloudFrontFunctionError(f"{function_arn} has no published (LIVE) code")
+ try:
+ # A function cannot call back into MiniStack, so this is run_offloop work.
+ return await run_offloop(cloudfront_js.evaluate, code, event)
+ except cloudfront_js.CloudFrontFunctionError as exc:
+ raise CloudFrontFunctionError(str(exc)) from exc
+
+
+# ---------------------------------------------------------------------------
+# Cache behavior matching — CloudFront path-pattern glob, first match wins.
+# ---------------------------------------------------------------------------
+
+
+def _glob_to_regex(pattern: str):
+ # A leading / in a path pattern is optional (Developer Guide, "Path pattern").
+ if not pattern.startswith("/"):
+ pattern = "/" + pattern
+ out = [".*" if ch == "*" else "." if ch == "?" else _re_escape(ch) for ch in pattern]
+ return _re_compile("^" + "".join(out) + "$")
+
+
+def _match_behavior(parsed: dict, path: str):
+ for behavior in parsed["ordered_behaviors"]:
+ if _glob_to_regex(behavior["path_pattern"]).match(path):
+ return behavior
+ return parsed["default_behavior"]
+
+
+# ---------------------------------------------------------------------------
+# Origin-request forwarding: the cache policy / origin request policy combination table (Developer Guide).
+# ---------------------------------------------------------------------------
+
+
+def _behavior_kind(behavior: str, names, lowercase: bool = False):
+ """A cache or origin request policy behavior value as a (kind, set) pair."""
+ names = [n.lower() for n in (names or [])] if lowercase else list(names or [])
+ if behavior in ("all", "allViewer", "allViewerAndWhitelistCloudFront"):
+ return ("all", None)
+ if behavior == "whitelist":
+ return ("whitelist", set(names))
+ if behavior == "allExcept":
+ return ("allExcept", set(names))
+ return ("none", None)
+
+
+def _combine_forward(cache_kind_set, orp_kind_set, name: str) -> bool:
+ """Whether ``name`` reaches the origin, per the cache-policy / origin-request-policy combination table."""
+ cache_kind, cache_set = cache_kind_set
+ if cache_kind == "all":
+ return True # a cache policy's "all" always wins, even over an ORP block list
+ orp_kind, orp_set = orp_kind_set if orp_kind_set is not None else ("none", None)
+ if orp_kind == "all":
+ return True
+ if cache_kind == "none":
+ if orp_kind == "whitelist":
+ return name in orp_set
+ if orp_kind == "allExcept":
+ return name not in orp_set
+ return False
+ if cache_kind == "whitelist":
+ if name in cache_set:
+ return True
+ if orp_kind == "whitelist":
+ return name in orp_set
+ if orp_kind == "allExcept":
+ return name not in orp_set
+ return False
+ if cache_kind == "allExcept":
+ if name not in cache_set:
+ if orp_kind == "allExcept":
+ return name not in orp_set
+ return True # cache policy already allows it through unless ORP also blocks it
+ return orp_kind == "whitelist" and name in orp_set # ORP allow-list rescues a cache-blocked name
+ return False
+
+
+def _legacy_predicates(fv: dict):
+ """Forwarding predicates for a legacy ``ForwardedValues`` behavior (no CachePolicyId)."""
+ header_names = {n.lower() for n in (fv.get("headers") or [])}
+ forward_all_headers = "*" in (fv.get("headers") or [])
+ cookies_forward = fv.get("cookies_forward", "none")
+ cookies_whitelist = fv.get("cookies_whitelist") or []
+ forward_qs = bool(fv.get("query_string"))
+
+ def header_allowed(name):
+ return forward_all_headers or name.lower() in header_names
+
+ def cookie_allowed(name):
+ if cookies_forward == "all":
+ return True
+ if cookies_forward == "whitelist":
+ return any(fnmatch.fnmatchcase(name, pattern) for pattern in cookies_whitelist)
+ return False
+
+ def qs_allowed(_name):
+ return forward_qs
+
+ return header_allowed, cookie_allowed, qs_allowed
+
+
+def _policy_predicates(behavior: dict):
+ cache_params = cache_policy_params(behavior.get("cache_policy_id")) or {}
+ orp_cfg = (
+ origin_request_policy_config(behavior["origin_request_policy_id"])
+ if behavior.get("origin_request_policy_id") else None
+ )
+
+ cache_headers = _behavior_kind(cache_params.get("HeaderBehavior", "none"), cache_params.get("Headers"), lowercase=True)
+ cache_cookies = _behavior_kind(cache_params.get("CookieBehavior", "none"), cache_params.get("Cookies"))
+ cache_qs = _behavior_kind(cache_params.get("QueryStringBehavior", "none"), cache_params.get("QueryStrings"))
+
+ if orp_cfg is not None:
+ orp_headers = _behavior_kind(orp_cfg["HeaderBehavior"], orp_cfg["Headers"], lowercase=True)
+ orp_cookies = _behavior_kind(orp_cfg["CookieBehavior"], orp_cfg["Cookies"])
+ orp_qs = _behavior_kind(orp_cfg["QueryStringBehavior"], orp_cfg["QueryStrings"])
+ else:
+ orp_headers = orp_cookies = orp_qs = None
+
+ def header_allowed(name):
+ return _combine_forward(cache_headers, orp_headers, name.lower())
+
+ def cookie_allowed(name):
+ return _combine_forward(cache_cookies, orp_cookies, name)
+
+ def qs_allowed(name):
+ return _combine_forward(cache_qs, orp_qs, name)
+
+ return header_allowed, cookie_allowed, qs_allowed
+
+
+def _forwarding_predicates(behavior: dict):
+ if behavior.get("cache_policy_id"):
+ return _policy_predicates(behavior)
+ if behavior.get("forwarded_values") is not None:
+ return _legacy_predicates(behavior["forwarded_values"])
+ # Neither a cache policy nor ForwardedValues: only the defaults (Host, User-Agent).
+ return (lambda _n: False), (lambda _n: False), (lambda _n: False)
+
+
+# ---------------------------------------------------------------------------
+# Response-headers policy
+# ---------------------------------------------------------------------------
+
+
+def _cors_origin_allowed(allow_origins: list, viewer_origin: str) -> bool:
+ """Whether ``viewer_origin`` matches an AccessControlAllowOrigins entry ("*" or a glob)."""
+ return any(p == "*" or fnmatch.fnmatchcase(viewer_origin, p) for p in allow_origins or [])
+
+
+def _apply_cors_headers(headers: dict, cors_cfg: dict, viewer_origin: str, method: str) -> dict:
+ """Apply a response headers policy's CORS headers (Developer Guide, "CORS headers")."""
+ if not cors_cfg or not viewer_origin or not _cors_origin_allowed(cors_cfg.get("AllowOrigins"), viewer_origin):
+ return headers
+ result = dict(headers)
+ override = cors_cfg.get("OriginOverride", False)
+
+ def _set(name, value):
+ key = name.lower()
+ if key in result and not override:
+ return
+ result[key] = value
+
+ _set("access-control-allow-origin", "*" if "*" in (cors_cfg.get("AllowOrigins") or []) else viewer_origin)
+ if cors_cfg.get("AllowCredentials"):
+ _set("access-control-allow-credentials", "true")
+ if cors_cfg.get("ExposeHeaders"):
+ _set("access-control-expose-headers", ", ".join(cors_cfg["ExposeHeaders"]))
+ if method == "OPTIONS":
+ if cors_cfg.get("AllowHeaders"):
+ _set("access-control-allow-headers", ", ".join(cors_cfg["AllowHeaders"]))
+ if cors_cfg.get("AllowMethods"):
+ _set("access-control-allow-methods", ", ".join(cors_cfg["AllowMethods"]))
+ if cors_cfg.get("MaxAgeSec") is not None:
+ _set("access-control-max-age", str(cors_cfg["MaxAgeSec"]))
+ return result
+
+
+def _apply_response_headers_policy(headers: dict, policy_cfg: dict, viewer_origin: str, method: str) -> dict:
+ if not policy_cfg:
+ return headers
+ result = _apply_cors_headers(headers, policy_cfg.get("Cors"), viewer_origin, method)
+
+ def _set(name, value, override):
+ key = name.lower()
+ if key in result and not override:
+ return
+ result[key] = value
+
+ # CustomHeaders/RemoveHeaders are None when the policy never supplied that block.
+ for item in policy_cfg.get("CustomHeaders") or []:
+ _set(item["Header"], item["Value"], item["Override"])
+
+ sec = policy_cfg.get("Security") or {}
+ if "ContentTypeOptions" in sec:
+ _set("x-content-type-options", "nosniff", sec["ContentTypeOptions"]["Override"])
+ if "FrameOptions" in sec:
+ _set("x-frame-options", sec["FrameOptions"]["FrameOption"], sec["FrameOptions"]["Override"])
+ if "XSSProtection" in sec:
+ xp = sec["XSSProtection"]
+ value = "1" if xp["Protection"] else "0"
+ if xp["Protection"] and xp.get("ModeBlock"):
+ value += "; mode=block"
+ if xp.get("ReportUri"):
+ value += f"; report={xp['ReportUri']}"
+ _set("x-xss-protection", value, xp["Override"])
+ if "ReferrerPolicy" in sec:
+ _set("referrer-policy", sec["ReferrerPolicy"]["ReferrerPolicy"], sec["ReferrerPolicy"]["Override"])
+ if "ContentSecurityPolicy" in sec:
+ _set("content-security-policy", sec["ContentSecurityPolicy"]["ContentSecurityPolicy"],
+ sec["ContentSecurityPolicy"]["Override"])
+ if "StrictTransportSecurity" in sec:
+ hsts = sec["StrictTransportSecurity"]
+ value = f"max-age={hsts['AccessControlMaxAgeSec']}"
+ if hsts.get("IncludeSubdomains"):
+ value += "; includeSubDomains"
+ if hsts.get("Preload"):
+ value += "; preload"
+ _set("strict-transport-security", value, hsts["Override"])
+
+ for item in policy_cfg.get("RemoveHeaders") or []:
+ result.pop(item["Header"].lower(), None)
+
+ return result
+
+
+# ---------------------------------------------------------------------------
+# CloudFront's own added headers and synthesized error pages
+# ---------------------------------------------------------------------------
+
+
+def _cf_request_id() -> str:
+ return base64.b64encode(os.urandom(42)).decode().rstrip("=")
+
+
+def _cf_via_hash() -> str:
+ return "".join(random.choices(string.ascii_lowercase + string.digits, k=13))
+
+
+def _add_cloudfront_headers(headers: dict, x_cache: str, request_id: str | None = None) -> dict:
+ result = dict(headers)
+ result["via"] = f"1.1 {_cf_via_hash()}.cloudfront.net (CloudFront)"
+ result["x-cache"] = x_cache
+ result["x-amz-cf-pop"] = "LOC50-C1"
+ result["x-amz-cf-id"] = request_id or _cf_request_id()
+ return result
+
+
+def _render_headers(headers: dict, title_case: bool) -> dict:
+ if not title_case:
+ return dict(headers)
+ return {_title_case_header_name(name): value for name, value in headers.items()}
+
+
+# The fixed body of CloudFront's own error pages ("The request could not be satisfied").
+_CF_ERROR_TEMPLATE = """<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
+<TITLE>ERROR: The request could not be satisfied</TITLE>
+</HEAD><BODY>
+<H1>{status} ERROR</H1>
+<H2>The request could not be satisfied.</H2>
+<HR noshade size="1px">
+{explanation}
+<BR clear="all">
+<HR noshade size="1px">
+<PRE>
+Generated by cloudfront (CloudFront)
+Request ID: {request_id}
+</PRE>
+</BODY></HTML>"""
+
+
+def _cf_error_response(status: int, explanation: str, x_cache: str) -> tuple:
+ request_id = _cf_request_id()
+ body = _CF_ERROR_TEMPLATE.format(status=status, explanation=explanation,
+ request_id=request_id).encode("utf-8")
+ headers = _add_cloudfront_headers({"content-type": "text/html"}, x_cache, request_id=request_id)
+ return status, _render_headers(headers, title_case=True), body
+
+
+# A failed origin TLS handshake is a 502 (Developer Guide, http-502-bad-gateway).
+_ERROR_X_CACHE = "Error from cloudfront"
+
+
+# ---------------------------------------------------------------------------
+# Origin forwarding
+# ---------------------------------------------------------------------------
+
+
+class OriginUnreachable(Exception):
+ pass
+
+
+class OriginTimeout(Exception):
+ """The origin didn't respond within OriginReadTimeout."""
+
+
+# An origin addressed by an AWS-owned hostname is refused with 502, so no traffic reaches real AWS.
+_AWS_OWNED_HOST_SUFFIXES = (".amazonaws.com", ".amazonaws.com.cn", ".on.aws", ".api.aws", ".cloudfront.net")
+
+
+def _normalize_origin_host(host: str) -> str:
+ h = (host or "").strip().lower()
+ return h[:-1] if h.endswith(".") else h
+
+
+def _is_aws_owned_host(host: str) -> bool:
+ return any(host == suffix.removeprefix(".") or host.endswith(suffix) for suffix in _AWS_OWNED_HOST_SUFFIXES)
+
+
+def _origin_connect_target(origin: dict, viewer_is_https: bool) -> tuple:
+ """(connect_host, connect_port, use_https) — the origin's own
+ DomainName and HTTPPort/HTTPSPort, dialed exactly as configured."""
+ policy = origin["protocol_policy"]
+ use_https = policy == "https-only" or (policy == "match-viewer" and viewer_is_https)
+ port = origin["https_port"] if use_https else origin["http_port"]
+ return origin["domain_name"], port, use_https
+
+
+def _origin_ssl_context() -> ssl.SSLContext:
+ """System roots, plus the gateway's own TLS cert under USE_SSL
+ (lambda_svc.py's container CA-trust precedent) — a custom https origin
+ may itself be a MiniStack-served endpoint sharing that cert."""
+ from ministack.core import tls
+
+ ctx = ssl.create_default_context()
+ if tls.use_ssl_enabled():
+ ctx.load_verify_locations(tls.resolve_tls_material()[0])
+ ctx.verify_flags |= ssl.VERIFY_X509_PARTIAL_CHAIN
+ return ctx
+
+
+def _set_header_ci(headers: dict, name: str, value) -> None:
+ """Set ``name: value``, replacing any case-insensitive match."""
+ for existing in [k for k in headers if k.lower() == name.lower()]:
+ del headers[existing]
+ headers[name] = value
+
+
+def _origin_x_forwarded_for(viewer_xff: str, client_ip: str) -> str:
+ """CloudFront Developer Guide, "Client IP addresses": appends the TCP
+ peer's address to an existing viewer-sent X-Forwarded-For, or adds a
+ fresh one carrying just that address."""
+ return f"{viewer_xff}, {client_ip}" if viewer_xff else client_ip
+
+
+def _origin_request_headers(origin: dict, headers: dict, client_ip: str, viewer_x_forwarded_for: str) -> dict:
+ """The header fields sent to a custom origin, one value per name."""
+ fwd_headers = {k: v for k, v in headers.items() if k not in _CONNECTION_MANAGEMENT_HEADERS}
+ for name, value in origin.get("custom_headers") or []:
+ _set_header_ci(fwd_headers, name, value)
+ # Host is the origin's DomainName unless the policies forward the viewer's Host.
+ _set_header_ci(fwd_headers, "Host", headers.get("host", origin["domain_name"]))
+ _set_header_ci(fwd_headers, "Via", f"1.1 {_cf_via_hash()}.cloudfront.net (CloudFront)")
+ _set_header_ci(fwd_headers, "X-Amz-Cf-Id", _cf_request_id())
+ # Always appended (Developer Guide, "Client IP addresses").
+ _set_header_ci(fwd_headers, "X-Forwarded-For",
+ _origin_x_forwarded_for(viewer_x_forwarded_for, client_ip))
+ # RFC 9110 §5.3: a repeated field value is one comma-joined line.
+ return {
+ name: ", ".join(str(v) for v in value) if isinstance(value, list) else value
+ for name, value in fwd_headers.items()
+ }
+
+
+def _forward_to_origin(origin: dict, method: str, uri: str, query_string: str,
+ headers: dict, body: bytes, viewer_is_https: bool, client_ip: str = "127.0.0.1",
+ viewer_x_forwarded_for: str = "") -> tuple:
+ """Blocking: run under core.concurrency.run_reentrant (the origin may be
+ MiniStack itself). Returns (status, reason, headers, body)."""
+ conn_host, conn_port, use_https = _origin_connect_target(origin, viewer_is_https)
+ if _is_aws_owned_host(_normalize_origin_host(conn_host)):
+ raise OriginUnreachable(f"refusing to dial AWS-owned origin host {conn_host!r}")
+ target_path = uri + (f"?{query_string}" if query_string else "")
+ send_headers = _origin_request_headers(origin, headers, client_ip, viewer_x_forwarded_for)
+
+ conn_cls = http.client.HTTPSConnection if use_https else http.client.HTTPConnection
+ timeout = origin.get("read_timeout", 30)
+ conn = (
+ conn_cls(conn_host, conn_port, timeout=timeout, context=_origin_ssl_context())
+ if use_https else conn_cls(conn_host, conn_port, timeout=timeout)
+ )
+ try:
+ conn.request(method, target_path, body=body if body else None, headers=send_headers)
+ resp = conn.getresponse()
+ data = resp.read()
+ return resp.status, resp.reason, resp.getheaders(), data
+ except TimeoutError as e:
+ raise OriginTimeout(str(e)) from e
+ except Exception as e:
+ # An unreachable origin, or an untrusted or self-signed origin certificate.
+ raise OriginUnreachable(str(e)) from e
+ finally:
+ conn.close()
+
+
+# ---------------------------------------------------------------------------
+# S3 origin — served from MiniStack's own S3 store in-process, never a socket.
+# ---------------------------------------------------------------------------
+
+
+def _s3_object_key(origin: dict, uri: str) -> str:
+ """OriginPath prefixed to the request URI, as CloudFront builds the origin path."""
+ return unquote((origin.get("origin_path") or "") + uri).lstrip("/")
+
+
+def _serve_s3_origin(origin: dict, method: str, uri: str, headers: dict, dist_arn: str = "") -> tuple:
+ """Fetch from an S3 origin in-process (blocking; run off the event loop)."""
+ from ministack.services import s3
+
+ return s3.serve_cloudfront_origin_fetch(
+ origin["s3_bucket"], _s3_object_key(origin, uri), method, headers,
+ source_arn=dist_arn if origin.get("oac_id") else "", oai_id=origin.get("oai_id", ""))
+
+
+# ---------------------------------------------------------------------------
+# Registered ALB origin: its AWS-owned DNSName is served in-process, never dialed.
+# ---------------------------------------------------------------------------
+
+
+def _registered_alb(origin: dict):
+ """The load balancer an origin's AWS-owned DomainName belongs to, or None."""
+ host = _normalize_origin_host(origin["domain_name"])
+ if not _is_aws_owned_host(host):
+ return None
+ # Imported here so distributions without such an origin never load the ALB module.
+ from ministack.services import alb
+
+ return alb.find_lb_for_host(host)
+
+
+async def _fetch_from_alb_origin(lb: dict, origin: dict, method: str, uri: str, query_string: str, headers: dict,
+ body: bytes, viewer_is_https: bool, client_ip: str, viewer_xff: str) -> tuple:
+ """(status, reason, headers, body) from the listener on the origin's port; raises OriginTimeout /
+ OriginUnreachable. The listener's rules and targets answer, as they would a request on the wire."""
+ from ministack.services import alb
+
+ _host, port, use_https = _origin_connect_target(origin, viewer_is_https)
+ # Without a TLS handshake there is no certificate for CloudFront to check, so only HTTP is served.
+ if use_https or not any(
+ listener["LoadBalancerArn"] == lb["LoadBalancerArn"] and listener.get("Port") == port
+ and listener.get("Protocol") == "HTTP" for listener in alb._listeners.values()
+ ):
+ raise OriginUnreachable(f"no HTTP listener on port {port} of {lb['DNSName']}")
+
+ # The ALB data plane takes the gateway's request shape: lowercase header names, parsed query.
+ alb_headers = {k.lower(): v for k, v in _origin_request_headers(origin, headers, client_ip, viewer_xff).items()}
+ read_timeout = origin.get("read_timeout", 30)
+ try:
+ status, response_headers, data = await asyncio.wait_for(
+ alb.dispatch_request(lb, method, uri, alb_headers, body or b"",
+ parse_qs(query_string, keep_blank_values=True), port,
+ client_ip="127.0.0.1"),
+ read_timeout,
+ )
+ except asyncio.TimeoutError as e:
+ raise OriginTimeout(f"no response from {lb['DNSName']} within {read_timeout}s") from e
+ except Exception as e:
+ raise OriginUnreachable(str(e)) from e
+ if isinstance(data, StreamingResponse):
+ data = await _collect_streaming_body(data, read_timeout)
+ pairs = [(name, value) for name, values in response_headers.items()
+ for value in (values if isinstance(values, list) else [values])]
+ return status, http.client.responses.get(status, ""), pairs, data
+
+
+def _registered_function_url(origin: dict):
+ """The Function URL host an origin's AWS-owned DomainName names, when that URL exists in this MiniStack."""
+ host = _normalize_origin_host(origin["domain_name"])
+ if not (_is_aws_owned_host(host) and ".lambda-url." in host):
+ return None
+ from ministack import app
+ from ministack.services import lambda_svc
+
+ parsed = app._parse_lambda_url(host, "/")
+ return host if parsed and lambda_svc.resolve_function_url(parsed[0]) is not None else None
+
+
+async def _fetch_from_function_url_origin(host: str, origin: dict, method: str, uri: str, query_string: str,
+ headers: dict, body: bytes, viewer_is_https: bool, client_ip: str,
+ viewer_xff: str) -> tuple:
+ """(status, reason, headers, body) from a Function URL, served as a request to its host would be."""
+ from ministack import app
+
+ _host, _port, use_https = _origin_connect_target(origin, viewer_is_https)
+ # A Function URL only answers HTTPS.
+ if not use_https:
+ raise OriginUnreachable(f"{host} does not answer HTTP")
+ url_headers = {k.lower(): v for k, v in _origin_request_headers(origin, headers, client_ip, viewer_xff).items()}
+ # A Function URL refuses a Host other than its own, as when a policy forwards the viewer's.
+ if url_headers.get("host", "").split(":", 1)[0].lower() != host:
+ return 403, "Forbidden", [("Content-Type", "application/json"), ("x-amzn-RequestId", new_uuid()),
+ ("x-amzn-ErrorType", "AccessDeniedException")], b'{"Message":null}'
+ read_timeout = origin.get("read_timeout", 30)
+ try:
+ status, response_headers, data = await asyncio.wait_for(
+ app._handle_lambda_url_request(host, uri, method, url_headers, body or b"",
+ parse_qs(query_string, keep_blank_values=True), source_ip="127.0.0.1"),
+ read_timeout,
+ )
+ except asyncio.TimeoutError as e:
+ raise OriginTimeout(f"no response from {host} within {read_timeout}s") from e
+ except Exception as e:
+ raise OriginUnreachable(str(e)) from e
+ pairs = [(name, value) for name, values in response_headers.items()
+ for value in (values if isinstance(values, list) else [values])]
+ return status, http.client.responses.get(status, ""), pairs, data
+
+
+async def _collect_streaming_body(stream: StreamingResponse, read_timeout: float) -> bytes:
+ """Buffer a relayed body; ``read_timeout`` bounds each chunk, and a missing final frame is a failure."""
+ frames: asyncio.Queue = asyncio.Queue()
+ finished = object()
+
+ async def receive():
+ # No viewer disconnect to report; the whole body is buffered.
+ await asyncio.Event().wait()
+
+ async def run():
+ try:
+ await stream.runner(frames.put, receive)
+ finally:
+ frames.put_nowait(finished)
+
+ runner = asyncio.ensure_future(run())
+ chunks, complete = [], False
+ try:
+ while (frame := await asyncio.wait_for(frames.get(), read_timeout)) is not finished:
+ chunks.append(frame.get("body", b""))
+ complete = not frame.get("more_body", False)
+ except asyncio.TimeoutError as e:
+ raise OriginTimeout(f"no response data within {read_timeout}s") from e
+ finally:
+ # Cancelling the runner closes the target connection behind it.
+ runner.cancel()
+ await asyncio.gather(runner, return_exceptions=True)
+ if not complete:
+ raise OriginUnreachable("origin response ended before its final body frame")
+ return b"".join(chunks)
+
+
+# ---------------------------------------------------------------------------
+# Multi-value headers: a repeated name (e.g. Set-Cookie) maps to a list.
+# ---------------------------------------------------------------------------
+
+
+async def _fetch_from_origin(origin: dict, method: str, uri: str, query_string: str, headers: dict, body: bytes,
+ viewer_is_https: bool, client_ip: str, viewer_xff: str, dist_arn: str) -> tuple:
+ """(status, reason, headers, body) from an S3 or custom origin; raises OriginTimeout / OriginUnreachable."""
+ if origin.get("s3_bucket"):
+ status, header_pairs, data = await run_offloop(_serve_s3_origin, origin, method, uri, headers, dist_arn)
+ pairs = header_pairs.items() if isinstance(header_pairs, dict) else header_pairs
+ return status, http.client.responses.get(status, ""), _multidict_from_pairs(pairs), data
+ lb = _registered_alb(origin)
+ if lb is not None:
+ status, reason, header_pairs, data = await _fetch_from_alb_origin(
+ lb, origin, method, (origin.get("origin_path") or "") + uri, query_string, headers, body,
+ viewer_is_https, client_ip, viewer_xff,
+ )
+ return status, reason, _multidict_from_pairs(header_pairs), data
+ url_host = _registered_function_url(origin)
+ if url_host is not None:
+ status, reason, header_pairs, data = await _fetch_from_function_url_origin(
+ url_host, origin, method, (origin.get("origin_path") or "") + uri, query_string, headers, body,
+ viewer_is_https, client_ip, viewer_xff,
+ )
+ return status, reason, _multidict_from_pairs(header_pairs), data
+ # The origin may be MiniStack's own gateway, so the call must be reentrant.
+ status, reason, header_pairs, data = await run_reentrant(
+ _forward_to_origin, origin, method, (origin.get("origin_path") or "") + uri, query_string, headers, body,
+ viewer_is_https, client_ip, viewer_xff,
+ )
+ return status, reason, _multidict_from_pairs(header_pairs), data
+
+
+def _multidict_from_pairs(pairs) -> dict:
+ out: dict = {}
+ for name, value in pairs:
+ key = name.lower()
+ if key in out:
+ existing = out[key]
+ if isinstance(existing, list):
+ existing.append(value)
+ else:
+ out[key] = [existing, value]
+ else:
+ out[key] = value
+ return out
+
+
+def _strip_hop_by_hop_response_headers(headers: dict) -> dict:
+ """Connection management is this module's concern, not the origin's."""
+ return {k: v for k, v in headers.items() if k not in _HOP_BY_HOP_RESPONSE_HEADERS
+ and not k.startswith("proxy-")}
+
+
+# ---------------------------------------------------------------------------
+# Behavior-level settings: ViewerProtocolPolicy, AllowedMethods, DefaultRootObject
+# ---------------------------------------------------------------------------
+
+
+def _check_viewer_protocol_policy(behavior: dict, method: str, viewer_is_https: bool, request_uri: str,
+ query_string: str, headers: dict):
+ """A redirect/forbid response, or None to continue."""
+ policy = behavior.get("viewer_protocol_policy", "allow-all")
+ if viewer_is_https or policy == "allow-all":
+ return None
+ if policy == "redirect-to-https":
+ host = headers.get("host", "")
+ location = f"https://{host}{request_uri}" + (f"?{query_string}" if query_string else "")
+ redirect_status = 301 if method in ("GET", "HEAD") else 307
+ status, resp_headers, body = _cf_error_response(redirect_status, "The document has moved.", _ERROR_X_CACHE)
+ resp_headers["Location"] = location
+ return status, resp_headers, body
+ if policy == "https-only":
+ return _cf_error_response(403, "Viewers must use HTTPS to request this object.", _ERROR_X_CACHE)
+ return None
+
+
+def _check_allowed_methods(behavior: dict, method: str):
+ allowed = behavior.get("allowed_methods")
+ if allowed is not None and method not in allowed:
+ # CloudFront's 403 page text for this case (see _CF_ERROR_TEMPLATE).
+ return _cf_error_response(
+ 403,
+ "This distribution is not configured to allow the HTTP request method that was used for this "
+ "request. The distribution supports only cachable requests.",
+ _ERROR_X_CACHE,
+ )
+ return None
+
+
+def _apply_default_root_object(default_root_object: str, request_uri: str) -> str:
+ """CloudFront Developer Guide, "Default root object": the configured
+ object is requested from the origin when a viewer requests the
+ distribution's root URL."""
+ if default_root_object and request_uri == "/":
+ return "/" + default_root_object
+ return request_uri
+
+
+def _check_stacked_distribution(headers: dict):
+ """403 for a request that already passed through this hop (Developer Guide, stacked distributions)."""
+ if "(cloudfront)" in headers.get("via", "").lower():
+ return _cf_error_response(
+ 403, "This distribution is configured to serve another CloudFront distribution as its origin.",
+ _ERROR_X_CACHE,
+ )
+ return None
+
+
+# ---------------------------------------------------------------------------
+# Main entry point
+# ---------------------------------------------------------------------------
+
+
+async def handle_viewer_request(dist: dict, method: str, path: str, raw_uri: str, raw_query_string: str,
+ headers: dict, body: bytes, query_params: dict, client_ip: str = "127.0.0.1") -> tuple:
+ """Serve one viewer request against ``dist`` (a cloudfront.py distribution record)."""
+ stacked_response = _check_stacked_distribution(headers)
+ if stacked_response is not None:
+ return stacked_response
+
+ dist_id = dist["Id"]
+ dist_domain = dist["DomainName"]
+ request_id = new_uuid()
+ viewer_is_https = headers.get("x-forwarded-proto", "http") == "https"
+
+ parsed = parse_distribution_dataplane_config(dist)
+ behavior = _match_behavior(parsed, path)
+ if behavior is None:
+ return _cf_error_response(502, "The request could not be satisfied.", _ERROR_X_CACHE)
+
+ protocol_response = _check_viewer_protocol_policy(
+ behavior, method, viewer_is_https, raw_uri, raw_query_string, headers,
+ )
+ if protocol_response is not None:
+ return protocol_response
+ methods_response = _check_allowed_methods(behavior, method)
+ if methods_response is not None:
+ return methods_response
+
+ request_headers = dict(headers)
+ request_uri = _apply_default_root_object(parsed.get("default_root_object", ""), raw_uri)
+ request_query = dict(query_params or {})
+ literal_query_string = None # set when a function rearranges querystring into a literal string
+ function_touched_querystring = False # set when a function returns a changed querystring object
+
+ request_function_arn = behavior["functions"].get("viewer-request")
+ if request_function_arn:
+ event = _build_function_event(
+ "viewer-request", dist_id, dist_domain, request_id,
+ method, request_uri, request_query, request_headers, client_ip,
+ )
+ try:
+ result = await _run_function(request_function_arn, event)
+ except CloudFrontFunctionError:
+ logger.exception("viewer-request function error (distribution=%s)", dist_id)
+ return _cf_error_response(503, "The Lambda function failed to execute.", _ERROR_X_CACHE)
+
+ if isinstance(result, dict) and "statusCode" in result:
+ # A function-generated response goes straight to the viewer, skipping the origin.
+ resp_headers = _merge_function_headers(None, result.get("headers"))
+ resp_headers = _add_cloudfront_headers(resp_headers, "FunctionGeneratedResponse from cloudfront")
+ set_cookie_values = _set_cookie_values_from_event(result.get("cookies"))
+ if set_cookie_values:
+ resp_headers["set-cookie"] = (
+ set_cookie_values if len(set_cookie_values) > 1 else set_cookie_values[0]
+ )
+ # statusDescription is dropped: ASGI carries no reason phrase.
+ resp_body = _body_from_response_event(result, b"")
+ return result["statusCode"], _render_headers(resp_headers, title_case=True), resp_body
+
+ # Request headers stay lowercase internally; responses are Title-Cased once, on return.
+ req = result if isinstance(result, dict) else event["request"]
+ request_uri = req.get("uri", request_uri)
+ request_headers = _merge_function_headers(event["request"]["headers"], req.get("headers"))
+ cookie_pairs = [f"{n}={(f.get('value') if isinstance(f, dict) else f)}"
+ for n, f in (req.get("cookies") or {}).items()]
+ if cookie_pairs:
+ request_headers["cookie"] = "; ".join(cookie_pairs)
+ qs_field = req.get("querystring")
+ if isinstance(qs_field, str):
+ # A function rewrote the querystring as a literal string (functions-event-structure).
+ literal_query_string = qs_field
+ request_query = {}
+ elif qs_field is not None and qs_field != event["request"]["querystring"]:
+ function_touched_querystring = True
+ request_query = {}
+ for name, field in qs_field.items():
+ if not isinstance(field, dict):
+ continue
+ values = [field.get("value", "")]
+ values += [v.get("value", "") for v in field.get("multiValue", [])[1:]]
+ request_query[name] = values
+
+ origin = parsed["origins"].get(behavior["target_origin_id"])
+ if origin is None:
+ return _cf_error_response(502, "The request could not be satisfied.", _ERROR_X_CACHE)
+
+ header_allowed, cookie_allowed, qs_allowed = _forwarding_predicates(behavior)
+ request_cookies = _event_cookies_from_header(request_headers.get("cookie", ""))
+ forward_host = header_allowed("host")
+
+ fwd_headers = {
+ name: value for name, value in request_headers.items()
+ if name not in _HOP_BY_HOP_REQUEST_HEADERS
+ and (header_allowed(name) or name in _ALWAYS_FORWARD_REQUEST_HEADERS)
+ }
+ if header_allowed("user-agent") and "user-agent" in request_headers:
+ fwd_headers["user-agent"] = request_headers["user-agent"]
+ else:
+ # User-Agent is "Amazon CloudFront" unless forwarded (Developer Guide, "User-Agent header").
+ fwd_headers["user-agent"] = "Amazon CloudFront"
+ if forward_host:
+ fwd_headers["host"] = request_headers.get("host", "")
+ cookie_pairs = [f"{name}={field['value']}" for name, field in request_cookies.items() if cookie_allowed(name)]
+ if cookie_pairs:
+ fwd_headers["cookie"] = "; ".join(cookie_pairs)
+ if literal_query_string is not None:
+ forward_qs = literal_query_string
+ elif not function_touched_querystring and all(qs_allowed(name) for name in request_query):
+ # Unrewritten and fully forwarded: keep the viewer's bytes.
+ forward_qs = raw_query_string
+ else:
+ forward_qs = urlencode(
+ [(name, v) for name, values in request_query.items() if qs_allowed(name) for v in values],
+ quote_via=quote,
+ )
+
+ try:
+ status, reason, origin_headers, origin_body = await _fetch_from_origin(
+ origin, method, request_uri, forward_qs, fwd_headers, body, viewer_is_https,
+ client_ip, request_headers.get("x-forwarded-for", ""), dist.get("ARN", ""),
+ )
+ origin_status = status
+ error_response = parsed["custom_error_responses"].get(status)
+ if error_response:
+ # The page is requested through the cache behavior its path matches (Developer Guide,
+ # "Store objects and custom error pages in different locations").
+ origin = parsed["origins"].get(_match_behavior(parsed, error_response["page"])["target_origin_id"])
+ if origin is None:
+ return _cf_error_response(502, "The request could not be satisfied.", _ERROR_X_CACHE)
+ status, reason, origin_headers, origin_body = await _fetch_from_origin(
+ origin, "GET", error_response["page"], "", {"user-agent": "Amazon CloudFront"}, b"",
+ viewer_is_https, client_ip, "", dist.get("ARN", ""),
+ )
+ # An unavailable page is answered with the status its origin returned (Developer Guide,
+ # "Generate custom error responses").
+ if status < 400:
+ status = error_response["response_code"]
+ except OriginTimeout:
+ logger.warning("Origin timed out for distribution=%s origin=%s", dist_id, origin["domain_name"])
+ return _cf_error_response(504, "The request could not be satisfied.", _ERROR_X_CACHE)
+ except OriginUnreachable:
+ logger.warning("Origin unreachable for distribution=%s origin=%s", dist_id, origin["domain_name"])
+ return _cf_error_response(502, "The request could not be satisfied.", _ERROR_X_CACHE)
+
+ origin_headers = _strip_hop_by_hop_response_headers(origin_headers)
+ policy_cfg = response_headers_policy_config(behavior.get("response_headers_policy_id"))
+ response_headers = _apply_response_headers_policy(
+ origin_headers, policy_cfg, request_headers.get("origin", ""), method,
+ )
+ response_body = origin_body
+
+ response_function_arn = behavior["functions"].get("viewer-response")
+ # No viewer-response function runs when the origin answers 400 or above.
+ if response_function_arn and origin_status < 400:
+ event = _build_function_event(
+ "viewer-response", dist_id, dist_domain, request_id,
+ method, request_uri, request_query, headers, client_ip,
+ )
+ event["response"] = _response_event_fields(status, reason, response_headers)
+ try:
+ result = await _run_function(response_function_arn, event)
+ except CloudFrontFunctionError:
+ logger.exception("viewer-response function error (distribution=%s)", dist_id)
+ return _cf_error_response(503, "The Lambda function failed to execute.", _ERROR_X_CACHE)
+ response = result if isinstance(result, dict) else event["response"]
+ status = response.get("statusCode", status)
+ # Replace, not merge: a header the function deleted stays gone.
+ response_headers = _merge_function_headers(event["response"]["headers"], response.get("headers"))
+ set_cookie_values = _set_cookie_values_from_event(response.get("cookies"))
+ if set_cookie_values:
+ response_headers["set-cookie"] = set_cookie_values if len(set_cookie_values) > 1 else set_cookie_values[0]
+ response_body = _body_from_response_event(response, origin_body)
+
+ final_headers = _add_cloudfront_headers(response_headers, "Miss from cloudfront")
+ return status, _render_headers(final_headers, title_case=True), response_body
diff --git a/ministack/services/cloudfront_dataplane.py b/ministack/services/cloudfront_dataplane.py
deleted file mode 100644
index e10d4751..00000000
--- a/ministack/services/cloudfront_dataplane.py
+++ /dev/null
@@ -1,873 +0,0 @@
-# Copyright (c) 2026 MiniStack Contributors. SPDX-License-Identifier: MIT
-# Copies or substantial portions, including AI-assisted ports or rewrites, must retain this notice (see LICENSE).
-"""CloudFront data plane: serves viewer requests to ``<label>.cloudfront.net`` / ``<label>.cloudfront.<MINISTACK_HOST>``.
-
-Not modelled: caching, WAF, logging, geo restrictions, signed URLs and cookies,
-Lambda@Edge, CORS preflight, synthetic ``CloudFront-Viewer-*`` headers, origins at AWS hostnames (502).
-"""
-
-import base64
-import fnmatch
-import http.client
-import logging
-import os
-import random
-import ssl
-import string
-from re import compile as _re_compile
-from re import escape as _re_escape
-from urllib.parse import quote, unquote, urlencode
-
-from ministack.core import cloudfront_js
-from ministack.core.concurrency import run_offloop, run_reentrant
-from ministack.core.responses import new_uuid
-from ministack.services import cloudfront, s3
-
-logger = logging.getLogger("cloudfront_dataplane")
-
-_HOP_BY_HOP_REQUEST_HEADERS = {"host", "cookie", "content-length", "connection", "transfer-encoding"}
-# Hop-by-hop headers dropped at the socket boundary (_forward_to_origin).
-_CONNECTION_MANAGEMENT_HEADERS = {"content-length", "connection", "transfer-encoding"}
-
-# Always forwarded, whatever the policies say (Developer Guide, request headers table).
-_ALWAYS_FORWARD_REQUEST_HEADERS = {
- "range", "if-match", "if-modified-since", "if-none-match", "if-range", "if-unmodified-since",
-}
-
-# Dropped from an origin's response before it reaches the viewer — connection
-# management is this module's concern, not the origin's.
-_HOP_BY_HOP_RESPONSE_HEADERS = {"connection", "keep-alive", "transfer-encoding", "upgrade", "trailer"}
-
-
-class CloudFrontFunctionError(Exception):
- """A CloudFront Function threw, timed out, or its code didn't define a
- handler — CloudFront answers every one of these with a 503."""
-
-
-# ---------------------------------------------------------------------------
-# CloudFront Functions event <-> HTTP conversion (functions-event-structure.html)
-# ---------------------------------------------------------------------------
-
-
-def _title_case_header_name(name: str) -> str:
- """"example-header-name" -> "Example-Header-Name", as CloudFront Functions title-case header names."""
- return "-".join(part[:1].upper() + part[1:] if part[:1].isascii() else part for part in name.split("-"))
-
-
-def _event_items(values: dict) -> dict:
- """A querystring/headers object: one field per name, carrying a
- ``multiValue`` array when the same name repeated (functions-event-
- structure.html, "Duplicate query strings, headers, and cookies")."""
- out = {}
- for name, value in values.items():
- items = value if isinstance(value, list) else [value]
- out[name] = {"value": items[0]}
- if len(items) > 1:
- out[name]["multiValue"] = [{"value": v} for v in items]
- return out
-
-
-def _event_cookies_from_header(cookie_header: str) -> dict:
- cookies = {}
- for part in (cookie_header or "").split(";"):
- name, sep, value = part.strip().partition("=")
- if sep and name:
- cookies[name] = {"value": value.strip()}
- return cookies
-
-
-def _event_cookies_from_set_cookie(set_cookie) -> dict:
- """The response.cookies event object built from raw Set-Cookie header values."""
- values = set_cookie if isinstance(set_cookie, list) else ([set_cookie] if set_cookie else [])
- cookies = {}
- for raw in values:
- name_value, _, attrs = raw.partition(";")
- name, _, value = name_value.partition("=")
- name = name.strip()
- if not name:
- continue
- entry = {"value": value.strip()}
- if attrs.strip():
- entry["attributes"] = attrs.strip()
- if name in cookies:
- existing = cookies[name]
- existing.setdefault("multiValue", [{k: v for k, v in existing.items() if k != "multiValue"}])
- existing["multiValue"].append(entry)
- else:
- cookies[name] = entry
- return cookies
-
-
-def _set_cookie_values_from_event(cookies: dict) -> list:
- out = []
- for name, field in (cookies or {}).items():
- if not isinstance(field, dict):
- continue
- entries = field.get("multiValue") or [field]
- for entry in entries:
- line = f"{name}={entry.get('value', '')}"
- if entry.get("attributes"):
- line += f"; {entry['attributes']}"
- out.append(line)
- return out
-
-
-def _event_querystring(query_params: dict) -> dict:
- return _event_items({name: values for name, values in (query_params or {}).items()})
-
-
-def _merge_function_headers(original_headers_obj: dict, returned_headers_obj: dict) -> dict:
- """A name(lowercase)->value dict built from a function's returned headers object."""
- original = original_headers_obj or {}
- out = {}
- for name, field in (returned_headers_obj or {}).items():
- if not isinstance(field, dict):
- continue
- orig_field = original.get(name)
- orig_multi = orig_field.get("multiValue") if isinstance(orig_field, dict) else None
- new_multi = field.get("multiValue")
- if new_multi and new_multi != orig_multi:
- out[name.lower()] = [str(v.get("value", "")) for v in new_multi]
- elif orig_multi:
- out[name.lower()] = [str(field.get("value", ""))] + [str(v.get("value", "")) for v in orig_multi[1:]]
- elif "value" in field:
- out[name.lower()] = str(field["value"])
- return out
-
-
-def _build_function_event(event_type: str, dist_id: str, dist_domain: str, request_id: str,
- method: str, uri: str, query_params: dict, headers: dict, client_ip: str) -> dict:
- request_headers = {k: v for k, v in headers.items() if k != "cookie"}
- return {
- "version": "1.0",
- "context": {
- "distributionDomainName": dist_domain,
- "distributionId": dist_id,
- "eventType": event_type,
- "requestId": request_id,
- },
- # functions-event-structure.html: the TCP peer, not a header value.
- "viewer": {"ip": client_ip},
- "request": {
- "method": method,
- "uri": uri,
- "querystring": _event_querystring(query_params),
- "headers": _event_items(request_headers),
- "cookies": _event_cookies_from_header(headers.get("cookie", "")),
- },
- }
-
-
-def _response_event_fields(status: int, reason: str, headers: dict) -> dict:
- response_headers = {k: v for k, v in headers.items() if k != "set-cookie"}
- return {
- "statusCode": status,
- "statusDescription": reason or "",
- "headers": _event_items(response_headers),
- "cookies": _event_cookies_from_set_cookie(headers.get("set-cookie")),
- }
-
-
-def _body_from_response_event(response: dict, fallback: bytes) -> bytes:
- body = response.get("body")
- if body is None:
- return fallback
- if isinstance(body, str):
- return body.encode("utf-8")
- if isinstance(body, dict):
- data = body.get("data", "")
- if body.get("encoding") == "base64":
- return base64.b64decode(data)
- return str(data).encode("utf-8")
- return fallback
-
-
-async def _run_function(function_arn: str, event: dict) -> dict:
- code = cloudfront.live_function_code(function_arn)
- if code is None:
- raise CloudFrontFunctionError(f"{function_arn} has no published (LIVE) code")
- try:
- # A function cannot call back into MiniStack, so this is run_offloop work.
- return await run_offloop(cloudfront_js.evaluate, code, event)
- except cloudfront_js.CloudFrontFunctionError as exc:
- raise CloudFrontFunctionError(str(exc)) from exc
-
-
-# ---------------------------------------------------------------------------
-# Cache behavior matching — CloudFront path-pattern glob, first match wins.
-# ---------------------------------------------------------------------------
-
-
-def _glob_to_regex(pattern: str):
- # A leading / in a path pattern is optional (Developer Guide, "Path pattern").
- if not pattern.startswith("/"):
- pattern = "/" + pattern
- out = [".*" if ch == "*" else "." if ch == "?" else _re_escape(ch) for ch in pattern]
- return _re_compile("^" + "".join(out) + "$")
-
-
-def _match_behavior(parsed: dict, path: str):
- for behavior in parsed["ordered_behaviors"]:
- if _glob_to_regex(behavior["path_pattern"]).match(path):
- return behavior
- return parsed["default_behavior"]
-
-
-# ---------------------------------------------------------------------------
-# Origin-request forwarding: the cache policy / origin request policy combination table (Developer Guide).
-# ---------------------------------------------------------------------------
-
-
-def _behavior_kind(behavior: str, names, lowercase: bool = False):
- """A cache or origin request policy behavior value as a (kind, set) pair."""
- names = [n.lower() for n in (names or [])] if lowercase else list(names or [])
- if behavior in ("all", "allViewer", "allViewerAndWhitelistCloudFront"):
- return ("all", None)
- if behavior == "whitelist":
- return ("whitelist", set(names))
- if behavior == "allExcept":
- return ("allExcept", set(names))
- return ("none", None)
-
-
-def _combine_forward(cache_kind_set, orp_kind_set, name: str) -> bool:
- """Whether ``name`` reaches the origin, per the cache-policy / origin-request-policy combination table."""
- cache_kind, cache_set = cache_kind_set
- if cache_kind == "all":
- return True # a cache policy's "all" always wins, even over an ORP block list
- orp_kind, orp_set = orp_kind_set if orp_kind_set is not None else ("none", None)
- if orp_kind == "all":
- return True
- if cache_kind == "none":
- if orp_kind == "whitelist":
- return name in orp_set
- if orp_kind == "allExcept":
- return name not in orp_set
- return False
- if cache_kind == "whitelist":
- if name in cache_set:
- return True
- if orp_kind == "whitelist":
- return name in orp_set
- if orp_kind == "allExcept":
- return name not in orp_set
- return False
- if cache_kind == "allExcept":
- if name not in cache_set:
- if orp_kind == "allExcept":
- return name not in orp_set
- return True # cache policy already allows it through unless ORP also blocks it
- return orp_kind == "whitelist" and name in orp_set # ORP allow-list rescues a cache-blocked name
- return False
-
-
-def _legacy_predicates(fv: dict):
- """Forwarding predicates for a legacy ``ForwardedValues`` behavior (no CachePolicyId)."""
- header_names = {n.lower() for n in (fv.get("headers") or [])}
- forward_all_headers = "*" in (fv.get("headers") or [])
- cookies_forward = fv.get("cookies_forward", "none")
- cookies_whitelist = fv.get("cookies_whitelist") or []
- forward_qs = bool(fv.get("query_string"))
-
- def header_allowed(name):
- return forward_all_headers or name.lower() in header_names
-
- def cookie_allowed(name):
- if cookies_forward == "all":
- return True
- if cookies_forward == "whitelist":
- return any(fnmatch.fnmatchcase(name, pattern) for pattern in cookies_whitelist)
- return False
-
- def qs_allowed(_name):
- return forward_qs
-
- return header_allowed, cookie_allowed, qs_allowed
-
-
-def _policy_predicates(behavior: dict):
- cache_params = cloudfront.cache_policy_params(behavior.get("cache_policy_id")) or {}
- orp_cfg = (
- cloudfront.origin_request_policy_config(behavior["origin_request_policy_id"])
- if behavior.get("origin_request_policy_id") else None
- )
-
- cache_headers = _behavior_kind(cache_params.get("HeaderBehavior", "none"), cache_params.get("Headers"), lowercase=True)
- cache_cookies = _behavior_kind(cache_params.get("CookieBehavior", "none"), cache_params.get("Cookies"))
- cache_qs = _behavior_kind(cache_params.get("QueryStringBehavior", "none"), cache_params.get("QueryStrings"))
-
- if orp_cfg is not None:
- orp_headers = _behavior_kind(orp_cfg["HeaderBehavior"], orp_cfg["Headers"], lowercase=True)
- orp_cookies = _behavior_kind(orp_cfg["CookieBehavior"], orp_cfg["Cookies"])
- orp_qs = _behavior_kind(orp_cfg["QueryStringBehavior"], orp_cfg["QueryStrings"])
- else:
- orp_headers = orp_cookies = orp_qs = None
-
- def header_allowed(name):
- return _combine_forward(cache_headers, orp_headers, name.lower())
-
- def cookie_allowed(name):
- return _combine_forward(cache_cookies, orp_cookies, name)
-
- def qs_allowed(name):
- return _combine_forward(cache_qs, orp_qs, name)
-
- return header_allowed, cookie_allowed, qs_allowed
-
-
-def _forwarding_predicates(behavior: dict):
- if behavior.get("cache_policy_id"):
- return _policy_predicates(behavior)
- if behavior.get("forwarded_values") is not None:
- return _legacy_predicates(behavior["forwarded_values"])
- # Neither a cache policy nor legacy ForwardedValues: nothing beyond the
- # defaults every origin request always carries (Host, User-Agent).
- return (lambda _n: False), (lambda _n: False), (lambda _n: False)
-
-
-# ---------------------------------------------------------------------------
-# Response-headers policy
-# ---------------------------------------------------------------------------
-
-
-def _cors_origin_allowed(allow_origins: list, viewer_origin: str) -> bool:
- """Whether ``viewer_origin`` matches an AccessControlAllowOrigins entry ("*" or a glob)."""
- return any(p == "*" or fnmatch.fnmatchcase(viewer_origin, p) for p in allow_origins or [])
-
-
-def _apply_cors_headers(headers: dict, cors_cfg: dict, viewer_origin: str, method: str) -> dict:
- """Apply a response headers policy's CORS headers (Developer Guide, "CORS headers")."""
- if not cors_cfg or not viewer_origin or not _cors_origin_allowed(cors_cfg.get("AllowOrigins"), viewer_origin):
- return headers
- result = dict(headers)
- override = cors_cfg.get("OriginOverride", False)
-
- def _set(name, value):
- key = name.lower()
- if key in result and not override:
- return
- result[key] = value
-
- _set("access-control-allow-origin", "*" if "*" in (cors_cfg.get("AllowOrigins") or []) else viewer_origin)
- if cors_cfg.get("AllowCredentials"):
- _set("access-control-allow-credentials", "true")
- if cors_cfg.get("ExposeHeaders"):
- _set("access-control-expose-headers", ", ".join(cors_cfg["ExposeHeaders"]))
- if method == "OPTIONS":
- if cors_cfg.get("AllowHeaders"):
- _set("access-control-allow-headers", ", ".join(cors_cfg["AllowHeaders"]))
- if cors_cfg.get("AllowMethods"):
- _set("access-control-allow-methods", ", ".join(cors_cfg["AllowMethods"]))
- if cors_cfg.get("MaxAgeSec") is not None:
- _set("access-control-max-age", str(cors_cfg["MaxAgeSec"]))
- return result
-
-
-def _apply_response_headers_policy(headers: dict, policy_cfg: dict, viewer_origin: str, method: str) -> dict:
- if not policy_cfg:
- return headers
- result = _apply_cors_headers(headers, policy_cfg.get("Cors"), viewer_origin, method)
-
- def _set(name, value, override):
- key = name.lower()
- if key in result and not override:
- return
- result[key] = value
-
- # CustomHeaders/RemoveHeaders are None when the policy never supplied that block.
- for item in policy_cfg.get("CustomHeaders") or []:
- _set(item["Header"], item["Value"], item["Override"])
-
- sec = policy_cfg.get("Security") or {}
- if "ContentTypeOptions" in sec:
- _set("x-content-type-options", "nosniff", sec["ContentTypeOptions"]["Override"])
- if "FrameOptions" in sec:
- _set("x-frame-options", sec["FrameOptions"]["FrameOption"], sec["FrameOptions"]["Override"])
- if "XSSProtection" in sec:
- xp = sec["XSSProtection"]
- value = "1" if xp["Protection"] else "0"
- if xp["Protection"] and xp.get("ModeBlock"):
- value += "; mode=block"
- if xp.get("ReportUri"):
- value += f"; report={xp['ReportUri']}"
- _set("x-xss-protection", value, xp["Override"])
- if "ReferrerPolicy" in sec:
- _set("referrer-policy", sec["ReferrerPolicy"]["ReferrerPolicy"], sec["ReferrerPolicy"]["Override"])
- if "ContentSecurityPolicy" in sec:
- _set("content-security-policy", sec["ContentSecurityPolicy"]["ContentSecurityPolicy"],
- sec["ContentSecurityPolicy"]["Override"])
- if "StrictTransportSecurity" in sec:
- hsts = sec["StrictTransportSecurity"]
- value = f"max-age={hsts['AccessControlMaxAgeSec']}"
- if hsts.get("IncludeSubdomains"):
- value += "; includeSubDomains"
- if hsts.get("Preload"):
- value += "; preload"
- _set("strict-transport-security", value, hsts["Override"])
-
- for item in policy_cfg.get("RemoveHeaders") or []:
- result.pop(item["Header"].lower(), None)
-
- return result
-
-
-# ---------------------------------------------------------------------------
-# CloudFront's own added headers and synthesized error pages
-# ---------------------------------------------------------------------------
-
-
-def _cf_request_id() -> str:
- return base64.b64encode(os.urandom(42)).decode().rstrip("=")
-
-
-def _cf_via_hash() -> str:
- return "".join(random.choices(string.ascii_lowercase + string.digits, k=13))
-
-
-def _add_cloudfront_headers(headers: dict, x_cache: str, request_id: str | None = None) -> dict:
- result = dict(headers)
- result["via"] = f"1.1 {_cf_via_hash()}.cloudfront.net (CloudFront)"
- result["x-cache"] = x_cache
- result["x-amz-cf-pop"] = "LOC50-C1"
- result["x-amz-cf-id"] = request_id or _cf_request_id()
- return result
-
-
-def _render_headers(headers: dict, title_case: bool) -> dict:
- if not title_case:
- return dict(headers)
- return {_title_case_header_name(name): value for name, value in headers.items()}
-
-
-# The fixed body of CloudFront's own error pages ("The request could not be satisfied").
-_CF_ERROR_TEMPLATE = """<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
-<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
-<TITLE>ERROR: The request could not be satisfied</TITLE>
-</HEAD><BODY>
-<H1>{status} ERROR</H1>
-<H2>The request could not be satisfied.</H2>
-<HR noshade size="1px">
-{explanation}
-<BR clear="all">
-<HR noshade size="1px">
-<PRE>
-Generated by cloudfront (CloudFront)
-Request ID: {request_id}
-</PRE>
-</BODY></HTML>"""
-
-
-def _cf_error_response(status: int, explanation: str, x_cache: str) -> tuple:
- request_id = _cf_request_id()
- body = _CF_ERROR_TEMPLATE.format(status=status, explanation=explanation,
- request_id=request_id).encode("utf-8")
- headers = _add_cloudfront_headers({"content-type": "text/html"}, x_cache, request_id=request_id)
- return status, _render_headers(headers, title_case=True), body
-
-
-# A failed origin TLS handshake is a 502 (Developer Guide, http-502-bad-gateway).
-_ERROR_X_CACHE = "Error from cloudfront"
-
-
-# ---------------------------------------------------------------------------
-# Origin forwarding
-# ---------------------------------------------------------------------------
-
-
-class OriginUnreachable(Exception):
- pass
-
-
-class OriginTimeout(Exception):
- """The origin didn't respond within OriginReadTimeout."""
-
-
-# An origin addressed by an AWS-owned hostname is refused with 502, so no traffic reaches real AWS.
-_AWS_OWNED_HOST_SUFFIXES = (".amazonaws.com", ".amazonaws.com.cn", ".on.aws", ".api.aws", ".cloudfront.net")
-
-
-def _normalize_origin_host(host: str) -> str:
- h = (host or "").strip().lower()
- return h[:-1] if h.endswith(".") else h
-
-
-def _is_aws_owned_host(host: str) -> bool:
- return any(host == suffix.removeprefix(".") or host.endswith(suffix) for suffix in _AWS_OWNED_HOST_SUFFIXES)
-
-
-def _origin_connect_target(origin: dict, viewer_is_https: bool) -> tuple:
- """(connect_host, connect_port, use_https) — the origin's own
- DomainName and HTTPPort/HTTPSPort, dialed exactly as configured."""
- policy = origin["protocol_policy"]
- use_https = policy == "https-only" or (policy == "match-viewer" and viewer_is_https)
- port = origin["https_port"] if use_https else origin["http_port"]
- return origin["domain_name"], port, use_https
-
-
-def _origin_ssl_context() -> ssl.SSLContext:
- """System roots, plus the gateway's own TLS cert under USE_SSL
- (lambda_svc.py's container CA-trust precedent) — a custom https origin
- may itself be a MiniStack-served endpoint sharing that cert."""
- from ministack.core import tls
-
- ctx = ssl.create_default_context()
- if tls.use_ssl_enabled():
- ctx.load_verify_locations(tls.resolve_tls_material()[0])
- ctx.verify_flags |= ssl.VERIFY_X509_PARTIAL_CHAIN
- return ctx
-
-
-def _set_header_ci(headers: dict, name: str, value) -> None:
- """Set ``name: value``, replacing any case-insensitive match."""
- for existing in [k for k in headers if k.lower() == name.lower()]:
- del headers[existing]
- headers[name] = value
-
-
-def _origin_x_forwarded_for(viewer_xff: str, client_ip: str) -> str:
- """CloudFront Developer Guide, "Client IP addresses": appends the TCP
- peer's address to an existing viewer-sent X-Forwarded-For, or adds a
- fresh one carrying just that address."""
- return f"{viewer_xff}, {client_ip}" if viewer_xff else client_ip
-
-
-def _forward_to_origin(origin: dict, method: str, uri: str, query_string: str,
- headers: dict, body: bytes, viewer_is_https: bool, client_ip: str = "127.0.0.1",
- viewer_x_forwarded_for: str = "") -> tuple:
- """Blocking: run under core.concurrency.run_reentrant (the origin may be
- MiniStack itself). Returns (status, reason, headers, body)."""
- conn_host, conn_port, use_https = _origin_connect_target(origin, viewer_is_https)
- if _is_aws_owned_host(_normalize_origin_host(conn_host)):
- raise OriginUnreachable(f"refusing to dial AWS-owned origin host {conn_host!r}")
- target_path = uri + (f"?{query_string}" if query_string else "")
-
- fwd_headers = {k: v for k, v in headers.items() if k not in _CONNECTION_MANAGEMENT_HEADERS}
- for name, value in origin.get("custom_headers") or []:
- _set_header_ci(fwd_headers, name, value)
- # Host is the origin's DomainName unless the policies forward the viewer's Host.
- _set_header_ci(fwd_headers, "Host", headers.get("host", origin["domain_name"]))
- _set_header_ci(fwd_headers, "Via", f"1.1 {_cf_via_hash()}.cloudfront.net (CloudFront)")
- _set_header_ci(fwd_headers, "X-Amz-Cf-Id", _cf_request_id())
- # Always appended (Developer Guide, "Client IP addresses").
- _set_header_ci(fwd_headers, "X-Forwarded-For",
- _origin_x_forwarded_for(viewer_x_forwarded_for, client_ip))
- # RFC 9110 §5.3: a repeated field value is one comma-joined line.
- send_headers = {
- name: ", ".join(str(v) for v in value) if isinstance(value, list) else value
- for name, value in fwd_headers.items()
- }
-
- conn_cls = http.client.HTTPSConnection if use_https else http.client.HTTPConnection
- timeout = origin.get("read_timeout", 30)
- conn = (
- conn_cls(conn_host, conn_port, timeout=timeout, context=_origin_ssl_context())
- if use_https else conn_cls(conn_host, conn_port, timeout=timeout)
- )
- try:
- conn.request(method, target_path, body=body if body else None, headers=send_headers)
- resp = conn.getresponse()
- data = resp.read()
- return resp.status, resp.reason, resp.getheaders(), data
- except TimeoutError as e:
- raise OriginTimeout(str(e)) from e
- except Exception as e:
- # An unreachable origin, or an untrusted or self-signed origin certificate.
- raise OriginUnreachable(str(e)) from e
- finally:
- conn.close()
-
-
-# ---------------------------------------------------------------------------
-# S3 origin — served from MiniStack's own S3 store in-process, never a socket.
-# ---------------------------------------------------------------------------
-
-
-def _s3_object_key(origin: dict, uri: str) -> str:
- """OriginPath prefixed to the request URI, as CloudFront builds the origin path."""
- return unquote((origin.get("origin_path") or "") + uri).lstrip("/")
-
-
-def _serve_s3_origin(origin: dict, method: str, uri: str, headers: dict, dist_arn: str = "") -> tuple:
- """Fetch from an S3 origin in-process (blocking; run off the event loop)."""
- return s3.serve_cloudfront_origin_fetch(
- origin["s3_bucket"], _s3_object_key(origin, uri), method, headers,
- source_arn=dist_arn if origin.get("oac_id") else "", oai_id=origin.get("oai_id", ""))
-
-
-# ---------------------------------------------------------------------------
-# Multi-value header handling — a name -> value map where a repeated name
-# (duplicate response headers, multiple Set-Cookie) becomes a name -> list.
-# ---------------------------------------------------------------------------
-
-
-async def _fetch_from_origin(origin: dict, method: str, uri: str, query_string: str, headers: dict, body: bytes,
- viewer_is_https: bool, client_ip: str, viewer_xff: str, dist_arn: str) -> tuple:
- """(status, reason, headers, body) from an S3 or custom origin; raises OriginTimeout / OriginUnreachable."""
- if origin.get("s3_bucket"):
- status, header_pairs, data = await run_offloop(_serve_s3_origin, origin, method, uri, headers, dist_arn)
- pairs = header_pairs.items() if isinstance(header_pairs, dict) else header_pairs
- return status, http.client.responses.get(status, ""), _multidict_from_pairs(pairs), data
- # The origin may be MiniStack's own gateway, so the call must be reentrant.
- status, reason, header_pairs, data = await run_reentrant(
- _forward_to_origin, origin, method, (origin.get("origin_path") or "") + uri, query_string, headers, body,
- viewer_is_https, client_ip, viewer_xff,
- )
- return status, reason, _multidict_from_pairs(header_pairs), data
-
-
-def _multidict_from_pairs(pairs) -> dict:
- out: dict = {}
- for name, value in pairs:
- key = name.lower()
- if key in out:
- existing = out[key]
- if isinstance(existing, list):
- existing.append(value)
- else:
- out[key] = [existing, value]
- else:
- out[key] = value
- return out
-
-
-def _strip_hop_by_hop_response_headers(headers: dict) -> dict:
- """Connection management is this module's concern, not the origin's."""
- return {k: v for k, v in headers.items() if k not in _HOP_BY_HOP_RESPONSE_HEADERS
- and not k.startswith("proxy-")}
-
-
-# ---------------------------------------------------------------------------
-# Behavior-level settings: ViewerProtocolPolicy, AllowedMethods, DefaultRootObject
-# ---------------------------------------------------------------------------
-
-
-def _check_viewer_protocol_policy(behavior: dict, method: str, viewer_is_https: bool, request_uri: str,
- query_string: str, headers: dict):
- """A redirect/forbid response, or None to continue."""
- policy = behavior.get("viewer_protocol_policy", "allow-all")
- if viewer_is_https or policy == "allow-all":
- return None
- if policy == "redirect-to-https":
- host = headers.get("host", "")
- location = f"https://{host}{request_uri}" + (f"?{query_string}" if query_string else "")
- redirect_status = 301 if method in ("GET", "HEAD") else 307
- status, resp_headers, body = _cf_error_response(redirect_status, "The document has moved.", _ERROR_X_CACHE)
- resp_headers["Location"] = location
- return status, resp_headers, body
- if policy == "https-only":
- return _cf_error_response(403, "Viewers must use HTTPS to request this object.", _ERROR_X_CACHE)
- return None
-
-
-def _check_allowed_methods(behavior: dict, method: str):
- allowed = behavior.get("allowed_methods")
- if allowed is not None and method not in allowed:
- # Community-observed CloudFront 403 page text for this exact case
- # (see _CF_ERROR_TEMPLATE's citation note).
- return _cf_error_response(
- 403,
- "This distribution is not configured to allow the HTTP request method that was used for this "
- "request. The distribution supports only cachable requests.",
- _ERROR_X_CACHE,
- )
- return None
-
-
-def _apply_default_root_object(default_root_object: str, request_uri: str) -> str:
- """CloudFront Developer Guide, "Default root object": the configured
- object is requested from the origin when a viewer requests the
- distribution's root URL."""
- if default_root_object and request_uri == "/":
- return "/" + default_root_object
- return request_uri
-
-
-def _check_stacked_distribution(headers: dict):
- """403 for a request that already passed through this hop (Developer Guide, stacked distributions)."""
- if "(cloudfront)" in headers.get("via", "").lower():
- return _cf_error_response(
- 403, "This distribution is configured to serve another CloudFront distribution as its origin.",
- _ERROR_X_CACHE,
- )
- return None
-
-
-# ---------------------------------------------------------------------------
-# Main entry point
-# ---------------------------------------------------------------------------
-
-
-async def handle_request(dist: dict, method: str, path: str, raw_uri: str, raw_query_string: str,
- headers: dict, body: bytes, query_params: dict, client_ip: str = "127.0.0.1") -> tuple:
- """Serve one viewer request against ``dist`` (a cloudfront.py distribution record)."""
- stacked_response = _check_stacked_distribution(headers)
- if stacked_response is not None:
- return stacked_response
-
- dist_id = dist["Id"]
- dist_domain = dist["DomainName"]
- request_id = new_uuid()
- viewer_is_https = headers.get("x-forwarded-proto", "http") == "https"
-
- parsed = cloudfront.parse_distribution_dataplane_config(dist)
- behavior = _match_behavior(parsed, path)
- if behavior is None:
- return _cf_error_response(502, "The request could not be satisfied.", _ERROR_X_CACHE)
-
- protocol_response = _check_viewer_protocol_policy(
- behavior, method, viewer_is_https, raw_uri, raw_query_string, headers,
- )
- if protocol_response is not None:
- return protocol_response
- methods_response = _check_allowed_methods(behavior, method)
- if methods_response is not None:
- return methods_response
-
- request_headers = dict(headers)
- request_uri = _apply_default_root_object(parsed.get("default_root_object", ""), raw_uri)
- request_query = dict(query_params or {})
- literal_query_string = None # set when a function rearranges querystring into a literal string
- function_touched_querystring = False # set when a function returns a changed querystring object
-
- request_function_arn = behavior["functions"].get("viewer-request")
- if request_function_arn:
- event = _build_function_event(
- "viewer-request", dist_id, dist_domain, request_id,
- method, request_uri, request_query, request_headers, client_ip,
- )
- try:
- result = await _run_function(request_function_arn, event)
- except CloudFrontFunctionError:
- logger.exception("viewer-request function error (distribution=%s)", dist_id)
- return _cf_error_response(503, "The Lambda function failed to execute.", _ERROR_X_CACHE)
-
- if isinstance(result, dict) and "statusCode" in result:
- # A function-generated response goes straight to the viewer, skipping the origin.
- resp_headers = _merge_function_headers(None, result.get("headers"))
- resp_headers = _add_cloudfront_headers(resp_headers, "FunctionGeneratedResponse from cloudfront")
- set_cookie_values = _set_cookie_values_from_event(result.get("cookies"))
- if set_cookie_values:
- resp_headers["set-cookie"] = (
- set_cookie_values if len(set_cookie_values) > 1 else set_cookie_values[0]
- )
- # statusDescription is dropped: ASGI carries no reason phrase.
- resp_body = _body_from_response_event(result, b"")
- return result["statusCode"], _render_headers(resp_headers, title_case=True), resp_body
-
- # Request headers stay lowercase internally; responses are Title-Cased once, on return.
- req = result if isinstance(result, dict) else event["request"]
- request_uri = req.get("uri", request_uri)
- request_headers = _merge_function_headers(event["request"]["headers"], req.get("headers"))
- cookie_pairs = [f"{n}={(f.get('value') if isinstance(f, dict) else f)}"
- for n, f in (req.get("cookies") or {}).items()]
- if cookie_pairs:
- request_headers["cookie"] = "; ".join(cookie_pairs)
- qs_field = req.get("querystring")
- if isinstance(qs_field, str):
- # A function rewrote the querystring as a literal string (functions-event-structure).
- literal_query_string = qs_field
- request_query = {}
- elif qs_field is not None and qs_field != event["request"]["querystring"]:
- function_touched_querystring = True
- request_query = {}
- for name, field in qs_field.items():
- if not isinstance(field, dict):
- continue
- values = [field.get("value", "")]
- values += [v.get("value", "") for v in field.get("multiValue", [])[1:]]
- request_query[name] = values
-
- origin = parsed["origins"].get(behavior["target_origin_id"])
- if origin is None:
- return _cf_error_response(502, "The request could not be satisfied.", _ERROR_X_CACHE)
-
- header_allowed, cookie_allowed, qs_allowed = _forwarding_predicates(behavior)
- request_cookies = _event_cookies_from_header(request_headers.get("cookie", ""))
- forward_host = header_allowed("host")
-
- fwd_headers = {
- name: value for name, value in request_headers.items()
- if name not in _HOP_BY_HOP_REQUEST_HEADERS
- and (header_allowed(name) or name in _ALWAYS_FORWARD_REQUEST_HEADERS)
- }
- if header_allowed("user-agent") and "user-agent" in request_headers:
- fwd_headers["user-agent"] = request_headers["user-agent"]
- else:
- # User-Agent is "Amazon CloudFront" unless forwarded (Developer Guide, "User-Agent header").
- fwd_headers["user-agent"] = "Amazon CloudFront"
- if forward_host:
- fwd_headers["host"] = request_headers.get("host", "")
- cookie_pairs = [f"{name}={field['value']}" for name, field in request_cookies.items() if cookie_allowed(name)]
- if cookie_pairs:
- fwd_headers["cookie"] = "; ".join(cookie_pairs)
- if literal_query_string is not None:
- forward_qs = literal_query_string
- elif not function_touched_querystring and all(qs_allowed(name) for name in request_query):
- # Byte-exact: nothing rewrote the querystring and the policy forwards
- # every parameter, so there is no reason to re-encode it.
- forward_qs = raw_query_string
- else:
- forward_qs = urlencode(
- [(name, v) for name, values in request_query.items() if qs_allowed(name) for v in values],
- quote_via=quote,
- )
-
- try:
- status, reason, origin_headers, origin_body = await _fetch_from_origin(
- origin, method, request_uri, forward_qs, fwd_headers, body, viewer_is_https,
- client_ip, request_headers.get("x-forwarded-for", ""), dist.get("ARN", ""),
- )
- origin_status = status
- error_response = parsed["custom_error_responses"].get(status)
- if error_response:
- # The page is requested through the cache behavior its path matches (Developer Guide,
- # "Store objects and custom error pages in different locations").
- origin = parsed["origins"].get(_match_behavior(parsed, error_response["page"])["target_origin_id"])
- if origin is None:
- return _cf_error_response(502, "The request could not be satisfied.", _ERROR_X_CACHE)
- status, reason, origin_headers, origin_body = await _fetch_from_origin(
- origin, "GET", error_response["page"], "", {"user-agent": "Amazon CloudFront"}, b"",
- viewer_is_https, client_ip, "", dist.get("ARN", ""),
- )
- # An unavailable page is answered with the status its origin returned (Developer Guide,
- # "Generate custom error responses").
- if status < 400:
- status = error_response["response_code"]
- except OriginTimeout:
- logger.warning("Origin timed out for distribution=%s origin=%s", dist_id, origin["domain_name"])
- return _cf_error_response(504, "The request could not be satisfied.", _ERROR_X_CACHE)
- except OriginUnreachable:
- logger.warning("Origin unreachable for distribution=%s origin=%s", dist_id, origin["domain_name"])
- return _cf_error_response(502, "The request could not be satisfied.", _ERROR_X_CACHE)
-
- origin_headers = _strip_hop_by_hop_response_headers(origin_headers)
- policy_cfg = cloudfront.response_headers_policy_config(behavior.get("response_headers_policy_id"))
- response_headers = _apply_response_headers_policy(
- origin_headers, policy_cfg, request_headers.get("origin", ""), method,
- )
- response_body = origin_body
-
- response_function_arn = behavior["functions"].get("viewer-response")
- # No viewer-response function runs when the origin answers 400 or above.
- if response_function_arn and origin_status < 400:
- event = _build_function_event(
- "viewer-response", dist_id, dist_domain, request_id,
- method, request_uri, request_query, headers, client_ip,
- )
- event["response"] = _response_event_fields(status, reason, response_headers)
- try:
- result = await _run_function(response_function_arn, event)
- except CloudFrontFunctionError:
- logger.exception("viewer-response function error (distribution=%s)", dist_id)
- return _cf_error_response(503, "The Lambda function failed to execute.", _ERROR_X_CACHE)
- response = result if isinstance(result, dict) else event["response"]
- status = response.get("statusCode", status)
- # Replace, not merge: a header the function deleted stays gone.
- response_headers = _merge_function_headers(event["response"]["headers"], response.get("headers"))
- set_cookie_values = _set_cookie_values_from_event(response.get("cookies"))
- if set_cookie_values:
- response_headers["set-cookie"] = set_cookie_values if len(set_cookie_values) > 1 else set_cookie_values[0]
- response_body = _body_from_response_event(response, origin_body)
-
- final_headers = _add_cloudfront_headers(response_headers, "Miss from cloudfront")
- return status, _render_headers(final_headers, title_case=True), response_body
diff --git a/ministack/services/ec2.py b/ministack/services/ec2.py
index b00dfc05..7b7d777c 100644
--- a/ministack/services/ec2.py
+++ b/ministack/services/ec2.py
@@ -7416,9 +7416,7 @@ _VA_STORES = {}
def _va_lookup(p, kind, prefix_only=True, listed=False):
- """(record, error) for the id member of ``kind``: format, then existence.
- Trust provider calls hold the whole id to the pattern and word NotFound as
- the Describe calls do."""
+ """(record, error) for the id member of ``kind``: format, then existence."""
value = _p(p, _VA_ID_MEMBERS[kind][0])
err = _va_bad_id(kind, value, prefix_only=prefix_only)
if err:
@@ -7490,18 +7488,7 @@ def _va_next_token_xml(token):
def _va_describe(p, kind, bounds, members=(), exclusive=False):
- """Run the checks shared by the Verified Access Describe* calls.
-
- ``kind`` names the id list (``<member>.N``), ``members`` the single-id
- members the call also takes, in the order AWS checks them, and
- ``exclusive`` that those members cannot be sent together. ``bounds`` is the
- MaxResults range AWS enforces, which is not always the botocore one.
- The checks run in AWS's order: MaxResults is an integer, the id list is not
- paged, ids are well formed, MaxResults is in range, the members are well
- formed and not combined, NextToken is one MiniStack issued, and ids and members exist.
- Filter names other than the tag filters are ignored, as AWS ignores them.
- Returns (selected_records, next_token, error_response).
- """
+ """Shared Describe* checks, in AWS's order; returns (page, next_token, error). ``bounds`` is AWS's MaxResults range."""
id_member = _VA_ID_MEMBERS[kind][0]
ids = _parse_member_list(p, id_member)
values = [(m, _p(p, _VA_ID_MEMBERS[m][0])) for m in members]
@@ -7871,10 +7858,7 @@ def _delete_verified_access_instance(p):
def _va_attachment_args(p, detach):
- """(instance, trust provider, error) for Attach/Detach. AWS looks up each
- well-formed id first, instance before provider, and reports a malformed id
- (the provider's first) only when no lookup failed. Attach holds the
- provider id's prefix to the pattern, Detach the whole id."""
+ """(instance, provider, error): each well-formed id is looked up first, then a malformed one is reported."""
err = _va_require(p, "VerifiedAccessInstanceId", "VerifiedAccessTrustProviderId")
if err:
return None, None, err
@@ -8197,9 +8181,7 @@ def _va_get_policy_xml(record):
def _va_modify_policy(record, p):
- """Apply a Modify*Policy request; returns an error response or None.
- A document needs PolicyEnabled=true, PolicyEnabled=true needs a document,
- and disabling the policy drops its document."""
+ """A document needs PolicyEnabled=true and vice versa; disabling drops the document."""
enabled = _p(p, "PolicyEnabled")
has_document = "PolicyDocument" in p
if enabled == "true" and not has_document:
@@ -8303,9 +8285,7 @@ def _va_live_endpoints():
def _va_settle(ep):
- """AWS answers pending on create and updating on modify, and the endpoint
- is active once AWS has applied the change; MiniStack applies it at once,
- so the next read sees active."""
+ """MiniStack applies a change at once, so the next read after pending/updating is active."""
if ep["Status"] in ("pending", "updating"):
ep["Status"] = "active"
@@ -8343,9 +8323,7 @@ _VA_PREFIX_RE = re.compile(r"[A-Za-z0-9_](?:[A-Za-z0-9_-]*[A-Za-z0-9_])?")
def _va_options_value_error(p, prefix, xml_prefix, tcp=True):
- """Protocol and port values, both reported when both are bad; on create,
- HTTP/HTTPS then refuses PortRanges, and tcp refuses a Port and needs
- PortRanges. AWS names the network-interface options EniOptions here."""
+ """Protocol and port values; on create, PortRanges only with tcp, which takes no Port."""
protocol = _p(p, f"{prefix}.Protocol")
port = _p(p, f"{prefix}.Port")
bad = []
diff --git a/ministack/services/ecs.py b/ministack/services/ecs.py
index 68740677..71bfdd1f 100644
--- a/ministack/services/ecs.py
+++ b/ministack/services/ecs.py
@@ -1239,7 +1239,8 @@ def _reconcile_service_tasks(cluster_name, svc_key):
desired = svc.get("desiredCount", 0)
td_arn = svc.get("taskDefinition", "")
cluster_arn = svc.get("clusterArn", "")
- launch_type = svc.get("launchType", "EC2")
+ launch_type = svc.get("launchType")
+ strategy = svc.get("capacityProviderStrategy")
network_cfg = svc.get("networkConfiguration", {})
if desired == 0:
@@ -1329,6 +1330,7 @@ def _reconcile_service_tasks(cluster_name, svc_key):
"group": f"service:{svc_name}",
"startedBy": svc_name,
"launchType": launch_type,
+ "capacityProviderStrategy": strategy,
"platformVersion": svc.get("platformVersion", ""),
"networkConfiguration": network_cfg,
"enableExecuteCommand": svc.get("enableExecuteCommand", False),
@@ -1351,6 +1353,32 @@ def _reconcile_service_tasks(cluster_name, svc_key):
_complete_service_deployment(cluster_name, svc_key)
+_BOTH_LAUNCH_CHOICES = ("Specifying both a launch type and capacity provider strategy is not supported. "
+ "Remove one and try again.")
+_FARGATE_PROVIDERS = ("FARGATE", "FARGATE_SPOT")
+
+
+def _launch_choice(data, cluster):
+ """(launchType, capacityProviderStrategy, error): a strategy omits the launch type, and with
+ neither the cluster's default strategy applies, else EC2."""
+ strategy = data.get("capacityProviderStrategy") or None
+ launch_type = data.get("launchType")
+ if strategy and launch_type:
+ return None, None, error_response_json("InvalidParameterException", _BOTH_LAUNCH_CHOICES, 400)
+ if strategy or launch_type:
+ return launch_type, strategy, None
+ default = (cluster or {}).get("defaultCapacityProviderStrategy") or None
+ return (None, default, None) if default else ("EC2", None, None)
+
+
+def _task_capacity(launch_type, strategy):
+ """The launch type a task runs on and the capacity provider it was placed with."""
+ if not strategy:
+ return launch_type or "EC2", None
+ provider = max(strategy, key=lambda item: (item.get("base", 0), item.get("weight", 0)))["capacityProvider"]
+ return ("FARGATE" if provider in _FARGATE_PROVIDERS else "EC2"), provider
+
+
def _create_service(data):
cluster_name = _resolve_cluster_name(data.get("cluster", "default"))
if cluster_name is None:
@@ -1379,10 +1407,13 @@ def _create_service(data):
desired = data.get("desiredCount", 1)
arn = f"arn:aws:ecs:{get_region()}:{get_account_id()}:service/{cluster_name}/{name}"
now = _iso()
- launch_type = data.get("launchType", "EC2")
+ launch_type, strategy, err = _launch_choice(data, _clusters[cluster_name])
+ if err:
+ return err
deployment = _make_deployment(td_arn, desired)
deployment["launchType"] = launch_type
+ deployment["capacityProviderStrategy"] = strategy
svc = {
"serviceArn": arn,
@@ -1394,6 +1425,7 @@ def _create_service(data):
"pendingCount": 0,
"status": "ACTIVE",
"launchType": launch_type,
+ "capacityProviderStrategy": strategy,
"platformVersion": data.get("platformVersion", ""),
"platformFamily": data.get("platformFamily", ""),
"networkConfiguration": data.get("networkConfiguration", {}),
@@ -1552,7 +1584,8 @@ def _update_service(data):
dep["status"] = "ACTIVE"
new_dep = _make_deployment(
td_arn, new_desired if new_desired is not None else svc["desiredCount"])
- new_dep["launchType"] = svc.get("launchType", "EC2")
+ new_dep["launchType"] = svc.get("launchType")
+ new_dep["capacityProviderStrategy"] = svc.get("capacityProviderStrategy")
# A replacement remains IN_PROGRESS until its own tasks stabilize.
new_dep["rolloutState"] = "IN_PROGRESS"
new_dep["rolloutStateReason"] = ""
@@ -1580,7 +1613,10 @@ def _update_service(data):
if "loadBalancers" in data:
svc["loadBalancers"] = data["loadBalancers"]
if "capacityProviderStrategy" in data:
- svc["capacityProviderStrategy"] = data["capacityProviderStrategy"]
+ # An empty strategy moves the service back to a launch type.
+ strategy = data["capacityProviderStrategy"] or None
+ svc["capacityProviderStrategy"] = strategy
+ svc["launchType"] = None if strategy else (svc.get("launchType") or "EC2")
if changed:
svc["events"].insert(0, {
@@ -2584,7 +2620,10 @@ def _run_task(data):
"overrides", {"containerOverrides": [], "inferenceAcceleratorOverrides": []}
)
container_overrides = copy.deepcopy(overrides.get("containerOverrides", []))
- launch_type = data.get("launchType", "EC2")
+ launch_type, strategy, err = _launch_choice(data, _clusters[cluster_name])
+ if err:
+ return err
+ launch_type, capacity_provider = _task_capacity(launch_type, strategy)
platform_version = data.get("platformVersion", "")
platform_family = ""
if launch_type == "FARGATE":
@@ -2639,6 +2678,7 @@ def _run_task(data):
"lastStatus": initial_status,
"desiredStatus": "RUNNING",
"launchType": launch_type,
+ "capacityProviderName": capacity_provider,
"cpu": td.get("cpu", "256"),
"memory": td.get("memory", "512"),
"platformVersion": platform_version,
diff --git a/ministack/services/iot.py b/ministack/services/iot.py
index 1366397c..29d184b5 100644
--- a/ministack/services/iot.py
+++ b/ministack/services/iot.py
@@ -11740,7 +11740,7 @@ def _ws_handshake_check(scope: dict) -> AuthError | ResolvedCredential | None:
from ministack.app import AUTH
from ministack.core.iam_evaluator import authenticate_handshake
- return authenticate_handshake(scope) if AUTH else None
+ return authenticate_handshake(scope, "iotdevicegateway") if AUTH else None
async def _refuse_ws_handshake(scope: dict, send) -> None:
diff --git a/ministack/services/lambda_svc.py b/ministack/services/lambda_svc.py
index ba1086d8..526654d4 100644
--- a/ministack/services/lambda_svc.py
+++ b/ministack/services/lambda_svc.py
@@ -46,6 +46,7 @@ import urllib.error
import urllib.request
import weakref
import zipfile
+import zlib
from datetime import datetime, timezone
from functools import wraps
from typing import Any
@@ -78,6 +79,7 @@ from ministack.core.responses import (
error_response_json,
get_account_id,
get_region,
+ get_request_client_ip,
json_response,
new_uuid,
)
@@ -4622,7 +4624,6 @@ def _spawn_lambda_container_impl(config: dict, code_zip: bytes | None,
run_kwargs: dict = {
"image": image,
"environment": container_env,
- "ports": {"8080/tcp": None},
"detach": True,
"stdin_open": False,
"labels": container_reaper.own_labels("lambda"),
@@ -4660,6 +4661,9 @@ def _spawn_lambda_container_impl(config: dict, code_zip: bytes | None,
run_kwargs["mounts"] = mounts
if LAMBDA_DOCKER_NETWORK:
run_kwargs["network"] = LAMBDA_DOCKER_NETWORK
+ else:
+ # The runtime is reached on its network address when there is one; otherwise on loopback only.
+ run_kwargs["ports"] = {"8080/tcp": ("127.0.0.1", None)}
# AWS runs a function on the architecture it declares. Docker otherwise
# picks the host's, so an arm64 function on an x86_64 host quietly ran as
@@ -8005,7 +8009,7 @@ def _create_function_url_config(func_name: str, data: dict, qualifier: str | Non
"ResourceConflictException", f"Function URL config already exists for {func_name}", 409
)
cfg = {
- "FunctionUrl": f"https://{new_uuid()}.lambda-url.{get_region()}.on.aws/",
+ "FunctionUrl": f"https://{_new_function_url_id()}.lambda-url.{get_region()}.on.aws/",
"FunctionArn": _func_arn(func_name),
"AuthType": data.get("AuthType", "NONE"),
"InvokeMode": data.get("InvokeMode", "BUFFERED"),
@@ -8077,6 +8081,15 @@ _STREAM_PRELUDE_SEPARATOR = b"\x00" * 8
_FUNCTION_URL_TEXT_TYPES = ("application/json", "application/xml", "application/javascript")
+def _new_function_url_id() -> str:
+ # AWS's URL ids are 32 lowercase letters and digits.
+ return "".join(secrets.choice("abcdefghijklmnopqrstuvwxyz0123456789") for _ in range(32))
+
+
+def _function_url_trace_root() -> str:
+ return "Root=1-%08x-%s" % (int(time.time()), secrets.token_hex(12))
+
+
def _function_url_id(function_url: str) -> str:
"""Return the URL id (first host label) of a stored ``FunctionUrl``."""
return function_url.split("://", 1)[-1].split(".", 1)[0]
@@ -8165,6 +8178,9 @@ def _build_function_url_event(
headers: dict,
body: bytes,
query_params: dict,
+ source_ip: str = "127.0.0.1",
+ anonymous: bool = False,
+ trace_root: str = "",
) -> dict:
"""Build the payload-format-2.0 event a Function URL delivers.
@@ -8184,14 +8200,28 @@ def _build_function_url_event(
req_body, is_base64 = None, False
domain_name = f"{url_id}.lambda-url.{region}.on.aws"
+ # The fields the Function URL front end sets: X-Forwarded-For keeps only the
+ # first address it was sent, or names the caller.
+ event_headers = {k.lower(): v for k, v in headers.items()}
+ first_hop = (event_headers.get("x-forwarded-for") or "").split(",", 1)[0].strip()
+ event_headers.update({
+ "host": domain_name,
+ "x-amzn-tls-cipher-suite": "TLS_AES_128_GCM_SHA256",
+ "x-amzn-tls-version": "TLSv1.3",
+ "x-amzn-trace-id": trace_root or _function_url_trace_root(),
+ "x-forwarded-for": first_hop or source_ip,
+ "x-forwarded-port": "443",
+ "x-forwarded-proto": "https",
+ })
event = {
"version": "2.0",
"routeKey": "$default",
"rawPath": path,
"rawQueryString": raw_qs,
- "headers": dict(headers),
+ "headers": event_headers,
"requestContext": {
- "accountId": account_id,
+ # A URL with AuthType NONE reports the caller's account as "anonymous".
+ "accountId": "anonymous" if anonymous else account_id,
"apiId": url_id,
"domainName": domain_name,
"domainPrefix": url_id,
@@ -8199,7 +8229,7 @@ def _build_function_url_event(
"method": method,
"path": path,
"protocol": "HTTP/1.1",
- "sourceIp": "127.0.0.1",
+ "sourceIp": source_ip,
"userAgent": headers.get("user-agent", ""),
},
"requestId": new_uuid(),
@@ -8221,7 +8251,7 @@ def _build_function_url_event(
async def handle_function_url_request(
url_id: str, method: str, path: str, headers: dict, body: bytes, query_params: dict,
- resolved: tuple | None = None,
+ resolved: tuple | None = None, source_ip: str | None = None,
) -> tuple:
"""Serve a Lambda Function URL data-plane request.
@@ -8239,24 +8269,35 @@ async def handle_function_url_request(
if method == "OPTIONS" and cfg.get("Cors"):
return 200, cors_headers, b""
+ request_id = new_uuid()
if cfg.get("AuthType") == "AWS_IAM" and not _function_url_is_signed(headers, query_params):
- return 403, {"Content-Type": "application/json", **cors_headers}, json.dumps({"Message": "Forbidden"}).encode()
+ return 403, {"Content-Type": "application/json", "x-amzn-RequestId": request_id,
+ "x-amzn-ErrorType": "AccessDeniedException", **cors_headers}, b'{"Message":"Forbidden"}'
function_ref = f"{func_name}:{qualifier}" if qualifier else func_name
func_data, func_config, _ = _get_func_record_for_ref_in_scope(function_ref, account_id=account_id, region=region)
if func_data is None or func_config is None:
return error_response_json("ResourceNotFoundException", f"Function not found: {func_name}", 404)
- event = _build_function_url_event(url_id, account_id, region, method, path, headers, body, query_params)
+ trace_root = _function_url_trace_root()
+ event = _build_function_url_event(url_id, account_id, region, method, path, headers, body, query_params,
+ source_ip=source_ip or get_request_client_ip(),
+ anonymous=cfg.get("AuthType") == "NONE", trace_root=trace_root)
+ event["requestContext"]["requestId"] = request_id
exec_record = _execution_record_for_config(func_data, func_config)
result = await run_reentrant(_execute_function_with_config_scope, exec_record, event,
thread_name="ministack-lambda-url")
+ lineage = "%08x" % zlib.crc32(func_config.get("FunctionArn", func_name).encode())
+ invoke_headers = {"x-amzn-RequestId": request_id,
+ "X-Amzn-Trace-Id": f"{trace_root};Parent={secrets.token_hex(8)};Sampled=0;Lineage=1:{lineage}:0"}
if cfg.get("InvokeMode") == "RESPONSE_STREAM" and not result.get("error") and not result.get("throttle"):
- return _function_url_stream_response(result, cors_headers)
-
- proxy_response, _ = lambda_execute_result_to_api_proxy_response(result)
- return _function_url_proxy_response(proxy_response, cors_headers)
+ status, resp_headers, data = _function_url_stream_response(result, cors_headers)
+ else:
+ proxy_response, _ = lambda_execute_result_to_api_proxy_response(result)
+ status, resp_headers, data = _function_url_proxy_response(proxy_response, cors_headers)
+ _apply_headers_case_insensitively(resp_headers, invoke_headers)
+ return status, resp_headers, data
def _apply_headers_case_insensitively(base: dict, overrides: dict) -> None:
diff --git a/ministack/services/stepfunctions.py b/ministack/services/stepfunctions.py
index b116fae4..0edd500f 100644
--- a/ministack/services/stepfunctions.py
+++ b/ministack/services/stepfunctions.py
@@ -142,6 +142,7 @@ _task_tokens = AccountRegionScopedDict()
_tags = AccountRegionScopedDict()
_activities = AccountRegionScopedDict()
_activity_tasks = AccountRegionScopedDict()
+_map_runs = AccountRegionScopedDict()
# version_arn -> {stateMachineVersionArn, stateMachineRevisionId,
# description, creationDate, definition, roleArn, type,
@@ -165,6 +166,7 @@ def get_state():
"activities": copy.deepcopy(_activities),
"state_machine_versions": copy.deepcopy(_state_machine_versions),
"state_machine_aliases": copy.deepcopy(_state_machine_aliases),
+ "map_runs": copy.deepcopy(_map_runs),
}
@@ -177,6 +179,7 @@ def _restore_state(data):
return
_state_machines.update(data.get("state_machines", {}))
_executions.update(data.get("executions", {}))
+ _map_runs.update(data.get("map_runs", {}))
_tags.update(data.get("tags", {}))
_activities.update(data.get("activities", {}))
_state_machine_versions.update(data.get("state_machine_versions", {}))
@@ -295,6 +298,9 @@ async def handle_request(method, path, headers, body, query_params):
"DeleteStateMachineAlias": _delete_state_machine_alias,
"DescribeStateMachineAlias": _describe_state_machine_alias,
"ListStateMachineAliases": _list_state_machine_aliases,
+ "ListMapRuns": _list_map_runs,
+ "DescribeMapRun": _describe_map_run,
+ "UpdateMapRun": _update_map_run,
}
handler = handlers.get(action)
@@ -962,9 +968,42 @@ def _describe_execution(data):
result["error"] = execution["error"]
if execution.get("cause"):
result["cause"] = execution["cause"]
+ if execution.get("mapRunArn"):
+ result["mapRunArn"] = execution["mapRunArn"]
return json_response(result)
+def _map_run_view(run):
+ return {k: v for k, v in run.items() if not k.startswith("_")}
+
+
+def _list_map_runs(data):
+ exec_arn = data.get("executionArn")
+ if exec_arn not in _executions:
+ return error_response_json("ExecutionDoesNotExist", f"Execution Does Not Exist: '{exec_arn}'", 400)
+ runs = [{"executionArn": r["executionArn"], "mapRunArn": r["mapRunArn"], "stateMachineArn": r["_stateMachineArn"],
+ "startDate": r["startDate"], **({"stopDate": r["stopDate"]} if r.get("stopDate") else {})}
+ for r in _map_runs.values() if r["executionArn"] == exec_arn]
+ return json_response({"mapRuns": runs})
+
+
+def _describe_map_run(data):
+ run = _map_runs.get(data.get("mapRunArn"))
+ if run is None:
+ return error_response_json("ResourceNotFound", f"Map Run Does Not Exist: '{data.get('mapRunArn')}'", 400)
+ return json_response(_map_run_view(run))
+
+
+def _update_map_run(data):
+ run = _map_runs.get(data.get("mapRunArn"))
+ if run is None:
+ return error_response_json("ResourceNotFound", f"Map Run Does Not Exist: '{data.get('mapRunArn')}'", 400)
+ for field in ("maxConcurrency", "toleratedFailurePercentage", "toleratedFailureCount"):
+ if field in data:
+ run[field] = data[field]
+ return json_response({})
+
+
def _execution_sort_key(execution):
"""AWS lists the most recent execution first: a running one by its
startDate, the rest by stopDate. The ARN breaks ties, so the order is
@@ -995,21 +1034,29 @@ def _list_executions(data):
if cursor is None:
return error_response_json("InvalidToken", "Invalid Token: 'Invalid token'", 400)
sm_arn = data.get("stateMachineArn")
+ map_run_arn = data.get("mapRunArn")
+ if map_run_arn and map_run_arn not in _map_runs:
+ return error_response_json("ResourceNotFound", f"Map Run Does Not Exist: '{map_run_arn}'", 400)
status_filter = data.get("statusFilter")
all_execs = []
for ex in _executions.values():
if sm_arn and ex["stateMachineArn"] != sm_arn:
continue
+ if map_run_arn and ex.get("mapRunArn") != map_run_arn:
+ continue
if status_filter and ex["status"] != status_filter:
continue
- all_execs.append({
+ item = {
"executionArn": ex["executionArn"],
"stateMachineArn": ex["stateMachineArn"],
"name": ex["name"],
"status": ex["status"],
"startDate": ex["startDate"],
"stopDate": ex.get("stopDate"),
- })
+ }
+ if ex.get("mapRunArn"):
+ item["mapRunArn"] = ex["mapRunArn"]
+ all_execs.append(item)
all_execs.sort(key=_execution_sort_key, reverse=True)
if cursor is not None:
all_execs = [e for e in all_execs if _execution_sort_key(e) < cursor]
@@ -2595,6 +2642,8 @@ def _run_map_items(state_def, raw_input, execution, ctx):
items = [items]
iterator = state_def.get("Iterator") or state_def.get("ItemProcessor", {})
+ if (iterator.get("ProcessorConfig") or {}).get("Mode") == "DISTRIBUTED":
+ return _run_distributed_map(state_def, raw_input, effective, iterator, execution, ctx)
iter_states = iterator.get("States", {})
iter_start = iterator.get("StartAt")
max_conc = state_def.get("MaxConcurrency", 0)
@@ -2640,6 +2689,169 @@ def _run_map_items(state_def, raw_input, execution, ctx):
return output, _next_or_end(state_def)
+_CSV_DELIMITERS = {"COMMA": ",", "PIPE": "|", "SEMICOLON": ";", "SPACE": " ", "TAB": "\t"}
+
+
+def _s3_object_text(bucket, key):
+ from ministack.services import s3
+
+ status, _headers, body = s3._get_object(bucket, key, {}, {})
+ if status != 200:
+ raise _ExecutionError("States.ItemReaderFailed", f"The specified key does not exist. (Bucket: {bucket}, Key: {key})")
+ return body.decode("utf-8") if isinstance(body, (bytes, bytearray)) else str(body)
+
+
+def _parse_reader_items(text, config):
+ """Items of a JSON, JSONL or CSV dataset, as the Distributed Map reads them."""
+ import csv
+ import io
+
+ input_type = config.get("InputType", "JSON")
+ if input_type == "JSONL":
+ return [json.loads(line) for line in text.splitlines() if line.strip()]
+ if input_type == "CSV":
+ rows = list(csv.reader(io.StringIO(text), delimiter=_CSV_DELIMITERS.get(config.get("CSVDelimiter", "COMMA"), ",")))
+ if config.get("CSVHeaderLocation") == "GIVEN":
+ headers = list(config.get("CSVHeaders") or [])
+ else:
+ headers, rows = (rows[0], rows[1:]) if rows else ([], [])
+ # Missing fields are empty strings and extra fields are dropped.
+ return [{h: (row[i] if i < len(row) else "") for i, h in enumerate(headers)} for row in rows if row]
+ data = json.loads(text)
+ pointer = config.get("ItemsPointer")
+ if pointer:
+ for part in pointer.strip("/").split("/"):
+ data = data[int(part)] if isinstance(data, list) else data[part]
+ return data
+
+
+def _read_map_items(reader, effective, ctx):
+ from ministack.services import s3
+
+ params = _resolve_params_obj(reader.get("Parameters") or reader.get("Arguments") or {}, effective, ctx)
+ config = reader.get("ReaderConfig") or {}
+ resource = reader.get("Resource", "")
+ if resource.endswith("s3:listObjectsV2"):
+ bucket = s3._buckets.get(params.get("Bucket"))
+ if bucket is None:
+ raise _ExecutionError("States.ItemReaderFailed", f"The specified bucket does not exist. (Bucket: {params.get('Bucket')})")
+ prefix = params.get("Prefix", "")
+ items = []
+ for key in sorted(k for k in bucket["objects"] if k.startswith(prefix)):
+ obj = bucket["objects"][key]
+ if config.get("Transformation") == "LOAD_AND_FLATTEN":
+ items.extend(_parse_reader_items(_s3_object_text(params["Bucket"], key), config))
+ continue
+ items.append({"Etag": obj.get("etag", ""), "Key": key, "LastModified": _epoch_seconds(obj.get("last_modified")),
+ "Size": obj.get("size", 0), "StorageClass": obj.get("storage_class") or "STANDARD"})
+ else:
+ items = _parse_reader_items(_s3_object_text(params.get("Bucket"), params.get("Key")), config)
+ max_items = config.get("MaxItems")
+ if config.get("MaxItemsPath"):
+ max_items = _resolve_path_or_ctx(config["MaxItemsPath"], effective, ctx)
+ return items[:int(max_items)] if max_items else items
+
+
+def _epoch_seconds(stamp):
+ from datetime import datetime
+
+ if isinstance(stamp, (int, float)):
+ return int(stamp)
+ try:
+ return int(datetime.fromisoformat(str(stamp).replace("Z", "+00:00")).timestamp())
+ except ValueError:
+ return 0
+
+
+def _run_distributed_map(state_def, raw_input, effective, iterator, execution, ctx):
+ """Each item runs as a child workflow execution of a Map Run, which ListMapRuns, DescribeMapRun and
+ ListExecutions(mapRunArn) report."""
+ if state_def.get("ItemReader"):
+ items = _read_map_items(state_def["ItemReader"], effective, ctx)
+ else:
+ items = _resolve_path_or_ctx(state_def.get("ItemsPath", "$"), effective, ctx)
+ if not isinstance(items, list):
+ items = [items]
+
+ parent_sm = execution["stateMachineArn"].split(":")
+ sm_name, region, account = parent_sm[6], parent_sm[3], parent_sm[4]
+ label = state_def.get("Label") or new_uuid()
+ child_sm_arn = f"arn:aws:states:{region}:{account}:stateMachine:{sm_name}/{label}"
+ map_run_arn = f"arn:aws:states:{region}:{account}:mapRun:{sm_name}/{label}:{new_uuid()}"
+ max_conc = int(state_def.get("MaxConcurrency") or 0)
+ tolerated_pct = float(state_def.get("ToleratedFailurePercentage") or 0)
+ tolerated_count = int(state_def.get("ToleratedFailureCount") or 0)
+ zero = {k: 0 for k in ("pending", "running", "succeeded", "failed", "timedOut", "aborted", "total", "resultsWritten",
+ "failuresNotRedrivable", "pendingRedrive")}
+ run = {
+ "mapRunArn": map_run_arn, "executionArn": execution["executionArn"], "status": "RUNNING",
+ "startDate": now_iso(), "maxConcurrency": max_conc, "toleratedFailurePercentage": tolerated_pct,
+ "toleratedFailureCount": tolerated_count, "itemCounts": dict(zero, total=len(items), pending=len(items)),
+ "executionCounts": dict(zero, total=len(items), pending=len(items)), "redriveCount": 0,
+ "_stateMachineArn": child_sm_arn,
+ }
+ _map_runs[map_run_arn] = run
+ item_selector = state_def.get("ItemSelector") or state_def.get("Parameters")
+ results = [None] * len(items)
+ failures = [None] * len(items)
+ lock = threading.Lock()
+
+ def run_child(idx, item):
+ item_ctx = copy.deepcopy(ctx)
+ item_ctx["Map"] = {"Item": {"Index": idx, "Value": item}}
+ child_input = _resolve_params_obj(item_selector, effective, item_ctx) if item_selector else item
+ name = new_uuid()
+ child = {
+ "executionArn": f"arn:aws:states:{region}:{account}:execution:{sm_name}/{label}:{name}",
+ "stateMachineArn": child_sm_arn, "name": name, "status": "RUNNING", "startDate": now_iso(),
+ "stopDate": None, "input": json.dumps(child_input), "inputDetails": {"included": True}, "output": None,
+ "outputDetails": {"included": True}, "mapRunArn": map_run_arn, "events": [],
+ "testCase": execution.get("testCase"), "mockAttempts": {},
+ }
+ _executions[child["executionArn"]] = child
+ with lock:
+ for counts in (run["itemCounts"], run["executionCounts"]):
+ counts["pending"] -= 1
+ counts["running"] += 1
+ try:
+ results[idx] = _run_sub_machine(iterator.get("States", {}), iterator.get("StartAt"),
+ child_input, child, item_ctx)
+ child.update(status="SUCCEEDED", output=json.dumps(results[idx]))
+ outcome = "succeeded"
+ except Exception as exc:
+ err = exc if isinstance(exc, _ExecutionError) else _ExecutionError("States.TaskFailed", str(exc))
+ failures[idx] = err
+ child.update(status="FAILED", error=err.error, cause=err.cause)
+ outcome = "failed"
+ child["stopDate"] = now_iso()
+ with lock:
+ for counts in (run["itemCounts"], run["executionCounts"]):
+ counts["running"] -= 1
+ counts[outcome] += 1
+
+ workers = max_conc if max_conc > 0 else (min(len(items), 64) or 1)
+ with ThreadPoolExecutor(max_workers=workers) as pool:
+ futures_wait([pool.submit(contextvars.copy_context().run, run_child, i, item) for i, item in enumerate(items)])
+
+ failed = sum(1 for f in failures if f is not None)
+ limits = []
+ if "ToleratedFailureCount" in state_def:
+ limits.append(failed > tolerated_count)
+ if "ToleratedFailurePercentage" in state_def:
+ limits.append(failed * 100 > tolerated_pct * len(items))
+ # With no threshold set, any failed item fails the Map Run.
+ exceeded = bool(failed) and (any(limits) if limits else True)
+ run["status"] = "FAILED" if exceeded else "SUCCEEDED"
+ run["stopDate"] = now_iso()
+ if exceeded:
+ raise _ExecutionError("States.ExceedToleratedFailureThreshold",
+ "The specified tolerated failure threshold was exceeded")
+ result = _apply_result_selector(state_def, results)
+ output = _apply_result_path(state_def, raw_input, result)
+ output = _apply_output_path(state_def, output, ctx)
+ return output, _next_or_end(state_def)
+
+
# ---------------------------------------------------------------------------
# Sub-machine runner (Parallel branches / Map iterations)
# ---------------------------------------------------------------------------
@@ -5329,6 +5541,7 @@ def reset():
_activity_tasks.clear()
_state_machine_versions.clear()
_state_machine_aliases.clear()
+ _map_runs.clear()
# ---------------------------------------------------------------------------
diff --git a/tests/test_alb.py b/tests/test_alb.py
index 3383274d..75dcc494 100644
--- a/tests/test_alb.py
+++ b/tests/test_alb.py
@@ -572,7 +572,10 @@ def test_elbv2_dataplane_event_shape(elbv2, lam):
assert body["httpMethod"] == "GET"
assert body["path"] == "/check"
assert body["queryStringParameters"].get("foo") == "bar"
- assert "headers" in body
+ assert body["headers"]["x-forwarded-for"] == "127.0.0.1"
+ assert (body["headers"]["x-forwarded-proto"], body["headers"]["x-forwarded-port"]) == ("http", "80")
+ assert re.fullmatch(r"Root=1-[0-9a-f]{8}-[0-9a-f]{24}", body["headers"]["x-amzn-trace-id"])
+ assert body["multiValueHeaders"]["x-forwarded-for"] == ["127.0.0.1"]
assert body["isBase64Encoded"] is False
finally:
_alb_teardown(elbv2, lam, lb_arn, tg_arn, l_arn, "dp-alb-evt-fn")
@@ -1060,6 +1063,108 @@ def test_elbv2_dataplane_preserves_repeated_target_headers(elbv2, chunked, cooki
thread.join(timeout=5)
+def _start_request_capture_server():
+ """Serve one connection: keep its raw request-header lines, answer 200 "ok"."""
+ import socket as _socket
+ import threading
+
+ sock = _socket.socket()
+ sock.setsockopt(_socket.SOL_SOCKET, _socket.SO_REUSEADDR, 1)
+ sock.bind(("127.0.0.1", 0))
+ sock.listen(1)
+ header_lines = []
+
+ def _serve():
+ try:
+ conn, _ = sock.accept()
+ with conn:
+ data = b""
+ while b"\r\n\r\n" not in data:
+ chunk = conn.recv(4096)
+ if not chunk:
+ break
+ data += chunk
+ header_lines.extend(data.partition(b"\r\n\r\n")[0].decode("latin-1").split("\r\n")[1:])
+ conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
+ except OSError:
+ pass
+
+ thread = threading.Thread(target=_serve, daemon=True)
+ thread.start()
+ return sock, thread, header_lines
+
+
+def test_elbv2_dataplane_sends_forwarded_headers_once(elbv2):
+ """A request that already carries X-Forwarded-* reaches the target with one field per name."""
+ sock, thread, header_lines = _start_request_capture_server()
+ lb_name = f"dp-alb-xff-{_uuid_mod.uuid4().hex[:8]}"
+ lb_arn, tg_arn, l_arn = _alb_http_target_setup(
+ elbv2, lb_name, "127.0.0.1", sock.getsockname()[1]
+ )
+ try:
+ request = _req.Request(f"{_endpoint}/_alb/{lb_name}/", headers={
+ "X-Forwarded-For": "203.0.113.5", "X-Forwarded-Proto": "https", "X-Forwarded-Port": "443",
+ })
+ with _req.urlopen(request, timeout=10) as resp:
+ assert resp.read() == b"ok"
+ thread.join(timeout=5)
+ fields = [(n.lower(), v.strip()) for n, _, v in (line.partition(":") for line in header_lines)]
+ names = [n for n, _ in fields]
+ for name in ("x-forwarded-for", "x-forwarded-proto", "x-forwarded-port"):
+ assert names.count(name) == 1, header_lines
+ values = dict(fields)
+ # The load balancer appends its client's address and states the listener's protocol and port.
+ assert values["x-forwarded-for"] == "203.0.113.5, 127.0.0.1"
+ assert (values["x-forwarded-proto"], values["x-forwarded-port"]) == ("http", "80")
+ assert re.fullmatch(r"Root=1-[0-9a-f]{8}-[0-9a-f]{24}", values["x-amzn-trace-id"])
+ finally:
+ _alb_http_target_teardown(elbv2, lb_arn, tg_arn, l_arn)
+ sock.close()
+ thread.join(timeout=5)
+
+
+def test_elbv2_target_connection_closed_when_the_caller_stops_waiting():
+ """A caller that gives up before the target answers closes the target connection
+ instead of leaving it open for the idle timeout."""
+ import asyncio
+ import socket as _socket
+ import threading
+
+ from ministack.services import alb
+
+ sock = _socket.socket()
+ sock.bind(("127.0.0.1", 0))
+ sock.listen(1)
+ port = sock.getsockname()[1]
+ closed = threading.Event()
+
+ def _serve():
+ conn, _ = sock.accept()
+ with conn:
+ conn.settimeout(30)
+ data = b""
+ while b"\r\n\r\n" not in data:
+ data += conn.recv(4096)
+ if conn.recv(4096) == b"":
+ closed.set()
+
+ thread = threading.Thread(target=_serve, daemon=True)
+ thread.start()
+
+ async def _give_up():
+ with pytest.raises(asyncio.TimeoutError):
+ await asyncio.wait_for(alb._proxy_http_target(
+ {"Id": "127.0.0.1", "Port": port}, {"Port": port}, "GET", "/", {}, b"", {},
+ ), 0.5)
+ return await asyncio.to_thread(closed.wait, 5)
+
+ try:
+ assert asyncio.run(_give_up())
+ finally:
+ sock.close()
+ thread.join(timeout=35)
+
+
@pytest.mark.serial
def test_elbv2_dataplane_connection_close_target_gets_the_idle_timeout(elbv2):
"""A `Connection: close` target must be bounded by the idle timeout too.
@@ -2718,3 +2823,33 @@ def test_elbv2_modify_rule_with_empty_conditions_or_actions_changes_nothing(elbv
raw_elbv2.modify_rule(RuleArn=rule["RuleArn"], **params)
_refused(exc, "You must provide either conditions, actions, or transforms")
assert _described_rule(elbv2, rule["RuleArn"])["Transforms"] == [_url_rewrite()]
+
+
+def test_elbv2_target_headers_follow_xff_mode_and_trace_id_rules():
+ from ministack.services import alb
+
+ lb_arn = "arn:aws:elasticloadbalancing:us-east-1:000000000000:loadbalancer/app/xff-unit/1"
+ listener = {"Protocol": "HTTPS", "Port": 443}
+ try:
+ h = alb._target_headers(lb_arn, listener, {"X-Forwarded-For": "127.0.0.4, 127.0.0.8"}, "127.0.0.1")
+ assert h["x-forwarded-for"] == "127.0.0.4, 127.0.0.8, 127.0.0.1"
+ assert (h["x-forwarded-proto"], h["x-forwarded-port"]) == ("https", "443")
+ assert alb._target_headers(lb_arn, listener, {}, "127.0.0.1")["x-forwarded-for"] == "127.0.0.1"
+ assert alb._target_headers(lb_arn, listener, {"x-forwarded-for": "127.0.0.4"},
+ "2001:db8::1")["x-forwarded-for"] == "127.0.0.4, [2001:db8::1]"
+
+ trace = alb._target_headers(lb_arn, listener, {"x-amzn-trace-id": "Root=1-67891233-abcdef012345678912345678;CalledFrom=app"},
+ "127.0.0.1")["x-amzn-trace-id"].split(";")
+ assert re.fullmatch(r"Self=1-[0-9a-f]{8}-[0-9a-f]{24}", trace[0])
+ assert trace[1:] == ["Root=1-67891233-abcdef012345678912345678", "CalledFrom=app"]
+ trace = alb._target_headers(lb_arn, listener, {"x-amzn-trace-id": "Self=1-00000000-000000000000000000000000;Root=1-67891233-abcdef012345678912345678"},
+ "127.0.0.1")["x-amzn-trace-id"].split(";")
+ assert trace[0] != "Self=1-00000000-000000000000000000000000" and len(trace) == 2
+
+ alb._lb_attrs[lb_arn] = [{"Key": "routing.http.xff_header_processing.mode", "Value": "preserve"}]
+ assert alb._target_headers(lb_arn, listener, {"x-forwarded-for": "127.0.0.4"}, "127.0.0.1")["x-forwarded-for"] == "127.0.0.4"
+ assert "x-forwarded-for" not in alb._target_headers(lb_arn, listener, {}, "127.0.0.1")
+ alb._lb_attrs[lb_arn] = [{"Key": "routing.http.xff_header_processing.mode", "Value": "remove"}]
+ assert "x-forwarded-for" not in alb._target_headers(lb_arn, listener, {"x-forwarded-for": "127.0.0.4"}, "127.0.0.1")
+ finally:
+ alb._lb_attrs.pop(lb_arn, None)
diff --git a/tests/test_cloudfront.py b/tests/test_cloudfront.py
index 8fb82b74..e24f3f73 100644
--- a/tests/test_cloudfront.py
+++ b/tests/test_cloudfront.py
@@ -1,16 +1,29 @@
+import asyncio
import copy
+import http.client
+import http.server
+import io
import json
import os
import re
+import socket
+import ssl
+import threading
import time
import urllib.error
import urllib.parse
import urllib.request
import uuid as _uuid_mod
+import zipfile
+from unittest import mock
import pytest
from botocore.exceptions import ClientError
-from conftest import make_client
+from conftest import GATEWAY_PORT, make_client
+
+from ministack.core import cloudfront_js
+from ministack.core.responses import StreamingResponse
+from ministack.services import cloudfront as cloudfront_svc
ENDPOINT = os.environ.get("MINISTACK_ENDPOINT", "http://localhost:4566")
@@ -4656,3 +4669,1685 @@ def test_cloudfront_update_function_invalid_base64(cloudfront):
assert re.search(r"<Message>(.*?)</Message>", text).group(1) == "Invalid Base64 encoding"
finally:
_cf_delete_function_if_present(cloudfront, name)
+
+# ---------------------------------------------------------------------------
+# Data plane: a distribution serving traffic over the wire at <label>.cloudfront.<MINISTACK_HOST>.
+# cf_stack fronts an HTTP API whose Lambda echoes what it receives.
+# ---------------------------------------------------------------------------
+
+# Managed policy ids (CloudFront Developer Guide, "Managed origin request
+# policies" / "Managed cache policies" / "Managed response headers
+# policies").
+_ALL_VIEWER = "216adef6-5c7f-47e4-b989-5492eafa07d3"
+_ALL_VIEWER_EXCEPT_HOST = "b689b0a8-53d0-40ab-baf2-68738e2966ac"
+_CACHING_DISABLED = "4135ea2d-6df8-44a3-9df3-4b5a84be39ad"
+_MANAGED_CORS_WITH_PREFLIGHT = "5cc3b908-e619-4b99-88e5-2cf7f45965bd"
+
+_ORIGIN_LAMBDA_CODE = (
+ b"import json\n"
+ b"def handler(event, context):\n"
+ b" path = event.get('rawPath', '')\n"
+ b" headers = event.get('headers', {})\n"
+ b" # MiniStack's apigatewayv2 (payload format 2.0) does not split an\n"
+ b" # incoming Cookie header into the top-level `cookies` array real AWS\n"
+ b" # uses; it stays in `headers['cookie']` like any other header, which\n"
+ b" # is also what cloudfront_svc.py actually forwards.\n"
+ b" cookie_header = headers.get('cookie', '')\n"
+ b" cookies = [c.strip() for c in cookie_header.split(';') if c.strip()]\n"
+ b" resp = {\n"
+ b" 'statusCode': 200,\n"
+ b" 'headers': {'Content-Type': 'application/json', 'X-Origin-Header': 'origin-value'},\n"
+ b" 'body': json.dumps({\n"
+ b" 'path': path,\n"
+ b" 'headers': headers,\n"
+ b" 'query': event.get('queryStringParameters') or {},\n"
+ b" 'cookies': cookies,\n"
+ b" }),\n"
+ b" }\n"
+ b" if path == '/cookies':\n"
+ b" resp['cookies'] = ['resp1=a; Path=/', 'resp2=b; Path=/']\n"
+ b" if path == '/status/500':\n"
+ b" resp['statusCode'] = 500\n"
+ b" return resp\n"
+)
+
+
+def _zip_lambda(code: bytes) -> bytes:
+ buf = io.BytesIO()
+ with zipfile.ZipFile(buf, "w") as zf:
+ zf.writestr("index.py", code)
+ return buf.getvalue()
+
+
+def _none_cache_policy_config(name: str) -> dict:
+ return {
+ "Name": name,
+ "MinTTL": 0,
+ "ParametersInCacheKeyAndForwardedToOrigin": {
+ "HeadersConfig": {"HeaderBehavior": "none"},
+ "CookiesConfig": {"CookieBehavior": "none"},
+ "QueryStringsConfig": {"QueryStringBehavior": "none"},
+ "EnableAcceptEncodingGzip": False,
+ },
+ }
+
+
+def _response_headers_policy(cloudfront, name: str, custom_headers: list) -> str:
+ resp = cloudfront.create_response_headers_policy(
+ ResponseHeadersPolicyConfig={
+ "Name": name,
+ "CustomHeadersConfig": {
+ "Quantity": len(custom_headers),
+ "Items": [{"Header": h, "Value": v, "Override": o} for (h, v, o) in custom_headers],
+ },
+ }
+ )
+ return resp["ResponseHeadersPolicy"]["Id"]
+
+
+def _create_function(cloudfront, name: str, code: bytes, runtime: str = "cloudfront-js-2.0") -> str:
+ created = cloudfront.create_function(
+ Name=name, FunctionConfig={"Comment": name, "Runtime": runtime}, FunctionCode=code,
+ )
+ published = cloudfront.publish_function(Name=name, IfMatch=created["ETag"])
+ return published["FunctionSummary"]["FunctionMetadata"]["FunctionARN"]
+
+
[email protected](scope="module")
+def cf_backend(apigw, lam):
+ """A custom API Gateway HTTP API, backed by a Lambda that echoes the
+ inbound request, addressed like a real CloudFront custom origin."""
+ suffix = _uuid_mod.uuid4().hex[:10]
+ lam.create_function(
+ FunctionName=f"cf-dp-origin-{suffix}",
+ Runtime="python3.12",
+ Role="arn:aws:iam::000000000000:role/test-role",
+ Handler="index.handler",
+ Code={"ZipFile": _zip_lambda(_ORIGIN_LAMBDA_CODE)},
+ )
+ api_id = apigw.create_api(Name=f"cf-dp-api-{suffix}", ProtocolType="HTTP")["ApiId"]
+ int_id = apigw.create_integration(
+ ApiId=api_id, IntegrationType="AWS_PROXY",
+ IntegrationUri=f"arn:aws:lambda:us-east-1:000000000000:function:cf-dp-origin-{suffix}",
+ PayloadFormatVersion="2.0",
+ )["IntegrationId"]
+ apigw.create_route(ApiId=api_id, RouteKey="ANY /{proxy+}", Target=f"integrations/{int_id}")
+ apigw.create_stage(ApiId=api_id, StageName="$default")
+ # "*.execute-api.localhost" resolves to 127.0.0.1 (RFC 6761); the origin
+ # is dialed there at the gateway's own listening port, exactly as any
+ # custom origin is dialed at its configured DomainName/HTTPPort.
+ return {"origin_domain": f"{api_id}.execute-api.localhost"}
+
+
[email protected](scope="module")
+def cf_stack(cloudfront, cf_backend):
+ """One distribution wired up per the module docstring."""
+ suffix = _uuid_mod.uuid4().hex[:10]
+ origin_domain = cf_backend["origin_domain"]
+
+ cp_none_id = cloudfront.create_cache_policy(
+ CachePolicyConfig=_none_cache_policy_config(f"cf-dp-cp-none-{suffix}")
+ )["CachePolicy"]["Id"]
+
+ default_policy_id = _response_headers_policy(
+ cloudfront, f"cf-dp-default-policy-{suffix}",
+ [("X-Origin-Header", "policy-value", True), ("X-Default-Marker", "yes", False)],
+ )
+ csv_policy_id = _response_headers_policy(
+ cloudfront, f"cf-dp-csv-policy-{suffix}", [("X-Csv-Marker", "yes", False)],
+ )
+
+ request_fn_arn = _create_function(
+ cloudfront, f"cf-dp-viewer-request-{suffix}",
+ b"function handler(event) {\n"
+ b" var request = event.request;\n"
+ b" if (request.uri === '/fn/blocked') {\n"
+ b" return { statusCode: 403, statusDescription: 'Forbidden',\n"
+ b" headers: { 'content-type': { value: 'text/plain' } },\n"
+ b" body: { encoding: 'text', data: 'blocked by viewer-request' } };\n"
+ b" }\n"
+ b" request.uri = request.uri.replace('/fn/', '/');\n"
+ b" return request;\n"
+ b"}\n",
+ )
+ response_fn_arn = _create_function(
+ cloudfront, f"cf-dp-viewer-response-{suffix}",
+ b"function handler(event) {\n"
+ b" var response = event.response;\n"
+ b" response.headers['x-viewer-response-added'] = { value: 'yes' };\n"
+ b" return response;\n"
+ b"}\n",
+ )
+ async_fn_arn = _create_function(
+ cloudfront, f"cf-dp-async-{suffix}",
+ b"async function handler(event) {\n"
+ b" var v = await Promise.resolve('yes');\n"
+ b" var request = event.request;\n"
+ b" request.headers['x-async-ran'] = { value: v };\n"
+ b" request.uri = request.uri.replace('/async/', '/');\n"
+ b" return request;\n"
+ b"}\n",
+ )
+ error_fn_arn = _create_function(
+ cloudfront, f"cf-dp-error-{suffix}",
+ b"function handler(event) { throw new Error('boom'); }\n",
+ )
+ response_delete_multi_fn_arn = _create_function(
+ cloudfront, f"cf-dp-response-delete-multi-{suffix}",
+ b"function handler(event) {\n"
+ b" var response = event.response;\n"
+ b" delete response.headers['x-origin-header'];\n"
+ b" response.headers['x-multi-resp'] = { value: 'a', multiValue: [{ value: 'a' }, { value: 'b' }] };\n"
+ b" return response;\n"
+ b"}\n",
+ )
+ viewer_ip_fn_arn = _create_function(
+ cloudfront, f"cf-dp-viewer-ip-{suffix}",
+ b"function handler(event) {\n"
+ b" var request = event.request;\n"
+ b" request.headers['x-viewer-ip'] = { value: event.viewer.ip };\n"
+ b" return request;\n"
+ b"}\n",
+ )
+
+ def behavior(path_pattern, target_origin_id="origin-apigw", **overrides):
+ b = {
+ "PathPattern": path_pattern,
+ "TargetOriginId": target_origin_id,
+ "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": cp_none_id,
+ "OriginRequestPolicyId": _ALL_VIEWER_EXCEPT_HOST,
+ "ResponseHeadersPolicyId": default_policy_id,
+ }
+ b.update(overrides)
+ return b
+
+ dist_config = {
+ "CallerReference": f"cf-dp-{suffix}",
+ "Comment": "data plane test distribution",
+ "Enabled": True,
+ "DefaultRootObject": "index.html",
+ "Origins": {
+ "Quantity": 2,
+ "Items": [
+ {
+ "Id": "origin-apigw",
+ "DomainName": origin_domain,
+ "CustomOriginConfig": {
+ "HTTPPort": int(GATEWAY_PORT), "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
+ },
+ },
+ {
+ "Id": "origin-apigw-prefixed",
+ "DomainName": origin_domain,
+ "OriginPath": "/extra",
+ "CustomHeaders": {
+ "Quantity": 1,
+ "Items": [{"HeaderName": "X-Added-By-Origin", "HeaderValue": "yes"}],
+ },
+ "CustomOriginConfig": {
+ "HTTPPort": int(GATEWAY_PORT), "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
+ },
+ },
+ ],
+ },
+ "DefaultCacheBehavior": {
+ k: v for k, v in behavior("*").items() if k != "PathPattern"
+ },
+ "CacheBehaviors": {
+ "Quantity": 8,
+ "Items": [
+ {**behavior("*.csv"), "ResponseHeadersPolicyId": csv_policy_id},
+ {
+ **behavior("/fn/*"),
+ "FunctionAssociations": {
+ "Quantity": 2,
+ "Items": [
+ {"FunctionARN": request_fn_arn, "EventType": "viewer-request"},
+ {"FunctionARN": response_fn_arn, "EventType": "viewer-response"},
+ ],
+ },
+ },
+ {
+ **behavior("/async/*"),
+ "FunctionAssociations": {
+ "Quantity": 1,
+ "Items": [{"FunctionARN": async_fn_arn, "EventType": "viewer-request"}],
+ },
+ },
+ {
+ **behavior("/err/*"),
+ "FunctionAssociations": {
+ "Quantity": 1,
+ "Items": [{"FunctionARN": error_fn_arn, "EventType": "viewer-request"}],
+ },
+ },
+ {**behavior("/cd/*"), "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": ""},
+ {
+ **{k: v for k, v in behavior("/legacy/*").items()
+ if k not in ("CachePolicyId", "OriginRequestPolicyId")},
+ "MinTTL": 0, "DefaultTTL": 0, "MaxTTL": 0,
+ "ForwardedValues": {
+ "QueryString": True,
+ "Cookies": {"Forward": "whitelist", "WhitelistedNames": {"Quantity": 1, "Items": ["keep"]}},
+ "Headers": {"Quantity": 1, "Items": ["X-Legacy"]},
+ },
+ },
+ {**behavior("/restricted/*"),
+ "AllowedMethods": {"Quantity": 2, "Items": ["GET", "HEAD"]}},
+ # No leading "/" — CloudFront treats it the same as "/api/*".
+ behavior("api/*"),
+ {
+ **behavior("/cors/*"), "ResponseHeadersPolicyId": _MANAGED_CORS_WITH_PREFLIGHT,
+ "AllowedMethods": {"Quantity": 3, "Items": ["GET", "HEAD", "OPTIONS"]},
+ },
+ {
+ **behavior("/respmulti/*"),
+ "FunctionAssociations": {
+ "Quantity": 1,
+ "Items": [{"FunctionARN": response_delete_multi_fn_arn, "EventType": "viewer-response"}],
+ },
+ },
+ {
+ **behavior("/viewerip/*"),
+ "FunctionAssociations": {
+ "Quantity": 1,
+ "Items": [{"FunctionARN": viewer_ip_fn_arn, "EventType": "viewer-request"}],
+ },
+ },
+ ],
+ },
+ }
+ # "/cd/*" carries no OriginRequestPolicyId at all; the empty-string
+ # placeholder above exists only so `behavior()`'s dict-merge has a key to
+ # drop — CloudFront's own XML omits the element entirely for "no ORP".
+ for item in dist_config["CacheBehaviors"]["Items"]:
+ if item.get("OriginRequestPolicyId") == "":
+ del item["OriginRequestPolicyId"]
+ dist_config["CacheBehaviors"]["Items"].append(
+ {**behavior("/redirect/*"), "ViewerProtocolPolicy": "redirect-to-https"}
+ )
+ dist_config["CacheBehaviors"]["Items"].append(
+ {**behavior("/prefixed/*", target_origin_id="origin-apigw-prefixed")}
+ )
+ dist_config["CacheBehaviors"]["Quantity"] = len(dist_config["CacheBehaviors"]["Items"])
+
+ dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
+ return {"dist_host": dist["DomainName"], "origin_domain": origin_domain}
+
+
+def _get(host: str, path: str, method: str = "GET", extra_headers: dict | None = None):
+ """``host`` is a distribution's real AWS-shaped DomainName
+ (``<label>.cloudfront.net``), sent verbatim as the Host header — exactly
+ what a real viewer sends. The connection itself targets the gateway's own
+ address directly, since ``*.cloudfront.net`` doesn't resolve locally (the
+ same way ``_raw_get`` below reaches an execute-api/ALB host)."""
+ conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
+ try:
+ headers = {"Host": f"{host}:{GATEWAY_PORT}"}
+ headers.update(extra_headers or {})
+ conn.request(method, path, headers=headers)
+ resp = conn.getresponse()
+ return resp.status, dict(resp.getheaders()), resp.read()
+ finally:
+ conn.close()
+
+
+def _skip_without_node():
+ if not cloudfront_js.available():
+ pytest.skip("node is not available")
+
+
+# ---------------------------------------------------------------------------
+# Routing
+# ---------------------------------------------------------------------------
+
+
+def test_routes_by_distribution_domain(cf_stack):
+ status, _headers, body = _get(cf_stack["dist_host"], "/hello")
+ assert status == 200
+ assert json.loads(body)["path"] == "/hello"
+
+
+def test_path_pattern_without_leading_slash_matches_as_if_it_had_one(cf_stack):
+ # PathPattern "api/*" (no leading /) — CloudFront Developer Guide,
+ # "Path pattern": "CloudFront treats the path the same with or without
+ # the leading /".
+ status, _headers, body = _get(cf_stack["dist_host"], "/api/hello")
+ assert status == 200
+ assert json.loads(body)["path"] == "/api/hello"
+
+
+def test_unknown_distribution_label_is_not_served_by_cloudfront():
+ # Not a 200, and not CloudFront's own data plane (no X-Cache): falls
+ # through to whatever the gateway's generic router answers for an
+ # otherwise-unrouted host — CloudFront's own control-plane 404, since no
+ # other special-case host pattern matches either.
+ status, headers, body = _get("d0000000000000.cloudfront.localhost", "/anything")
+ assert "X-Cache" not in headers
+ assert status == 404
+ assert b"<Code>NoSuchResource</Code>" in body
+
+
+# ---------------------------------------------------------------------------
+# CloudFront Functions
+# ---------------------------------------------------------------------------
+
+
+def test_viewer_request_function_rewrites_uri(cf_stack):
+ _skip_without_node()
+ status, _headers, body = _get(cf_stack["dist_host"], "/fn/hello")
+ assert status == 200
+ assert json.loads(body)["path"] == "/hello"
+
+
+def test_function_generated_response_skips_the_origin(cf_stack):
+ _skip_without_node()
+ status, headers, body = _get(cf_stack["dist_host"], "/fn/blocked")
+ assert status == 403
+ assert body == b"blocked by viewer-request"
+ assert headers.get("X-Cache") == "FunctionGeneratedResponse from cloudfront"
+ assert "X-Origin-Header" not in headers # never reached the origin
+
+
+def test_viewer_response_function_adds_a_header(cf_stack):
+ _skip_without_node()
+ status, headers, _body = _get(cf_stack["dist_host"], "/fn/hello")
+ assert status == 200
+ assert headers.get("X-Viewer-Response-Added") == "yes"
+
+
+def test_async_2_0_handler_runs_and_awaits(cf_stack):
+ _skip_without_node()
+ status, _headers, body = _get(cf_stack["dist_host"], "/async/hello")
+ assert status == 200
+ payload = json.loads(body)
+ assert payload["path"] == "/hello"
+ assert payload["headers"].get("x-async-ran") == "yes"
+
+
+def test_function_error_answers_503(cf_stack):
+ _skip_without_node()
+ status, headers, _body = _get(cf_stack["dist_host"], "/err/hello")
+ assert status == 503
+ assert headers.get("X-Cache") == "Error from cloudfront"
+
+
+def test_viewer_response_function_skipped_when_origin_status_is_400_or_above(cf_stack):
+ # functions-event-structure.html, "Status code and body" note: CloudFront
+ # does not invoke a viewer-response function when the origin answers 400+.
+ _skip_without_node()
+ status, headers, _body = _get(cf_stack["dist_host"], "/fn/status/500")
+ assert status == 500
+ assert "X-Viewer-Response-Added" not in headers
+
+
+def test_viewer_response_function_deletes_header_and_expands_multivalue(cf_stack):
+ # Replace, not merge (a deleted header stays gone), and a multiValue
+ # response header becomes repeated Set-Cookie-style lines.
+ _skip_without_node()
+ conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
+ try:
+ conn.putrequest("GET", "/respmulti/hello", skip_host=True, skip_accept_encoding=True)
+ conn.putheader("Host", f"{cf_stack['dist_host']}:{GATEWAY_PORT}")
+ conn.endheaders()
+ resp = conn.getresponse()
+ resp.read()
+ multi_values = resp.msg.get_all("X-Multi-Resp") or []
+ has_origin_header = "X-Origin-Header" in resp.msg
+ finally:
+ conn.close()
+ assert sorted(multi_values) == ["a", "b"]
+ assert not has_origin_header
+
+
+def test_function_event_viewer_ip_is_the_tcp_peer_not_the_xff_header(cf_stack):
+ # functions-event-structure.html: viewer.ip is the TCP peer, regardless
+ # of what X-Forwarded-For the viewer sent.
+ _skip_without_node()
+ status, _headers, body = _get(
+ cf_stack["dist_host"], "/viewerip/hello", extra_headers={"X-Forwarded-For": "203.0.113.9"},
+ )
+ assert status == 200
+ assert json.loads(body)["headers"].get("x-viewer-ip") == "127.0.0.1"
+
+
+# ---------------------------------------------------------------------------
+# Cache behaviors and response-headers policy
+# ---------------------------------------------------------------------------
+
+
+def test_ordered_behavior_matches_csv_glob_with_its_own_policy(cf_stack):
+ status, headers, _body = _get(cf_stack["dist_host"], "/report.csv")
+ assert status == 200
+ assert headers.get("X-Csv-Marker") == "yes"
+ assert "X-Default-Marker" not in headers
+
+
+def test_response_headers_policy_overrides_origin_header_and_adds_marker(cf_stack):
+ # The origin always emits X-Origin-Header: origin-value; the default
+ # behavior's response-headers policy names the same header with
+ # Override=True, and adds X-Default-Marker unconditionally.
+ status, headers, _body = _get(cf_stack["dist_host"], "/hello")
+ assert status == 200
+ assert headers.get("X-Origin-Header") == "policy-value"
+ assert headers.get("X-Default-Marker") == "yes"
+
+
+def test_cors_headers_only_applied_when_origin_header_present(cf_stack):
+ status, headers, _body = _get(cf_stack["dist_host"], "/cors/hello")
+ assert status == 200
+ assert "Access-Control-Allow-Origin" not in headers
+
+ status, headers, _body = _get(
+ cf_stack["dist_host"], "/cors/hello", extra_headers={"Origin": "https://example.com"},
+ )
+ assert status == 200
+ assert headers.get("Access-Control-Allow-Origin") == "*"
+ assert headers.get("Access-Control-Expose-Headers") == "*"
+ # Preflight-only headers (Allow-Methods/-Headers/-Max-Age) never appear
+ # on a plain GET.
+ assert "Access-Control-Allow-Methods" not in headers
+
+
+def test_cors_preflight_headers_only_on_options(cf_stack):
+ # The response-headers policy applies to whatever status the request
+ # produces (here, the backing API Gateway HTTP API's own unconfigured-
+ # CORS OPTIONS rejection — orthogonal to the response-headers policy
+ # under test), so only the header is asserted, not the status.
+ _status, headers, _body = _get(
+ cf_stack["dist_host"], "/cors/hello", method="OPTIONS",
+ extra_headers={"Origin": "https://example.com"},
+ )
+ assert headers.get("Access-Control-Allow-Methods") == "DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT"
+
+
+def test_cloudfront_added_headers_present(cf_stack):
+ status, headers, _body = _get(cf_stack["dist_host"], "/hello")
+ assert status == 200
+ assert headers.get("X-Cache") == "Miss from cloudfront"
+ assert headers.get("Via", "").endswith(".cloudfront.net (CloudFront)")
+ assert "X-Amz-Cf-Id" in headers
+ assert "X-Amz-Cf-Pop" in headers
+
+
+# ---------------------------------------------------------------------------
+# Cache-policy / origin-request-policy forwarding
+# ---------------------------------------------------------------------------
+
+
+def test_all_viewer_except_host_header_forwards_viewer_header_not_host(cf_stack):
+ status, _headers, body = _get(
+ cf_stack["dist_host"], "/hello", extra_headers={"X-Custom-Test": "present"},
+ )
+ assert status == 200
+ payload = json.loads(body)
+ assert payload["headers"].get("x-custom-test") == "present"
+ assert payload["headers"].get("host") == cf_stack["origin_domain"]
+
+
+def test_all_viewer_forwards_the_distributions_own_host(cf_raw_origin_stack):
+ # AllViewer forwards every viewer header including Host, so the origin
+ # sees the *distribution's* Host, not a default derived from its own
+ # DomainName. A real network-distinct origin doesn't care; this uses the
+ # raw http.server origin (not the execute-api one) because MiniStack's
+ # own execute-api addressing is itself Host-routed, so forwarding the
+ # viewer's Host there would recurse into the gateway's own routing
+ # instead of reaching the intended origin — an artifact of this
+ # emulator's single-port addressing, not of CloudFront's own behavior.
+ status, _headers, body = _get(
+ cf_raw_origin_stack, "/av/hello", extra_headers={"X-Custom-Test": "present"},
+ )
+ assert status == 200
+ payload = json.loads(body)
+ assert payload["headers"].get("x-custom-test") == "present"
+ assert payload["headers"].get("host") == f"{cf_raw_origin_stack}:{GATEWAY_PORT}"
+
+
+def test_caching_disabled_with_no_origin_request_policy_forwards_only_defaults(cf_stack):
+ # CachingDisabled's ParametersInCacheKeyAndForwardedToOrigin is
+ # none/none/none, and the behavior carries no origin request policy at
+ # all: nothing beyond the always-included defaults (Host, User-Agent)
+ # reaches the origin.
+ status, _headers, body = _get(
+ cf_stack["dist_host"], "/cd/hello", extra_headers={"X-Custom-Test": "present"},
+ )
+ assert status == 200
+ payload = json.loads(body)
+ assert "x-custom-test" not in payload["headers"]
+ assert payload["headers"].get("host") == cf_stack["origin_domain"]
+
+
+def test_x_forwarded_for_always_appends_regardless_of_policy(cf_stack):
+ # CloudFront Developer Guide, "Client IP addresses": X-Forwarded-For is
+ # always the viewer's own chain plus the client IP, even under
+ # CachingDisabled with no origin request policy forwarding it.
+ status, _headers, body = _get(
+ cf_stack["dist_host"], "/cd/hello", extra_headers={"X-Forwarded-For": "203.0.113.9"},
+ )
+ assert status == 200
+ payload = json.loads(body)
+ assert payload["headers"].get("x-forwarded-for") == "203.0.113.9, 127.0.0.1"
+
+
+def test_user_agent_defaults_to_amazon_cloudfront_when_not_forwarded(cf_stack):
+ # CloudFront Developer Guide, "User-Agent header": replaced unconditionally
+ # unless the forwarding policy carries the viewer's own.
+ status, _headers, body = _get(
+ cf_stack["dist_host"], "/cd/hello", extra_headers={"User-Agent": "custom-client/1.0"},
+ )
+ assert status == 200
+ assert json.loads(body)["headers"].get("user-agent") == "Amazon CloudFront"
+
+
+def test_legacy_forwarded_values_forwards_only_its_own_selections(cf_stack):
+ status, _headers, body = _get(
+ cf_stack["dist_host"], "/legacy/hello?a=1&b=2",
+ extra_headers={"X-Legacy": "yes", "X-Not-Forwarded": "no", "Cookie": "keep=1; drop=2"},
+ )
+ assert status == 200
+ payload = json.loads(body)
+ assert payload["query"] == {"a": "1", "b": "2"} # ForwardedValues.QueryString=true
+ assert payload["headers"].get("x-legacy") == "yes"
+ assert "x-not-forwarded" not in payload["headers"]
+ assert payload["cookies"] == ["keep=1"] # only the whitelisted cookie name
+
+
+# ---------------------------------------------------------------------------
+# _combine_forward — the cache-policy / origin-request-policy combinator
+# (same function serves headers, cookies and query strings). Exercised
+# end-to-end above for "all"/"none"; "whitelist" and "allExcept" are cheaper
+# to prove directly against the pure combinator than via more distributions.
+# ---------------------------------------------------------------------------
+
+
+ "cache_kind_set,orp_kind_set,name,expected",
+ [
+ (("whitelist", {"a"}), None, "a", True),
+ (("whitelist", {"a"}), None, "b", False),
+ (("whitelist", {"a"}), ("whitelist", {"b"}), "b", True), # ORP allow-list adds to cache's
+ (("allExcept", {"a"}), None, "a", False), # blocked by the cache policy
+ (("allExcept", {"a"}), None, "b", True),
+ (("allExcept", {"a"}), ("allExcept", {"b"}), "a", False), # ORP allExcept doesn't rescue it
+ (("allExcept", {"a"}), ("whitelist", {"a"}), "a", True), # ORP allow-list rescues a blocked name
+ (("none", None), ("allExcept", {"a"}), "a", False),
+ (("none", None), ("allExcept", {"a"}), "b", True),
+ (("none", None), None, "a", False),
+ ],
+)
+def test_combine_forward_whitelist_and_allexcept_branches(cache_kind_set, orp_kind_set, name, expected):
+ assert cloudfront_svc._combine_forward(cache_kind_set, orp_kind_set, name) == expected
+
+
+# ---------------------------------------------------------------------------
+# Behavior-level settings
+# ---------------------------------------------------------------------------
+
+
+def test_redirect_to_https_sends_301_with_https_location(cf_stack):
+ status, headers, _body = _get(cf_stack["dist_host"], "/redirect/hello")
+ assert status == 301
+ assert headers.get("Location") == f"https://{cf_stack['dist_host']}:{GATEWAY_PORT}/redirect/hello"
+
+
+def test_allowed_methods_rejects_a_disallowed_method(cf_stack):
+ status, _headers, _body = _get(cf_stack["dist_host"], "/restricted/hello", method="POST")
+ assert status == 403
+
+
+def test_default_root_object_is_requested_for_the_root_url(cf_stack):
+ status, _headers, body = _get(cf_stack["dist_host"], "/")
+ assert status == 200
+ assert json.loads(body)["path"] == "/index.html"
+
+
+def test_origin_path_and_custom_headers_reach_the_origin(cf_stack):
+ status, _headers, body = _get(cf_stack["dist_host"], "/prefixed/hello")
+ assert status == 200
+ payload = json.loads(body)
+ assert payload["path"] == "/extra/prefixed/hello"
+ assert payload["headers"].get("x-added-by-origin") == "yes"
+
+
+def test_multiple_set_cookie_headers_all_carried_through(cf_stack):
+ conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
+ try:
+ conn.putrequest("GET", "/cookies", skip_host=True, skip_accept_encoding=True)
+ conn.putheader("Host", f"{cf_stack['dist_host']}:{GATEWAY_PORT}")
+ conn.endheaders()
+ resp = conn.getresponse()
+ resp.read()
+ cookies = resp.msg.get_all("Set-Cookie") or []
+ finally:
+ conn.close()
+ assert sorted(cookies) == ["resp1=a; Path=/", "resp2=b; Path=/"]
+
+
+# ---------------------------------------------------------------------------
+# AWS-owned origin hosts — refused before a socket opens; no server or
+# distribution needed, calls the pure function. An emulator must never open
+# a socket to a real AWS host.
+# ---------------------------------------------------------------------------
+
+
+def test_aws_owned_origin_host_refused_before_any_socket():
+ origin = {
+ "domain_name": "abcd1234.execute-api.us-east-1.amazonaws.com.", # trailing dot
+ "origin_path": "", "custom_headers": [], "s3_bucket": None,
+ "http_port": 80, "https_port": 443, "protocol_policy": "https-only",
+ }
+ with mock.patch("http.client.HTTPSConnection") as https_conn, \
+ mock.patch("http.client.HTTPConnection") as http_conn:
+ with pytest.raises(cloudfront_svc.OriginUnreachable):
+ cloudfront_svc._forward_to_origin(
+ origin, "GET", "/anything", "", {}, b"", viewer_is_https=True,
+ )
+ https_conn.assert_not_called()
+ http_conn.assert_not_called()
+
+
+def test_origin_ssl_context_trusts_gateway_cert_under_use_ssl(tmp_path):
+ # Precedent: lambda_svc.py's TLS trust for Lambda containers — system
+ # roots plus the gateway's own cert, not a replacement of the store.
+ from ministack.core.x509_utils import generate_ca
+
+ ca_pem, _ca_key_pem = generate_ca(common_name="cf-dp-origin-ssl-test")
+ cert_path = tmp_path / "ca.pem"
+ cert_path.write_text(ca_pem)
+ with mock.patch("ministack.core.tls.use_ssl_enabled", return_value=True), \
+ mock.patch("ministack.core.tls.resolve_tls_material", return_value=(str(cert_path), "key")):
+ ctx = cloudfront_svc._origin_ssl_context()
+ assert ctx.verify_flags & ssl.VERIFY_X509_PARTIAL_CHAIN
+ subjects = [dict(rdn for pair in cert["subject"] for rdn in pair) for cert in ctx.get_ca_certs()]
+ assert any(s.get("commonName") == "cf-dp-origin-ssl-test" for s in subjects)
+
+
+def test_stacked_distribution_via_header_answers_403(cloudfront):
+ # CloudFront Developer Guide, http-403-permission-denied.html, "Stacked
+ # distributions cause a 403 error": an origin pointed back at this same
+ # distribution (via the resolvable `<label>.cloudfront.localhost` form)
+ # must answer 403 on the second hop rather than recurse into a hang.
+ suffix = _uuid_mod.uuid4().hex[:10]
+ dist_config = {
+ "CallerReference": f"cf-dp-stacked-{suffix}",
+ "Comment": "data plane stacked-distribution test",
+ "Enabled": True,
+ "Origins": {
+ "Quantity": 1,
+ "Items": [{
+ "Id": "self-origin",
+ "DomainName": "placeholder.cloudfront.localhost",
+ "CustomOriginConfig": {
+ "HTTPPort": int(GATEWAY_PORT), "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
+ },
+ }],
+ },
+ "DefaultCacheBehavior": {
+ "TargetOriginId": "self-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED,
+ },
+ }
+ created = cloudfront.create_distribution(DistributionConfig=dist_config)
+ dist = created["Distribution"]
+ label = dist["DomainName"].removesuffix(".cloudfront.net")
+ dist_config["Origins"]["Items"][0]["DomainName"] = f"{label}.cloudfront.localhost"
+ cloudfront.update_distribution(DistributionConfig=dist_config, Id=dist["Id"], IfMatch=created["ETag"])
+
+ status, _headers, _body = _get(dist["DomainName"], "/anything")
+ assert status == 403
+
+
+def test_unreachable_origin_answers_502(cloudfront):
+ suffix = _uuid_mod.uuid4().hex[:10]
+ dist_config = {
+ "CallerReference": f"cf-dp-dead-{suffix}",
+ "Comment": "data plane dead-origin test",
+ "Enabled": True,
+ "Origins": {
+ "Quantity": 1,
+ "Items": [{
+ "Id": "dead-origin",
+ "DomainName": "127.0.0.1",
+ "CustomOriginConfig": {
+ "HTTPPort": 1, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only", # nothing listens on port 1
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
+ },
+ }],
+ },
+ "DefaultCacheBehavior": {
+ "TargetOriginId": "dead-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED,
+ },
+ }
+ dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
+ status, headers, _body = _get(dist["DomainName"], "/anything")
+ assert status == 502
+ assert headers.get("X-Cache") == "Error from cloudfront"
+
+
+def test_forward_to_origin_timeout_raises_origin_timeout():
+ # CloudFront Developer Guide, http-504-gateway-timeout.html: "The origin
+ # didn't respond before the request expired." OriginReadTimeout default
+ # 30s (DownloadDistValuesOrigin.md, "Response timeout").
+ origin = {
+ "domain_name": "127.0.0.1", "origin_path": "", "custom_headers": [], "s3_bucket": None,
+ "http_port": 1, "https_port": 443, "protocol_policy": "http-only", "read_timeout": 30,
+ }
+ with mock.patch("http.client.HTTPConnection") as mock_conn_cls:
+ mock_conn_cls.return_value.request.side_effect = TimeoutError("timed out")
+ with pytest.raises(cloudfront_svc.OriginTimeout):
+ cloudfront_svc._forward_to_origin(
+ origin, "GET", "/anything", "", {}, b"", viewer_is_https=False,
+ )
+
+
+def test_handle_request_maps_origin_timeout_to_504():
+ parsed = {
+ "default_root_object": "",
+ "origins": {"o1": {
+ "domain_name": "127.0.0.1", "origin_path": "", "custom_headers": [],
+ "s3_bucket": None, "http_port": 1, "https_port": 443,
+ "protocol_policy": "http-only", "read_timeout": 30,
+ }},
+ "default_behavior": {
+ "path_pattern": None, "target_origin_id": "o1", "viewer_protocol_policy": "allow-all",
+ "allowed_methods": None, "cache_policy_id": _CACHING_DISABLED,
+ "origin_request_policy_id": None, "response_headers_policy_id": None,
+ "forwarded_values": None, "functions": {},
+ },
+ "ordered_behaviors": [],
+ }
+ dist = {"Id": "EDUMMY00000000", "DomainName": "ddummy0000000.cloudfront.net"}
+ with mock.patch("ministack.services.cloudfront.parse_distribution_dataplane_config", return_value=parsed), \
+ mock.patch("ministack.services.cloudfront._forward_to_origin",
+ side_effect=cloudfront_svc.OriginTimeout("slow")):
+ status, headers, _body = asyncio.run(
+ cloudfront_svc.handle_viewer_request(dist, "GET", "/anything", "/anything", "", {}, b"", {})
+ )
+ assert status == 504
+ assert headers.get("X-Cache") == "Error from cloudfront"
+
+
+# ---------------------------------------------------------------------------
+# S3 origin — served from MiniStack's own S3 store in-process.
+# ---------------------------------------------------------------------------
+
+
[email protected](scope="module")
+def cf_s3_stack(cloudfront, s3):
+ suffix = _uuid_mod.uuid4().hex[:10]
+ bucket = f"cf-dp-s3-origin-{suffix}"
+ s3.create_bucket(Bucket=bucket)
+ body = b"hello from the bucket\n"
+ s3.put_object(Bucket=bucket, Key="hello.txt", Body=body, ContentType="text/plain")
+
+ dist_config = {
+ "CallerReference": f"cf-dp-s3-{suffix}",
+ "Comment": "data plane S3-origin test distribution",
+ "Enabled": True,
+ "Origins": {
+ "Quantity": 1,
+ "Items": [{
+ "Id": "s3-origin",
+ "DomainName": f"{bucket}.s3.amazonaws.com",
+ "S3OriginConfig": {"OriginAccessIdentity": ""},
+ "OriginAccessControlId": "test-oac-id",
+ }],
+ },
+ "DefaultCacheBehavior": {
+ "TargetOriginId": "s3-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED,
+ },
+ }
+ dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
+ s3.put_bucket_policy(Bucket=bucket, Policy=json.dumps({"Statement": [{
+ "Effect": "Allow", "Principal": {"Service": "cloudfront.amazonaws.com"},
+ "Action": "s3:GetObject", "Resource": f"arn:aws:s3:::{bucket}/*",
+ "Condition": {"StringEquals": {"AWS:SourceArn": dist["ARN"]}},
+ }]}))
+ return {"dist_host": dist["DomainName"], "body": body}
+
+
+def test_s3_origin_without_a_bucket_policy_grant_is_denied(cloudfront, s3):
+ """An OAC origin whose bucket policy does not grant the distribution answers 403."""
+ suffix = _uuid_mod.uuid4().hex[:10]
+ bucket = f"cf-dp-s3-nogrant-{suffix}"
+ s3.create_bucket(Bucket=bucket)
+ s3.put_object(Bucket=bucket, Key="hello.txt", Body=b"x")
+ dist = cloudfront.create_distribution(DistributionConfig={
+ "CallerReference": f"cf-dp-s3-nogrant-{suffix}", "Comment": "", "Enabled": True,
+ "Origins": {"Quantity": 1, "Items": [{
+ "Id": "s3-origin", "DomainName": f"{bucket}.s3.amazonaws.com",
+ "S3OriginConfig": {"OriginAccessIdentity": ""}, "OriginAccessControlId": "test-oac-id",
+ }]},
+ "DefaultCacheBehavior": {"TargetOriginId": "s3-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED},
+ })["Distribution"]
+ status, _headers, body = _get(dist["DomainName"], "/hello.txt")
+ assert status == 403
+ assert b"<Code>AccessDenied</Code>" in body
+
+
+def test_s3_origin_serves_bucket_object(cf_s3_stack):
+ status, headers, body = _get(cf_s3_stack["dist_host"], "/hello.txt")
+ assert status == 200
+ assert body == cf_s3_stack["body"]
+ assert headers.get("Content-Type") == "text/plain"
+
+
+def test_s3_origin_range_get_returns_partial_content(cf_s3_stack):
+ # Range is forwarded regardless of cache/origin-request-policy header
+ # configuration (CloudFront Developer Guide, "HTTP request headers and
+ # CloudFront behavior"), even under CachingDisabled with no ORP here.
+ status, headers, body = _get(
+ cf_s3_stack["dist_host"], "/hello.txt", extra_headers={"Range": "bytes=0-4"},
+ )
+ assert status == 206
+ assert body == cf_s3_stack["body"][:5]
+ assert headers.get("Content-Range", "").startswith("bytes 0-4/")
+
+
+def test_s3_origin_missing_key_returns_access_denied(cf_s3_stack):
+ # The OAC bucket policy CloudFront's console generates grants only
+ # s3:GetObject, never s3:ListBucket — without ListBucket, S3 answers a
+ # missing key with 403 AccessDenied rather than 404 NoSuchKey.
+ status, _headers, body = _get(cf_s3_stack["dist_host"], "/does-not-exist.txt")
+ assert status == 403
+ assert b"<Code>AccessDenied</Code>" in body
+
+
+def _s3_oac_origin(origin_id: str, bucket: str) -> dict:
+ return {"Id": origin_id, "DomainName": f"{bucket}.s3.amazonaws.com",
+ "S3OriginConfig": {"OriginAccessIdentity": ""}, "OriginAccessControlId": "test-oac-id"}
+
+
+def _custom_error_dist(cloudfront, suffix: str, origins: list, error_responses: list, behaviors: list | None = None):
+ return cloudfront.create_distribution(DistributionConfig={
+ "CallerReference": f"cf-dp-errors-{suffix}", "Comment": "", "Enabled": True,
+ "Origins": {"Quantity": len(origins), "Items": origins},
+ "DefaultCacheBehavior": {"TargetOriginId": origins[0]["Id"], "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED},
+ "CacheBehaviors": {"Quantity": len(behaviors or []), "Items": behaviors or []},
+ "CustomErrorResponses": {"Quantity": len(error_responses), "Items": error_responses},
+ })["Distribution"]
+
+
+def test_custom_error_response_serves_the_page_with_the_configured_status(cloudfront, s3):
+ """A single-page app: S3 answers 403 for a client route, CloudFront serves /index.html with 200."""
+ suffix = _uuid_mod.uuid4().hex[:10]
+ bucket = f"cf-dp-spa-{suffix}"
+ s3.create_bucket(Bucket=bucket)
+ s3.put_object(Bucket=bucket, Key="index.html", Body=b"<html>app</html>", ContentType="text/html")
+ dist = _custom_error_dist(cloudfront, suffix, [_s3_oac_origin("site", bucket)], [
+ {"ErrorCode": 403, "ResponsePagePath": "/index.html", "ResponseCode": "200", "ErrorCachingMinTTL": 10},
+ ])
+ s3.put_bucket_policy(Bucket=bucket, Policy=json.dumps({"Statement": [{
+ "Effect": "Allow", "Principal": {"Service": "cloudfront.amazonaws.com"},
+ "Action": "s3:GetObject", "Resource": f"arn:aws:s3:::{bucket}/*",
+ "Condition": {"StringEquals": {"AWS:SourceArn": dist["ARN"]}},
+ }]}))
+
+ status, headers, body = _get(dist["DomainName"], "/auth/callback")
+ assert status == 200
+ assert body == b"<html>app</html>"
+ assert headers.get("Content-Type") == "text/html"
+
+
+def test_unavailable_custom_error_page_answers_the_status_its_origin_returned(cloudfront, s3, cf_backend):
+ """The Developer Guide's example ("Generate custom error responses"): the custom origin answers 500 and the
+ page is missing from its S3 bucket, so the viewer gets S3's status, neither 500 nor the configured 200."""
+ suffix = _uuid_mod.uuid4().hex[:10]
+ pages = f"cf-dp-pages-{suffix}"
+ s3.create_bucket(Bucket=pages)
+ dist = _custom_error_dist(
+ cloudfront, suffix,
+ [{"Id": "api", "DomainName": cf_backend["origin_domain"], "CustomOriginConfig": {
+ "HTTPPort": int(GATEWAY_PORT), "HTTPSPort": 443, "OriginProtocolPolicy": "http-only"}},
+ _s3_oac_origin("pages", pages)],
+ [{"ErrorCode": 500, "ResponsePagePath": "/errors/500.html", "ResponseCode": "200"}],
+ [{"PathPattern": "/errors/*", "TargetOriginId": "pages", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED}],
+ )
+
+ status, _headers, body = _get(dist["DomainName"], "/status/500")
+ assert status == 403
+ assert b"<Code>AccessDenied</Code>" in body
+
+
+# ---------------------------------------------------------------------------
+# Percent-encoding fidelity (CloudFront Developer Guide, "Restrictions on all
+# edge functions" > "URI, query string, and headers encoding").
+# ---------------------------------------------------------------------------
+
+
+class _PathEchoHandler(http.server.BaseHTTPRequestHandler):
+ def do_GET(self):
+ body = json.dumps({
+ "path": self.path,
+ "headers": {k.lower(): v for k, v in self.headers.items()},
+ }).encode()
+ self.send_response(200)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(body)))
+ if self.path.endswith("/hopbyhop"):
+ self.send_header("Connection", "keep-alive")
+ self.send_header("X-Should-Survive", "yes")
+ if self.path.endswith("/respdup"):
+ self.send_header("X-Resp-Dup", "first")
+ self.send_header("X-Resp-Dup", "second")
+ self.end_headers()
+ self.wfile.write(body)
+
+ def do_POST(self):
+ length = int(self.headers.get("Content-Length", 0) or 0)
+ posted_body = self.rfile.read(length) if length else b""
+ body = json.dumps({
+ "path": self.path,
+ "body": posted_body.decode("utf-8", errors="replace"),
+ "content_length_header": self.headers.get("Content-Length"),
+ }).encode()
+ self.send_response(200)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, format, *args):
+ pass
+
+
[email protected](scope="module")
+def path_echo_origin_port():
+ server = http.server.HTTPServer(("127.0.0.1", 0), _PathEchoHandler)
+ port = server.server_address[1]
+ thread = threading.Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ try:
+ yield port
+ finally:
+ server.shutdown()
+ thread.join(timeout=5)
+ server.server_close()
+
+
[email protected](scope="module")
+def cf_raw_origin_stack(cloudfront, path_echo_origin_port):
+ suffix = _uuid_mod.uuid4().hex[:10]
+ passthrough_fn_arn = None
+ if cloudfront_js.available():
+ passthrough_fn_arn = _create_function(
+ cloudfront, f"cf-dp-passthrough-{suffix}",
+ b"function handler(event) { return event.request; }\n",
+ )
+ dist_config = {
+ "CallerReference": f"cf-dp-raw-{suffix}",
+ "Comment": "data plane raw-path-fidelity test distribution",
+ "Enabled": True,
+ "Origins": {
+ "Quantity": 1,
+ "Items": [{
+ "Id": "raw-origin",
+ "DomainName": "127.0.0.1",
+ "CustomOriginConfig": {
+ "HTTPPort": path_echo_origin_port, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
+ },
+ }],
+ },
+ "DefaultCacheBehavior": {
+ "TargetOriginId": "raw-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED,
+ },
+ "CacheBehaviors": {
+ "Quantity": 1,
+ "Items": [{
+ "PathPattern": "/av/*",
+ "TargetOriginId": "raw-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": _ALL_VIEWER,
+ }],
+ },
+ }
+ if passthrough_fn_arn:
+ dist_config["CacheBehaviors"]["Quantity"] = 2
+ dist_config["CacheBehaviors"]["Items"].append({
+ "PathPattern": "/fn/*",
+ "TargetOriginId": "raw-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": _ALL_VIEWER,
+ "FunctionAssociations": {
+ "Quantity": 1, "Items": [{"FunctionARN": passthrough_fn_arn, "EventType": "viewer-request"}],
+ },
+ })
+ dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
+ return dist["DomainName"]
+
+
+def _raw_get(dist_host: str, raw_path: str):
+ conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
+ try:
+ conn.putrequest("GET", raw_path, skip_host=True, skip_accept_encoding=True)
+ conn.putheader("Host", f"{dist_host}:{GATEWAY_PORT}")
+ conn.endheaders()
+ resp = conn.getresponse()
+ return resp.status, resp.read()
+ finally:
+ conn.close()
+
+
+def test_percent_encoded_path_forwarded_byte_exact(cf_raw_origin_stack):
+ for raw_path in (
+ "/echo/%21%24%26%27%28%29%2A%2B%2C%3B%3D%40",
+ "/ready_pin_guests/123%2Ejson",
+ ):
+ status, body = _raw_get(cf_raw_origin_stack, raw_path)
+ assert status == 200
+ assert json.loads(body)["path"] == raw_path
+
+
+def test_query_string_forwarded_byte_exact_when_unmodified_and_fully_allowed(cf_raw_origin_stack):
+ # "/av/*" carries AllViewer (forwards every query-string parameter) and
+ # no function: nothing should re-encode the viewer's own percent-encoding.
+ raw_path = "/av/hello?a=%68%65%6c%6c%6f&b=1"
+ status, body = _raw_get(cf_raw_origin_stack, raw_path)
+ assert status == 200
+ assert json.loads(body)["path"] == raw_path
+
+
+def test_function_returning_the_request_keeps_a_bare_query_key(cf_raw_origin_stack):
+ """AWS: observed 2026-10-10, a distribution whose viewer-request function returns event.request:
+ GET /locations?type[] got the response a Rails origin gives the bare key (?type[]= gets another)."""
+ _skip_without_node()
+ raw_path = "/fn/hello?flag&a=1"
+ status, body = _raw_get(cf_raw_origin_stack, raw_path)
+ assert status == 200
+ assert json.loads(body)["path"] == raw_path
+
+
+def test_post_body_round_trips_with_content_length(cf_raw_origin_stack):
+ conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
+ try:
+ payload = b'{"hello": "world"}'
+ conn.putrequest("POST", "/av/echo", skip_host=True, skip_accept_encoding=True)
+ conn.putheader("Host", f"{cf_raw_origin_stack}:{GATEWAY_PORT}")
+ conn.putheader("Content-Type", "application/json")
+ conn.putheader("Content-Length", str(len(payload)))
+ conn.endheaders(payload)
+ resp = conn.getresponse()
+ body = resp.read()
+ finally:
+ conn.close()
+ echoed = json.loads(body)
+ assert echoed["body"] == payload.decode()
+ assert echoed["content_length_header"] == str(len(payload))
+
+
+def test_hop_by_hop_response_headers_stripped(cf_raw_origin_stack):
+ status, headers, _body = _get(cf_raw_origin_stack, "/av/hopbyhop")
+ assert status == 200
+ assert "Connection" not in headers
+ assert headers.get("X-Should-Survive") == "yes"
+
+
+# ---------------------------------------------------------------------------
+# Raw request-header fidelity toward the origin — a plain http.server or
+# http.client doesn't distinguish "one folded line" from "two lines with the
+# same name", so this reads the exact bytes off the socket.
+# ---------------------------------------------------------------------------
+
+
+class _RawHeaderCaptureServer:
+ """Accepts one connection at a time, records the raw request-header
+ block exactly as received, and answers a trivial 200."""
+
+ def __init__(self):
+ self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+ self.sock.bind(("127.0.0.1", 0))
+ self.sock.listen(5)
+ self.port = self.sock.getsockname()[1]
+ self.last_header_lines: list = []
+ self._stop = False
+ self.thread = threading.Thread(target=self._serve, daemon=True)
+ self.thread.start()
+
+ def _serve(self):
+ self.sock.settimeout(0.5)
+ while not self._stop:
+ try:
+ conn, _ = self.sock.accept()
+ except socket.timeout:
+ continue
+ except OSError:
+ break
+ with conn:
+ conn.settimeout(5)
+ data = b""
+ while b"\r\n\r\n" not in data:
+ chunk = conn.recv(4096)
+ if not chunk:
+ break
+ data += chunk
+ head, _, _rest = data.partition(b"\r\n\r\n")
+ lines = head.decode("latin-1").split("\r\n")
+ self.last_header_lines = lines[1:] # drop the request line
+ body = b"ok"
+ conn.sendall(
+ b"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: "
+ + str(len(body)).encode() + b"\r\n\r\n" + body
+ )
+
+ def stop(self):
+ self._stop = True
+ self.sock.close()
+ self.thread.join(timeout=5)
+
+
[email protected](scope="module")
+def raw_header_server():
+ server = _RawHeaderCaptureServer()
+ try:
+ yield server
+ finally:
+ server.stop()
+
+
[email protected](scope="module")
+def cf_raw_header_stack(cloudfront, raw_header_server):
+ suffix = _uuid_mod.uuid4().hex[:10]
+ dup_edit_fn_arn = None
+ if cloudfront_js.available():
+ # Viewer headers are already comma-joined before this module ever
+ # sees them (app.py, RFC 9110 §5.2), so a function is the only way to
+ # produce a genuine multiValue request header here.
+ dup_edit_fn_arn = _create_function(
+ cloudfront, f"cf-dp-dup-header-edit-{suffix}",
+ b"function handler(event) {\n"
+ b" var request = event.request;\n"
+ b" request.headers['x-dup'] = { value: 'a', multiValue: [{ value: 'a' }, { value: 'b' }] };\n"
+ b" return request;\n"
+ b"}\n",
+ )
+ dist_config = {
+ "CallerReference": f"cf-dp-rawheaders-{suffix}",
+ "Comment": "data plane raw request-header test distribution",
+ "Enabled": True,
+ "Origins": {
+ "Quantity": 1,
+ "Items": [{
+ "Id": "raw-header-origin",
+ "DomainName": "127.0.0.1",
+ "CustomOriginConfig": {
+ "HTTPPort": raw_header_server.port, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
+ },
+ # Lowercase, viewer sends Title-Case — proves the overwrite is
+ # case-insensitive, not merely a literal-name match.
+ "CustomHeaders": {"Quantity": 1, "Items": [{"HeaderName": "x-custom-test", "HeaderValue": "from-origin"}]},
+ }],
+ },
+ "DefaultCacheBehavior": {
+ "TargetOriginId": "raw-header-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": _ALL_VIEWER,
+ },
+ }
+ if dup_edit_fn_arn:
+ dist_config["CacheBehaviors"] = {
+ "Quantity": 1,
+ "Items": [{
+ "PathPattern": "/dupheader/*",
+ "TargetOriginId": "raw-header-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": _ALL_VIEWER,
+ "FunctionAssociations": {
+ "Quantity": 1,
+ "Items": [{"FunctionARN": dup_edit_fn_arn, "EventType": "viewer-request"}],
+ },
+ }],
+ }
+ dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
+ return dist["DomainName"]
+
+
+def test_origin_request_headers_no_duplicates_case_insensitive_overwrite_and_xff(
+ cf_raw_header_stack, raw_header_server,
+):
+ conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
+ try:
+ conn.putrequest("GET", "/anything", skip_host=True, skip_accept_encoding=True)
+ conn.putheader("Host", f"{cf_raw_header_stack}:{GATEWAY_PORT}")
+ conn.putheader("X-Custom-Test", "viewer-value")
+ conn.putheader("X-Forwarded-For", "203.0.113.5")
+ conn.endheaders()
+ resp = conn.getresponse()
+ resp.read()
+ finally:
+ conn.close()
+
+ lines = raw_header_server.last_header_lines
+ names_lower = [line.split(":", 1)[0].lower() for line in lines if line]
+ # AllViewer forwards Host too: exactly one line, not a lowercase
+ # viewer-forwarded one plus a Title-Cased one this module also adds.
+ assert names_lower.count("host") == 1
+
+ custom_test_lines = [line for line in lines if line.lower().startswith("x-custom-test:")]
+ assert len(custom_test_lines) == 1
+ assert custom_test_lines[0].split(":", 1)[1].strip() == "from-origin"
+
+ xff_lines = [line for line in lines if line.lower().startswith("x-forwarded-for:")]
+ assert len(xff_lines) == 1
+ assert xff_lines[0].split(":", 1)[1].strip() == "203.0.113.5, 127.0.0.1"
+
+
+def test_function_set_multivalue_header_forwarded_as_one_comma_joined_line(
+ cf_raw_header_stack, raw_header_server,
+):
+ # RFC 9110 §5.3: a repeated field value is one comma-joined line when
+ # forwarded to the origin, not several lines of the same name.
+ _skip_without_node()
+ conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
+ try:
+ conn.putrequest("GET", "/dupheader/hello", skip_host=True, skip_accept_encoding=True)
+ conn.putheader("Host", f"{cf_raw_header_stack}:{GATEWAY_PORT}")
+ conn.endheaders()
+ resp = conn.getresponse()
+ resp.read()
+ finally:
+ conn.close()
+
+ dup_lines = [line for line in raw_header_server.last_header_lines if line.lower().startswith("x-dup:")]
+ assert len(dup_lines) == 1
+ assert dup_lines[0].split(":", 1)[1].strip() == "a, b"
+
+
[email protected](scope="module")
+def cf_response_dup_stack(cloudfront, path_echo_origin_port):
+ """A distribution whose origin (the raw path-echo server) sends a
+ genuinely repeated response header, for the response-side multiValue
+ edit-only merge rule below."""
+ suffix = _uuid_mod.uuid4().hex[:10]
+ fn_arn = None
+ if cloudfront_js.available():
+ fn_arn = _create_function(
+ cloudfront, f"cf-dp-response-dup-edit-{suffix}",
+ b"function handler(event) {\n"
+ b" var response = event.response;\n"
+ b" if (response.headers['x-resp-dup']) {\n"
+ b" response.headers['x-resp-dup'].value = 'edited';\n"
+ b" }\n"
+ b" return response;\n"
+ b"}\n",
+ )
+ behavior = {
+ "TargetOriginId": "respdup-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED,
+ }
+ if fn_arn:
+ behavior["FunctionAssociations"] = {
+ "Quantity": 1, "Items": [{"FunctionARN": fn_arn, "EventType": "viewer-response"}],
+ }
+ dist_config = {
+ "CallerReference": f"cf-dp-respdup-{suffix}",
+ "Comment": "data plane response multiValue edit-only test",
+ "Enabled": True,
+ "Origins": {
+ "Quantity": 1,
+ "Items": [{
+ "Id": "respdup-origin",
+ "DomainName": "127.0.0.1",
+ "CustomOriginConfig": {
+ "HTTPPort": path_echo_origin_port, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
+ },
+ }],
+ },
+ "DefaultCacheBehavior": behavior,
+ }
+ dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
+ return dist["DomainName"]
+
+
+def test_viewer_response_function_edits_only_value_of_repeated_header_keeps_remaining_values(
+ cf_response_dup_stack,
+):
+ # functions-event-structure.html: a header whose multiValue is unchanged
+ # from the event is read only for its primary value; the rest of its
+ # original values survive.
+ _skip_without_node()
+ conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
+ try:
+ conn.putrequest("GET", "/respdup", skip_host=True, skip_accept_encoding=True)
+ conn.putheader("Host", f"{cf_response_dup_stack}:{GATEWAY_PORT}")
+ conn.endheaders()
+ resp = conn.getresponse()
+ resp.read()
+ dup_values = resp.msg.get_all("X-Resp-Dup") or []
+ finally:
+ conn.close()
+ assert dup_values == ["edited", "second"]
+
+
+# ---------------------------------------------------------------------------
+# TLS to a custom https origin.
+# ---------------------------------------------------------------------------
+
+
+def untrusted_https_origin_port(tmp_path):
+ pytest.importorskip("cryptography")
+ from ministack.core.x509_utils import generate_ca, sign_leaf_certificate
+
+ ca_pem, ca_key_pem = generate_ca(common_name="cf-dp-untrusted-ca")
+ leaf_pem, leaf_key_pem, _public = sign_leaf_certificate(
+ ca_pem, ca_key_pem, common_name="127.0.0.1", san_ips=["127.0.0.1"],
+ )
+ cert_path = tmp_path / "leaf.pem"
+ cert_path.write_text(leaf_pem + leaf_key_pem)
+
+ server = http.server.HTTPServer(("127.0.0.1", 0), _PathEchoHandler)
+ ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+ ctx.load_cert_chain(str(cert_path))
+ server.socket = ctx.wrap_socket(server.socket, server_side=True)
+ port = server.server_address[1]
+ thread = threading.Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ try:
+ yield port
+ finally:
+ server.shutdown()
+ thread.join(timeout=5)
+ server.server_close()
+
+
+def test_untrusted_origin_certificate_answers_502(cloudfront, untrusted_https_origin_port):
+ # http-502-bad-gateway.html's self-signed/untrusted-chain case: the
+ # origin's own CA isn't in the system trust store (and USE_SSL is off in
+ # this test harness, so the gateway cert isn't trusted either).
+ suffix = _uuid_mod.uuid4().hex[:10]
+ dist_config = {
+ "CallerReference": f"cf-dp-untrusted-{suffix}",
+ "Comment": "data plane untrusted-origin-cert test",
+ "Enabled": True,
+ "Origins": {
+ "Quantity": 1,
+ "Items": [{
+ "Id": "untrusted-origin",
+ "DomainName": "127.0.0.1",
+ "CustomOriginConfig": {
+ "HTTPPort": 80, "HTTPSPort": untrusted_https_origin_port,
+ "OriginProtocolPolicy": "https-only",
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
+ },
+ }],
+ },
+ "DefaultCacheBehavior": {
+ "TargetOriginId": "untrusted-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED,
+ },
+ }
+ dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
+ status, headers, _body = _get(dist["DomainName"], "/anything")
+ assert status == 502
+ assert headers.get("X-Cache") == "Error from cloudfront"
+
+
+# ---------------------------------------------------------------------------
+# Registered ALB origin — an ALB's DNSName is AWS-shaped, so it is served
+# in-process through the ALB data plane instead of being dialed.
+# ---------------------------------------------------------------------------
+
+
+def _alb_origin_dist(cloudfront, lb_dns: str, port: int, custom_headers: list, read_timeout: int = 30) -> str:
+ suffix = _uuid_mod.uuid4().hex[:10]
+ origin = {
+ "Id": "alb-origin",
+ "DomainName": lb_dns,
+ "CustomOriginConfig": {
+ "HTTPPort": port, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": read_timeout, "OriginKeepaliveTimeout": 5,
+ },
+ }
+ if custom_headers:
+ origin["CustomHeaders"] = {
+ "Quantity": len(custom_headers),
+ "Items": [{"HeaderName": name, "HeaderValue": value} for name, value in custom_headers],
+ }
+ dist_config = {
+ "CallerReference": f"cf-dp-alb-{suffix}",
+ "Comment": "data plane registered-ALB-origin test",
+ "Enabled": True,
+ "Origins": {"Quantity": 1, "Items": [origin]},
+ "DefaultCacheBehavior": {
+ "TargetOriginId": "alb-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED,
+ },
+ }
+ return cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]["DomainName"]
+
+
+def _alb_fixed(status: str, body: str) -> dict:
+ return {"Type": "fixed-response",
+ "FixedResponseConfig": {"StatusCode": status, "ContentType": "text/plain", "MessageBody": body}}
+
+
+def alb_origin(elbv2):
+ """A load balancer with one HTTP:80 listener whose default action answers 403."""
+ name = f"cf-alb-{_uuid_mod.uuid4().hex[:8]}"
+ lb = elbv2.create_load_balancer(Name=name, Subnets=["subnet-1"])["LoadBalancers"][0]
+ listener_arn = elbv2.create_listener(
+ LoadBalancerArn=lb["LoadBalancerArn"], Protocol="HTTP", Port=80,
+ DefaultActions=[_alb_fixed("403", "denied")],
+ )["Listeners"][0]["ListenerArn"]
+ try:
+ yield {"dns": lb["DNSName"], "listener_arn": listener_arn}
+ finally:
+ elbv2.delete_listener(ListenerArn=listener_arn)
+ elbv2.delete_load_balancer(LoadBalancerArn=lb["LoadBalancerArn"])
+
+
+def test_registered_alb_origin_is_served_through_its_listener_rules(cloudfront, elbv2, alb_origin):
+ # Restricting an ALB to CloudFront with an origin custom header
+ # (restrict-access-to-load-balancer.html).
+ elbv2.create_rule(
+ ListenerArn=alb_origin["listener_arn"], Priority=1,
+ Conditions=[{"Field": "http-header",
+ "HttpHeaderConfig": {"HttpHeaderName": "X-Origin-Verify", "Values": ["origin-token"]}}],
+ Actions=[_alb_fixed("200", "registered-alb-origin")],
+ )
+ with_header = _alb_origin_dist(cloudfront, alb_origin["dns"], 80, [("X-Origin-Verify", "origin-token")])
+ without_header = _alb_origin_dist(cloudfront, alb_origin["dns"], 80, [])
+
+ status, _headers, body = _get(with_header, "/anything")
+ assert (status, body) == (200, b"registered-alb-origin")
+ status, _headers, body = _get(without_header, "/anything")
+ assert (status, body) == (403, b"denied")
+
+
+def test_registered_alb_origin_without_a_listener_on_the_origin_port_answers_502(cloudfront, alb_origin):
+ dist_host = _alb_origin_dist(cloudfront, alb_origin["dns"], 8080, [])
+ status, headers, _body = _get(dist_host, "/anything")
+ assert status == 502
+ assert headers.get("X-Cache") == "Error from cloudfront"
+
+
+def test_unregistered_alb_hostname_is_still_refused(cloudfront):
+ dist_host = _alb_origin_dist(cloudfront, "cf-alb-missing-1234567890.us-east-1.elb.amazonaws.com", 80, [])
+ status, headers, _body = _get(dist_host, "/anything")
+ assert status == 502
+ assert headers.get("X-Cache") == "Error from cloudfront"
+
+
+def test_registered_alb_origin_relays_an_ip_target_response(cloudfront, elbv2, alb_origin, raw_header_server):
+ tg_arn = elbv2.create_target_group(
+ Name=f"cf-alb-{_uuid_mod.uuid4().hex[:8]}", Protocol="HTTP", Port=raw_header_server.port,
+ VpcId="vpc-00000001", TargetType="ip",
+ )["TargetGroups"][0]["TargetGroupArn"]
+ try:
+ elbv2.register_targets(TargetGroupArn=tg_arn, Targets=[{"Id": "127.0.0.1", "Port": raw_header_server.port}])
+ elbv2.modify_listener(ListenerArn=alb_origin["listener_arn"],
+ DefaultActions=[{"Type": "forward", "TargetGroupArn": tg_arn}])
+ dist_host = _alb_origin_dist(cloudfront, alb_origin["dns"], 80, [("X-Origin-Verify", "origin-token")])
+
+ status, _headers, body = _get(dist_host, "/anything", extra_headers={"X-Forwarded-For": "203.0.113.5"})
+ assert (status, body) == (200, b"ok")
+
+ lines = raw_header_server.last_header_lines
+ names_lower = [line.split(":", 1)[0].lower() for line in lines if line]
+ assert names_lower.count("x-origin-verify") == 1
+ xff_lines = [line for line in lines if line.lower().startswith("x-forwarded-for:")]
+ assert len(xff_lines) == 1
+ # CloudFront appends the viewer, then the load balancer appends CloudFront's address.
+ assert xff_lines[0].split(":", 1)[1].strip() == "203.0.113.5, 127.0.0.1, 127.0.0.1"
+ finally:
+ elbv2.delete_target_group(TargetGroupArn=tg_arn)
+
+
+def test_registered_alb_origin_timeout_answers_504_and_closes_the_target_connection(cloudfront, elbv2, alb_origin):
+ # A target that reads the request and never answers.
+ listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ listener.bind(("127.0.0.1", 0))
+ listener.listen(1)
+ target_port = listener.getsockname()[1]
+ closed = threading.Event()
+
+ def serve():
+ conn, _ = listener.accept()
+ with conn:
+ conn.settimeout(10)
+ data = b""
+ while b"\r\n\r\n" not in data:
+ data += conn.recv(4096)
+ if conn.recv(4096) == b"":
+ closed.set()
+
+ thread = threading.Thread(target=serve, daemon=True)
+ thread.start()
+ tg_arn = elbv2.create_target_group(
+ Name=f"cf-alb-{_uuid_mod.uuid4().hex[:8]}", Protocol="HTTP", Port=target_port,
+ VpcId="vpc-00000001", TargetType="ip",
+ )["TargetGroups"][0]["TargetGroupArn"]
+ try:
+ elbv2.register_targets(TargetGroupArn=tg_arn, Targets=[{"Id": "127.0.0.1", "Port": target_port}])
+ elbv2.modify_listener(ListenerArn=alb_origin["listener_arn"],
+ DefaultActions=[{"Type": "forward", "TargetGroupArn": tg_arn}])
+ dist_host = _alb_origin_dist(cloudfront, alb_origin["dns"], 80, [], read_timeout=1)
+
+ status, headers, _body = _get(dist_host, "/anything")
+ assert status == 504
+ assert headers.get("X-Cache") == "Error from cloudfront"
+ assert closed.wait(5)
+ finally:
+ elbv2.delete_target_group(TargetGroupArn=tg_arn)
+ listener.close()
+ thread.join(timeout=10)
+
+
+def test_streamed_alb_body_without_its_final_frame_is_an_origin_failure():
+ async def truncated(send, _receive):
+ await send({"type": "http.response.body", "body": b"partial", "more_body": True})
+
+ with pytest.raises(cloudfront_svc.OriginUnreachable):
+ asyncio.run(cloudfront_svc._collect_streaming_body(StreamingResponse(truncated), 5))
+
+
+def test_streamed_alb_body_idle_past_the_read_timeout_is_an_origin_timeout():
+ async def stalled(send, _receive):
+ await send({"type": "http.response.body", "body": b"first", "more_body": True})
+ await asyncio.sleep(5)
+ await send({"type": "http.response.body", "body": b"", "more_body": False})
+
+ with pytest.raises(cloudfront_svc.OriginTimeout):
+ asyncio.run(cloudfront_svc._collect_streaming_body(StreamingResponse(stalled), 0.2))
+
+
+def _function_url_dist(cloudfront, url_host: str, protocol_policy: str = "https-only",
+ origin_request_policy: str = _ALL_VIEWER_EXCEPT_HOST) -> str:
+ origin = {
+ "Id": "url-origin", "DomainName": url_host,
+ "CustomHeaders": {"Quantity": 1, "Items": [{"HeaderName": "X-Origin-Verify", "HeaderValue": "origin-token"}]},
+ "CustomOriginConfig": {
+ "HTTPPort": 80, "HTTPSPort": 443, "OriginProtocolPolicy": protocol_policy,
+ "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
+ "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
+ },
+ }
+ dist_config = {
+ "CallerReference": f"cf-dp-url-{_uuid_mod.uuid4().hex[:10]}",
+ "Comment": "data plane Function URL origin test",
+ "Enabled": True,
+ "Origins": {"Quantity": 1, "Items": [origin]},
+ "DefaultCacheBehavior": {
+ "TargetOriginId": "url-origin", "ViewerProtocolPolicy": "allow-all",
+ "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": origin_request_policy,
+ },
+ }
+ return cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]["DomainName"]
+
+
+def function_url_origin(lam):
+ name = f"cf-dp-url-{_uuid_mod.uuid4().hex[:10]}"
+ lam.create_function(FunctionName=name, Runtime="python3.12", Role="arn:aws:iam::000000000000:role/test-role",
+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_ORIGIN_LAMBDA_CODE)})
+ urls = {}
+ for alias, auth in (("open", "NONE"), ("iam", "AWS_IAM")):
+ version = lam.publish_version(FunctionName=name)["Version"]
+ lam.create_alias(FunctionName=name, Name=alias, FunctionVersion=version)
+ url = lam.create_function_url_config(FunctionName=name, Qualifier=alias, AuthType=auth)["FunctionUrl"]
+ urls[alias] = url.split("://", 1)[1].rstrip("/")
+ yield urls
+ lam.delete_function(FunctionName=name)
+
+
+def test_function_url_origin_is_served_with_the_origin_custom_headers(cloudfront, function_url_origin):
+ dist_host = _function_url_dist(cloudfront, function_url_origin["open"])
+ status, _headers, body = _get(dist_host, "/hello?a=1")
+ assert status == 200, body
+ payload = json.loads(body)
+ assert payload["path"] == "/hello"
+ assert payload["query"] == {"a": "1"}
+ assert payload["headers"]["x-origin-verify"] == "origin-token"
+ headers = payload["headers"]
+ assert headers["host"] == function_url_origin["open"]
+ assert "(CloudFront)" in headers["via"]
+ # Captured on AWS: the Function URL keeps the first X-Forwarded-For address and sets the rest.
+ assert (headers["x-forwarded-for"], headers["x-forwarded-port"], headers["x-forwarded-proto"]) == (
+ "127.0.0.1", "443", "https")
+ assert (headers["x-amzn-tls-version"], headers["x-amzn-tls-cipher-suite"]) == ("TLSv1.3", "TLS_AES_128_GCM_SHA256")
+ assert re.fullmatch(r"Root=1-[0-9a-f]{8}-[0-9a-f]{24}", headers["x-amzn-trace-id"])
+ status, _headers, body = _get(dist_host, "/hello", extra_headers={"X-Forwarded-For": "203.0.113.9"})
+ assert json.loads(body)["headers"]["x-forwarded-for"] == "203.0.113.9"
+
+
+def test_function_url_origin_refuses_the_viewer_host(cloudfront, function_url_origin):
+ dist_host = _function_url_dist(cloudfront, function_url_origin["open"], origin_request_policy=_ALL_VIEWER)
+ status, headers, body = _get(dist_host, "/hello")
+ assert (status, body) == (403, b'{"Message":null}')
+ assert {k.lower(): v for k, v in headers.items()}.get("x-amzn-errortype") == "AccessDeniedException"
+
+
+def test_function_url_origin_with_aws_iam_auth_answers_forbidden_without_signing(cloudfront, function_url_origin):
+ status, headers, body = _get(_function_url_dist(cloudfront, function_url_origin["iam"]), "/hello")
+ assert (status, body) == (403, b'{"Message":"Forbidden"}')
+ assert {k.lower(): v for k, v in headers.items()}.get("x-amzn-errortype") == "AccessDeniedException"
+
+
+def test_function_url_origin_over_http_is_unreachable(cloudfront, function_url_origin):
+ status, _headers, _body = _get(_function_url_dist(cloudfront, function_url_origin["open"], "http-only"), "/hello")
+ assert status == 502
+
+
+def test_unregistered_function_url_hostname_is_still_refused(cloudfront):
+ host = f"{_uuid_mod.uuid4()}.lambda-url.us-east-1.on.aws"
+ assert _get(_function_url_dist(cloudfront, host), "/hello")[0] == 502
diff --git a/tests/test_cloudfront_dataplane.py b/tests/test_cloudfront_dataplane.py
deleted file mode 100644
index 105a9204..00000000
--- a/tests/test_cloudfront_dataplane.py
+++ /dev/null
@@ -1,1459 +0,0 @@
-# Copyright (c) 2026 MiniStack Contributors. SPDX-License-Identifier: MIT
-# Copies or substantial portions, including AI-assisted ports or rewrites, must retain this notice (see LICENSE).
-"""
-CloudFront data plane tests — a distribution actually serving traffic.
-
-A CloudFront distribution's DomainName is always AWS-shaped
-(``<label>.cloudfront.net``); routing matches that host, or the resolvable
-local form built on MINISTACK_HOST (``<label>.cloudfront.localhost``), which
-this file uses to reach a distribution over the real wire.
-
-One distribution (``cf_stack``) is built once per module and exercised by
-most tests; it fronts a custom API Gateway HTTP API backed by a Lambda that
-echoes the inbound path, headers, query string, and cookies, with several
-cache behaviors covering the request-forwarding, protocol-policy, allowed-
-methods, and origin-path/custom-header surfaces. A handful of tests need
-their own distribution (S3 origin, a raw path-echo origin, a dead origin).
-"""
-
-import asyncio
-import http.client
-import http.server
-import io
-import json
-import socket
-import ssl
-import threading
-import uuid as _uuid_mod
-import zipfile
-from unittest import mock
-
-import pytest
-from conftest import GATEWAY_PORT
-
-from ministack.core import cloudfront_js
-from ministack.services import cloudfront_dataplane
-
-# Managed policy ids (CloudFront Developer Guide, "Managed origin request
-# policies" / "Managed cache policies" / "Managed response headers
-# policies").
-_ALL_VIEWER = "216adef6-5c7f-47e4-b989-5492eafa07d3"
-_ALL_VIEWER_EXCEPT_HOST = "b689b0a8-53d0-40ab-baf2-68738e2966ac"
-_CACHING_DISABLED = "4135ea2d-6df8-44a3-9df3-4b5a84be39ad"
-_MANAGED_CORS_WITH_PREFLIGHT = "5cc3b908-e619-4b99-88e5-2cf7f45965bd"
-
-_ORIGIN_LAMBDA_CODE = (
- b"import json\n"
- b"def handler(event, context):\n"
- b" path = event.get('rawPath', '')\n"
- b" headers = event.get('headers', {})\n"
- b" # MiniStack's apigatewayv2 (payload format 2.0) does not split an\n"
- b" # incoming Cookie header into the top-level `cookies` array real AWS\n"
- b" # uses; it stays in `headers['cookie']` like any other header, which\n"
- b" # is also what cloudfront_dataplane.py actually forwards.\n"
- b" cookie_header = headers.get('cookie', '')\n"
- b" cookies = [c.strip() for c in cookie_header.split(';') if c.strip()]\n"
- b" resp = {\n"
- b" 'statusCode': 200,\n"
- b" 'headers': {'Content-Type': 'application/json', 'X-Origin-Header': 'origin-value'},\n"
- b" 'body': json.dumps({\n"
- b" 'path': path,\n"
- b" 'headers': headers,\n"
- b" 'query': event.get('queryStringParameters') or {},\n"
- b" 'cookies': cookies,\n"
- b" }),\n"
- b" }\n"
- b" if path == '/cookies':\n"
- b" resp['cookies'] = ['resp1=a; Path=/', 'resp2=b; Path=/']\n"
- b" if path == '/status/500':\n"
- b" resp['statusCode'] = 500\n"
- b" return resp\n"
-)
-
-
-def _zip_lambda(code: bytes) -> bytes:
- buf = io.BytesIO()
- with zipfile.ZipFile(buf, "w") as zf:
- zf.writestr("index.py", code)
- return buf.getvalue()
-
-
-def _none_cache_policy_config(name: str) -> dict:
- return {
- "Name": name,
- "MinTTL": 0,
- "ParametersInCacheKeyAndForwardedToOrigin": {
- "HeadersConfig": {"HeaderBehavior": "none"},
- "CookiesConfig": {"CookieBehavior": "none"},
- "QueryStringsConfig": {"QueryStringBehavior": "none"},
- "EnableAcceptEncodingGzip": False,
- },
- }
-
-
-def _response_headers_policy(cloudfront, name: str, custom_headers: list) -> str:
- resp = cloudfront.create_response_headers_policy(
- ResponseHeadersPolicyConfig={
- "Name": name,
- "CustomHeadersConfig": {
- "Quantity": len(custom_headers),
- "Items": [{"Header": h, "Value": v, "Override": o} for (h, v, o) in custom_headers],
- },
- }
- )
- return resp["ResponseHeadersPolicy"]["Id"]
-
-
-def _create_function(cloudfront, name: str, code: bytes, runtime: str = "cloudfront-js-2.0") -> str:
- created = cloudfront.create_function(
- Name=name, FunctionConfig={"Comment": name, "Runtime": runtime}, FunctionCode=code,
- )
- published = cloudfront.publish_function(Name=name, IfMatch=created["ETag"])
- return published["FunctionSummary"]["FunctionMetadata"]["FunctionARN"]
-
-
[email protected](scope="module")
-def cf_backend(apigw, lam):
- """A custom API Gateway HTTP API, backed by a Lambda that echoes the
- inbound request, addressed like a real CloudFront custom origin."""
- suffix = _uuid_mod.uuid4().hex[:10]
- lam.create_function(
- FunctionName=f"cf-dp-origin-{suffix}",
- Runtime="python3.12",
- Role="arn:aws:iam::000000000000:role/test-role",
- Handler="index.handler",
- Code={"ZipFile": _zip_lambda(_ORIGIN_LAMBDA_CODE)},
- )
- api_id = apigw.create_api(Name=f"cf-dp-api-{suffix}", ProtocolType="HTTP")["ApiId"]
- int_id = apigw.create_integration(
- ApiId=api_id, IntegrationType="AWS_PROXY",
- IntegrationUri=f"arn:aws:lambda:us-east-1:000000000000:function:cf-dp-origin-{suffix}",
- PayloadFormatVersion="2.0",
- )["IntegrationId"]
- apigw.create_route(ApiId=api_id, RouteKey="ANY /{proxy+}", Target=f"integrations/{int_id}")
- apigw.create_stage(ApiId=api_id, StageName="$default")
- # "*.execute-api.localhost" resolves to 127.0.0.1 (RFC 6761); the origin
- # is dialed there at the gateway's own listening port, exactly as any
- # custom origin is dialed at its configured DomainName/HTTPPort.
- return {"origin_domain": f"{api_id}.execute-api.localhost"}
-
-
[email protected](scope="module")
-def cf_stack(cloudfront, cf_backend):
- """One distribution wired up per the module docstring."""
- suffix = _uuid_mod.uuid4().hex[:10]
- origin_domain = cf_backend["origin_domain"]
-
- cp_none_id = cloudfront.create_cache_policy(
- CachePolicyConfig=_none_cache_policy_config(f"cf-dp-cp-none-{suffix}")
- )["CachePolicy"]["Id"]
-
- default_policy_id = _response_headers_policy(
- cloudfront, f"cf-dp-default-policy-{suffix}",
- [("X-Origin-Header", "policy-value", True), ("X-Default-Marker", "yes", False)],
- )
- csv_policy_id = _response_headers_policy(
- cloudfront, f"cf-dp-csv-policy-{suffix}", [("X-Csv-Marker", "yes", False)],
- )
-
- request_fn_arn = _create_function(
- cloudfront, f"cf-dp-viewer-request-{suffix}",
- b"function handler(event) {\n"
- b" var request = event.request;\n"
- b" if (request.uri === '/fn/blocked') {\n"
- b" return { statusCode: 403, statusDescription: 'Forbidden',\n"
- b" headers: { 'content-type': { value: 'text/plain' } },\n"
- b" body: { encoding: 'text', data: 'blocked by viewer-request' } };\n"
- b" }\n"
- b" request.uri = request.uri.replace('/fn/', '/');\n"
- b" return request;\n"
- b"}\n",
- )
- response_fn_arn = _create_function(
- cloudfront, f"cf-dp-viewer-response-{suffix}",
- b"function handler(event) {\n"
- b" var response = event.response;\n"
- b" response.headers['x-viewer-response-added'] = { value: 'yes' };\n"
- b" return response;\n"
- b"}\n",
- )
- async_fn_arn = _create_function(
- cloudfront, f"cf-dp-async-{suffix}",
- b"async function handler(event) {\n"
- b" var v = await Promise.resolve('yes');\n"
- b" var request = event.request;\n"
- b" request.headers['x-async-ran'] = { value: v };\n"
- b" request.uri = request.uri.replace('/async/', '/');\n"
- b" return request;\n"
- b"}\n",
- )
- error_fn_arn = _create_function(
- cloudfront, f"cf-dp-error-{suffix}",
- b"function handler(event) { throw new Error('boom'); }\n",
- )
- response_delete_multi_fn_arn = _create_function(
- cloudfront, f"cf-dp-response-delete-multi-{suffix}",
- b"function handler(event) {\n"
- b" var response = event.response;\n"
- b" delete response.headers['x-origin-header'];\n"
- b" response.headers['x-multi-resp'] = { value: 'a', multiValue: [{ value: 'a' }, { value: 'b' }] };\n"
- b" return response;\n"
- b"}\n",
- )
- viewer_ip_fn_arn = _create_function(
- cloudfront, f"cf-dp-viewer-ip-{suffix}",
- b"function handler(event) {\n"
- b" var request = event.request;\n"
- b" request.headers['x-viewer-ip'] = { value: event.viewer.ip };\n"
- b" return request;\n"
- b"}\n",
- )
-
- def behavior(path_pattern, target_origin_id="origin-apigw", **overrides):
- b = {
- "PathPattern": path_pattern,
- "TargetOriginId": target_origin_id,
- "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": cp_none_id,
- "OriginRequestPolicyId": _ALL_VIEWER_EXCEPT_HOST,
- "ResponseHeadersPolicyId": default_policy_id,
- }
- b.update(overrides)
- return b
-
- dist_config = {
- "CallerReference": f"cf-dp-{suffix}",
- "Comment": "cloudfront_dataplane test distribution",
- "Enabled": True,
- "DefaultRootObject": "index.html",
- "Origins": {
- "Quantity": 2,
- "Items": [
- {
- "Id": "origin-apigw",
- "DomainName": origin_domain,
- "CustomOriginConfig": {
- "HTTPPort": int(GATEWAY_PORT), "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
- "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
- "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
- },
- },
- {
- "Id": "origin-apigw-prefixed",
- "DomainName": origin_domain,
- "OriginPath": "/extra",
- "CustomHeaders": {
- "Quantity": 1,
- "Items": [{"HeaderName": "X-Added-By-Origin", "HeaderValue": "yes"}],
- },
- "CustomOriginConfig": {
- "HTTPPort": int(GATEWAY_PORT), "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
- "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
- "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
- },
- },
- ],
- },
- "DefaultCacheBehavior": {
- k: v for k, v in behavior("*").items() if k != "PathPattern"
- },
- "CacheBehaviors": {
- "Quantity": 8,
- "Items": [
- {**behavior("*.csv"), "ResponseHeadersPolicyId": csv_policy_id},
- {
- **behavior("/fn/*"),
- "FunctionAssociations": {
- "Quantity": 2,
- "Items": [
- {"FunctionARN": request_fn_arn, "EventType": "viewer-request"},
- {"FunctionARN": response_fn_arn, "EventType": "viewer-response"},
- ],
- },
- },
- {
- **behavior("/async/*"),
- "FunctionAssociations": {
- "Quantity": 1,
- "Items": [{"FunctionARN": async_fn_arn, "EventType": "viewer-request"}],
- },
- },
- {
- **behavior("/err/*"),
- "FunctionAssociations": {
- "Quantity": 1,
- "Items": [{"FunctionARN": error_fn_arn, "EventType": "viewer-request"}],
- },
- },
- {**behavior("/cd/*"), "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": ""},
- {
- **{k: v for k, v in behavior("/legacy/*").items()
- if k not in ("CachePolicyId", "OriginRequestPolicyId")},
- "MinTTL": 0, "DefaultTTL": 0, "MaxTTL": 0,
- "ForwardedValues": {
- "QueryString": True,
- "Cookies": {"Forward": "whitelist", "WhitelistedNames": {"Quantity": 1, "Items": ["keep"]}},
- "Headers": {"Quantity": 1, "Items": ["X-Legacy"]},
- },
- },
- {**behavior("/restricted/*"),
- "AllowedMethods": {"Quantity": 2, "Items": ["GET", "HEAD"]}},
- # No leading "/" — CloudFront treats it the same as "/api/*".
- behavior("api/*"),
- {
- **behavior("/cors/*"), "ResponseHeadersPolicyId": _MANAGED_CORS_WITH_PREFLIGHT,
- "AllowedMethods": {"Quantity": 3, "Items": ["GET", "HEAD", "OPTIONS"]},
- },
- {
- **behavior("/respmulti/*"),
- "FunctionAssociations": {
- "Quantity": 1,
- "Items": [{"FunctionARN": response_delete_multi_fn_arn, "EventType": "viewer-response"}],
- },
- },
- {
- **behavior("/viewerip/*"),
- "FunctionAssociations": {
- "Quantity": 1,
- "Items": [{"FunctionARN": viewer_ip_fn_arn, "EventType": "viewer-request"}],
- },
- },
- ],
- },
- }
- # "/cd/*" carries no OriginRequestPolicyId at all; the empty-string
- # placeholder above exists only so `behavior()`'s dict-merge has a key to
- # drop — CloudFront's own XML omits the element entirely for "no ORP".
- for item in dist_config["CacheBehaviors"]["Items"]:
- if item.get("OriginRequestPolicyId") == "":
- del item["OriginRequestPolicyId"]
- dist_config["CacheBehaviors"]["Items"].append(
- {**behavior("/redirect/*"), "ViewerProtocolPolicy": "redirect-to-https"}
- )
- dist_config["CacheBehaviors"]["Items"].append(
- {**behavior("/prefixed/*", target_origin_id="origin-apigw-prefixed")}
- )
- dist_config["CacheBehaviors"]["Quantity"] = len(dist_config["CacheBehaviors"]["Items"])
-
- dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
- return {"dist_host": dist["DomainName"], "origin_domain": origin_domain}
-
-
-def _get(host: str, path: str, method: str = "GET", extra_headers: dict | None = None):
- """``host`` is a distribution's real AWS-shaped DomainName
- (``<label>.cloudfront.net``), sent verbatim as the Host header — exactly
- what a real viewer sends. The connection itself targets the gateway's own
- address directly, since ``*.cloudfront.net`` doesn't resolve locally (the
- same way ``_raw_get`` below reaches an execute-api/ALB host)."""
- conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
- try:
- headers = {"Host": f"{host}:{GATEWAY_PORT}"}
- headers.update(extra_headers or {})
- conn.request(method, path, headers=headers)
- resp = conn.getresponse()
- return resp.status, dict(resp.getheaders()), resp.read()
- finally:
- conn.close()
-
-
-def _skip_without_node():
- if not cloudfront_js.available():
- pytest.skip("node is not available")
-
-
-# ---------------------------------------------------------------------------
-# Routing
-# ---------------------------------------------------------------------------
-
-
-def test_routes_by_distribution_domain(cf_stack):
- status, _headers, body = _get(cf_stack["dist_host"], "/hello")
- assert status == 200
- assert json.loads(body)["path"] == "/hello"
-
-
-def test_path_pattern_without_leading_slash_matches_as_if_it_had_one(cf_stack):
- # PathPattern "api/*" (no leading /) — CloudFront Developer Guide,
- # "Path pattern": "CloudFront treats the path the same with or without
- # the leading /".
- status, _headers, body = _get(cf_stack["dist_host"], "/api/hello")
- assert status == 200
- assert json.loads(body)["path"] == "/api/hello"
-
-
-def test_unknown_distribution_label_is_not_served_by_cloudfront():
- # Not a 200, and not CloudFront's own data plane (no X-Cache): falls
- # through to whatever the gateway's generic router answers for an
- # otherwise-unrouted host — CloudFront's own control-plane 404, since no
- # other special-case host pattern matches either.
- status, headers, body = _get("d0000000000000.cloudfront.localhost", "/anything")
- assert "X-Cache" not in headers
- assert status == 404
- assert b"<Code>NoSuchResource</Code>" in body
-
-
-# ---------------------------------------------------------------------------
-# CloudFront Functions
-# ---------------------------------------------------------------------------
-
-
-def test_viewer_request_function_rewrites_uri(cf_stack):
- _skip_without_node()
- status, _headers, body = _get(cf_stack["dist_host"], "/fn/hello")
- assert status == 200
- assert json.loads(body)["path"] == "/hello"
-
-
-def test_function_generated_response_skips_the_origin(cf_stack):
- _skip_without_node()
- status, headers, body = _get(cf_stack["dist_host"], "/fn/blocked")
- assert status == 403
- assert body == b"blocked by viewer-request"
- assert headers.get("X-Cache") == "FunctionGeneratedResponse from cloudfront"
- assert "X-Origin-Header" not in headers # never reached the origin
-
-
-def test_viewer_response_function_adds_a_header(cf_stack):
- _skip_without_node()
- status, headers, _body = _get(cf_stack["dist_host"], "/fn/hello")
- assert status == 200
- assert headers.get("X-Viewer-Response-Added") == "yes"
-
-
-def test_async_2_0_handler_runs_and_awaits(cf_stack):
- _skip_without_node()
- status, _headers, body = _get(cf_stack["dist_host"], "/async/hello")
- assert status == 200
- payload = json.loads(body)
- assert payload["path"] == "/hello"
- assert payload["headers"].get("x-async-ran") == "yes"
-
-
-def test_function_error_answers_503(cf_stack):
- _skip_without_node()
- status, headers, _body = _get(cf_stack["dist_host"], "/err/hello")
- assert status == 503
- assert headers.get("X-Cache") == "Error from cloudfront"
-
-
-def test_viewer_response_function_skipped_when_origin_status_is_400_or_above(cf_stack):
- # functions-event-structure.html, "Status code and body" note: CloudFront
- # does not invoke a viewer-response function when the origin answers 400+.
- _skip_without_node()
- status, headers, _body = _get(cf_stack["dist_host"], "/fn/status/500")
- assert status == 500
- assert "X-Viewer-Response-Added" not in headers
-
-
-def test_viewer_response_function_deletes_header_and_expands_multivalue(cf_stack):
- # Replace, not merge (a deleted header stays gone), and a multiValue
- # response header becomes repeated Set-Cookie-style lines.
- _skip_without_node()
- conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
- try:
- conn.putrequest("GET", "/respmulti/hello", skip_host=True, skip_accept_encoding=True)
- conn.putheader("Host", f"{cf_stack['dist_host']}:{GATEWAY_PORT}")
- conn.endheaders()
- resp = conn.getresponse()
- resp.read()
- multi_values = resp.msg.get_all("X-Multi-Resp") or []
- has_origin_header = "X-Origin-Header" in resp.msg
- finally:
- conn.close()
- assert sorted(multi_values) == ["a", "b"]
- assert not has_origin_header
-
-
-def test_function_event_viewer_ip_is_the_tcp_peer_not_the_xff_header(cf_stack):
- # functions-event-structure.html: viewer.ip is the TCP peer, regardless
- # of what X-Forwarded-For the viewer sent.
- _skip_without_node()
- status, _headers, body = _get(
- cf_stack["dist_host"], "/viewerip/hello", extra_headers={"X-Forwarded-For": "203.0.113.9"},
- )
- assert status == 200
- assert json.loads(body)["headers"].get("x-viewer-ip") == "127.0.0.1"
-
-
-# ---------------------------------------------------------------------------
-# Cache behaviors and response-headers policy
-# ---------------------------------------------------------------------------
-
-
-def test_ordered_behavior_matches_csv_glob_with_its_own_policy(cf_stack):
- status, headers, _body = _get(cf_stack["dist_host"], "/report.csv")
- assert status == 200
- assert headers.get("X-Csv-Marker") == "yes"
- assert "X-Default-Marker" not in headers
-
-
-def test_response_headers_policy_overrides_origin_header_and_adds_marker(cf_stack):
- # The origin always emits X-Origin-Header: origin-value; the default
- # behavior's response-headers policy names the same header with
- # Override=True, and adds X-Default-Marker unconditionally.
- status, headers, _body = _get(cf_stack["dist_host"], "/hello")
- assert status == 200
- assert headers.get("X-Origin-Header") == "policy-value"
- assert headers.get("X-Default-Marker") == "yes"
-
-
-def test_cors_headers_only_applied_when_origin_header_present(cf_stack):
- status, headers, _body = _get(cf_stack["dist_host"], "/cors/hello")
- assert status == 200
- assert "Access-Control-Allow-Origin" not in headers
-
- status, headers, _body = _get(
- cf_stack["dist_host"], "/cors/hello", extra_headers={"Origin": "https://example.com"},
- )
- assert status == 200
- assert headers.get("Access-Control-Allow-Origin") == "*"
- assert headers.get("Access-Control-Expose-Headers") == "*"
- # Preflight-only headers (Allow-Methods/-Headers/-Max-Age) never appear
- # on a plain GET.
- assert "Access-Control-Allow-Methods" not in headers
-
-
-def test_cors_preflight_headers_only_on_options(cf_stack):
- # The response-headers policy applies to whatever status the request
- # produces (here, the backing API Gateway HTTP API's own unconfigured-
- # CORS OPTIONS rejection — orthogonal to the response-headers policy
- # under test), so only the header is asserted, not the status.
- _status, headers, _body = _get(
- cf_stack["dist_host"], "/cors/hello", method="OPTIONS",
- extra_headers={"Origin": "https://example.com"},
- )
- assert headers.get("Access-Control-Allow-Methods") == "DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT"
-
-
-def test_cloudfront_added_headers_present(cf_stack):
- status, headers, _body = _get(cf_stack["dist_host"], "/hello")
- assert status == 200
- assert headers.get("X-Cache") == "Miss from cloudfront"
- assert headers.get("Via", "").endswith(".cloudfront.net (CloudFront)")
- assert "X-Amz-Cf-Id" in headers
- assert "X-Amz-Cf-Pop" in headers
-
-
-# ---------------------------------------------------------------------------
-# Cache-policy / origin-request-policy forwarding
-# ---------------------------------------------------------------------------
-
-
-def test_all_viewer_except_host_header_forwards_viewer_header_not_host(cf_stack):
- status, _headers, body = _get(
- cf_stack["dist_host"], "/hello", extra_headers={"X-Custom-Test": "present"},
- )
- assert status == 200
- payload = json.loads(body)
- assert payload["headers"].get("x-custom-test") == "present"
- assert payload["headers"].get("host") == cf_stack["origin_domain"]
-
-
-def test_all_viewer_forwards_the_distributions_own_host(cf_raw_origin_stack):
- # AllViewer forwards every viewer header including Host, so the origin
- # sees the *distribution's* Host, not a default derived from its own
- # DomainName. A real network-distinct origin doesn't care; this uses the
- # raw http.server origin (not the execute-api one) because MiniStack's
- # own execute-api addressing is itself Host-routed, so forwarding the
- # viewer's Host there would recurse into the gateway's own routing
- # instead of reaching the intended origin — an artifact of this
- # emulator's single-port addressing, not of CloudFront's own behavior.
- status, _headers, body = _get(
- cf_raw_origin_stack, "/av/hello", extra_headers={"X-Custom-Test": "present"},
- )
- assert status == 200
- payload = json.loads(body)
- assert payload["headers"].get("x-custom-test") == "present"
- assert payload["headers"].get("host") == f"{cf_raw_origin_stack}:{GATEWAY_PORT}"
-
-
-def test_caching_disabled_with_no_origin_request_policy_forwards_only_defaults(cf_stack):
- # CachingDisabled's ParametersInCacheKeyAndForwardedToOrigin is
- # none/none/none, and the behavior carries no origin request policy at
- # all: nothing beyond the always-included defaults (Host, User-Agent)
- # reaches the origin.
- status, _headers, body = _get(
- cf_stack["dist_host"], "/cd/hello", extra_headers={"X-Custom-Test": "present"},
- )
- assert status == 200
- payload = json.loads(body)
- assert "x-custom-test" not in payload["headers"]
- assert payload["headers"].get("host") == cf_stack["origin_domain"]
-
-
-def test_x_forwarded_for_always_appends_regardless_of_policy(cf_stack):
- # CloudFront Developer Guide, "Client IP addresses": X-Forwarded-For is
- # always the viewer's own chain plus the client IP, even under
- # CachingDisabled with no origin request policy forwarding it.
- status, _headers, body = _get(
- cf_stack["dist_host"], "/cd/hello", extra_headers={"X-Forwarded-For": "203.0.113.9"},
- )
- assert status == 200
- payload = json.loads(body)
- assert payload["headers"].get("x-forwarded-for") == "203.0.113.9, 127.0.0.1"
-
-
-def test_user_agent_defaults_to_amazon_cloudfront_when_not_forwarded(cf_stack):
- # CloudFront Developer Guide, "User-Agent header": replaced unconditionally
- # unless the forwarding policy carries the viewer's own.
- status, _headers, body = _get(
- cf_stack["dist_host"], "/cd/hello", extra_headers={"User-Agent": "custom-client/1.0"},
- )
- assert status == 200
- assert json.loads(body)["headers"].get("user-agent") == "Amazon CloudFront"
-
-
-def test_legacy_forwarded_values_forwards_only_its_own_selections(cf_stack):
- status, _headers, body = _get(
- cf_stack["dist_host"], "/legacy/hello?a=1&b=2",
- extra_headers={"X-Legacy": "yes", "X-Not-Forwarded": "no", "Cookie": "keep=1; drop=2"},
- )
- assert status == 200
- payload = json.loads(body)
- assert payload["query"] == {"a": "1", "b": "2"} # ForwardedValues.QueryString=true
- assert payload["headers"].get("x-legacy") == "yes"
- assert "x-not-forwarded" not in payload["headers"]
- assert payload["cookies"] == ["keep=1"] # only the whitelisted cookie name
-
-
-# ---------------------------------------------------------------------------
-# _combine_forward — the cache-policy / origin-request-policy combinator
-# (same function serves headers, cookies and query strings). Exercised
-# end-to-end above for "all"/"none"; "whitelist" and "allExcept" are cheaper
-# to prove directly against the pure combinator than via more distributions.
-# ---------------------------------------------------------------------------
-
-
- "cache_kind_set,orp_kind_set,name,expected",
- [
- (("whitelist", {"a"}), None, "a", True),
- (("whitelist", {"a"}), None, "b", False),
- (("whitelist", {"a"}), ("whitelist", {"b"}), "b", True), # ORP allow-list adds to cache's
- (("allExcept", {"a"}), None, "a", False), # blocked by the cache policy
- (("allExcept", {"a"}), None, "b", True),
- (("allExcept", {"a"}), ("allExcept", {"b"}), "a", False), # ORP allExcept doesn't rescue it
- (("allExcept", {"a"}), ("whitelist", {"a"}), "a", True), # ORP allow-list rescues a blocked name
- (("none", None), ("allExcept", {"a"}), "a", False),
- (("none", None), ("allExcept", {"a"}), "b", True),
- (("none", None), None, "a", False),
- ],
-)
-def test_combine_forward_whitelist_and_allexcept_branches(cache_kind_set, orp_kind_set, name, expected):
- assert cloudfront_dataplane._combine_forward(cache_kind_set, orp_kind_set, name) == expected
-
-
-# ---------------------------------------------------------------------------
-# Behavior-level settings
-# ---------------------------------------------------------------------------
-
-
-def test_redirect_to_https_sends_301_with_https_location(cf_stack):
- status, headers, _body = _get(cf_stack["dist_host"], "/redirect/hello")
- assert status == 301
- assert headers.get("Location") == f"https://{cf_stack['dist_host']}:{GATEWAY_PORT}/redirect/hello"
-
-
-def test_allowed_methods_rejects_a_disallowed_method(cf_stack):
- status, _headers, _body = _get(cf_stack["dist_host"], "/restricted/hello", method="POST")
- assert status == 403
-
-
-def test_default_root_object_is_requested_for_the_root_url(cf_stack):
- status, _headers, body = _get(cf_stack["dist_host"], "/")
- assert status == 200
- assert json.loads(body)["path"] == "/index.html"
-
-
-def test_origin_path_and_custom_headers_reach_the_origin(cf_stack):
- status, _headers, body = _get(cf_stack["dist_host"], "/prefixed/hello")
- assert status == 200
- payload = json.loads(body)
- assert payload["path"] == "/extra/prefixed/hello"
- assert payload["headers"].get("x-added-by-origin") == "yes"
-
-
-def test_multiple_set_cookie_headers_all_carried_through(cf_stack):
- conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
- try:
- conn.putrequest("GET", "/cookies", skip_host=True, skip_accept_encoding=True)
- conn.putheader("Host", f"{cf_stack['dist_host']}:{GATEWAY_PORT}")
- conn.endheaders()
- resp = conn.getresponse()
- resp.read()
- cookies = resp.msg.get_all("Set-Cookie") or []
- finally:
- conn.close()
- assert sorted(cookies) == ["resp1=a; Path=/", "resp2=b; Path=/"]
-
-
-# ---------------------------------------------------------------------------
-# AWS-owned origin hosts — refused before a socket opens; no server or
-# distribution needed, calls the pure function. An emulator must never open
-# a socket to a real AWS host.
-# ---------------------------------------------------------------------------
-
-
-def test_aws_owned_origin_host_refused_before_any_socket():
- origin = {
- "domain_name": "abcd1234.execute-api.us-east-1.amazonaws.com.", # trailing dot
- "origin_path": "", "custom_headers": [], "s3_bucket": None,
- "http_port": 80, "https_port": 443, "protocol_policy": "https-only",
- }
- with mock.patch("http.client.HTTPSConnection") as https_conn, \
- mock.patch("http.client.HTTPConnection") as http_conn:
- with pytest.raises(cloudfront_dataplane.OriginUnreachable):
- cloudfront_dataplane._forward_to_origin(
- origin, "GET", "/anything", "", {}, b"", viewer_is_https=True,
- )
- https_conn.assert_not_called()
- http_conn.assert_not_called()
-
-
-def test_origin_ssl_context_trusts_gateway_cert_under_use_ssl(tmp_path):
- # Precedent: lambda_svc.py's TLS trust for Lambda containers — system
- # roots plus the gateway's own cert, not a replacement of the store.
- from ministack.core.x509_utils import generate_ca
-
- ca_pem, _ca_key_pem = generate_ca(common_name="cf-dp-origin-ssl-test")
- cert_path = tmp_path / "ca.pem"
- cert_path.write_text(ca_pem)
- with mock.patch("ministack.core.tls.use_ssl_enabled", return_value=True), \
- mock.patch("ministack.core.tls.resolve_tls_material", return_value=(str(cert_path), "key")):
- ctx = cloudfront_dataplane._origin_ssl_context()
- assert ctx.verify_flags & ssl.VERIFY_X509_PARTIAL_CHAIN
- subjects = [dict(rdn for pair in cert["subject"] for rdn in pair) for cert in ctx.get_ca_certs()]
- assert any(s.get("commonName") == "cf-dp-origin-ssl-test" for s in subjects)
-
-
-def test_stacked_distribution_via_header_answers_403(cloudfront):
- # CloudFront Developer Guide, http-403-permission-denied.html, "Stacked
- # distributions cause a 403 error": an origin pointed back at this same
- # distribution (via the resolvable `<label>.cloudfront.localhost` form)
- # must answer 403 on the second hop rather than recurse into a hang.
- suffix = _uuid_mod.uuid4().hex[:10]
- dist_config = {
- "CallerReference": f"cf-dp-stacked-{suffix}",
- "Comment": "cloudfront_dataplane stacked-distribution test",
- "Enabled": True,
- "Origins": {
- "Quantity": 1,
- "Items": [{
- "Id": "self-origin",
- "DomainName": "placeholder.cloudfront.localhost",
- "CustomOriginConfig": {
- "HTTPPort": int(GATEWAY_PORT), "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
- "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
- "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
- },
- }],
- },
- "DefaultCacheBehavior": {
- "TargetOriginId": "self-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED,
- },
- }
- created = cloudfront.create_distribution(DistributionConfig=dist_config)
- dist = created["Distribution"]
- label = dist["DomainName"].removesuffix(".cloudfront.net")
- dist_config["Origins"]["Items"][0]["DomainName"] = f"{label}.cloudfront.localhost"
- cloudfront.update_distribution(DistributionConfig=dist_config, Id=dist["Id"], IfMatch=created["ETag"])
-
- status, _headers, _body = _get(dist["DomainName"], "/anything")
- assert status == 403
-
-
-def test_unreachable_origin_answers_502(cloudfront):
- suffix = _uuid_mod.uuid4().hex[:10]
- dist_config = {
- "CallerReference": f"cf-dp-dead-{suffix}",
- "Comment": "cloudfront_dataplane dead-origin test",
- "Enabled": True,
- "Origins": {
- "Quantity": 1,
- "Items": [{
- "Id": "dead-origin",
- "DomainName": "127.0.0.1",
- "CustomOriginConfig": {
- "HTTPPort": 1, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only", # nothing listens on port 1
- "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
- "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
- },
- }],
- },
- "DefaultCacheBehavior": {
- "TargetOriginId": "dead-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED,
- },
- }
- dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
- status, headers, _body = _get(dist["DomainName"], "/anything")
- assert status == 502
- assert headers.get("X-Cache") == "Error from cloudfront"
-
-
-def test_forward_to_origin_timeout_raises_origin_timeout():
- # CloudFront Developer Guide, http-504-gateway-timeout.html: "The origin
- # didn't respond before the request expired." OriginReadTimeout default
- # 30s (DownloadDistValuesOrigin.md, "Response timeout").
- origin = {
- "domain_name": "127.0.0.1", "origin_path": "", "custom_headers": [], "s3_bucket": None,
- "http_port": 1, "https_port": 443, "protocol_policy": "http-only", "read_timeout": 30,
- }
- with mock.patch("http.client.HTTPConnection") as mock_conn_cls:
- mock_conn_cls.return_value.request.side_effect = TimeoutError("timed out")
- with pytest.raises(cloudfront_dataplane.OriginTimeout):
- cloudfront_dataplane._forward_to_origin(
- origin, "GET", "/anything", "", {}, b"", viewer_is_https=False,
- )
-
-
-def test_handle_request_maps_origin_timeout_to_504():
- parsed = {
- "default_root_object": "",
- "origins": {"o1": {
- "domain_name": "127.0.0.1", "origin_path": "", "custom_headers": [],
- "s3_bucket": None, "http_port": 1, "https_port": 443,
- "protocol_policy": "http-only", "read_timeout": 30,
- }},
- "default_behavior": {
- "path_pattern": None, "target_origin_id": "o1", "viewer_protocol_policy": "allow-all",
- "allowed_methods": None, "cache_policy_id": _CACHING_DISABLED,
- "origin_request_policy_id": None, "response_headers_policy_id": None,
- "forwarded_values": None, "functions": {},
- },
- "ordered_behaviors": [],
- }
- dist = {"Id": "EDUMMY00000000", "DomainName": "ddummy0000000.cloudfront.net"}
- with mock.patch("ministack.services.cloudfront.parse_distribution_dataplane_config", return_value=parsed), \
- mock.patch("ministack.services.cloudfront_dataplane._forward_to_origin",
- side_effect=cloudfront_dataplane.OriginTimeout("slow")):
- status, headers, _body = asyncio.run(
- cloudfront_dataplane.handle_request(dist, "GET", "/anything", "/anything", "", {}, b"", {})
- )
- assert status == 504
- assert headers.get("X-Cache") == "Error from cloudfront"
-
-
-# ---------------------------------------------------------------------------
-# S3 origin — served from MiniStack's own S3 store in-process.
-# ---------------------------------------------------------------------------
-
-
[email protected](scope="module")
-def cf_s3_stack(cloudfront, s3):
- suffix = _uuid_mod.uuid4().hex[:10]
- bucket = f"cf-dp-s3-origin-{suffix}"
- s3.create_bucket(Bucket=bucket)
- body = b"hello from the bucket\n"
- s3.put_object(Bucket=bucket, Key="hello.txt", Body=body, ContentType="text/plain")
-
- dist_config = {
- "CallerReference": f"cf-dp-s3-{suffix}",
- "Comment": "cloudfront_dataplane S3-origin test distribution",
- "Enabled": True,
- "Origins": {
- "Quantity": 1,
- "Items": [{
- "Id": "s3-origin",
- "DomainName": f"{bucket}.s3.amazonaws.com",
- "S3OriginConfig": {"OriginAccessIdentity": ""},
- "OriginAccessControlId": "test-oac-id",
- }],
- },
- "DefaultCacheBehavior": {
- "TargetOriginId": "s3-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED,
- },
- }
- dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
- s3.put_bucket_policy(Bucket=bucket, Policy=json.dumps({"Statement": [{
- "Effect": "Allow", "Principal": {"Service": "cloudfront.amazonaws.com"},
- "Action": "s3:GetObject", "Resource": f"arn:aws:s3:::{bucket}/*",
- "Condition": {"StringEquals": {"AWS:SourceArn": dist["ARN"]}},
- }]}))
- return {"dist_host": dist["DomainName"], "body": body}
-
-
-def test_s3_origin_without_a_bucket_policy_grant_is_denied(cloudfront, s3):
- """An OAC origin whose bucket policy does not grant the distribution answers 403."""
- suffix = _uuid_mod.uuid4().hex[:10]
- bucket = f"cf-dp-s3-nogrant-{suffix}"
- s3.create_bucket(Bucket=bucket)
- s3.put_object(Bucket=bucket, Key="hello.txt", Body=b"x")
- dist = cloudfront.create_distribution(DistributionConfig={
- "CallerReference": f"cf-dp-s3-nogrant-{suffix}", "Comment": "", "Enabled": True,
- "Origins": {"Quantity": 1, "Items": [{
- "Id": "s3-origin", "DomainName": f"{bucket}.s3.amazonaws.com",
- "S3OriginConfig": {"OriginAccessIdentity": ""}, "OriginAccessControlId": "test-oac-id",
- }]},
- "DefaultCacheBehavior": {"TargetOriginId": "s3-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED},
- })["Distribution"]
- status, _headers, body = _get(dist["DomainName"], "/hello.txt")
- assert status == 403
- assert b"<Code>AccessDenied</Code>" in body
-
-
-def test_s3_origin_serves_bucket_object(cf_s3_stack):
- status, headers, body = _get(cf_s3_stack["dist_host"], "/hello.txt")
- assert status == 200
- assert body == cf_s3_stack["body"]
- assert headers.get("Content-Type") == "text/plain"
-
-
-def test_s3_origin_range_get_returns_partial_content(cf_s3_stack):
- # Range is forwarded regardless of cache/origin-request-policy header
- # configuration (CloudFront Developer Guide, "HTTP request headers and
- # CloudFront behavior"), even under CachingDisabled with no ORP here.
- status, headers, body = _get(
- cf_s3_stack["dist_host"], "/hello.txt", extra_headers={"Range": "bytes=0-4"},
- )
- assert status == 206
- assert body == cf_s3_stack["body"][:5]
- assert headers.get("Content-Range", "").startswith("bytes 0-4/")
-
-
-def test_s3_origin_missing_key_returns_access_denied(cf_s3_stack):
- # The OAC bucket policy CloudFront's console generates grants only
- # s3:GetObject, never s3:ListBucket — without ListBucket, S3 answers a
- # missing key with 403 AccessDenied rather than 404 NoSuchKey.
- status, _headers, body = _get(cf_s3_stack["dist_host"], "/does-not-exist.txt")
- assert status == 403
- assert b"<Code>AccessDenied</Code>" in body
-
-
-def _s3_oac_origin(origin_id: str, bucket: str) -> dict:
- return {"Id": origin_id, "DomainName": f"{bucket}.s3.amazonaws.com",
- "S3OriginConfig": {"OriginAccessIdentity": ""}, "OriginAccessControlId": "test-oac-id"}
-
-
-def _custom_error_dist(cloudfront, suffix: str, origins: list, error_responses: list, behaviors: list | None = None):
- return cloudfront.create_distribution(DistributionConfig={
- "CallerReference": f"cf-dp-errors-{suffix}", "Comment": "", "Enabled": True,
- "Origins": {"Quantity": len(origins), "Items": origins},
- "DefaultCacheBehavior": {"TargetOriginId": origins[0]["Id"], "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED},
- "CacheBehaviors": {"Quantity": len(behaviors or []), "Items": behaviors or []},
- "CustomErrorResponses": {"Quantity": len(error_responses), "Items": error_responses},
- })["Distribution"]
-
-
-def test_custom_error_response_serves_the_page_with_the_configured_status(cloudfront, s3):
- """A single-page app: S3 answers 403 for a client route, CloudFront serves /index.html with 200."""
- suffix = _uuid_mod.uuid4().hex[:10]
- bucket = f"cf-dp-spa-{suffix}"
- s3.create_bucket(Bucket=bucket)
- s3.put_object(Bucket=bucket, Key="index.html", Body=b"<html>app</html>", ContentType="text/html")
- dist = _custom_error_dist(cloudfront, suffix, [_s3_oac_origin("site", bucket)], [
- {"ErrorCode": 403, "ResponsePagePath": "/index.html", "ResponseCode": "200", "ErrorCachingMinTTL": 10},
- ])
- s3.put_bucket_policy(Bucket=bucket, Policy=json.dumps({"Statement": [{
- "Effect": "Allow", "Principal": {"Service": "cloudfront.amazonaws.com"},
- "Action": "s3:GetObject", "Resource": f"arn:aws:s3:::{bucket}/*",
- "Condition": {"StringEquals": {"AWS:SourceArn": dist["ARN"]}},
- }]}))
-
- status, headers, body = _get(dist["DomainName"], "/auth/callback")
- assert status == 200
- assert body == b"<html>app</html>"
- assert headers.get("Content-Type") == "text/html"
-
-
-def test_unavailable_custom_error_page_answers_the_status_its_origin_returned(cloudfront, s3, cf_backend):
- """The Developer Guide's example ("Generate custom error responses"): the custom origin answers 500 and the
- page is missing from its S3 bucket, so the viewer gets S3's status, neither 500 nor the configured 200."""
- suffix = _uuid_mod.uuid4().hex[:10]
- pages = f"cf-dp-pages-{suffix}"
- s3.create_bucket(Bucket=pages)
- dist = _custom_error_dist(
- cloudfront, suffix,
- [{"Id": "api", "DomainName": cf_backend["origin_domain"], "CustomOriginConfig": {
- "HTTPPort": int(GATEWAY_PORT), "HTTPSPort": 443, "OriginProtocolPolicy": "http-only"}},
- _s3_oac_origin("pages", pages)],
- [{"ErrorCode": 500, "ResponsePagePath": "/errors/500.html", "ResponseCode": "200"}],
- [{"PathPattern": "/errors/*", "TargetOriginId": "pages", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED}],
- )
-
- status, _headers, body = _get(dist["DomainName"], "/status/500")
- assert status == 403
- assert b"<Code>AccessDenied</Code>" in body
-
-
-# ---------------------------------------------------------------------------
-# Percent-encoding fidelity (CloudFront Developer Guide, "Restrictions on all
-# edge functions" > "URI, query string, and headers encoding").
-# ---------------------------------------------------------------------------
-
-
-class _PathEchoHandler(http.server.BaseHTTPRequestHandler):
- def do_GET(self):
- body = json.dumps({
- "path": self.path,
- "headers": {k.lower(): v for k, v in self.headers.items()},
- }).encode()
- self.send_response(200)
- self.send_header("Content-Type", "application/json")
- self.send_header("Content-Length", str(len(body)))
- if self.path.endswith("/hopbyhop"):
- self.send_header("Connection", "keep-alive")
- self.send_header("X-Should-Survive", "yes")
- if self.path.endswith("/respdup"):
- self.send_header("X-Resp-Dup", "first")
- self.send_header("X-Resp-Dup", "second")
- self.end_headers()
- self.wfile.write(body)
-
- def do_POST(self):
- length = int(self.headers.get("Content-Length", 0) or 0)
- posted_body = self.rfile.read(length) if length else b""
- body = json.dumps({
- "path": self.path,
- "body": posted_body.decode("utf-8", errors="replace"),
- "content_length_header": self.headers.get("Content-Length"),
- }).encode()
- self.send_response(200)
- self.send_header("Content-Type", "application/json")
- self.send_header("Content-Length", str(len(body)))
- self.end_headers()
- self.wfile.write(body)
-
- def log_message(self, format, *args):
- pass
-
-
[email protected](scope="module")
-def path_echo_origin_port():
- server = http.server.HTTPServer(("127.0.0.1", 0), _PathEchoHandler)
- port = server.server_address[1]
- thread = threading.Thread(target=server.serve_forever, daemon=True)
- thread.start()
- try:
- yield port
- finally:
- server.shutdown()
- thread.join(timeout=5)
- server.server_close()
-
-
[email protected](scope="module")
-def cf_raw_origin_stack(cloudfront, path_echo_origin_port):
- suffix = _uuid_mod.uuid4().hex[:10]
- passthrough_fn_arn = None
- if cloudfront_js.available():
- passthrough_fn_arn = _create_function(
- cloudfront, f"cf-dp-passthrough-{suffix}",
- b"function handler(event) { return event.request; }\n",
- )
- dist_config = {
- "CallerReference": f"cf-dp-raw-{suffix}",
- "Comment": "cloudfront_dataplane raw-path-fidelity test distribution",
- "Enabled": True,
- "Origins": {
- "Quantity": 1,
- "Items": [{
- "Id": "raw-origin",
- "DomainName": "127.0.0.1",
- "CustomOriginConfig": {
- "HTTPPort": path_echo_origin_port, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
- "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
- "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
- },
- }],
- },
- "DefaultCacheBehavior": {
- "TargetOriginId": "raw-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED,
- },
- "CacheBehaviors": {
- "Quantity": 1,
- "Items": [{
- "PathPattern": "/av/*",
- "TargetOriginId": "raw-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": _ALL_VIEWER,
- }],
- },
- }
- if passthrough_fn_arn:
- dist_config["CacheBehaviors"]["Quantity"] = 2
- dist_config["CacheBehaviors"]["Items"].append({
- "PathPattern": "/fn/*",
- "TargetOriginId": "raw-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": _ALL_VIEWER,
- "FunctionAssociations": {
- "Quantity": 1, "Items": [{"FunctionARN": passthrough_fn_arn, "EventType": "viewer-request"}],
- },
- })
- dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
- return dist["DomainName"]
-
-
-def _raw_get(dist_host: str, raw_path: str):
- conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
- try:
- conn.putrequest("GET", raw_path, skip_host=True, skip_accept_encoding=True)
- conn.putheader("Host", f"{dist_host}:{GATEWAY_PORT}")
- conn.endheaders()
- resp = conn.getresponse()
- return resp.status, resp.read()
- finally:
- conn.close()
-
-
-def test_percent_encoded_path_forwarded_byte_exact(cf_raw_origin_stack):
- for raw_path in (
- "/echo/%21%24%26%27%28%29%2A%2B%2C%3B%3D%40",
- "/ready_pin_guests/123%2Ejson",
- ):
- status, body = _raw_get(cf_raw_origin_stack, raw_path)
- assert status == 200
- assert json.loads(body)["path"] == raw_path
-
-
-def test_query_string_forwarded_byte_exact_when_unmodified_and_fully_allowed(cf_raw_origin_stack):
- # "/av/*" carries AllViewer (forwards every query-string parameter) and
- # no function: nothing should re-encode the viewer's own percent-encoding.
- raw_path = "/av/hello?a=%68%65%6c%6c%6f&b=1"
- status, body = _raw_get(cf_raw_origin_stack, raw_path)
- assert status == 200
- assert json.loads(body)["path"] == raw_path
-
-
-def test_function_returning_the_request_keeps_a_bare_query_key(cf_raw_origin_stack):
- """AWS: observed 2026-10-10, a distribution whose viewer-request function returns event.request:
- GET /locations?type[] got the response a Rails origin gives the bare key (?type[]= gets another)."""
- _skip_without_node()
- raw_path = "/fn/hello?flag&a=1"
- status, body = _raw_get(cf_raw_origin_stack, raw_path)
- assert status == 200
- assert json.loads(body)["path"] == raw_path
-
-
-def test_post_body_round_trips_with_content_length(cf_raw_origin_stack):
- conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
- try:
- payload = b'{"hello": "world"}'
- conn.putrequest("POST", "/av/echo", skip_host=True, skip_accept_encoding=True)
- conn.putheader("Host", f"{cf_raw_origin_stack}:{GATEWAY_PORT}")
- conn.putheader("Content-Type", "application/json")
- conn.putheader("Content-Length", str(len(payload)))
- conn.endheaders(payload)
- resp = conn.getresponse()
- body = resp.read()
- finally:
- conn.close()
- echoed = json.loads(body)
- assert echoed["body"] == payload.decode()
- assert echoed["content_length_header"] == str(len(payload))
-
-
-def test_hop_by_hop_response_headers_stripped(cf_raw_origin_stack):
- status, headers, _body = _get(cf_raw_origin_stack, "/av/hopbyhop")
- assert status == 200
- assert "Connection" not in headers
- assert headers.get("X-Should-Survive") == "yes"
-
-
-# ---------------------------------------------------------------------------
-# Raw request-header fidelity toward the origin — a plain http.server or
-# http.client doesn't distinguish "one folded line" from "two lines with the
-# same name", so this reads the exact bytes off the socket.
-# ---------------------------------------------------------------------------
-
-
-class _RawHeaderCaptureServer:
- """Accepts one connection at a time, records the raw request-header
- block exactly as received, and answers a trivial 200."""
-
- def __init__(self):
- self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
- self.sock.bind(("127.0.0.1", 0))
- self.sock.listen(5)
- self.port = self.sock.getsockname()[1]
- self.last_header_lines: list = []
- self._stop = False
- self.thread = threading.Thread(target=self._serve, daemon=True)
- self.thread.start()
-
- def _serve(self):
- self.sock.settimeout(0.5)
- while not self._stop:
- try:
- conn, _ = self.sock.accept()
- except socket.timeout:
- continue
- except OSError:
- break
- with conn:
- conn.settimeout(5)
- data = b""
- while b"\r\n\r\n" not in data:
- chunk = conn.recv(4096)
- if not chunk:
- break
- data += chunk
- head, _, _rest = data.partition(b"\r\n\r\n")
- lines = head.decode("latin-1").split("\r\n")
- self.last_header_lines = lines[1:] # drop the request line
- body = b"ok"
- conn.sendall(
- b"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: "
- + str(len(body)).encode() + b"\r\n\r\n" + body
- )
-
- def stop(self):
- self._stop = True
- self.sock.close()
- self.thread.join(timeout=5)
-
-
[email protected](scope="module")
-def raw_header_server():
- server = _RawHeaderCaptureServer()
- try:
- yield server
- finally:
- server.stop()
-
-
[email protected](scope="module")
-def cf_raw_header_stack(cloudfront, raw_header_server):
- suffix = _uuid_mod.uuid4().hex[:10]
- dup_edit_fn_arn = None
- if cloudfront_js.available():
- # Viewer headers are already comma-joined before this module ever
- # sees them (app.py, RFC 9110 §5.2), so a function is the only way to
- # produce a genuine multiValue request header here.
- dup_edit_fn_arn = _create_function(
- cloudfront, f"cf-dp-dup-header-edit-{suffix}",
- b"function handler(event) {\n"
- b" var request = event.request;\n"
- b" request.headers['x-dup'] = { value: 'a', multiValue: [{ value: 'a' }, { value: 'b' }] };\n"
- b" return request;\n"
- b"}\n",
- )
- dist_config = {
- "CallerReference": f"cf-dp-rawheaders-{suffix}",
- "Comment": "cloudfront_dataplane raw request-header test distribution",
- "Enabled": True,
- "Origins": {
- "Quantity": 1,
- "Items": [{
- "Id": "raw-header-origin",
- "DomainName": "127.0.0.1",
- "CustomOriginConfig": {
- "HTTPPort": raw_header_server.port, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
- "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
- "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
- },
- # Lowercase, viewer sends Title-Case — proves the overwrite is
- # case-insensitive, not merely a literal-name match.
- "CustomHeaders": {"Quantity": 1, "Items": [{"HeaderName": "x-custom-test", "HeaderValue": "from-origin"}]},
- }],
- },
- "DefaultCacheBehavior": {
- "TargetOriginId": "raw-header-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": _ALL_VIEWER,
- },
- }
- if dup_edit_fn_arn:
- dist_config["CacheBehaviors"] = {
- "Quantity": 1,
- "Items": [{
- "PathPattern": "/dupheader/*",
- "TargetOriginId": "raw-header-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED, "OriginRequestPolicyId": _ALL_VIEWER,
- "FunctionAssociations": {
- "Quantity": 1,
- "Items": [{"FunctionARN": dup_edit_fn_arn, "EventType": "viewer-request"}],
- },
- }],
- }
- dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
- return dist["DomainName"]
-
-
-def test_origin_request_headers_no_duplicates_case_insensitive_overwrite_and_xff(
- cf_raw_header_stack, raw_header_server,
-):
- conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
- try:
- conn.putrequest("GET", "/anything", skip_host=True, skip_accept_encoding=True)
- conn.putheader("Host", f"{cf_raw_header_stack}:{GATEWAY_PORT}")
- conn.putheader("X-Custom-Test", "viewer-value")
- conn.putheader("X-Forwarded-For", "203.0.113.5")
- conn.endheaders()
- resp = conn.getresponse()
- resp.read()
- finally:
- conn.close()
-
- lines = raw_header_server.last_header_lines
- names_lower = [line.split(":", 1)[0].lower() for line in lines if line]
- # AllViewer forwards Host too: exactly one line, not a lowercase
- # viewer-forwarded one plus a Title-Cased one this module also adds.
- assert names_lower.count("host") == 1
-
- custom_test_lines = [line for line in lines if line.lower().startswith("x-custom-test:")]
- assert len(custom_test_lines) == 1
- assert custom_test_lines[0].split(":", 1)[1].strip() == "from-origin"
-
- xff_lines = [line for line in lines if line.lower().startswith("x-forwarded-for:")]
- assert len(xff_lines) == 1
- assert xff_lines[0].split(":", 1)[1].strip() == "203.0.113.5, 127.0.0.1"
-
-
-def test_function_set_multivalue_header_forwarded_as_one_comma_joined_line(
- cf_raw_header_stack, raw_header_server,
-):
- # RFC 9110 §5.3: a repeated field value is one comma-joined line when
- # forwarded to the origin, not several lines of the same name.
- _skip_without_node()
- conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
- try:
- conn.putrequest("GET", "/dupheader/hello", skip_host=True, skip_accept_encoding=True)
- conn.putheader("Host", f"{cf_raw_header_stack}:{GATEWAY_PORT}")
- conn.endheaders()
- resp = conn.getresponse()
- resp.read()
- finally:
- conn.close()
-
- dup_lines = [line for line in raw_header_server.last_header_lines if line.lower().startswith("x-dup:")]
- assert len(dup_lines) == 1
- assert dup_lines[0].split(":", 1)[1].strip() == "a, b"
-
-
[email protected](scope="module")
-def cf_response_dup_stack(cloudfront, path_echo_origin_port):
- """A distribution whose origin (the raw path-echo server) sends a
- genuinely repeated response header, for the response-side multiValue
- edit-only merge rule below."""
- suffix = _uuid_mod.uuid4().hex[:10]
- fn_arn = None
- if cloudfront_js.available():
- fn_arn = _create_function(
- cloudfront, f"cf-dp-response-dup-edit-{suffix}",
- b"function handler(event) {\n"
- b" var response = event.response;\n"
- b" if (response.headers['x-resp-dup']) {\n"
- b" response.headers['x-resp-dup'].value = 'edited';\n"
- b" }\n"
- b" return response;\n"
- b"}\n",
- )
- behavior = {
- "TargetOriginId": "respdup-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED,
- }
- if fn_arn:
- behavior["FunctionAssociations"] = {
- "Quantity": 1, "Items": [{"FunctionARN": fn_arn, "EventType": "viewer-response"}],
- }
- dist_config = {
- "CallerReference": f"cf-dp-respdup-{suffix}",
- "Comment": "cloudfront_dataplane response multiValue edit-only test",
- "Enabled": True,
- "Origins": {
- "Quantity": 1,
- "Items": [{
- "Id": "respdup-origin",
- "DomainName": "127.0.0.1",
- "CustomOriginConfig": {
- "HTTPPort": path_echo_origin_port, "HTTPSPort": 443, "OriginProtocolPolicy": "http-only",
- "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
- "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
- },
- }],
- },
- "DefaultCacheBehavior": behavior,
- }
- dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
- return dist["DomainName"]
-
-
-def test_viewer_response_function_edits_only_value_of_repeated_header_keeps_remaining_values(
- cf_response_dup_stack,
-):
- # functions-event-structure.html: a header whose multiValue is unchanged
- # from the event is read only for its primary value; the rest of its
- # original values survive.
- _skip_without_node()
- conn = http.client.HTTPConnection("127.0.0.1", int(GATEWAY_PORT), timeout=10)
- try:
- conn.putrequest("GET", "/respdup", skip_host=True, skip_accept_encoding=True)
- conn.putheader("Host", f"{cf_response_dup_stack}:{GATEWAY_PORT}")
- conn.endheaders()
- resp = conn.getresponse()
- resp.read()
- dup_values = resp.msg.get_all("X-Resp-Dup") or []
- finally:
- conn.close()
- assert dup_values == ["edited", "second"]
-
-
-# ---------------------------------------------------------------------------
-# TLS to a custom https origin.
-# ---------------------------------------------------------------------------
-
-
-def untrusted_https_origin_port(tmp_path):
- pytest.importorskip("cryptography")
- from ministack.core.x509_utils import generate_ca, sign_leaf_certificate
-
- ca_pem, ca_key_pem = generate_ca(common_name="cf-dp-untrusted-ca")
- leaf_pem, leaf_key_pem, _public = sign_leaf_certificate(
- ca_pem, ca_key_pem, common_name="127.0.0.1", san_ips=["127.0.0.1"],
- )
- cert_path = tmp_path / "leaf.pem"
- cert_path.write_text(leaf_pem + leaf_key_pem)
-
- server = http.server.HTTPServer(("127.0.0.1", 0), _PathEchoHandler)
- ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
- ctx.load_cert_chain(str(cert_path))
- server.socket = ctx.wrap_socket(server.socket, server_side=True)
- port = server.server_address[1]
- thread = threading.Thread(target=server.serve_forever, daemon=True)
- thread.start()
- try:
- yield port
- finally:
- server.shutdown()
- thread.join(timeout=5)
- server.server_close()
-
-
-def test_untrusted_origin_certificate_answers_502(cloudfront, untrusted_https_origin_port):
- # http-502-bad-gateway.html's self-signed/untrusted-chain case: the
- # origin's own CA isn't in the system trust store (and USE_SSL is off in
- # this test harness, so the gateway cert isn't trusted either).
- suffix = _uuid_mod.uuid4().hex[:10]
- dist_config = {
- "CallerReference": f"cf-dp-untrusted-{suffix}",
- "Comment": "cloudfront_dataplane untrusted-origin-cert test",
- "Enabled": True,
- "Origins": {
- "Quantity": 1,
- "Items": [{
- "Id": "untrusted-origin",
- "DomainName": "127.0.0.1",
- "CustomOriginConfig": {
- "HTTPPort": 80, "HTTPSPort": untrusted_https_origin_port,
- "OriginProtocolPolicy": "https-only",
- "OriginSslProtocols": {"Quantity": 1, "Items": ["TLSv1.2"]},
- "OriginReadTimeout": 30, "OriginKeepaliveTimeout": 5,
- },
- }],
- },
- "DefaultCacheBehavior": {
- "TargetOriginId": "untrusted-origin", "ViewerProtocolPolicy": "allow-all",
- "CachePolicyId": _CACHING_DISABLED,
- },
- }
- dist = cloudfront.create_distribution(DistributionConfig=dist_config)["Distribution"]
- status, headers, _body = _get(dist["DomainName"], "/anything")
- assert status == 502
- assert headers.get("X-Cache") == "Error from cloudfront"
diff --git a/tests/test_ec2_verified_access.py b/tests/test_ec2_verified_access.py
index c4a82254..5687dfa7 100644
--- a/tests/test_ec2_verified_access.py
+++ b/tests/test_ec2_verified_access.py
@@ -1946,3 +1946,55 @@ def test_verified_access_create_permission_contract(auth_enabled, iam_user):
else:
inst_id = _instance(denied)
denied.delete_verified_access_instance(VerifiedAccessInstanceId=inst_id)
+
+
+def test_verified_access_deny_on_each_authorized_resource_holds(auth_enabled, iam_user):
+ """AWS authorizes a group move on the group and the instance, and a detach on the instance and the provider."""
+ admin = make_client("ec2")
+ inst_id, oidc_id = _instance_with_provider(admin)
+ device_id = _device_provider(admin)["VerifiedAccessTrustProviderId"]
+ _attach(admin, inst_id, device_id)
+ group_id = _group(admin, inst_id, Description="before")
+ prefix = "arn:aws:ec2:us-east-1:000000000000:"
+ client = iam_user(policy={"Version": "2012-10-17", "Statement": [
+ {"Effect": "Allow", "Action": "ec2:*", "Resource": "*"},
+ {"Effect": "Deny", "Action": "ec2:ModifyVerifiedAccessGroup", "Resource": f"{prefix}verified-access-group/{group_id}"},
+ {"Effect": "Deny", "Action": "ec2:DetachVerifiedAccessTrustProvider",
+ "Resource": f"{prefix}verified-access-trust-provider/{device_id}"},
+ ]}).client("ec2")
+ try:
+ calls = (
+ lambda: client.modify_verified_access_group(VerifiedAccessGroupId=group_id, VerifiedAccessInstanceId=inst_id,
+ Description="after"),
+ lambda: client.detach_verified_access_trust_provider(VerifiedAccessInstanceId=inst_id,
+ VerifiedAccessTrustProviderId=device_id),
+ )
+ for call in calls:
+ if auth_enabled:
+ with pytest.raises(ClientError) as exc:
+ call()
+ assert (_status(exc), _error(exc)[0]) == (403, "UnauthorizedOperation")
+ else:
+ call()
+ group = admin.describe_verified_access_groups(VerifiedAccessGroupIds=[group_id])["VerifiedAccessGroups"][0]
+ assert group["Description"] == ("before" if auth_enabled else "after")
+ assert (device_id in _provider_ids(admin, inst_id)) is auth_enabled
+ finally:
+ _cleanup(admin, groups=[group_id], instances=[inst_id], providers=[oidc_id, device_id])
+
+
+def test_verified_access_resource_arns_cover_every_authorized_resource():
+ from ministack.core.iam_actions import verified_access_resource_arns
+
+ prefix = "arn:aws:ec2:us-east-1:000000000000:"
+ params = {"Action": "CreateVerifiedAccessEndpoint", "VerifiedAccessGroupId": "vagr-00000000000000009",
+ "LoadBalancerOptions.SubnetId.1": "subnet-a", "LoadBalancerOptions.SubnetId.2": "subnet-b",
+ "SecurityGroupId.1": "sg-a"}
+ arns = verified_access_resource_arns(params, "us-east-1", "000000000000")
+ assert set(arns) == {prefix + r for r in ("security-group/sg-a", "subnet/subnet-a", "subnet/subnet-b",
+ "verified-access-endpoint/*", "verified-access-group/*")}
+ assert verified_access_resource_arns({"Action": "CreateVerifiedAccessGroup", "VerifiedAccessInstanceId": "vai-00000000000000009"},
+ "us-east-1", "000000000000") == [
+ prefix + "verified-access-instance/*", prefix + "verified-access-group/*"]
+ assert verified_access_resource_arns({"Action": "DescribeVerifiedAccessGroups"}, "us-east-1", "000000000000") == []
diff --git a/tests/test_ecs.py b/tests/test_ecs.py
index e9f3f637..09521989 100644
--- a/tests/test_ecs.py
+++ b/tests/test_ecs.py
@@ -4459,3 +4459,52 @@ def test_ecs_unavailable_registry_secret_fails_start_without_exposing_secret(
assert not client.images.registry_calls
if secret:
assert secret not in task["stoppedReason"]
+
+
+def test_ecs_service_capacity_provider_strategy_replaces_the_launch_type(ecs):
+ cluster = f"cps-{_uuid_mod.uuid4().hex[:8]}"
+ bare = f"{cluster}-bare"
+ fargate = [{"capacityProvider": "FARGATE", "weight": 1}]
+ ecs.create_cluster(clusterName=cluster)
+ ecs.create_cluster(clusterName=bare)
+ ecs.put_cluster_capacity_providers(cluster=cluster, capacityProviders=["FARGATE", "FARGATE_SPOT"],
+ defaultCapacityProviderStrategy=fargate)
+ ecs.register_task_definition(family=cluster, requiresCompatibilities=["FARGATE"], networkMode="awsvpc",
+ cpu="256", memory="512",
+ containerDefinitions=[{"name": "app", "image": "nginx:alpine", "essential": True}])
+ net = {"awsvpcConfiguration": {"subnets": ["subnet-00000001"]}}
+ try:
+ created = ecs.create_service(cluster=cluster, serviceName="given", taskDefinition=cluster, desiredCount=0,
+ capacityProviderStrategy=fargate, networkConfiguration=net)["service"]
+ described = ecs.describe_services(cluster=cluster, services=["given"])["services"][0]
+ for svc in (created, described, created["deployments"][0]):
+ assert svc["capacityProviderStrategy"] == fargate
+ assert "launchType" not in svc
+
+ default = ecs.create_service(cluster=cluster, serviceName="default", taskDefinition=cluster,
+ desiredCount=0, networkConfiguration=net)["service"]
+ assert default["capacityProviderStrategy"] == fargate and "launchType" not in default
+ plain = ecs.create_service(cluster=bare, serviceName="plain", taskDefinition=cluster, desiredCount=0)["service"]
+ assert plain["launchType"] == "EC2" and "capacityProviderStrategy" not in plain
+
+ with pytest.raises(ClientError) as exc:
+ ecs.create_service(cluster=cluster, serviceName="both", taskDefinition=cluster, launchType="FARGATE",
+ capacityProviderStrategy=fargate, networkConfiguration=net)
+ assert exc.value.response["Error"]["Code"] == "InvalidParameterException"
+ assert exc.value.response["Error"]["Message"] == (
+ "Specifying both a launch type and capacity provider strategy is not supported. Remove one and try again.")
+
+ reverted = ecs.update_service(cluster=cluster, service="given", capacityProviderStrategy=[])["service"]
+ assert reverted["launchType"] == "EC2" and "capacityProviderStrategy" not in reverted
+
+ task = ecs.run_task(cluster=cluster, taskDefinition=cluster, networkConfiguration=net,
+ capacityProviderStrategy=[{"capacityProvider": "FARGATE_SPOT", "weight": 1}])["tasks"][0]
+ assert (task["launchType"], task["capacityProviderName"]) == ("FARGATE", "FARGATE_SPOT")
+ ecs.stop_task(cluster=cluster, task=task["taskArn"])
+ finally:
+ for c, names in ((cluster, ("given", "default")), (bare, ("plain",))):
+ for name in names:
+ try:
+ ecs.delete_service(cluster=c, service=name, force=True)
+ except ClientError:
+ pass
diff --git a/tests/test_iam_auth.py b/tests/test_iam_auth.py
index 567fd42d..69ee72e9 100644
--- a/tests/test_iam_auth.py
+++ b/tests/test_iam_auth.py
@@ -5507,6 +5507,13 @@ class TestQuerySignatures:
assert (code, status) == ("IncompleteSignature", 400)
assert "Credential must have exactly 5 slash-delimited elements" in body
+ @pytest.mark.parametrize("query,service", [(_LIST_USERS, "iam"), (_LIST_QUEUES, "sqs")])
+ def test_s3_scoped_signature_is_refused_by_another_service(self, user_key, query, service):
+ """Only S3 takes a query signed for s3; a correct one is refused anywhere else (#2338)."""
+ code, status, body = _presigned(*user_key, "s3", query)
+ assert (code, status) == ("SignatureDoesNotMatch", 403)
+ assert f"Credential should be scoped to correct service: '{service}'. " in body
+
def test_root_key_signs_with_any_secret(self, user_key):
assert _presigned("test", "anything", "iam", _LIST_USERS)[:2] == (None, 200)
diff --git a/tests/test_lambda.py b/tests/test_lambda.py
index c6e742e0..48a76c95 100644
--- a/tests/test_lambda.py
+++ b/tests/test_lambda.py
@@ -7946,6 +7946,19 @@ def _spawn_capture_run_kwargs(monkeypatch, *, endpoint, docker_flags="", config_
return captured
[email protected]("network", ["", "example-net"])
+def test_lambda_container_publishes_its_runtime_port_on_loopback_only(monkeypatch, network):
+ """With DOCKER_NETWORK the runtime is reached on that network and no port is published (#2365)."""
+ monkeypatch.setattr(lsvc, "LAMBDA_DOCKER_NETWORK", network)
+ monkeypatch.setattr(lsvc, "_running_in_container", lambda: False)
+ captured = _spawn_capture_run_kwargs(monkeypatch, endpoint="http://localhost:4566")
+ if network:
+ assert captured["network"] == network
+ assert "ports" not in captured
+ else:
+ assert captured["ports"] == {"8080/tcp": ("127.0.0.1", None)}
+
+
@pytest.mark.parametrize("timeout", [3, 301, 900])
@pytest.mark.parametrize("package_type", ["Zip", "Image"])
@pytest.mark.parametrize("in_container", [False, True])
@@ -15641,3 +15654,31 @@ def test_slow_code_update_does_not_block_independent_resources(functions, monkey
def test_missing_resource_keeps_not_found_response(functions, operation):
assert _update(operation, "stale", name="missing")[0] == 404
assert _lambda_mod._update_alias("revision-test", "missing", {"RevisionId": "stale"})[0] == 404
+
+
+def test_lambda_function_url_event_and_response_follow_aws(lam):
+ """Captured on AWS: the URL id, the front end's request fields, accountId and the response ids."""
+ name = f"fu-aws-{_uuid_mod.uuid4().hex[:10]}"
+ code = ("import json\n"
+ "def handler(event, context):\n"
+ " return {'statusCode': 200, 'body': json.dumps(event)}\n")
+ lam.create_function(FunctionName=name, Runtime="python3.12", Role=_LAMBDA_ROLE, Handler="index.handler",
+ Code={"ZipFile": _make_zip(code)})
+ try:
+ url = lam.create_function_url_config(FunctionName=name, AuthType="NONE")["FunctionUrl"]
+ url_id = _function_url_id(url)
+ assert re.fullmatch(r"https://[a-z0-9]{32}\.lambda-url\.us-east-1\.on\.aws/", url)
+ req = _function_url_request(url_id, path="/echo", headers={"X-Forwarded-For": "203.0.113.9, 198.51.100.1"})
+ with _urlreq.urlopen(req) as resp:
+ event = json.loads(resp.read())
+ request_id, trace = resp.headers["x-amzn-RequestId"], resp.headers["X-Amzn-Trace-Id"]
+ headers = event["headers"]
+ assert headers["host"] == f"{url_id}.lambda-url.us-east-1.on.aws"
+ assert (headers["x-forwarded-for"], headers["x-forwarded-port"], headers["x-forwarded-proto"]) == (
+ "203.0.113.9", "443", "https")
+ assert event["requestContext"]["accountId"] == "anonymous"
+ assert event["requestContext"]["requestId"] == request_id
+ assert trace.startswith(headers["x-amzn-trace-id"] + ";Parent=")
+ assert re.fullmatch(r"Root=1-[0-9a-f]{8}-[0-9a-f]{24};Parent=[0-9a-f]{16};Sampled=0;Lineage=1:[0-9a-f]{8}:0", trace)
+ finally:
+ lam.delete_function(FunctionName=name)
diff --git a/tests/test_opensearchserverless.py b/tests/test_opensearchserverless.py
index 27a4d687..8eecdc21 100644
--- a/tests/test_opensearchserverless.py
+++ b/tests/test_opensearchserverless.py
@@ -492,7 +492,7 @@ def test_aoss_data_plane_denylist(method, path, query, ctype, supported):
def _gateway_request(host_header, method, path, headers=None, body=b""):
"""Connect to the gateway's own address, sending ``host_header`` as Host —
- the way test_cloudfront_dataplane.py's ``_get`` reaches a virtual host
+ the way test_cloudfront.py's ``_get`` reaches a virtual host
that doesn't resolve locally."""
conn = http.client.HTTPConnection(urllib.parse.urlparse(ENDPOINT).hostname, int(GATEWAY_PORT), timeout=10)
try:
diff --git a/tests/test_stepfunctions.py b/tests/test_stepfunctions.py
index 62c31925..93b5eda2 100644
--- a/tests/test_stepfunctions.py
+++ b/tests/test_stepfunctions.py
@@ -9978,3 +9978,47 @@ def test_sfn_describe_for_execution_through_an_alias_reports_the_version(sfn, en
sfn.delete_state_machine_alias(stateMachineAliasArn=alias)
sfn.delete_state_machine(stateMachineArn=created["stateMachineArn"])
+
+
+def test_sfn_distributed_map_runs_child_executions_of_a_map_run(sfn, s3):
+ """A DISTRIBUTED Map creates a Map Run of child executions and reads an ItemReader dataset (#2366)."""
+ uid = _uuid_mod.uuid4().hex[:8]
+ bucket = f"dmap-{uid}"
+ s3.create_bucket(Bucket=bucket)
+ s3.put_object(Bucket=bucket, Key="items.json", Body=b"[1, 2, 3]")
+ s3.put_object(Bucket=bucket, Key="rows.csv", Body=b"a,b\n1,2\n3\n")
+ processor = {"ProcessorConfig": {"Mode": "DISTRIBUTED", "ExecutionType": "STANDARD"},
+ "StartAt": "P", "States": {"P": {"Type": "Pass", "End": True}}}
+
+ def run(name, map_fields, payload):
+ definition = {"StartAt": "Map", "States": {"Map": {**map_fields, "Type": "Map", "ItemProcessor": processor,
+ "End": True}}}
+ arn = sfn.create_state_machine(name=f"{name}-{uid}", definition=json.dumps(definition),
+ roleArn="arn:aws:iam::000000000000:role/r")["stateMachineArn"]
+ execution = sfn.start_execution(stateMachineArn=arn, input=json.dumps(payload))["executionArn"]
+ return arn, execution, _wait_sfn(sfn, execution, timeout=20)
+
+ arn, execution, desc = run("dmap-input", {"ItemsPath": "$.items", "Label": "lbl"}, {"items": [1, 2, 3]})
+ assert desc["status"] == "SUCCEEDED" and json.loads(desc["output"]) == [1, 2, 3]
+ (map_run,) = sfn.list_map_runs(executionArn=execution)["mapRuns"]
+ assert map_run["mapRunArn"].startswith(f"{arn.replace(':stateMachine:', ':mapRun:')}/lbl:")
+ assert map_run["stateMachineArn"] == f"{arn}/lbl"
+ described = sfn.describe_map_run(mapRunArn=map_run["mapRunArn"])
+ assert described["status"] == "SUCCEEDED"
+ assert (described["itemCounts"]["succeeded"], described["executionCounts"]["total"]) == (3, 3)
+ children = sfn.list_executions(mapRunArn=map_run["mapRunArn"])["executions"]
+ assert len(children) == 3 and all(c["stateMachineArn"] == f"{arn}/lbl" for c in children)
+ assert len(sfn.list_executions(stateMachineArn=arn)["executions"]) == 1
+
+ reader = {"Resource": "arn:aws:states:::s3:getObject", "Parameters": {"Bucket": bucket, "Key": "items.json"},
+ "ReaderConfig": {"InputType": "JSON"}}
+ _, _, desc = run("dmap-json", {"ItemReader": reader}, {"note": "the items are in S3"})
+ assert json.loads(desc["output"]) == [1, 2, 3]
+ csv_reader = {"Resource": "arn:aws:states:::s3:getObject", "Parameters": {"Bucket": bucket, "Key": "rows.csv"},
+ "ReaderConfig": {"InputType": "CSV", "CSVHeaderLocation": "FIRST_ROW"}}
+ _, _, desc = run("dmap-csv", {"ItemReader": csv_reader}, {})
+ assert json.loads(desc["output"]) == [{"a": "1", "b": "2"}, {"a": "3", "b": ""}]
+ list_reader = {"Resource": "arn:aws:states:::s3:listObjectsV2", "Parameters": {"Bucket": bucket}}
+ _, _, desc = run("dmap-list", {"ItemReader": list_reader}, {})
+ assert [o["Key"] for o in json.loads(desc["output"])] == ["items.json", "rows.csv"]
+ assert set(json.loads(desc["output"])[0]) == {"Etag", "Key", "LastModified", "Size", "StorageClass"}