Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(merges): correct validations per doc/aws (#1706)

Nahuel Nuceraab658aba5f9e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d6b68161..61a394b8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,14 +14,17 @@ Versioning follows [Semantic Versioning](https://semver.org/).
### Fixed
- **S3 — the `s3:TestEvent` no longer reaches Lambda targets** — `PutBucketNotificationConfiguration` fanned the test event out to every destination, Lambda included, but AWS sends it to SQS queues and SNS topics only; a Lambda destination is verified through its function permissions, not by invoking it. The payload carries no `Records` array, so each S3-triggered function raised on it, and because delivery goes through the async path the event was retried to `MaximumRetryAttempts` (default 2) and then routed to the function's DLQ or `OnFailure` destination, where a synthetic event AWS never sends looked like a genuinely dropped message. Any CDK stack with an S3-triggered Lambda hit this through `BucketNotificationsHandler`. Queue and topic destinations still receive the test event, and real object events still invoke the function. Contributed by @ppettitau.
### Changed
+
- **Lambda — warm local custom runtimes** — `provided.*` bootstraps now reuse the existing subprocess worker pool instead of restarting on every invocation. Each invocation receives its request metadata through the Lambda Runtime API, concurrent calls use separate workers, and failed environments are cleaned up before reuse. Durable `provided.*` invocations retain their one-shot executor, because a bootstrap process reads its environment once, at spawn; Docker, image, and proxy execution are unchanged.
### Fixed
+
+- **S3 — the `s3:TestEvent` no longer reaches Lambda targets** — `PutBucketNotificationConfiguration` fanned the test event out to every destination, Lambda included, but AWS sends it to SQS queues and SNS topics only; a Lambda destination is verified through its function permissions, not by invoking it. The payload carries no `Records` array, so each S3-triggered function raised on it, and because delivery goes through the async path the event was retried to `MaximumRetryAttempts` (default 2) and then routed to the function's DLQ or `OnFailure` destination, where a synthetic event AWS never sends looked like a genuinely dropped message. Any CDK stack with an S3-triggered Lambda hit this through `BucketNotificationsHandler`. Queue and topic destinations still receive the test event, and real object events still invoke the function. Contributed by @ppettitau.
- **S3 — a presigned upload is no longer rejected for the checksum in its query string** — since v3.729.0 the JavaScript SDK signs `x-amz-checksum-crc32` into every presigned `PutObject` by default, computed over the *empty* body, because whoever holds the URL chooses the body later. MiniStack hoisted that parameter into the request headers alongside the user metadata a presigned URL legitimately carries there, and the header path then verified it against the real body, so a stock `getSignedUrl` upload failed with `400 BadDigest` where real S3 answers `200` and stores the object. The `x-amz-checksum-*` value parameters are now excluded from hoisting: they still take part in signature verification, so rewriting one in the URL is still `403`, but they are no longer read as a supplied integrity value. A checksum sent as a real request header is verified exactly as before. `x-amz-checksum-algorithm` and `x-amz-sdk-checksum-algorithm` keep hoisting — they name an algorithm for the server to compute rather than carrying a value, so they cannot disagree with a body, and a presigned `CreateMultipartUpload` uses either to choose the algorithm its parts are digested with.
- **CloudFormation — an `AWS::ApiGateway::Method`'s `MethodResponses` and `IntegrationResponses` are provisioned** — the provisioner read the method and integration properties and discarded both response lists, so every REST API deployed from a template lost its mapped response headers. The visible casualty was the CDK's `defaultCorsPreflightOptions`, whose generated `OPTIONS` method returns the `Access-Control-Allow-*` headers through a MOCK integration's `responseParameters`: the preflight answered without a single CORS header and the browser blocked the request. That became load-bearing once `COGNITO_USER_POOLS` authorizers were enforced in 1.5.10, because the preflight can no longer fall through to a proxy integration and be answered by the application's own CORS middleware. Both lists are now provisioned onto the method, where the existing request-time mapping picks them up, and a stack update reprovisions them from the template. Contributed by @ppettitau.
-- **Transcribe — `StartTranscriptionJob` returns a job already `IN_PROGRESS`** — the start response reported `QUEUED` with no `StartTime`, and the job reached `IN_PROGRESS` only from the background worker, after the response had been sent. A caller that reads the status off the start response and then waits for the terminal `Transcribe Job State Change` event, rather than polling, therefore never saw an in-progress state at all: the job went straight from queued to complete. Jobs now start `IN_PROGRESS` with `StartTime` set, matching the response the AWS CLI walkthrough documents, and `TRANSCRIBE_JOB_RUN_SECONDS` covers the whole run rather than half of it. `QUEUED`, which AWS reaches only for a request that opted into job queueing through `JobExecutionSettings.AllowDeferredExecution` while the account is at its concurrent job limit, stays reachable through the new `TRANSCRIBE_JOB_QUEUE_SECONDS` (default `0`); both knobs are settable at runtime through `/_ministack/config`. Contributed by @ppettitau.
+- **Transcribe — `StartTranscriptionJob` returns a job already `IN_PROGRESS`** — the start response reported `QUEUED` with no `StartTime`, and the job reached `IN_PROGRESS` only from the background worker, after the response had been sent. A caller that reads the status off the start response and then waits for the terminal `Transcribe Job State Change` event, rather than polling, therefore never saw an in-progress state at all: the job went straight from queued to complete. Jobs now start `IN_PROGRESS` with `StartTime` set, matching the response the AWS CLI walkthrough documents, and `TRANSCRIBE_JOB_RUN_SECONDS` covers the whole run rather than half of it. `QUEUED`, which AWS reaches only for a request that opted into job queueing through `JobExecutionSettings.AllowDeferredExecution` while the account is at its concurrent job limit, is not modelled, so a job never queues. `TRANSCRIBE_JOB_RUN_SECONDS` is settable at runtime through `/_ministack/config`. Contributed by @ppettitau.
- **CloudFormation — `UpdateReplacePolicy: Retain` holds for every handler-side replacement** — 24 update handlers that replace a resource themselves (a renamed SSM parameter, CloudWatch dashboard, IoT policy or provisioning template, a Cognito user pool group or client moved to another pool, an S3 bucket policy moved to another bucket, the API Gateway REST resources, a Lambda permission, URL, event source mapping or event invoke config, a Firehose delivery stream, an OpenSearch domain) deleted the predecessor inline, so a template that retains it lost the resource, and where the physical id changed the stack still recorded `DELETE_SKIPPED` for it; only the handlers routed through the shared rename helper honoured the policy. Every inline delete now goes through one helper that reads the policy the engine publishes, the three handlers that deleted before creating create first, and the predecessor stays, with its `DELETE_SKIPPED` event where the replacement changes the physical id, as on AWS. Contributed by @iot-rocket.
@@ -62,9 +65,11 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- **CloudFormation — an unknown `Capabilities` value is refused** — `CreateStack`, `UpdateStack` and `CreateChangeSet` accepted any string, so a typo or a capability of another tool (`CAPABILITY_RESOURCE_POLICY`, which the CloudFormation API does not define) was kept without effect and the deploy behaved as if it had been acknowledged. The member is now checked against the three documented values with the API's parameter validation message, joined with the other request-level problems. Contributed by @iot-rocket.
### Fixed
- **CloudFormation — the template and stack quotas are enforced** — a template of any size, with any number of resources, parameters, outputs or mappings, and a stack name of any shape were accepted, so a template that a real account refuses deployed. `CreateStack`, `UpdateStack`, `CreateChangeSet`, `ValidateTemplate` and `GetTemplateSummary` now refuse a `TemplateBody` over 51,200 bytes and a template behind `TemplateURL` over 1,000,000 bytes; the engine refuses more than 500 resources, 200 parameters, 200 outputs, 200 mappings or 200 attributes in a mapping, a name over 255 characters, a description over 1,024 bytes and a parameter default or value over 4,096 bytes; `CreateStack` and a `CREATE` change set refuse a stack name that is not alphanumeric-and-hyphens starting with a letter or longer than 128 characters, joined with an oversized body into the one message the API's parameter validation gives. All of it before a stack record exists, with the messages of the API's parameter validation and the template format errors. `ValidateTemplate` also takes `TemplateURL` now. Contributed by @iot-rocket.
+
- **Lambda — a Node custom resource signals its stack in the docker executor** — the response submitters the CDK bundles into its custom-resource handlers (`nodejs-entrypoint`, the provider framework, `AwsCustomResource`) build the `ResponseURL` PUT from the URL's hostname and path only and hand it to `https.request`, so under `LAMBDA_EXECUTOR=docker` the callback went out over TLS to port 443, nothing answered, and an `AwsCustomResource`-backed resource ran its API call and then hung its stack until `ServiceTimeout` (the local executor already downgraded https for the gateway). The shim the docker executor injects now turns https to the gateway hosts (the endpoint host, `host.docker.internal`, `localhost` and `MINISTACK_HOST`) on the https default port or the gateway port into plain http on the gateway port; other hosts, and those hosts on other ports, keep TLS. The local executor's own list gains `MINISTACK_HOST` as well. Contributed by @iot-rocket.
-### Fixed
+
- **CloudFormation — `Capabilities` are enforced under `AUTH=true`** — `CreateStack`, `UpdateStack` and `CreateChangeSet` accepted a template with IAM resources or a `Transform` whatever the request acknowledged, so a deploy that CloudFormation refuses with `InsufficientCapabilitiesException` went through. With `AUTH=true` they now answer that error (HTTP 400, `Requires capabilities : [CAPABILITY_IAM]`, measured) and create nothing: an IAM resource needs `CAPABILITY_IAM` or `CAPABILITY_NAMED_IAM`, one with a custom name `CAPABILITY_NAMED_IAM`, a template with a `Transform` `CAPABILITY_AUTO_EXPAND` on `CreateStack` and `UpdateStack` (not on a change set, as the API documents). Without `AUTH` nothing changes; `GetTemplateSummary` keeps reporting the same set. Contributed by @iot-rocket.
+
- **Lambda — a durable invocation runs on the warm pool** — durable Python and Node.js invocations were sent to the one-shot executor, because the three `AWS_LAMBDA_DURABLE_*` variables change on every call while a pooled worker's environment is fixed when it spawns. Each of them therefore paid a fresh interpreter start and the botocore import inside the function's `Timeout`, whose default is 3 seconds, so on a loaded machine every durable invocation ended as a timeout. The values now travel in the event, the channel the X-Ray trace header and the invoke-depth counter already use, and both worker bootstraps move them into the environment before the handler runs and drop them again when the next invocation is not durable. Contributed by @iot-rocket.
## [1.5.10] — 2026-09-10
diff --git a/README.md b/README.md
index ae6bf3ce..3025bd7d 100644
--- a/README.md
+++ b/README.md
@@ -427,7 +427,7 @@ subnet = ec2.create_subnet(
| **Bedrock Agent** | 72 operations — agents (versions, aliases, action groups, collaborators, agent knowledge bases, PrepareAgent), knowledge bases (data sources, ingestion jobs — `StartIngestionJob` reads the S3 data source and reports real statistics, `FAILED` with `failureReasons` when the source is unreadable — documents), flows (aliases, versions, ValidateFlowDefinition, PrepareFlow), prompts with versions, tagging | All 72 operations verified against botocore `bedrock-agent-2023-06-05` |
| **Bedrock Agent Runtime** | InvokeAgent, InvokeInlineAgent, GetAgentMemory, DeleteAgentMemory, Retrieve, RetrieveAndGenerate, RetrieveAndGenerateStream, Rerank, CreateSession, GetSession, UpdateSession, DeleteSession, EndSession, ListSessions, CreateInvocation, ListInvocations, PutInvocationStep, GetInvocationStep, ListInvocationSteps, InvokeFlow, StartFlowExecution, StopFlowExecution, GetFlowExecution, ListFlowExecutions, ListFlowExecutionEvents, GetExecutionFlowSnapshot, OptimizePrompt | All 31 operations verified against botocore `bedrock-agent-runtime-2023-07-26`; eventstream responses on streaming operations; `Retrieve` searches the documents Knowledge Base ingestion stored (lexical scoring — no embeddings) and returns content, `s3Location` and score; `InvokeAgent` answers a canned reply without orchestration and emits no trace events — it never fabricates an `ORCHESTRATE` trace |
| **Bedrock AgentCore** | Control plane (`bedrock-agentcore-control`): CreateAgentRuntime, GetAgentRuntime, ListAgentRuntimes, UpdateAgentRuntime, DeleteAgentRuntime, ListAgentRuntimeVersions, CreateAgentRuntimeEndpoint, GetAgentRuntimeEndpoint, ListAgentRuntimeEndpoints, UpdateAgentRuntimeEndpoint, DeleteAgentRuntimeEndpoint. Data plane (`bedrock-agentcore`): InvokeAgentRuntime | Agent-runtime lifecycle with `CREATING`/`UPDATING`→`READY` transitions, versioned ARNs (`agent/{uuid}:{version}`, `agentEndpoint/{uuid}`); request/response shapes verified against botocore `bedrock-agentcore-control` / `bedrock-agentcore`; account- and region-scoped state; `InvokeAgentRuntime` is a deterministic echo (no model inference), matching the requested emulation scope |
-| **Transcribe** | StartTranscriptionJob, GetTranscriptionJob, ListTranscriptionJobs, DeleteTranscriptionJob | Batch transcription jobs only; streaming is a separate endpoint (`transcribestreaming`). All four operations verified against botocore `transcribe-2017-10-26`. A job starts `IN_PROGRESS` with `StartTime` set, as on AWS, and reaches `COMPLETED` over `TRANSCRIBE_JOB_RUN_SECONDS` (default 2, `0` completes immediately); `QUEUED`, which AWS reaches only for a request that opted into job queueing while the account is at its concurrent job limit, is reachable by setting `TRANSCRIBE_JOB_QUEUE_SECONDS`; the media object is read from MiniStack S3 via an `s3://`, path-style or virtual-hosted HTTPS URI, and an unreadable URI fails the job with a `FailureReason` rather than erroring at `Start`, as on AWS; the transcript is written back to S3 in the real result format (`results.transcripts`, word-level `results.items`, `results.audio_segments`, plus `results.segments` and `results.speaker_labels` when the matching `Settings` are set) at `OutputBucketName`/`OutputKey`, or to an auto-created `ministack-transcribe-output` bucket. `IdentifyLanguage` and `IdentifyMultipleLanguages` resolve a language on completion, `ContentRedaction` controls whether a `redacted-` copy replaces or accompanies the transcript, and `Subtitles` writes WebVTT and SubRip files, redacted alongside the transcript when redaction is on. Reaching a terminal state publishes a `Transcribe Job State Change` event, so EventBridge rules downstream of a job fire. There is no speech recognition, so the transcript is deterministic canned text keyed on a digest of the media URI |
+| **Transcribe** | StartTranscriptionJob, GetTranscriptionJob, ListTranscriptionJobs, DeleteTranscriptionJob | Batch transcription jobs only; streaming is a separate endpoint (`transcribestreaming`). All four operations verified against botocore `transcribe-2017-10-26`. A job starts `IN_PROGRESS` with `StartTime` set, as on AWS, and reaches `COMPLETED` over `TRANSCRIBE_JOB_RUN_SECONDS` (default 2, `0` completes immediately); `QUEUED`, which AWS reaches only for a request that opted into job queueing (`JobExecutionSettings.AllowDeferredExecution`) while the account is at its concurrent job limit, is not modelled; the media object is read from MiniStack S3 via an `s3://`, path-style or virtual-hosted HTTPS URI, and an unreadable URI fails the job with a `FailureReason` rather than erroring at `Start`, as on AWS; the transcript is written back to S3 in the real result format (`results.transcripts`, word-level `results.items`, `results.audio_segments`, plus `results.segments` and `results.speaker_labels` when the matching `Settings` are set) at `OutputBucketName`/`OutputKey`, or to an auto-created `ministack-transcribe-output` bucket. `IdentifyLanguage` and `IdentifyMultipleLanguages` resolve a language on completion, `ContentRedaction` controls whether a `redacted-` copy replaces or accompanies the transcript, and `Subtitles` writes WebVTT and SubRip files, redacted alongside the transcript when redaction is on. Reaching a terminal state publishes a `Transcribe Job State Change` event, so EventBridge rules downstream of a job fire. There is no speech recognition, so the transcript is deterministic canned text keyed on a digest of the media URI |
| **Translate** | StartTextTranslationJob, DescribeTextTranslationJob, ListTextTranslationJobs, StopTextTranslationJob | Batch text translation jobs only; the synchronous `TranslateText` / `TranslateDocument` pair and the custom terminology / parallel data families are not implemented. All four operations verified against botocore `translate-2017-07-01`. Jobs walk `SUBMITTED` → `IN_PROGRESS` → `COMPLETED` over `TRANSLATE_JOB_RUN_SECONDS` (default 2, `0` completes immediately); every document under `InputDataConfig.S3Uri` is read from MiniStack S3 and written back to the rewritten output location AWS uses — `OutputDataConfig.S3Uri` is reported as `s3://<bucket>/<prefix><account-id>-TranslateText-<JobId>/` — as `<target-code>.<input name>`, with a `details/<code>.auxiliary-translation-details.json` summary per target language. `text/plain` is translated line by line, `text/html` keeps its markup and translates only text nodes, `application/x-xliff+xml` gets a populated `<target>` in every `<trans-unit>` plus `target-language` on every `<file>`, and the three Office ZIP types are copied through. A document that can't be read as its content type is counted in `JobDetails` and the job ends `COMPLETED_WITH_ERROR`; an empty or unreadable input location fails the job with a `Message` rather than erroring at `Start`. `ClientToken` is idempotent, `StopTextTranslationJob` reaches a running worker, and reaching a terminal state publishes a `Translate TextTranslationJob State Change` event, so EventBridge rules downstream of a job fire. There is no machine translation, so the output is deterministic canned text derived from the source |
| **MSK** | CreateCluster, ListClusters, DescribeCluster, DeleteCluster, GetBootstrapBrokers, ListNodes, CreateConfiguration, ListConfigurations, DescribeConfiguration, ListConfigurationRevisions, DescribeConfigurationRevision, BatchAssociateScramSecret, BatchDisassociateScramSecret, ListScramSecrets, TagResource, UntagResource, ListTagsForResource | Kafka control plane; `GetBootstrapBrokers` honors `MINISTACK_MSK_BOOTSTRAP` so clients route to a real broker you bring (Redpanda, Kafka, KRaft) — the Kafka wire protocol itself is not emulated |
| **WAF v2** | CreateWebACL, GetWebACL, UpdateWebACL, DeleteWebACL, ListWebACLs, AssociateWebACL, DisassociateWebACL, GetWebACLForResource, ListResourcesForWebACL, CreateIPSet, GetIPSet, UpdateIPSet, DeleteIPSet, ListIPSets, CreateRuleGroup, GetRuleGroup, UpdateRuleGroup, DeleteRuleGroup, ListRuleGroups, TagResource, UntagResource, ListTagsForResource, CheckCapacity, DescribeManagedRuleGroup | LockToken enforced on Update/Delete; resource associations tracked |
@@ -847,7 +847,6 @@ end-to-end without any client config.
| `CODEBUILD_DOCKER_FLAGS` | _(unset)_ | Extra `docker run` flags for the CodeBuild local-agent container, same syntax and parser as `LAMBDA_DOCKER_FLAGS`. On an SELinux-enforcing host the agent needs `--security-opt label=disable` to reach the Docker socket it is handed |
| `MINISTACK_IMAGE_PREFIX` | _(unset)_ | Private-registry prefix prepended to every nested container image (RDS postgres/mysql/mariadb, ElastiCache redis/memcached, EKS k3s, Lambda runtimes). Testcontainers' `hub.image.name.prefix` is auto-forwarded into this var by the Java/Python modules |
| `LAMBDA_WARM_TTL_SECONDS` | `300` | How long an idle warm Lambda container stays in the pool before the reaper evicts it |
-| `LAMBDA_PROVIDED_INIT_TIMEOUT` | `20` | Seconds a local warm `provided.*` bootstrap has to issue its first Runtime API `/next` request |
| `LAMBDA_ACCOUNT_CONCURRENCY` | `0` | Account-level concurrent-invocation cap (0 = unbounded). Match real AWS by setting to `1000`. Used to simulate `ConcurrentInvocationLimitExceeded` throttles |
| `SFN_MOCK_CONFIG` | _(unset)_ | Path to JSON file for Step Functions mock testing; compatible with AWS SFN Local format. Also accepts `LOCALSTACK_SFN_MOCK_CONFIG` |
| `ATHENA_ENGINE` | `auto` | SQL engine for Athena: `auto`, `duckdb`, `mock` |
@@ -865,7 +864,6 @@ end-to-end without any client config.
| `MINISTACK_MSK_BOOTSTRAP_SASL_SCRAM` | _(unset)_ | SASL/SCRAM bootstrap string (`BootstrapBrokerStringSaslScram`) |
| `MINISTACK_MSK_BOOTSTRAP_SASL_IAM` | _(unset)_ | SASL/IAM bootstrap string (`BootstrapBrokerStringSaslIam`) |
| `TRANSCRIBE_JOB_RUN_SECONDS` | `2` | Wall-clock time a Transcribe job spends between `IN_PROGRESS` and `COMPLETED`. Set `0` to complete immediately, or raise it when a test needs to observe `IN_PROGRESS`. Also settable without a restart by posting `{"transcribe._JOB_RUN_SECONDS": 10}` to `/_ministack/config` |
-| `TRANSCRIBE_JOB_QUEUE_SECONDS` | `0` | Wall-clock time a Transcribe job sits `QUEUED` before it starts. On AWS a job is queued only if the request opted in through `JobExecutionSettings.AllowDeferredExecution` and the account is at its concurrent job limit, so the default is `0` and `StartTranscriptionJob` returns `IN_PROGRESS` as it does on AWS. Raise it to exercise a caller that handles `QUEUED`. Also settable through `/_ministack/config` |
| `TRANSLATE_JOB_RUN_SECONDS` | `2` | Wall-clock time a Translate batch job spends between `SUBMITTED` and `COMPLETED`. Set `0` to complete immediately, or raise it when a test needs to observe `IN_PROGRESS` or stop a running job. Also settable without a restart by posting `{"translate._JOB_RUN_SECONDS": 10}` to `/_ministack/config` |
### API Gateway HTTP proxy execution model
diff --git a/ministack/app.py b/ministack/app.py
index 6d625852..9092a492 100644
--- a/ministack/app.py
+++ b/ministack/app.py
@@ -197,12 +197,12 @@ _NON_S3_VHOST_NAMES = frozenset(
)
from ministack.core import container_reaper
-from ministack.core.aws_credentials import (
+from ministack.core.concurrency import spawn_background
+from ministack.core.hypercorn_compat import install as _install_hypercorn_compat
+from ministack.core.iam_evaluator import (
AmbiguousAccessKeyError,
find_iam_access_key_account,
)
-from ministack.core.concurrency import spawn_background
-from ministack.core.hypercorn_compat import install as _install_hypercorn_compat
from ministack.core.persistence import PERSIST_STATE, load_state, save_all
from ministack.core.responses import (
_12_DIGIT_RE,
@@ -1246,7 +1246,6 @@ async def _handle_admin_config_request(path: str, method: str, body: bytes):
"stepfunctions._SFN_WAIT_SCALE",
"translate._JOB_RUN_SECONDS",
"transcribe._JOB_RUN_SECONDS",
- "transcribe._JOB_QUEUE_SECONDS",
"lambda_svc.LAMBDA_EXECUTOR",
"cloudtrail._recording_enabled",
"alb.TARGET_CONNECT_TIMEOUT",
@@ -1272,7 +1271,6 @@ async def _handle_admin_config_request(path: str, method: str, body: bytes):
"stepfunctions._SFN_WAIT_SCALE",
"translate._JOB_RUN_SECONDS",
"transcribe._JOB_RUN_SECONDS",
- "transcribe._JOB_QUEUE_SECONDS",
):
try:
float_value = float(value)
@@ -2191,6 +2189,8 @@ async def _dispatch_service_request(
from ministack.core.iam_actions import (
access_denied_response,
dynamodb_resource_arns,
+ dynamodb_service_context,
+ eventbridge_resource_arns,
extract_iam_action,
extract_resource_arn,
)
@@ -2203,30 +2203,12 @@ async def _dispatch_service_request(
resource_arn = extract_resource_arn(
service, method, path, headers, body, routing_params, region, get_account_id()
)
- service_context = None
- if service == "dynamodb":
- try:
- dynamodb_body = json.loads(body or b"{}")
- except (json.JSONDecodeError, TypeError, UnicodeDecodeError):
- dynamodb_body = {}
- projection = dynamodb_body.get("ProjectionExpression")
- names = dynamodb_body.get("ExpressionAttributeNames") or {}
- if projection:
- attributes = [
- names.get(part.strip(), part.strip())
- for part in projection.split(",")
- ]
- service_context = {
- "dynamodb:Attributes": attributes,
- "dynamodb:Select": dynamodb_body.get(
- "Select", "SPECIFIC_ATTRIBUTES"
- ),
- }
- enforce_kwargs = {"resource_arn": resource_arn}
- if service_context is not None:
- enforce_kwargs["service_context"] = service_context
+ service_context = (
+ dynamodb_service_context(body) if service == "dynamodb" else None
+ )
denied = enforce(
- access_key, iam_action, service, region, **enforce_kwargs
+ access_key, iam_action, service, region,
+ resource_arn=resource_arn, service_context=service_context,
)
# A copy also reads its source, a batch delete is one check per
# key, an attributes call is a pair, a governance bypass its own action.
@@ -2251,6 +2233,17 @@ async def _dispatch_service_request(
)
if denied:
break
+ # PutEvents carries one entry per event, and entries may name
+ # different buses: AWS authorizes each against its own bus.
+ if service == "events" and not denied:
+ for extra_arn in eventbridge_resource_arns(
+ body, region, get_account_id())[1:]:
+ denied = enforce(
+ access_key, iam_action, service, region,
+ resource_arn=extra_arn,
+ )
+ if denied:
+ break
if denied:
if isinstance(denied, AuthError):
return access_denied_response(
diff --git a/ministack/core/aws_credentials.py b/ministack/core/aws_credentials.py
deleted file mode 100644
index b6fbdb26..00000000
--- a/ministack/core/aws_credentials.py
+++ /dev/null
@@ -1,240 +0,0 @@
-# Copyright (c) 2026 MiniStack Contributors. SPDX-License-Identifier: MIT
-# Copies or substantial portions, including AI-assisted ports or rewrites, must retain this notice (see LICENSE).
-"""Resolve AWS access keys to the credentials and principals MiniStack issued."""
-
-import hmac
-import os
-import re
-import time
-from dataclasses import dataclass
-
-from ministack.core.responses import get_account_id
-
-_ACCOUNT_ID_RE = re.compile(r"^\d{12}$")
-_SESSION_TOKEN_NOT_CHECKED = object()
-
-
-@dataclass(frozen=True)
-class ResolvedCredential:
- access_key_id: str
- secret_access_key: str
- account_id: str
- principal_arn: str
- principal_id: str
- principal_type: str
- principal_name: str = ""
- session_token: str | None = None
- expiration: float | None = None
- source_access_key_id: str | None = None
-
-
-@dataclass(frozen=True)
-class CredentialResolutionError:
- code: str
- message: str
-
-
-class AmbiguousAccessKeyError(ValueError):
- """Raised when more than one tenant owns the same IAM access key."""
-
-
-def is_root_access_key(access_key_id: str) -> bool:
- """Return whether a key represents MiniStack's configured root caller."""
- configured_key = os.environ.get("AWS_ACCESS_KEY_ID", "test")
- return (
- not access_key_id
- or access_key_id == "test"
- or access_key_id == configured_key
- or bool(_ACCOUNT_ID_RE.fullmatch(access_key_id))
- )
-
-
-def _unknown_access_key() -> CredentialResolutionError:
- return CredentialResolutionError(
- "UnrecognizedClientException",
- "The security token included in the request is invalid.",
- )
-
-
-def _invalid_session_token() -> CredentialResolutionError:
- return CredentialResolutionError(
- "InvalidToken",
- "The provided token is malformed or otherwise invalid.",
- )
-
-
-def _account_from_arn(arn: str) -> str | None:
- parts = arn.split(":")
- if len(parts) > 4 and _ACCOUNT_ID_RE.fullmatch(parts[4]):
- return parts[4]
- return None
-
-
-def _principal_type(arn: str, recorded_type: str) -> str:
- if recorded_type:
- return recorded_type
- if ":assumed-role/" in arn:
- return "AssumedRole"
- if ":user/" in arn:
- return "User"
- if arn.endswith(":root"):
- return "Root"
- return ""
-
-
-def _validate_session_token(expected: str | None, supplied) -> CredentialResolutionError | None:
- if supplied is _SESSION_TOKEN_NOT_CHECKED:
- return None
- supplied = supplied or ""
- if expected is None:
- return _invalid_session_token() if supplied else None
- if not isinstance(expected, str) or not isinstance(supplied, str) or not hmac.compare_digest(
- expected.encode("utf-8"), supplied.encode("utf-8")
- ):
- return _invalid_session_token()
- return None
-
-
-def find_iam_access_key_account(access_key_id: str) -> str | None:
- """Return the sole account that owns an IAM access key, if known.
-
- AWS access key IDs are globally unique. Duplicate emulator records raise
- instead of selecting either tenant.
- """
- if is_root_access_key(access_key_id):
- return None
-
- from ministack.services import sts as sts_svc
-
- if access_key_id in sts_svc._sessions:
- return None
-
- from ministack.services import iam as iam_svc
-
- accounts = {
- account_id
- for (account_id, stored_key) in iam_svc._access_keys.to_dict()
- if stored_key == access_key_id
- }
- if len(accounts) > 1:
- raise AmbiguousAccessKeyError(access_key_id)
- if accounts:
- return accounts.pop()
- return None
-
-
-def resolve_credential(
- access_key_id: str,
- account_id: str | None = None,
- session_token=_SESSION_TOKEN_NOT_CHECKED,
-) -> ResolvedCredential | CredentialResolutionError:
- """Resolve a root, IAM, or STS key in an explicit account.
-
- Passing ``session_token`` also validates whether the caller supplied the
- exact token attached to temporary credentials. Omitting it leaves token
- validation to the caller, which keeps policy-only identity lookups usable.
- """
- account_id = account_id or get_account_id()
-
- if is_root_access_key(access_key_id):
- if _ACCOUNT_ID_RE.fullmatch(access_key_id):
- if account_id != access_key_id:
- return _unknown_access_key()
- account_id = access_key_id
- configured_key = os.environ.get("AWS_ACCESS_KEY_ID", "test")
- configured_token = os.environ.get("AWS_SESSION_TOKEN") or None
- numeric_token_supplied = (
- bool(_ACCOUNT_ID_RE.fullmatch(access_key_id))
- and isinstance(session_token, str)
- and bool(session_token)
- )
- expected_token = (
- configured_token
- if access_key_id == configured_key or numeric_token_supplied
- else None
- )
- token_error = _validate_session_token(expected_token, session_token)
- if token_error:
- return token_error
- return ResolvedCredential(
- access_key_id=access_key_id,
- secret_access_key=os.environ.get("AWS_SECRET_ACCESS_KEY", "test"),
- session_token=expected_token,
- account_id=account_id,
- principal_arn=f"arn:aws:iam::{account_id}:root",
- principal_id=account_id,
- principal_type="Root",
- source_access_key_id=access_key_id,
- )
-
- from ministack.services import sts as sts_svc
-
- session = sts_svc._sessions.get(access_key_id)
- if session is not None:
- principal_arn = str(session.get("Arn", ""))
- session_account = session.get("AccountId") or _account_from_arn(principal_arn)
- if session_account and session_account != account_id:
- return _unknown_access_key()
- expiration = session.get("Expiration")
- if isinstance(expiration, (int, float)) and time.time() > expiration:
- return CredentialResolutionError(
- "ExpiredTokenException",
- "The security token included in the request is expired",
- )
- expected_token = session.get("SessionToken")
- token_error = _validate_session_token(expected_token, session_token)
- if token_error:
- return token_error
- secret = session.get("SecretAccessKey")
- principal_type = _principal_type(principal_arn, session.get("PrincipalType", ""))
- if not secret or not principal_type:
- return _unknown_access_key()
- principal_name = ""
- if principal_type == "User" and ":user/" in principal_arn:
- principal_name = str(session.get("PrincipalName") or "")
- if not principal_name:
- principal_name = principal_arn.rsplit("/", 1)[-1]
- return ResolvedCredential(
- access_key_id=access_key_id,
- secret_access_key=secret,
- session_token=expected_token,
- expiration=expiration if isinstance(expiration, (int, float)) else None,
- account_id=session_account or account_id,
- principal_arn=principal_arn,
- principal_id=str(session.get("UserId", "")),
- principal_type=principal_type,
- principal_name=principal_name,
- source_access_key_id=session.get("SourceAccessKeyId"),
- )
-
- from ministack.services import iam as iam_svc
-
- key_record = iam_svc._access_keys.get_scoped(account_id, None, access_key_id)
- if key_record is None:
- return _unknown_access_key()
- if key_record.get("Status") != "Active":
- return CredentialResolutionError(
- "InvalidClientTokenId",
- "The security token included in the request is invalid.",
- )
- token_error = _validate_session_token(None, session_token)
- if token_error:
- return token_error
- secret = key_record.get("SecretAccessKey")
- if not secret:
- return _unknown_access_key()
- user_name = str(key_record.get("UserName", ""))
- user = iam_svc._users.get_scoped(account_id, None, user_name) or {}
- principal_arn = str(
- user.get("Arn") or f"arn:aws:iam::{account_id}:user/{user_name}"
- )
- return ResolvedCredential(
- access_key_id=access_key_id,
- secret_access_key=secret,
- account_id=account_id,
- principal_arn=principal_arn,
- principal_id=str(user.get("UserId", "")),
- principal_type="User",
- principal_name=user_name,
- source_access_key_id=access_key_id,
- )
diff --git a/ministack/core/iam_actions.py b/ministack/core/iam_actions.py
index 21e8f1b8..80a5b333 100644
--- a/ministack/core/iam_actions.py
+++ b/ministack/core/iam_actions.py
@@ -920,12 +920,9 @@ def extract_resource_arn(service: str, method: str, path: str,
event_data = json.loads(body or b"{}")
except (json.JSONDecodeError, TypeError, UnicodeDecodeError):
event_data = {}
- entries = event_data.get("Entries") if isinstance(event_data, dict) else None
- if isinstance(entries, list) and entries:
- event_bus = entries[0].get("EventBusName") or "default"
- if event_bus.startswith("arn:"):
- return event_bus
- return f"arn:aws:events:{region}:{account_id}:event-bus/{event_bus}"
+ buses = eventbridge_resource_arns(body, region, account_id)
+ if buses:
+ return buses[0]
name = _safe_json_field(body, "Name") or _safe_json_field(body, "RuleName")
bus = _safe_json_field(body, "EventBusName") or "default"
if name:
@@ -1526,6 +1523,35 @@ def extract_resource_arn(service: str, method: str, path: str,
return "*"
+def eventbridge_resource_arns(body: bytes, region: str, account_id: str) -> list[str]:
+ """Every event-bus ARN a ``PutEvents`` request addresses, in request order.
+
+ AWS authorizes ``events:PutEvents`` per bus, and one call can carry entries
+ for several, so the caller checks each. Entries that are not objects, and
+ bus names that are not strings, are skipped rather than crashing the
+ enforcement path on a malformed request.
+ """
+ try:
+ data = json.loads(body or b"{}")
+ except (json.JSONDecodeError, TypeError, UnicodeDecodeError):
+ return []
+ entries = data.get("Entries") if isinstance(data, dict) else None
+ if not isinstance(entries, list):
+ return []
+ arns = []
+ for entry in entries:
+ if not isinstance(entry, dict):
+ continue
+ bus = entry.get("EventBusName")
+ if not isinstance(bus, str) or not bus:
+ bus = "default"
+ arn = bus if bus.startswith("arn:") else (
+ f"arn:aws:events:{region}:{account_id}:event-bus/{bus}")
+ if arn not in arns:
+ arns.append(arn)
+ return arns
+
+
def dynamodb_resource_arns(body: bytes, region: str, account_id: str) -> list[str]:
"""Return every table ARN addressed by a DynamoDB JSON request."""
try:
@@ -1549,6 +1575,53 @@ def dynamodb_resource_arns(body: bytes, region: str, account_id: str) -> list[st
]
+def dynamodb_service_context(body: bytes) -> dict:
+ """The DynamoDB condition keys a request carries.
+
+ ``dynamodb:Attributes`` is the list of *top-level* attributes the request
+ names: AWS resolves a ``ProjectionExpression`` of ``"Name, Address.City"``
+ to ``["Name", "Address"]``, and a placeholder is substituted per path
+ segment, not on the whole path. The key is omitted when the request names
+ no attributes, which is how AWS evaluates it ("evaluated only on the
+ attributes specified in the request").
+
+ ``dynamodb:Select`` always has a value on AWS for the operations that
+ return attributes (``ALL_ATTRIBUTES`` unless the request says otherwise),
+ so it is set on every request: leaving it unresolved lets a policy that
+ conditions on it with ``StringEqualsIfExists`` pass a request AWS refuses.
+ """
+ try:
+ data = json.loads(body or b"{}")
+ except (json.JSONDecodeError, TypeError, UnicodeDecodeError):
+ return {}
+ if not isinstance(data, dict):
+ return {}
+ names = data.get("ExpressionAttributeNames")
+ names = names if isinstance(names, dict) else {}
+
+ attributes = []
+ projection = data.get("ProjectionExpression")
+ if isinstance(projection, str):
+ for part in projection.split(","):
+ # Top level only: the path's first segment, before any "." or "[".
+ head = re.split(r"[.\[]", part.strip(), maxsplit=1)[0].strip()
+ if head:
+ attributes.append(names.get(head, head))
+ # AttributesToGet is the legacy form of the same projection.
+ legacy = data.get("AttributesToGet")
+ if isinstance(legacy, list):
+ attributes.extend(a for a in legacy if isinstance(a, str) and a)
+
+ context = {}
+ if attributes:
+ context["dynamodb:Attributes"] = attributes
+ select = data.get("Select")
+ if not isinstance(select, str) or not select:
+ select = "SPECIFIC_ATTRIBUTES" if attributes else "ALL_ATTRIBUTES"
+ context["dynamodb:Select"] = select
+ return context
+
+
def access_denied_response(service: str, action: str, principal_arn: str,
request_id: str, *, error_code: str = "",
message: str = "", headers: dict | None = None) -> tuple:
diff --git a/ministack/core/iam_evaluator.py b/ministack/core/iam_evaluator.py
index 2b1e1382..1d70487b 100644
--- a/ministack/core/iam_evaluator.py
+++ b/ministack/core/iam_evaluator.py
@@ -12,19 +12,16 @@ Evaluation order:
"""
import datetime as _dt
+import hmac
import ipaddress
import json
import logging
+import os
import re
+import time
from dataclasses import dataclass, field
from typing import Any
-from ministack.core.aws_credentials import (
- CredentialResolutionError,
- is_root_access_key,
- resolve_credential,
-)
-
logger = logging.getLogger("ministack")
@@ -637,6 +634,242 @@ def _resolve_managed_policy_document(policy_arn: str,
return None
+# ---------------------------------------------------------------------------
+# Access-key resolution — the credentials and principals MiniStack issued
+# ---------------------------------------------------------------------------
+
+_ACCOUNT_ID_RE = re.compile(r"^\d{12}$")
+_SESSION_TOKEN_NOT_CHECKED = object()
+
+
+@dataclass(frozen=True)
+class ResolvedCredential:
+ access_key_id: str
+ secret_access_key: str
+ account_id: str
+ principal_arn: str
+ principal_id: str
+ principal_type: str
+ principal_name: str = ""
+ session_token: str | None = None
+ expiration: float | None = None
+ source_access_key_id: str | None = None
+
+
+@dataclass(frozen=True)
+class CredentialResolutionError:
+ code: str
+ message: str
+
+
+class AmbiguousAccessKeyError(ValueError):
+ """Raised when more than one tenant owns the same IAM access key."""
+
+
+def is_root_access_key(access_key_id: str) -> bool:
+ """Return whether a key represents MiniStack's configured root caller."""
+ configured_key = os.environ.get("AWS_ACCESS_KEY_ID", "test")
+ return (
+ not access_key_id
+ or access_key_id == "test"
+ or access_key_id == configured_key
+ or bool(_ACCOUNT_ID_RE.fullmatch(access_key_id))
+ )
+
+
+def _unknown_access_key() -> CredentialResolutionError:
+ return CredentialResolutionError(
+ "UnrecognizedClientException",
+ "The security token included in the request is invalid.",
+ )
+
+
+def _invalid_session_token() -> CredentialResolutionError:
+ return CredentialResolutionError(
+ "InvalidToken",
+ "The provided token is malformed or otherwise invalid.",
+ )
+
+
+def _account_from_arn(arn: str) -> str | None:
+ parts = arn.split(":")
+ if len(parts) > 4 and _ACCOUNT_ID_RE.fullmatch(parts[4]):
+ return parts[4]
+ return None
+
+
+def _principal_type(arn: str, recorded_type: str) -> str:
+ if recorded_type:
+ return recorded_type
+ if ":assumed-role/" in arn:
+ return "AssumedRole"
+ if ":user/" in arn:
+ return "User"
+ if arn.endswith(":root"):
+ return "Root"
+ return ""
+
+
+def _validate_session_token(expected: str | None, supplied) -> CredentialResolutionError | None:
+ if supplied is _SESSION_TOKEN_NOT_CHECKED:
+ return None
+ supplied = supplied or ""
+ if expected is None:
+ return _invalid_session_token() if supplied else None
+ if not isinstance(expected, str) or not isinstance(supplied, str) or not hmac.compare_digest(
+ expected.encode("utf-8"), supplied.encode("utf-8")
+ ):
+ return _invalid_session_token()
+ return None
+
+
+def find_iam_access_key_account(access_key_id: str) -> str | None:
+ """Return the sole account that owns an IAM access key, if known.
+
+ AWS access key IDs are globally unique. Duplicate emulator records raise
+ instead of selecting either tenant.
+ """
+ if is_root_access_key(access_key_id):
+ return None
+
+ from ministack.services import sts as sts_svc
+
+ if access_key_id in sts_svc._sessions:
+ return None
+
+ from ministack.services import iam as iam_svc
+
+ accounts = {
+ account_id
+ for (account_id, stored_key) in iam_svc._access_keys.to_dict()
+ if stored_key == access_key_id
+ }
+ if len(accounts) > 1:
+ raise AmbiguousAccessKeyError(access_key_id)
+ if accounts:
+ return accounts.pop()
+ return None
+
+
+def resolve_credential(
+ access_key_id: str,
+ account_id: str | None = None,
+ session_token=_SESSION_TOKEN_NOT_CHECKED,
+) -> ResolvedCredential | CredentialResolutionError:
+ """Resolve a root, IAM, or STS key in an explicit account.
+
+ Passing ``session_token`` also validates whether the caller supplied the
+ exact token attached to temporary credentials. Omitting it leaves token
+ validation to the caller, which keeps policy-only identity lookups usable.
+ """
+ from ministack.core.responses import get_account_id
+
+ account_id = account_id or get_account_id()
+
+ if is_root_access_key(access_key_id):
+ if _ACCOUNT_ID_RE.fullmatch(access_key_id):
+ if account_id != access_key_id:
+ return _unknown_access_key()
+ account_id = access_key_id
+ configured_key = os.environ.get("AWS_ACCESS_KEY_ID", "test")
+ configured_token = os.environ.get("AWS_SESSION_TOKEN") or None
+ numeric_token_supplied = (
+ bool(_ACCOUNT_ID_RE.fullmatch(access_key_id))
+ and isinstance(session_token, str)
+ and bool(session_token)
+ )
+ expected_token = (
+ configured_token
+ if access_key_id == configured_key or numeric_token_supplied
+ else None
+ )
+ token_error = _validate_session_token(expected_token, session_token)
+ if token_error:
+ return token_error
+ return ResolvedCredential(
+ access_key_id=access_key_id,
+ secret_access_key=os.environ.get("AWS_SECRET_ACCESS_KEY", "test"),
+ session_token=expected_token,
+ account_id=account_id,
+ principal_arn=f"arn:aws:iam::{account_id}:root",
+ principal_id=account_id,
+ principal_type="Root",
+ source_access_key_id=access_key_id,
+ )
+
+ from ministack.services import sts as sts_svc
+
+ session = sts_svc._sessions.get(access_key_id)
+ if session is not None:
+ principal_arn = str(session.get("Arn", ""))
+ session_account = session.get("AccountId") or _account_from_arn(principal_arn)
+ if session_account and session_account != account_id:
+ return _unknown_access_key()
+ expiration = session.get("Expiration")
+ if isinstance(expiration, (int, float)) and time.time() > expiration:
+ return CredentialResolutionError(
+ "ExpiredTokenException",
+ "The security token included in the request is expired",
+ )
+ expected_token = session.get("SessionToken")
+ token_error = _validate_session_token(expected_token, session_token)
+ if token_error:
+ return token_error
+ secret = session.get("SecretAccessKey")
+ principal_type = _principal_type(principal_arn, session.get("PrincipalType", ""))
+ if not secret or not principal_type:
+ return _unknown_access_key()
+ principal_name = ""
+ if principal_type == "User" and ":user/" in principal_arn:
+ principal_name = str(session.get("PrincipalName") or "")
+ if not principal_name:
+ principal_name = principal_arn.rsplit("/", 1)[-1]
+ return ResolvedCredential(
+ access_key_id=access_key_id,
+ secret_access_key=secret,
+ session_token=expected_token,
+ expiration=expiration if isinstance(expiration, (int, float)) else None,
+ account_id=session_account or account_id,
+ principal_arn=principal_arn,
+ principal_id=str(session.get("UserId", "")),
+ principal_type=principal_type,
+ principal_name=principal_name,
+ source_access_key_id=session.get("SourceAccessKeyId"),
+ )
+
+ from ministack.services import iam as iam_svc
+
+ key_record = iam_svc._access_keys.get_scoped(account_id, None, access_key_id)
+ if key_record is None:
+ return _unknown_access_key()
+ if key_record.get("Status") != "Active":
+ return CredentialResolutionError(
+ "InvalidClientTokenId",
+ "The security token included in the request is invalid.",
+ )
+ token_error = _validate_session_token(None, session_token)
+ if token_error:
+ return token_error
+ secret = key_record.get("SecretAccessKey")
+ if not secret:
+ return _unknown_access_key()
+ user_name = str(key_record.get("UserName", ""))
+ user = iam_svc._users.get_scoped(account_id, None, user_name) or {}
+ principal_arn = str(
+ user.get("Arn") or f"arn:aws:iam::{account_id}:user/{user_name}"
+ )
+ return ResolvedCredential(
+ access_key_id=access_key_id,
+ secret_access_key=secret,
+ account_id=account_id,
+ principal_arn=principal_arn,
+ principal_id=str(user.get("UserId", "")),
+ principal_type="User",
+ principal_name=user_name,
+ source_access_key_id=access_key_id,
+ )
+
+
def _is_root_key(access_key_id: str) -> bool:
"""Keys treated as root: empty, ``test``, configured, or account IDs."""
return is_root_access_key(access_key_id)
diff --git a/ministack/core/lambda_runtime.py b/ministack/core/lambda_runtime.py
index ac57f94d..5c6ac83e 100644
--- a/ministack/core/lambda_runtime.py
+++ b/ministack/core/lambda_runtime.py
@@ -138,23 +138,20 @@ _LOCAL_WORKER_TTL = float(os.environ.get("LAMBDA_WARM_TTL_SECONDS", "300"))
# because both the worker script and lambda_svc's one-shot wrappers embed it.
INVOKE_DEPTH_ENV = "_MINISTACK_INVOKE_DEPTH"
INVOKE_DEPTH_HEADER = "X-Ministack-Invoke-Depth"
-# The warm worker's env is fixed at spawn time, so its depth rides in the
-# event and is popped off before the handler sees it — same trick the X-Ray
-# trace header uses.
-INVOKE_DEPTH_EVENT_KEY = "_ministack_invoke_depth"
+# The warm worker's env is fixed at spawn time, so its depth travels in the
+# invocation envelope beside the payload — same channel as the X-Ray trace
+# header and the durable-execution variables below.
# The durable-execution variables change on every invocation of the same
# function, so a pooled worker cannot carry them in its spawn environment
-# either. They ride in the event under this key, as a mapping of the three
-# names below, and both worker bootstraps move them into the environment
-# before the handler runs.
+# either. They travel in the envelope as a mapping of the three names below,
+# and both worker bootstraps move them into the environment before the
+# handler runs.
#
# One declaration for both readers: lambda_svc builds the overlay from the
# durable context under these keys, the bootstraps iterate the names to set
# them and to delete them again. A fourth variable added to only one of the
-# two would be set and never cleared, which is the leak the event carrier
-# exists to avoid.
-DURABLE_CTX_EVENT_KEY = "_ministack_durable_ctx"
+# two would be set and never cleared.
DURABLE_ENV_VARS = {
"AWS_LAMBDA_DURABLE_EXECUTION_ARN": "arn",
"AWS_LAMBDA_DURABLE_CHECKPOINT_TOKEN": "token",
@@ -167,8 +164,6 @@ def _sub_runtime_tokens(script: str) -> str:
return (
script.replace("__DEPTH_ENV__", INVOKE_DEPTH_ENV)
.replace("__DEPTH_HEADER__", INVOKE_DEPTH_HEADER)
- .replace("__DEPTH_EVENT_KEY__", INVOKE_DEPTH_EVENT_KEY)
- .replace("__DURABLE_EVENT_KEY__", DURABLE_CTX_EVENT_KEY)
.replace("__DURABLE_ENV_VARS__", json.dumps(list(DURABLE_ENV_VARS)))
)
@@ -274,26 +269,33 @@ def run():
line = sys.stdin.readline()
if not line:
break
- event = json.loads(line)
- # X-Ray active tracing: ministack injects the per-invocation trace
- # header into the event; pop it into os.environ so the AWS X-Ray SDK
- # can read _X_AMZN_TRACE_ID on import.
- _xray_tid = event.pop("_x_amzn_trace_id", None)
+ # One invocation, one envelope: {"event": <payload>, "ctx": {...}}.
+ # The payload is the caller's and reaches the handler exactly as it
+ # was sent, whatever its JSON type — on AWS the per-invocation values
+ # below are in the execution environment and the context object, never
+ # in the event.
+ _ms_msg = json.loads(line)
+ event = _ms_msg["event"]
+ _ms_ctx = _ms_msg.get("ctx") or {}
+ # X-Ray active tracing: _X_AMZN_TRACE_ID is a reserved environment
+ # variable that "changes with each invocation", so a pooled worker
+ # sets it per call.
+ _xray_tid = _ms_ctx.get("trace_id")
if _xray_tid:
os.environ["_X_AMZN_TRACE_ID"] = _xray_tid
elif "_X_AMZN_TRACE_ID" in os.environ:
del os.environ["_X_AMZN_TRACE_ID"]
# Recursive-loop depth: same per-invocation channel as the trace
# header, read by the before-call handler when this function calls out.
- _ms_depth = event.pop("__DEPTH_EVENT_KEY__", None)
+ _ms_depth = _ms_ctx.get("depth")
if _ms_depth is not None:
os.environ["__DEPTH_ENV__"] = str(_ms_depth)
elif "__DEPTH_ENV__" in os.environ:
del os.environ["__DEPTH_ENV__"]
# Durable execution: the ARN, the checkpoint token and the execution
- # name belong to this invocation only, so they arrive with the event
- # and are dropped again when the next one is not durable.
- _ms_durable = event.pop("__DURABLE_EVENT_KEY__", None) or {}
+ # name belong to this invocation only, and are dropped again when the
+ # next one is not durable.
+ _ms_durable = _ms_ctx.get("durable") or {}
for _ms_var in __DURABLE_ENV_VARS__:
if _ms_var in _ms_durable:
os.environ[_ms_var] = str(_ms_durable[_ms_var])
@@ -306,7 +308,7 @@ def run():
"function_version": os.environ.get("AWS_LAMBDA_FUNCTION_VERSION", "$LATEST"),
"memory_limit_in_mb": init.get("memory", 128),
"invoked_function_arn": init.get("arn", ""),
- "aws_request_id": event.pop("_request_id", ""),
+ "aws_request_id": _ms_ctx.get("request_id", ""),
"log_group_name": os.environ.get("AWS_LAMBDA_LOG_GROUP_NAME", "/aws/lambda/" + _function_name),
"log_stream_name": os.environ.get("AWS_LAMBDA_LOG_STREAM_NAME", ""),
"identity": None,
@@ -938,37 +940,49 @@ rl.on("line", async (line) => {
return;
}
- // Subsequent lines are event invocations
- const event = msg;
+ // Subsequent lines are one invocation each: {"event": <payload>, "ctx": {...}}.
+ // The payload is the caller's and reaches the handler exactly as it was
+ // sent, whatever its JSON type — on AWS the values below live in the
+ // execution environment and the context object, never in the event.
+ const event = msg.event;
+ const ctx = msg.ctx || {};
+ const _msDeadline = Date.now() + Number(ctx.timeout_ms || 300000);
const context = {
- functionName: event._function_name || "",
- memoryLimitInMB: event._memory || "128",
- invokedFunctionArn: event._arn || "",
- awsRequestId: event._request_id || "",
- getRemainingTimeInMillis: () => 300000,
+ functionName: process.env.AWS_LAMBDA_FUNCTION_NAME || "",
+ functionVersion: process.env.AWS_LAMBDA_FUNCTION_VERSION || "$LATEST",
+ memoryLimitInMB: process.env.AWS_LAMBDA_FUNCTION_MEMORY_SIZE || "128",
+ invokedFunctionArn: process.env._LAMBDA_FUNCTION_ARN || "",
+ awsRequestId: ctx.request_id || "",
+ logGroupName: process.env.AWS_LAMBDA_LOG_GROUP_NAME
+ || ("/aws/lambda/" + (process.env.AWS_LAMBDA_FUNCTION_NAME || "")),
+ logStreamName: process.env.AWS_LAMBDA_LOG_STREAM_NAME || "",
+ identity: null,
+ clientContext: null,
+ callbackWaitsForEmptyEventLoop: true,
+ getRemainingTimeInMillis: () => Math.max(0, _msDeadline - Date.now()),
done: () => {},
succeed: () => {},
fail: () => {},
};
- // X-Ray active tracing: ministack injects the per-invocation trace
- // header into the event; promote it to process.env so the AWS X-Ray SDK
- // can read _X_AMZN_TRACE_ID on require().
- if (event._x_amzn_trace_id) {
- process.env._X_AMZN_TRACE_ID = event._x_amzn_trace_id;
+ // X-Ray active tracing: _X_AMZN_TRACE_ID is a reserved environment
+ // variable that "changes with each invocation", so a pooled worker sets
+ // it per call and the AWS X-Ray SDK reads it on require().
+ if (ctx.trace_id) {
+ process.env._X_AMZN_TRACE_ID = ctx.trace_id;
} else if ("_X_AMZN_TRACE_ID" in process.env) {
delete process.env._X_AMZN_TRACE_ID;
}
// Recursive-loop depth rides the same per-invocation channel, and the
// bundled Lambda stub reads it back off process.env when the handler
// invokes another function.
- if (event.__DEPTH_EVENT_KEY__ !== undefined) {
- process.env.__DEPTH_ENV__ = String(event.__DEPTH_EVENT_KEY__);
+ if (ctx.depth !== undefined && ctx.depth !== null) {
+ process.env.__DEPTH_ENV__ = String(ctx.depth);
} else if ("__DEPTH_ENV__" in process.env) {
delete process.env.__DEPTH_ENV__;
}
- // Durable execution: per-invocation values, so they ride in the event and
- // are dropped again when the next invocation is not durable.
- const _msDurable = event.__DURABLE_EVENT_KEY__ || {};
+ // Durable execution: per-invocation values, dropped again when the next
+ // invocation is not durable.
+ const _msDurable = ctx.durable || {};
for (const _msVar of __DURABLE_ENV_VARS__) {
if (_msVar in _msDurable) {
process.env[_msVar] = String(_msDurable[_msVar]);
@@ -976,13 +990,6 @@ rl.on("line", async (line) => {
delete process.env[_msVar];
}
}
- delete event.__DURABLE_EVENT_KEY__;
- delete event.__DEPTH_EVENT_KEY__;
- delete event._x_amzn_trace_id;
- delete event._request_id;
- delete event._function_name;
- delete event._memory;
- delete event._arn;
try {
let settled = false;
@@ -1320,7 +1327,15 @@ class Worker:
time.sleep(0.001)
return "\n".join(lines)
- def invoke(self, event: dict, request_id: str) -> dict:
+ def invoke(self, event, request_id: str, *, trace_id: str | None = None,
+ depth=None, durable: dict | None = None) -> dict:
+ """Run one invocation on this worker.
+
+ The per-invocation values travel beside the payload, not inside it:
+ on AWS they are the execution environment (`_X_AMZN_TRACE_ID` is
+ reserved and "changes with each invocation") and the context object,
+ so the handler's event is the caller's payload of any JSON type.
+ """
with self._lock:
cold = self._cold
@@ -1332,12 +1347,25 @@ class Worker:
cold = False
timeout = self.config.get("Timeout", 30)
- event["_request_id"] = request_id
+ # Only what genuinely differs per invocation. The function's own
+ # identity (name, version, memory, ARN, log group and stream) is
+ # already in the worker's environment, where AWS puts it, and both
+ # bootstraps build their context object from there.
+ envelope = {
+ "event": event,
+ "ctx": {
+ "request_id": request_id,
+ "timeout_ms": int(float(timeout) * 1000),
+ "trace_id": trace_id,
+ "depth": depth,
+ "durable": durable or {},
+ },
+ }
result_box: list = []
def _read_response():
try:
- self._proc.stdin.write(json.dumps(event) + "\n")
+ self._proc.stdin.write(json.dumps(envelope) + "\n")
self._proc.stdin.flush()
for _ in range(200):
response_line = self._proc.stdout.readline()
@@ -1661,8 +1689,13 @@ def reset():
# ---------------------------------------------------------------------------
# Seconds the bootstrap binary gets to complete its cold start and issue its
-# first GET /runtime/invocation/next before we give up on the environment.
-_PROVIDED_INIT_TIMEOUT = float(os.environ.get("LAMBDA_PROVIDED_INIT_TIMEOUT", "20"))
+# first GET /runtime/invocation/next. AWS fixes this: "The Init phase is
+# limited to 10 seconds. If all three tasks do not complete within 10 seconds,
+# Lambda retries the Init phase at the time of the first function invocation
+# with the configured function timeout" (lambda-runtime-environment.html), so
+# the number is not ours to tune — and blowing it is not a failed invocation,
+# it re-runs init under the function's own Timeout (see _spawn).
+_PROVIDED_INIT_TIMEOUT = 10.0
# How often init/invocation waits check whether the bootstrap is still alive.
_PROVIDED_POLL = 0.05
@@ -1675,6 +1708,18 @@ _PROVIDED_LOG_MAX_LINES = 10000
_INVOCATION_RESULT_RE = re.compile(r"/runtime/invocation/([^/]+)/(response|error)/?$")
+class ProvidedInitTimeout(RuntimeError):
+ """The bootstrap started but did not reach the Runtime API in time."""
+
+
+class ProvidedRuntimeError(RuntimeError):
+ """A provided-runtime failure that carries the error type AWS reports."""
+
+ def __init__(self, message: str, error_type: str):
+ super().__init__(message)
+ self.error_type = error_type
+
+
class ProvidedWorker(Worker):
"""A reusable execution environment for a ``provided.*`` Lambda.
@@ -1859,8 +1904,12 @@ class ProvidedWorker(Worker):
return
request_id, kind = match.group(1), match.group(2)
if not worker._record_result(generation, request_id, kind, parsed):
+ # The shape AWS's own Runtime Interface Emulator renders:
+ # 400 with {"errorMessage": "Invalid request ID",
+ # "errorType": "InvalidRequestID"} (RIE
+ # internal/lambda/rapi/rendering/render_error.go).
self._respond(400, json.dumps({
- "errorMessage": f"Invalid request ID: {request_id}",
+ "errorMessage": "Invalid request ID",
"errorType": "InvalidRequestID",
}).encode())
return
@@ -1908,7 +1957,7 @@ class ProvidedWorker(Worker):
# -- Lifecycle --------------------------------------------------------
- def _spawn(self):
+ def _spawn(self, init_timeout: float | None = None):
import socketserver
from ministack.services.lambda_svc import (
@@ -1923,8 +1972,14 @@ class ProvidedWorker(Worker):
code_dir = _provided_runtime_code_dir(self.code_zip)
bootstrap_path = os.path.join(code_dir, "bootstrap")
- if not os.path.exists(bootstrap_path):
- raise RuntimeError("no bootstrap binary found")
+ if not os.path.exists(bootstrap_path) or not os.access(bootstrap_path, os.X_OK):
+ # "If the bootstrap file doesn't exist or isn't executable, your
+ # function returns a Runtime.InvalidEntrypoint error upon
+ # invocation" (runtimes-custom.html).
+ raise ProvidedRuntimeError(
+ "No bootstrap binary found in the deployment package.",
+ "Runtime.InvalidEntrypoint",
+ )
pending: queue.Queue = queue.Queue(maxsize=1)
log_queue: queue.Queue = queue.Queue(maxsize=_PROVIDED_LOG_MAX_LINES)
@@ -1969,14 +2024,22 @@ class ProvidedWorker(Worker):
# elsewhere, or the child inherits the open write fd and execve fails
# with ETXTBSY (#1051).
with _provided_code_lock:
- self._proc = subprocess.Popen(
- [bootstrap_path],
- cwd=code_dir,
- env=proc_env,
- stdin=subprocess.DEVNULL,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT,
- )
+ try:
+ self._proc = subprocess.Popen(
+ [bootstrap_path],
+ cwd=code_dir,
+ env=proc_env,
+ stdin=subprocess.DEVNULL,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ )
+ except OSError as exc:
+ # A bootstrap the host cannot execute is the same class of
+ # failure as a missing one, and AWS names it the same way.
+ raise ProvidedRuntimeError(
+ f"Couldn't execute the bootstrap binary: {exc}",
+ "Runtime.InvalidEntrypoint",
+ ) from exc
self._log_thread = threading.Thread(
target=self._pump_output, args=(self._proc.stdout, log_queue),
@@ -1984,7 +2047,7 @@ class ProvidedWorker(Worker):
self._log_thread.start()
self._stderr_thread = self._log_thread
- self._await_init()
+ self._await_init(init_timeout)
except BaseException:
# Never leave a half-built environment behind.
self._teardown()
@@ -1994,14 +2057,16 @@ class ProvidedWorker(Worker):
logger.info("Lambda provided-runtime environment spawned for %s (cold start)",
self.func_name)
- def _await_init(self) -> None:
+ def _await_init(self, init_timeout: float | None = None) -> None:
"""Wait for the bootstrap to reach /next, bounded and interruptible.
Returns as soon as init settles: a first poll, an /init/error POST, or
- the process exiting. Only a live but silent binary waits out
- ``_PROVIDED_INIT_TIMEOUT``.
+ the process exiting. Only a live but silent binary waits out the
+ deadline, and that raises ``ProvidedInitTimeout`` so the caller can do
+ what AWS does — run init again under the function's own timeout.
"""
- deadline = time.monotonic() + _PROVIDED_INIT_TIMEOUT
+ budget = _PROVIDED_INIT_TIMEOUT if init_timeout is None else init_timeout
+ deadline = time.monotonic() + budget
while not self._init_settled.wait(_PROVIDED_POLL):
exit_code = self._proc.poll() if self._proc is not None else -1
if exit_code is not None:
@@ -2009,9 +2074,9 @@ class ProvidedWorker(Worker):
f"bootstrap exited during init with code {exit_code}"
f"{self._init_log_suffix()}")
if time.monotonic() >= deadline:
- raise RuntimeError(
+ raise ProvidedInitTimeout(
f"bootstrap did not reach the Runtime API within "
- f"{_PROVIDED_INIT_TIMEOUT:g}s{self._init_log_suffix()}")
+ f"{budget:g}s{self._init_log_suffix()}")
if self._init_error is not None:
raise RuntimeError(f"init error: {self._init_error}")
@@ -2029,12 +2094,23 @@ class ProvidedWorker(Worker):
"""
with self._lock:
cold = False
+ timeout = self.config.get("Timeout", 30)
if self._proc is None or self._proc.poll() is not None:
- self._spawn()
+ try:
+ self._spawn()
+ except ProvidedInitTimeout:
+ # AWS: an Init phase that overruns its 10 seconds is not a
+ # failed invocation — "Lambda retries the Init phase at the
+ # time of the first function invocation with the configured
+ # function timeout" (lambda-runtime-environment.html).
+ logger.info(
+ "Lambda %s: init exceeded %gs; re-running it under the "
+ "function timeout (%ss)",
+ self.func_name, _PROVIDED_INIT_TIMEOUT, timeout)
+ self._spawn(init_timeout=timeout)
cold = True
self._cold = False
- timeout = self.config.get("Timeout", 30)
generation = self._generation
with self._state_lock:
diff --git a/ministack/services/apigateway.py b/ministack/services/apigateway.py
index 20bb8034..e8aa80e1 100644
--- a/ministack/services/apigateway.py
+++ b/ministack/services/apigateway.py
@@ -391,6 +391,10 @@ async def handle_request(method, path, headers, body, query_params):
if method == "PATCH":
return _update_api(api_id, data)
+ # /v2/apis/{apiId}/cors
+ if api_id and sub == "cors" and method == "DELETE":
+ return _delete_cors_configuration(api_id)
+
# /v2/apis/{apiId}/routes[/{routeId}[/routeresponses[/{routeResponseId}]]]
if api_id and sub == "routes":
rr_segment = parts[5] if len(parts) > 5 else None
@@ -1843,16 +1847,24 @@ def _update_api(api_id, data):
"description"):
if k in data:
api[k] = data[k]
- # UpdateApi takes the whole CORS configuration, so an empty one removes it
- # rather than storing a blank block the reader would have to interpret.
- if "corsConfiguration" in data:
- if data["corsConfiguration"]:
- api["corsConfiguration"] = data["corsConfiguration"]
- else:
- api.pop("corsConfiguration", None)
+ # A CORS configuration is replaced wholesale, never removed here: the API
+ # has DeleteCorsConfiguration for that, so an empty one is nothing to
+ # apply. A caller that means to remove it deletes it.
+ if data.get("corsConfiguration"):
+ api["corsConfiguration"] = data["corsConfiguration"]
return _apigw_response(api)
+def _delete_cors_configuration(api_id):
+ """DeleteCorsConfiguration: the operation that removes an HTTP API's CORS
+ configuration (UpdateApi replaces it, it cannot clear it)."""
+ api = _apis.get(api_id)
+ if not api:
+ return _apigw_error("NotFoundException", f"API {api_id} not found", 404)
+ api.pop("corsConfiguration", None)
+ return 204, {}, b""
+
+
# ---- Control plane: Routes ----
def _create_route(api_id, data):
diff --git a/ministack/services/cloudformation/provisioners.py b/ministack/services/cloudformation/provisioners.py
index cd3b1c5f..0217f1db 100644
--- a/ministack/services/cloudformation/provisioners.py
+++ b/ministack/services/cloudformation/provisioners.py
@@ -382,6 +382,31 @@ def _cf_oac_delete(physical_id, props):
_cf._oacs.pop(physical_id, None)
+def _cf_function_live_body(cfg, code):
+ """The snapshot PublishFunction freezes for the LIVE stage."""
+ return {
+ "comment": cfg["comment"],
+ "runtime": cfg["runtime"],
+ "kvs_arns": list(cfg["kvs_arns"]),
+ "code": code,
+ }
+
+
+def _cf_function_tags(name, props):
+ """`Tags` on the type is an array of Tag and is "No interruption" on an
+ update, so the template's list replaces whatever the function carried."""
+ tags = [
+ {"Key": str(t.get("Key", "")), "Value": str(t.get("Value", ""))}
+ for t in (props.get("Tags") or [])
+ if isinstance(t, dict) and t.get("Key")
+ ]
+ arn = _cf._func_arn(name)
+ if tags:
+ _cf._tags[arn] = tags
+ else:
+ _cf._tags.pop(arn, None)
+
+
def _cf_function_body(name, props):
"""The FunctionConfig, source and AutoPublish flag a template carries."""
cfg_el = _cf_props_to_element("FunctionConfig", props.get("FunctionConfig") or {})
@@ -426,7 +451,9 @@ def _cf_function_create(logical_id, props, stack_name):
"last_modified_live": now if auto_publish else None,
"dev_etag": dev_etag,
"live_etag": new_uuid() if auto_publish else None,
+ "live_body": _cf_function_live_body(cfg, code) if auto_publish else None,
}
+ _cf_function_tags(name, props)
# The reference lists only FunctionARN and FunctionMetadata.FunctionARN as
# GetAtt attributes — no Stage.
arn = _cf._func_arn(name)
@@ -442,13 +469,12 @@ def _cf_function_update(physical_id, old_props, new_props, stack_name, logical_i
removed, while everything else keeps the physical id — the function name,
which is what its ARN is built from — and its creation time.
- The stage handling follows MiniStack's own UpdateFunction model, which the
- reference does not describe: the new source lands in DEVELOPMENT and the
- LIVE stage is dropped, because the record holds one body for both stages
- and an unpublished change must not be served as published. `AutoPublish:
- true` republishes right after, which is what the reference means by
- "updating the AWS::CloudFront::Function resource with the AutoPublish
- property set to true".
+ The new source lands in DEVELOPMENT and the published body keeps serving
+ LIVE, as UpdateFunction does ("The changes are made only to the version of
+ the function that is in the DEVELOPMENT stage"). `AutoPublish: true`
+ republishes right after, which is what the reference means by "updating the
+ AWS::CloudFront::Function resource with the AutoPublish property set to
+ true".
"""
name = new_props.get("Name") or _physical_name(stack_name,
logical_id or physical_id, max_len=64)
@@ -469,14 +495,18 @@ def _cf_function_update(physical_id, old_props, new_props, stack_name, logical_i
record["code"] = code
record["last_modified_dev"] = now
record["dev_etag"] = new_uuid()
- record["last_modified_live"] = now if auto_publish else None
- record["live_etag"] = new_uuid() if auto_publish else None
+ if auto_publish:
+ record["last_modified_live"] = now
+ record["live_etag"] = new_uuid()
+ record["live_body"] = _cf_function_live_body(cfg, code)
+ _cf_function_tags(name, new_props)
arn = _cf._func_arn(name)
return physical_id, {"FunctionARN": arn, "FunctionMetadata.FunctionARN": arn}
def _cf_function_delete(physical_id, props):
_cf._functions.pop(physical_id, None)
+ _cf._tags.pop(_cf._func_arn(physical_id), None)
# ===========================================================================
@@ -672,8 +702,11 @@ def _rename_replacement(physical_id, old_props, new_props, stack_name, logical_i
return None
created = create_fn(logical_id or physical_id, new_props, stack_name)
replaced = created[0] != physical_id or delete_when_id_unchanged
- if current_name is not None and replaced and not _RETAIN_REPLACED.get():
- delete_fn(physical_id, old_props)
+ if current_name is not None and replaced:
+ # Through the shared helper, so the retaining policy has one reader:
+ # a second copy of the check here is the drift _delete_predecessor
+ # exists to prevent.
+ _delete_predecessor(delete_fn, physical_id, old_props)
return created
@@ -7769,10 +7802,17 @@ def _apigw_v2_api_update(physical_id, old_props, new_props, stack_name, logical_
if replaced is not None:
return replaced
payload = _apigw_v2_api_props(new_props, stack_name, logical_id or physical_id)
- payload["corsConfiguration"] = new_props.get("CorsConfiguration") or {}
+ if new_props.get("CorsConfiguration"):
+ payload["corsConfiguration"] = new_props["CorsConfiguration"]
resp = _apigw_v2._update_api(physical_id, payload)
if resp[0] >= 400:
raise ValueError(f"AWS::ApiGatewayV2::Api update failed: {resp[2]!r}")
+ if not new_props.get("CorsConfiguration"):
+ # A dropped property reverts to the create's default, which for CORS
+ # is no configuration at all. UpdateApi replaces a configuration and
+ # cannot clear one — DeleteCorsConfiguration is the call that removes
+ # it — so the removal happens here rather than inside UpdateApi.
+ _apigw_v2._delete_cors_configuration(physical_id)
_reconcile_tag_map(api.setdefault("tags", {}), old_props, new_props)
return physical_id, {"ApiId": physical_id, "ApiEndpoint": api["apiEndpoint"]}
diff --git a/ministack/services/cloudfront.py b/ministack/services/cloudfront.py
index 9b04e85d..a5914511 100644
--- a/ministack/services/cloudfront.py
+++ b/ministack/services/cloudfront.py
@@ -647,18 +647,34 @@ def _resolve_taggable_cloudfront_arn(arn: str):
return arn, None
+def _function_view(fn: dict, stage: str) -> dict:
+ """The function body a stage serves.
+
+ An update lands in DEVELOPMENT only — "The changes are made only to the
+ version of the function that is in the DEVELOPMENT stage. To copy the
+ updates from the DEVELOPMENT stage to LIVE, you must publish the function"
+ — so LIVE keeps serving the body captured at the last publish. A record
+ from before this snapshot existed falls back to the live body.
+ """
+ if stage == "LIVE":
+ return fn.get("live_body") or fn
+ return fn
+
+
def _function_summary_builder(fn: dict, stage: str, status: str, last_modified: str):
+ view = _function_view(fn, stage)
+
def build(root):
fc = SubElement(root, "FunctionConfig")
- SubElement(fc, "Comment").text = fn.get("comment", "")
- kvs_arns = fn.get("kvs_arns", [])
+ SubElement(fc, "Comment").text = view.get("comment", "")
+ kvs_arns = view.get("kvs_arns", [])
kvs = SubElement(fc, "KeyValueStoreAssociations")
SubElement(kvs, "Quantity").text = str(len(kvs_arns))
items_el = SubElement(kvs, "Items")
for arn in kvs_arns:
assoc = SubElement(items_el, "KeyValueStoreAssociation")
SubElement(assoc, "KeyValueStoreARN").text = arn
- SubElement(fc, "Runtime").text = fn["runtime"]
+ SubElement(fc, "Runtime").text = view.get("runtime", fn["runtime"])
md = SubElement(root, "FunctionMetadata")
SubElement(md, "CreatedTime").text = fn["created"]
SubElement(md, "FunctionARN").text = fn["arn"]
@@ -731,6 +747,8 @@ def _cf_create_function(headers, body):
"last_modified_live": None,
"dev_etag": dev_etag,
"live_etag": None,
+ # The body PublishFunction froze; LIVE serves this, not the working copy.
+ "live_body": None,
}
_functions[name] = fn
logger.info("CreateFunction name=%s", name)
@@ -801,7 +819,7 @@ def _cf_get_function(name: str, stage: str):
if not fn["live_etag"]:
return _error("NoSuchFunctionExists", "The specified function does not exist.", 404)
etag = fn["live_etag"]
- code = fn["code"]
+ code = _function_view(fn, "LIVE").get("code", fn["code"])
elif stage == "DEVELOPMENT":
etag = fn["dev_etag"]
code = fn["code"]
@@ -828,6 +846,14 @@ def _cf_publish_function(name: str, headers):
now = _now_iso()
fn["live_etag"] = new_uuid()
fn["last_modified_live"] = now
+ # Publishing copies DEVELOPMENT to LIVE; later updates to the development
+ # body must not reach the stage that is serving traffic.
+ fn["live_body"] = {
+ "comment": fn["comment"],
+ "runtime": fn["runtime"],
+ "kvs_arns": list(fn.get("kvs_arns", [])),
+ "code": fn["code"],
+ }
logger.info("PublishFunction name=%s", name)
lm = fn["last_modified_live"]
@@ -870,8 +896,8 @@ def _cf_update_function(name: str, headers, body):
fn["code"] = code
fn["last_modified_dev"] = now
fn["dev_etag"] = new_uuid()
- fn["live_etag"] = None
- fn["last_modified_live"] = None
+ # LIVE is untouched: an update changes the DEVELOPMENT stage only, and the
+ # published version keeps serving until PublishFunction copies this one.
logger.info("UpdateFunction name=%s", name)
return _xml_response(
diff --git a/ministack/services/ecs.py b/ministack/services/ecs.py
index 5ab74a89..fa434413 100644
--- a/ministack/services/ecs.py
+++ b/ministack/services/ecs.py
@@ -94,6 +94,13 @@ _ecs_reaper_started = False
_ecs_reaper_lock = threading.Lock()
+# A task that is registered but not stopped, in task-lifecycle order. A
+# PENDING or ACTIVATING task already counts against a service's desired
+# capacity, so reconciliation must not launch a second one while its images
+# are still pulling.
+_PRE_STOP_STATUSES = ("PENDING", "ACTIVATING", "RUNNING")
+
+
def _live_container_ids():
"""Container ids still owned by a live task.
@@ -854,7 +861,7 @@ def _reconcile_service_tasks(cluster_name, svc_key):
for arn, t in _tasks.items():
if (t.get("group") == f"service:{svc_name}"
and t.get("clusterArn") == cluster_arn
- and t.get("lastStatus") in ("PENDING", "RUNNING")):
+ and t.get("lastStatus") in _PRE_STOP_STATUSES):
if t.get("taskDefinitionArn") == target_td_arn:
current_tasks.append((arn, t))
else:
@@ -1003,7 +1010,7 @@ def _delete_service(data):
a for a, t in _tasks.items()
if t.get("group") == f"service:{svc_name}"
and t.get("clusterArn") == cluster_arn
- and t.get("lastStatus") in ("PENDING", "RUNNING")
+ and t.get("lastStatus") in _PRE_STOP_STATUSES
]:
_stop_task({"task": task_arn, "cluster": cluster_name, "reason": "Service deleted"})
@@ -1453,6 +1460,32 @@ def _mark_task_stopped(task_arn, task, reason, stop_code, exit_code=None):
return True
+def _mark_task_activating(task_arn, task):
+ """Move a registered task to ACTIVATING, where the image pull happens.
+
+ "ACTIVATING: This is a transition state where Amazon ECS has to perform
+ additional steps after the task is launched but before the task can
+ transition to the RUNNING state. This is the state where Amazon ECS pulls
+ the container images, creates the containers, configures the task
+ networking, registers load balancer target groups" (task-lifecycle). The
+ task sits PENDING only until its worker picks it up.
+ """
+ with resource_lock("ecs-task", task_arn):
+ if not _task_is_active(task_arn, task):
+ return False
+ task["lastStatus"] = "ACTIVATING"
+ task["pullStartedAt"] = task.get("pullStartedAt") or _iso()
+
+ cluster_name = _cluster_name_from_arn(task.get("clusterArn", ""))
+ if cluster_name:
+ # pendingTasksCount / pendingCount are "the number of tasks in the
+ # cluster that are in the PENDING state" per the model, so an
+ # ACTIVATING task leaves those counts rather than joining running.
+ _recount_cluster(cluster_name)
+ _refresh_service_state(cluster_name, task.get("group", ""))
+ return True
+
+
def _mark_task_running(task_arn, task):
with resource_lock("ecs-task", task_arn):
if not _task_is_active(task_arn, task):
@@ -1553,11 +1586,7 @@ def _start_task_worker(task, td, container_overrides, docker_client):
cluster_arn = task["clusterArn"]
launch_type = task.get("launchType", "EC2")
- with resource_lock("ecs-task", task_arn):
- active = _task_is_active(task_arn, task)
- if active:
- task["pullStartedAt"] = task.get("pullStartedAt") or _iso()
- if not active:
+ if not _mark_task_activating(task_arn, task):
return
ecs_network = None
@@ -2323,9 +2352,20 @@ def _ecs_single_resource_tail(ref, resource_type):
def _sanitize(obj):
- """Remove internal keys (prefixed with _) from a dict for API responses."""
+ """Project a stored record onto its wire shape.
+
+ Drops internal keys (prefixed with _) and members with no value: AWS omits
+ a member it has nothing for rather than sending an explicit null, and a
+ task holds placeholders for the timestamps it has not reached yet
+ (startedAt and the pull markers while it is PENDING, stoppedAt until it
+ stops). boto3 discards a null member on the way in, so only a client that
+ reads the wire sees the difference.
+ """
if isinstance(obj, dict):
- return {k: _sanitize(v) for k, v in obj.items() if not k.startswith("_")}
+ return {
+ k: _sanitize(v) for k, v in obj.items()
+ if not k.startswith("_") and v is not None
+ }
if isinstance(obj, list):
return [_sanitize(i) for i in obj]
return obj
diff --git a/ministack/services/lambda_durable.py b/ministack/services/lambda_durable.py
index c8ba786c..f80e069f 100644
--- a/ministack/services/lambda_durable.py
+++ b/ministack/services/lambda_durable.py
@@ -23,6 +23,7 @@ from __future__ import annotations
import base64
import contextvars
import copy
+import hashlib as _hashlib
import json
import secrets
import time
@@ -84,6 +85,27 @@ _resume_thread_started = False
# start.
_callback_index = AccountScopedDict()
+
+def _callback_id(arn: str, op_id: str) -> str:
+ """The CallbackId an execution's CALLBACK operation is handed out under.
+
+ Operation ids come from the client and are deterministic per workflow
+ position, so two executions of the same function reach the same callback
+ with the same operation id. Keying the callback on the operation id alone
+ let the second registration overwrite the first, and the id the first
+ execution had been given then resolved to the second one. The id is ours
+ to mint (the SDK reads it back out of ``CallbackDetails.CallbackId``), so
+ it is derived from the execution ARN as well — unique per execution, and
+ stable across replays and restarts, which a random token would not be.
+
+ The alphabet is the model's (``[A-Za-z0-9+/]+={0,2}``) narrowed to hex:
+ a standard-base64 id can contain ``/``, and ``scope["path"]`` reaches the
+ router already percent-decoded, so such an id would split the URI segment
+ and never resolve.
+ """
+ return _hashlib.sha256(f"{arn}\n{op_id}".encode()).hexdigest()
+
+
# Function-level DurableConfig is stored on the function config in lambda_svc;
# we expose helpers here for serialization parity.
@@ -141,7 +163,15 @@ def restore_state(data):
if op.get("Type") == "CALLBACK" and op.get("Status") == "STARTED":
op_id = op.get("Id")
if op_id:
- _callback_index.set_scoped(account_id, region, op_id, (arn, op_id))
+ _callback_index.set_scoped(
+ account_id, region, _callback_id(arn, op_id), (arn, op_id))
+ # A record written before the id became execution-scoped
+ # carries the old id; keep it resolvable so a callback
+ # handed out before the upgrade still completes.
+ stored = (op.get("CallbackDetails") or {}).get("CallbackId")
+ if stored and stored != _callback_id(arn, op_id):
+ _callback_index.set_scoped(
+ account_id, region, stored, (arn, op_id))
# Re-arm WAIT and CALLBACK timers for executions that were still
# RUNNING when the process went down. Without this, restored
# executions stall forever — timers never fire.
@@ -712,7 +742,8 @@ def _apply_update(rec: dict, upd: dict) -> None:
# resume scheduler can poll them alongside WAIT expiries.
cb_opts = upd.get("CallbackOptions") or {}
if action == "START":
- details["CallbackId"] = op_id # SDK uses Operation.Id as CallbackId
+ # Unique per (execution, operation): see _callback_id.
+ details["CallbackId"] = _callback_id(rec["DurableExecutionArn"], op_id)
timeout_s = cb_opts.get("TimeoutSeconds")
if timeout_s is not None:
details["TimeoutDeadline"] = now + float(timeout_s)
@@ -720,11 +751,14 @@ def _apply_update(rec: dict, upd: dict) -> None:
if hb_s is not None:
details["HeartbeatTimeoutSeconds"] = float(hb_s)
details["HeartbeatDeadline"] = now + float(hb_s)
- # Index so Send*Callback handlers can look us up by the bare id.
- _callback_index[op_id] = (rec["DurableExecutionArn"], op_id)
+ # Index so Send*Callback handlers can look us up by the id.
+ _callback_index[details["CallbackId"]] = (rec["DurableExecutionArn"], op_id)
elif action in ("SUCCEED", "FAIL", "CANCEL"):
- # Callback resolved internally — drop from the live index.
- _callback_index.pop(op_id, None)
+ # Callback resolved internally — drop from the live index. The
+ # stored id is read back rather than recomputed so a record
+ # written before the id became execution-scoped still clears.
+ _callback_index.pop(details.get("CallbackId") or op_id, None)
+ _callback_index.pop(_callback_id(rec["DurableExecutionArn"], op_id), None)
elif op_type == "CONTEXT":
details = existing.setdefault("ContextDetails", {})
if payload is not None and action == "SUCCEED":
@@ -749,7 +783,13 @@ def _apply_update(rec: dict, upd: dict) -> None:
if rec.get("InputPayload"):
details["InputPayload"] = rec["InputPayload"]
- # History event mirror.
+ # History event mirror. CallbackStarted reports the id the operation was
+ # handed out under, which is execution-scoped (see _callback_id), not the
+ # client's operation id.
+ callback_id = ""
+ if op_type == "CALLBACK":
+ callback_id = (existing.get("CallbackDetails") or {}).get("CallbackId") or ""
+
event_type_map = {
("STEP", "START"): ("StepStarted", "StepStartedDetails", {}),
("STEP", "SUCCEED"): ("StepSucceeded", "StepSucceededDetails",
@@ -763,7 +803,7 @@ def _apply_update(rec: dict, upd: dict) -> None:
("WAIT", "CANCEL"): ("WaitCancelled", "WaitCancelledDetails",
{"Error": {"Payload": err or {}, "Truncated": False}}),
("CALLBACK", "START"): ("CallbackStarted", "CallbackStartedDetails",
- {"CallbackId": op_id or ""}),
+ {"CallbackId": callback_id or ""}),
("CALLBACK", "SUCCEED"): ("CallbackSucceeded", "CallbackSucceededDetails",
{"Result": {"Payload": payload or "", "Truncated": False}}),
("CALLBACK", "FAIL"): ("CallbackFailed", "CallbackFailedDetails",
diff --git a/ministack/services/lambda_svc.py b/ministack/services/lambda_svc.py
index 1d27f4fe..d0a8e0be 100644
--- a/ministack/services/lambda_svc.py
+++ b/ministack/services/lambda_svc.py
@@ -53,11 +53,9 @@ from ministack.core import container_reaper
from ministack.core.arn import ArnParseError, parse_arn
from ministack.core.concurrency import run_reentrant
from ministack.core.lambda_runtime import (
- DURABLE_CTX_EVENT_KEY,
DURABLE_ENV_VARS,
INVOKE_DEPTH_BOOTSTRAP,
INVOKE_DEPTH_ENV,
- INVOKE_DEPTH_EVENT_KEY,
INVOKE_DEPTH_HEADER,
acquire_worker,
ensure_spawned,
@@ -4096,6 +4094,14 @@ try {
.filter(Boolean));
const origHttpsRequest = https.request;
https.request = function (input, options, callback) {
+ // Keep the caller's own arguments for the pass-through below: Node's
+ // ClientRequest reads (input, options, cb) positionally and, for a
+ // non-string input, takes cb from the SECOND argument — so replaying a
+ // normalised (input, undefined, callback) would drop the callback and the
+ // handler would never see its response. A copy, not `arguments` itself:
+ // this file is not in strict mode, so `arguments` stays aliased to the
+ // parameters and the normalisation below would rewrite it too.
+ const original = Array.prototype.slice.call(arguments);
if (typeof options === "function") { callback = options; options = undefined; }
let opts;
if (typeof input === "string" || input instanceof URL) {
@@ -4111,7 +4117,7 @@ try {
// port itself: a handler that dials its own TLS sidecar on another port
// of localhost keeps TLS.
if (!PLAIN_HOSTS.has(host) || (rawPort && rawPort !== "443" && rawPort !== EP_PORT)) {
- return origHttpsRequest.call(https, input, options, callback);
+ return origHttpsRequest.apply(https, original);
}
opts.protocol = "http:";
opts.hostname = host;
@@ -4123,8 +4129,9 @@ try {
delete opts._defaultAgent;
return http.request(opts, callback);
};
- https.get = function (input, options, callback) {
- const req = https.request(input, options, callback);
+ https.get = function () {
+ // Same argument shapes as request(), forwarded untouched.
+ const req = https.request.apply(https, arguments);
req.end();
return req;
};
@@ -5079,25 +5086,19 @@ def _execute_function_warm(func: dict, event: dict) -> dict:
# can set ``_X_AMZN_TRACE_ID`` in os.environ before calling the
# handler. Per-invocation, not bake-time, so it can't live in the
# worker's spawn env.
- _xray = _xray_trace_id_for_invocation(config)
- if _xray:
- event["_x_amzn_trace_id"] = _xray
- # Same channel for the recursive-loop depth: the worker's env is
- # fixed at spawn time, so it has to ride in the event. Both worker
- # bootstraps move it to the environment and drop the key before the
- # handler runs. Non-dict payloads have nowhere to carry it, and lose
- # the counter.
- if isinstance(event, dict):
- event[INVOKE_DEPTH_EVENT_KEY] = _invoke_depth.get()
- # Durable executions ride the same channel: the ARN, the checkpoint
- # token and the execution name differ per invocation, which is why
- # they cannot be part of the worker's spawn environment. The
- # bootstrap clears them again when the key is absent, so a worker
- # reused for a non-durable call does not see the previous one's.
- durable = _durable_env_overlay()
- if durable:
- event[DURABLE_CTX_EVENT_KEY] = durable
- result = worker.invoke(event, new_uuid())
+ # The per-invocation values travel beside the payload, never inside
+ # it: on AWS the trace header is a reserved environment variable that
+ # "changes with each invocation", the request id is on the context
+ # object, and the payload the handler receives is the caller's, of
+ # whatever JSON type. A pooled worker's spawn environment is fixed, so
+ # the values ride the envelope and the bootstrap applies them per call.
+ result = worker.invoke(
+ event,
+ new_uuid(),
+ trace_id=_xray_trace_id_for_invocation(config),
+ depth=_invoke_depth.get(),
+ durable=_durable_env_overlay(),
+ )
if result.get("status") == "ok":
return {"body": result.get("result"), "log": result.get("log", "")}
else:
@@ -5207,10 +5208,14 @@ def _execute_function_provided_warm(func: dict, event: dict,
account, region = _account_region_from_function_config(config)
invalidate_worker(func_name, qualifier=qualifier, account=account, region=region)
worker = None # invalidation already removed and reaped the worker
+ # A Python class name is not an AWS error type: a bootstrap that is
+ # missing or cannot be executed is `Runtime.InvalidEntrypoint`, and the
+ # environment failures AWS names Runtime.* carry their own type.
+ error_type = getattr(e, "error_type", "") or "Runtime.Unknown"
# Do not transparently invoke again: the handler may already have
# performed side effects before its environment failed.
return {
- "body": {"errorMessage": str(e), "errorType": type(e).__name__},
+ "body": {"errorMessage": str(e), "errorType": error_type},
"error": True,
"log": "",
}
diff --git a/ministack/services/s3.py b/ministack/services/s3.py
index d4ab88ad..624acf26 100644
--- a/ministack/services/s3.py
+++ b/ministack/services/s3.py
@@ -50,7 +50,7 @@ from xml.sax.saxutils import escape as _esc
from defusedxml.ElementTree import fromstring
from ministack.core.arn import ArnParseError, parse_arn
-from ministack.core.aws_credentials import (
+from ministack.core.iam_evaluator import (
AmbiguousAccessKeyError,
CredentialResolutionError,
find_iam_access_key_account,
@@ -1576,17 +1576,24 @@ def _verify_presigned_sigv4(method, path, headers, query_params):
try:
owner = find_iam_access_key_account(_akid)
except AmbiguousAccessKeyError:
+ # Two tenants holding one key is an emulator-only state; S3 has no
+ # error for it, so it answers as it does for a key it cannot resolve.
return _error(
- "InvalidAccessKeyId", "The AWS Access Key Id is ambiguous.", 403, path
+ "InvalidAccessKeyId",
+ "The AWS Access Key Id you provided does not exist in our records.",
+ 403,
+ path,
)
if owner:
set_request_account_id(owner)
credential = resolve_credential(_akid, get_account_id(), session_token)
if isinstance(credential, CredentialResolutionError):
+ # S3's own error table: ExpiredToken and InvalidToken are 400, while
+ # InvalidAccessKeyId is 403.
if credential.code == "ExpiredTokenException":
- return _error("ExpiredToken", credential.message, 403, path)
+ return _error("ExpiredToken", "The provided token has expired.", 400, path)
if credential.code == "InvalidToken":
- return _error("InvalidToken", credential.message, 403, path)
+ return _error("InvalidToken", credential.message, 400, path)
return _error(
"InvalidAccessKeyId",
"The AWS Access Key Id you provided does not exist in our records.",
diff --git a/ministack/services/sts.py b/ministack/services/sts.py
index 39d84747..339f58dd 100644
--- a/ministack/services/sts.py
+++ b/ministack/services/sts.py
@@ -16,7 +16,7 @@ import time
from urllib.parse import parse_qs
from ministack.core.arn import ArnParseError, parse_arn
-from ministack.core.aws_credentials import CredentialResolutionError, resolve_credential
+from ministack.core.iam_evaluator import CredentialResolutionError, resolve_credential
from ministack.core.responses import get_account_id, json_response, new_uuid
from ministack.core.router import extract_access_key_id
diff --git a/ministack/services/transcribe.py b/ministack/services/transcribe.py
index 56dbe8b3..60284b63 100644
--- a/ministack/services/transcribe.py
+++ b/ministack/services/transcribe.py
@@ -12,9 +12,9 @@ Implemented:
StartTranscriptionJob returns the job already IN_PROGRESS with StartTime set,
as the documented AWS response does. QUEUED is reached on AWS only by a
-request that opted into job queueing while the account is at its concurrent
-job limit; here it is reachable by setting ``TRANSCRIBE_JOB_QUEUE_SECONDS``.
-A background task
+request that opted into job queueing (``JobExecutionSettings.AllowDeferredExecution``)
+while the account is at its concurrent job limit; neither the quota nor the
+opt-in is modelled here, so a job never queues. A background task
walks the job to COMPLETED or FAILED over ``TRANSCRIBE_JOB_RUN_SECONDS``
(0 completes immediately), reads the media
object out of MiniStack's S3 store, and writes a transcript document back to
@@ -84,13 +84,6 @@ logger = logging.getLogger("transcribe")
# non-zero value, tests that just want a result set it to 0.
_JOB_RUN_SECONDS = float(os.environ.get("TRANSCRIBE_JOB_RUN_SECONDS", "2"))
-# How long a job sits QUEUED before it starts. On AWS a job is queued only if
-# the request opted into queueing (JobExecutionSettings.AllowDeferredExecution)
-# and the account is at its concurrent job limit, so the default is 0 and the
-# start response reports IN_PROGRESS. Raise it to exercise a caller that
-# handles QUEUED. Also settable at runtime through /_ministack/config.
-_JOB_QUEUE_SECONDS = float(os.environ.get("TRANSCRIBE_JOB_QUEUE_SECONDS", "0"))
-
# Where transcripts land when the caller supplies no OutputBucketName. Real
# Transcribe uses a service-managed bucket and hands back a presigned URL; the
# local equivalent is a bucket MiniStack owns, so the URI is always fetchable.
@@ -621,13 +614,13 @@ def _live_job(job_name, run_id):
return job
-async def _run_job(job_name, run_id, account_id, region, netloc, queue_seconds):
+async def _run_job(job_name, run_id, account_id, region, netloc):
"""Walk a job to a terminal state. Runs as a background task, so it pins
the account and region it was started for rather than inheriting whatever
request happens to be in flight."""
with request_scope(account_id, region):
try:
- await _run_job_inner(job_name, run_id, netloc, queue_seconds)
+ await _run_job_inner(job_name, run_id, netloc)
except Exception:
logger.exception("Transcribe: job %s crashed", job_name)
job = _live_job(job_name, run_id)
@@ -638,21 +631,9 @@ async def _run_job(job_name, run_id, account_id, region, netloc, queue_seconds):
_emit_state_change(job_name, "FAILED")
-async def _run_job_inner(job_name, run_id, netloc, queue_seconds):
- # queue_seconds is passed in rather than read here: the start handler
- # decided on it when it chose the job's initial status, and /_ministack/config
- # can change the module value in between, which would otherwise leave a
- # QUEUED job with no worker to start it.
- if queue_seconds > 0:
- await asyncio.sleep(queue_seconds)
-
- job = _live_job(job_name, run_id)
- if job is None:
- return
-
- job["TranscriptionJobStatus"] = "IN_PROGRESS"
- job["StartTime"] = time.time()
-
+async def _run_job_inner(job_name, run_id, netloc):
+ # The job is already IN_PROGRESS with StartTime set: the start handler put
+ # it there, as the AWS response does. There is no queued phase to wait out.
if _JOB_RUN_SECONDS > 0:
await asyncio.sleep(_JOB_RUN_SECONDS)
@@ -833,20 +814,18 @@ def _start_transcription_job(data):
location_type = "SERVICE_BUCKET"
run_id = new_uuid()
- queue_seconds = _JOB_QUEUE_SECONDS
- queued = queue_seconds > 0
# One timestamp for both members: a job that starts immediately has not
# started before it was created, and two time.time() calls can skew.
created = time.time()
job = {
"TranscriptionJobName": name,
- "TranscriptionJobStatus": "QUEUED" if queued else "IN_PROGRESS",
+ "TranscriptionJobStatus": "IN_PROGRESS",
"LanguageCode": language_code,
"MediaSampleRateHertz": data.get("MediaSampleRateHertz"),
"MediaFormat": data.get("MediaFormat"),
"Media": copy.deepcopy(media),
"Transcript": None,
- "StartTime": None if queued else created,
+ "StartTime": created,
"CreationTime": created,
"CompletionTime": None,
"FailureReason": None,
@@ -874,9 +853,7 @@ def _start_transcription_job(data):
_jobs[name] = job
asyncio.create_task(
- _run_job(
- name, run_id, get_account_id(), get_region(), _external_netloc(), queue_seconds
- )
+ _run_job(name, run_id, get_account_id(), get_region(), _external_netloc())
)
return json_response({"TranscriptionJob": _public_job(job)})
diff --git a/tests/test_apigatewayv2.py b/tests/test_apigatewayv2.py
index 16c945d3..0cdaf33b 100644
--- a/tests/test_apigatewayv2.py
+++ b/tests/test_apigatewayv2.py
@@ -3142,6 +3142,38 @@ def _create_cors_api(apigw, *, name: str, cors: dict | None):
return api_id
+def test_apigwv2_update_api_replaces_cors_and_delete_removes_it(apigw):
+ """UpdateApi replaces a CORS configuration; it cannot clear one. The API
+ has DeleteCorsConfiguration for that, so an UpdateApi that carries no
+ usable configuration leaves the existing one alone."""
+ import urllib.request
+
+ api_id = _create_cors_api(apigw, name="cors-update", cors={
+ "AllowOrigins": ["https://a.example"], "AllowMethods": ["GET"],
+ })
+ assert apigw.get_api(ApiId=api_id)["CorsConfiguration"]["AllowOrigins"] == [
+ "https://a.example"]
+
+ # A configuration on the call replaces the stored one.
+ apigw.update_api(ApiId=api_id, CorsConfiguration={
+ "AllowOrigins": ["https://b.example"], "AllowMethods": ["GET", "POST"]})
+ cors = apigw.get_api(ApiId=api_id)["CorsConfiguration"]
+ assert cors["AllowOrigins"] == ["https://b.example"]
+
+ # Another property on its own does not disturb it.
+ apigw.update_api(ApiId=api_id, Description="unrelated")
+ assert apigw.get_api(ApiId=api_id)["CorsConfiguration"]["AllowOrigins"] == [
+ "https://b.example"]
+
+ # DeleteCorsConfiguration is what removes it (boto3 models it; the call
+ # goes out as DELETE /v2/apis/{apiId}/cors).
+ req = urllib.request.Request(
+ f"{ENDPOINT}/v2/apis/{api_id}/cors", method="DELETE")
+ with urllib.request.urlopen(req) as r:
+ assert r.status == 204
+ assert "CorsConfiguration" not in apigw.get_api(ApiId=api_id)
+
+
def test_apigwv2_cors_preflight_echoes_configured_origin(apigw):
"""OPTIONS preflight returns allow_origin from cors_configuration, not wildcard (#406)."""
import urllib.request
diff --git a/tests/test_aws_credentials.py b/tests/test_aws_credentials.py
deleted file mode 100644
index 0a30be11..00000000
--- a/tests/test_aws_credentials.py
+++ /dev/null
@@ -1,407 +0,0 @@
-# Copyright (c) 2026 MiniStack Contributors. SPDX-License-Identifier: MIT
-# Copies or substantial portions, including AI-assisted ports or rewrites, must retain this notice (see LICENSE).
-
-import asyncio
-import time
-
-import pytest
-
-from ministack.core.aws_credentials import (
- AmbiguousAccessKeyError,
- CredentialResolutionError,
- ResolvedCredential,
- find_iam_access_key_account,
- resolve_credential,
-)
-from ministack.core.iam_evaluator import PrincipalInfo, resolve_principal
-from ministack.services import iam as iam_svc
-from ministack.services import sts as sts_svc
-
-
-def test_resolve_root_credential_from_environment(monkeypatch):
- monkeypatch.setenv("AWS_ACCESS_KEY_ID", "configured-root")
- monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "configured-secret")
- monkeypatch.setenv("AWS_SESSION_TOKEN", "configured-token")
-
- credential = resolve_credential(
- "configured-root", "123456789012", "configured-token"
- )
-
- assert isinstance(credential, ResolvedCredential)
- assert credential.secret_access_key == "configured-secret"
- assert credential.session_token == "configured-token"
- assert credential.principal_arn == "arn:aws:iam::123456789012:root"
-
-
-def test_resolve_root_credential_treats_empty_environment_token_as_absent(monkeypatch):
- monkeypatch.setenv("AWS_ACCESS_KEY_ID", "configured-root")
- monkeypatch.setenv("AWS_SESSION_TOKEN", "")
-
- credential = resolve_credential("configured-root", "123456789012", "")
-
- assert isinstance(credential, ResolvedCredential)
- assert credential.session_token is None
-
-
-def test_resolve_numeric_root_accepts_optional_ambient_session_token(monkeypatch):
- account_id = "123456789012"
- monkeypatch.setenv("AWS_ACCESS_KEY_ID", "configured-root")
- monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "configured-secret")
- monkeypatch.setenv("AWS_SESSION_TOKEN", "ambient-token")
-
- with_token = resolve_credential(account_id, account_id, "ambient-token")
- without_token = resolve_credential(account_id, account_id, "")
- wrong_token = resolve_credential(account_id, account_id, "wrong-token")
-
- assert isinstance(with_token, ResolvedCredential)
- assert with_token.session_token == "ambient-token"
- assert isinstance(without_token, ResolvedCredential)
- assert without_token.session_token is None
- assert isinstance(wrong_token, CredentialResolutionError)
- assert wrong_token.code == "InvalidToken"
-
-
-def test_resolve_iam_credential_is_account_scoped_and_requires_active_status():
- access_key = "AKIATESTSCOPED00001"
- first_account = "111111111111"
- second_account = "222222222222"
- iam_svc._access_keys.set_scoped(first_account, None, access_key, {
- "AccessKeyId": access_key,
- "SecretAccessKey": "first-secret",
- "Status": "Active",
- "UserName": "first-user",
- })
- iam_svc._access_keys.set_scoped(second_account, None, access_key, {
- "AccessKeyId": access_key,
- "SecretAccessKey": "second-secret",
- "Status": "Inactive",
- "UserName": "second-user",
- })
- try:
- first = resolve_credential(access_key, first_account, "")
- second = resolve_credential(access_key, second_account, "")
-
- assert isinstance(first, ResolvedCredential)
- assert first.secret_access_key == "first-secret"
- assert first.principal_arn == (
- "arn:aws:iam::111111111111:user/first-user"
- )
- assert isinstance(second, CredentialResolutionError)
- assert second.code == "InvalidClientTokenId"
- with pytest.raises(AmbiguousAccessKeyError):
- find_iam_access_key_account(access_key)
- finally:
- iam_svc._access_keys.pop_scoped(first_account, None, access_key, None)
- iam_svc._access_keys.pop_scoped(second_account, None, access_key, None)
-
-
-def test_resolve_sts_credential_checks_token_expiry_and_origin():
- access_key = "ASIATESTSESSION0001"
- account_id = "123456789012"
- sts_svc._sessions[access_key] = {
- "Arn": f"arn:aws:iam::{account_id}:user/alice",
- "UserId": "AIDAALICE",
- "SecretAccessKey": "session-secret",
- "SessionToken": "session-token",
- "Expiration": time.time() + 60,
- "AccountId": account_id,
- "PrincipalType": "User",
- "SourceAccessKeyId": "AKIAALICE",
- }
- try:
- credential = resolve_credential(access_key, account_id, "session-token")
- wrong = resolve_credential(access_key, account_id, "wrong-token")
- missing = resolve_credential(access_key, account_id, "")
- non_ascii = resolve_credential(access_key, account_id, "not-valid-☃")
-
- assert isinstance(credential, ResolvedCredential)
- assert credential.principal_type == "User"
- assert credential.principal_name == "alice"
- assert credential.source_access_key_id == "AKIAALICE"
- assert isinstance(wrong, CredentialResolutionError)
- assert wrong.code == "InvalidToken"
- assert isinstance(missing, CredentialResolutionError)
- assert missing.code == "InvalidToken"
- assert isinstance(non_ascii, CredentialResolutionError)
- assert non_ascii.code == "InvalidToken"
-
- sts_svc._sessions[access_key]["Expiration"] = time.time() - 1
- expired = resolve_credential(access_key, account_id, "session-token")
- assert isinstance(expired, CredentialResolutionError)
- assert expired.code == "ExpiredTokenException"
- finally:
- sts_svc._sessions.pop(access_key, None)
-
-
-def test_find_iam_access_key_account_returns_unique_owner():
- access_key = "test-account-lookup-key"
- account_id = "123456789012"
- iam_svc._access_keys.set_scoped(account_id, None, access_key, {
- "AccessKeyId": access_key,
- "SecretAccessKey": "secret",
- "Status": "Active",
- "UserName": "alice",
- })
- try:
- assert find_iam_access_key_account(access_key) == account_id
- finally:
- iam_svc._access_keys.pop_scoped(account_id, None, access_key, None)
-
-
-
-
-def test_ambiguous_iam_access_key_is_rejected_before_http_routing(monkeypatch):
- from ministack import app as app_mod
- from ministack.core.responses import get_account_id, set_request_account_id
-
- monkeypatch.setattr(app_mod, "AUTH", True)
- access_key = "test-ambiguous-http-key"
- accounts = ("000000000000", "123456789012")
- original_account = get_account_id()
- sent = []
- for account_id in accounts:
- iam_svc._access_keys.set_scoped(account_id, None, access_key, {
- "AccessKeyId": access_key,
- "SecretAccessKey": f"secret-{account_id}",
- "Status": "Active",
- "UserName": f"user-{account_id}",
- })
- scope = {
- "type": "http",
- "method": "GET",
- "path": "/",
- "headers": [(b"host", b"s3.localhost")],
- "query_string": (
- f"X-Amz-Credential={access_key}/20260908/us-east-1/s3/aws4_request"
- ).encode(),
- }
-
- async def receive():
- return {"type": "http.request", "body": b"", "more_body": False}
-
- async def send(message):
- sent.append(message)
-
- try:
- asyncio.run(app_mod.app(scope, receive, send))
-
- assert sent[0]["type"] == "http.response.start"
- assert sent[0]["status"] == 403
- assert b"InvalidClientTokenId" in sent[1]["body"]
- finally:
- for account_id in accounts:
- iam_svc._access_keys.pop_scoped(account_id, None, access_key, None)
- set_request_account_id(original_account)
-
-
-
-
[email protected]("auth_enabled", [False, True])
[email protected]("ambiguous", [False, True])
-def test_http_iam_routing_respects_auth_mode(monkeypatch, auth_enabled, ambiguous):
- from ministack import app as app_mod
- from ministack.core.responses import get_account_id, request_scope
-
- key = "test-routing-mode-key"
- owner = "123456789012"
- accounts = [owner, "234567890123"] if ambiguous else [owner]
- monkeypatch.setattr(app_mod, "AUTH", auth_enabled)
- monkeypatch.setenv("MINISTACK_ACCOUNT_ID", "000000000000")
- routed = []
- sent = []
-
- async def capture(*args):
- routed.append(get_account_id())
- return 200, {}, b"ok"
-
- async def receive():
- return {"type": "http.request", "body": b"", "more_body": False}
-
- async def send(message):
- sent.append(message)
-
- monkeypatch.setattr(app_mod, "_handle_pre_body_request", capture)
- for account in accounts:
- iam_svc._access_keys.set_scoped(account, None, key, {"UserName": "alice"})
- try:
- with request_scope("000000000000", "us-east-1"):
- asyncio.run(app_mod.app({
- "type": "http", "method": "GET", "path": "/",
- "headers": [(b"host", b"sts.localhost"), (
- b"authorization",
- f"AWS4-HMAC-SHA256 Credential={key}/20260911/us-east-1/sts/aws4_request".encode(),
- )],
- "query_string": b"",
- }, receive, send))
- assert sent[0]["status"] == (403 if auth_enabled and ambiguous else 200)
- assert routed == ([] if auth_enabled and ambiguous else [
- owner if auth_enabled else "000000000000"
- ])
- finally:
- for account in accounts:
- iam_svc._access_keys.pop_scoped(account, None, key, None)
-
-
[email protected]("auth_enabled", [False, True])
[email protected]("bad_token", [False, True])
-def test_s3_presign_verifies_in_both_auth_modes(monkeypatch, auth_enabled, bad_token):
- from urllib.parse import parse_qs, urlsplit
-
- import boto3
- from botocore.config import Config
-
- from ministack import app as app_mod
- from ministack.core.responses import get_account_id, request_scope
- from ministack.services import s3 as s3_svc
-
- key = "test-presign-mode-key"
- owner = "123456789012"
- monkeypatch.setattr(app_mod, "AUTH", auth_enabled)
- iam_svc._access_keys.set_scoped(owner, None, key, {
- "UserName": "alice", "Status": "Active", "SecretAccessKey": "secret",
- })
- client = boto3.client(
- "s3", endpoint_url="http://localhost:4566", region_name="us-east-1",
- aws_access_key_id=key, aws_secret_access_key="secret",
- aws_session_token="unexpected-token" if bad_token else None,
- config=Config(signature_version="s3v4"),
- )
- url = urlsplit(client.generate_presigned_url(
- "get_object", Params={"Bucket": "test-bucket", "Key": "object"},
- ))
- try:
- with request_scope("000000000000", "us-east-1"):
- error = s3_svc._verify_presigned_sigv4(
- "GET", url.path, {"host": url.netloc}, parse_qs(url.query),
- )
- assert get_account_id() == owner
- if bad_token:
- assert error[0] == 403
- else:
- assert error is None
- finally:
- iam_svc._access_keys.pop_scoped(owner, None, key, None)
-
-
[email protected]("auth_enabled", [False, True])
[email protected]("action", ["GetSessionToken", "AssumeRole"])
[email protected]("credential_kind", ["unknown", "inactive", "expired"])
-def test_sts_credential_rejections_require_auth(monkeypatch, auth_enabled, action, credential_kind):
- from ministack import app as app_mod
- from ministack.core.responses import request_scope
-
- key = "test-sts-mode-key"
- account = "000000000000"
- monkeypatch.setattr(app_mod, "AUTH", auth_enabled)
- if credential_kind == "inactive":
- iam_svc._access_keys.set_scoped(account, None, key, {
- "UserName": "alice", "Status": "Inactive", "SecretAccessKey": "secret",
- })
- if credential_kind == "expired":
- sts_svc._sessions[key] = {
- "Arn": f"arn:aws:sts::{account}:assumed-role/role/session",
- "UserId": "role:session", "SecretAccessKey": "secret",
- "Expiration": time.time() - 60,
- }
- previous = set(sts_svc._sessions)
- try:
- with request_scope(account, "us-east-1"):
- status, _, _ = asyncio.run(sts_svc.handle_request(
- "GET", "/", {
- "authorization": f"AWS4-HMAC-SHA256 Credential={key}/20260911/us-east-1/sts/aws4_request",
- }, b"", {
- "Action": [action], "RoleArn": [f"arn:aws:iam::{account}:role/role"],
- "RoleSessionName": ["session"],
- },
- ))
- assert status == (403 if auth_enabled else 200)
- finally:
- iam_svc._access_keys.pop_scoped(account, None, key, None)
- for created in set(sts_svc._sessions) - previous:
- sts_svc._sessions.pop(created, None)
- sts_svc._sessions.pop(key, None)
-
-
-def test_presigned_mrap_resolves_alias_in_iam_owner_account(monkeypatch):
- from urllib.parse import parse_qs, urlsplit
-
- from botocore.auth import S3SigV4QueryAuth
- from botocore.awsrequest import AWSRequest
- from botocore.credentials import Credentials
-
- from ministack import app as app_mod
- from ministack.core.responses import get_account_id, request_scope
- from ministack.services import s3 as s3_svc
-
- key, owner, alias = "test-mrap-owner-key", "123456789012", "testalias.mrap"
- host = f"{alias}.accesspoint.s3-global.amazonaws.com"
- monkeypatch.setattr(app_mod, "AUTH", False)
- request = AWSRequest(method="GET", url=f"http://{host}/object")
- S3SigV4QueryAuth(Credentials(key, "secret"), "s3", "us-east-1").add_auth(request)
- url = urlsplit(request.url)
- routed = []
-
- async def capture(method, path, headers, body, query_params, **kwargs):
- routed.append((get_account_id(), path))
- return 200, {}, b"object"
-
- monkeypatch.setattr(s3_svc, "handle_request", capture)
- iam_svc._access_keys.set_scoped(owner, None, key, {
- "UserName": "alice", "Status": "Active", "SecretAccessKey": "secret",
- })
- s3_svc._mraps.set_scoped(owner, None, alias, {"Regions": ["member-bucket"]})
- try:
- with request_scope("000000000000", "us-east-1"):
- result = asyncio.run(app_mod._handle_s3_vhost_request(
- host, url.path, "GET", {"host": host}, b"", parse_qs(url.query),
- ))
- assert result[0] == 200
- assert routed == [(owner, "/member-bucket/object")]
- finally:
- iam_svc._access_keys.pop_scoped(owner, None, key, None)
- s3_svc._mraps.pop_scoped(owner, None, alias, None)
-
-
-def test_resolve_get_session_token_principal_retains_user_policies():
- access_key = "test-session-access-key"
- account_id = "123456789012"
- user_name = "alice"
- iam_svc._users.set_scoped(account_id, None, user_name, {
- "UserName": user_name,
- "UserId": "AIDAALICE",
- "AttachedPolicies": [],
- })
- iam_svc._user_inline_policies[user_name] = {
- "allow-s3": {
- "Statement": [{
- "Effect": "Allow",
- "Action": "s3:GetObject",
- "Resource": "*",
- }],
- },
- }
- sts_svc._sessions[access_key] = {
- "Arn": f"arn:aws:iam::{account_id}:user/team/{user_name}",
- "UserId": "AIDAALICE",
- "SecretAccessKey": "session-secret",
- "SessionToken": "session-token",
- "Expiration": time.time() + 60,
- "AccountId": account_id,
- "PrincipalType": "User",
- "PrincipalName": user_name,
- "SourceAccessKeyId": "AKIAALICE",
- }
- try:
- principal = resolve_principal(access_key, account_id)
-
- assert isinstance(principal, PrincipalInfo)
- assert principal.type == "User"
- assert principal.arn == (
- f"arn:aws:iam::{account_id}:user/team/{user_name}"
- )
- assert principal.policies
- assert principal.policies[0][0].actions == ["s3:GetObject"]
- finally:
- sts_svc._sessions.pop(access_key, None)
- iam_svc._user_inline_policies.pop(user_name, None)
- iam_svc._users.pop_scoped(account_id, None, user_name, None)
diff --git a/tests/test_cfn.py b/tests/test_cfn.py
index bc66b694..fee10ab5 100644
--- a/tests/test_cfn.py
+++ b/tests/test_cfn.py
@@ -10501,8 +10501,9 @@ def test_cfn_cloudfront_function_updates_in_place(cfn, cloudfront):
body = cloudfront.get_function(Name=name, Stage="LIVE")["FunctionCode"].read()
assert b"updated-by-cfn" in body
- # Dropping AutoPublish leaves the new code unpublished, as the service's
- # own UpdateFunction does.
+ # Dropping AutoPublish leaves the new code unpublished: the update
+ # reaches DEVELOPMENT only and LIVE keeps serving what was published,
+ # as UpdateFunction does.
_cfn_cf_stack(cfn, stack_name,
_cf_function_template(name, comment="v3", code=new_code,
auto_publish=None),
@@ -10510,8 +10511,8 @@ def test_cfn_cloudfront_function_updates_in_place(cfn, cloudfront):
dev = cloudfront.describe_function(Name=name, Stage="DEVELOPMENT")["FunctionSummary"]
assert dev["FunctionConfig"]["Comment"] == "v3"
assert dev["FunctionMetadata"]["CreatedTime"] == created
- with pytest.raises(ClientError):
- cloudfront.describe_function(Name=name, Stage="LIVE")
+ still_live = cloudfront.describe_function(Name=name, Stage="LIVE")["FunctionSummary"]
+ assert still_live["FunctionConfig"]["Comment"] == "v2"
finally:
_delete_cfn_test_stack(cfn, stack_name)
diff --git a/tests/test_cloudfront.py b/tests/test_cloudfront.py
index 659385a4..dd488c9d 100644
--- a/tests/test_cloudfront.py
+++ b/tests/test_cloudfront.py
@@ -826,6 +826,47 @@ def test_cloudfront_function_create_publish_describe_get_delete(cloudfront):
assert exc.value.response["Error"]["Code"] == "NoSuchFunctionExists"
+def test_cloudfront_function_update_keeps_the_published_live_stage(cloudfront):
+ """UpdateFunction changes the DEVELOPMENT stage only: "To copy the updates
+ from the DEVELOPMENT stage to LIVE, you must publish the function". The
+ published version keeps serving until PublishFunction runs again."""
+ name = f"fn-live-{_uuid_mod.uuid4().hex[:8]}"
+ published = b"function handler(event) { return 'published'; }"
+ updated = b"function handler(event) { return 'updated'; }"
+
+ cr = cloudfront.create_function(
+ Name=name,
+ FunctionConfig={"Comment": "v1", "Runtime": "cloudfront-js-1.0"},
+ FunctionCode=published,
+ )
+ cloudfront.publish_function(Name=name, IfMatch=_cf_resp_etag(cr))
+
+ upd = cloudfront.update_function(
+ Name=name,
+ IfMatch=_cf_resp_etag(cloudfront.describe_function(Name=name, Stage="DEVELOPMENT")),
+ FunctionConfig={"Comment": "v2", "Runtime": "cloudfront-js-1.0"},
+ FunctionCode=updated,
+ )
+
+ live = cloudfront.describe_function(Name=name, Stage="LIVE")["FunctionSummary"]
+ assert live["FunctionConfig"]["Comment"] == "v1"
+ live_body = cloudfront.get_function(Name=name, Stage="LIVE")["FunctionCode"]
+ assert (live_body.read() if hasattr(live_body, "read") else live_body) == published
+ dev_body = cloudfront.get_function(Name=name, Stage="DEVELOPMENT")["FunctionCode"]
+ assert (dev_body.read() if hasattr(dev_body, "read") else dev_body) == updated
+
+ cloudfront.publish_function(Name=name, IfMatch=_cf_resp_etag(upd))
+ live_body = cloudfront.get_function(Name=name, Stage="LIVE")["FunctionCode"]
+ assert (live_body.read() if hasattr(live_body, "read") else live_body) == updated
+ live = cloudfront.describe_function(Name=name, Stage="LIVE")["FunctionSummary"]
+ assert live["FunctionConfig"]["Comment"] == "v2"
+
+ cloudfront.delete_function(
+ Name=name,
+ IfMatch=_cf_resp_etag(cloudfront.describe_function(Name=name, Stage="DEVELOPMENT")),
+ )
+
+
def test_cloudfront_function_duplicate_name(cloudfront):
name = f"fn-dup-{_uuid_mod.uuid4().hex[:8]}"
cloudfront.create_function(
diff --git a/tests/test_ecs.py b/tests/test_ecs.py
index 16ef90a1..47a04a35 100644
--- a/tests/test_ecs.py
+++ b/tests/test_ecs.py
@@ -1482,7 +1482,8 @@ def _fake_docker_recorder(run_impl=None):
def test_ecs_run_task_returns_pending_before_docker_start(monkeypatch):
- """Registration is visible while a slow Docker start is still blocked."""
+ """Registration is visible while a slow Docker start is still blocked, and
+ the task reports ACTIVATING for the length of the pull."""
import threading
from ministack.services import ecs as _ecs
@@ -1541,8 +1542,16 @@ def test_ecs_run_task_returns_pending_before_docker_start(monkeypatch):
task = json.loads(response[2])["tasks"][0]
assert task["lastStatus"] == "PENDING"
assert task["containers"][0]["lastStatus"] == "PENDING"
+ # AWS omits a timestamp it has no value for rather than sending a null:
+ # the task has not started, pulled or stopped yet.
+ for member in ("startedAt", "pullStartedAt", "pullStoppedAt",
+ "stoppingAt", "stoppedAt"):
+ assert member not in task
assert started.wait(timeout=2)
- assert _ecs._tasks[task["taskArn"]]["lastStatus"] == "PENDING"
+ # The image pull is the ACTIVATING phase on AWS: "This is the state where
+ # Amazon ECS pulls the container images, creates the containers ...".
+ assert _ecs._tasks[task["taskArn"]]["lastStatus"] == "ACTIVATING"
+ assert _ecs._tasks[task["taskArn"]]["pullStartedAt"]
release.set()
_wait_until(lambda: _ecs._tasks[task["taskArn"]]["lastStatus"] == "RUNNING")
diff --git a/tests/test_iam_auth.py b/tests/test_iam_auth.py
index d97d4b93..a0e603f9 100644
--- a/tests/test_iam_auth.py
+++ b/tests/test_iam_auth.py
@@ -10,6 +10,7 @@ Integration tests (SimulateCustomPolicy, policy validation) run against
the MiniStack server and work with AUTH=false.
"""
+import asyncio
import json
import time
@@ -17,15 +18,20 @@ import pytest
from botocore.exceptions import ClientError
from ministack.core.iam_evaluator import (
+ AmbiguousAccessKeyError,
AuthError,
+ CredentialResolutionError,
EvalContext,
EvalResult,
PrincipalInfo,
+ ResolvedCredential,
enforce,
evaluate,
evaluate_trust_policy,
+ find_iam_access_key_account,
fnmatch_iam,
parse_policy_document,
+ resolve_credential,
resolve_principal,
validate_policy_document,
)
@@ -931,6 +937,55 @@ class TestResourceArn:
"arn:aws:dynamodb:us-east-1:123:table/snapshots",
]
+ def test_eventbridge_put_events_returns_every_bus(self):
+ from ministack.core.iam_actions import eventbridge_resource_arns
+ body = json.dumps({"Entries": [
+ {"EventBusName": "orders"},
+ {"Source": "example"},
+ {"EventBusName": "orders"},
+ {"EventBusName": "arn:aws:events:us-east-1:123:event-bus/audit"},
+ ]}).encode()
+ assert eventbridge_resource_arns(body, "us-east-1", "123") == [
+ "arn:aws:events:us-east-1:123:event-bus/orders",
+ "arn:aws:events:us-east-1:123:event-bus/default",
+ "arn:aws:events:us-east-1:123:event-bus/audit",
+ ]
+
+ def test_eventbridge_put_events_survives_a_malformed_entry(self):
+ from ministack.core.iam_actions import eventbridge_resource_arns
+ body = json.dumps({"Entries": ["junk", {"EventBusName": 7}]}).encode()
+ assert eventbridge_resource_arns(body, "us-east-1", "123") == [
+ "arn:aws:events:us-east-1:123:event-bus/default",
+ ]
+
+ def test_dynamodb_attributes_are_top_level_only(self):
+ """AWS resolves a ProjectionExpression of "Name, Address.City" to
+ ["Name", "Address"], substituting a placeholder per path segment."""
+ from ministack.core.iam_actions import dynamodb_service_context
+ body = json.dumps({
+ "TableName": "users",
+ "ProjectionExpression": "Name, Address.City, #a.#c, Items[0]",
+ "ExpressionAttributeNames": {"#a": "Addr", "#c": "City"},
+ }).encode()
+ context = dynamodb_service_context(body)
+ assert context["dynamodb:Attributes"] == ["Name", "Address", "Addr", "Items"]
+ assert context["dynamodb:Select"] == "SPECIFIC_ATTRIBUTES"
+
+ def test_dynamodb_select_is_always_resolved(self):
+ """Select always has a value on AWS, so a policy conditioning on it
+ with StringEqualsIfExists must not pass a projection-less request."""
+ from ministack.core.iam_actions import dynamodb_service_context
+ assert dynamodb_service_context(
+ json.dumps({"TableName": "users"}).encode()
+ ) == {"dynamodb:Select": "ALL_ATTRIBUTES"}
+ assert dynamodb_service_context(
+ json.dumps({"TableName": "users", "Select": "COUNT"}).encode()
+ )["dynamodb:Select"] == "COUNT"
+ assert dynamodb_service_context(
+ json.dumps({"TableName": "users", "AttributesToGet": ["a", "b"]}).encode()
+ ) == {"dynamodb:Attributes": ["a", "b"], "dynamodb:Select": "SPECIFIC_ATTRIBUTES"}
+ assert dynamodb_service_context(b"not json") == {}
+
def test_eventbridge_put_events_defaults_to_default_bus(self):
from ministack.core.iam_actions import extract_resource_arn
body = json.dumps({"Entries": [{"Source": "example"}]}).encode()
@@ -1474,7 +1529,8 @@ class TestS3EnforcementSites:
stmts = parse_policy_document(policy)
seen = []
- def enforce_stub(access_key_id, iam_action, service, region, resource_arn="*"):
+ def enforce_stub(access_key_id, iam_action, service, region, resource_arn="*",
+ service_context=None):
seen.append((iam_action, resource_arn))
result = evaluate(_ctx(action=iam_action, resource=resource_arn), [stmts])
if result.decision == "Allow":
@@ -1958,3 +2014,284 @@ def test_lambda_execution_role_explicit_deny_overrides_allow(monkeypatch):
iam_svc._roles.pop(role_name, None)
sts_svc._sessions.clear()
_request_account_id.reset(token)
+
+
+# ---------------------------------------------------------------------------
+# Access-key resolution (root / IAM user / STS session)
+# ---------------------------------------------------------------------------
+
+
+def test_resolve_root_credential_from_environment(monkeypatch):
+ monkeypatch.setenv("AWS_ACCESS_KEY_ID", "configured-root")
+ monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "configured-secret")
+ monkeypatch.setenv("AWS_SESSION_TOKEN", "configured-token")
+
+ credential = resolve_credential(
+ "configured-root", "123456789012", "configured-token"
+ )
+
+ assert isinstance(credential, ResolvedCredential)
+ assert credential.secret_access_key == "configured-secret"
+ assert credential.session_token == "configured-token"
+ assert credential.principal_arn == "arn:aws:iam::123456789012:root"
+
+
+def test_resolve_root_credential_treats_empty_environment_token_as_absent(monkeypatch):
+ monkeypatch.setenv("AWS_ACCESS_KEY_ID", "configured-root")
+ monkeypatch.setenv("AWS_SESSION_TOKEN", "")
+
+ credential = resolve_credential("configured-root", "123456789012", "")
+
+ assert isinstance(credential, ResolvedCredential)
+ assert credential.session_token is None
+
+
+def test_resolve_numeric_root_accepts_optional_ambient_session_token(monkeypatch):
+ account_id = "123456789012"
+ monkeypatch.setenv("AWS_ACCESS_KEY_ID", "configured-root")
+ monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "configured-secret")
+ monkeypatch.setenv("AWS_SESSION_TOKEN", "ambient-token")
+
+ with_token = resolve_credential(account_id, account_id, "ambient-token")
+ without_token = resolve_credential(account_id, account_id, "")
+ wrong_token = resolve_credential(account_id, account_id, "wrong-token")
+
+ assert isinstance(with_token, ResolvedCredential)
+ assert with_token.session_token == "ambient-token"
+ assert isinstance(without_token, ResolvedCredential)
+ assert without_token.session_token is None
+ assert isinstance(wrong_token, CredentialResolutionError)
+ assert wrong_token.code == "InvalidToken"
+
+
+def test_resolve_iam_credential_is_account_scoped_and_requires_active_status():
+ from ministack.services import iam as iam_svc
+
+ access_key = "AKIATESTSCOPED00001"
+ first_account = "111111111111"
+ second_account = "222222222222"
+ iam_svc._access_keys.set_scoped(first_account, None, access_key, {
+ "AccessKeyId": access_key,
+ "SecretAccessKey": "first-secret",
+ "Status": "Active",
+ "UserName": "first-user",
+ })
+ iam_svc._access_keys.set_scoped(second_account, None, access_key, {
+ "AccessKeyId": access_key,
+ "SecretAccessKey": "second-secret",
+ "Status": "Inactive",
+ "UserName": "second-user",
+ })
+ try:
+ first = resolve_credential(access_key, first_account, "")
+ second = resolve_credential(access_key, second_account, "")
+
+ assert isinstance(first, ResolvedCredential)
+ assert first.secret_access_key == "first-secret"
+ assert first.principal_arn == (
+ "arn:aws:iam::111111111111:user/first-user"
+ )
+ assert isinstance(second, CredentialResolutionError)
+ assert second.code == "InvalidClientTokenId"
+ with pytest.raises(AmbiguousAccessKeyError):
+ find_iam_access_key_account(access_key)
+ finally:
+ iam_svc._access_keys.pop_scoped(first_account, None, access_key, None)
+ iam_svc._access_keys.pop_scoped(second_account, None, access_key, None)
+
+
+def test_resolve_sts_credential_checks_token_expiry_and_origin():
+ from ministack.services import sts as sts_svc
+
+ access_key = "ASIATESTSESSION0001"
+ account_id = "123456789012"
+ sts_svc._sessions[access_key] = {
+ "Arn": f"arn:aws:iam::{account_id}:user/alice",
+ "UserId": "AIDAALICE",
+ "SecretAccessKey": "session-secret",
+ "SessionToken": "session-token",
+ "Expiration": time.time() + 60,
+ "AccountId": account_id,
+ "PrincipalType": "User",
+ "SourceAccessKeyId": "AKIAALICE",
+ }
+ try:
+ credential = resolve_credential(access_key, account_id, "session-token")
+ wrong = resolve_credential(access_key, account_id, "wrong-token")
+ missing = resolve_credential(access_key, account_id, "")
+ non_ascii = resolve_credential(access_key, account_id, "not-valid-☃")
+
+ assert isinstance(credential, ResolvedCredential)
+ assert credential.principal_type == "User"
+ assert credential.principal_name == "alice"
+ assert credential.source_access_key_id == "AKIAALICE"
+ assert isinstance(wrong, CredentialResolutionError)
+ assert wrong.code == "InvalidToken"
+ assert isinstance(missing, CredentialResolutionError)
+ assert missing.code == "InvalidToken"
+ assert isinstance(non_ascii, CredentialResolutionError)
+ assert non_ascii.code == "InvalidToken"
+
+ sts_svc._sessions[access_key]["Expiration"] = time.time() - 1
+ expired = resolve_credential(access_key, account_id, "session-token")
+ assert isinstance(expired, CredentialResolutionError)
+ assert expired.code == "ExpiredTokenException"
+ finally:
+ sts_svc._sessions.pop(access_key, None)
+
+
+def test_find_iam_access_key_account_returns_unique_owner():
+ from ministack.services import iam as iam_svc
+
+ access_key = "test-account-lookup-key"
+ account_id = "123456789012"
+ iam_svc._access_keys.set_scoped(account_id, None, access_key, {
+ "AccessKeyId": access_key,
+ "SecretAccessKey": "secret",
+ "Status": "Active",
+ "UserName": "alice",
+ })
+ try:
+ assert find_iam_access_key_account(access_key) == account_id
+ finally:
+ iam_svc._access_keys.pop_scoped(account_id, None, access_key, None)
+
+
+def test_resolve_get_session_token_principal_retains_user_policies():
+ from ministack.services import iam as iam_svc
+ from ministack.services import sts as sts_svc
+
+ access_key = "test-session-access-key"
+ account_id = "123456789012"
+ user_name = "alice"
+ iam_svc._users.set_scoped(account_id, None, user_name, {
+ "UserName": user_name,
+ "UserId": "AIDAALICE",
+ "AttachedPolicies": [],
+ })
+ iam_svc._user_inline_policies[user_name] = {
+ "allow-s3": {
+ "Statement": [{
+ "Effect": "Allow",
+ "Action": "s3:GetObject",
+ "Resource": "*",
+ }],
+ },
+ }
+ sts_svc._sessions[access_key] = {
+ "Arn": f"arn:aws:iam::{account_id}:user/team/{user_name}",
+ "UserId": "AIDAALICE",
+ "SecretAccessKey": "session-secret",
+ "SessionToken": "session-token",
+ "Expiration": time.time() + 60,
+ "AccountId": account_id,
+ "PrincipalType": "User",
+ "PrincipalName": user_name,
+ "SourceAccessKeyId": "AKIAALICE",
+ }
+ try:
+ principal = resolve_principal(access_key, account_id)
+
+ assert isinstance(principal, PrincipalInfo)
+ assert principal.type == "User"
+ assert principal.arn == (
+ f"arn:aws:iam::{account_id}:user/team/{user_name}"
+ )
+ assert principal.policies
+ assert principal.policies[0][0].actions == ["s3:GetObject"]
+ finally:
+ sts_svc._sessions.pop(access_key, None)
+ iam_svc._user_inline_policies.pop(user_name, None)
+ iam_svc._users.pop_scoped(account_id, None, user_name, None)
+
+
+def test_ambiguous_iam_access_key_is_rejected_before_http_routing(monkeypatch):
+ from ministack import app as app_mod
+ from ministack.core.responses import get_account_id, set_request_account_id
+ from ministack.services import iam as iam_svc
+
+ monkeypatch.setattr(app_mod, "AUTH", True)
+ access_key = "test-ambiguous-http-key"
+ accounts = ("000000000000", "123456789012")
+ original_account = get_account_id()
+ sent = []
+ for account_id in accounts:
+ iam_svc._access_keys.set_scoped(account_id, None, access_key, {
+ "AccessKeyId": access_key,
+ "SecretAccessKey": f"secret-{account_id}",
+ "Status": "Active",
+ "UserName": f"user-{account_id}",
+ })
+ scope = {
+ "type": "http",
+ "method": "GET",
+ "path": "/",
+ "headers": [(b"host", b"s3.localhost")],
+ "query_string": (
+ f"X-Amz-Credential={access_key}/20260908/us-east-1/s3/aws4_request"
+ ).encode(),
+ }
+
+ async def receive():
+ return {"type": "http.request", "body": b"", "more_body": False}
+
+ async def send(message):
+ sent.append(message)
+
+ try:
+ asyncio.run(app_mod.app(scope, receive, send))
+
+ assert sent[0]["type"] == "http.response.start"
+ assert sent[0]["status"] == 403
+ assert b"InvalidClientTokenId" in sent[1]["body"]
+ finally:
+ for account_id in accounts:
+ iam_svc._access_keys.pop_scoped(account_id, None, access_key, None)
+ set_request_account_id(original_account)
+
+
[email protected]("auth_enabled", [False, True])
[email protected]("ambiguous", [False, True])
+def test_http_iam_routing_respects_auth_mode(monkeypatch, auth_enabled, ambiguous):
+ from ministack import app as app_mod
+ from ministack.core.responses import get_account_id, request_scope
+ from ministack.services import iam as iam_svc
+
+ key = "test-routing-mode-key"
+ owner = "123456789012"
+ accounts = [owner, "234567890123"] if ambiguous else [owner]
+ monkeypatch.setattr(app_mod, "AUTH", auth_enabled)
+ monkeypatch.setenv("MINISTACK_ACCOUNT_ID", "000000000000")
+ routed = []
+ sent = []
+
+ async def capture(*args):
+ routed.append(get_account_id())
+ return 200, {}, b"ok"
+
+ async def receive():
+ return {"type": "http.request", "body": b"", "more_body": False}
+
+ async def send(message):
+ sent.append(message)
+
+ monkeypatch.setattr(app_mod, "_handle_pre_body_request", capture)
+ for account in accounts:
+ iam_svc._access_keys.set_scoped(account, None, key, {"UserName": "alice"})
+ try:
+ with request_scope("000000000000", "us-east-1"):
+ asyncio.run(app_mod.app({
+ "type": "http", "method": "GET", "path": "/",
+ "headers": [(b"host", b"sts.localhost"), (
+ b"authorization",
+ f"AWS4-HMAC-SHA256 Credential={key}/20260911/us-east-1/sts/aws4_request".encode(),
+ )],
+ "query_string": b"",
+ }, receive, send))
+ assert sent[0]["status"] == (403 if auth_enabled and ambiguous else 200)
+ assert routed == ([] if auth_enabled and ambiguous else [
+ owner if auth_enabled else "000000000000"
+ ])
+ finally:
+ for account in accounts:
+ iam_svc._access_keys.pop_scoped(account, None, key, None)
diff --git a/tests/test_lambda.py b/tests/test_lambda.py
index 7b8e2c02..d9d5539b 100644
--- a/tests/test_lambda.py
+++ b/tests/test_lambda.py
@@ -3,6 +3,9 @@ import contextlib
import io
import json
import os
+import socket
+import sys
+import threading
import shutil
import sys
import time
@@ -10,7 +13,7 @@ import urllib.error as _urlerr
import urllib.request as _urlreq
import uuid as _uuid_mod
import zipfile
-from unittest.mock import patch
+from unittest.mock import Mock, patch
from urllib.parse import urlparse
import boto3
@@ -6259,6 +6262,113 @@ def test_route_async_failure_to_sqs_dlq():
set_request_region(original_region)
[email protected]("payload", [b'[1,2,3]', b'"a string"', b'42', b'{"k":"v"}'])
+def test_warm_worker_context_reaches_every_payload_shape(lam, payload):
+ """On AWS the per-invocation context is the execution environment and the
+ context object, not the event: `_X_AMZN_TRACE_ID` is a reserved variable
+ that "changes with each invocation" (configuration-envvars) and the payload
+ reaches the handler as sent, whatever its JSON type. The pooled worker
+ carries those values beside the payload, so a list or a bare string is
+ served exactly like an object."""
+ fname = f"warm-ctx-{_uuid_mod.uuid4().hex[:8]}"
+ code = (
+ "import os\n"
+ "def handler(event, context):\n"
+ " return {'event': event,\n"
+ " 'trace': os.environ.get('_X_AMZN_TRACE_ID'),\n"
+ " 'req': context.aws_request_id}\n"
+ )
+ lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _make_zip(code)},
+ Timeout=10, TracingConfig={"Mode": "Active"},
+ )
+ try:
+ body = json.loads(lam.invoke(FunctionName=fname, Payload=payload)["Payload"].read())
+ assert body["event"] == json.loads(payload)
+ assert body["trace"], "the trace header did not reach a non-dict payload"
+ assert body["req"]
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+def test_nodejs_warm_context_object_is_populated(lam):
+ """The properties nodejs-context.html documents, on the warm pool: the
+ bootstrap read them off the event, where nothing ever set them, so every
+ one was empty and getRemainingTimeInMillis() was a constant."""
+ fname = f"warm-jsctx-{_uuid_mod.uuid4().hex[:8]}"
+ code = (
+ "exports.handler = async (event, context) => ({\n"
+ " name: context.functionName, version: context.functionVersion,\n"
+ " mem: context.memoryLimitInMB, arn: context.invokedFunctionArn,\n"
+ " req: context.awsRequestId, group: context.logGroupName,\n"
+ " stream: context.logStreamName, remaining: context.getRemainingTimeInMillis(),\n"
+ " waits: context.callbackWaitsForEmptyEventLoop});\n"
+ )
+ lam.create_function(
+ FunctionName=fname, Runtime="nodejs20.x", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _make_zip_js(code)},
+ Timeout=7, MemorySize=256,
+ )
+ try:
+ first = json.loads(lam.invoke(FunctionName=fname, Payload=b"{}")["Payload"].read())
+ assert first["name"] == fname
+ assert first["version"] == "$LATEST"
+ assert first["mem"] == "256"
+ assert first["arn"].endswith(f":function:{fname}")
+ assert first["group"] == f"/aws/lambda/{fname}"
+ assert first["stream"]
+ assert first["waits"] is True
+ # Counts down from the configured timeout rather than a constant.
+ assert 0 < first["remaining"] <= 7000
+ # The request id is per invocation; the log stream is per environment.
+ second = json.loads(lam.invoke(FunctionName=fname, Payload=b"{}")["Payload"].read())
+ assert second["req"] != first["req"]
+ assert second["stream"] == first["stream"]
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+def test_warm_invocation_leaves_the_caller_event_untouched(monkeypatch):
+ """The per-invocation values (trace header, invoke depth, durable context)
+ travel beside the payload, so the handler's event is the caller's and
+ nothing ministack-shaped survives the call — the async retry path hands
+ the very same object to `_route_async_failure`, which puts it in the DLQ
+ envelope as `requestPayload`."""
+ seen = {}
+ worker = Mock()
+
+ def _invoke(event, request_id, **ctx):
+ seen["event"] = event
+ seen["ctx"] = ctx
+ return {"status": "ok", "result": {"ok": True}, "log": ""}
+
+ worker.invoke.side_effect = _invoke
+ monkeypatch.setattr(lsvc, "_ensure_reaper_thread", lambda: None)
+ monkeypatch.setattr(lsvc, "acquire_worker", Mock(return_value=(worker, "reuse")))
+ monkeypatch.setattr(lsvc, "release_worker", Mock())
+ monkeypatch.setattr(lsvc, "_emit_lambda_logs", Mock())
+ config = {
+ "FunctionName": "warm-carrier-copy",
+ "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:warm-carrier-copy",
+ "Runtime": "python3.12", "Handler": "index.handler", "Timeout": 3,
+ }
+ event = {"input": "hi"}
+ token = lsvc._durable_ctx.set({"arn": "exec-arn", "token": "tok", "name": "exec"})
+ try:
+ result = lsvc._execute_function_warm({"config": config, "code_zip": b"zip"}, event)
+ finally:
+ lsvc._durable_ctx.reset(token)
+
+ assert result["body"] == {"ok": True}
+ # The per-invocation values reached the worker beside the payload...
+ assert seen["ctx"]["depth"] is not None
+ assert seen["ctx"]["durable"]["AWS_LAMBDA_DURABLE_EXECUTION_ARN"] == "exec-arn"
+ # ...and the payload is the caller's, untouched.
+ assert seen["event"] == {"input": "hi"}
+ assert event == {"input": "hi"}
+
+
def test_route_async_failure_to_sqs_does_not_tail_match_foreign_region():
"""A stale foreign-Region target ARN must not route to a same-named local queue."""
import ministack.services.sqs as _sqs
@@ -8765,6 +8875,95 @@ def _start_callback(lam):
return fname, rec["DurableExecutionArn"], cb_id, body["CheckpointToken"]
+def test_lambda_durable_concurrent_executions_get_their_own_callback(lam):
+ """Two executions of one function reach the same workflow position, so the
+ SDK checkpoints the same operation id for both. The CallbackId is the
+ emulator's to mint and must be unique per execution: completing the first
+ execution's callback must resolve that execution and leave the second one
+ waiting. Reported by @Nhollas."""
+ import base64 as _b64
+ from urllib.parse import quote
+
+ fname = f"durable-cb-iso-{_uuid_mod.uuid4().hex[:8]}"
+ zip_b64 = _b64.b64encode(_make_zip("def handler(e,c): return e")).decode()
+ _raw_durable("POST", "/2015-03-31/functions", body={
+ "FunctionName": fname,
+ "Runtime": "python3.12",
+ "Role": _LAMBDA_ROLE,
+ "Handler": "index.handler",
+ "Code": {"ZipFile": zip_b64},
+ "DurableConfig": {"Enabled": True},
+ })
+ op_id = "cbsharedaaaaaaaaaaaaaaaaaaaaaa"
+
+ def start(payload):
+ """Invoke, then checkpoint a CALLBACK START under the shared op id."""
+ req = urllib.request.Request(
+ f"{_ms_endpoint()}/2015-03-31/functions/{fname}/invocations",
+ method="POST", data=payload, headers={"Content-Type": "application/json"})
+ with urllib.request.urlopen(req) as r:
+ arn = r.headers.get("X-Amz-Durable-Execution-Arn")
+ token = r.headers.get("X-Amz-Durable-Checkpoint-Token")
+ r.read()
+ code, body = _raw_durable(
+ "POST", f"/2025-12-01/durable-executions/{quote(arn, safe='/:$')}/checkpoint",
+ body={"CheckpointToken": token, "Updates": [{
+ "Id": op_id, "Type": "CALLBACK", "Action": "START", "Name": "answer",
+ "CallbackOptions": {"TimeoutSeconds": 120},
+ }]})
+ assert code == 200, body
+ op = next(o for o in body["NewExecutionState"]["Operations"] if o["Id"] == op_id)
+ return arn, (op.get("CallbackDetails") or {}).get("CallbackId")
+
+ try:
+ arn_a, cb_a = start(b'{"runId":"A"}')
+ arn_b, cb_b = start(b'{"runId":"B"}')
+ assert arn_a != arn_b
+ assert cb_a and cb_b
+ assert cb_a != cb_b, "two executions were handed the same CallbackId"
+
+ # Completing A's callback resolves A...
+ req = urllib.request.Request(
+ f"{_ms_endpoint()}/2025-12-01/durable-execution-callbacks/"
+ f"{quote(cb_a, safe='')}/succeed",
+ method="POST", data=b'{"answer":"A"}',
+ headers={"Content-Type": "application/json"})
+ with urllib.request.urlopen(req) as r:
+ assert r.status == 200
+
+ def events(arn):
+ code, hist = _raw_durable(
+ "GET", f"/2025-12-01/durable-executions/{quote(arn, safe='/:$')}/history")
+ assert code == 200, hist
+ return hist["Events"]
+
+ succeeded_a = [e for e in events(arn_a) if e["EventType"] == "CallbackSucceeded"]
+ assert len(succeeded_a) == 1
+ assert succeeded_a[0]["CallbackSucceededDetails"]["Result"]["Payload"] == (
+ '{"answer":"A"}')
+ # ...and leaves B waiting: no CallbackSucceeded of A's answer on B.
+ assert not [e for e in events(arn_b) if e["EventType"] == "CallbackSucceeded"], (
+ "the second execution received the first execution's answer")
+
+ # B's own id still resolves B.
+ req = urllib.request.Request(
+ f"{_ms_endpoint()}/2025-12-01/durable-execution-callbacks/"
+ f"{quote(cb_b, safe='')}/succeed",
+ method="POST", data=b'{"answer":"B"}',
+ headers={"Content-Type": "application/json"})
+ with urllib.request.urlopen(req) as r:
+ assert r.status == 200
+ succeeded_b = [e for e in events(arn_b) if e["EventType"] == "CallbackSucceeded"]
+ assert len(succeeded_b) == 1
+ assert succeeded_b[0]["CallbackSucceededDetails"]["Result"]["Payload"] == (
+ '{"answer":"B"}')
+ finally:
+ try:
+ lam.delete_function(FunctionName=fname)
+ except Exception:
+ pass
+
+
def test_lambda_durable_send_callback_success_then_already_closed(lam):
"""First succeed returns 200; second call against the same closed callback
must return CallbackTimeoutException (400) per the spec."""
@@ -11588,6 +11787,7 @@ def test_extract_cache_sweep_is_reference_based(monkeypatch, tmp_path):
from stored CodeSha256 values (base64), never by re-hashing blobs."""
import base64 as _b64
import hashlib as _hashlib
+
from ministack.core.responses import AccountRegionScopedDict
_fresh_extract_cache(monkeypatch, tmp_path)
code_a, code_b = _make_zip("def handler(e,c): return 1"), _make_zip("def handler(e,c): return 2")
@@ -11894,3 +12094,989 @@ def test_lambda_rie_sentinel_arn_resolves_in_caller_scope():
lsvc._functions.pop_scoped(account_id, "eu-central-1", function_name, None)
set_request_account_id(original_account)
set_request_region(original_region)
+
+
+# ---------------------------------------------------------------------------
+# provided.* (custom runtime) local execution: warm environment and dispatch
+#
+# Every test drives a *real* bootstrap — a stdlib-only Python script that
+# speaks the Lambda Runtime API over ``AWS_LAMBDA_RUNTIME_API`` exactly as a Go
+# or Rust binary would. No Go toolchain, no Docker, no running MiniStack.
+# ---------------------------------------------------------------------------
+
+from ministack.core import lambda_runtime
+from ministack.services import lambda_svc
+
+# ---------------------------------------------------------------------------
+# Bootstrap fixtures (stdlib only, run by the host Python interpreter)
+# ---------------------------------------------------------------------------
+
+_PREAMBLE = '''#!{python}
+import json, os, sys, time, urllib.request, urllib.error
+
+API = os.environ["AWS_LAMBDA_RUNTIME_API"]
+PORT = int(API.rsplit(":", 1)[1])
+
+
+def _call(path, data=None, method=None):
+ url = "http://%s/2018-06-01%s" % (API, path)
+ req = urllib.request.Request(url, data=data, method=method)
+ try:
+ resp = urllib.request.urlopen(req, timeout=30)
+ return resp.getcode(), resp.headers, resp.read()
+ except urllib.error.HTTPError as exc:
+ return exc.code, exc.headers, exc.read()
+
+
+def next_invocation():
+ code, headers, body = _call("/runtime/invocation/next")
+ if code != 200:
+ raise SystemExit(0)
+ return headers, json.loads(body or b"null")
+
+
+def post_response(rid, payload):
+ return _call("/runtime/invocation/%s/response" % rid,
+ data=json.dumps(payload).encode(), method="POST")[0]
+
+
+def post_error(rid, payload):
+ return _call("/runtime/invocation/%s/error" % rid,
+ data=json.dumps(payload).encode(), method="POST")[0]
+
+
+def post_init_error(payload):
+ return _call("/runtime/init/error",
+ data=json.dumps(payload).encode(), method="POST")[0]
+'''
+
+
+def _bootstrap(body: str) -> bytes:
+ """Zip a bootstrap script whose body follows the Runtime API preamble."""
+ script = _PREAMBLE.format(python=sys.executable) + "\n" + body
+ buf = io.BytesIO()
+ with zipfile.ZipFile(buf, "w") as archive:
+ archive.writestr("bootstrap", script)
+ return buf.getvalue()
+
+
+ECHO_BOOTSTRAP = _bootstrap('''
+n = 0
+while True:
+ headers, event = next_invocation()
+ n += 1
+ rid = headers["Lambda-Runtime-Aws-Request-Id"]
+ post_response(rid, {
+ "pid": os.getpid(),
+ "n": n,
+ "port": PORT,
+ "event": event,
+ "request_id": rid,
+ "trace": headers.get("Lambda-Runtime-Trace-Id"),
+ "arn": headers.get("Lambda-Runtime-Invoked-Function-Arn"),
+ "deadline": headers.get("Lambda-Runtime-Deadline-Ms"),
+ })
+''')
+
+
+def _provided_worker_config(**overrides):
+ config = {
+ "Runtime": "provided.al2023",
+ "Handler": "bootstrap",
+ "FunctionName": "provided-fn",
+ "FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:provided-fn",
+ "Timeout": 10,
+ "MemorySize": 128,
+ }
+ config.update(overrides)
+ return config
+
+
+def isolated_runtime(monkeypatch, tmp_path):
+ """Pool isolation for the provided-runtime tests.
+
+ Requested explicitly rather than autouse: these tests now sit beside the
+ rest of the Lambda suite, which shares the warm pool with the live server.
+ """
+ from ministack.services import lambda_svc
+
+ monkeypatch.setattr(lambda_runtime, "_workers", {})
+ monkeypatch.setattr(lambda_svc, "_PROVIDED_CODE_CACHE", str(tmp_path / "code"))
+ monkeypatch.setattr(lambda_svc, "_provided_code_dirs", {})
+ yield
+ lambda_runtime.reset()
+
+
+def worker_factory(isolated_runtime):
+ """Build ProvidedWorkers and guarantee they are torn down."""
+ built: list = []
+
+ def _make(code_zip: bytes, **config_overrides) -> lambda_runtime.ProvidedWorker:
+ worker = lambda_runtime.ProvidedWorker("provided-fn", _provided_worker_config(**config_overrides), code_zip)
+ built.append(worker)
+ return worker
+
+ yield _make
+ lambda_runtime.kill_workers(built)
+ lambda_runtime.reset()
+
+
+def _port_is_closed(port: int) -> bool:
+ for _ in range(50):
+ sock = socket.socket()
+ sock.settimeout(0.5)
+ try:
+ sock.connect(("127.0.0.1", port))
+ except OSError:
+ return True
+ finally:
+ sock.close()
+ time.sleep(0.05)
+ return False
+
+
+# ---------------------------------------------------------------------------
+# Reuse + per-invocation trace context
+# ---------------------------------------------------------------------------
+
+
+def test_environment_is_reused_and_trace_is_per_invocation(worker_factory):
+ """Second invocation lands on the same process, with its own trace ID."""
+ worker = worker_factory(ECHO_BOOTSTRAP)
+
+ first = worker.invoke({"k": 1}, "req-1", trace_id="Root=1-aaa;Sampled=1")
+ second = worker.invoke({"k": 2}, "req-2", trace_id="Root=1-bbb;Sampled=1")
+ third = worker.invoke({"k": 3}, "req-3")
+
+ assert first["status"] == "ok" and second["status"] == "ok"
+ assert first["cold_start"] is True
+ assert second["cold_start"] is False and third["cold_start"] is False
+
+ assert first["result"]["pid"] == second["result"]["pid"] == third["result"]["pid"]
+ assert [r["result"]["n"] for r in (first, second, third)] == [1, 2, 3]
+
+ # Request IDs and trace IDs are per invocation, never carried over.
+ assert first["result"]["request_id"] == "req-1"
+ assert second["result"]["request_id"] == "req-2"
+ assert first["result"]["trace"] == "Root=1-aaa;Sampled=1"
+ assert second["result"]["trace"] == "Root=1-bbb;Sampled=1"
+ assert third["result"]["trace"] is None
+ assert first["result"]["arn"] == worker.config["FunctionArn"]
+
+
+def test_trace_id_does_not_touch_the_event_payload(worker_factory):
+ """The event is the caller's; trace context rides the Runtime API header.
+
+ A user event that happens to contain the key the old implementation
+ reserved must reach the handler untouched.
+ """
+ worker = worker_factory(ECHO_BOOTSTRAP)
+ event = {"_x_amzn_trace_id": "user-supplied", "hello": "world"}
+
+ result = worker.invoke(event, "req-1", trace_id="Root=1-ccc")
+
+ assert result["status"] == "ok"
+ # Delivered verbatim ...
+ assert result["result"]["event"] == {
+ "_x_amzn_trace_id": "user-supplied", "hello": "world"}
+ # ... and the header carries the real trace ID.
+ assert result["result"]["trace"] == "Root=1-ccc"
+ # The caller's dict was not mutated.
+ assert event == {"_x_amzn_trace_id": "user-supplied", "hello": "world"}
+
+
+# ---------------------------------------------------------------------------
+# Logs: both pipes drained continuously, buffer bounded
+# ---------------------------------------------------------------------------
+
+
+CHATTY_BOOTSTRAP = _bootstrap('''
+LINE = "x" * 120
+while True:
+ headers, event = next_invocation()
+ rid = headers["Lambda-Runtime-Aws-Request-Id"]
+ count = event["lines"]
+ for i in range(count):
+ sys.stdout.write("OUT-%d %s\\n" % (i, LINE))
+ sys.stderr.write("ERR-%d %s\\n" % (i, LINE))
+ sys.stdout.flush()
+ sys.stderr.flush()
+ time.sleep(0.2)
+ post_response(rid, {"logged": count})
+''')
+
+
+def test_large_log_volume_does_not_wedge_the_bootstrap(worker_factory):
+ """~500KB across both pipes: an undrained pipe would block the child.
+
+ stdout is drained as logs too — a custom runtime writes there — and both
+ streams are merged into one log buffer.
+ """
+ worker = worker_factory(CHATTY_BOOTSTRAP, Timeout=30)
+
+ result = worker.invoke({"lines": 2000}, "req-1")
+
+ assert result["status"] == "ok"
+ assert result["result"] == {"logged": 2000}
+ log = result["log"]
+ assert "OUT-0 " in log and "ERR-0 " in log
+ assert "OUT-1999 " in log and "ERR-1999 " in log
+
+ # And the environment survives it.
+ again = worker.invoke({"lines": 1}, "req-2")
+ assert again["status"] == "ok"
+ assert again["cold_start"] is False
+
+
+def test_log_buffer_is_bounded_and_keeps_the_newest_lines(worker_factory, monkeypatch):
+ monkeypatch.setattr(lambda_runtime, "_PROVIDED_LOG_MAX_LINES", 50)
+ worker = worker_factory(CHATTY_BOOTSTRAP, Timeout=30)
+
+ result = worker.invoke({"lines": 400}, "req-1")
+
+ assert result["status"] == "ok"
+ lines = [line for line in result["log"].splitlines() if line]
+ assert len(lines) <= 50
+ # Oldest dropped, newest kept.
+ assert "OUT-0 " not in result["log"]
+ assert any(line.startswith("ERR-399 ") or line.startswith("OUT-399 ")
+ for line in lines)
+
+
+# ---------------------------------------------------------------------------
+# Crash / restart
+# ---------------------------------------------------------------------------
+
+
+CRASH_BOOTSTRAP = _bootstrap('''
+headers, event = next_invocation()
+rid = headers["Lambda-Runtime-Aws-Request-Id"]
+post_response(rid, {"pid": os.getpid(), "port": PORT})
+sys.stderr.write("bye\\n")
+sys.stderr.flush()
+os._exit(9)
+''')
+
+
+def test_crashed_bootstrap_is_replaced_without_leaking_its_server(worker_factory):
+ worker = worker_factory(CRASH_BOOTSTRAP)
+ threads_before = threading.active_count()
+
+ first = worker.invoke({}, "req-1")
+ assert first["status"] == "ok"
+ first_port = first["result"]["port"]
+ first_pid = first["result"]["pid"]
+
+ # The bootstrap exits right after responding; the next invocation gets a
+ # brand new environment, and the dead one's HTTP server is gone.
+ second = worker.invoke({}, "req-2")
+ assert second["status"] == "ok"
+ assert second["cold_start"] is True
+ assert second["result"]["pid"] != first_pid
+ assert second["result"]["port"] != first_port
+ assert _port_is_closed(first_port), "leaked Runtime API server from dead generation"
+
+ worker.kill()
+ assert _port_is_closed(second["result"]["port"])
+ # Two generations, no accumulated threads.
+ for _ in range(50):
+ if threading.active_count() <= threads_before + 1:
+ break
+ time.sleep(0.1)
+ assert threading.active_count() <= threads_before + 1
+
+
+# ---------------------------------------------------------------------------
+# Init failures are detected promptly
+# ---------------------------------------------------------------------------
+
+
+INIT_ERROR_BOOTSTRAP = _bootstrap('''
+sys.stderr.write("init blew up\\n")
+sys.stderr.flush()
+post_init_error({"errorMessage": "cannot load config", "errorType": "Init.Error"})
+sys.exit(1)
+''')
+
+INSTANT_EXIT_BOOTSTRAP = _bootstrap('''
+sys.stderr.write("missing shared library\\n")
+sys.stderr.flush()
+sys.exit(3)
+''')
+
+
+def test_init_error_fails_fast_and_leaves_nothing_running(worker_factory):
+ worker = worker_factory(INIT_ERROR_BOOTSTRAP)
+
+ started = time.monotonic()
+ with pytest.raises(RuntimeError, match="init error"):
+ worker.invoke({}, "req-1")
+ elapsed = time.monotonic() - started
+
+ assert elapsed < lambda_runtime._PROVIDED_INIT_TIMEOUT / 2
+ assert worker._proc is None
+ assert worker._server is None
+ assert worker._server_thread is None
+
+
+def test_bootstrap_that_exits_immediately_fails_fast(worker_factory):
+ worker = worker_factory(INSTANT_EXIT_BOOTSTRAP)
+
+ started = time.monotonic()
+ with pytest.raises(RuntimeError, match="exited during init with code 3"):
+ worker.invoke({}, "req-1")
+ elapsed = time.monotonic() - started
+
+ assert elapsed < lambda_runtime._PROVIDED_INIT_TIMEOUT / 2
+ assert worker._proc is None and worker._server is None
+
+
+def test_init_overrun_reruns_init_under_the_function_timeout(worker_factory, monkeypatch):
+ """AWS does not fail the call when init overruns its budget: "Lambda
+ retries the Init phase at the time of the first function invocation with
+ the configured function timeout". Only the second overrun gives up."""
+ monkeypatch.setattr(lambda_runtime, "_PROVIDED_INIT_TIMEOUT", 0.5)
+ silent = _bootstrap('time.sleep(60)\n')
+ worker = worker_factory(silent, Timeout=2)
+
+ started = time.monotonic()
+ with pytest.raises(lambda_runtime.ProvidedInitTimeout,
+ match="did not reach the Runtime API"):
+ worker.invoke({}, "req-1")
+ elapsed = time.monotonic() - started
+
+ # Both budgets were spent: the 0.5s init, then a re-init under Timeout=2.
+ assert 2.5 <= elapsed < 10.0
+ assert worker._proc is None and worker._server is None
+
+
+def test_a_missing_bootstrap_is_runtime_invalid_entrypoint(worker_factory):
+ """"If the bootstrap file doesn't exist or isn't executable, your function
+ returns a Runtime.InvalidEntrypoint error upon invocation"."""
+ buf = io.BytesIO()
+ with zipfile.ZipFile(buf, "w") as archive:
+ archive.writestr("not-bootstrap", "noop")
+ worker = worker_factory(buf.getvalue())
+
+ with pytest.raises(lambda_runtime.ProvidedRuntimeError) as excinfo:
+ worker.invoke({}, "req-1")
+ assert excinfo.value.error_type == "Runtime.InvalidEntrypoint"
+
+
+# ---------------------------------------------------------------------------
+# Timeout + recovery
+# ---------------------------------------------------------------------------
+
+
+SLOW_ONCE_BOOTSTRAP_TEMPLATE = '''
+MARKER = {marker!r}
+while True:
+ headers, event = next_invocation()
+ rid = headers["Lambda-Runtime-Aws-Request-Id"]
+ if not os.path.exists(MARKER):
+ open(MARKER, "w").close()
+ sys.stderr.write("going to sleep\\n")
+ sys.stderr.flush()
+ time.sleep(30)
+ post_response(rid, {{"pid": os.getpid(), "port": PORT}})
+'''
+
+
+def test_timeout_kills_environment_and_next_invocation_recovers(worker_factory, tmp_path):
+ marker = str(tmp_path / "slept")
+ worker = worker_factory(
+ _bootstrap(SLOW_ONCE_BOOTSTRAP_TEMPLATE.format(marker=marker)),
+ Timeout=1,
+ )
+
+ started = time.monotonic()
+ timed_out = worker.invoke({}, "req-1")
+ elapsed = time.monotonic() - started
+
+ assert timed_out["status"] == "error"
+ assert timed_out["error"] == "Task timed out after 1.00 seconds"
+ assert "going to sleep" in timed_out["log"]
+ assert elapsed < 8.0
+ assert worker._proc is None and worker._server is None
+
+ # A fresh environment serves the next invocation, and the marker makes the
+ # replacement take the fast path.
+ recovered = worker.invoke({}, "req-2")
+ assert recovered["status"] == "ok"
+ assert recovered["cold_start"] is True
+
+
+# ---------------------------------------------------------------------------
+# Request-ID validation and stale-response isolation
+# ---------------------------------------------------------------------------
+
+
+BAD_REQUEST_ID_BOOTSTRAP = _bootstrap('''
+while True:
+ headers, event = next_invocation()
+ rid = headers["Lambda-Runtime-Aws-Request-Id"]
+ codes = {
+ "bogus": post_response("not-a-real-request-id", {"stolen": True}),
+ "empty": post_error("", {"errorMessage": "nope"}),
+ }
+ codes["real"] = post_response(rid, {"codes": codes, "request_id": rid})
+ if codes["real"] != 202:
+ post_error(rid, {"errorMessage": "response rejected: %s" % codes})
+''')
+
+
+def test_unknown_request_ids_are_rejected_not_delivered(worker_factory):
+ worker = worker_factory(BAD_REQUEST_ID_BOOTSTRAP)
+
+ result = worker.invoke({}, "req-1")
+
+ assert result["status"] == "ok"
+ # The forged IDs got 400s; only the real one was accepted.
+ assert result["result"]["codes"] == {"bogus": 400, "empty": 404}
+ assert result["result"]["request_id"] == "req-1"
+
+
+DUPLICATE_RESPONSE_BOOTSTRAP = _bootstrap('''
+while True:
+ headers, event = next_invocation()
+ rid = headers["Lambda-Runtime-Aws-Request-Id"]
+ first = post_response(rid, {"which": "first"})
+ second = post_response(rid, {"which": "second", "first_code": first})
+ sys.stderr.write("dup-code=%s\\n" % second)
+ sys.stderr.flush()
+''')
+
+
+def test_duplicate_response_for_same_request_id_is_rejected(worker_factory):
+ worker = worker_factory(DUPLICATE_RESPONSE_BOOTSTRAP)
+
+ first = worker.invoke({}, "req-1")
+ assert first["status"] == "ok"
+ assert first["result"] == {"which": "first"}
+
+ # The duplicate got a 400 and, crucially, did not become the *next*
+ # invocation's result.
+ second = worker.invoke({}, "req-2")
+ assert second["status"] == "ok"
+ assert second["result"] == {"which": "first"}
+ assert "dup-code=400" in "\n".join([first["log"], second["log"]])
+
+
+def test_results_from_a_dead_generation_are_ignored(isolated_runtime):
+ """A late POST from a torn-down environment cannot satisfy its successor."""
+ worker = lambda_runtime.ProvidedWorker("provided-fn", _provided_worker_config(), b"")
+ worker._generation = 7
+ worker._current_request_id = "req-live"
+
+ # Same request ID, previous generation.
+ assert worker._record_result(6, "req-live", "response", {"stale": True}) is False
+ assert worker._response_ready.is_set() is False
+ assert worker._result == {}
+
+ # Wrong request ID, current generation.
+ assert worker._record_result(7, "req-old", "response", {"stale": True}) is False
+ assert worker._response_ready.is_set() is False
+
+ # The live one is accepted exactly once.
+ assert worker._record_result(7, "req-live", "response", {"ok": True}) is True
+ assert worker._result == {"response": {"ok": True}}
+ assert worker._record_result(7, "req-live", "response", {"again": True}) is False
+ assert worker._result == {"response": {"ok": True}}
+
+
+def test_init_error_from_a_dead_generation_is_ignored(isolated_runtime):
+ worker = lambda_runtime.ProvidedWorker("provided-fn", _provided_worker_config(), b"")
+ worker._generation = 3
+
+ worker._record_init_error(2, {"errorMessage": "stale"})
+ assert worker._init_error is None
+ assert worker._init_settled.is_set() is False
+
+ worker._record_init_error(3, {"errorMessage": "live"})
+ assert worker._init_error == {"errorMessage": "live"}
+ assert worker._init_settled.is_set() is True
+
+
+# ---------------------------------------------------------------------------
+# Cleanup / pool contract
+# ---------------------------------------------------------------------------
+
+
+def test_kill_reaps_process_server_and_threads(worker_factory):
+ worker = worker_factory(ECHO_BOOTSTRAP)
+ threads_before = threading.active_count()
+
+ result = worker.invoke({}, "req-1")
+ port = result["result"]["port"]
+ proc = worker._proc
+ log_thread = worker._log_thread
+ assert proc.poll() is None
+
+ worker.kill()
+
+ assert proc.poll() is not None, "bootstrap process not reaped"
+ assert worker._proc is None
+ assert worker._server is None and worker._server_thread is None
+ assert worker._log_thread is None
+ assert _port_is_closed(port), "Runtime API socket still listening after kill"
+ log_thread.join(timeout=2.0)
+ assert not log_thread.is_alive()
+ for _ in range(50):
+ if threading.active_count() <= threads_before:
+ break
+ time.sleep(0.1)
+ assert threading.active_count() <= threads_before
+
+
+def test_pool_contract_lease_reuse_and_reset(isolated_runtime):
+ """provided.* functions use the same acquire/release pool as the rest."""
+ config = _provided_worker_config()
+ try:
+ worker, reason = lambda_runtime.acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
+ assert isinstance(worker, lambda_runtime.ProvidedWorker)
+ assert reason == "spawn"
+ assert worker.in_use is True
+
+ first = worker.invoke({}, "req-1")
+ assert first["status"] == "ok"
+ port = first["result"]["port"]
+
+ # A second lease while the first is held gets a *separate* environment.
+ other, other_reason = lambda_runtime.acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
+ assert other is not worker and other_reason == "spawn"
+ lambda_runtime.release_worker(other)
+
+ lambda_runtime.release_worker(worker)
+ again, reason = lambda_runtime.acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
+ assert reason == "reused"
+ assert again is worker
+ assert again.invoke({}, "req-2")["result"]["port"] == port
+ lambda_runtime.release_worker(again)
+ finally:
+ lambda_runtime.reset()
+
+ assert _port_is_closed(port)
+
+
+def test_concurrent_invocations_use_separate_leased_environments(isolated_runtime):
+ """Two leases run at the same time on two processes, results not crossed."""
+ config = _provided_worker_config()
+ barrier = threading.Barrier(2, timeout=30)
+ results: dict = {}
+
+ try:
+ workers = []
+ for _ in range(2):
+ worker, reason = lambda_runtime.acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
+ assert reason == "spawn"
+ workers.append(worker)
+
+ def _run(index, worker):
+ barrier.wait()
+ results[index] = worker.invoke({"i": index}, f"req-{index}",
+ trace_id=f"Root=1-{index}")
+
+ threads = [threading.Thread(target=_run, args=(i, w))
+ for i, w in enumerate(workers)]
+ for thread in threads:
+ thread.start()
+ for thread in threads:
+ thread.join(timeout=60)
+ assert not thread.is_alive()
+
+ assert set(results) == {0, 1}
+ assert all(r["status"] == "ok" for r in results.values())
+ assert results[0]["result"]["pid"] != results[1]["result"]["pid"]
+ for index, result in results.items():
+ assert result["result"]["event"] == {"i": index}
+ assert result["result"]["request_id"] == f"req-{index}"
+ assert result["result"]["trace"] == f"Root=1-{index}"
+ for worker in workers:
+ lambda_runtime.release_worker(worker)
+ finally:
+ lambda_runtime.reset()
+
+ for result in results.values():
+ assert _port_is_closed(result["result"]["port"])
+
+
+def test_idle_reaper_and_reset_release_provided_environments(isolated_runtime):
+ config = _provided_worker_config()
+ try:
+ first, _ = lambda_runtime.acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
+ second, _ = lambda_runtime.acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
+ ports = [w.invoke({}, "req-1")["result"]["port"] for w in (first, second)]
+ procs = [first._proc, second._proc]
+ lambda_runtime.release_worker(first)
+ lambda_runtime.release_worker(second)
+
+ # ttl=0 reaps every surplus environment; the first per key stays warm.
+ assert lambda_runtime.reap_idle_workers(ttl=0) == 1
+ assert procs[1].poll() is not None
+ assert _port_is_closed(ports[1])
+ assert second._server is None
+ assert procs[0].poll() is None
+
+ lambda_runtime.reset()
+ assert procs[0].poll() is not None
+ assert _port_is_closed(ports[0])
+ assert first._server is None and first._log_thread is None
+ finally:
+ lambda_runtime.reset()
+
+
+def test_invoke_signature_keeps_trace_out_of_the_positional_contract(isolated_runtime):
+ """``trace_id`` is keyword-only, so no caller can pass it as the event."""
+ import inspect
+
+ sig = inspect.signature(lambda_runtime.ProvidedWorker.invoke)
+ assert list(sig.parameters) == ["self", "event", "request_id", "trace_id"]
+ assert sig.parameters["trace_id"].kind is inspect.Parameter.KEYWORD_ONLY
+ assert sig.parameters["trace_id"].default is None
+
+
+def test_json_only_bootstrap_output_is_not_parsed_as_protocol(worker_factory):
+ """stdout is logs, not a protocol channel: JSON there must not confuse us."""
+ noisy = _bootstrap('''
+while True:
+ headers, event = next_invocation()
+ rid = headers["Lambda-Runtime-Aws-Request-Id"]
+ sys.stdout.write(json.dumps({"status": "ok", "result": "from-stdout"}) + "\\n")
+ sys.stdout.flush()
+ post_response(rid, {"real": True})
+''')
+ worker = worker_factory(noisy)
+
+ result = worker.invoke({}, "req-1")
+
+ assert result["status"] == "ok"
+ assert result["result"] == {"real": True}
+ assert "from-stdout" in result["log"]
+
+
+def test_handler_error_does_not_poison_warm_environment(worker_factory):
+ code = _bootstrap('''
+n = 0
+while True:
+ headers, event = next_invocation()
+ n += 1
+ rid = headers["Lambda-Runtime-Aws-Request-Id"]
+ if event.get("fail"):
+ post_error(rid, {"errorMessage": "handler failed", "errorType": "Test.Error"})
+ else:
+ post_response(rid, {"n": n})
+''')
+ worker = worker_factory(code)
+ failed = worker.invoke({"fail": True}, "req-1")
+ assert failed["status"] == "error"
+ assert failed["error_payload"]["errorType"] == "Test.Error"
+ recovered = worker.invoke({}, "req-2")
+ assert recovered["status"] == "ok"
+ assert recovered["cold_start"] is False
+ assert recovered["result"] == {"n": 2}
+
+
+def test_exit_during_invocation_fails_without_waiting_for_timeout(worker_factory):
+ worker = worker_factory(_bootstrap('''
+next_invocation()
+sys.stderr.write("crashed during handler\\n")
+sys.stderr.flush()
+sys.exit(9)
+'''), Timeout=30)
+ started = time.monotonic()
+ result = worker.invoke({}, "req-1")
+ assert time.monotonic() - started < 8
+ assert result["status"] == "error"
+ assert result["error_payload"]["errorType"] == "Runtime.ExitError"
+ assert "crashed during handler" in result["log"]
+ assert worker._proc is None and worker._server is None
+
+
+def test_response_recorded_just_after_process_exit_is_preserved(worker_factory, monkeypatch):
+ from unittest.mock import Mock
+
+ worker = worker_factory(ECHO_BOOTSTRAP)
+ proc = Mock()
+ statuses = iter([None, 0])
+ proc.poll.side_effect = lambda: next(statuses, 0)
+ worker._proc = proc
+ ready = Mock()
+ waits = iter([False, True])
+
+ def wait(timeout):
+ settled = next(waits)
+ if settled:
+ worker._result = {"response": {"ok": True}}
+ return settled
+
+ ready.wait.side_effect = wait
+ worker._response_ready = ready
+ monkeypatch.setattr(worker, "_drain_stderr_bounded", lambda: "")
+ result = worker.invoke({}, "req-1")
+ assert result["status"] == "ok"
+ assert result["result"] == {"ok": True}
+ assert ready.wait.call_count == 2
+ assert worker._proc is None
+
+
+def test_spawn_failure_closes_the_already_started_server(worker_factory, monkeypatch):
+ worker = worker_factory(ECHO_BOOTSTRAP)
+ resources = []
+
+ def failed_popen(*args, **kwargs):
+ resources.extend([worker._server, worker._server_thread])
+ raise OSError("cannot execute bootstrap")
+
+ monkeypatch.setattr(lambda_runtime.subprocess, "Popen", failed_popen)
+ # A bootstrap the host cannot exec is Runtime.InvalidEntrypoint on AWS, so
+ # the OSError is re-raised carrying that type rather than as itself.
+ with pytest.raises(lambda_runtime.ProvidedRuntimeError,
+ match="cannot execute bootstrap") as excinfo:
+ worker.invoke({}, "req-1")
+ assert excinfo.value.error_type == "Runtime.InvalidEntrypoint"
+ server, thread = resources
+ assert server.fileno() == -1
+ assert not thread.is_alive()
+ assert worker._proc is None and worker._server is None
+
+
+def _provided_dispatch_config(account="111122223333", region="us-east-1", version="$LATEST"):
+ return {
+ "FunctionName": "provided-pool-test",
+ "FunctionArn": f"arn:aws:lambda:{region}:{account}:function:provided-pool-test",
+ "Runtime": "provided.al2023",
+ "Version": version,
+ "Timeout": 2,
+ "Handler": "bootstrap",
+ }
+
+
+def isolated_pool(monkeypatch):
+ monkeypatch.setattr(lambda_runtime, "_workers", {})
+ monkeypatch.setattr(lambda_svc, "_ensure_reaper_thread", lambda: None)
+ try:
+ yield
+ finally:
+ lambda_runtime.reset()
+
+
+ {"_x_amzn_trace_id": "user data", "value": 1}, [1, 2], "text", None,
+])
+def test_provided_metadata_does_not_mutate_payload(monkeypatch, event):
+ worker = Mock()
+ worker.invoke.return_value = {"status": "ok", "result": event, "log": "handler log"}
+ monkeypatch.setattr(lambda_svc, "_ensure_reaper_thread", lambda: None)
+ monkeypatch.setattr(lambda_svc, "acquire_worker", Mock(return_value=(worker, "spawn")))
+ release = Mock()
+ monkeypatch.setattr(lambda_svc, "release_worker", release)
+ monkeypatch.setattr(lambda_svc, "_xray_trace_id_for_invocation", lambda config: "trace-1")
+ result = lambda_svc._execute_function_provided_warm(
+ {"config": _provided_dispatch_config(), "code_zip": b"zip"}, event, "request-1",
+ )
+ worker.invoke.assert_called_once_with(event, "request-1", trace_id="trace-1")
+ release.assert_called_once_with(worker)
+ assert result == {"body": event, "log": "handler log"}
+ if isinstance(event, dict):
+ assert event["_x_amzn_trace_id"] == "user data"
+
+
+def test_provided_failure_is_scoped_and_not_retried(monkeypatch, isolated_pool):
+ config = _provided_dispatch_config()
+ failed, _ = lambda_runtime.acquire_worker(config["FunctionName"], config, b"zip")
+ lambda_runtime.release_worker(failed)
+ fail = Mock(side_effect=RuntimeError("bootstrap crashed"))
+ monkeypatch.setattr(failed, "invoke", fail)
+ unrelated = []
+ for account, region, version in [
+ ("999900001111", "us-east-1", "$LATEST"),
+ ("111122223333", "eu-west-1", "$LATEST"),
+ ("111122223333", "us-east-1", "1"),
+ ]:
+ other = _provided_dispatch_config(account, region, version)
+ worker, _ = lambda_runtime.acquire_worker(
+ other["FunctionName"], other, b"zip", qualifier=version,
+ )
+ unrelated.append(worker)
+ fallback = Mock()
+ monkeypatch.setattr(lambda_svc, "_execute_function_provided", fallback)
+ result = lambda_svc._execute_function_provided_warm({"config": config, "code_zip": b"zip"}, {})
+ assert result["error"] is True
+ assert result["body"]["errorMessage"] == "bootstrap crashed"
+ assert fail.call_count == 1
+ fallback.assert_not_called()
+ remaining = [w for entries in lambda_runtime._workers.values() for w in entries]
+ assert remaining == unrelated
+ assert all(w.in_use for w in unrelated)
+
+
+def test_provided_concurrent_leases_and_reuse(isolated_pool):
+ config = _provided_dispatch_config()
+ first, _ = lambda_runtime.acquire_worker(config["FunctionName"], config, b"zip")
+ second, _ = lambda_runtime.acquire_worker(config["FunctionName"], config, b"zip")
+ assert isinstance(first, lambda_runtime.ProvidedWorker)
+ assert isinstance(second, lambda_runtime.ProvidedWorker)
+ assert first is not second
+ lambda_runtime.release_worker(first)
+ reused, reason = lambda_runtime.acquire_worker(config["FunctionName"], config, b"zip")
+ assert reused is first
+ assert reason == "reused"
+ lambda_runtime.release_worker(reused)
+ lambda_runtime.release_worker(second)
+
+
[email protected]("mode,target", [
+ ("local", "_execute_function_provided_warm"),
+ ("durable", "_execute_function_provided"),
+ ("docker", "_execute_function_docker"),
+ ("strict", "_execute_function_docker"),
+ ("image", "_execute_function_docker"),
+ ("proxy", "_execute_function_proxy"),
+])
+def test_provided_dispatch_preserves_other_executors(monkeypatch, mode, target):
+ monkeypatch.setattr(lambda_svc, "LAMBDA_EXECUTOR", "docker" if mode == "docker" else "local")
+ monkeypatch.setattr(lambda_svc, "LAMBDA_STRICT", mode == "strict")
+ monkeypatch.setattr(lambda_svc, "_proxy_url_for", lambda config: "http://proxy" if mode == "proxy" else None)
+ monkeypatch.setattr(lambda_svc, "_emit_lambda_logs", Mock())
+ names = ["_execute_function_provided_warm", "_execute_function_provided",
+ "_execute_function_docker", "_execute_function_proxy"]
+ executors = {name: Mock(return_value={"body": name}) for name in names}
+ for name, executor in executors.items():
+ monkeypatch.setattr(lambda_svc, name, executor)
+ config = _provided_dispatch_config()
+ if mode == "image":
+ config.update(PackageType="Image", ImageUri="example:latest")
+ func = {"config": config, "code_zip": b"zip"}
+ token = lambda_svc._durable_ctx.set({"test": True} if mode == "durable" else None)
+ try:
+ result = lambda_svc._execute_function_dispatch(func, config, {}, "request-1", time.time())
+ finally:
+ lambda_svc._durable_ctx.reset(token)
+ assert result == {"body": target}
+ for name, executor in executors.items():
+ assert executor.call_count == (1 if name == target else 0)
+ if mode == "local":
+ executors[target].assert_called_once_with(func, {}, "request-1")
+
+
[email protected]("runtime", ["python3.12", "nodejs20.x"])
[email protected]("durable", [False, True])
+def test_python_and_nodejs_dispatch_to_the_warm_pool_durable_or_not(
+ monkeypatch, runtime, durable):
+ """The sibling of the provided.* routing above: python and nodejs reach
+ _execute_function_warm whether or not the invocation is durable, because
+ their per-call durable context rides in the event rather than in the
+ worker's spawn environment."""
+ monkeypatch.setattr(lambda_svc, "LAMBDA_EXECUTOR", "local")
+ monkeypatch.setattr(lambda_svc, "LAMBDA_STRICT", False)
+ monkeypatch.setattr(lambda_svc, "_proxy_url_for", lambda config: None)
+ monkeypatch.setattr(lambda_svc, "_emit_lambda_logs", Mock())
+ names = ["_execute_function_warm", "_execute_function_local",
+ "_execute_function_provided", "_execute_function_provided_warm",
+ "_execute_function_docker", "_execute_function_proxy"]
+ executors = {name: Mock(return_value={"body": name}) for name in names}
+ for name, executor in executors.items():
+ monkeypatch.setattr(lambda_svc, name, executor)
+ config = _provided_dispatch_config()
+ config["Runtime"] = runtime
+ func = {"config": config, "code_zip": b"zip"}
+ token = lambda_svc._durable_ctx.set({"test": True} if durable else None)
+ try:
+ result = lambda_svc._execute_function_dispatch(func, config, {}, "request-1", time.time())
+ finally:
+ lambda_svc._durable_ctx.reset(token)
+ assert result == {"body": "_execute_function_warm"}
+ for name, executor in executors.items():
+ assert executor.call_count == (1 if name == "_execute_function_warm" else 0)
+
+
[email protected]("unavailable", ["sdk", "daemon"])
+def test_docker_unavailable_fallback_carries_the_durable_context(monkeypatch, unavailable):
+ """Both permissive Docker fallbacks send python and nodejs to the warm
+ pool, so the durable context reaches the handler there too. Before the
+ carrier moved into the event they invoked with the three variables unset."""
+ monkeypatch.setattr(lambda_svc, "_docker_available", unavailable != "sdk")
+ monkeypatch.setattr(lambda_svc, "_get_docker_client",
+ lambda: None if unavailable == "daemon" else object())
+ monkeypatch.setattr(lambda_svc, "LAMBDA_STRICT", False)
+ warm = Mock(return_value={"body": "warm"})
+ one_shot = Mock(return_value={"body": "one-shot"})
+ monkeypatch.setattr(lambda_svc, "_execute_function_warm", warm)
+ monkeypatch.setattr(lambda_svc, "_execute_function_local", one_shot)
+ config = _provided_dispatch_config()
+ config["Runtime"] = "python3.12"
+ func = {"config": config, "code_zip": b"zip"}
+ token = lambda_svc._durable_ctx.set({"arn": "exec-arn", "token": "tok", "name": "exec"})
+ try:
+ assert lambda_svc._execute_function_docker(func, {}) == {"body": "warm"}
+ overlay = lambda_svc._durable_env_overlay()
+ finally:
+ lambda_svc._durable_ctx.reset(token)
+ warm.assert_called_once_with(func, {})
+ one_shot.assert_not_called()
+ assert overlay == {
+ "AWS_LAMBDA_DURABLE_EXECUTION_ARN": "exec-arn",
+ "AWS_LAMBDA_DURABLE_CHECKPOINT_TOKEN": "tok",
+ "AWS_LAMBDA_DURABLE_EXECUTION_NAME": "exec",
+ }
+
+
+def test_provided_env_keeps_function_vars_and_endpoint_precedence(monkeypatch):
+ config = _provided_dispatch_config()
+ config.update(MemorySize=256, Environment={"Variables": {
+ "CUSTOM": "value", "AWS_ENDPOINT_URL": "http://wrong:4566",
+ }})
+ monkeypatch.setenv("AWS_ENDPOINT_URL", "http://ministack:4566")
+ monkeypatch.setattr(lambda_svc, "get_region", lambda: "us-east-1")
+ token = lambda_svc._durable_ctx.set(None)
+ try:
+ env = lambda_svc._provided_worker_env(config, "/code", 12345)
+ finally:
+ lambda_svc._durable_ctx.reset(token)
+ assert env["AWS_LAMBDA_RUNTIME_API"] == "127.0.0.1:12345"
+ assert env["AWS_LAMBDA_FUNCTION_NAME"] == config["FunctionName"]
+ assert env["AWS_LAMBDA_FUNCTION_MEMORY_SIZE"] == "256"
+ assert env["AWS_ACCESS_KEY_ID"] == "111122223333"
+ assert env["AWS_REGION"] == "us-east-1"
+ assert env["LAMBDA_TASK_ROOT"] == "/code"
+ assert env["CUSTOM"] == "value"
+ assert env["AWS_ENDPOINT_URL"] == "http://ministack:4566"
+
+
+def test_provided_env_uses_execution_role_credentials(monkeypatch):
+ config = _provided_dispatch_config()
+ credentials = {
+ "AWS_ACCESS_KEY_ID": "ASIATESTROLE",
+ "AWS_SECRET_ACCESS_KEY": "role-secret",
+ "AWS_SESSION_TOKEN": "role-session",
+ }
+ resolve = Mock(return_value=credentials)
+ monkeypatch.setattr(lambda_svc, "execution_credentials", resolve)
+ env = lambda_svc._provided_worker_env(config, "/code", 12345)
+ resolve.assert_called_once_with(config)
+ assert {key: env[key] for key in credentials} == credentials
+
+
[email protected]("operation", ["code", "configuration", "delete"])
+def test_function_changes_invalidate_provided_workers(monkeypatch, isolated_pool, operation):
+ config = _provided_dispatch_config()
+ name = config["FunctionName"]
+ monkeypatch.setattr(lambda_svc, "_functions", {name: {"config": config, "code_zip": b"zip"}})
+ monkeypatch.setattr(lambda_svc, "get_account_id", lambda: "111122223333")
+ monkeypatch.setattr(lambda_svc, "get_region", lambda: "us-east-1")
+ monkeypatch.setattr(lambda_svc, "_pool_kill_function", Mock())
+ monkeypatch.setattr(lambda_svc, "_sweep_extract_cache", Mock())
+ monkeypatch.setattr(lambda_svc, "_schedule_state_transition", Mock())
+ worker, _ = lambda_runtime.acquire_worker(name, config, b"zip")
+ lambda_runtime.release_worker(worker)
+ if operation == "code":
+ result = lambda_svc._update_code(name, {})
+ elif operation == "configuration":
+ result = lambda_svc._update_config(name, {"Environment": {"Variables": {"UPDATED": "yes"}}})
+ else:
+ result = lambda_svc._delete_function(name, {})
+ assert result[0] in (200, 204)
+ assert not lambda_runtime._workers
diff --git a/tests/test_lambda_provided_dispatch.py b/tests/test_lambda_provided_dispatch.py
deleted file mode 100644
index 80b9b265..00000000
--- a/tests/test_lambda_provided_dispatch.py
+++ /dev/null
@@ -1,249 +0,0 @@
-"""Local custom-runtime routing and pool integration; no running server required."""
-
-import time
-from unittest.mock import Mock
-
-import pytest
-
-from ministack.core import lambda_runtime
-from ministack.services import lambda_svc
-
-
-def _config(account="111122223333", region="us-east-1", version="$LATEST"):
- return {
- "FunctionName": "provided-pool-test",
- "FunctionArn": f"arn:aws:lambda:{region}:{account}:function:provided-pool-test",
- "Runtime": "provided.al2023",
- "Version": version,
- "Timeout": 2,
- "Handler": "bootstrap",
- }
-
-
-def isolated_pool(monkeypatch):
- monkeypatch.setattr(lambda_runtime, "_workers", {})
- monkeypatch.setattr(lambda_svc, "_ensure_reaper_thread", lambda: None)
- try:
- yield
- finally:
- lambda_runtime.reset()
-
-
- {"_x_amzn_trace_id": "user data", "value": 1}, [1, 2], "text", None,
-])
-def test_provided_metadata_does_not_mutate_payload(monkeypatch, event):
- worker = Mock()
- worker.invoke.return_value = {"status": "ok", "result": event, "log": "handler log"}
- monkeypatch.setattr(lambda_svc, "_ensure_reaper_thread", lambda: None)
- monkeypatch.setattr(lambda_svc, "acquire_worker", Mock(return_value=(worker, "spawn")))
- release = Mock()
- monkeypatch.setattr(lambda_svc, "release_worker", release)
- monkeypatch.setattr(lambda_svc, "_xray_trace_id_for_invocation", lambda config: "trace-1")
- result = lambda_svc._execute_function_provided_warm(
- {"config": _config(), "code_zip": b"zip"}, event, "request-1",
- )
- worker.invoke.assert_called_once_with(event, "request-1", trace_id="trace-1")
- release.assert_called_once_with(worker)
- assert result == {"body": event, "log": "handler log"}
- if isinstance(event, dict):
- assert event["_x_amzn_trace_id"] == "user data"
-
-
-def test_provided_failure_is_scoped_and_not_retried(monkeypatch, isolated_pool):
- config = _config()
- failed, _ = lambda_runtime.acquire_worker(config["FunctionName"], config, b"zip")
- lambda_runtime.release_worker(failed)
- fail = Mock(side_effect=RuntimeError("bootstrap crashed"))
- monkeypatch.setattr(failed, "invoke", fail)
- unrelated = []
- for account, region, version in [
- ("999900001111", "us-east-1", "$LATEST"),
- ("111122223333", "eu-west-1", "$LATEST"),
- ("111122223333", "us-east-1", "1"),
- ]:
- other = _config(account, region, version)
- worker, _ = lambda_runtime.acquire_worker(
- other["FunctionName"], other, b"zip", qualifier=version,
- )
- unrelated.append(worker)
- fallback = Mock()
- monkeypatch.setattr(lambda_svc, "_execute_function_provided", fallback)
- result = lambda_svc._execute_function_provided_warm({"config": config, "code_zip": b"zip"}, {})
- assert result["error"] is True
- assert result["body"]["errorMessage"] == "bootstrap crashed"
- assert fail.call_count == 1
- fallback.assert_not_called()
- remaining = [w for entries in lambda_runtime._workers.values() for w in entries]
- assert remaining == unrelated
- assert all(w.in_use for w in unrelated)
-
-
-def test_provided_concurrent_leases_and_reuse(isolated_pool):
- config = _config()
- first, _ = lambda_runtime.acquire_worker(config["FunctionName"], config, b"zip")
- second, _ = lambda_runtime.acquire_worker(config["FunctionName"], config, b"zip")
- assert isinstance(first, lambda_runtime.ProvidedWorker)
- assert isinstance(second, lambda_runtime.ProvidedWorker)
- assert first is not second
- lambda_runtime.release_worker(first)
- reused, reason = lambda_runtime.acquire_worker(config["FunctionName"], config, b"zip")
- assert reused is first
- assert reason == "reused"
- lambda_runtime.release_worker(reused)
- lambda_runtime.release_worker(second)
-
-
[email protected]("mode,target", [
- ("local", "_execute_function_provided_warm"),
- ("durable", "_execute_function_provided"),
- ("docker", "_execute_function_docker"),
- ("strict", "_execute_function_docker"),
- ("image", "_execute_function_docker"),
- ("proxy", "_execute_function_proxy"),
-])
-def test_provided_dispatch_preserves_other_executors(monkeypatch, mode, target):
- monkeypatch.setattr(lambda_svc, "LAMBDA_EXECUTOR", "docker" if mode == "docker" else "local")
- monkeypatch.setattr(lambda_svc, "LAMBDA_STRICT", mode == "strict")
- monkeypatch.setattr(lambda_svc, "_proxy_url_for", lambda config: "http://proxy" if mode == "proxy" else None)
- monkeypatch.setattr(lambda_svc, "_emit_lambda_logs", Mock())
- names = ["_execute_function_provided_warm", "_execute_function_provided",
- "_execute_function_docker", "_execute_function_proxy"]
- executors = {name: Mock(return_value={"body": name}) for name in names}
- for name, executor in executors.items():
- monkeypatch.setattr(lambda_svc, name, executor)
- config = _config()
- if mode == "image":
- config.update(PackageType="Image", ImageUri="example:latest")
- func = {"config": config, "code_zip": b"zip"}
- token = lambda_svc._durable_ctx.set({"test": True} if mode == "durable" else None)
- try:
- result = lambda_svc._execute_function_dispatch(func, config, {}, "request-1", time.time())
- finally:
- lambda_svc._durable_ctx.reset(token)
- assert result == {"body": target}
- for name, executor in executors.items():
- assert executor.call_count == (1 if name == target else 0)
- if mode == "local":
- executors[target].assert_called_once_with(func, {}, "request-1")
-
-
[email protected]("runtime", ["python3.12", "nodejs20.x"])
[email protected]("durable", [False, True])
-def test_python_and_nodejs_dispatch_to_the_warm_pool_durable_or_not(
- monkeypatch, runtime, durable):
- """The sibling of the provided.* routing above: python and nodejs reach
- _execute_function_warm whether or not the invocation is durable, because
- their per-call durable context rides in the event rather than in the
- worker's spawn environment."""
- monkeypatch.setattr(lambda_svc, "LAMBDA_EXECUTOR", "local")
- monkeypatch.setattr(lambda_svc, "LAMBDA_STRICT", False)
- monkeypatch.setattr(lambda_svc, "_proxy_url_for", lambda config: None)
- monkeypatch.setattr(lambda_svc, "_emit_lambda_logs", Mock())
- names = ["_execute_function_warm", "_execute_function_local",
- "_execute_function_provided", "_execute_function_provided_warm",
- "_execute_function_docker", "_execute_function_proxy"]
- executors = {name: Mock(return_value={"body": name}) for name in names}
- for name, executor in executors.items():
- monkeypatch.setattr(lambda_svc, name, executor)
- config = _config()
- config["Runtime"] = runtime
- func = {"config": config, "code_zip": b"zip"}
- token = lambda_svc._durable_ctx.set({"test": True} if durable else None)
- try:
- result = lambda_svc._execute_function_dispatch(func, config, {}, "request-1", time.time())
- finally:
- lambda_svc._durable_ctx.reset(token)
- assert result == {"body": "_execute_function_warm"}
- for name, executor in executors.items():
- assert executor.call_count == (1 if name == "_execute_function_warm" else 0)
-
-
[email protected]("unavailable", ["sdk", "daemon"])
-def test_docker_unavailable_fallback_carries_the_durable_context(monkeypatch, unavailable):
- """Both permissive Docker fallbacks send python and nodejs to the warm
- pool, so the durable context reaches the handler there too. Before the
- carrier moved into the event they invoked with the three variables unset."""
- monkeypatch.setattr(lambda_svc, "_docker_available", unavailable != "sdk")
- monkeypatch.setattr(lambda_svc, "_get_docker_client",
- lambda: None if unavailable == "daemon" else object())
- monkeypatch.setattr(lambda_svc, "LAMBDA_STRICT", False)
- warm = Mock(return_value={"body": "warm"})
- one_shot = Mock(return_value={"body": "one-shot"})
- monkeypatch.setattr(lambda_svc, "_execute_function_warm", warm)
- monkeypatch.setattr(lambda_svc, "_execute_function_local", one_shot)
- config = _config()
- config["Runtime"] = "python3.12"
- func = {"config": config, "code_zip": b"zip"}
- token = lambda_svc._durable_ctx.set({"arn": "exec-arn", "token": "tok", "name": "exec"})
- try:
- assert lambda_svc._execute_function_docker(func, {}) == {"body": "warm"}
- overlay = lambda_svc._durable_env_overlay()
- finally:
- lambda_svc._durable_ctx.reset(token)
- warm.assert_called_once_with(func, {})
- one_shot.assert_not_called()
- assert overlay == {
- "AWS_LAMBDA_DURABLE_EXECUTION_ARN": "exec-arn",
- "AWS_LAMBDA_DURABLE_CHECKPOINT_TOKEN": "tok",
- "AWS_LAMBDA_DURABLE_EXECUTION_NAME": "exec",
- }
-
-
-def test_provided_env_keeps_function_vars_and_endpoint_precedence(monkeypatch):
- config = _config()
- config.update(MemorySize=256, Environment={"Variables": {
- "CUSTOM": "value", "AWS_ENDPOINT_URL": "http://wrong:4566",
- }})
- monkeypatch.setenv("AWS_ENDPOINT_URL", "http://ministack:4566")
- monkeypatch.setattr(lambda_svc, "get_region", lambda: "us-east-1")
- token = lambda_svc._durable_ctx.set(None)
- try:
- env = lambda_svc._provided_worker_env(config, "/code", 12345)
- finally:
- lambda_svc._durable_ctx.reset(token)
- assert env["AWS_LAMBDA_RUNTIME_API"] == "127.0.0.1:12345"
- assert env["AWS_LAMBDA_FUNCTION_NAME"] == config["FunctionName"]
- assert env["AWS_LAMBDA_FUNCTION_MEMORY_SIZE"] == "256"
- assert env["AWS_ACCESS_KEY_ID"] == "111122223333"
- assert env["AWS_REGION"] == "us-east-1"
- assert env["LAMBDA_TASK_ROOT"] == "/code"
- assert env["CUSTOM"] == "value"
- assert env["AWS_ENDPOINT_URL"] == "http://ministack:4566"
-
-
-def test_provided_env_uses_execution_role_credentials(monkeypatch):
- config = _config()
- credentials = {
- "AWS_ACCESS_KEY_ID": "ASIATESTROLE",
- "AWS_SECRET_ACCESS_KEY": "role-secret",
- "AWS_SESSION_TOKEN": "role-session",
- }
- resolve = Mock(return_value=credentials)
- monkeypatch.setattr(lambda_svc, "execution_credentials", resolve)
- env = lambda_svc._provided_worker_env(config, "/code", 12345)
- resolve.assert_called_once_with(config)
- assert {key: env[key] for key in credentials} == credentials
-
-
[email protected]("operation", ["code", "configuration", "delete"])
-def test_function_changes_invalidate_provided_workers(monkeypatch, isolated_pool, operation):
- config = _config()
- name = config["FunctionName"]
- monkeypatch.setattr(lambda_svc, "_functions", {name: {"config": config, "code_zip": b"zip"}})
- monkeypatch.setattr(lambda_svc, "get_account_id", lambda: "111122223333")
- monkeypatch.setattr(lambda_svc, "get_region", lambda: "us-east-1")
- monkeypatch.setattr(lambda_svc, "_pool_kill_function", Mock())
- monkeypatch.setattr(lambda_svc, "_sweep_extract_cache", Mock())
- monkeypatch.setattr(lambda_svc, "_schedule_state_transition", Mock())
- worker, _ = lambda_runtime.acquire_worker(name, config, b"zip")
- lambda_runtime.release_worker(worker)
- if operation == "code":
- result = lambda_svc._update_code(name, {})
- elif operation == "configuration":
- result = lambda_svc._update_config(name, {"Environment": {"Variables": {"UPDATED": "yes"}}})
- else:
- result = lambda_svc._delete_function(name, {})
- assert result[0] in (200, 204)
- assert not lambda_runtime._workers
diff --git a/tests/test_lambda_provided_worker.py b/tests/test_lambda_provided_worker.py
deleted file mode 100644
index f5498c66..00000000
--- a/tests/test_lambda_provided_worker.py
+++ /dev/null
@@ -1,745 +0,0 @@
-"""
-Unit tests for the warm ``provided.*`` execution environment
-(``ministack.core.lambda_runtime.ProvidedWorker``).
-
-Every test drives a *real* bootstrap: a stdlib-only Python script that speaks
-the Lambda Runtime API over ``AWS_LAMBDA_RUNTIME_API`` exactly as a Go or Rust
-binary would. No Go toolchain, no Docker and no running MiniStack are needed.
-
-Coverage:
- * environment reuse across invocations, with per-invocation trace context
- * the caller's event is never mutated to carry the trace ID
- * continuous draining of stdout *and* stderr (a pipe that fills would
- otherwise wedge the bootstrap), plus the bounded log buffer
- * a crashed bootstrap is replaced, and its server/socket/threads go with it
- * init errors and instant exits fail fast instead of burning the init timeout
- * a timed-out invocation tears the environment down and the next one recovers
- * stale / duplicate / cross-generation request IDs are rejected (400)
- * teardown leaves no process, listening socket or thread behind
-"""
-
-import socket
-import sys
-import threading
-import time
-import zipfile
-from io import BytesIO
-
-import pytest
-
-from ministack.core import lambda_runtime
-from ministack.core.lambda_runtime import (
- ProvidedWorker,
- acquire_worker,
- kill_workers,
- release_worker,
- reset,
-)
-
-# ---------------------------------------------------------------------------
-# Bootstrap fixtures (stdlib only, run by the host Python interpreter)
-# ---------------------------------------------------------------------------
-
-_PREAMBLE = '''#!{python}
-import json, os, sys, time, urllib.request, urllib.error
-
-API = os.environ["AWS_LAMBDA_RUNTIME_API"]
-PORT = int(API.rsplit(":", 1)[1])
-
-
-def _call(path, data=None, method=None):
- url = "http://%s/2018-06-01%s" % (API, path)
- req = urllib.request.Request(url, data=data, method=method)
- try:
- resp = urllib.request.urlopen(req, timeout=30)
- return resp.getcode(), resp.headers, resp.read()
- except urllib.error.HTTPError as exc:
- return exc.code, exc.headers, exc.read()
-
-
-def next_invocation():
- code, headers, body = _call("/runtime/invocation/next")
- if code != 200:
- raise SystemExit(0)
- return headers, json.loads(body or b"null")
-
-
-def post_response(rid, payload):
- return _call("/runtime/invocation/%s/response" % rid,
- data=json.dumps(payload).encode(), method="POST")[0]
-
-
-def post_error(rid, payload):
- return _call("/runtime/invocation/%s/error" % rid,
- data=json.dumps(payload).encode(), method="POST")[0]
-
-
-def post_init_error(payload):
- return _call("/runtime/init/error",
- data=json.dumps(payload).encode(), method="POST")[0]
-'''
-
-
-def _bootstrap(body: str) -> bytes:
- """Zip a bootstrap script whose body follows the Runtime API preamble."""
- script = _PREAMBLE.format(python=sys.executable) + "\n" + body
- buf = BytesIO()
- with zipfile.ZipFile(buf, "w") as archive:
- archive.writestr("bootstrap", script)
- return buf.getvalue()
-
-
-ECHO_BOOTSTRAP = _bootstrap('''
-n = 0
-while True:
- headers, event = next_invocation()
- n += 1
- rid = headers["Lambda-Runtime-Aws-Request-Id"]
- post_response(rid, {
- "pid": os.getpid(),
- "n": n,
- "port": PORT,
- "event": event,
- "request_id": rid,
- "trace": headers.get("Lambda-Runtime-Trace-Id"),
- "arn": headers.get("Lambda-Runtime-Invoked-Function-Arn"),
- "deadline": headers.get("Lambda-Runtime-Deadline-Ms"),
- })
-''')
-
-
-def _config(**overrides):
- config = {
- "Runtime": "provided.al2023",
- "Handler": "bootstrap",
- "FunctionName": "provided-fn",
- "FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:provided-fn",
- "Timeout": 10,
- "MemorySize": 128,
- }
- config.update(overrides)
- return config
-
-
[email protected](autouse=True)
-def isolated_runtime(monkeypatch, tmp_path):
- from ministack.services import lambda_svc
-
- monkeypatch.setattr(lambda_runtime, "_workers", {})
- monkeypatch.setattr(lambda_svc, "_PROVIDED_CODE_CACHE", str(tmp_path / "code"))
- monkeypatch.setattr(lambda_svc, "_provided_code_dirs", {})
- yield
- reset()
-
-
-def worker_factory():
- """Build ProvidedWorkers and guarantee they are torn down."""
- built: list = []
-
- def _make(code_zip: bytes, **config_overrides) -> ProvidedWorker:
- worker = ProvidedWorker("provided-fn", _config(**config_overrides), code_zip)
- built.append(worker)
- return worker
-
- yield _make
- kill_workers(built)
- reset()
-
-
-def _port_is_closed(port: int) -> bool:
- for _ in range(50):
- sock = socket.socket()
- sock.settimeout(0.5)
- try:
- sock.connect(("127.0.0.1", port))
- except OSError:
- return True
- finally:
- sock.close()
- time.sleep(0.05)
- return False
-
-
-# ---------------------------------------------------------------------------
-# Reuse + per-invocation trace context
-# ---------------------------------------------------------------------------
-
-
-def test_environment_is_reused_and_trace_is_per_invocation(worker_factory):
- """Second invocation lands on the same process, with its own trace ID."""
- worker = worker_factory(ECHO_BOOTSTRAP)
-
- first = worker.invoke({"k": 1}, "req-1", trace_id="Root=1-aaa;Sampled=1")
- second = worker.invoke({"k": 2}, "req-2", trace_id="Root=1-bbb;Sampled=1")
- third = worker.invoke({"k": 3}, "req-3")
-
- assert first["status"] == "ok" and second["status"] == "ok"
- assert first["cold_start"] is True
- assert second["cold_start"] is False and third["cold_start"] is False
-
- assert first["result"]["pid"] == second["result"]["pid"] == third["result"]["pid"]
- assert [r["result"]["n"] for r in (first, second, third)] == [1, 2, 3]
-
- # Request IDs and trace IDs are per invocation, never carried over.
- assert first["result"]["request_id"] == "req-1"
- assert second["result"]["request_id"] == "req-2"
- assert first["result"]["trace"] == "Root=1-aaa;Sampled=1"
- assert second["result"]["trace"] == "Root=1-bbb;Sampled=1"
- assert third["result"]["trace"] is None
- assert first["result"]["arn"] == worker.config["FunctionArn"]
-
-
-def test_trace_id_does_not_touch_the_event_payload(worker_factory):
- """The event is the caller's; trace context rides the Runtime API header.
-
- A user event that happens to contain the key the old implementation
- reserved must reach the handler untouched.
- """
- worker = worker_factory(ECHO_BOOTSTRAP)
- event = {"_x_amzn_trace_id": "user-supplied", "hello": "world"}
-
- result = worker.invoke(event, "req-1", trace_id="Root=1-ccc")
-
- assert result["status"] == "ok"
- # Delivered verbatim ...
- assert result["result"]["event"] == {
- "_x_amzn_trace_id": "user-supplied", "hello": "world"}
- # ... and the header carries the real trace ID.
- assert result["result"]["trace"] == "Root=1-ccc"
- # The caller's dict was not mutated.
- assert event == {"_x_amzn_trace_id": "user-supplied", "hello": "world"}
-
-
-# ---------------------------------------------------------------------------
-# Logs: both pipes drained continuously, buffer bounded
-# ---------------------------------------------------------------------------
-
-
-CHATTY_BOOTSTRAP = _bootstrap('''
-LINE = "x" * 120
-while True:
- headers, event = next_invocation()
- rid = headers["Lambda-Runtime-Aws-Request-Id"]
- count = event["lines"]
- for i in range(count):
- sys.stdout.write("OUT-%d %s\\n" % (i, LINE))
- sys.stderr.write("ERR-%d %s\\n" % (i, LINE))
- sys.stdout.flush()
- sys.stderr.flush()
- time.sleep(0.2)
- post_response(rid, {"logged": count})
-''')
-
-
-def test_large_log_volume_does_not_wedge_the_bootstrap(worker_factory):
- """~500KB across both pipes: an undrained pipe would block the child.
-
- stdout is drained as logs too — a custom runtime writes there — and both
- streams are merged into one log buffer.
- """
- worker = worker_factory(CHATTY_BOOTSTRAP, Timeout=30)
-
- result = worker.invoke({"lines": 2000}, "req-1")
-
- assert result["status"] == "ok"
- assert result["result"] == {"logged": 2000}
- log = result["log"]
- assert "OUT-0 " in log and "ERR-0 " in log
- assert "OUT-1999 " in log and "ERR-1999 " in log
-
- # And the environment survives it.
- again = worker.invoke({"lines": 1}, "req-2")
- assert again["status"] == "ok"
- assert again["cold_start"] is False
-
-
-def test_log_buffer_is_bounded_and_keeps_the_newest_lines(worker_factory, monkeypatch):
- monkeypatch.setattr(lambda_runtime, "_PROVIDED_LOG_MAX_LINES", 50)
- worker = worker_factory(CHATTY_BOOTSTRAP, Timeout=30)
-
- result = worker.invoke({"lines": 400}, "req-1")
-
- assert result["status"] == "ok"
- lines = [line for line in result["log"].splitlines() if line]
- assert len(lines) <= 50
- # Oldest dropped, newest kept.
- assert "OUT-0 " not in result["log"]
- assert any(line.startswith("ERR-399 ") or line.startswith("OUT-399 ")
- for line in lines)
-
-
-# ---------------------------------------------------------------------------
-# Crash / restart
-# ---------------------------------------------------------------------------
-
-
-CRASH_BOOTSTRAP = _bootstrap('''
-headers, event = next_invocation()
-rid = headers["Lambda-Runtime-Aws-Request-Id"]
-post_response(rid, {"pid": os.getpid(), "port": PORT})
-sys.stderr.write("bye\\n")
-sys.stderr.flush()
-os._exit(9)
-''')
-
-
-def test_crashed_bootstrap_is_replaced_without_leaking_its_server(worker_factory):
- worker = worker_factory(CRASH_BOOTSTRAP)
- threads_before = threading.active_count()
-
- first = worker.invoke({}, "req-1")
- assert first["status"] == "ok"
- first_port = first["result"]["port"]
- first_pid = first["result"]["pid"]
-
- # The bootstrap exits right after responding; the next invocation gets a
- # brand new environment, and the dead one's HTTP server is gone.
- second = worker.invoke({}, "req-2")
- assert second["status"] == "ok"
- assert second["cold_start"] is True
- assert second["result"]["pid"] != first_pid
- assert second["result"]["port"] != first_port
- assert _port_is_closed(first_port), "leaked Runtime API server from dead generation"
-
- worker.kill()
- assert _port_is_closed(second["result"]["port"])
- # Two generations, no accumulated threads.
- for _ in range(50):
- if threading.active_count() <= threads_before + 1:
- break
- time.sleep(0.1)
- assert threading.active_count() <= threads_before + 1
-
-
-# ---------------------------------------------------------------------------
-# Init failures are detected promptly
-# ---------------------------------------------------------------------------
-
-
-INIT_ERROR_BOOTSTRAP = _bootstrap('''
-sys.stderr.write("init blew up\\n")
-sys.stderr.flush()
-post_init_error({"errorMessage": "cannot load config", "errorType": "Init.Error"})
-sys.exit(1)
-''')
-
-INSTANT_EXIT_BOOTSTRAP = _bootstrap('''
-sys.stderr.write("missing shared library\\n")
-sys.stderr.flush()
-sys.exit(3)
-''')
-
-
-def test_init_error_fails_fast_and_leaves_nothing_running(worker_factory):
- worker = worker_factory(INIT_ERROR_BOOTSTRAP)
-
- started = time.monotonic()
- with pytest.raises(RuntimeError, match="init error"):
- worker.invoke({}, "req-1")
- elapsed = time.monotonic() - started
-
- assert elapsed < lambda_runtime._PROVIDED_INIT_TIMEOUT / 2
- assert worker._proc is None
- assert worker._server is None
- assert worker._server_thread is None
-
-
-def test_bootstrap_that_exits_immediately_fails_fast(worker_factory):
- worker = worker_factory(INSTANT_EXIT_BOOTSTRAP)
-
- started = time.monotonic()
- with pytest.raises(RuntimeError, match="exited during init with code 3"):
- worker.invoke({}, "req-1")
- elapsed = time.monotonic() - started
-
- assert elapsed < lambda_runtime._PROVIDED_INIT_TIMEOUT / 2
- assert worker._proc is None and worker._server is None
-
-
-def test_init_timeout_is_bounded(worker_factory, monkeypatch):
- """A binary that starts but never polls /next is given up on."""
- monkeypatch.setattr(lambda_runtime, "_PROVIDED_INIT_TIMEOUT", 1.0)
- silent = _bootstrap('time.sleep(60)\n')
- worker = worker_factory(silent)
-
- started = time.monotonic()
- with pytest.raises(RuntimeError, match="did not reach the Runtime API"):
- worker.invoke({}, "req-1")
- elapsed = time.monotonic() - started
-
- assert 1.0 <= elapsed < 8.0
- assert worker._proc is None and worker._server is None
-
-
-# ---------------------------------------------------------------------------
-# Timeout + recovery
-# ---------------------------------------------------------------------------
-
-
-SLOW_ONCE_BOOTSTRAP_TEMPLATE = '''
-MARKER = {marker!r}
-while True:
- headers, event = next_invocation()
- rid = headers["Lambda-Runtime-Aws-Request-Id"]
- if not os.path.exists(MARKER):
- open(MARKER, "w").close()
- sys.stderr.write("going to sleep\\n")
- sys.stderr.flush()
- time.sleep(30)
- post_response(rid, {{"pid": os.getpid(), "port": PORT}})
-'''
-
-
-def test_timeout_kills_environment_and_next_invocation_recovers(worker_factory, tmp_path):
- marker = str(tmp_path / "slept")
- worker = worker_factory(
- _bootstrap(SLOW_ONCE_BOOTSTRAP_TEMPLATE.format(marker=marker)),
- Timeout=1,
- )
-
- started = time.monotonic()
- timed_out = worker.invoke({}, "req-1")
- elapsed = time.monotonic() - started
-
- assert timed_out["status"] == "error"
- assert timed_out["error"] == "Task timed out after 1.00 seconds"
- assert "going to sleep" in timed_out["log"]
- assert elapsed < 8.0
- assert worker._proc is None and worker._server is None
-
- # A fresh environment serves the next invocation, and the marker makes the
- # replacement take the fast path.
- recovered = worker.invoke({}, "req-2")
- assert recovered["status"] == "ok"
- assert recovered["cold_start"] is True
-
-
-# ---------------------------------------------------------------------------
-# Request-ID validation and stale-response isolation
-# ---------------------------------------------------------------------------
-
-
-BAD_REQUEST_ID_BOOTSTRAP = _bootstrap('''
-while True:
- headers, event = next_invocation()
- rid = headers["Lambda-Runtime-Aws-Request-Id"]
- codes = {
- "bogus": post_response("not-a-real-request-id", {"stolen": True}),
- "empty": post_error("", {"errorMessage": "nope"}),
- }
- codes["real"] = post_response(rid, {"codes": codes, "request_id": rid})
- if codes["real"] != 202:
- post_error(rid, {"errorMessage": "response rejected: %s" % codes})
-''')
-
-
-def test_unknown_request_ids_are_rejected_not_delivered(worker_factory):
- worker = worker_factory(BAD_REQUEST_ID_BOOTSTRAP)
-
- result = worker.invoke({}, "req-1")
-
- assert result["status"] == "ok"
- # The forged IDs got 400s; only the real one was accepted.
- assert result["result"]["codes"] == {"bogus": 400, "empty": 404}
- assert result["result"]["request_id"] == "req-1"
-
-
-DUPLICATE_RESPONSE_BOOTSTRAP = _bootstrap('''
-while True:
- headers, event = next_invocation()
- rid = headers["Lambda-Runtime-Aws-Request-Id"]
- first = post_response(rid, {"which": "first"})
- second = post_response(rid, {"which": "second", "first_code": first})
- sys.stderr.write("dup-code=%s\\n" % second)
- sys.stderr.flush()
-''')
-
-
-def test_duplicate_response_for_same_request_id_is_rejected(worker_factory):
- worker = worker_factory(DUPLICATE_RESPONSE_BOOTSTRAP)
-
- first = worker.invoke({}, "req-1")
- assert first["status"] == "ok"
- assert first["result"] == {"which": "first"}
-
- # The duplicate got a 400 and, crucially, did not become the *next*
- # invocation's result.
- second = worker.invoke({}, "req-2")
- assert second["status"] == "ok"
- assert second["result"] == {"which": "first"}
- assert "dup-code=400" in "\n".join([first["log"], second["log"]])
-
-
-def test_results_from_a_dead_generation_are_ignored():
- """A late POST from a torn-down environment cannot satisfy its successor."""
- worker = ProvidedWorker("provided-fn", _config(), b"")
- worker._generation = 7
- worker._current_request_id = "req-live"
-
- # Same request ID, previous generation.
- assert worker._record_result(6, "req-live", "response", {"stale": True}) is False
- assert worker._response_ready.is_set() is False
- assert worker._result == {}
-
- # Wrong request ID, current generation.
- assert worker._record_result(7, "req-old", "response", {"stale": True}) is False
- assert worker._response_ready.is_set() is False
-
- # The live one is accepted exactly once.
- assert worker._record_result(7, "req-live", "response", {"ok": True}) is True
- assert worker._result == {"response": {"ok": True}}
- assert worker._record_result(7, "req-live", "response", {"again": True}) is False
- assert worker._result == {"response": {"ok": True}}
-
-
-def test_init_error_from_a_dead_generation_is_ignored():
- worker = ProvidedWorker("provided-fn", _config(), b"")
- worker._generation = 3
-
- worker._record_init_error(2, {"errorMessage": "stale"})
- assert worker._init_error is None
- assert worker._init_settled.is_set() is False
-
- worker._record_init_error(3, {"errorMessage": "live"})
- assert worker._init_error == {"errorMessage": "live"}
- assert worker._init_settled.is_set() is True
-
-
-# ---------------------------------------------------------------------------
-# Cleanup / pool contract
-# ---------------------------------------------------------------------------
-
-
-def test_kill_reaps_process_server_and_threads(worker_factory):
- worker = worker_factory(ECHO_BOOTSTRAP)
- threads_before = threading.active_count()
-
- result = worker.invoke({}, "req-1")
- port = result["result"]["port"]
- proc = worker._proc
- log_thread = worker._log_thread
- assert proc.poll() is None
-
- worker.kill()
-
- assert proc.poll() is not None, "bootstrap process not reaped"
- assert worker._proc is None
- assert worker._server is None and worker._server_thread is None
- assert worker._log_thread is None
- assert _port_is_closed(port), "Runtime API socket still listening after kill"
- log_thread.join(timeout=2.0)
- assert not log_thread.is_alive()
- for _ in range(50):
- if threading.active_count() <= threads_before:
- break
- time.sleep(0.1)
- assert threading.active_count() <= threads_before
-
-
-def test_pool_contract_lease_reuse_and_reset():
- """provided.* functions use the same acquire/release pool as the rest."""
- config = _config()
- try:
- worker, reason = acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
- assert isinstance(worker, ProvidedWorker)
- assert reason == "spawn"
- assert worker.in_use is True
-
- first = worker.invoke({}, "req-1")
- assert first["status"] == "ok"
- port = first["result"]["port"]
-
- # A second lease while the first is held gets a *separate* environment.
- other, other_reason = acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
- assert other is not worker and other_reason == "spawn"
- release_worker(other)
-
- release_worker(worker)
- again, reason = acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
- assert reason == "reused"
- assert again is worker
- assert again.invoke({}, "req-2")["result"]["port"] == port
- release_worker(again)
- finally:
- reset()
-
- assert _port_is_closed(port)
-
-
-def test_concurrent_invocations_use_separate_leased_environments():
- """Two leases run at the same time on two processes, results not crossed."""
- config = _config()
- barrier = threading.Barrier(2, timeout=30)
- results: dict = {}
-
- try:
- workers = []
- for _ in range(2):
- worker, reason = acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
- assert reason == "spawn"
- workers.append(worker)
-
- def _run(index, worker):
- barrier.wait()
- results[index] = worker.invoke({"i": index}, f"req-{index}",
- trace_id=f"Root=1-{index}")
-
- threads = [threading.Thread(target=_run, args=(i, w))
- for i, w in enumerate(workers)]
- for thread in threads:
- thread.start()
- for thread in threads:
- thread.join(timeout=60)
- assert not thread.is_alive()
-
- assert set(results) == {0, 1}
- assert all(r["status"] == "ok" for r in results.values())
- assert results[0]["result"]["pid"] != results[1]["result"]["pid"]
- for index, result in results.items():
- assert result["result"]["event"] == {"i": index}
- assert result["result"]["request_id"] == f"req-{index}"
- assert result["result"]["trace"] == f"Root=1-{index}"
- for worker in workers:
- release_worker(worker)
- finally:
- reset()
-
- for result in results.values():
- assert _port_is_closed(result["result"]["port"])
-
-
-def test_idle_reaper_and_reset_release_provided_environments():
- config = _config()
- try:
- first, _ = acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
- second, _ = acquire_worker("provided-fn", config, ECHO_BOOTSTRAP)
- ports = [w.invoke({}, "req-1")["result"]["port"] for w in (first, second)]
- procs = [first._proc, second._proc]
- release_worker(first)
- release_worker(second)
-
- # ttl=0 reaps every surplus environment; the first per key stays warm.
- assert lambda_runtime.reap_idle_workers(ttl=0) == 1
- assert procs[1].poll() is not None
- assert _port_is_closed(ports[1])
- assert second._server is None
- assert procs[0].poll() is None
-
- reset()
- assert procs[0].poll() is not None
- assert _port_is_closed(ports[0])
- assert first._server is None and first._log_thread is None
- finally:
- reset()
-
-
-def test_invoke_signature_keeps_trace_out_of_the_positional_contract():
- """``trace_id`` is keyword-only, so no caller can pass it as the event."""
- import inspect
-
- sig = inspect.signature(ProvidedWorker.invoke)
- assert list(sig.parameters) == ["self", "event", "request_id", "trace_id"]
- assert sig.parameters["trace_id"].kind is inspect.Parameter.KEYWORD_ONLY
- assert sig.parameters["trace_id"].default is None
-
-
-def test_json_only_bootstrap_output_is_not_parsed_as_protocol(worker_factory):
- """stdout is logs, not a protocol channel: JSON there must not confuse us."""
- noisy = _bootstrap('''
-while True:
- headers, event = next_invocation()
- rid = headers["Lambda-Runtime-Aws-Request-Id"]
- sys.stdout.write(json.dumps({"status": "ok", "result": "from-stdout"}) + "\\n")
- sys.stdout.flush()
- post_response(rid, {"real": True})
-''')
- worker = worker_factory(noisy)
-
- result = worker.invoke({}, "req-1")
-
- assert result["status"] == "ok"
- assert result["result"] == {"real": True}
- assert "from-stdout" in result["log"]
-
-
-def test_handler_error_does_not_poison_warm_environment(worker_factory):
- code = _bootstrap('''
-n = 0
-while True:
- headers, event = next_invocation()
- n += 1
- rid = headers["Lambda-Runtime-Aws-Request-Id"]
- if event.get("fail"):
- post_error(rid, {"errorMessage": "handler failed", "errorType": "Test.Error"})
- else:
- post_response(rid, {"n": n})
-''')
- worker = worker_factory(code)
- failed = worker.invoke({"fail": True}, "req-1")
- assert failed["status"] == "error"
- assert failed["error_payload"]["errorType"] == "Test.Error"
- recovered = worker.invoke({}, "req-2")
- assert recovered["status"] == "ok"
- assert recovered["cold_start"] is False
- assert recovered["result"] == {"n": 2}
-
-
-def test_exit_during_invocation_fails_without_waiting_for_timeout(worker_factory):
- worker = worker_factory(_bootstrap('''
-next_invocation()
-sys.stderr.write("crashed during handler\\n")
-sys.stderr.flush()
-sys.exit(9)
-'''), Timeout=30)
- started = time.monotonic()
- result = worker.invoke({}, "req-1")
- assert time.monotonic() - started < 8
- assert result["status"] == "error"
- assert result["error_payload"]["errorType"] == "Runtime.ExitError"
- assert "crashed during handler" in result["log"]
- assert worker._proc is None and worker._server is None
-
-
-def test_response_recorded_just_after_process_exit_is_preserved(worker_factory, monkeypatch):
- from unittest.mock import Mock
-
- worker = worker_factory(ECHO_BOOTSTRAP)
- proc = Mock()
- statuses = iter([None, 0])
- proc.poll.side_effect = lambda: next(statuses, 0)
- worker._proc = proc
- ready = Mock()
- waits = iter([False, True])
-
- def wait(timeout):
- settled = next(waits)
- if settled:
- worker._result = {"response": {"ok": True}}
- return settled
-
- ready.wait.side_effect = wait
- worker._response_ready = ready
- monkeypatch.setattr(worker, "_drain_stderr_bounded", lambda: "")
- result = worker.invoke({}, "req-1")
- assert result["status"] == "ok"
- assert result["result"] == {"ok": True}
- assert ready.wait.call_count == 2
- assert worker._proc is None
-
-
-def test_spawn_failure_closes_the_already_started_server(worker_factory, monkeypatch):
- worker = worker_factory(ECHO_BOOTSTRAP)
- resources = []
-
- def failed_popen(*args, **kwargs):
- resources.extend([worker._server, worker._server_thread])
- raise OSError("cannot execute bootstrap")
-
- monkeypatch.setattr(lambda_runtime.subprocess, "Popen", failed_popen)
- with pytest.raises(OSError, match="cannot execute bootstrap"):
- worker.invoke({}, "req-1")
- server, thread = resources
- assert server.fileno() == -1
- assert not thread.is_alive()
- assert worker._proc is None and worker._server is None
diff --git a/tests/test_s3.py b/tests/test_s3.py
index 318339d5..1575702d 100644
--- a/tests/test_s3.py
+++ b/tests/test_s3.py
@@ -3888,7 +3888,8 @@ def test_s3_presigned_url_requires_exact_sts_session_token(s3, sts):
for bad_token in (None, "wrong-session-token"):
with pytest.raises(urllib.error.HTTPError) as exc:
urllib.request.urlopen(presign(bad_token))
- assert exc.value.code == 403
+ # S3's error table: InvalidToken is 400, not 403.
+ assert exc.value.code == 400
assert b"<Code>InvalidToken</Code>" in exc.value.read()
@@ -3928,6 +3929,90 @@ def test_s3_presigned_url_accepts_lenient_session_origin(s3):
assert urllib.request.urlopen(url).status == 200
[email protected]("auth_enabled", [False, True])
[email protected]("bad_token", [False, True])
+def test_s3_presign_verifies_in_both_auth_modes(monkeypatch, auth_enabled, bad_token):
+ from urllib.parse import parse_qs, urlsplit
+
+ import boto3
+ from botocore.config import Config
+
+ from ministack import app as app_mod
+ from ministack.core.responses import get_account_id, request_scope
+ from ministack.services import iam as iam_svc
+ from ministack.services import s3 as s3_svc
+
+ key = "test-presign-mode-key"
+ owner = "123456789012"
+ monkeypatch.setattr(app_mod, "AUTH", auth_enabled)
+ iam_svc._access_keys.set_scoped(owner, None, key, {
+ "UserName": "alice", "Status": "Active", "SecretAccessKey": "secret",
+ })
+ client = boto3.client(
+ "s3", endpoint_url="http://localhost:4566", region_name="us-east-1",
+ aws_access_key_id=key, aws_secret_access_key="secret",
+ aws_session_token="unexpected-token" if bad_token else None,
+ config=Config(signature_version="s3v4"),
+ )
+ url = urlsplit(client.generate_presigned_url(
+ "get_object", Params={"Bucket": "test-bucket", "Key": "object"},
+ ))
+ try:
+ with request_scope("000000000000", "us-east-1"):
+ error = s3_svc._verify_presigned_sigv4(
+ "GET", url.path, {"host": url.netloc}, parse_qs(url.query),
+ )
+ assert get_account_id() == owner
+ if bad_token:
+ assert error[0] == 400
+ else:
+ assert error is None
+ finally:
+ iam_svc._access_keys.pop_scoped(owner, None, key, None)
+
+
+def test_presigned_mrap_resolves_alias_in_iam_owner_account(monkeypatch):
+ import asyncio
+ from urllib.parse import parse_qs, urlsplit
+
+ from botocore.auth import S3SigV4QueryAuth
+ from botocore.awsrequest import AWSRequest
+ from botocore.credentials import Credentials
+
+ from ministack import app as app_mod
+ from ministack.core.responses import get_account_id, request_scope
+ from ministack.services import iam as iam_svc
+ from ministack.services import s3 as s3_svc
+
+ key, owner, alias = "test-mrap-owner-key", "123456789012", "testalias.mrap"
+ host = f"{alias}.accesspoint.s3-global.amazonaws.com"
+ monkeypatch.setattr(app_mod, "AUTH", False)
+ request = AWSRequest(method="GET", url=f"http://{host}/object")
+ S3SigV4QueryAuth(Credentials(key, "secret"), "s3", "us-east-1").add_auth(request)
+ url = urlsplit(request.url)
+ routed = []
+
+ async def capture(method, path, headers, body, query_params, **kwargs):
+ routed.append((get_account_id(), path))
+ return 200, {}, b"object"
+
+ monkeypatch.setattr(s3_svc, "handle_request", capture)
+ iam_svc._access_keys.set_scoped(owner, None, key, {
+ "UserName": "alice", "Status": "Active", "SecretAccessKey": "secret",
+ })
+ s3_svc._mraps.set_scoped(owner, None, alias, {"Regions": ["member-bucket"]})
+ try:
+ with request_scope("000000000000", "us-east-1"):
+ result = asyncio.run(app_mod._handle_s3_vhost_request(
+ host, url.path, "GET", {"host": host}, b"", parse_qs(url.query),
+ ))
+ assert result[0] == 200
+ assert routed == [(owner, "/member-bucket/object")]
+ finally:
+ iam_svc._access_keys.pop_scoped(owner, None, key, None)
+ s3_svc._mraps.pop_scoped(owner, None, alias, None)
+
+
def test_s3_presigned_url_virtual_hosted_style_is_verified():
"""A virtual-hosted-style presigned URL signs the bucket-less canonical URI
against the `{bucket}.host` Host header. MiniStack rewrites vhost → path-style
diff --git a/tests/test_sts.py b/tests/test_sts.py
index 5dc1c484..dd23eeab 100644
--- a/tests/test_sts.py
+++ b/tests/test_sts.py
@@ -332,6 +332,50 @@ def test_sts_get_session_token_rejects_temporary_caller_when_auth_enabled(monkey
sts_mod._sessions.pop(source_key, None)
[email protected]("auth_enabled", [False, True])
[email protected]("action", ["GetSessionToken", "AssumeRole"])
[email protected]("credential_kind", ["unknown", "inactive", "expired"])
+def test_sts_credential_rejections_require_auth(monkeypatch, auth_enabled, action, credential_kind):
+ import asyncio
+ import time
+
+ from ministack import app as app_mod
+ from ministack.core.responses import request_scope
+ from ministack.services import iam as iam_svc
+ from ministack.services import sts as sts_svc
+
+ key = "test-sts-mode-key"
+ account = "000000000000"
+ monkeypatch.setattr(app_mod, "AUTH", auth_enabled)
+ if credential_kind == "inactive":
+ iam_svc._access_keys.set_scoped(account, None, key, {
+ "UserName": "alice", "Status": "Inactive", "SecretAccessKey": "secret",
+ })
+ if credential_kind == "expired":
+ sts_svc._sessions[key] = {
+ "Arn": f"arn:aws:sts::{account}:assumed-role/role/session",
+ "UserId": "role:session", "SecretAccessKey": "secret",
+ "Expiration": time.time() - 60,
+ }
+ previous = set(sts_svc._sessions)
+ try:
+ with request_scope(account, "us-east-1"):
+ status, _, _ = asyncio.run(sts_svc.handle_request(
+ "GET", "/", {
+ "authorization": f"AWS4-HMAC-SHA256 Credential={key}/20260911/us-east-1/sts/aws4_request",
+ }, b"", {
+ "Action": [action], "RoleArn": [f"arn:aws:iam::{account}:role/role"],
+ "RoleSessionName": ["session"],
+ },
+ ))
+ assert status == (403 if auth_enabled else 200)
+ finally:
+ iam_svc._access_keys.pop_scoped(account, None, key, None)
+ for created in set(sts_svc._sessions) - previous:
+ sts_svc._sessions.pop(created, None)
+ sts_svc._sessions.pop(key, None)
+
+
def test_sts_assume_role_with_web_identity(sts, iam):
iam.create_role(
RoleName="test-oidc-role",
diff --git a/tests/test_transcribe.py b/tests/test_transcribe.py
index e3f28a61..41e12c91 100644
--- a/tests/test_transcribe.py
+++ b/tests/test_transcribe.py
@@ -101,38 +101,6 @@ def test_transcribe_start_returns_a_job_already_in_progress(transcribe, media):
assert started["StartTime"] >= started["CreationTime"]
-def test_transcribe_queue_seconds_puts_a_job_through_queued_first(transcribe, media):
- """QUEUED is off by default because AWS reaches it only for a request that
- opted into job queueing while the account is at its concurrent job limit,
- but a caller that handles the state still needs a way to exercise it. With
- TRANSCRIBE_JOB_QUEUE_SECONDS set, the job starts QUEUED with no StartTime
- and the worker moves it to IN_PROGRESS."""
- from conftest import _ministack_config
-
- _, _, media_uri = media
- job_name = _unique("job")
-
- _ministack_config({"transcribe._JOB_QUEUE_SECONDS": 3})
- try:
- started = transcribe.start_transcription_job(
- TranscriptionJobName=job_name,
- LanguageCode="en-US",
- Media={"MediaFileUri": media_uri},
- )["TranscriptionJob"]
-
- assert started["TranscriptionJobStatus"] == "QUEUED"
- # AWS omits StartTime until the job leaves the queue.
- assert "StartTime" not in started
-
- running = _wait_for_status(transcribe, job_name, statuses=("IN_PROGRESS",))
- assert running["StartTime"] >= running["CreationTime"]
- finally:
- _ministack_config({"transcribe._JOB_QUEUE_SECONDS": 0})
-
- job = _wait_for_status(transcribe, job_name)
- assert job["TranscriptionJobStatus"] == "COMPLETED"
-
-
def test_transcribe_transcript_is_deterministic_per_media(transcribe, s3, media):
"""There is no ASR here, so the value of the mock rests entirely on being
reproducible. The same media must transcribe identically on every run or