Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
release v1.4.1 (#1063)
Nahuel Nucerad6561c0ef37a
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 3a71c67b..369c4dc9 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -23,7 +23,7 @@ jobs:- name: Set up Pythonuses: actions/setup-python@v6with:- python-version: "3.12"+ python-version: "3.13"cache: "pip"- name: Install dependencies@@ -54,7 +54,7 @@ jobs:- name: Set up Pythonuses: actions/setup-python@v6with:- python-version: "3.12"+ python-version: "3.13"cache: "pip"- name: Install dependenciesdiff --git a/.github/workflows/pypi-publish.yml b/.github/workflows/pypi-publish.ymlindex 9f17221c..8b50bca7 100644--- a/.github/workflows/pypi-publish.yml+++ b/.github/workflows/pypi-publish.yml@@ -18,7 +18,7 @@ jobs:- name: Set up Pythonuses: actions/setup-python@v5with:- python-version: "3.12"+ python-version: "3.13"- name: Install build toolsrun: pip install builddiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 172ae467..cd10f2bd 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,23 @@ All notable changes to MiniStack will be documented here.Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).Versioning follows [Semantic Versioning](https://semver.org/).+## [1.4.1] — 2026-07-09++### Added+- **EKS — clusters pull images from local ECR** — every k3s cluster now boots with an auto-generated `/etc/rancher/k3s/registries.yaml` that mirrors the cluster's ECR registry hostname (`<account>.dkr.ecr.<region>.amazonaws.com`) to the MiniStack gateway, so `kubectl run --image <account>.dkr.ecr.<region>.amazonaws.com/repo:tag` pulls an image you pushed to local ECR — no manual registry wiring. The mirror endpoint is MiniStack's own address on the shared Docker network when one is detected, and `host.docker.internal` (added to the k3s container via `host-gateway`) when MiniStack runs on the host. Verified end-to-end: image pushed to local ECR, pod pulled and reached `Running`. Reported by @L3337.+- **RDS — Aurora MySQL version-to-image mapping and MySQL 8.4 support** — Aurora MySQL containers now boot the MySQL image matching the requested engine version track (5.7 / 8.0 / 8.4) instead of the floating `mysql:8` tag, which had started silently booting MySQL 8.4 for Aurora MySQL 8.0 test cases. `DescribeDBEngineVersions` returns the full creatable Aurora MySQL catalog with correct per-family parameter groups — including the new `8.4.mysql_aurora.8.4.7` / `aurora-mysql8.4` (GA 2026-05-21) — an explicit `EngineVersion` not in the catalog is rejected with the AWS error shape, and `aurora-mysql8.4` parameter-group defaults omit `skip-character-set-client-handshake` (removed in MySQL 8.4) while older families keep it. The default Aurora MySQL version moves from the no-longer-creatable `8.0.mysql_aurora.3.03.0` to `8.0.mysql_aurora.3.10.3`. Contributed by @Areson.++### Changed+- **EventBridge Pipes — region-isolated state** — pipe records and stream positions move to account+region scope (continuing the 1.4.0 multi-region work), so same-name pipes in different regions no longer collide, and the background poller processes each pipe under its own account and region. Legacy persisted state migrates by recovering each pipe's region from its ARN. Contributed by @Areson.+- **Docker images — Python 3.13 base** — both images (`Dockerfile` Alpine and `Dockerfile.full` Debian) move from `python:3.12` to `python:3.13`, clearing the CPython 3.12 interpreter CVEs flagged by image scanners; CI now runs the test suite on 3.13 to match the shipped interpreter. `Dockerfile.full` also runs `apt-get upgrade` so Debian base packages pick up security patches on every rebuild, as its comment always claimed and as the Alpine image already did. Contributed by @scottschreckengaust.++### Fixed+- **Lambda — concurrent `provided.*` invocations no longer fail with `ETXTBSY` ("Text file busy")** — each invocation extracted the function zip into its own temp directory and exec'd the `bootstrap` binary from it; while one thread still held the extraction's write descriptor, another thread's process spawn let the child inherit it, and executing the binary failed with `ETXTBSY` (a classic CPython fork/exec race). Extraction is now content-addressed — one shared read-only directory per code sha256, matching real Lambda's read-only `/var/task` — and spawns are serialized against extraction so the race cannot occur. Reported by @crestonbunch.+- **Lambda — `provided.*` runtime env restores `AWS_LAMBDA_FUNCTION_MEMORY_SIZE`, `AWS_LAMBDA_FUNCTION_VERSION`, and `AWS_LAMBDA_LOG_STREAM_NAME`** — 1.4.0's reserved-name filtering stripped these from the user environment, and the provided-runtime executor (unlike the container, local, and warm-worker paths) never re-injected them from the function config, so the Rust `lambda_runtime` crate — which requires `AWS_LAMBDA_FUNCTION_MEMORY_SIZE` — crashed on startup. All three are now set from the function configuration, matching the other executors and real AWS. Reported by @crestonbunch.+- **Lambda — SQS event source mappings deliver `messageAttributes` in camelCase** — records forwarded to functions carried the PascalCase inner keys SQS uses on the wire (`StringValue`/`DataType`/`BinaryValue`), but real Lambda re-serializes them to camelCase (`stringValue`/`dataType`/…) before invoking. Typed handler bindings — e.g. Java's `SQSEvent.MessageAttribute`, which Jackson populates case-strictly — saw all-null attributes and failed on messages whose attributes were set correctly. The ESM now performs the same camelCase transformation as AWS at the delivery boundary; the SQS API surface itself is unchanged. Reported by @w-zx.+- **S3 — `versionId` survives payload persistence to a volume** — with `S3_PERSIST=1`, the on-disk `.meta.json` sidecar never recorded the object's `version_id`, and all four write paths (PutObject, POST upload, CopyObject, CompleteMultipartUpload) persisted the sidecar *before* the version id was assigned — so after a restart, GET/HEAD returned no `x-amz-version-id` for objects on the volume. The sidecar now stores the version id and is written after version assignment, so the current version id of every object survives restarts. (Historical version data is still memory-only; full version-history persistence is tracked separately.) Reported by @adzcodemi.+- **Cognito — `ListUsers` parses quoted attribute names in `Filter`** — the filter parser only matched unquoted attribute names, but AWS's documented syntax also accepts the quoted form (`"email" = "value"` — used in AWS's own API-reference sample request), and on any parse failure MiniStack silently returned **all** users. Both forms now parse (including the docs' no-space `"email"^="value"` shape), and an unparseable filter logs a warning. Contributed by @kjdev.+## [1.4.0] "Areson" — 2026-07-08### Addeddiff --git a/Dockerfile b/Dockerfileindex be241ea7..51fc4fcd 100644--- a/Dockerfile+++ b/Dockerfile@@ -1,4 +1,4 @@-FROM python:3.12-alpine AS builder+FROM python:3.13-alpine AS builderRUN pip install --no-cache-dir --no-compile \hypercorn==0.18.0 \@@ -13,25 +13,25 @@ RUN pip install --no-cache-dir --no-compile \"awscli"# Strip awscli help examples (~25 MB) and Python cache files (~15 MB).-RUN rm -rf /usr/local/lib/python3.12/site-packages/awscli/examples \- && find /usr/local/lib/python3.12/site-packages -type d -name '__pycache__' -exec rm -rf {} + 2>/dev/null \- && rm -rf /usr/local/lib/python3.12/site-packages/pip*.dist-info \- && rm -rf /usr/local/lib/python3.12/site-packages/pip*+RUN rm -rf /usr/local/lib/python3.13/site-packages/awscli/examples \+ && find /usr/local/lib/python3.13/site-packages -type d -name '__pycache__' -exec rm -rf {} + 2>/dev/null \+ && rm -rf /usr/local/lib/python3.13/site-packages/pip*.dist-info \+ && rm -rf /usr/local/lib/python3.13/site-packages/pip*-FROM python:3.12-alpine+FROM python:3.13-alpineLABEL maintainer="MiniStack" \description="Local AWS Service Emulator — drop-in LocalStack replacement"# Upgrade base packages to pick up latest security patches.RUN apk upgrade --no-cache && apk add --no-cache nodejs bash openssl && rm -f /usr/bin/wget /bin/wget \- && rm -rf /usr/local/lib/python3.12/site-packages/pip* \+ && rm -rf /usr/local/lib/python3.13/site-packages/pip* \/usr/local/bin/pip*WORKDIR /opt/ministack# Copy cleaned Python packages and CLI entrypoints from builder.-COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages+COPY --from=builder /usr/local/lib/python3.13/site-packages /usr/local/lib/python3.13/site-packagesCOPY --from=builder /usr/local/bin/aws /usr/local/bin/awsCOPY --from=builder /usr/local/bin/aws_completer /usr/local/bin/aws_completerCOPY --from=builder /usr/local/bin/hypercorn /usr/local/bin/hypercorndiff --git a/Dockerfile.full b/Dockerfile.fullindex d0b811a3..b38956fb 100644--- a/Dockerfile.full+++ b/Dockerfile.full@@ -1,4 +1,4 @@-FROM python:3.12-slim AS builder+FROM python:3.13-slim AS builder# Full variant — Debian/glibc base so DuckDB and psycopg2 install from# published manylinux wheels instead of building from source against musl.@@ -21,14 +21,14 @@ RUN pip install --no-cache-dir --no-compile \# Strip awscli help examples (~25 MB), Python cache files (~15 MB),# pip metadata, and DuckDB stub / ADBC artefacts that aren't used at runtime.-RUN rm -rf /usr/local/lib/python3.12/site-packages/awscli/examples \- && find /usr/local/lib/python3.12/site-packages -type d -name '__pycache__' -exec rm -rf {} + 2>/dev/null \- && rm -rf /usr/local/lib/python3.12/site-packages/pip*.dist-info \- && rm -rf /usr/local/lib/python3.12/site-packages/pip* \- && rm -rf /usr/local/lib/python3.12/site-packages/_duckdb-stubs \- && rm -rf /usr/local/lib/python3.12/site-packages/adbc_driver_duckdb+RUN rm -rf /usr/local/lib/python3.13/site-packages/awscli/examples \+ && find /usr/local/lib/python3.13/site-packages -type d -name '__pycache__' -exec rm -rf {} + 2>/dev/null \+ && rm -rf /usr/local/lib/python3.13/site-packages/pip*.dist-info \+ && rm -rf /usr/local/lib/python3.13/site-packages/pip* \+ && rm -rf /usr/local/lib/python3.13/site-packages/_duckdb-stubs \+ && rm -rf /usr/local/lib/python3.13/site-packages/adbc_driver_duckdb-FROM python:3.12-slim+FROM python:3.13-slimLABEL maintainer="MiniStack" \description="Local AWS Service Emulator — full variant with DuckDB (Athena), PostgreSQL, MySQL drivers, and SFTP"@@ -41,13 +41,13 @@ RUN apt-get update \&& apt-get install -y --no-install-recommends nodejs bash \&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \&& rm -rf /var/lib/apt/lists/* /var/cache/apt/* /tmp/* \- && rm -rf /usr/local/lib/python3.12/site-packages/pip* /usr/local/bin/pip* \+ && rm -rf /usr/local/lib/python3.13/site-packages/pip* /usr/local/bin/pip* \&& rm -rf /usr/share/doc /usr/share/man /usr/share/info /usr/share/localeWORKDIR /opt/ministack# Copy cleaned Python packages and CLI entrypoints from builder.-COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages+COPY --from=builder /usr/local/lib/python3.13/site-packages /usr/local/lib/python3.13/site-packagesCOPY --from=builder /usr/local/bin/aws /usr/local/bin/awsCOPY --from=builder /usr/local/bin/aws_completer /usr/local/bin/aws_completerCOPY --from=builder /usr/local/bin/hypercorn /usr/local/bin/hypercorndiff --git a/ministack/services/eks.py b/ministack/services/eks.pyindex 00abee97..2b44e680 100644--- a/ministack/services/eks.py+++ b/ministack/services/eks.py@@ -370,10 +370,96 @@ def _k3s_run_kwargs(name: str, port: int, ms_network: str | None = None, oidc_ar)if ms_network:run_kwargs["network"] = ms_network+ # host-gateway lets the k3s node reach a host-run MiniStack (the ECR+ # registry mirror, #1054) even on Linux Docker where the name doesn't+ # resolve natively. Harmless when a shared network is used instead.+ run_kwargs["extra_hosts"] = {"host.docker.internal": "host-gateway"}return run_kwargs+def _ecr_registry_hosts(cluster: dict) -> list[str]:+ """The local ECR registry hostnames this cluster's nodes should resolve+ to MiniStack — derived from the cluster ARN (not contextvars) so the+ restore/restart paths, which run outside a request context, get the+ same answer as create."""+ try:+ spec = parse_arn(cluster.get("arn", ""))+ return [f"{spec.account_id}.dkr.ecr.{spec.region}.amazonaws.com"]+ except ArnParseError:+ return []+++def _k3s_registries_yaml(client, ms_network: str | None, ecr_hosts: list[str]) -> bytes | None:+ """Build /etc/rancher/k3s/registries.yaml mapping this cluster's local+ ECR registry hostnames to the MiniStack gateway, so pods with images+ like ``<account>.dkr.ecr.<region>.amazonaws.com/repo:tag`` pull from+ local ECR (#1054). ECR's Docker Registry V2 endpoints serve anonymously,+ so no auth config is needed.++ The mirror endpoint must be reachable from INSIDE the k3s container:+ - shared user-defined network: MiniStack's own IP on that network+ (``host-gateway`` resolves to docker0, which iptables typically+ blocks from a sibling bridge — same reasoning as the ECS metadata+ server wiring)+ - otherwise: ``host.docker.internal``, resolvable through the+ host-gateway extra_host added in ``_k3s_run_kwargs``+ """+ if not ecr_hosts:+ return None+ port = os.environ.get("GATEWAY_PORT") or os.environ.get("EDGE_PORT") or "4566"+ host = None+ if ms_network and client is not None:+ try:+ self_container = client.containers.get(os.environ.get("HOSTNAME", ""))+ nets = self_container.attrs["NetworkSettings"]["Networks"]+ host = (nets.get(ms_network) or {}).get("IPAddress") or None+ except Exception:+ host = None+ if not host:+ host = "host.docker.internal"+ endpoint = f"http://{host}:{port}"+ lines = ["mirrors:"]+ for reg in ecr_hosts:+ lines.append(f' "{reg}":')+ lines.append(" endpoint:")+ lines.append(f' - "{endpoint}"')+ return ("\n".join(lines) + "\n").encode()+++def _start_k3s_container(client, run_kwargs: dict, registries_yaml: bytes | None):+ """Start the k3s container, injecting registries.yaml before boot.++ k3s reads /etc/rancher/k3s/registries.yaml once at startup, so the file+ must exist before the entrypoint runs: create the container stopped,+ upload the file with put_archive, then start it."""+ if not registries_yaml:+ return client.containers.run(**run_kwargs)+ import io+ import tarfile++ create_kwargs = dict(run_kwargs)+ create_kwargs.pop("detach", None) # run()-only kwarg+ try:+ container = client.containers.create(**create_kwargs)+ except Exception:+ # create() does not auto-pull missing images the way run() does.+ client.images.pull(create_kwargs["image"])+ container = client.containers.create(**create_kwargs)+ try:+ buf = io.BytesIO()+ with tarfile.open(fileobj=buf, mode="w") as tar:+ info = tarfile.TarInfo("etc/rancher/k3s/registries.yaml")+ info.size = len(registries_yaml)+ info.mode = 0o644+ tar.addfile(info, io.BytesIO(registries_yaml))+ container.put_archive("/", buf.getvalue())+ except Exception as e:+ logger.warning("EKS: could not inject ECR registries.yaml: %s", e)+ container.start()+ return container++def _stop_all_k3s():"""Stop all k3s containers managed by MiniStack."""client = _get_docker()@@ -483,7 +569,8 @@ def _create_cluster(body):ms_network = _get_ministack_network(client)run_kwargs = _k3s_run_kwargs(name=name, port=port, ms_network=ms_network, oidc_args=oidc_args, node_labels=node_labels)- container = client.containers.run(**run_kwargs)+ registries_yaml = _k3s_registries_yaml(client, ms_network, _ecr_registry_hosts(cluster))+ container = _start_k3s_container(client, run_kwargs, registries_yaml)cluster["_docker_id"] = container.idcluster["endpoint"] = _cluster_endpoint(port)@@ -973,7 +1060,8 @@ def _restart_k3s(cluster_name, oidc_args=None, idp_cfg_refs=None):ms_network = _get_ministack_network(client)run_kwargs = _k3s_run_kwargs(name=cluster_name, port=cluster["_port"], ms_network=ms_network, oidc_args=oidc_args, node_labels=node_labels)- container = client.containers.run(**run_kwargs)+ registries_yaml = _k3s_registries_yaml(client, ms_network, _ecr_registry_hosts(cluster))+ container = _start_k3s_container(client, run_kwargs, registries_yaml)cluster["_docker_id"] = container.idcluster["endpoint"] = _cluster_endpoint(cluster["_port"])diff --git a/ministack/services/lambda_svc.py b/ministack/services/lambda_svc.pyindex 977f447f..697decb9 100644--- a/ministack/services/lambda_svc.py+++ b/ministack/services/lambda_svc.py@@ -251,6 +251,44 @@ def _runtime_env_vars(config: dict) -> dict:return {key: value for key, value in env.items() if key not in _RESERVED_RUNTIME_ENV_VARS}+# ── provided.* runtime code cache (#1051) ──────────────────+# Concurrent provided.* invocations used to extract the code zip into a+# fresh TemporaryDirectory per invocation and exec the bootstrap from it.+# While one thread still holds the extraction's write fd on ``bootstrap``,+# another thread's ``Popen`` fork lets the child inherit that fd, and the+# child's ``execve`` then fails with ETXTBSY (CPython bpo-2320). Extraction+# is therefore content-addressed — one shared read-only directory per code+# sha256, matching real Lambda's read-only /var/task — and both extraction+# and every spawn take ``_provided_code_lock`` so no fork can overlap an+# extraction write. Directories accumulate per distinct code sha within+# ``tempfile.gettempdir()``; ``reset()`` removes them.+_PROVIDED_CODE_CACHE = os.path.join(tempfile.gettempdir(), "ministack-provided-code")+_provided_code_lock = threading.Lock()+_provided_code_dirs: dict[str, str] = {}+++def _provided_runtime_code_dir(code_zip: bytes) -> str:+ """Extract ``code_zip`` once into a shared per-sha directory and return it."""+ sha = hashlib.sha256(code_zip).hexdigest()+ with _provided_code_lock:+ cached = _provided_code_dirs.get(sha)+ if cached and os.path.isdir(cached):+ return cached+ code_dir = os.path.join(_PROVIDED_CODE_CACHE, sha)+ if os.path.isdir(code_dir):+ # Leftover from a previous process — contents may be partial.+ import shutil+ shutil.rmtree(code_dir, ignore_errors=True)+ os.makedirs(code_dir, exist_ok=True)+ with zipfile.ZipFile(io.BytesIO(code_zip)) as zf:+ _extract_zip_preserving_mode(zf, code_dir)+ bootstrap_path = os.path.join(code_dir, "bootstrap")+ if os.path.exists(bootstrap_path):+ os.chmod(bootstrap_path, 0o755)+ _provided_code_dirs[sha] = code_dir+ return code_dir++# ── Persistence ────────────────────────────────────────────# Lambda code zips are stored on disk as content-addressed blob files under@@ -3878,150 +3916,147 @@ def _execute_function_provided(func: dict, event: dict) -> dict:try:import http.serverimport socketserver- with tempfile.TemporaryDirectory() as tmpdir:- # Extract bootstrap binary- zip_path = os.path.join(tmpdir, "code.zip")- with open(zip_path, "wb") as f:- f.write(code_zip)- code_dir = os.path.join(tmpdir, "code")- os.makedirs(code_dir)- with zipfile.ZipFile(zip_path) as zf:- _extract_zip_preserving_mode(zf, code_dir)-- bootstrap_path = os.path.join(code_dir, "bootstrap")- if not os.path.exists(bootstrap_path):- return {"body": {"statusCode": 200, "body": "Mock response - no bootstrap binary found"}}- os.chmod(bootstrap_path, 0o755)-- # Shared state for the Runtime API- result_holder = {"response": None, "error": None}- event_json = json.dumps(event)- request_id = new_uuid()- event_served = threading.Event()- response_received = threading.Event()- server_ready = threading.Event()-- class RuntimeAPIHandler(http.server.BaseHTTPRequestHandler):- def log_message(self, format, *args):- pass # Suppress logs-- def _read_body(self):- """Read request body, handling both Content-Length and chunked transfer encoding."""- transfer_encoding = self.headers.get("Transfer-Encoding", "")- if "chunked" in transfer_encoding.lower():- chunks = []- while True:- line = self.rfile.readline().strip()- chunk_size = int(line, 16)- if chunk_size == 0:- self.rfile.readline() # trailing CRLF- break- chunks.append(self.rfile.read(chunk_size))+ code_dir = _provided_runtime_code_dir(code_zip)+ bootstrap_path = os.path.join(code_dir, "bootstrap")+ if not os.path.exists(bootstrap_path):+ return {"body": {"statusCode": 200, "body": "Mock response - no bootstrap binary found"}}++ # Shared state for the Runtime API+ result_holder = {"response": None, "error": None}+ event_json = json.dumps(event)+ request_id = new_uuid()+ event_served = threading.Event()+ response_received = threading.Event()+ server_ready = threading.Event()++ class RuntimeAPIHandler(http.server.BaseHTTPRequestHandler):+ def log_message(self, format, *args):+ pass # Suppress logs++ def _read_body(self):+ """Read request body, handling both Content-Length and chunked transfer encoding."""+ transfer_encoding = self.headers.get("Transfer-Encoding", "")+ if "chunked" in transfer_encoding.lower():+ chunks = []+ while True:+ line = self.rfile.readline().strip()+ chunk_size = int(line, 16)+ if chunk_size == 0:self.rfile.readline() # trailing CRLF- return b"".join(chunks)- content_length = int(self.headers.get("Content-Length", 0))- return self.rfile.read(content_length) if content_length else b""-- def do_GET(self):- # GET /2018-06-01/runtime/invocation/next- if "/runtime/invocation/next" in self.path:- self.send_response(200)- self.send_header("Lambda-Runtime-Aws-Request-Id", request_id)- self.send_header("Lambda-Runtime-Deadline-Ms",- str(int((time.time() + timeout) * 1000)))- self.send_header("Content-Type", "application/json")- self.end_headers()- self.wfile.write(event_json.encode())- event_served.set()- else:- self.send_response(404)- self.end_headers()+ break+ chunks.append(self.rfile.read(chunk_size))+ self.rfile.readline() # trailing CRLF+ return b"".join(chunks)+ content_length = int(self.headers.get("Content-Length", 0))+ return self.rfile.read(content_length) if content_length else b""++ def do_GET(self):+ # GET /2018-06-01/runtime/invocation/next+ if "/runtime/invocation/next" in self.path:+ self.send_response(200)+ self.send_header("Lambda-Runtime-Aws-Request-Id", request_id)+ self.send_header("Lambda-Runtime-Deadline-Ms",+ str(int((time.time() + timeout) * 1000)))+ self.send_header("Content-Type", "application/json")+ self.end_headers()+ self.wfile.write(event_json.encode())+ event_served.set()+ else:+ self.send_response(404)+ self.end_headers()- def do_POST(self):- body = self._read_body()- if f"/runtime/invocation/{request_id}/response" in self.path:- try:- result_holder["response"] = json.loads(body)- except json.JSONDecodeError:- result_holder["response"] = body.decode("utf-8", errors="replace")- self.send_response(202)- self.end_headers()- response_received.set()- elif f"/runtime/invocation/{request_id}/error" in self.path:- try:- result_holder["error"] = json.loads(body)- except json.JSONDecodeError:- result_holder["error"] = body.decode("utf-8", errors="replace")- self.send_response(202)- self.end_headers()- response_received.set()- elif "/runtime/init/error" in self.path:- try:- result_holder["error"] = json.loads(body)- except json.JSONDecodeError:- result_holder["error"] = body.decode("utf-8", errors="replace")- self.send_response(202)- self.end_headers()- response_received.set()- else:- self.send_response(404)- self.end_headers()+ def do_POST(self):+ body = self._read_body()+ if f"/runtime/invocation/{request_id}/response" in self.path:+ try:+ result_holder["response"] = json.loads(body)+ except json.JSONDecodeError:+ result_holder["response"] = body.decode("utf-8", errors="replace")+ self.send_response(202)+ self.end_headers()+ response_received.set()+ elif f"/runtime/invocation/{request_id}/error" in self.path:+ try:+ result_holder["error"] = json.loads(body)+ except json.JSONDecodeError:+ result_holder["error"] = body.decode("utf-8", errors="replace")+ self.send_response(202)+ self.end_headers()+ response_received.set()+ elif "/runtime/init/error" in self.path:+ try:+ result_holder["error"] = json.loads(body)+ except json.JSONDecodeError:+ result_holder["error"] = body.decode("utf-8", errors="replace")+ self.send_response(202)+ self.end_headers()+ response_received.set()+ else:+ self.send_response(404)+ self.end_headers()- # Bind to port 0 — OS assigns a free port atomically, no race window- class _QuietTCPServer(socketserver.TCPServer):- def handle_error(self, request, client_address):- import sys- _, exc, _ = sys.exc_info()- if isinstance(exc, (BrokenPipeError, ConnectionResetError, ConnectionAbortedError)):- return- super().handle_error(request, client_address)+ # Bind to port 0 — OS assigns a free port atomically, no race window+ class _QuietTCPServer(socketserver.TCPServer):+ def handle_error(self, request, client_address):+ import sys+ _, exc, _ = sys.exc_info()+ if isinstance(exc, (BrokenPipeError, ConnectionResetError, ConnectionAbortedError)):+ return+ super().handle_error(request, client_address)- server = _QuietTCPServer(("127.0.0.1", 0), RuntimeAPIHandler)- port = server.server_address[1]+ server = _QuietTCPServer(("127.0.0.1", 0), RuntimeAPIHandler)+ port = server.server_address[1]- def _serve():- server_ready.set()- server.serve_forever()+ def _serve():+ server_ready.set()+ server.serve_forever()- server_thread = threading.Thread(target=_serve, daemon=True)- server_thread.start()- server_ready.wait(timeout=5)+ server_thread = threading.Thread(target=_serve, daemon=True)+ server_thread.start()+ server_ready.wait(timeout=5)- try:- # Build environment for the Lambda binary- proc_env = dict(os.environ)- proc_env.update({- "AWS_LAMBDA_RUNTIME_API": f"127.0.0.1:{port}",- "AWS_DEFAULT_REGION": get_region(),- "AWS_REGION": get_region(),- "AWS_ACCESS_KEY_ID": _account_region_from_function_config(config)[0],- "AWS_SECRET_ACCESS_KEY": os.environ.get("AWS_SECRET_ACCESS_KEY", "test"),- "AWS_LAMBDA_FUNCTION_NAME": config.get("FunctionName", "unknown"),- "LAMBDA_TASK_ROOT": code_dir,- "_HANDLER": config.get("Handler", "bootstrap"),- })- proc_env.update(env_vars)- proc_env.update(_durable_env_overlay())- # X-Ray active tracing. ``_execute_function_provided`` builds- # ``proc_env`` per-invocation, so a per-call trace ID is safe- # here (unlike the RIE pool). aws-xray-sdk reads this env var- # per-segment via ``os.getenv``, so the runtime sees it.- _xray_trace_id = _xray_trace_id_for_invocation(config)- if _xray_trace_id:- proc_env["_X_AMZN_TRACE_ID"] = _xray_trace_id- # Override AWS_ENDPOINT_URL *after* function env vars so- # Lambda binaries always call back to this MiniStack- # instance. Function-level env vars may carry the- # host-mapped URL which is unreachable from inside the- # container.- endpoint = os.environ.get("AWS_ENDPOINT_URL", "")- if not endpoint:- hostname = os.environ.get("LOCALSTACK_HOSTNAME", "")- if hostname:- endpoint = _normalize_endpoint_url(hostname)- if endpoint:- proc_env["AWS_ENDPOINT_URL"] = endpoint+ try:+ # Build environment for the Lambda binary+ proc_env = dict(os.environ)+ proc_env.update({+ "AWS_LAMBDA_RUNTIME_API": f"127.0.0.1:{port}",+ "AWS_DEFAULT_REGION": get_region(),+ "AWS_REGION": get_region(),+ "AWS_ACCESS_KEY_ID": _account_region_from_function_config(config)[0],+ "AWS_SECRET_ACCESS_KEY": os.environ.get("AWS_SECRET_ACCESS_KEY", "test"),+ "AWS_LAMBDA_FUNCTION_NAME": config.get("FunctionName", "unknown"),+ "AWS_LAMBDA_FUNCTION_MEMORY_SIZE": str(config.get("MemorySize", 128)),+ "AWS_LAMBDA_FUNCTION_VERSION": config.get("Version", "$LATEST"),+ "AWS_LAMBDA_LOG_STREAM_NAME": new_uuid(),+ "LAMBDA_TASK_ROOT": code_dir,+ "_HANDLER": config.get("Handler", "bootstrap"),+ })+ proc_env.update(env_vars)+ proc_env.update(_durable_env_overlay())+ # X-Ray active tracing. ``_execute_function_provided`` builds+ # ``proc_env`` per-invocation, so a per-call trace ID is safe+ # here (unlike the RIE pool). aws-xray-sdk reads this env var+ # per-segment via ``os.getenv``, so the runtime sees it.+ _xray_trace_id = _xray_trace_id_for_invocation(config)+ if _xray_trace_id:+ proc_env["_X_AMZN_TRACE_ID"] = _xray_trace_id+ # Override AWS_ENDPOINT_URL *after* function env vars so+ # Lambda binaries always call back to this MiniStack+ # instance. Function-level env vars may carry the+ # host-mapped URL which is unreachable from inside the+ # container.+ endpoint = os.environ.get("AWS_ENDPOINT_URL", "")+ if not endpoint:+ hostname = os.environ.get("LOCALSTACK_HOSTNAME", "")+ if hostname:+ endpoint = _normalize_endpoint_url(hostname)+ if endpoint:+ proc_env["AWS_ENDPOINT_URL"] = endpoint+ # Spawn under the code lock: no fork may overlap an extraction+ # write elsewhere, or the child inherits the open write fd and+ # execve fails with ETXTBSY (#1051).+ with _provided_code_lock:proc = subprocess.Popen([bootstrap_path],cwd=code_dir,@@ -4030,27 +4065,27 @@ def _execute_function_provided(func: dict, event: dict) -> dict:stderr=subprocess.PIPE,)- if response_received.wait(timeout=timeout):- proc.terminate()- try:- _, stderr_out = proc.communicate(timeout=5)- if stderr_out:- logger.info("Lambda %s stderr: %s", config.get("FunctionName", "?"), stderr_out.decode("utf-8", errors="replace")[:500])- except Exception:- pass- if result_holder["error"]:- err = result_holder["error"]- if isinstance(err, dict):- return {"body": err, "error": True}- return {"body": {"errorMessage": str(err), "errorType": "Runtime.HandlerError"}, "error": True}- return {"body": result_holder["response"]}- else:- proc.kill()- stdout, stderr = proc.communicate(timeout=5)- logs = (stdout.decode("utf-8", errors="replace") + stderr.decode("utf-8", errors="replace")).strip()- return {"body": {"errorMessage": f"Lambda timed out after {timeout}s: {logs[:500]}", "errorType": "Runtime.ExitError"}, "error": True}- finally:- server.shutdown()+ if response_received.wait(timeout=timeout):+ proc.terminate()+ try:+ _, stderr_out = proc.communicate(timeout=5)+ if stderr_out:+ logger.info("Lambda %s stderr: %s", config.get("FunctionName", "?"), stderr_out.decode("utf-8", errors="replace")[:500])+ except Exception:+ pass+ if result_holder["error"]:+ err = result_holder["error"]+ if isinstance(err, dict):+ return {"body": err, "error": True}+ return {"body": {"errorMessage": str(err), "errorType": "Runtime.HandlerError"}, "error": True}+ return {"body": result_holder["response"]}+ else:+ proc.kill()+ stdout, stderr = proc.communicate(timeout=5)+ logs = (stdout.decode("utf-8", errors="replace") + stderr.decode("utf-8", errors="replace")).strip()+ return {"body": {"errorMessage": f"Lambda timed out after {timeout}s: {logs[:500]}", "errorType": "Runtime.ExitError"}, "error": True}+ finally:+ server.shutdown()except Exception as e:logger.error("provided runtime execution error: %s", e)@@ -5501,6 +5536,19 @@ def _iter_all_esms():yield acct_id, region, esm+def _sqs_message_attributes_to_camel_case(attrs: dict) -> dict:+ """SQS stores attributes with the PascalCase inner keys its API uses on+ the wire (StringValue/DataType/BinaryValue/...). The Lambda ESM+ re-serializes them to camelCase before invoking the function; typed+ handler bindings (e.g. Java's SQSEvent.MessageAttribute) reject the+ PascalCase form. First-char lowercasing also covers StringListValues/+ BinaryListValues without an allowlist."""+ return {+ name: {(k[0].lower() + k[1:] if k else k): v for k, v in inner.items()}+ for name, inner in (attrs or {}).items()+ }++def _poll_sqs():from ministack.services import sqs as _sqs@@ -5556,7 +5604,9 @@ def _poll_sqs():"SenderId": get_account_id(),"ApproximateFirstReceiveTimestamp": str(int(first_recv * 1000)),},- "messageAttributes": msg.get("message_attributes", {}),+ "messageAttributes": _sqs_message_attributes_to_camel_case(+ msg.get("message_attributes", {})+ ),"md5OfBody": msg.get("md5_body") or msg.get("md5") or "","eventSource": "aws:sqs","eventSourceARN": source_arn,@@ -5935,6 +5985,8 @@ def _list_function_url_configs(func_name: str, query_params: dict):def reset():+ import shutil+from ministack.core import lambda_runtime_functions.clear()@@ -5945,6 +5997,9 @@ def reset():_dynamodb_stream_positions.clear()_pool_clear_all()lambda_runtime.reset()+ with _provided_code_lock:+ _provided_code_dirs.clear()+ shutil.rmtree(_PROVIDED_CODE_CACHE, ignore_errors=True)# ---------------------------------------------------------------------------diff --git a/ministack/services/s3.py b/ministack/services/s3.pyindex 080e31db..a359f2fa 100644--- a/ministack/services/s3.py+++ b/ministack/services/s3.py@@ -2201,9 +2201,6 @@ def _put_object(bucket_name: str, key: str, body: bytes, headers: dict):if len(pending_tags) > 10:return _error("BadRequest", "Object tags cannot be greater than 10", 400)- if S3_PERSIST:- _persist_object(bucket_name, key, obj)-_fire_s3_event_async(bucket_name, key, "s3:ObjectCreated:Put", size=obj["size"], etag=obj["etag"])@@ -2230,6 +2227,11 @@ def _put_object(bucket_name: str, key: str, body: bytes, headers: dict):for v in _object_versions[vkey][:-1]:v["is_latest"] = False+ # Persist only after the versioning block: the .meta.json sidecar must+ # carry the version_id assigned above (#1058).+ if S3_PERSIST:+ _persist_object(bucket_name, key, obj)+if pending_tags is not None:_object_tags[(bucket_name, key, obj.get("version_id"))] = pending_tagsreturn 200, resp_headers, b""@@ -2397,9 +2399,6 @@ def _post_object(bucket_name: str, body: bytes, headers: dict):if len(parsed) <= 10:pending_tags = parsed- if S3_PERSIST:- _persist_object(bucket_name, key, obj)-_fire_s3_event_async(bucket_name, key, "s3:ObjectCreated:Post", size=obj["size"], etag=etag)@@ -2423,6 +2422,11 @@ def _post_object(bucket_name: str, body: bytes, headers: dict):for v in _object_versions[vkey][:-1]:v["is_latest"] = False+ # Persist only after the versioning block: the .meta.json sidecar must+ # carry the version_id assigned above (#1058).+ if S3_PERSIST:+ _persist_object(bucket_name, key, obj)+if pending_tags is not None:_object_tags[(bucket_name, key, version_id)] = pending_tags@@ -2819,9 +2823,6 @@ def _copy_object(bucket_name: str, dest_key: str, headers: dict):else:_object_legal_hold.pop((bucket_name, dest_key), None)- if S3_PERSIST:- _persist_object(bucket_name, dest_key, dest_obj)-_fire_s3_event_async(bucket_name,dest_key,@@ -2852,6 +2853,11 @@ def _copy_object(bucket_name: str, dest_key: str, headers: dict):for v in _object_versions[vkey][:-1]:v["is_latest"] = False+ # Persist only after the versioning block: the .meta.json sidecar must+ # carry the version_id assigned above (#1058).+ if S3_PERSIST:+ _persist_object(bucket_name, dest_key, dest_obj)+dest_version_id = dest_obj.get("version_id")if pending_dest_tags is not None:_object_tags[(bucket_name, dest_key, dest_version_id)] = pending_dest_tags@@ -3942,9 +3948,6 @@ def _complete_multipart_upload(}bucket["objects"][key] = obj- if S3_PERSIST:- _persist_object(bucket_name, key, obj)-del _multipart_uploads[upload_id]_fire_s3_event_async(@@ -3974,6 +3977,11 @@ def _complete_multipart_upload(for v in _object_versions[vkey][:-1]:v["is_latest"] = False+ # Persist only after the versioning block: the .meta.json sidecar must+ # carry the version_id assigned above (#1058).+ if S3_PERSIST:+ _persist_object(bucket_name, key, obj)+root = Element("CompleteMultipartUploadResult", xmlns=S3_NS)s3_host = os.environ.get("MINISTACK_HOST", os.environ.get("AWS_ENDPOINT_URL", "http://localhost:4566"))SubElement(root, "Location").text = f"{s3_host}/{bucket_name}/{key}"@@ -4195,6 +4203,7 @@ def _persist_object(bucket: str, key: str, obj):"preserved_headers": obj.get("preserved_headers", {}),"storage_class": obj.get("storage_class", "STANDARD"),"checksums": obj.get("checksums", {}),+ "version_id": obj.get("version_id"),}_atomic_write(fpath + ".meta.json", json.dumps(meta), text=True)# Drop body from in-memory record to save RAM@@ -4357,6 +4366,8 @@ def _load_persisted_bucket(account_id, bucket_name, bucket_path):"storage_class": meta.get("storage_class", "STANDARD"),"checksums": meta.get("checksums", {}),}+ if meta.get("version_id"):+ bucket["objects"][key]["version_id"] = meta["version_id"]_load_persisted_data()diff --git a/pyproject.toml b/pyproject.tomlindex 42c0f76b..30eadc50 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"[project]name = "ministack"-version = "1.4.0"+version = "1.4.1"description = "Free, open-source local AWS emulator — drop-in LocalStack replacement"readme = "README.md"license = { file = "LICENSE" }@@ -18,6 +18,7 @@ classifiers = ["Programming Language :: Python :: 3.10","Programming Language :: Python :: 3.11","Programming Language :: Python :: 3.12",+ "Programming Language :: Python :: 3.13","Topic :: Software Development :: Testing","Topic :: Internet",]diff --git a/tests/test_eks.py b/tests/test_eks.pyindex f8900d5a..4a7e8b56 100644--- a/tests/test_eks.py+++ b/tests/test_eks.py@@ -478,6 +478,45 @@ def test_eks_k3s_run_kwargs_container_name_and_labels():assert kwargs["labels"] == {"ministack": "eks", "cluster_name": "my-cluster"}+def test_eks_k3s_run_kwargs_host_gateway_extra_host():+ """The k3s node must be able to reach a host-run MiniStack for ECR+ registry mirroring (#1054) — host.docker.internal via host-gateway."""+ from ministack.services.eks import _k3s_run_kwargs++ kwargs = _k3s_run_kwargs(name="c1", port=16443)+ assert kwargs["extra_hosts"] == {"host.docker.internal": "host-gateway"}+++def test_eks_ecr_registry_hosts_from_cluster_arn():+ """ECR mirror hostnames derive from the cluster ARN, not contextvars,+ so restore/restart paths (no request context) behave like create (#1054)."""+ from ministack.services.eks import _ecr_registry_hosts++ cluster = {"arn": "arn:aws:eks:eu-west-1:123456789012:cluster/my-cluster"}+ assert _ecr_registry_hosts(cluster) == ["123456789012.dkr.ecr.eu-west-1.amazonaws.com"]+ assert _ecr_registry_hosts({"arn": "not-an-arn"}) == []+ assert _ecr_registry_hosts({}) == []+++def test_eks_k3s_registries_yaml_shape(monkeypatch):+ """registries.yaml maps the cluster's ECR hostname to the gateway; with+ no shared network the endpoint goes through host.docker.internal (#1054)."""+ from ministack.services.eks import _k3s_registries_yaml++ monkeypatch.delenv("GATEWAY_PORT", raising=False)+ monkeypatch.delenv("EDGE_PORT", raising=False)+ yaml_bytes = _k3s_registries_yaml(+ None, None, ["123456789012.dkr.ecr.eu-west-1.amazonaws.com"])+ text = yaml_bytes.decode()+ assert text == (+ "mirrors:\n"+ ' "123456789012.dkr.ecr.eu-west-1.amazonaws.com":\n'+ " endpoint:\n"+ ' - "http://host.docker.internal:4566"\n'+ )+ assert _k3s_registries_yaml(None, None, []) is None++def test_eks_addon_lifecycle(eks):"""CreateAddon / DescribeAddon / ListAddons / UpdateAddon / DeleteAddon.Issue #752: terraform aws_eks_addon fails on missing POST /clusters/{name}/addons."""diff --git a/tests/test_lambda.py b/tests/test_lambda.pyindex 2dfcf1cf..f2351047 100644--- a/tests/test_lambda.py+++ b/tests/test_lambda.py@@ -893,6 +893,93 @@ def test_lambda_esm_sqs(lam, sqs):# Cleanuplam.delete_event_source_mapping(UUID=esm_uuid)++def test_sqs_esm_message_attributes_to_camel_case_helper():+ """SQS PascalCase inner keys become Lambda-event camelCase (#1059)."""+ from ministack.services.lambda_svc import _sqs_message_attributes_to_camel_case as conv++ attrs = {+ "version": {"DataType": "String", "StringValue": "1.0"},+ "blob": {"DataType": "Binary", "BinaryValue": "aGk="},+ "lists": {"DataType": "String", "StringListValues": ["a"], "BinaryListValues": []},+ }+ out = conv(attrs)+ assert out["version"] == {"dataType": "String", "stringValue": "1.0"}+ assert out["blob"] == {"dataType": "Binary", "binaryValue": "aGk="}+ assert out["lists"] == {"dataType": "String", "stringListValues": ["a"], "binaryListValues": []}+ assert conv({}) == {}+ assert conv(None) == {}+++def test_lambda_esm_sqs_message_attributes_camel_case(lam, sqs):+ """SQS → Lambda ESM delivers messageAttributes with camelCase inner keys (#1059).++ The handler raises on PascalCase input, so the message is only consumed+ (deleted from the queue) when the transformation happened."""+ try:+ lam.delete_function(FunctionName="esm-attr-func")+ except ClientError:+ pass++ code = (+ "def handler(event, context):\n"+ " for r in event['Records']:\n"+ " attr = r['messageAttributes']['version']\n"+ " assert attr['stringValue'] == '1.0', attr\n"+ " assert attr['dataType'] == 'String', attr\n"+ " assert 'StringValue' not in attr, attr\n"+ " return 'ok'\n"+ )+ lam.create_function(+ FunctionName="esm-attr-func",+ Runtime="python3.12",+ Role=_LAMBDA_ROLE,+ Handler="index.handler",+ Code={"ZipFile": _make_zip(code)},+ )+ q_url = sqs.create_queue(QueueName="esm-attr-queue")["QueueUrl"]+ q_arn = sqs.get_queue_attributes(+ QueueUrl=q_url, AttributeNames=["QueueArn"]+ )["Attributes"]["QueueArn"]+ resp = lam.create_event_source_mapping(+ EventSourceArn=q_arn,+ FunctionName="esm-attr-func",+ BatchSize=1,+ Enabled=True,+ )+ esm_uuid = resp["UUID"]+ try:+ sqs.send_message(+ QueueUrl=q_url,+ MessageBody="attr-check",+ MessageAttributes={"version": {"DataType": "String", "StringValue": "1.0"}},+ )+ # Poll queue counters (not receive_message — that would race the poller).+ deadline = time.time() + 15+ remaining = None+ while time.time() < deadline:+ attrs = sqs.get_queue_attributes(+ QueueUrl=q_url,+ AttributeNames=[+ "ApproximateNumberOfMessages",+ "ApproximateNumberOfMessagesNotVisible",+ ],+ )["Attributes"]+ remaining = (int(attrs["ApproximateNumberOfMessages"])+ + int(attrs["ApproximateNumberOfMessagesNotVisible"]))+ if remaining == 0:+ break+ time.sleep(0.5)+ assert remaining == 0, (+ "message not consumed — handler rejected messageAttributes "+ "(inner keys not camelCase?)"+ )+ finally:+ lam.delete_event_source_mapping(UUID=esm_uuid)+ lam.delete_function(FunctionName="esm-attr-func")+ sqs.delete_queue(QueueUrl=q_url)++def test_lambda_create_function(lam):resp = lam.create_function(FunctionName="lam-create-test",@@ -3243,6 +3330,113 @@ def test_lambda_provided_runtime_docker_invoke(lam):lam.delete_function(FunctionName=func_name)+def test_lambda_provided_runtime_env_has_function_vars():+ """_execute_function_provided re-injects AWS_LAMBDA_FUNCTION_MEMORY_SIZE /+ _VERSION / LOG_STREAM_NAME from the function config (#1060).++ _runtime_env_vars() strips these reserved names from the user env, so the+ executor must set them itself — the Rust lambda_runtime crate panics when+ AWS_LAMBDA_FUNCTION_MEMORY_SIZE is absent. The bootstrap here echoes the+ env back through the Runtime API."""+ from ministack.services import lambda_svc as lmod++ bootstrap_script = (+ "#!/usr/bin/env python3\n"+ "import json, os, urllib.request\n"+ "api = os.environ['AWS_LAMBDA_RUNTIME_API']\n"+ "nxt = urllib.request.urlopen(f'http://{api}/2018-06-01/runtime/invocation/next')\n"+ "rid = nxt.headers['Lambda-Runtime-Aws-Request-Id']\n"+ "body = json.dumps({\n"+ " 'memory': os.environ.get('AWS_LAMBDA_FUNCTION_MEMORY_SIZE'),\n"+ " 'version': os.environ.get('AWS_LAMBDA_FUNCTION_VERSION'),\n"+ " 'log_stream': os.environ.get('AWS_LAMBDA_LOG_STREAM_NAME'),\n"+ "}).encode()\n"+ "urllib.request.urlopen(urllib.request.Request(\n"+ " f'http://{api}/2018-06-01/runtime/invocation/{rid}/response',\n"+ " data=body, method='POST'))\n"+ )+ buf = io.BytesIO()+ with zipfile.ZipFile(buf, "w") as zf:+ info = zipfile.ZipInfo("bootstrap")+ info.external_attr = 0o755 << 16+ zf.writestr(info, bootstrap_script)++ func = {+ "config": {+ "FunctionName": "provided-env-1060",+ "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:provided-env-1060",+ "MemorySize": 512,+ "Version": "$LATEST",+ "Timeout": 20,+ "Handler": "bootstrap",+ # Reserved names are stripped from the user env — the runtime+ # values must come from the function config, not from here.+ "Environment": {"Variables": {"AWS_LAMBDA_FUNCTION_MEMORY_SIZE": "9999"}},+ },+ "code_zip": buf.getvalue(),+ }+ result = lmod._execute_function_provided(func, {"ping": "pong"})+ assert not result.get("error"), result+ body = result["body"]+ assert body["memory"] == "512"+ assert body["version"] == "$LATEST"+ assert body["log_stream"]+++def test_lambda_provided_runtime_parallel_invocations():+ """Concurrent provided.* invocations must not fail with ETXTBSY (#1051).++ Pre-fix, every invocation extracted the code zip into its own tempdir;+ while one thread still held the extraction's write fd on ``bootstrap``,+ another thread's Popen fork let the child inherit it and execve failed+ with 'Text file busy'. Post-fix all invocations share one read-only+ per-sha extraction and spawns are serialized against extraction."""+ import concurrent.futures+ import hashlib+ from ministack.services import lambda_svc as lmod++ bootstrap_script = (+ "#!/usr/bin/env python3\n"+ "import json, os, urllib.request\n"+ "api = os.environ['AWS_LAMBDA_RUNTIME_API']\n"+ "nxt = urllib.request.urlopen(f'http://{api}/2018-06-01/runtime/invocation/next')\n"+ "rid = nxt.headers['Lambda-Runtime-Aws-Request-Id']\n"+ "event = json.loads(nxt.read())\n"+ "urllib.request.urlopen(urllib.request.Request(\n"+ " f'http://{api}/2018-06-01/runtime/invocation/{rid}/response',\n"+ " data=json.dumps({'echo': event.get('n')}).encode(), method='POST'))\n"+ )+ buf = io.BytesIO()+ with zipfile.ZipFile(buf, "w") as zf:+ info = zipfile.ZipInfo("bootstrap")+ info.external_attr = 0o755 << 16+ zf.writestr(info, bootstrap_script)++ func = {+ "config": {+ "FunctionName": "provided-parallel-1051",+ "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:provided-parallel-1051",+ "MemorySize": 128,+ "Timeout": 20,+ "Handler": "bootstrap",+ },+ "code_zip": buf.getvalue(),+ }++ with concurrent.futures.ThreadPoolExecutor(max_workers=8) as ex:+ results = list(ex.map(+ lambda i: lmod._execute_function_provided(func, {"n": i}), range(8)))++ for i, r in enumerate(results):+ assert not r.get("error"), f"invocation {i} failed: {r}"+ assert r["body"]["echo"] == i++ # All invocations shared a single per-sha extraction directory.+ sha = hashlib.sha256(func["code_zip"]).hexdigest()+ code_dir = lmod._provided_code_dirs.get(sha)+ assert code_dir and os.path.isdir(code_dir)++def test_apigwv2_nodejs_lambda_proxy(lam, apigw):"""API Gateway v2 HTTP API should invoke Node.js Lambda via warm worker, not return mock."""import urllib.request as _urlreqdiff --git a/tests/test_s3.py b/tests/test_s3.pyindex 8beacd6a..b8fe20f4 100644--- a/tests/test_s3.py+++ b/tests/test_s3.py@@ -2363,6 +2363,67 @@ def test_s3_storage_class_persisted_to_disk(tmp_path, monkeypatch):assert restored["storage_class"] == "GLACIER"+def test_s3_version_id_persisted_to_disk(tmp_path, monkeypatch):+ """version_id survives _persist_object → _load_persisted_bucket round-trip (#1058)."""+ from ministack.services import s3 as s3mod+ monkeypatch.setattr(s3mod, "DATA_DIR", str(tmp_path))+ monkeypatch.setattr(s3mod, "S3_PERSIST", True)+ monkeypatch.setattr(s3mod, "get_account_id", lambda: "000000000000")++ obj = {+ "body": b"hello",+ "content_type": "application/octet-stream",+ "content_encoding": None,+ "etag": '"abc"',+ "last_modified": s3mod.now_iso(),+ "size": 5,+ "metadata": {},+ "preserved_headers": {},+ "storage_class": "STANDARD",+ "version_id": "test-version-1058",+ }+ s3mod._persist_object("ver-bucket", "k", obj)++ meta_path = os.path.join(str(tmp_path), "000000000000", "ver-bucket", "k.meta.json")+ with open(meta_path) as mf:+ assert json.load(mf)["version_id"] == "test-version-1058"++ s3mod._buckets._data.pop(("000000000000", "ver-bucket"), None)+ try:+ s3mod._load_persisted_bucket(+ "000000000000", "ver-bucket",+ os.path.join(str(tmp_path), "000000000000", "ver-bucket"))+ restored = s3mod._buckets._data[("000000000000", "ver-bucket")]["objects"]["k"]+ assert restored["version_id"] == "test-version-1058"+ finally:+ s3mod._buckets._data.pop(("000000000000", "ver-bucket"), None)+++def test_s3_put_object_sidecar_carries_version_id(tmp_path, monkeypatch):+ """PutObject on a versioned bucket must persist AFTER version_id assignment,+ so the on-disk .meta.json carries the id returned in x-amz-version-id (#1058)."""+ from ministack.core import responses as respmod+ from ministack.services import s3 as s3mod+ monkeypatch.setattr(s3mod, "DATA_DIR", str(tmp_path))+ monkeypatch.setattr(s3mod, "S3_PERSIST", True)+ monkeypatch.setattr(s3mod, "get_account_id", lambda: "000000000000")+ monkeypatch.setattr(respmod, "get_account_id", lambda: "000000000000")+ try:+ s3mod._create_bucket("ver-put-bucket", b"")+ s3mod._bucket_versioning["ver-put-bucket"] = "Enabled"+ status, resp_headers, _ = s3mod._put_object("ver-put-bucket", "k", b"hello", {})+ assert status == 200+ version_id = resp_headers["x-amz-version-id"]+ meta_path = os.path.join(+ str(tmp_path), "000000000000", "ver-put-bucket", "k.meta.json")+ with open(meta_path) as mf:+ assert json.load(mf)["version_id"] == version_id+ finally:+ s3mod._buckets._data.pop(("000000000000", "ver-put-bucket"), None)+ s3mod._bucket_versioning.pop("ver-put-bucket", None)+ s3mod._object_versions.pop(("ver-put-bucket", "k"), None)++def test_s3_create_bucket_persists_account_scoped(tmp_path, monkeypatch):"""CreateBucket persists under DATA_DIR/<account>/<bucket>, never DATA_DIR/<bucket> (#824)."""from ministack.core import responses as respmod