Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.
Python awsaws-emulatoraws-localaws-sdkdevtoolsdockerdynamodbec2emulatorlambdalocalstacklocalstack-alternativeministackmock-awsopen-sourcepythons3sqsterraform
feat(lambda): add snapstart (#1603)
Nahuel Nucerab5a2e2528d15
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9d6f817b..a8ecacd5 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -19,6 +19,9 @@ Versioning follows [Semantic Versioning](https://semver.org/).- **Amazon Transcribe (`transcribe`)** is a new service emulator covering batch transcription jobs: `StartTranscriptionJob`, `GetTranscriptionJob`, `ListTranscriptionJobs` and `DeleteTranscriptionJob`, verified against botocore `transcribe-2017-10-26`. A job walks `QUEUED` → `IN_PROGRESS` → `COMPLETED` on a background task paced by `TRANSCRIBE_JOB_RUN_SECONDS` (default 2, `0` completes immediately), reads its media object from MiniStack's S3 store through an `s3://`, path-style or virtual-hosted HTTPS URI, and writes a transcript document back to S3 in the real Transcribe result format: `results.transcripts`, word-level `results.items`, `results.audio_segments`, and `results.segments` / `results.speaker_labels` when `ShowAlternatives` and `ShowSpeakerLabels` are set. `IdentifyLanguage` fills in `LanguageCode` and `IdentifiedLanguageScore` on completion while `IdentifyMultipleLanguages` reports `LanguageCodes` and leaves `LanguageCode` unset, as AWS does. `ContentRedaction` decides whether the `redacted-` copy replaces the transcript or sits beside it, and `Subtitles` writes WebVTT and SubRip files reported as `SubtitleFileUris`, redacted and renamed to match when redaction is on. Output lands at `OutputBucketName` / `OutputKey`, or in an auto-created `ministack-transcribe-output` bucket when neither is given, so the reported URIs are always fetchable. Media that cannot be read fails the job asynchronously with a `FailureReason` rather than erroring at `Start`, matching AWS, and a job left mid-flight by a restart fails on restore instead of stranding pollers. Reaching a terminal state publishes a `Transcribe Job State Change` event on the default bus, so an S3 → Lambda → Transcribe → EventBridge pipeline runs end to end locally. There is **no speech recognition**: the transcript is deterministic canned text keyed on a digest of the media URI, the same approach Bedrock Runtime takes for model output, which keeps integration tests reproducible. Streaming transcription, the call-analytics and medical job families, custom vocabularies and the tagging operations are not implemented. State is account- and region-scoped and persisted. Contributed by @ppettitau.+### Fixed+- **Lambda — a failed Docker cold start no longer leaks its extraction directory** — the executor unpacks the function zip and its layers into a `ministack-lambda-docker-*` temp directory before talking to Docker, but only three specific failure paths cleaned it up; anything else raised between the extraction and the container start — a corrupt layer zip, or a Docker socket read-timeout out of the image lookup (which is not `ImageNotFound`) — orphaned the directory. Under Docker-API pressure that compounds: each timed-out spawn leaves a full unpacked tree behind, measured at gigabytes of `/tmp` in minutes on a large stack. Cleanup now covers every exit: the directory is owned by the spawn until the container is handed to the warm pool, and removed on any failure in between. Reported by @iot-rocket.+## [1.5.5] — 2026-09-01### Addeddiff --git a/ministack/core/lambda_runtime.py b/ministack/core/lambda_runtime.pyindex ff91e0ba..113e16e7 100644--- a/ministack/core/lambda_runtime.py+++ b/ministack/core/lambda_runtime.py@@ -183,6 +183,25 @@ def run():try:mod = importlib.import_module(module_name)handler_fn = getattr(mod, handler_name)+ if init.get("snapstart"):+ # SnapStart runtime hooks (snapshot-restore-py, bundled in AWS's+ # python3.12+ runtimes). Each environment of a published SnapStart+ # version runs one snapshot/restore cycle: before-snapshot hooks+ # in reverse registration order, then after-restore hooks in+ # registration order, before the first invocation. A hook that+ # raises fails the init, which fails the publish, as on AWS.+ try:+ from snapshot_restore_py import (+ get_after_restore,+ get_before_snapshot,+ )+ except ImportError:+ pass+ else:+ for _fn, _args, _kwargs in reversed(get_before_snapshot()):+ _fn(*_args, **_kwargs)+ for _fn, _args, _kwargs in get_after_restore():+ _fn(*_args, **_kwargs)_real_stdout.write(json.dumps({"status": "ready", "cold": True}) + "\\n")_real_stdout.flush()except Exception as e:@@ -1132,6 +1151,11 @@ class Worker:"function_name": self.config.get("FunctionName", ""),"memory": self.config.get("MemorySize", 128),"arn": self.config.get("FunctionArn", ""),+ # A published SnapStart version runs its snapshot/restore runtime+ # hooks during init (see the worker script). $LATEST never does —+ # it has no snapshot on AWS either.+ "snapstart": (self.config.get("SnapStart") or {}).get(+ "OptimizationStatus") == "On",}self._proc.stdin.write(json.dumps(init) + "\n")self._proc.stdin.flush()@@ -1426,7 +1450,18 @@ def reap_idle_workers(ttl: float = None) -> int:keep = []for i, worker in enumerate(entries):idle = now - getattr(worker, "last_used", now)- if i == 0 or worker.in_use or idle < ttl:+ # The first worker per key is kept warm indefinitely — except+ # for a published SnapStart version's worker, which exists+ # because PublishVersion pre-initialized it: without the+ # exception every publish would pin one subprocess forever.+ # Reaping it costs a re-init on the next invoke, the same as+ # the docker pool's TTL eviction.+ snapstart_version = (+ (worker.config.get("SnapStart") or {}).get(+ "OptimizationStatus") == "On"+ and worker.config.get("Version", "$LATEST") != "$LATEST"+ )+ if worker.in_use or idle < ttl or (i == 0 and not snapstart_version):keep.append(worker)else:killed.append(worker)@@ -1438,6 +1473,20 @@ def reap_idle_workers(ttl: float = None) -> int:return len(killed)+def ensure_spawned(worker: Worker) -> None:+ """Spawn the worker's subprocess now if it isn't running.++ SnapStart moves initialization to PublishVersion: the publish path leases+ a worker for the new version and forces the cold start here, so an init+ failure fails the publish (version State=Failed, as on AWS) and the first+ invoke finds the environment already warm.+ """+ with worker._lock:+ if worker._proc is None or worker._proc.poll() is not None:+ worker._spawn()+ worker._cold = False++def release_worker(worker: Worker) -> None:"""Return a leased worker to its pool."""if worker is None:diff --git a/ministack/services/cloudformation/provisioners.py b/ministack/services/cloudformation/provisioners.pyindex 32a774ba..b796856d 100644--- a/ministack/services/cloudformation/provisioners.py+++ b/ministack/services/cloudformation/provisioners.py@@ -1084,6 +1084,7 @@ def _lambda_create(logical_id, props, stack_name):"EphemeralStorage": {"Size": props.get("EphemeralStorage", {}).get("Size", 512)},"TracingConfig": props.get("TracingConfig", {"Mode": "PassThrough"}),"LoggingConfig": props.get("LoggingConfig", {"LogFormat": "Text", "LogGroup": f"/aws/lambda/{name}"}),+ "SnapStart": _lambda_svc._snapstart_response(props.get("SnapStart")),"RevisionId": new_uuid(),},"code_zip": code_zip,@@ -1193,6 +1194,7 @@ def _lambda_update(physical_id, old_props, new_props, stack_name, logical_id=Non"LoggingConfig", {"LogFormat": "Text", "LogGroup": f"/aws/lambda/{name}"}),"Architectures": new_props.get("Architectures", ["x86_64"]),+ "SnapStart": new_props.get("SnapStart", {"ApplyOn": "None"}),}if new_props.get("ImageConfig"):config_data["ImageConfig"] = new_props["ImageConfig"]diff --git a/ministack/services/lambda_svc.py b/ministack/services/lambda_svc.pyindex 62386c24..160d7e99 100644--- a/ministack/services/lambda_svc.py+++ b/ministack/services/lambda_svc.py@@ -56,6 +56,7 @@ from ministack.core.lambda_runtime import (INVOKE_DEPTH_EVENT_KEY,INVOKE_DEPTH_HEADER,acquire_worker,+ ensure_spawned,invalidate_worker,reap_idle_workers,release_worker,@@ -563,6 +564,20 @@ def restore_state(data):_function_urls.update(data.get("function_urls", {}))_restore_esm_positions(_kinesis_positions, data.get("kinesis_positions", {}))_restore_esm_positions(_dynamodb_stream_positions, data.get("dynamodb_stream_positions", {}))+ # A SnapStart version persisted mid-publish restores as State=Pending+ # with no provisioning thread behind it — and Pending SnapStart+ # versions answer 409 on Invoke and are skipped by the generic state+ # flipper, so without re-provisioning here the version would be+ # uninvokable forever. Re-run the publish-time initialization.+ for scoped_key, func in list(_functions._data.items()):+ fn_name = scoped_key[-1]+ for ver_record in (func.get("versions") or {}).values():+ cfg = ver_record.get("config") or {}+ if (+ cfg.get("State") == "Pending"+ and (cfg.get("SnapStart") or {}).get("OptimizationStatus") == "On"+ ):+ _snapstart_provision_version_async(fn_name, ver_record)if _esms.has_any():_ensure_poller()@@ -1285,6 +1300,160 @@ def _durable_arn_lookup(name_or_arn: str) -> str | None:return func.get("FunctionArn") or _func_arn(canonical)+# ---------------------------------------------------------------------------+# SnapStart+# ---------------------------------------------------------------------------+# AWS accepts ``SnapStart.ApplyOn`` on CreateFunction and+# UpdateFunctionConfiguration and reports ``{ApplyOn, OptimizationStatus}`` on+# every read. $LATEST always reports OptimizationStatus=Off — a snapshot only+# exists per published version — while a version published with+# ApplyOn=PublishedVersions reports On, sits in State=Pending while the+# snapshot is created, then transitions to Active. Supported runtimes per the+# SnapStart docs: Java 11+, Python 3.12+, .NET 8+, for both Zip and+# container-image packages; incompatible with ephemeral storage above 512 MB.++_SNAPSTART_APPLY_ON_VALUES = ("PublishedVersions", "None")+++def _snapstart_runtime_supported(runtime: str) -> bool:+ m = re.match(r"^(java|python|dotnet)(\d+)(?:\.(\d+))?", runtime or "")+ if not m:+ return False+ family, major, minor = m.group(1), int(m.group(2)), int(m.group(3) or 0)+ if family == "java":+ return major >= 11+ if family == "python":+ return (major, minor) >= (3, 12)+ return major >= 8 # dotnet+++def _validate_snapstart(snap: dict | None, *, runtime: str, package_type: str,+ ephemeral_size) -> tuple | None:+ """Validate a SnapStart request block against AWS's documented constraints."""+ if snap is None:+ return None+ apply_on = (snap or {}).get("ApplyOn", "None")+ if apply_on not in _SNAPSTART_APPLY_ON_VALUES:+ return error_response_json(+ "ValidationException",+ f"1 validation error detected: Value '{apply_on}' at "+ "'snapStart.applyOn' failed to satisfy constraint: Member must "+ "satisfy enum value set: [PublishedVersions, None]",+ 400,+ )+ if apply_on == "None":+ return None+ if package_type != "Image" and not _snapstart_runtime_supported(runtime):+ return error_response_json(+ "InvalidParameterValueException",+ f"SnapStart is not supported for the {runtime} runtime.",+ 400,+ )+ try:+ too_big = ephemeral_size is not None and int(ephemeral_size) > 512+ except (TypeError, ValueError):+ too_big = False+ if too_big:+ return error_response_json(+ "InvalidParameterValueException",+ "SnapStart is not supported with ephemeral storage greater than "+ "512 MB.",+ 400,+ )+ return None+++def _snapstart_response(snap: dict | None) -> dict:+ apply_on = (snap or {}).get("ApplyOn") or "None"+ return {"ApplyOn": apply_on, "OptimizationStatus": "Off"}+++def _stamp_snapstart_published_version(ver_config: dict) -> bool:+ """AWS creates the snapshot when a version is published: the version+ reports OptimizationStatus=On and stays Pending until the snapshot is+ ready. Returns True when the version is SnapStart-enabled so the caller+ schedules the Pending→Active transition."""+ if (ver_config.get("SnapStart") or {}).get("ApplyOn") != "PublishedVersions":+ return False+ ver_config["SnapStart"] = {+ "ApplyOn": "PublishedVersions",+ "OptimizationStatus": "On",+ }+ ver_config["State"] = "Pending"+ ver_config["StateReason"] = "The function is being created."+ ver_config["StateReasonCode"] = "Creating"+ return True+++def _snapstart_provision_version_async(name: str, ver_record: dict) -> None:+ """SnapStart moves initialization to PublishVersion: create the version's+ execution environment now — a warm worker for python, a pooled RIE+ container for java/dotnet/Image — instead of at first invoke.++ Success flips the version Pending → Active with the environment already+ warm (the snapshot's observable effect); an init failure — a handler that+ won't import, a before-snapshot/after-restore hook that raises — lands the+ version in State=Failed, as AWS surfaces a failed snapshot. This thread+ owns the version's state transition; the generic Pending flipper skips+ SnapStart versions.+ """+ ver_config = ver_record["config"]+ code_zip = ver_record.get("code_zip")++ def _provision():+ try:+ account_id, region = _account_region_from_function_config(ver_config)+ _request_account_id.set(account_id)+ _request_region.set(region)+ except ArnParseError:+ pass+ try:+ runtime = ver_config.get("Runtime", "")+ package_type = ver_config.get("PackageType", "Zip")+ use_docker = (+ LAMBDA_STRICT+ or package_type == "Image"+ or LAMBDA_EXECUTOR == "docker"+ or not runtime.startswith("python")+ )+ if use_docker:+ if _docker_available and _get_docker_client() is not None:+ container, tmpdir = _spawn_lambda_container(ver_config, code_zip)+ key = _warm_pool_key(ver_config["FunctionName"], ver_config)+ entry = _pool_register(key, container, tmpdir)+ _pool_release(entry)+ _ensure_reaper_thread()+ # Docker unreachable: nothing to pre-warm; the invoke path+ # falls back exactly as it always did.+ elif code_zip:+ worker, _reason = acquire_worker(+ ver_config["FunctionName"], ver_config, code_zip,+ qualifier=ver_config.get("Version", "$LATEST"),+ )+ try:+ ensure_spawned(worker)+ finally:+ release_worker(worker)+ _ensure_reaper_thread()+ except Exception as exc:+ logger.warning("Lambda %s: SnapStart init failed at publish: %s",+ name, exc)+ ver_config["State"] = "Failed"+ ver_config["StateReason"] = str(exc)+ ver_config["StateReasonCode"] = "FunctionError"+ return+ ver_config["State"] = "Active"+ ver_config["StateReason"] = ""+ ver_config["StateReasonCode"] = ""+ ver_config["LastUpdateStatus"] = "Successful"++ ctx_snapshot = contextvars.copy_context()+ threading.Thread(+ target=ctx_snapshot.run, args=(_provision,), daemon=True,+ name="ministack-lambda-snapstart-init",+ ).start()++def _build_config(name: str, data: dict, code_zip: bytes | None = None) -> dict:code_size = len(code_zip) if code_zip else 0code_sha = base64.b64encode(hashlib.sha256(code_zip).digest()).decode() if code_zip else ""@@ -1343,7 +1512,7 @@ def _build_config(name: str, data: dict, code_zip: bytes | None = None) -> dict:"KMSKeyArn": data.get("KMSKeyArn", ""),"RevisionId": new_uuid(),"EphemeralStorage": data.get("EphemeralStorage", {"Size": 512}),- "SnapStart": {"ApplyOn": "None", "OptimizationStatus": "Off"},+ "SnapStart": _snapstart_response(data.get("SnapStart")),"LoggingConfig": data.get("LoggingConfig",{@@ -1967,6 +2136,15 @@ def _create_function(data: dict):data = dict(data)data["Layers"] = layers_cfg+ err = _validate_snapstart(+ data.get("SnapStart"),+ runtime=data.get("Runtime", ""),+ package_type=data.get("PackageType", "Zip"),+ ephemeral_size=(data.get("EphemeralStorage") or {}).get("Size", 512),+ )+ if err:+ return err+# Validate the execution role before building anything: _build_config# returns a config dict, so the error has to be raised by the caller, and# nothing may be persisted for a function that is going to be refused.@@ -2020,10 +2198,13 @@ def _create_function(data: dict):_functions[name]["next_version"] = ver_num + 1ver_config = copy.deepcopy(config)ver_config["Version"] = str(ver_num)- _functions[name]["versions"][str(ver_num)] = {+ ver_record = {"config": ver_config,"code_zip": code_zip,}+ _functions[name]["versions"][str(ver_num)] = ver_record+ if _stamp_snapstart_published_version(ver_config):+ _snapstart_provision_version_async(name, ver_record)config["Version"] = str(ver_num)_schedule_state_transition(name, _LAMBDA_STATE_TRANSITION_DELAY)@@ -2346,6 +2527,10 @@ def _delete_function(name: str, query_params: dict, path_qualifier: str | None =409,)_functions[name]["versions"].pop(qualifier, None)+ # The version's warm worker (SnapStart pre-initialized it at publish,+ # or a past invoke spawned it) has nothing left to serve.+ invalidate_worker(name, qualifier=qualifier,+ account=get_account_id(), region=get_region())else:del _functions[name]invalidate_worker(name, account=get_account_id(), region=get_region())@@ -2430,10 +2615,13 @@ def _update_code(name: str, data: dict):func["next_version"] = ver_num + 1ver_config = copy.deepcopy(func["config"])ver_config["Version"] = str(ver_num)- func["versions"][str(ver_num)] = {+ ver_record = {"config": ver_config,"code_zip": func.get("code_zip"),}+ func["versions"][str(ver_num)] = ver_record+ if _stamp_snapstart_published_version(ver_config):+ _snapstart_provision_version_async(name, ver_record)func["config"]["Version"] = str(ver_num)return json_response(func["config"])@@ -2456,6 +2644,21 @@ def _update_config(name: str, data: dict):data = dict(data)data["Layers"] = layers_cfgconfig = _functions[name]["config"]+ if "SnapStart" in data or "Runtime" in data or "EphemeralStorage" in data:+ # Validate the *effective* combination, so enabling SnapStart, moving+ # to an unsupported runtime, or raising ephemeral storage past 512 MB+ # while SnapStart is on are all refused before any field mutates.+ effective_snap = data.get("SnapStart", config.get("SnapStart"))+ err = _validate_snapstart(+ effective_snap,+ runtime=data.get("Runtime") or config.get("Runtime", ""),+ package_type=config.get("PackageType", "Zip"),+ ephemeral_size=(data.get("EphemeralStorage")+ or config.get("EphemeralStorage")+ or {}).get("Size", 512),+ )+ if err:+ return errfor key in ("Runtime","Handler",@@ -2476,6 +2679,7 @@ def _update_config(name: str, data: dict):"DurableConfig","TenancyConfig","CapacityProviderConfig",+ "SnapStart",):if key in data:if key == "Layers":@@ -2489,6 +2693,10 @@ def _update_config(name: str, data: dict):layer["CodeSize"] = _layer_codesize_for_arn(layer["Arn"])layers_cfg.append(layer)config["Layers"] = layers_cfg+ elif key == "SnapStart":+ # Request carries only ApplyOn; the stored/echoed shape adds+ # OptimizationStatus, which is always Off on $LATEST.+ config["SnapStart"] = _snapstart_response(data["SnapStart"])else:config[key] = data[key]if "Architectures" in data:@@ -2564,6 +2772,21 @@ async def _invoke(name: str, event: dict, headers: dict, path_qualifier: str | N404,)+ exec_config = exec_record.get("config") or {}+ if (+ executed_version != "$LATEST"+ and (exec_config.get("SnapStart") or {}).get("OptimizationStatus") == "On"+ and exec_config.get("State") == "Pending"+ ):+ # AWS: invoking a SnapStart version while its snapshot is being+ # created answers ResourceConflictException until it goes Active.+ return error_response_json(+ "ResourceConflictException",+ "The operation cannot be performed at this time. "+ "The function is currently in the following state: Pending",+ 409,+ )+if invocation_type == "DryRun":return 204, {"X-Amz-Executed-Version": executed_version}, b""@@ -3094,6 +3317,13 @@ def _schedule_state_transition(func_name: str, delay: float) -> None:and cfg.get("LastUpdateStatus") not in (None, "", "InProgress")):continue+ # A pending SnapStart version is mid-initialization: its+ # provisioning thread owns the Pending → Active/Failed decision.+ if (+ cfg.get("Version") not in (None, "", "$LATEST")+ and (cfg.get("SnapStart") or {}).get("OptimizationStatus") == "On"+ ):+ continuecfg["State"] = "Active"cfg["StateReason"] = ""cfg["StateReasonCode"] = ""@@ -3675,6 +3905,29 @@ def _declared_docker_platform(config: dict):def _spawn_lambda_container(config: dict, code_zip: bytes | None,_pin_platform: bool = True):+ """Create and start a Lambda container, never leaking the extraction dir.++ Everything between the extraction tmpdir's creation and the pool+ registration can raise — a corrupt code or layer zip, or any Docker API+ failure that is not one of the specifically-handled cases (e.g. a socket+ read timeout out of ``images.get``, which is not ``ImageNotFound``).+ Ownership of the tmpdir passes to the caller only on a successful return;+ on any exception it is removed here instead of orphaning one extraction+ per failed cold start (issue #1600).+ """+ made_tmpdirs: list[str] = []+ try:+ return _spawn_lambda_container_impl(config, code_zip, _pin_platform,+ made_tmpdirs)+ except BaseException:+ import shutil+ for d in made_tmpdirs:+ shutil.rmtree(d, ignore_errors=True)+ raise+++def _spawn_lambda_container_impl(config: dict, code_zip: bytes | None,+ _pin_platform: bool, _made_tmpdirs: list):"""Create and start a Lambda container for the given config.A function that explicitly declared an architecture is pinned to it — but@@ -3723,6 +3976,7 @@ def _spawn_lambda_container(config: dict, code_zip: bytes | None,if not code_zip:raise ValueError("Zip PackageType requires code_zip bytes")tmpdir = tempfile.mkdtemp(prefix="ministack-lambda-docker-")+ _made_tmpdirs.append(tmpdir)code_dir = os.path.join(tmpdir, "code")os.makedirs(code_dir)code_zip_path = os.path.join(tmpdir, "code.zip")@@ -5034,10 +5288,13 @@ def _publish_version(name: str, data: dict):if data.get("Description"):ver_config["Description"] = data["Description"]- func["versions"][str(ver_num)] = {+ ver_record = {"config": ver_config,"code_zip": func.get("code_zip"),}+ func["versions"][str(ver_num)] = ver_record+ if _stamp_snapstart_published_version(ver_config):+ _snapstart_provision_version_async(name, ver_record)return json_response(ver_config, 201)diff --git a/tests/test_lambda.py b/tests/test_lambda.pyindex f938a7de..30b338b9 100644--- a/tests/test_lambda.py+++ b/tests/test_lambda.py@@ -10878,3 +10878,417 @@ def test_lambda_platform_pinned_only_when_architectures_declared(monkeypatch):assert _lam._declared_docker_platform({"FunctionName": "pre-marker-record", "Architectures": ["x86_64"]}) is Noneassert _lam._declared_docker_platform({"FunctionName": "never-created"}) is None+++# ─────────────────────────────── SnapStart ───────────────────────────────+++def _snap_wait_active(lam, fname, qualifier=None):+ kw = {"FunctionName": fname}+ if qualifier:+ kw["Qualifier"] = qualifier+ for _ in range(40):+ cfg = lam.get_function_configuration(**kw)+ if cfg["State"] == "Active":+ return cfg+ time.sleep(0.25)+ raise AssertionError(f"{fname}:{qualifier} never became Active")+++def test_snapstart_defaults_off(lam):+ fname = f"snap-default-{_uuid_mod.uuid4().hex[:8]}"+ lam.create_function(+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},+ )+ try:+ cfg = lam.get_function_configuration(FunctionName=fname)+ assert cfg["SnapStart"] == {"ApplyOn": "None", "OptimizationStatus": "Off"}+ finally:+ lam.delete_function(FunctionName=fname)+++def test_snapstart_lifecycle_published_version_reports_on(lam):+ """AWS: $LATEST echoes ApplyOn with OptimizationStatus Off; publishing+ takes the snapshot, so the version reports On and transitions+ Pending → Active."""+ fname = f"snap-life-{_uuid_mod.uuid4().hex[:8]}"+ created = lam.create_function(+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},+ SnapStart={"ApplyOn": "PublishedVersions"},+ )+ try:+ assert created["SnapStart"] == {+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "Off",+ }+ _snap_wait_active(lam, fname)++ ver = lam.publish_version(FunctionName=fname)+ assert ver["SnapStart"] == {+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "On",+ }+ vcfg = _snap_wait_active(lam, fname, qualifier=ver["Version"])+ assert vcfg["SnapStart"]["OptimizationStatus"] == "On"++ # $LATEST never has a snapshot.+ latest = lam.get_function_configuration(FunctionName=fname)+ assert latest["SnapStart"] == {+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "Off",+ }+ finally:+ lam.delete_function(FunctionName=fname)+++def test_snapstart_create_with_publish_stamps_version_one(lam):+ fname = f"snap-pub1-{_uuid_mod.uuid4().hex[:8]}"+ lam.create_function(+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},+ SnapStart={"ApplyOn": "PublishedVersions"}, Publish=True,+ )+ try:+ vcfg = _snap_wait_active(lam, fname, qualifier="1")+ assert vcfg["SnapStart"] == {+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "On",+ }+ finally:+ lam.delete_function(FunctionName=fname)+++def test_snapstart_update_function_configuration_roundtrip(lam):+ fname = f"snap-upd-{_uuid_mod.uuid4().hex[:8]}"+ lam.create_function(+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},+ )+ try:+ _snap_wait_active(lam, fname)+ updated = lam.update_function_configuration(+ FunctionName=fname, SnapStart={"ApplyOn": "PublishedVersions"},+ )+ assert updated["SnapStart"] == {+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "Off",+ }+ _snap_wait_active(lam, fname)+ back = lam.update_function_configuration(+ FunctionName=fname, SnapStart={"ApplyOn": "None"},+ )+ assert back["SnapStart"] == {"ApplyOn": "None", "OptimizationStatus": "Off"}+ finally:+ lam.delete_function(FunctionName=fname)+++def test_snapstart_rejected_for_unsupported_runtime(lam):+ fname = f"snap-badrt-{_uuid_mod.uuid4().hex[:8]}"+ with pytest.raises(ClientError) as exc:+ lam.create_function(+ FunctionName=fname, Runtime="nodejs20.x", Role=_LAMBDA_ROLE,+ Handler="index.handler", Code={"ZipFile": _make_zip_js(_NODE_CODE)},+ SnapStart={"ApplyOn": "PublishedVersions"},+ )+ assert exc.value.response["Error"]["Code"] == "InvalidParameterValueException"++ # Enabling it later on an unsupported runtime is refused the same way.+ lam.create_function(+ FunctionName=fname, Runtime="nodejs20.x", Role=_LAMBDA_ROLE,+ Handler="index.handler", Code={"ZipFile": _make_zip_js(_NODE_CODE)},+ )+ try:+ _snap_wait_active(lam, fname)+ with pytest.raises(ClientError) as exc:+ lam.update_function_configuration(+ FunctionName=fname, SnapStart={"ApplyOn": "PublishedVersions"},+ )+ assert exc.value.response["Error"]["Code"] == "InvalidParameterValueException"+ finally:+ lam.delete_function(FunctionName=fname)+++def test_snapstart_rejected_with_large_ephemeral_storage(lam):+ fname = f"snap-eph-{_uuid_mod.uuid4().hex[:8]}"+ with pytest.raises(ClientError) as exc:+ lam.create_function(+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},+ SnapStart={"ApplyOn": "PublishedVersions"},+ EphemeralStorage={"Size": 1024},+ )+ assert exc.value.response["Error"]["Code"] == "InvalidParameterValueException"++ # Raising ephemeral storage past 512 MB while SnapStart is on is refused.+ lam.create_function(+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},+ SnapStart={"ApplyOn": "PublishedVersions"},+ )+ try:+ _snap_wait_active(lam, fname)+ with pytest.raises(ClientError) as exc:+ lam.update_function_configuration(+ FunctionName=fname, EphemeralStorage={"Size": 1024},+ )+ assert exc.value.response["Error"]["Code"] == "InvalidParameterValueException"+ finally:+ lam.delete_function(FunctionName=fname)+++# ──────────────── docker executor: no extraction-dir leak ────────────────+++_SPAWN_LEAK_CONFIG = {+ "FunctionName": "leak-fn", "Runtime": "python3.12",+ "Handler": "index.handler", "PackageType": "Zip", "Timeout": 3,+ "MemorySize": 128,+ "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:leak-fn",+}+++def _capture_mkdtemp(monkeypatch):+ import tempfile as _tf+ created = []+ real = _tf.mkdtemp++ def _capture(*a, **kw):+ d = real(*a, **kw)+ created.append(d)+ return d++ monkeypatch.setattr(_tf, "mkdtemp", _capture)+ return created+++def test_spawn_failure_on_corrupt_zip_leaves_no_tmpdir(monkeypatch):+ """A corrupt code zip raises out of the extraction block; the tmpdir made+ just before must not be orphaned (issue #1600)."""+ monkeypatch.setattr(lsvc, "_docker_available", True)+ monkeypatch.setattr(lsvc, "_get_docker_client", lambda: object())+ created = _capture_mkdtemp(monkeypatch)++ with pytest.raises(zipfile.BadZipFile):+ lsvc._spawn_lambda_container(dict(_SPAWN_LEAK_CONFIG), b"this is not a zip")++ assert created, "spawn never created an extraction dir"+ assert not any(os.path.exists(d) for d in created)+++def test_spawn_failure_on_docker_api_timeout_leaves_no_tmpdir(monkeypatch):+ """images.get raising anything other than ImageNotFound (the reported+ case: a socket read timeout) propagates after extraction; the tmpdir must+ be removed on the way out (issue #1600)."""+ monkeypatch.setattr(lsvc, "_docker_available", True)+ fake_client = MagicMock()+ fake_client.images.get.side_effect = TimeoutError("Read timed out.")+ monkeypatch.setattr(lsvc, "_get_docker_client", lambda: fake_client)+ created = _capture_mkdtemp(monkeypatch)++ with pytest.raises(TimeoutError):+ lsvc._spawn_lambda_container(+ dict(_SPAWN_LEAK_CONFIG), _make_zip("def handler(e, c): pass"),+ )++ assert created, "spawn never created an extraction dir"+ assert not any(os.path.exists(d) for d in created)+++def _make_zip_multi(files: dict) -> bytes:+ buf = io.BytesIO()+ with zipfile.ZipFile(buf, "w") as zf:+ for name, code in files.items():+ zf.writestr(name, code)+ return buf.getvalue()+++def _snap_wait_state(lam, fname, qualifier, want, attempts=60):+ for _ in range(attempts):+ cfg = lam.get_function_configuration(FunctionName=fname, Qualifier=qualifier)+ if cfg["State"] == want:+ return cfg+ time.sleep(0.25)+ raise AssertionError(f"{fname}:{qualifier} never reached {want} (last: {cfg['State']})")+++def test_snapstart_publish_initializes_and_pending_version_conflicts(lam):+ """SnapStart moves init to PublishVersion: while the environment spins up+ the version is Pending and Invoke answers ResourceConflictException; once+ Active the first invoke hits the pre-warmed environment."""+ fname = f"snap-pend-{_uuid_mod.uuid4().hex[:8]}"+ code = (+ "import time\n"+ "time.sleep(2)\n"+ "def handler(event, context):\n"+ " return {'ok': True}\n"+ )+ lam.create_function(+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,+ Handler="index.handler", Code={"ZipFile": _zip_lambda(code)},+ SnapStart={"ApplyOn": "PublishedVersions"},+ )+ try:+ _snap_wait_active(lam, fname)+ ver = lam.publish_version(FunctionName=fname)+ assert ver["State"] == "Pending"+ with pytest.raises(ClientError) as exc:+ lam.invoke(FunctionName=fname, Qualifier=ver["Version"], Payload=b"{}")+ assert exc.value.response["Error"]["Code"] == "ResourceConflictException"++ _snap_wait_state(lam, fname, ver["Version"], "Active")+ resp = lam.invoke(FunctionName=fname, Qualifier=ver["Version"], Payload=b"{}")+ assert json.loads(resp["Payload"].read()) == {"ok": True}+ finally:+ lam.delete_function(FunctionName=fname)+++def test_snapstart_publish_fails_version_on_broken_init(lam):+ """An init that raises fails the *publish*: the version lands State=Failed+ (AWS's failed-snapshot surface) instead of erroring at first invoke."""+ fname = f"snap-fail-{_uuid_mod.uuid4().hex[:8]}"+ lam.create_function(+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,+ Handler="index.handler",+ Code={"ZipFile": _zip_lambda("raise RuntimeError('boom-init')\n")},+ SnapStart={"ApplyOn": "PublishedVersions"},+ )+ try:+ _snap_wait_active(lam, fname)+ ver = lam.publish_version(FunctionName=fname)+ cfg = _snap_wait_state(lam, fname, ver["Version"], "Failed")+ assert cfg["StateReasonCode"] == "FunctionError"+ assert "boom-init" in cfg["StateReason"]+ finally:+ lam.delete_function(FunctionName=fname)+++_SNAP_HOOK_LIB = (+ "_before = []\n"+ "_after = []\n"+ "def register_before_snapshot(func, *args, **kwargs):\n"+ " _before.append((func, args, kwargs))\n"+ " return func\n"+ "def register_after_restore(func, *args, **kwargs):\n"+ " _after.append((func, args, kwargs))\n"+ " return func\n"+ "def get_before_snapshot():\n"+ " return _before\n"+ "def get_after_restore():\n"+ " return _after\n"+)++_SNAP_HOOK_HANDLER = (+ "import snapshot_restore_py as srp\n"+ "CALLS = []\n"+ "srp.register_before_snapshot(lambda: CALLS.append('before1'))\n"+ "srp.register_before_snapshot(lambda: CALLS.append('before2'))\n"+ "srp.register_after_restore(lambda: CALLS.append('after1'))\n"+ "srp.register_after_restore(lambda: CALLS.append('after2'))\n"+ "def handler(event, context):\n"+ " return {'calls': CALLS}\n"+)+++def test_snapstart_runtime_hooks_fire_on_published_version_only(lam):+ """snapshot-restore-py hooks run during a published version's init —+ before-snapshot hooks in reverse registration order, after-restore hooks+ in registration order — and never for $LATEST, which has no snapshot."""+ fname = f"snap-hook-{_uuid_mod.uuid4().hex[:8]}"+ lam.create_function(+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,+ Handler="index.handler",+ Code={"ZipFile": _make_zip_multi({+ "index.py": _SNAP_HOOK_HANDLER,+ "snapshot_restore_py.py": _SNAP_HOOK_LIB,+ })},+ SnapStart={"ApplyOn": "PublishedVersions"},+ )+ try:+ _snap_wait_active(lam, fname)+ ver = lam.publish_version(FunctionName=fname)+ _snap_wait_state(lam, fname, ver["Version"], "Active")++ resp = lam.invoke(FunctionName=fname, Qualifier=ver["Version"], Payload=b"{}")+ payload = json.loads(resp["Payload"].read())+ assert payload == {"calls": ["before2", "before1", "after1", "after2"]}++ latest = lam.invoke(FunctionName=fname, Payload=b"{}")+ assert json.loads(latest["Payload"].read()) == {"calls": []}+ finally:+ lam.delete_function(FunctionName=fname)+++def test_snapstart_version_worker_is_reapable():+ """A published SnapStart version's pre-warmed worker must not be pinned by+ the keep-the-first-worker rule, or every publish holds a subprocess+ forever; the $LATEST first worker keeps its exemption."""+ from ministack.core import lambda_runtime as lr++ def _mk(version, snap_status):+ cfg = {+ "FunctionName": "reap-probe", "Runtime": "python3.12",+ "Version": version,+ "SnapStart": {"ApplyOn": "PublishedVersions",+ "OptimizationStatus": snap_status},+ "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:reap-probe",+ }+ w = lr.Worker("reap-probe", cfg, b"")+ w.in_use = False+ w.last_used = time.time() - 10_000+ return w++ latest = _mk("$LATEST", "Off")+ snap_ver = _mk("1", "On")+ with lr._lock:+ lr._workers["000000000000:us-east-1:reap-probe:$LATEST"] = [latest]+ lr._workers["000000000000:us-east-1:reap-probe:1"] = [snap_ver]+ try:+ lr.reap_idle_workers(ttl=1)+ with lr._lock:+ assert lr._workers.get("000000000000:us-east-1:reap-probe:$LATEST") == [latest]+ assert "000000000000:us-east-1:reap-probe:1" not in lr._workers+ finally:+ with lr._lock:+ lr._workers.pop("000000000000:us-east-1:reap-probe:$LATEST", None)+ lr._workers.pop("000000000000:us-east-1:reap-probe:1", None)+++def test_snapstart_pending_version_reprovisions_on_restore():+ """A SnapStart version persisted while Pending (crash mid-publish) must be+ re-provisioned by restore_state — otherwise it answers 409 forever."""+ import copy as _copy++ arn = "arn:aws:lambda:us-east-1:000000000000:function:snap-restore-fn"+ ver_cfg = {+ "FunctionName": "snap-restore-fn", "FunctionArn": f"{arn}:1",+ "Runtime": "python3.12", "Handler": "index.handler",+ "MemorySize": 128, "Timeout": 3, "Version": "1",+ "State": "Pending", "StateReason": "The function is being created.",+ "StateReasonCode": "Creating",+ "SnapStart": {"ApplyOn": "PublishedVersions", "OptimizationStatus": "On"},+ "PackageType": "Zip",+ }+ func = {+ "config": {**_copy.deepcopy(ver_cfg), "FunctionArn": arn,+ "Version": "$LATEST", "State": "Active",+ "SnapStart": {"ApplyOn": "PublishedVersions",+ "OptimizationStatus": "Off"}},+ "code_zip": _make_zip("def handler(e, c):\n return {}\n"),+ "versions": {"1": {"config": ver_cfg,+ "code_zip": _make_zip("def handler(e, c):\n return {}\n")}},+ "next_version": 2, "tags": {}, "aliases": {},+ "policy": {"Version": "2012-10-17", "Id": "default", "Statement": []},+ "event_invoke_config": None, "event_invoke_configs": {},+ "concurrency": None, "provisioned_concurrency": {},+ }+ key = ("000000000000", "us-east-1", "snap-restore-fn")+ try:+ lsvc.restore_state({"functions": {"snap-restore-fn": func}})+ for _ in range(60):+ state = ver_cfg["State"]+ if state != "Pending":+ break+ time.sleep(0.25)+ assert ver_cfg["State"] == "Active", f"stuck in {ver_cfg['State']}"+ finally:+ lsvc._functions._data.pop(key, None)+ from ministack.core import lambda_runtime as _lr+ _lr.invalidate_worker("snap-restore-fn", account="000000000000",+ region="us-east-1")