Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

feat(lambda): add snapstart (#1603)

Nahuel Nucerab5a2e2528d15
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9d6f817b..a8ecacd5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -19,6 +19,9 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- **Amazon Transcribe (`transcribe`)** is a new service emulator covering batch transcription jobs: `StartTranscriptionJob`, `GetTranscriptionJob`, `ListTranscriptionJobs` and `DeleteTranscriptionJob`, verified against botocore `transcribe-2017-10-26`. A job walks `QUEUED` → `IN_PROGRESS` → `COMPLETED` on a background task paced by `TRANSCRIBE_JOB_RUN_SECONDS` (default 2, `0` completes immediately), reads its media object from MiniStack's S3 store through an `s3://`, path-style or virtual-hosted HTTPS URI, and writes a transcript document back to S3 in the real Transcribe result format: `results.transcripts`, word-level `results.items`, `results.audio_segments`, and `results.segments` / `results.speaker_labels` when `ShowAlternatives` and `ShowSpeakerLabels` are set. `IdentifyLanguage` fills in `LanguageCode` and `IdentifiedLanguageScore` on completion while `IdentifyMultipleLanguages` reports `LanguageCodes` and leaves `LanguageCode` unset, as AWS does. `ContentRedaction` decides whether the `redacted-` copy replaces the transcript or sits beside it, and `Subtitles` writes WebVTT and SubRip files reported as `SubtitleFileUris`, redacted and renamed to match when redaction is on. Output lands at `OutputBucketName` / `OutputKey`, or in an auto-created `ministack-transcribe-output` bucket when neither is given, so the reported URIs are always fetchable. Media that cannot be read fails the job asynchronously with a `FailureReason` rather than erroring at `Start`, matching AWS, and a job left mid-flight by a restart fails on restore instead of stranding pollers. Reaching a terminal state publishes a `Transcribe Job State Change` event on the default bus, so an S3 → Lambda → Transcribe → EventBridge pipeline runs end to end locally. There is **no speech recognition**: the transcript is deterministic canned text keyed on a digest of the media URI, the same approach Bedrock Runtime takes for model output, which keeps integration tests reproducible. Streaming transcription, the call-analytics and medical job families, custom vocabularies and the tagging operations are not implemented. State is account- and region-scoped and persisted. Contributed by @ppettitau.
+### Fixed
+- **Lambda — a failed Docker cold start no longer leaks its extraction directory** — the executor unpacks the function zip and its layers into a `ministack-lambda-docker-*` temp directory before talking to Docker, but only three specific failure paths cleaned it up; anything else raised between the extraction and the container start — a corrupt layer zip, or a Docker socket read-timeout out of the image lookup (which is not `ImageNotFound`) — orphaned the directory. Under Docker-API pressure that compounds: each timed-out spawn leaves a full unpacked tree behind, measured at gigabytes of `/tmp` in minutes on a large stack. Cleanup now covers every exit: the directory is owned by the spawn until the container is handed to the warm pool, and removed on any failure in between. Reported by @iot-rocket.
+
## [1.5.5] — 2026-09-01
### Added
diff --git a/ministack/core/lambda_runtime.py b/ministack/core/lambda_runtime.py
index ff91e0ba..113e16e7 100644
--- a/ministack/core/lambda_runtime.py
+++ b/ministack/core/lambda_runtime.py
@@ -183,6 +183,25 @@ def run():
try:
mod = importlib.import_module(module_name)
handler_fn = getattr(mod, handler_name)
+ if init.get("snapstart"):
+ # SnapStart runtime hooks (snapshot-restore-py, bundled in AWS's
+ # python3.12+ runtimes). Each environment of a published SnapStart
+ # version runs one snapshot/restore cycle: before-snapshot hooks
+ # in reverse registration order, then after-restore hooks in
+ # registration order, before the first invocation. A hook that
+ # raises fails the init, which fails the publish, as on AWS.
+ try:
+ from snapshot_restore_py import (
+ get_after_restore,
+ get_before_snapshot,
+ )
+ except ImportError:
+ pass
+ else:
+ for _fn, _args, _kwargs in reversed(get_before_snapshot()):
+ _fn(*_args, **_kwargs)
+ for _fn, _args, _kwargs in get_after_restore():
+ _fn(*_args, **_kwargs)
_real_stdout.write(json.dumps({"status": "ready", "cold": True}) + "\\n")
_real_stdout.flush()
except Exception as e:
@@ -1132,6 +1151,11 @@ class Worker:
"function_name": self.config.get("FunctionName", ""),
"memory": self.config.get("MemorySize", 128),
"arn": self.config.get("FunctionArn", ""),
+ # A published SnapStart version runs its snapshot/restore runtime
+ # hooks during init (see the worker script). $LATEST never does —
+ # it has no snapshot on AWS either.
+ "snapstart": (self.config.get("SnapStart") or {}).get(
+ "OptimizationStatus") == "On",
}
self._proc.stdin.write(json.dumps(init) + "\n")
self._proc.stdin.flush()
@@ -1426,7 +1450,18 @@ def reap_idle_workers(ttl: float = None) -> int:
keep = []
for i, worker in enumerate(entries):
idle = now - getattr(worker, "last_used", now)
- if i == 0 or worker.in_use or idle < ttl:
+ # The first worker per key is kept warm indefinitely — except
+ # for a published SnapStart version's worker, which exists
+ # because PublishVersion pre-initialized it: without the
+ # exception every publish would pin one subprocess forever.
+ # Reaping it costs a re-init on the next invoke, the same as
+ # the docker pool's TTL eviction.
+ snapstart_version = (
+ (worker.config.get("SnapStart") or {}).get(
+ "OptimizationStatus") == "On"
+ and worker.config.get("Version", "$LATEST") != "$LATEST"
+ )
+ if worker.in_use or idle < ttl or (i == 0 and not snapstart_version):
keep.append(worker)
else:
killed.append(worker)
@@ -1438,6 +1473,20 @@ def reap_idle_workers(ttl: float = None) -> int:
return len(killed)
+def ensure_spawned(worker: Worker) -> None:
+ """Spawn the worker's subprocess now if it isn't running.
+
+ SnapStart moves initialization to PublishVersion: the publish path leases
+ a worker for the new version and forces the cold start here, so an init
+ failure fails the publish (version State=Failed, as on AWS) and the first
+ invoke finds the environment already warm.
+ """
+ with worker._lock:
+ if worker._proc is None or worker._proc.poll() is not None:
+ worker._spawn()
+ worker._cold = False
+
+
def release_worker(worker: Worker) -> None:
"""Return a leased worker to its pool."""
if worker is None:
diff --git a/ministack/services/cloudformation/provisioners.py b/ministack/services/cloudformation/provisioners.py
index 32a774ba..b796856d 100644
--- a/ministack/services/cloudformation/provisioners.py
+++ b/ministack/services/cloudformation/provisioners.py
@@ -1084,6 +1084,7 @@ def _lambda_create(logical_id, props, stack_name):
"EphemeralStorage": {"Size": props.get("EphemeralStorage", {}).get("Size", 512)},
"TracingConfig": props.get("TracingConfig", {"Mode": "PassThrough"}),
"LoggingConfig": props.get("LoggingConfig", {"LogFormat": "Text", "LogGroup": f"/aws/lambda/{name}"}),
+ "SnapStart": _lambda_svc._snapstart_response(props.get("SnapStart")),
"RevisionId": new_uuid(),
},
"code_zip": code_zip,
@@ -1193,6 +1194,7 @@ def _lambda_update(physical_id, old_props, new_props, stack_name, logical_id=Non
"LoggingConfig", {"LogFormat": "Text", "LogGroup": f"/aws/lambda/{name}"}
),
"Architectures": new_props.get("Architectures", ["x86_64"]),
+ "SnapStart": new_props.get("SnapStart", {"ApplyOn": "None"}),
}
if new_props.get("ImageConfig"):
config_data["ImageConfig"] = new_props["ImageConfig"]
diff --git a/ministack/services/lambda_svc.py b/ministack/services/lambda_svc.py
index 62386c24..160d7e99 100644
--- a/ministack/services/lambda_svc.py
+++ b/ministack/services/lambda_svc.py
@@ -56,6 +56,7 @@ from ministack.core.lambda_runtime import (
INVOKE_DEPTH_EVENT_KEY,
INVOKE_DEPTH_HEADER,
acquire_worker,
+ ensure_spawned,
invalidate_worker,
reap_idle_workers,
release_worker,
@@ -563,6 +564,20 @@ def restore_state(data):
_function_urls.update(data.get("function_urls", {}))
_restore_esm_positions(_kinesis_positions, data.get("kinesis_positions", {}))
_restore_esm_positions(_dynamodb_stream_positions, data.get("dynamodb_stream_positions", {}))
+ # A SnapStart version persisted mid-publish restores as State=Pending
+ # with no provisioning thread behind it — and Pending SnapStart
+ # versions answer 409 on Invoke and are skipped by the generic state
+ # flipper, so without re-provisioning here the version would be
+ # uninvokable forever. Re-run the publish-time initialization.
+ for scoped_key, func in list(_functions._data.items()):
+ fn_name = scoped_key[-1]
+ for ver_record in (func.get("versions") or {}).values():
+ cfg = ver_record.get("config") or {}
+ if (
+ cfg.get("State") == "Pending"
+ and (cfg.get("SnapStart") or {}).get("OptimizationStatus") == "On"
+ ):
+ _snapstart_provision_version_async(fn_name, ver_record)
if _esms.has_any():
_ensure_poller()
@@ -1285,6 +1300,160 @@ def _durable_arn_lookup(name_or_arn: str) -> str | None:
return func.get("FunctionArn") or _func_arn(canonical)
+# ---------------------------------------------------------------------------
+# SnapStart
+# ---------------------------------------------------------------------------
+# AWS accepts ``SnapStart.ApplyOn`` on CreateFunction and
+# UpdateFunctionConfiguration and reports ``{ApplyOn, OptimizationStatus}`` on
+# every read. $LATEST always reports OptimizationStatus=Off — a snapshot only
+# exists per published version — while a version published with
+# ApplyOn=PublishedVersions reports On, sits in State=Pending while the
+# snapshot is created, then transitions to Active. Supported runtimes per the
+# SnapStart docs: Java 11+, Python 3.12+, .NET 8+, for both Zip and
+# container-image packages; incompatible with ephemeral storage above 512 MB.
+
+_SNAPSTART_APPLY_ON_VALUES = ("PublishedVersions", "None")
+
+
+def _snapstart_runtime_supported(runtime: str) -> bool:
+ m = re.match(r"^(java|python|dotnet)(\d+)(?:\.(\d+))?", runtime or "")
+ if not m:
+ return False
+ family, major, minor = m.group(1), int(m.group(2)), int(m.group(3) or 0)
+ if family == "java":
+ return major >= 11
+ if family == "python":
+ return (major, minor) >= (3, 12)
+ return major >= 8 # dotnet
+
+
+def _validate_snapstart(snap: dict | None, *, runtime: str, package_type: str,
+ ephemeral_size) -> tuple | None:
+ """Validate a SnapStart request block against AWS's documented constraints."""
+ if snap is None:
+ return None
+ apply_on = (snap or {}).get("ApplyOn", "None")
+ if apply_on not in _SNAPSTART_APPLY_ON_VALUES:
+ return error_response_json(
+ "ValidationException",
+ f"1 validation error detected: Value '{apply_on}' at "
+ "'snapStart.applyOn' failed to satisfy constraint: Member must "
+ "satisfy enum value set: [PublishedVersions, None]",
+ 400,
+ )
+ if apply_on == "None":
+ return None
+ if package_type != "Image" and not _snapstart_runtime_supported(runtime):
+ return error_response_json(
+ "InvalidParameterValueException",
+ f"SnapStart is not supported for the {runtime} runtime.",
+ 400,
+ )
+ try:
+ too_big = ephemeral_size is not None and int(ephemeral_size) > 512
+ except (TypeError, ValueError):
+ too_big = False
+ if too_big:
+ return error_response_json(
+ "InvalidParameterValueException",
+ "SnapStart is not supported with ephemeral storage greater than "
+ "512 MB.",
+ 400,
+ )
+ return None
+
+
+def _snapstart_response(snap: dict | None) -> dict:
+ apply_on = (snap or {}).get("ApplyOn") or "None"
+ return {"ApplyOn": apply_on, "OptimizationStatus": "Off"}
+
+
+def _stamp_snapstart_published_version(ver_config: dict) -> bool:
+ """AWS creates the snapshot when a version is published: the version
+ reports OptimizationStatus=On and stays Pending until the snapshot is
+ ready. Returns True when the version is SnapStart-enabled so the caller
+ schedules the Pending→Active transition."""
+ if (ver_config.get("SnapStart") or {}).get("ApplyOn") != "PublishedVersions":
+ return False
+ ver_config["SnapStart"] = {
+ "ApplyOn": "PublishedVersions",
+ "OptimizationStatus": "On",
+ }
+ ver_config["State"] = "Pending"
+ ver_config["StateReason"] = "The function is being created."
+ ver_config["StateReasonCode"] = "Creating"
+ return True
+
+
+def _snapstart_provision_version_async(name: str, ver_record: dict) -> None:
+ """SnapStart moves initialization to PublishVersion: create the version's
+ execution environment now — a warm worker for python, a pooled RIE
+ container for java/dotnet/Image — instead of at first invoke.
+
+ Success flips the version Pending → Active with the environment already
+ warm (the snapshot's observable effect); an init failure — a handler that
+ won't import, a before-snapshot/after-restore hook that raises — lands the
+ version in State=Failed, as AWS surfaces a failed snapshot. This thread
+ owns the version's state transition; the generic Pending flipper skips
+ SnapStart versions.
+ """
+ ver_config = ver_record["config"]
+ code_zip = ver_record.get("code_zip")
+
+ def _provision():
+ try:
+ account_id, region = _account_region_from_function_config(ver_config)
+ _request_account_id.set(account_id)
+ _request_region.set(region)
+ except ArnParseError:
+ pass
+ try:
+ runtime = ver_config.get("Runtime", "")
+ package_type = ver_config.get("PackageType", "Zip")
+ use_docker = (
+ LAMBDA_STRICT
+ or package_type == "Image"
+ or LAMBDA_EXECUTOR == "docker"
+ or not runtime.startswith("python")
+ )
+ if use_docker:
+ if _docker_available and _get_docker_client() is not None:
+ container, tmpdir = _spawn_lambda_container(ver_config, code_zip)
+ key = _warm_pool_key(ver_config["FunctionName"], ver_config)
+ entry = _pool_register(key, container, tmpdir)
+ _pool_release(entry)
+ _ensure_reaper_thread()
+ # Docker unreachable: nothing to pre-warm; the invoke path
+ # falls back exactly as it always did.
+ elif code_zip:
+ worker, _reason = acquire_worker(
+ ver_config["FunctionName"], ver_config, code_zip,
+ qualifier=ver_config.get("Version", "$LATEST"),
+ )
+ try:
+ ensure_spawned(worker)
+ finally:
+ release_worker(worker)
+ _ensure_reaper_thread()
+ except Exception as exc:
+ logger.warning("Lambda %s: SnapStart init failed at publish: %s",
+ name, exc)
+ ver_config["State"] = "Failed"
+ ver_config["StateReason"] = str(exc)
+ ver_config["StateReasonCode"] = "FunctionError"
+ return
+ ver_config["State"] = "Active"
+ ver_config["StateReason"] = ""
+ ver_config["StateReasonCode"] = ""
+ ver_config["LastUpdateStatus"] = "Successful"
+
+ ctx_snapshot = contextvars.copy_context()
+ threading.Thread(
+ target=ctx_snapshot.run, args=(_provision,), daemon=True,
+ name="ministack-lambda-snapstart-init",
+ ).start()
+
+
def _build_config(name: str, data: dict, code_zip: bytes | None = None) -> dict:
code_size = len(code_zip) if code_zip else 0
code_sha = base64.b64encode(hashlib.sha256(code_zip).digest()).decode() if code_zip else ""
@@ -1343,7 +1512,7 @@ def _build_config(name: str, data: dict, code_zip: bytes | None = None) -> dict:
"KMSKeyArn": data.get("KMSKeyArn", ""),
"RevisionId": new_uuid(),
"EphemeralStorage": data.get("EphemeralStorage", {"Size": 512}),
- "SnapStart": {"ApplyOn": "None", "OptimizationStatus": "Off"},
+ "SnapStart": _snapstart_response(data.get("SnapStart")),
"LoggingConfig": data.get(
"LoggingConfig",
{
@@ -1967,6 +2136,15 @@ def _create_function(data: dict):
data = dict(data)
data["Layers"] = layers_cfg
+ err = _validate_snapstart(
+ data.get("SnapStart"),
+ runtime=data.get("Runtime", ""),
+ package_type=data.get("PackageType", "Zip"),
+ ephemeral_size=(data.get("EphemeralStorage") or {}).get("Size", 512),
+ )
+ if err:
+ return err
+
# Validate the execution role before building anything: _build_config
# returns a config dict, so the error has to be raised by the caller, and
# nothing may be persisted for a function that is going to be refused.
@@ -2020,10 +2198,13 @@ def _create_function(data: dict):
_functions[name]["next_version"] = ver_num + 1
ver_config = copy.deepcopy(config)
ver_config["Version"] = str(ver_num)
- _functions[name]["versions"][str(ver_num)] = {
+ ver_record = {
"config": ver_config,
"code_zip": code_zip,
}
+ _functions[name]["versions"][str(ver_num)] = ver_record
+ if _stamp_snapstart_published_version(ver_config):
+ _snapstart_provision_version_async(name, ver_record)
config["Version"] = str(ver_num)
_schedule_state_transition(name, _LAMBDA_STATE_TRANSITION_DELAY)
@@ -2346,6 +2527,10 @@ def _delete_function(name: str, query_params: dict, path_qualifier: str | None =
409,
)
_functions[name]["versions"].pop(qualifier, None)
+ # The version's warm worker (SnapStart pre-initialized it at publish,
+ # or a past invoke spawned it) has nothing left to serve.
+ invalidate_worker(name, qualifier=qualifier,
+ account=get_account_id(), region=get_region())
else:
del _functions[name]
invalidate_worker(name, account=get_account_id(), region=get_region())
@@ -2430,10 +2615,13 @@ def _update_code(name: str, data: dict):
func["next_version"] = ver_num + 1
ver_config = copy.deepcopy(func["config"])
ver_config["Version"] = str(ver_num)
- func["versions"][str(ver_num)] = {
+ ver_record = {
"config": ver_config,
"code_zip": func.get("code_zip"),
}
+ func["versions"][str(ver_num)] = ver_record
+ if _stamp_snapstart_published_version(ver_config):
+ _snapstart_provision_version_async(name, ver_record)
func["config"]["Version"] = str(ver_num)
return json_response(func["config"])
@@ -2456,6 +2644,21 @@ def _update_config(name: str, data: dict):
data = dict(data)
data["Layers"] = layers_cfg
config = _functions[name]["config"]
+ if "SnapStart" in data or "Runtime" in data or "EphemeralStorage" in data:
+ # Validate the *effective* combination, so enabling SnapStart, moving
+ # to an unsupported runtime, or raising ephemeral storage past 512 MB
+ # while SnapStart is on are all refused before any field mutates.
+ effective_snap = data.get("SnapStart", config.get("SnapStart"))
+ err = _validate_snapstart(
+ effective_snap,
+ runtime=data.get("Runtime") or config.get("Runtime", ""),
+ package_type=config.get("PackageType", "Zip"),
+ ephemeral_size=(data.get("EphemeralStorage")
+ or config.get("EphemeralStorage")
+ or {}).get("Size", 512),
+ )
+ if err:
+ return err
for key in (
"Runtime",
"Handler",
@@ -2476,6 +2679,7 @@ def _update_config(name: str, data: dict):
"DurableConfig",
"TenancyConfig",
"CapacityProviderConfig",
+ "SnapStart",
):
if key in data:
if key == "Layers":
@@ -2489,6 +2693,10 @@ def _update_config(name: str, data: dict):
layer["CodeSize"] = _layer_codesize_for_arn(layer["Arn"])
layers_cfg.append(layer)
config["Layers"] = layers_cfg
+ elif key == "SnapStart":
+ # Request carries only ApplyOn; the stored/echoed shape adds
+ # OptimizationStatus, which is always Off on $LATEST.
+ config["SnapStart"] = _snapstart_response(data["SnapStart"])
else:
config[key] = data[key]
if "Architectures" in data:
@@ -2564,6 +2772,21 @@ async def _invoke(name: str, event: dict, headers: dict, path_qualifier: str | N
404,
)
+ exec_config = exec_record.get("config") or {}
+ if (
+ executed_version != "$LATEST"
+ and (exec_config.get("SnapStart") or {}).get("OptimizationStatus") == "On"
+ and exec_config.get("State") == "Pending"
+ ):
+ # AWS: invoking a SnapStart version while its snapshot is being
+ # created answers ResourceConflictException until it goes Active.
+ return error_response_json(
+ "ResourceConflictException",
+ "The operation cannot be performed at this time. "
+ "The function is currently in the following state: Pending",
+ 409,
+ )
+
if invocation_type == "DryRun":
return 204, {"X-Amz-Executed-Version": executed_version}, b""
@@ -3094,6 +3317,13 @@ def _schedule_state_transition(func_name: str, delay: float) -> None:
and cfg.get("LastUpdateStatus") not in (None, "", "InProgress")
):
continue
+ # A pending SnapStart version is mid-initialization: its
+ # provisioning thread owns the Pending → Active/Failed decision.
+ if (
+ cfg.get("Version") not in (None, "", "$LATEST")
+ and (cfg.get("SnapStart") or {}).get("OptimizationStatus") == "On"
+ ):
+ continue
cfg["State"] = "Active"
cfg["StateReason"] = ""
cfg["StateReasonCode"] = ""
@@ -3675,6 +3905,29 @@ def _declared_docker_platform(config: dict):
def _spawn_lambda_container(config: dict, code_zip: bytes | None,
_pin_platform: bool = True):
+ """Create and start a Lambda container, never leaking the extraction dir.
+
+ Everything between the extraction tmpdir's creation and the pool
+ registration can raise — a corrupt code or layer zip, or any Docker API
+ failure that is not one of the specifically-handled cases (e.g. a socket
+ read timeout out of ``images.get``, which is not ``ImageNotFound``).
+ Ownership of the tmpdir passes to the caller only on a successful return;
+ on any exception it is removed here instead of orphaning one extraction
+ per failed cold start (issue #1600).
+ """
+ made_tmpdirs: list[str] = []
+ try:
+ return _spawn_lambda_container_impl(config, code_zip, _pin_platform,
+ made_tmpdirs)
+ except BaseException:
+ import shutil
+ for d in made_tmpdirs:
+ shutil.rmtree(d, ignore_errors=True)
+ raise
+
+
+def _spawn_lambda_container_impl(config: dict, code_zip: bytes | None,
+ _pin_platform: bool, _made_tmpdirs: list):
"""Create and start a Lambda container for the given config.
A function that explicitly declared an architecture is pinned to it — but
@@ -3723,6 +3976,7 @@ def _spawn_lambda_container(config: dict, code_zip: bytes | None,
if not code_zip:
raise ValueError("Zip PackageType requires code_zip bytes")
tmpdir = tempfile.mkdtemp(prefix="ministack-lambda-docker-")
+ _made_tmpdirs.append(tmpdir)
code_dir = os.path.join(tmpdir, "code")
os.makedirs(code_dir)
code_zip_path = os.path.join(tmpdir, "code.zip")
@@ -5034,10 +5288,13 @@ def _publish_version(name: str, data: dict):
if data.get("Description"):
ver_config["Description"] = data["Description"]
- func["versions"][str(ver_num)] = {
+ ver_record = {
"config": ver_config,
"code_zip": func.get("code_zip"),
}
+ func["versions"][str(ver_num)] = ver_record
+ if _stamp_snapstart_published_version(ver_config):
+ _snapstart_provision_version_async(name, ver_record)
return json_response(ver_config, 201)
diff --git a/tests/test_lambda.py b/tests/test_lambda.py
index f938a7de..30b338b9 100644
--- a/tests/test_lambda.py
+++ b/tests/test_lambda.py
@@ -10878,3 +10878,417 @@ def test_lambda_platform_pinned_only_when_architectures_declared(monkeypatch):
assert _lam._declared_docker_platform(
{"FunctionName": "pre-marker-record", "Architectures": ["x86_64"]}) is None
assert _lam._declared_docker_platform({"FunctionName": "never-created"}) is None
+
+
+# ─────────────────────────────── SnapStart ───────────────────────────────
+
+
+def _snap_wait_active(lam, fname, qualifier=None):
+ kw = {"FunctionName": fname}
+ if qualifier:
+ kw["Qualifier"] = qualifier
+ for _ in range(40):
+ cfg = lam.get_function_configuration(**kw)
+ if cfg["State"] == "Active":
+ return cfg
+ time.sleep(0.25)
+ raise AssertionError(f"{fname}:{qualifier} never became Active")
+
+
+def test_snapstart_defaults_off(lam):
+ fname = f"snap-default-{_uuid_mod.uuid4().hex[:8]}"
+ lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},
+ )
+ try:
+ cfg = lam.get_function_configuration(FunctionName=fname)
+ assert cfg["SnapStart"] == {"ApplyOn": "None", "OptimizationStatus": "Off"}
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+def test_snapstart_lifecycle_published_version_reports_on(lam):
+ """AWS: $LATEST echoes ApplyOn with OptimizationStatus Off; publishing
+ takes the snapshot, so the version reports On and transitions
+ Pending → Active."""
+ fname = f"snap-life-{_uuid_mod.uuid4().hex[:8]}"
+ created = lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},
+ SnapStart={"ApplyOn": "PublishedVersions"},
+ )
+ try:
+ assert created["SnapStart"] == {
+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "Off",
+ }
+ _snap_wait_active(lam, fname)
+
+ ver = lam.publish_version(FunctionName=fname)
+ assert ver["SnapStart"] == {
+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "On",
+ }
+ vcfg = _snap_wait_active(lam, fname, qualifier=ver["Version"])
+ assert vcfg["SnapStart"]["OptimizationStatus"] == "On"
+
+ # $LATEST never has a snapshot.
+ latest = lam.get_function_configuration(FunctionName=fname)
+ assert latest["SnapStart"] == {
+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "Off",
+ }
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+def test_snapstart_create_with_publish_stamps_version_one(lam):
+ fname = f"snap-pub1-{_uuid_mod.uuid4().hex[:8]}"
+ lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},
+ SnapStart={"ApplyOn": "PublishedVersions"}, Publish=True,
+ )
+ try:
+ vcfg = _snap_wait_active(lam, fname, qualifier="1")
+ assert vcfg["SnapStart"] == {
+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "On",
+ }
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+def test_snapstart_update_function_configuration_roundtrip(lam):
+ fname = f"snap-upd-{_uuid_mod.uuid4().hex[:8]}"
+ lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},
+ )
+ try:
+ _snap_wait_active(lam, fname)
+ updated = lam.update_function_configuration(
+ FunctionName=fname, SnapStart={"ApplyOn": "PublishedVersions"},
+ )
+ assert updated["SnapStart"] == {
+ "ApplyOn": "PublishedVersions", "OptimizationStatus": "Off",
+ }
+ _snap_wait_active(lam, fname)
+ back = lam.update_function_configuration(
+ FunctionName=fname, SnapStart={"ApplyOn": "None"},
+ )
+ assert back["SnapStart"] == {"ApplyOn": "None", "OptimizationStatus": "Off"}
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+def test_snapstart_rejected_for_unsupported_runtime(lam):
+ fname = f"snap-badrt-{_uuid_mod.uuid4().hex[:8]}"
+ with pytest.raises(ClientError) as exc:
+ lam.create_function(
+ FunctionName=fname, Runtime="nodejs20.x", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _make_zip_js(_NODE_CODE)},
+ SnapStart={"ApplyOn": "PublishedVersions"},
+ )
+ assert exc.value.response["Error"]["Code"] == "InvalidParameterValueException"
+
+ # Enabling it later on an unsupported runtime is refused the same way.
+ lam.create_function(
+ FunctionName=fname, Runtime="nodejs20.x", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _make_zip_js(_NODE_CODE)},
+ )
+ try:
+ _snap_wait_active(lam, fname)
+ with pytest.raises(ClientError) as exc:
+ lam.update_function_configuration(
+ FunctionName=fname, SnapStart={"ApplyOn": "PublishedVersions"},
+ )
+ assert exc.value.response["Error"]["Code"] == "InvalidParameterValueException"
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+def test_snapstart_rejected_with_large_ephemeral_storage(lam):
+ fname = f"snap-eph-{_uuid_mod.uuid4().hex[:8]}"
+ with pytest.raises(ClientError) as exc:
+ lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},
+ SnapStart={"ApplyOn": "PublishedVersions"},
+ EphemeralStorage={"Size": 1024},
+ )
+ assert exc.value.response["Error"]["Code"] == "InvalidParameterValueException"
+
+ # Raising ephemeral storage past 512 MB while SnapStart is on is refused.
+ lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _zip_lambda(_LAMBDA_CODE)},
+ SnapStart={"ApplyOn": "PublishedVersions"},
+ )
+ try:
+ _snap_wait_active(lam, fname)
+ with pytest.raises(ClientError) as exc:
+ lam.update_function_configuration(
+ FunctionName=fname, EphemeralStorage={"Size": 1024},
+ )
+ assert exc.value.response["Error"]["Code"] == "InvalidParameterValueException"
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+# ──────────────── docker executor: no extraction-dir leak ────────────────
+
+
+_SPAWN_LEAK_CONFIG = {
+ "FunctionName": "leak-fn", "Runtime": "python3.12",
+ "Handler": "index.handler", "PackageType": "Zip", "Timeout": 3,
+ "MemorySize": 128,
+ "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:leak-fn",
+}
+
+
+def _capture_mkdtemp(monkeypatch):
+ import tempfile as _tf
+ created = []
+ real = _tf.mkdtemp
+
+ def _capture(*a, **kw):
+ d = real(*a, **kw)
+ created.append(d)
+ return d
+
+ monkeypatch.setattr(_tf, "mkdtemp", _capture)
+ return created
+
+
+def test_spawn_failure_on_corrupt_zip_leaves_no_tmpdir(monkeypatch):
+ """A corrupt code zip raises out of the extraction block; the tmpdir made
+ just before must not be orphaned (issue #1600)."""
+ monkeypatch.setattr(lsvc, "_docker_available", True)
+ monkeypatch.setattr(lsvc, "_get_docker_client", lambda: object())
+ created = _capture_mkdtemp(monkeypatch)
+
+ with pytest.raises(zipfile.BadZipFile):
+ lsvc._spawn_lambda_container(dict(_SPAWN_LEAK_CONFIG), b"this is not a zip")
+
+ assert created, "spawn never created an extraction dir"
+ assert not any(os.path.exists(d) for d in created)
+
+
+def test_spawn_failure_on_docker_api_timeout_leaves_no_tmpdir(monkeypatch):
+ """images.get raising anything other than ImageNotFound (the reported
+ case: a socket read timeout) propagates after extraction; the tmpdir must
+ be removed on the way out (issue #1600)."""
+ monkeypatch.setattr(lsvc, "_docker_available", True)
+ fake_client = MagicMock()
+ fake_client.images.get.side_effect = TimeoutError("Read timed out.")
+ monkeypatch.setattr(lsvc, "_get_docker_client", lambda: fake_client)
+ created = _capture_mkdtemp(monkeypatch)
+
+ with pytest.raises(TimeoutError):
+ lsvc._spawn_lambda_container(
+ dict(_SPAWN_LEAK_CONFIG), _make_zip("def handler(e, c): pass"),
+ )
+
+ assert created, "spawn never created an extraction dir"
+ assert not any(os.path.exists(d) for d in created)
+
+
+def _make_zip_multi(files: dict) -> bytes:
+ buf = io.BytesIO()
+ with zipfile.ZipFile(buf, "w") as zf:
+ for name, code in files.items():
+ zf.writestr(name, code)
+ return buf.getvalue()
+
+
+def _snap_wait_state(lam, fname, qualifier, want, attempts=60):
+ for _ in range(attempts):
+ cfg = lam.get_function_configuration(FunctionName=fname, Qualifier=qualifier)
+ if cfg["State"] == want:
+ return cfg
+ time.sleep(0.25)
+ raise AssertionError(f"{fname}:{qualifier} never reached {want} (last: {cfg['State']})")
+
+
+def test_snapstart_publish_initializes_and_pending_version_conflicts(lam):
+ """SnapStart moves init to PublishVersion: while the environment spins up
+ the version is Pending and Invoke answers ResourceConflictException; once
+ Active the first invoke hits the pre-warmed environment."""
+ fname = f"snap-pend-{_uuid_mod.uuid4().hex[:8]}"
+ code = (
+ "import time\n"
+ "time.sleep(2)\n"
+ "def handler(event, context):\n"
+ " return {'ok': True}\n"
+ )
+ lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler", Code={"ZipFile": _zip_lambda(code)},
+ SnapStart={"ApplyOn": "PublishedVersions"},
+ )
+ try:
+ _snap_wait_active(lam, fname)
+ ver = lam.publish_version(FunctionName=fname)
+ assert ver["State"] == "Pending"
+ with pytest.raises(ClientError) as exc:
+ lam.invoke(FunctionName=fname, Qualifier=ver["Version"], Payload=b"{}")
+ assert exc.value.response["Error"]["Code"] == "ResourceConflictException"
+
+ _snap_wait_state(lam, fname, ver["Version"], "Active")
+ resp = lam.invoke(FunctionName=fname, Qualifier=ver["Version"], Payload=b"{}")
+ assert json.loads(resp["Payload"].read()) == {"ok": True}
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+def test_snapstart_publish_fails_version_on_broken_init(lam):
+ """An init that raises fails the *publish*: the version lands State=Failed
+ (AWS's failed-snapshot surface) instead of erroring at first invoke."""
+ fname = f"snap-fail-{_uuid_mod.uuid4().hex[:8]}"
+ lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler",
+ Code={"ZipFile": _zip_lambda("raise RuntimeError('boom-init')\n")},
+ SnapStart={"ApplyOn": "PublishedVersions"},
+ )
+ try:
+ _snap_wait_active(lam, fname)
+ ver = lam.publish_version(FunctionName=fname)
+ cfg = _snap_wait_state(lam, fname, ver["Version"], "Failed")
+ assert cfg["StateReasonCode"] == "FunctionError"
+ assert "boom-init" in cfg["StateReason"]
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+_SNAP_HOOK_LIB = (
+ "_before = []\n"
+ "_after = []\n"
+ "def register_before_snapshot(func, *args, **kwargs):\n"
+ " _before.append((func, args, kwargs))\n"
+ " return func\n"
+ "def register_after_restore(func, *args, **kwargs):\n"
+ " _after.append((func, args, kwargs))\n"
+ " return func\n"
+ "def get_before_snapshot():\n"
+ " return _before\n"
+ "def get_after_restore():\n"
+ " return _after\n"
+)
+
+_SNAP_HOOK_HANDLER = (
+ "import snapshot_restore_py as srp\n"
+ "CALLS = []\n"
+ "srp.register_before_snapshot(lambda: CALLS.append('before1'))\n"
+ "srp.register_before_snapshot(lambda: CALLS.append('before2'))\n"
+ "srp.register_after_restore(lambda: CALLS.append('after1'))\n"
+ "srp.register_after_restore(lambda: CALLS.append('after2'))\n"
+ "def handler(event, context):\n"
+ " return {'calls': CALLS}\n"
+)
+
+
+def test_snapstart_runtime_hooks_fire_on_published_version_only(lam):
+ """snapshot-restore-py hooks run during a published version's init —
+ before-snapshot hooks in reverse registration order, after-restore hooks
+ in registration order — and never for $LATEST, which has no snapshot."""
+ fname = f"snap-hook-{_uuid_mod.uuid4().hex[:8]}"
+ lam.create_function(
+ FunctionName=fname, Runtime="python3.12", Role=_LAMBDA_ROLE,
+ Handler="index.handler",
+ Code={"ZipFile": _make_zip_multi({
+ "index.py": _SNAP_HOOK_HANDLER,
+ "snapshot_restore_py.py": _SNAP_HOOK_LIB,
+ })},
+ SnapStart={"ApplyOn": "PublishedVersions"},
+ )
+ try:
+ _snap_wait_active(lam, fname)
+ ver = lam.publish_version(FunctionName=fname)
+ _snap_wait_state(lam, fname, ver["Version"], "Active")
+
+ resp = lam.invoke(FunctionName=fname, Qualifier=ver["Version"], Payload=b"{}")
+ payload = json.loads(resp["Payload"].read())
+ assert payload == {"calls": ["before2", "before1", "after1", "after2"]}
+
+ latest = lam.invoke(FunctionName=fname, Payload=b"{}")
+ assert json.loads(latest["Payload"].read()) == {"calls": []}
+ finally:
+ lam.delete_function(FunctionName=fname)
+
+
+def test_snapstart_version_worker_is_reapable():
+ """A published SnapStart version's pre-warmed worker must not be pinned by
+ the keep-the-first-worker rule, or every publish holds a subprocess
+ forever; the $LATEST first worker keeps its exemption."""
+ from ministack.core import lambda_runtime as lr
+
+ def _mk(version, snap_status):
+ cfg = {
+ "FunctionName": "reap-probe", "Runtime": "python3.12",
+ "Version": version,
+ "SnapStart": {"ApplyOn": "PublishedVersions",
+ "OptimizationStatus": snap_status},
+ "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:reap-probe",
+ }
+ w = lr.Worker("reap-probe", cfg, b"")
+ w.in_use = False
+ w.last_used = time.time() - 10_000
+ return w
+
+ latest = _mk("$LATEST", "Off")
+ snap_ver = _mk("1", "On")
+ with lr._lock:
+ lr._workers["000000000000:us-east-1:reap-probe:$LATEST"] = [latest]
+ lr._workers["000000000000:us-east-1:reap-probe:1"] = [snap_ver]
+ try:
+ lr.reap_idle_workers(ttl=1)
+ with lr._lock:
+ assert lr._workers.get("000000000000:us-east-1:reap-probe:$LATEST") == [latest]
+ assert "000000000000:us-east-1:reap-probe:1" not in lr._workers
+ finally:
+ with lr._lock:
+ lr._workers.pop("000000000000:us-east-1:reap-probe:$LATEST", None)
+ lr._workers.pop("000000000000:us-east-1:reap-probe:1", None)
+
+
+def test_snapstart_pending_version_reprovisions_on_restore():
+ """A SnapStart version persisted while Pending (crash mid-publish) must be
+ re-provisioned by restore_state — otherwise it answers 409 forever."""
+ import copy as _copy
+
+ arn = "arn:aws:lambda:us-east-1:000000000000:function:snap-restore-fn"
+ ver_cfg = {
+ "FunctionName": "snap-restore-fn", "FunctionArn": f"{arn}:1",
+ "Runtime": "python3.12", "Handler": "index.handler",
+ "MemorySize": 128, "Timeout": 3, "Version": "1",
+ "State": "Pending", "StateReason": "The function is being created.",
+ "StateReasonCode": "Creating",
+ "SnapStart": {"ApplyOn": "PublishedVersions", "OptimizationStatus": "On"},
+ "PackageType": "Zip",
+ }
+ func = {
+ "config": {**_copy.deepcopy(ver_cfg), "FunctionArn": arn,
+ "Version": "$LATEST", "State": "Active",
+ "SnapStart": {"ApplyOn": "PublishedVersions",
+ "OptimizationStatus": "Off"}},
+ "code_zip": _make_zip("def handler(e, c):\n return {}\n"),
+ "versions": {"1": {"config": ver_cfg,
+ "code_zip": _make_zip("def handler(e, c):\n return {}\n")}},
+ "next_version": 2, "tags": {}, "aliases": {},
+ "policy": {"Version": "2012-10-17", "Id": "default", "Statement": []},
+ "event_invoke_config": None, "event_invoke_configs": {},
+ "concurrency": None, "provisioned_concurrency": {},
+ }
+ key = ("000000000000", "us-east-1", "snap-restore-fn")
+ try:
+ lsvc.restore_state({"functions": {"snap-restore-fn": func}})
+ for _ in range(60):
+ state = ver_cfg["State"]
+ if state != "Pending":
+ break
+ time.sleep(0.25)
+ assert ver_cfg["State"] == "Active", f"stuck in {ver_cfg['State']}"
+ finally:
+ lsvc._functions._data.pop(key, None)
+ from ministack.core import lambda_runtime as _lr
+ _lr.invalidate_worker("snap-restore-fn", account="000000000000",
+ region="us-east-1")