Skip to content

Ministack: Free, open-source local AWS emulator - 60+ services, Terraform compatible, real databases. Free forever. MIT licensed.

fix(glue): complete Glue 5 file writes when --extra-py-files is in the output bucket (#2448)

mishukdutta-cz0de8bdfa6435
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 05960780..d14e9543 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,7 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- **Athena — the result file is saved at the reported `OutputLocation`** — a query's `<id>.csv` and `<id>.csv.metadata` are now saved directly under the result location, at the path that `GetQueryExecution` reports, as AWS does. Before, they were saved one folder too deep, at `<location>/<id>.csv/<id>.csv`, so a client that read the reported path got `NoSuchKey`.
- **Athena — an enforced workgroup overrides the client-side result location** — when a workgroup has `EnforceWorkGroupConfiguration: true`, `StartQueryExecution` now writes results to the workgroup's `OutputLocation`, even when the request gives its own `ResultConfiguration.OutputLocation`, as AWS does. Before, the request's location was used.
- **Lambda — a partial ARN as the `FunctionName`** — `123456789012:function:my-function[:qualifier]` resolved to the account ID as the function name, so `Invoke` and the other function operations failed with `Function not found: ...:function:123456789012`; it now resolves as the full ARN in the caller's region, as on AWS, and gives the same result as that full ARN.
+- **Glue — Glue 5 file writes to the bucket that holds `--extra-py-files`** — a CSV (or other plain-file) write to that bucket stayed an open multipart upload, so the job succeeded but the output folder had only a 0-byte `part-*` marker under `__magic_job-…/`. `spark-submit` downloads `s3://` `--extra-py-files` before Spark starts, and the S3A filesystem it makes for that bucket saved the magicv2 committer's upload list in S3, where the commit did not look. Every S3A filesystem now tracks these uploads in memory, so the commit completes them. Glue 4 jobs use `DirectFileOutputCommitter` and do not change.
- **Step Functions — state events inside a Map or a Parallel branch** — the states in an inline Map item or a Parallel branch wrote no `<Type>StateEntered` or `<Type>StateExited` events, so `GetExecutionHistory` did not show them. They are now in the execution history, as on AWS; for a Distributed Map they are in each child execution's history and not in the parent's. Events added at the same time from several Map items or branches can no longer get the same `id`.
## [1.5.33] — 2026-10-26
diff --git a/ministack/services/glue.py b/ministack/services/glue.py
index 50f2e422..b94dafbb 100644
--- a/ministack/services/glue.py
+++ b/ministack/services/glue.py
@@ -2055,6 +2055,10 @@ def _execute_spark_docker(run, job, job_name, args, script_path, docker_client):
"spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem",
"--conf",
"spark.hadoop.fs.s3a.connection.ssl.enabled=false",
+ # Glue 5's magicv2 committer completes only the uploads it tracks in memory. Without this, a write
+ # to the bucket that spark-submit fetched s3:// --py-files from is never completed.
+ "--conf",
+ "spark.hadoop.fs.s3a.committer.magic.track.commits.in.memory.enabled=true",
]
# Iceberg GlueCatalog conf -- add when the user's job or --conf references
diff --git a/tests/test_glue.py b/tests/test_glue.py
index a00cf7bd..6d055c5b 100644
--- a/tests/test_glue.py
+++ b/tests/test_glue.py
@@ -2790,6 +2790,54 @@ def test_glue_spark_container_env_points_sdk_at_ministack(tmp_path, monkeypatch)
assert run["JobRunState"] == "SUCCEEDED"
+def test_glue_spark_submit_tracks_magic_commits_in_memory(tmp_path, monkeypatch):
+ """spark-submit fetches s3:// --py-files before Spark starts. Without in-memory tracking on every
+ S3A filesystem, Glue 5's magicv2 committer never completes a write to that bucket."""
+ from ministack.services import glue as _glue
+
+ monkeypatch.setattr(_glue, "_ministack_network", "")
+ monkeypatch.delenv("MINISTACK_HOST", raising=False)
+
+ created = {}
+
+ class _FakeContainer:
+ def put_archive(self, path, data):
+ pass
+
+ def start(self):
+ pass
+
+ def wait(self, timeout=None):
+ return {"StatusCode": 0}
+
+ def logs(self, tail=None):
+ return b""
+
+ def remove(self, force=False, v=False):
+ pass
+
+ class _FakeContainers:
+ def get(self, name):
+ raise RuntimeError("no such container")
+
+ def create(self, **kwargs):
+ created.update(kwargs)
+ return _FakeContainer()
+
+ class _FakeDocker:
+ containers = _FakeContainers()
+
+ script = tmp_path / "job.py"
+ script.write_text("print('hi')")
+ args = {"--extra-py-files": "s3://out-bucket/deps.zip"}
+ _glue._execute_spark_docker({"Id": "jr_magic"}, {"Timeout": 1}, "magicjob", args, str(script), _FakeDocker())
+
+ command = created["command"]
+ setting = "spark.hadoop.fs.s3a.committer.magic.track.commits.in.memory.enabled=true"
+ assert setting in command
+ assert command[command.index(setting) - 1] == "--conf"
+
+
# A small stand-in for botocore: create_client returns the endpoint_url it gets.
_FAKE_BOTOCORE_SESSION = (
"ORIGIN = {origin!r}\n"