Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
fix(video): measure extend/edit output length from the staged bytes; docs, changelog, v2.3.0
Soohong Kim123d180fed83
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e74377a..cb1c565 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,8 +2,12 @@All notable public changes to `sprite-gen` are recorded here. Versions track the `version:` field in `SKILL.md` and `pyproject.toml`.-## Unreleased (v2.2.1)+## Unreleased (v2.3.0)+- `sprite-gen video` gains the other Grok Imagine generation modes: `--last-frame` pins the closing frame (alone or with `--image`), and `--reference` (repeatable, up to 7) guides the clip with reference images named `<IMAGE_0>`, `<IMAGE_1>`, … in the prompt. `--image` is no longer required on its own. The image-only request body is unchanged and pinned by a snapshot test; the video → loop pipeline (`video-canvas`, `video-frames`, `video-loop`, `video-set`) is untouched.+- New `video-extend` (`POST /v1/videos/extensions`) continues a 2–15 s clip by 2–10 s and new `video-edit` (`POST /v1/videos/edits`) changes a clip of at most 8.7 s with a prompt. Both force the classic `grok-imagine-video` model (the 1.5 model refuses both endpoints), take no resolution / aspect / model flags because the output inherits the input's, check the input with ffprobe before uploading, and `video-extend` refuses a result shorter than its input.+- Local pre-checks refuse, before any call: no input at all, more than 7 references, `<IMAGE_n>` tags outside the given references (or with none), 1080p with references, and `last_frame` / `image`+`reference` on the classic model.+- The `sprite-gen-video-report` carries `mode` (`image-to-video`, `first-last`, `last-frame`, `reference`, `extend`, `edit`), `inputs` (local paths by role) and, for extend / edit, `input_duration`; every existing field stays.- The expired-login prescription for Grok video and images no longer names a bare prompt (`grok -p ok`): on the grok CLI that starts the coding agent in the current directory, which can read, write and spend on its own. The prescription is now a non-agent round-trip (`grok models`) run from an empty directory, with `grok login` as the fallback, and the message says so. A test pins the prescription to a non-agent command.## v2.2.0 - Independent asset tools and optional scenesdiff --git a/SKILL.md b/SKILL.mdindex c12060e..720be66 100644--- a/SKILL.md+++ b/SKILL.md@@ -1,7 +1,7 @@---name: sprite-gen-version: 2.2.1-description: "Generates images and game sprites through GPT or Grok with guided provider choices, separate saved defaults, automatic cleanup and optional curation. Handles sprite requests, ordinary image generation/editing, standalone image-to-video clips (i2v, animate this still, 그록 영상, 이매진 비디오, 스틸 움직여줘), chroma removal, animation atlases, video loops, 큐레이션뷰, image candidates, 팔레트 스왑, palette swap, recolor, rig layers, engine exports, repeating backgrounds, projected shadows, motion/contact inspection and optional scene composition from existing assets."+version: 2.3.0+description: "Generates images and game sprites through GPT or Grok with guided provider choices, separate saved defaults, automatic cleanup and optional curation. Handles sprite requests, ordinary image generation/editing, standalone image-to-video clips (i2v, animate this still, 그록 영상, 이매진 비디오, 스틸 움직여줘, first/last frame, reference-to-video, 영상 이어붙이기, 영상 편집, extend/edit a clip), chroma removal, animation atlases, video loops, 큐레이션뷰, image candidates, 팔레트 스왑, palette swap, recolor, rig layers, engine exports, repeating backgrounds, projected shadows, motion/contact inspection and optional scene composition from existing assets."license: Apache-2.0depends_on:required_bins:@@ -64,7 +64,8 @@ Pass choices already stated in the request. The guide checks access, combines ex| Task | Entry | Contract ||---|---|---|| GPT image sprites | `prepare`, `gen-set --provider codex`, `extract`, compose and QA | [atlas-workflow](docs/atlas-workflow.md) |-| Standalone video / animate a still | `video` | [video](docs/video.md) |+| Standalone video / animate a still, pin a last frame, reference images | `video` (`--image`, `--last-frame`, `--reference`) | [video](docs/video.md) |+| Continue or edit an existing clip | `video-extend`, `video-edit` | [video](docs/video.md) || Grok video sprites | `video-set` | [video-pipeline](docs/video-pipeline.md) || Ordinary image or edit | `gen --provider codex` or `gen --provider grok` | [gen](docs/gen.md) || Base and direction anchors | `anchor` | [directional-anchor-workflow](docs/directional-anchor-workflow.md) |diff --git a/docs/README.md b/docs/README.mdindex 6050faa..89e954a 100644--- a/docs/README.md+++ b/docs/README.md@@ -74,7 +74,7 @@ grouping is derived from `sprite_gen/_modules.py`, the one taxonomy table.| Doc | Owns ||---|---|| [gen.md](gen.md) | `sprite-gen gen` / `gen-set`: providers, default resolution, transparency strategy per provider, row usage |-| [video.md](video.md) | `sprite-gen video`: image to mp4 through Grok Imagine with the user's own credential |+| [video.md](video.md) | `sprite-gen video` / `video-extend` / `video-edit`: stills and clips to mp4 through Grok Imagine (image-to-video, last-frame pin, references, extension, editing) with the user's own credential || [video-pipeline.md](video-pipeline.md) | Pipeline B engine contract: state canvas, keyed frames, true-period and one-shot cycles, strip/GIF/WebP, the batch || [frame-interpolation.md](frame-interpolation.md) | Generative in-betweens for sprite frames, recorded as a take || [seamless-video-loop.md](seamless-video-loop.md) | Making a non-looping ambient clip loop forever (RIFE seam bridge) — a different job from pipeline B |diff --git a/docs/gen.md b/docs/gen.mdindex 0e03335..193007a 100644--- a/docs/gen.md+++ b/docs/gen.md@@ -56,7 +56,7 @@ transport. The user's Grok subscription login (`GROK_HOME` or `~/.grok`) takesprecedence even when `XAI_API_KEY` is set. Only an absent login file permits theAPI key (console credits); an invalid, expired or rejected login never does. See[authentication and expiry](video.md#setup--pick-one-credential)-for setup. No Grok executable is needed during generation with a valid credential.+for setup; the video modes (last-frame pin, references, `video-extend`, `video-edit`) are in [video.md](video.md#modes--which-flags-call-what). No Grok executable is needed during generation with a valid credential.No agent, credential fallback or automatic retry is started on failure.New images use `/v1/images/generations`; one reference uses `/v1/images/edits`diff --git a/docs/video.md b/docs/video.mdindex 7209f1a..9596282 100644--- a/docs/video.md+++ b/docs/video.md@@ -109,47 +109,101 @@ the API itself and never routes through the agent-side tool.## CLI+### `sprite-gen video` — generations (image-to-video, first/last frame, references)+```bashsprite-gen video \- --image still.png \ # PNG / JPEG / WebP; sent inline as a data URL+ [--image still.png] \ # first frame / the still to animate (PNG / JPEG / WebP, inline data URL)+ [--last-frame end.png] \ # pin the closing frame (grok-imagine-video-1.5 only); alone or with --image+ [--reference a.png --reference b.png …] \ # 1..7 references; name them in the prompt as <IMAGE_0>, <IMAGE_1>, …--prompt "Camera locked. Gentle idle sway, tail flick." \ # or --prompt-file--out clip.mp4 \[--duration 6] # 1..15 seconds (default 6)- [--resolution 720p] # 480p | 720p | 1080p (default 720p)+ [--resolution 720p] # 480p | 720p | 1080p (default 720p; 1080p is refused with --reference)[--aspect-ratio 1:1] # 1:1 16:9 9:16 4:3 3:4 3:2 2:3 (default: the still's ratio)[--audio | --no-audio] # default: the API's default (audio on)[--model grok-imagine-video-1.5][--report clip.report.json]```+At least one of `--image`, `--last-frame`, `--reference` is required. The mode is+derived from the flags (see the table above) and written to the report as `mode`.+Backward-compatible wrapper: `$SPRITE_GEN_ROOT/.venv/bin/python $SPRITE_GEN_ROOT/scripts/generate_sprite_video.py …` (same args).What happens, in order:-1. Validate the request (prompt, still, duration, resolution, aspect ratio) and- resolve the credential — all before any network call.+1. Validate the request (prompt, stills, duration, resolution, aspect ratio, the+ mode rules: ≤ 7 references, `<IMAGE_n>` tags in `0..count-1`, no tag without a+ reference, no 1080p with references, no `last_frame` / `image`+`reference` on the+ classic model) and resolve the credential — all before any network call.2. `POST /v1/videos/generations` with `model`, `prompt`, `duration`, `resolution`,- optional `aspect_ratio` / `generate_audio`, and `image.url` as a base64 data URL.- A `401/403` names the credential source and its fix; any reply without a- `request_id` fails with the API's own error text.+ then `image.url` / `last_frame.url` / `reference_images[].url` as base64 data+ URLs, optional `aspect_ratio` / `generate_audio`. A `401/403` names the+ credential source and its fix; any reply without a `request_id` fails with the+ API's own error text.3. `GET /v1/videos/{request_id}` every 4 s until `status` is `done`. `failed`,`expired`, a non-2xx poll, or the timeout (`SPRITE_GEN_VIDEO_TIMEOUT_SECONDS`,default 600) fail loudly. Nothing is written on any failure path.4. Download `video.url`, verify the bytes start with an mp4 `ftyp` box, then movethe file into `--out` atomically (`.part` staging).-The report (`sprite-gen-video-report`) carries `auth_source`, `model`,-`request_id`, `bytes`, `host`, requested/reported duration, resolution, aspect-ratio, audio flag, `elapsed_seconds`, and `polls`. **Tokens and download URLs are-never printed or written** — only the download host (`vidgen.x.ai`).+Reference-to-video tends to lock the opening composition to a referenced scene+image (probe m3: the clip opened on the third reference's framing), so put+character references first and scene references last, or leave the scene out.++### `sprite-gen video-extend` — continue a clip++```bash+sprite-gen video-extend \+ --video clip.mp4 \ # 2..15 s mp4 (a 15-second clip's 15.04 s container is accepted)+ --prompt "she presses the attack, camera tracks low" \+ --out longer.mp4 \+ [--duration 6] # seconds ADDED, 2..10 (default 6)+ [--report longer.report.json]+```++Model is fixed to the classic `grok-imagine-video` (`grok-imagine-video-1.5`+answers `400 Video extension is not supported for this model`). There are no+`--resolution` / `--aspect-ratio` / `--model` flags: the output inherits the+input's, capped at 720p. The input is checked locally (`.mp4`, `ftyp` box,+ffprobe length) before anything is uploaded; ffprobe missing is an error, not a+skipped check. The API returns **input + extension as one clip** (15.04 s in →+20.04 s out); a result shorter than the input is refused and not written. The+report carries `mode: "extend"`, `inputs.video`, `input_duration` (ffprobe+seconds) and `duration_requested` (the seconds added).++### `sprite-gen video-edit` — change a clip with a prompt++```bash+sprite-gen video-edit \+ --video clip.mp4 \ # at most 8.7 s+ --prompt "change her hakama to solid black, keep everything else" \+ --out edited.mp4 \+ [--report edited.report.json]+```++Same fixed classic model, same inherited resolution / aspect / length (the probe's+8.00 s input came back 7.71 s — the API trims the tail). Inputs over 8.7 s are+refused locally with an ffmpeg trim hint; there is no `--duration` at all.++### The report++`sprite-gen-video-report` carries `auth_source`, `model`, `mode`, `request_id`,+`inputs` (local paths by role: `image`, `last_frame`, `reference_images`,+`video`), `bytes`, `host`, requested/reported duration (`input_duration` for+extend / edit), resolution, aspect ratio, audio flag, `elapsed_seconds`, and+`polls`. The pre-modes `image` field stays. **Tokens and download URLs are never+printed or written** — only the download host (`vidgen.x.ai`).## Quotas and limits- Imagine quota is a weekly SuperGrok allowance; when it is exhausted the APIrefuses the POST and the run fails with that message (no retry loop here).- Duration 1–15 s, resolutions 480p/720p/1080p, the seven aspect ratios above —- from the xAI video docs as of 2026-09-08. A value outside those is rejected- locally before the call.+ from the xAI video docs as of 2026-09-08. Reference-to-video: ≤ 7 images, 720p+ max; extension: input 2–15 s, 2–10 s added; editing: input ≤ 8.7 s (docs as of+ 2026-09-13). A value outside those is rejected locally before the call.- Output is whatever the model returns (typically H.264 mp4 with audio unless`--no-audio`). Downstream frame extraction is a separate step and not part ofthis command.diff --git a/pyproject.toml b/pyproject.tomlindex f579e7c..4e379c2 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -12,7 +12,7 @@ build-backend = "setuptools.build_meta"name = "sprite-gen"# Release discipline: keep this package metadata version synchronized with# SKILL.md's `version:` field in the same release commit.-version = "2.2.1"+version = "2.3.0"description = "Component-row pipeline for clean 2D game sprites and animation atlases"readme = "README.md"license = "Apache-2.0"diff --git a/sprite_gen/gen/video.py b/sprite_gen/gen/video.pyindex 1d53b43..7bc6644 100644--- a/sprite_gen/gen/video.py+++ b/sprite_gen/gen/video.py@@ -243,6 +243,7 @@ class VideoResult:mode: str = MODE_IMAGE_TO_VIDEOinputs: dict[str, Any] = field(default_factory=dict) # role -> local path(s); never URLsinput_duration: float | None = None # extend / edit: ffprobe seconds of the input clip+ output_duration: float | None = None # extend / edit: ffprobe seconds of the published clipextra: dict[str, Any] = field(default_factory=dict)def to_dict(self) -> dict[str, Any]:@@ -262,6 +263,7 @@ class VideoResult:"duration_requested": self.duration_requested,"duration_reported": self.duration_reported,**({"input_duration": self.input_duration} if self.input_duration is not None else {}),+ **({"output_duration": self.output_duration} if self.output_duration is not None else {}),"resolution": self.resolution,"aspect_ratio": self.aspect_ratio,"generate_audio": self.generate_audio,@@ -357,12 +359,13 @@ def _submit_poll_publish(download: Callable[[str, str], bytes],sleep: Callable[[float], None],poll_timeout: float | None,- accept: Callable[[bytes, dict[str, Any]], None] | None = None,+ accept: Callable[[Path, dict[str, Any]], None] | None = None,) -> _Published:"""POST `body` to `endpoint`, poll `/videos/{id}`, download, verify, publish atomically.- `accept(data, final_poll)` may refuse the clip (raise SystemExit) before anything- is written — `video-extend` uses it to verify the returned length."""+ `accept(staged_path, final_poll)` may refuse the clip (raise SystemExit) while it is+ still the `.part` file — `video-extend` measures the staged bytes with ffprobe; the+ staged file is removed and `out` is never written."""started = time.monotonic()status, reply = call("POST", f"{API_BASE}/videos/{endpoint}", credential.token, body)if status in (401, 403):@@ -410,13 +413,17 @@ def _submit_poll_publish(raise SystemExit(f"{verb}: generation {request_id} is done but reported no video url")data = download(video_url, credential.token)verify_mp4(data)- if accept is not None:- accept(data, final)elapsed = time.monotonic() - startedout.parent.mkdir(parents=True, exist_ok=True)tmp = out.with_name(out.name + ".part")tmp.write_bytes(data)+ if accept is not None:+ try:+ accept(tmp, final)+ except BaseException:+ tmp.unlink(missing_ok=True)+ raiseos.replace(tmp, out)return _Published(data=data, request_id=str(request_id), model=model_reported, duration_reported=duration_reported,@@ -525,14 +532,17 @@ def extend_video("video": {"url": _data_url(clip)},}- def accept(data: bytes, final: dict[str, Any]) -> None:- # The API returns original + extension as one clip; anything shorter than the- # input means it did not extend and must not be published as if it had.- video = final.get("video") or {}- reported = video.get("duration") if isinstance(video, dict) else None- if isinstance(reported, (int, float)) and reported + 0.5 < input_seconds:+ measured: dict[str, float] = {}++ def accept(staged: Path, final: dict[str, Any]) -> None:+ # The API returns original + extension as one clip, but its poll reports+ # `video.duration` = the seconds ADDED (5 for a 15.04 s → 20.04 s run, live+ # 2026-09-13), so the length check measures the bytes, never the status.+ seconds = probe(staged)+ measured["output"] = seconds+ if seconds + EXTEND_INPUT_SLACK_SECONDS < input_seconds:raise SystemExit(- f"video-extend: the API reported a {reported}s clip for a {input_seconds:.2f}s input — not an extension; nothing was written"+ f"video-extend: the returned clip is {seconds:.2f}s for a {input_seconds:.2f}s input — not an extension; nothing was written")published = _submit_poll_publish(@@ -544,6 +554,7 @@ def extend_video(auth_source=credential.source, elapsed_seconds=published.elapsed_seconds, polls=published.polls,duration_requested=request.duration, duration_reported=published.duration_reported, host=published.host,prompt=request.prompt, mode=MODE_EXTEND, inputs={"video": str(clip)}, input_duration=input_seconds,+ output_duration=measured.get("output"),)@@ -587,15 +598,23 @@ def edit_video("prompt": request.prompt,"video": {"url": _data_url(clip)},}+ measured: dict[str, float] = {}++ def accept(staged: Path, final: dict[str, Any]) -> None:+ # No length rule for edits (the API trims the tail: 8.00 s in → 7.71 s out),+ # but the report says what came back, measured from the bytes.+ measured["output"] = probe(staged)+published = _submit_poll_publish(verb="video-edit", endpoint="edits", body=body, out=out, credential=credential,- call=call, download=download, sleep=sleep, poll_timeout=poll_timeout,+ call=call, download=download, sleep=sleep, poll_timeout=poll_timeout, accept=accept,)return VideoResult(out=out, bytes=len(published.data), model=published.model or request.model, request_id=published.request_id,auth_source=credential.source, elapsed_seconds=published.elapsed_seconds, polls=published.polls,duration_requested=0, duration_reported=published.duration_reported, host=published.host,prompt=request.prompt, mode=MODE_EDIT, inputs={"video": str(clip)}, input_duration=input_seconds,+ output_duration=measured.get("output"),)diff --git a/tests/gen/test_video_modes.py b/tests/gen/test_video_modes.pyindex 85f4628..55a4e37 100644--- a/tests/gen/test_video_modes.py+++ b/tests/gen/test_video_modes.py@@ -186,9 +186,15 @@ def _ext(tmp_path: Path, **overrides) -> video.ExtendRequest:return video.ExtendRequest(**kwargs)+def _probe(input_seconds: float, output_seconds: float):+ """ffprobe stand-in: the input clip vs the staged `.part` download."""+ return lambda clip: output_seconds if clip.name.endswith(".part") else input_seconds++def test_extend_posts_classic_model_to_extensions_and_inherits_the_rest(tmp_path: Path) -> None:- api = _FakeApi(done={"status": "done", "video": {"url": "https://vidgen.x.ai/v/2.mp4", "duration": 20.04}, "model": "grok-imagine-video"})- result = video.extend_video(_ext(tmp_path, duration=5), probe=lambda clip: 15.04, **api.kw())+ # Live 2026-09-13: the poll's video.duration is the seconds ADDED (5), the bytes are 20.04 s.+ api = _FakeApi(done={"status": "done", "video": {"url": "https://vidgen.x.ai/v/2.mp4", "duration": 5}, "model": "grok-imagine-video"})+ result = video.extend_video(_ext(tmp_path, duration=5), probe=_probe(15.04, 20.04), **api.kw())assert api.endpoint == f"{video.API_BASE}/videos/extensions"assert list(api.body) == ["model", "prompt", "duration", "video"]assert api.body["model"] == "grok-imagine-video" and api.body["duration"] == 5@@ -196,22 +202,24 @@ def test_extend_posts_classic_model_to_extensions_and_inherits_the_rest(tmp_pathassert "resolution" not in api.body and "aspect_ratio" not in api.bodypayload = result.to_dict()assert payload["mode"] == "extend" and payload["inputs"] == {"video": str((tmp_path / "in.mp4").resolve())}- assert payload["input_duration"] == 15.04 and payload["duration_reported"] == 20.04 and payload["duration_requested"] == 5+ assert payload["input_duration"] == 15.04 and payload["output_duration"] == 20.04+ assert payload["duration_reported"] == 5 and payload["duration_requested"] == 5assert result.out.read_bytes() == MP4def test_extend_accepts_the_frame_padded_fifteen_second_clip(tmp_path: Path) -> None:# probe m4c: a 15.04 s container went through and came back 20.04 s.api = _FakeApi(done={"status": "done", "video": {"url": "https://vidgen.x.ai/v/2.mp4", "duration": 20.04}, "model": "grok-imagine-video"})- video.extend_video(_ext(tmp_path, duration=5), probe=lambda clip: 15.04, **api.kw())+ video.extend_video(_ext(tmp_path, duration=5), probe=_probe(15.04, 20.04), **api.kw())assert api.body["duration"] == 5-def test_extend_refuses_a_result_shorter_than_the_input(tmp_path: Path) -> None:- api = _FakeApi(done={"status": "done", "video": {"url": "https://vidgen.x.ai/v/2.mp4", "duration": 5.0}, "model": "grok-imagine-video"})+def test_extend_refuses_a_result_shorter_than_the_input_measured_from_bytes(tmp_path: Path) -> None:+ # The poll claims 17 s; the downloaded bytes are 5 s. Bytes win, nothing is published.+ api = _FakeApi(done={"status": "done", "video": {"url": "https://vidgen.x.ai/v/2.mp4", "duration": 17.0}, "model": "grok-imagine-video"})request = _ext(tmp_path)- with pytest.raises(SystemExit, match="not an extension"):- video.extend_video(request, probe=lambda clip: 12.0, **api.kw())+ with pytest.raises(SystemExit, match="returned clip is 5.00s for a 12.00s input — not an extension"):+ video.extend_video(request, probe=_probe(12.0, 5.0), **api.kw())assert not request.out.exists() and not list(tmp_path.glob("*.part"))@@ -247,12 +255,12 @@ def _edit(tmp_path: Path, **overrides) -> video.EditRequest:def test_edit_posts_classic_model_to_edits_without_duration(tmp_path: Path) -> None:api = _FakeApi(done={"status": "done", "video": {"url": "https://vidgen.x.ai/v/3.mp4", "duration": 7.71}, "model": "grok-imagine-video"})- result = video.edit_video(_edit(tmp_path), probe=lambda clip: 8.0, **api.kw())+ result = video.edit_video(_edit(tmp_path), probe=_probe(8.0, 7.71), **api.kw())assert api.endpoint == f"{video.API_BASE}/videos/edits"assert list(api.body) == ["model", "prompt", "video"]assert api.body["model"] == "grok-imagine-video"payload = result.to_dict()- assert payload["mode"] == "edit" and payload["input_duration"] == 8.0 and payload["duration_reported"] == 7.71+ assert payload["mode"] == "edit" and payload["input_duration"] == 8.0 and payload["output_duration"] == 7.71assert payload["inputs"] == {"video": str((tmp_path / "in.mp4").resolve())}@@ -347,11 +355,12 @@ def test_run_extend_kwargs_writes_report(tmp_path: Path, monkeypatch) -> None:monkeypatch.setattr(video, "http_json", api.call)monkeypatch.setattr(video, "http_download", api.download)monkeypatch.setattr(video.time, "sleep", lambda s: None)- monkeypatch.setattr(video, "probe_duration_seconds", lambda clip, *, verb: 6.0)+ monkeypatch.setattr(video, "probe_duration_seconds", lambda clip, *, verb: 11.0 if clip.name.endswith(".part") else 6.0)out, report = tmp_path / "longer.mp4", tmp_path / "longer.report.json"rc = video.run_extend(video=_clip(tmp_path), prompt="go on", prompt_file=None, out=out, duration=5, report=report)assert rc == 0 and out.read_bytes() == MP4payload = json.loads(report.read_text(encoding="utf-8"))- assert payload["mode"] == "extend" and payload["model"] == "grok-imagine-video" and payload["input_duration"] == 6.0+ assert payload["mode"] == "extend" and payload["model"] == "grok-imagine-video"+ assert payload["input_duration"] == 6.0 and payload["output_duration"] == 11.0with pytest.raises(TypeError, match="unexpected keyword"):video.run_edit(video=_clip(tmp_path), prompt="p", prompt_file=None, out=out, report=None, duration=5)