Skip to content

Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.

docs: sprite-gen video setup contract, v1.61.0

Soohong Kim9f32d30d2b2e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 85bb765..3337fbd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,13 @@
All notable public changes to `sprite-gen` are recorded here. Versions track the `version:` field in `SKILL.md` and `pyproject.toml`.
+## v1.61.0 - Image to Video
+
+- Added `sprite-gen video`: one still + prompt → a verified mp4 through Grok Imagine (`POST /v1/videos/generations`), with duration 1–15 s, 480p/720p/1080p, optional aspect ratio and audio flag, and a `sprite-gen-video-report` JSON.
+- Credentials are the user's own and never part of the repo: `XAI_API_KEY` when set, otherwise the `grok` CLI login file (`~/.grok/auth.json`, `GROK_HOME` honoured). The report records `auth_source`; tokens and download URLs are never printed or written.
+- An expired grok login fails before any upload with the exact refresh command; a set-but-empty `XAI_API_KEY`, a missing credential, a refused request, a failed/expired generation, a poll timeout, or a non-mp4 download each fail by name and write nothing.
+- New wrapper `scripts/generate_sprite_video.py` and docs at `docs/video.md`.
+
## v1.60.0 - Native Alpha
- `sprite-gen gen --transparent` now follows a per-provider transparency strategy declared once on each adapter (`Provider.transparency`). `codex` asks `image_gen` for a genuinely transparent background and publishes the measured alpha (`native`, first choice); `grok` keeps deterministic chroma keying because Grok Imagine returns JPEG only.
diff --git a/README.md b/README.md
index 1bb7a3e..789573d 100644
--- a/README.md
+++ b/README.md
@@ -252,6 +252,15 @@ implementation. Transparent stills follow a per-provider strategy: `codex` asks
is published; `grok` is keyed out of a chroma background. See
[`docs/gen.md`](docs/gen.md) for the CLI and verification contract.
+### Image to video
+
+`sprite-gen video --image still.png --prompt "…" --out clip.mp4` animates one
+still through Grok Imagine using **your own** credential — the `grok` CLI login
+(`grok login`, SuperGrok Imagine quota) or an `XAI_API_KEY`. Nothing is shipped
+with the repo; the report says which one ran. An expired login stops before
+uploading and tells you the exact refresh command. Setup, parameters, and the
+verification contract: [`docs/video.md`](docs/video.md).
+
## Attribution
The component-row workflow is inspired by the Apache-2.0 licensed `hatch-pet` skill, but targets generic game sprite atlases and includes no pet packages or pet visual assets.
diff --git a/SKILL.md b/SKILL.md
index 21947f8..9768904 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -1,6 +1,6 @@
---
name: sprite-gen
-version: 1.60.0
+version: 1.61.0
description: "Generate clean 2D game sprites and animation atlases with a component-row pipeline: base identity, numeric sprite-request SSoT, per-state layout guides, image-gen row strips, chroma-key alpha cleanup, connected-component frame extraction, cell-based atlas composition, QA reports, and runtime manifest frame_layout. Its curation webview also serves ANY image-candidate set (icons, logos, generated drafts) — agent chat can't render images, this can: unpack_atlas_run --pngs-dir import, then serve_curation side-by-side compare/pick. Palette-swap bake (`sprite-gen recolor`) turns a base sheet + palette map into N colourway sheets; the curation view blink-compares and adopts a pick into curation.json.recolor.picked. Curation triggers (KR/EN): 큐레이션, 큐레이션뷰, 큐레이션 해줘, 이미지 후보 보여줘/안 보임, 나란히 비교, 골라볼게 띄워줘, curation view, show image candidates side by side, let me pick. Recolor triggers (KR/EN): 팔레트 스왑, 팔레트 베이크, 리컬러, 색깔 바꾸기, 컬러웨이, 색 변형, 팔레트 맵, 색갈이, palette swap, recolor, colourway, colorway, bake variants, palette map."
license: Apache-2.0
depends_on:
@@ -12,6 +12,7 @@ depends_on:
required_scripts:
- scripts/prepare_sprite_run.py
- scripts/generate_sprite_image.py
+ - scripts/generate_sprite_video.py
- scripts/extract_sprite_row_frames.py
- scripts/interpolate_frames.py
- scripts/compose_sprite_atlas.py
@@ -195,6 +196,7 @@ Scripts are explicit pipeline commands, not hidden imports. One job each (stage
- `sprite_gen/compose/compose_layers.py` (`sprite-gen compose-layers`) — deterministic composite bake for a run that declares a **rig**. Stacks curated rows onto each other by integer pivot translation + arbitrary alpha masks (no resampling, rotation or scale of its own) into `<run-dir>/layers/<name>.png` + `<name>.manifest.json` + `layers.report.json`. Optional and opt-in: a request with no `rig` / `track` / `layers` is not a layer run and is refused by name rather than composed. All-or-nothing — declaration, composition **and** publish: one violation reports every violation and writes nothing. Declaration schema, CLI usage, and what `prepare` carries: [`docs/layer-tracks.md`](docs/layer-tracks.md).
- `unpack_atlas_run.py` — inverse of compose: rebuild a curator-ready run dir from a finished sheet (`--grid` > `--manifest` > auto-detect) or import a PNG folder (`--pngs-dir`, with sibling `meta.json` labels/iso grid).
- `export_curated_pngs.py` — export curated frames back to named PNGs with the transform baked in, into `<run-dir>/curated/`; the deliverable for imported still sets.
+- `sprite_gen/gen/video.py` (`sprite-gen video`) — one still → verified mp4 through Grok Imagine (`POST /v1/videos/generations`) with the user's own credential: `XAI_API_KEY` if set, else the `grok` CLI login (`~/.grok/auth.json`, SuperGrok Imagine quota). Expired login fails before upload with the refresh command; tokens/URLs never reach the report. The `grok-imagine-video` skill is a thin shuttle over this. Contract: [`docs/video.md`](docs/video.md).
- `cutout.py` (`sprite-gen cutout`) — background remover for **imported** images (not pipeline output, which is already keyed). Routes on the corner background colour (`--key auto|white|magenta|green`): **white/ivory** → position matte (corner flood-fill keeps interior highlights unholed → decontaminated soft-alpha border + soft erode); **magenta/green key** → reuse the verified `extract.remove_chroma_background` engine as-is (no drift — key colours are absent from objects so its colour-only cut is safe there). `--white-check` writes cyan/magenta/yellow verification composites. No Silent Fallback (leftover non-zero RGB under transparency raises).
- `slice_sheet_cells.py` — slice a multi-figure grid sheet (same character, N expressions/variants in one image) into per-cell standing cuts: v1.13 chroma alpha + centroid cell assignment + merged-figure split/in-cell re-label + neighbour-debris drop + per-cell height normalization + shared feet baseline. For dialogue cut-in portraits (立ち絵), not animation rows. Detail: [`docs/sheet-slicing.md`](docs/sheet-slicing.md).
- `check_visible_magenta.py` — optional screenshot QA guard for visible chroma-key leakage.
@@ -505,6 +507,7 @@ sprite-gen (this SKILL.md = behavior contract + hub)
│
├─ GENERATION ── "raw/<state>.png from prompts (the one AI step)"
│ ├─ docs/gen.md # sprite-gen gen provider CLI · verified PNG/report · image-gen shuttle
+│ ├─ docs/video.md # sprite-gen video · Grok Imagine i2v with the user's own grok login / XAI_API_KEY
│ ├─ docs/frame-interpolation.md # generative in-between (codex/grok) → take raw · auth prereqs · RIFE retire rationale
│ └─ docs/seamless-video-loop.md # non-looping AI video clip → seamless loop: flow-matched cut + RIFE seam bridge
│
diff --git a/docs/video.md b/docs/video.md
new file mode 100644
index 0000000..5d922fa
--- /dev/null
+++ b/docs/video.md
@@ -0,0 +1,96 @@
+# `sprite-gen video` — image to video with your own Grok login (engine SSoT)
+
+`sprite-gen video` animates one still into a short mp4 through **Grok Imagine**
+(xAI `POST /v1/videos/generations`). It is the video counterpart of
+[`sprite-gen gen`](gen.md): one call = one still (+ prompt) → one **verified** mp4
+on disk plus a machine-readable report. The `grok-imagine-video` skill is a thin
+shuttle over this command.
+
+No credential is shipped with this repository. You bring your own, in one of two
+forms, and every run reports which one it used.
+
+## Setup — pick one credential
+
+| `auth_source` | What you need | Billing | How to set it up |
+|---|---|---|---|
+| `grok-login` (default) | the `grok` CLI signed in once | your SuperGrok **Imagine quota** (no console spend) | install the grok CLI, run `grok login` (`--oauth` for a browser, `--device-auth` for a headless box). It writes `~/.grok/auth.json`; this tool only reads it. |
+| `XAI_API_KEY` | an xAI console API key | console credit | `export XAI_API_KEY=xai-…` |
+
+Resolution order is fixed: **`XAI_API_KEY` wins when set**, otherwise the grok
+login file (`GROK_HOME` relocates `~/.grok`). Neither is a fallback for the other —
+a set-but-empty `XAI_API_KEY` is an error, and with no key and no login the run
+stops with both setup paths spelled out.
+
+### The login token expires — and this tool does not refresh it
+
+The grok CLI stores an OIDC access token that lasts about six hours
+(2026-09-08 measurement: `expires_at` 10:56Z, refreshed to 18:26Z by the next grok
+command). `sprite-gen video` reads `expires_at` **before uploading anything**; if
+it has passed, the run fails with the refresh prescription instead of gambling on
+a 403 mid-upload:
+
+```
+video: the grok login token expired at 2026-09-08T10:56:34Z (now …); nothing was uploaded.
+ refresh it with any grok CLI command that reaches the API, e.g. `grok -p ok --output-format plain`,
+ or sign in again with `grok login`. This tool never rewrites ~/.grok/auth.json itself.
+```
+
+Why not refresh it here: `auth.json` is the grok CLI's file, the refresh token in
+it may rotate, and a second writer would break the login the user relies on
+everywhere else. The same shape as `sprite-gen gen`'s `codex login status` gate.
+
+### Why the direct API call and not Grok Build's `image_to_video` tool
+
+Grok Build's built-in `image_to_video` tool posts without `output.upload_url`, and
+on Zero-Data-Retention teams the API answers `HTTP 400 — Zero Data Retention teams
+must provide output.upload_url for video generation`. The same login calling
+`/v1/videos/generations` directly (no `output.upload_url`) succeeds and bills the
+Imagine quota (verified 2026-08-22, re-verified 2026-09-08). So this engine calls
+the API itself and never routes through the agent-side tool.
+
+## CLI
+
+```bash
+sprite-gen video \
+ --image still.png \ # PNG / JPEG / WebP; sent inline as a data URL
+ --prompt "Camera locked. Gentle idle sway, tail flick." \ # or --prompt-file
+ --out clip.mp4 \
+ [--duration 6] # 1..15 seconds (default 6)
+ [--resolution 720p] # 480p | 720p | 1080p (default 720p)
+ [--aspect-ratio 1:1] # 1:1 16:9 9:16 4:3 3:4 3:2 2:3 (default: the still's ratio)
+ [--audio | --no-audio] # default: the API's default (audio on)
+ [--model grok-imagine-video-1.5]
+ [--report clip.report.json]
+```
+
+Backward-compatible wrapper: `$SPRITE_GEN_ROOT/.venv/bin/python $SPRITE_GEN_ROOT/scripts/generate_sprite_video.py …` (same args).
+
+What happens, in order:
+
+1. Validate the request (prompt, still, duration, resolution, aspect ratio) and
+ resolve the credential — all before any network call.
+2. `POST /v1/videos/generations` with `model`, `prompt`, `duration`, `resolution`,
+ optional `aspect_ratio` / `generate_audio`, and `image.url` as a base64 data URL.
+ A `401/403` names the credential source and its fix; any reply without a
+ `request_id` fails with the API's own error text.
+3. `GET /v1/videos/{request_id}` every 4 s until `status` is `done`. `failed`,
+ `expired`, a non-2xx poll, or the timeout (`SPRITE_GEN_VIDEO_TIMEOUT_SECONDS`,
+ default 600) fail loudly. Nothing is written on any failure path.
+4. Download `video.url`, verify the bytes start with an mp4 `ftyp` box, then move
+ the file into `--out` atomically (`.part` staging).
+
+The report (`sprite-gen-video-report`) carries `auth_source`, `model`,
+`request_id`, `bytes`, `host`, requested/reported duration, resolution, aspect
+ratio, audio flag, `elapsed_seconds`, and `polls`. **Tokens and download URLs are
+never printed or written** — only the download host (`vidgen.x.ai`).
+
+## Quotas and limits
+
+- Imagine quota is a weekly SuperGrok allowance; when it is exhausted the API
+ refuses the POST and the run fails with that message (no retry loop here).
+- Duration 1–15 s, resolutions 480p/720p/1080p, the seven aspect ratios above —
+ from the xAI video docs as of 2026-09-08. A value outside those is rejected
+ locally before the call.
+- Output is whatever the model returns (typically H.264 mp4 with audio unless
+ `--no-audio`). Downstream frame extraction is a separate step and not part of
+ this command.
diff --git a/pyproject.toml b/pyproject.toml
index 83b5fd0..98d5f96 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -12,7 +12,7 @@ build-backend = "setuptools.build_meta"
name = "sprite-gen"
# Release discipline: keep this package metadata version synchronized with
# SKILL.md's `version:` field in the same release commit.
-version = "1.60.0"
+version = "1.61.0"
description = "Component-row pipeline for clean 2D game sprites and animation atlases"
readme = "README.md"
license = "Apache-2.0"