Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
fix(video): a poll answer cut short, garbled or not JSON is asked again on the same id; a cut POST ends on the transport's line
Soohong Kimd63f474a3ba3
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c4a18a2..eb4e7b3 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -11,6 +11,8 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the- **Fix: a poll that gets no answer, or whose credential xAI rejects, leaves the request's id.** After xAI had accepted the request, a poll whose connection failed or timed out ended the run with the transport's line and no `request_id`, and a poll answered `401` ended it with its status. A poll that gets no answer is a GET, which buys nothing, so it is now asked again with the refused polls' backoff (a refused poll and a lost one are two failures in a row) and a stderr note, `poll N of generation <id> got no answer (<reason>); asking again in 8s`; lost until the poll timeout, the run is unfinished. A poll answered `401` or `403` — the credential stopped working after the POST, typically an expired login — ends the run unfinished at once, naming the credential's fix (refresh the grok login, or check `XAI_API_KEY`) before the request to ask for. A POST that gets no answer ends as before: no `request_id` came back. `xai.http_json` raises `xai.TransportError` for a request that got no answer, still a `SystemExit` with the same line, the socket's words in `reason`.- **Output change: the unfinished line names the command that asks again; reports count the polls that got no answer.** The line reads ``ask for request_id=<id> again with `sprite-gen video-resume --request-id <id> --out <out>` instead of submitting a new request``, `request_id=<id>` kept as it was. The success and unfinished reports gain `polls_unanswered`, how many of `polls` got no answer, each asked again; the unfinished report's `last_poll` gains `error`, what the socket said, when its last poll got none (`http` then `null`).- **`tests/gen/test_video.py`** asks again by id and loses polls: `xai.http_json` raises `xai.TransportError` with the old line for a DNS failure, a timeout and a reset; lost, pending, refused, lost, done is asked on the one request with waits 8, 4, 8, 16 and published with `polls_unanswered` 2; lost until a 30 s timeout waits 8, 16, 6 and ends unfinished with `last_poll.error`; a lost POST is sent once and is not unfinished; a 401 or 403 poll, on the login or on the key, ends unfinished at once naming its fix; `video-resume` polls with no POST and announces no charge, ends unfinished on the same id when refused, lost or pending past the timeout or when its credential is rejected, ends with `(HTTP 404)` for an unknown id, refuses eight ids that cannot be one before any call, and its CLI writes the resume report, the unfinished report and an empty stdout; the unified CLI routes it under `gen`.+- **Fix: a poll whose answer is cut short, garbled or not JSON is asked again on the same request — the run no longer ends with no `request_id`.** A poll answer that broke after it began — a body that closed before its `Content-Length` (`http.client.IncompleteRead`), a status line that is not one (`BadStatusLine`), a refusal's body cut the same way — escaped `video`, `video-extend`, `video-edit` and `video-resume` as a raw exception, with no `request_id`, no unfinished report and no resume command, so a caller could only submit the clip again and buy it twice; a `200` poll whose body is not JSON ended the run on `xai: response was not valid JSON`, also with no id. Both are now a poll that got no answer it could read, as a lost one is: asked again on the same `request_id` with the same backoff and stderr note (`got no answer (IncompleteRead(10 bytes read, 190 more expected))`, `got no answer (response was not valid JSON)`), counted in `polls_unanswered`, and unfinished at the poll timeout with that reason in `last_poll.error`. A POST so answered still ends at once and is never sent again — no `request_id` could be read: on the transport's line for a broken answer (a raw exception before), on the not-JSON line it always had. `xai.http_json` raises `xai.TransportError` for every answer it cannot read whole, each with the line it always had; `reason` is the `http.client` exception's repr for a broken answer, `response was not valid JSON` for the other.+- **`tests/gen/test_video.py`** answers over a real socket from a local server: `xai.http_json` raises `xai.TransportError` for a body closed 10 bytes into its 200, a 503 body cut the same way, a garbage status line (the transport's line) and a whole non-JSON `200` (the not-JSON line); cut, garbled, not JSON, cut refusal, pending, done is asked on the one request with waits 8, 16, 32, 60, 4 and published with `polls_unanswered` 4; a resumed poll cut short is asked again and published; cut until a 30 s timeout, the CLI leaves the unfinished report with `last_poll.error` the `IncompleteRead` and an empty stdout; a cut POST is sent once and is not unfinished.- **Output change: an `openai` image report carries the response's rate-limit headers — `extra.rate_limit`.** The provider dropped the response headers, so what the account may spend per minute and has left could not be read off real traffic. `extra.rate_limit` now holds every `x-ratelimit-*` header the response carried, names lower-cased and values as sent, and `{}` when none came (docs/gen.md). `openai_provider.http_image` returns the response headers as a third value.- **`tests/gen/test_openai_provider.py`** answers with headers: the report holds exactly the `x-ratelimit-*` ones among others of any case, and `{}` for a response without any.diff --git a/docs/video.md b/docs/video.mdindex 86d915e..81488c8 100644--- a/docs/video.md+++ b/docs/video.md@@ -148,18 +148,22 @@ What happens, in order:API's own error text.3. `GET /v1/videos/{request_id}` every 4 s until `status` is `done`. `failed`,`expired` or a non-2xx poll fails loudly — except a poll answered `429` or `500`,- `502`, `503`, `504`, or one that gets no answer at all (the connection failed or- timed out): the request was accepted, so that poll is asked again on the same- `request_id` (a new POST would buy the clip twice; a GET buys nothing), after a wait- that doubles from 8 s up to 60 s while the polls fail and goes back to 4 s on an- answer. A `[video] poll N of generation <id> was refused (429); asking again in 8s`- line — or `… got no answer (timed out); asking again in 8s` — goes to stderr each- time. The poll timeout (`SPRITE_GEN_VIDEO_TIMEOUT_SECONDS`, default 600, counted- from the POST's answer) bounds the whole poll: no wait runs past it, one last poll is- asked at it, and a generation still pending, refused or unanswered then is- *unfinished* (below). A poll answered `401` or `403` — the credential stopped- working after the POST, typically a login that expired — is unfinished at once. No- clip is written on any failure path.+ `502`, `503`, `504`, or one that gets no answer it can read (the connection failed+ or timed out; the answer broke on the way — a body shorter than its+ `Content-Length`, a status line that is not one, a refusal's body cut the same way;+ or a `200` body that is not JSON): the request was accepted, so that poll is asked+ again on the same `request_id` (a new POST would buy the clip twice; a GET buys+ nothing), after a wait that doubles from 8 s up to 60 s while the polls fail and goes+ back to 4 s on an answer. A `[video] poll N of generation <id> was refused (429);+ asking again in 8s` line — or `… got no answer (timed out); asking again in 8s`,+ the reason `IncompleteRead(10 bytes read, 190 more expected)` or `response was not+ valid JSON` for an answer it could not read — goes to stderr each time. The poll+ timeout (`SPRITE_GEN_VIDEO_TIMEOUT_SECONDS`, default 600, counted from the POST's+ answer) bounds the whole poll: no wait runs past it, one last poll is asked at it,+ and a generation still pending, refused or unanswered then is *unfinished* (below).+ A poll answered `401` or `403` — the credential stopped working after the POST,+ typically a login that expired — is unfinished at once. No clip is written on any+ failure path.4. Download `video.url`, verify the bytes start with an mp4 `ftyp` box, then movethe file into `--out` atomically (`.part` staging).@@ -236,8 +240,8 @@ the request — the unfinished report's `auth_source`; refresh an expired login`video`), `bytes`, `host`, requested/reported duration (`input_duration` forextend / edit), resolution, aspect ratio, audio flag, `elapsed_seconds`, and`polls`, `polls_refused` (how many of `polls` xAI answered 429 or 5xx, each asked-again) and `polls_unanswered` (how many got no answer, each asked again). The-pre-modes `image` field stays. **Tokens and download URLs are never+again) and `polls_unanswered` (how many got no answer they could read, each asked+again). The pre-modes `image` field stays. **Tokens and download URLs are neverprinted or written** — only the download host (`vidgen.x.ai`).### An unfinished generation@@ -267,16 +271,20 @@ same reason. With `--report` the run leaves a"last_poll": {"http": 429, "status": null}, "elapsed_seconds": 600.0}```-`last_poll` gains `"error"` (what the socket said) when the last poll got no answer,-its `http` then `null`. `auth_source` is the credential that submitted it — the one-that can ask for it again. A report already at that path is replaced, as on success.+`last_poll` gains `"error"` (what the transport said: `timed out`,+`IncompleteRead(10 bytes read, 190 more expected)`, `response was not valid JSON`, …)+when the last poll got no answer it could read, its `http` then `null`. `auth_source`+is the credential that submitted it — the one that can ask for it again. A report+already at that path is replaced, as on success.Every verb that polls (`video`, `video-extend`, `video-edit`, `video-resume`) doesthe same, `verb` naming it; a `video-resume` report's `model` is `null`.-A POST that gets no answer is not unfinished: no `request_id` came back, and the line-is the transport's own (`xai: request failed or timed out; no automatic retry (the-server may have accepted it)`). Python callers catch `xai.TransportError`, a-`SystemExit` with that line and the socket's words in `reason`.+A POST that gets no answer it can read is not unfinished: no `request_id` could be+read, so it ends at once and is never sent again. Its line is the transport's own+(`xai: request failed or timed out; no automatic retry (the server may have accepted+it)`), or `xai: response was not valid JSON` for a whole answer that is not JSON.+Python callers catch `xai.TransportError`, a `SystemExit` with that line and the+transport's words in `reason`.## Quotas and limitsdiff --git a/sprite_gen/gen/video.py b/sprite_gen/gen/video.pyindex 2329d46..ac5fd43 100644--- a/sprite_gen/gen/video.py+++ b/sprite_gen/gen/video.py@@ -101,8 +101,9 @@ MODE_RESUME = "resume"REQUEST_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*")POLL_INTERVAL_SECONDS = 4.0POLL_TIMEOUT_SECONDS = int(os.environ.get("SPRITE_GEN_VIDEO_TIMEOUT_SECONDS", "600"))-# A poll xAI throttles (429) or fails on its own side (5xx), or one that gets no answer at all (the-# connection failed or timed out), is asked again, on the same request: the POST was accepted, so a+# A poll xAI throttles (429) or fails on its own side (5xx), or one that gets no answer it can read (the+# connection failed or timed out, the answer broke on the way or was not JSON — xai.TransportError), is+# asked again, on the same request: the POST was accepted, so a# new submission would buy the clip twice, and a GET buys nothing. The wait doubles from# POLL_INTERVAL_SECONDS while the polls fail, up to the ceiling; no wait runs past the poll timeout,# which bounds the whole poll however its polls are answered.@@ -138,7 +139,7 @@ class GenerationUnfinished(SystemExit):self.polls_unanswered = polls_unansweredself.last_http = last_httpself.last_status = last_status- self.last_error = last_error # what the socket said when the last poll got no answer+ self.last_error = last_error # what the transport said when the last poll got no answer it could readself.elapsed_seconds = elapsed_secondsdef to_dict(self) -> dict[str, Any]:@@ -316,7 +317,7 @@ class VideoResult:duration_requested: int | None # video-resume: None, this run requested nothingduration_reported: float | None = Nonepolls_refused: int = 0 # polls xAI answered 429 or 5xx, each asked again- polls_unanswered: int = 0 # polls that got no answer (connection failed or timed out), each asked again+ polls_unanswered: int = 0 # polls that got no answer they could read (xai.TransportError), each asked againhost: str | None = Noneimage: Path | None = Noneprompt: str = ""diff --git a/sprite_gen/gen/xai.py b/sprite_gen/gen/xai.pyindex a745cba..f8b87c4 100644--- a/sprite_gen/gen/xai.py+++ b/sprite_gen/gen/xai.py@@ -7,6 +7,7 @@ that login. Never spawn an agent or switch sources after a login failure."""from __future__ import annotations+import http.clientimport jsonimport osimport urllib.error@@ -131,18 +132,36 @@ def resolve_credential(*, env: dict[str, str] | None = None, now: datetime | NonHttpCall = Callable[[str, str, str, dict | None], tuple[int, Any]]TRANSPORT_FAILED = "xai: request failed or timed out; no automatic retry (the server may have accepted it)"+NOT_JSON = "xai: response was not valid JSON"class TransportError(SystemExit):- """No HTTP answer came back: the connection failed or timed out, and the server may have taken the- request. Still the SystemExit and line it always was; a caller that may safely ask again (a poll of- an accepted request) catches this type. `reason` is what the socket said."""-- def __init__(self, reason: str) -> None:- super().__init__(TRANSPORT_FAILED)+ """No answer that could be read came back: the connection failed or timed out, the answer broke+ after it began (a body shorter than its Content-Length, a status line that is not one), or it came+ whole and was not JSON — and the server may have taken the request. Still the SystemExit and line+ it always was (`NOT_JSON` for the last); a caller that may safely ask again (a poll of an accepted+ request) catches this type. `reason` is what the transport said."""++ def __init__(self, reason: str, line: str = TRANSPORT_FAILED) -> None:+ super().__init__(line)self.reason = reason+# Everything that ends a request before its answer is read whole. http.client raises its own+# HTTPException family (IncompleteRead, BadStatusLine, LineTooLong, …) past urllib once the answer+# has begun; the socket's errors are OSError, inside a URLError while the request is still sent.+_NO_ANSWER = (OSError, http.client.HTTPException)+++def _no_answer(exc: BaseException) -> TransportError:+ said = exc.reason if isinstance(exc, urllib.error.URLError) else exc+ # An http.client exception carries what the server sent (BadStatusLine its line, CRLF included);+ # its repr names the kind and keeps that on one line. RemoteDisconnected is a reset too: its words.+ if isinstance(said, http.client.HTTPException) and not isinstance(said, OSError):+ return TransportError(repr(said)[:200])+ return TransportError((str(said) or type(said).__name__)[:200])++def http_json(method: str, url: str, token: str, body: dict | None = None, *, timeout: float = HTTP_TIMEOUT_SECONDS) -> tuple[int, Any]:data = None if body is None else json.dumps(body).encode("utf-8")request = urllib.request.Request(url, data=data, method=method)@@ -155,13 +174,16 @@ def http_json(method: str, url: str, token: str, body: dict | None = None, *, tiraw = response.read().decode("utf-8")return response.status, (json.loads(raw) if raw else {})except (json.JSONDecodeError, UnicodeDecodeError) as exc:- raise SystemExit("xai: response was not valid JSON") from exc+ raise TransportError("response was not valid JSON", NOT_JSON) from excexcept urllib.error.HTTPError as exc:- raw = exc.read().decode("utf-8", "replace")+ # A refusal's status alone is not an answer read whole: its body broken on the way is no answer too.+ try:+ raw = exc.read().decode("utf-8", "replace")+ except _NO_ANSWER as cut:+ raise _no_answer(cut) from cuttry:return exc.code, json.loads(raw)except json.JSONDecodeError:return exc.code, refusal.RawBody(raw)- except (urllib.error.URLError, TimeoutError, OSError) as exc:- said = exc.reason if isinstance(exc, urllib.error.URLError) else exc- raise TransportError((str(said) or type(said).__name__)[:200]) from exc+ except _NO_ANSWER as exc:+ raise _no_answer(exc) from exc