Skip to content

Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.

fix(video): a poll answer cut short, garbled or not JSON is asked again on the same id; a cut POST ends on the transport's line

Soohong Kimd63f474a3ba3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c4a18a2..eb4e7b3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,8 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the
- **Fix: a poll that gets no answer, or whose credential xAI rejects, leaves the request's id.** After xAI had accepted the request, a poll whose connection failed or timed out ended the run with the transport's line and no `request_id`, and a poll answered `401` ended it with its status. A poll that gets no answer is a GET, which buys nothing, so it is now asked again with the refused polls' backoff (a refused poll and a lost one are two failures in a row) and a stderr note, `poll N of generation <id> got no answer (<reason>); asking again in 8s`; lost until the poll timeout, the run is unfinished. A poll answered `401` or `403` — the credential stopped working after the POST, typically an expired login — ends the run unfinished at once, naming the credential's fix (refresh the grok login, or check `XAI_API_KEY`) before the request to ask for. A POST that gets no answer ends as before: no `request_id` came back. `xai.http_json` raises `xai.TransportError` for a request that got no answer, still a `SystemExit` with the same line, the socket's words in `reason`.
- **Output change: the unfinished line names the command that asks again; reports count the polls that got no answer.** The line reads ``ask for request_id=<id> again with `sprite-gen video-resume --request-id <id> --out <out>` instead of submitting a new request``, `request_id=<id>` kept as it was. The success and unfinished reports gain `polls_unanswered`, how many of `polls` got no answer, each asked again; the unfinished report's `last_poll` gains `error`, what the socket said, when its last poll got none (`http` then `null`).
- **`tests/gen/test_video.py`** asks again by id and loses polls: `xai.http_json` raises `xai.TransportError` with the old line for a DNS failure, a timeout and a reset; lost, pending, refused, lost, done is asked on the one request with waits 8, 4, 8, 16 and published with `polls_unanswered` 2; lost until a 30 s timeout waits 8, 16, 6 and ends unfinished with `last_poll.error`; a lost POST is sent once and is not unfinished; a 401 or 403 poll, on the login or on the key, ends unfinished at once naming its fix; `video-resume` polls with no POST and announces no charge, ends unfinished on the same id when refused, lost or pending past the timeout or when its credential is rejected, ends with `(HTTP 404)` for an unknown id, refuses eight ids that cannot be one before any call, and its CLI writes the resume report, the unfinished report and an empty stdout; the unified CLI routes it under `gen`.
+- **Fix: a poll whose answer is cut short, garbled or not JSON is asked again on the same request — the run no longer ends with no `request_id`.** A poll answer that broke after it began — a body that closed before its `Content-Length` (`http.client.IncompleteRead`), a status line that is not one (`BadStatusLine`), a refusal's body cut the same way — escaped `video`, `video-extend`, `video-edit` and `video-resume` as a raw exception, with no `request_id`, no unfinished report and no resume command, so a caller could only submit the clip again and buy it twice; a `200` poll whose body is not JSON ended the run on `xai: response was not valid JSON`, also with no id. Both are now a poll that got no answer it could read, as a lost one is: asked again on the same `request_id` with the same backoff and stderr note (`got no answer (IncompleteRead(10 bytes read, 190 more expected))`, `got no answer (response was not valid JSON)`), counted in `polls_unanswered`, and unfinished at the poll timeout with that reason in `last_poll.error`. A POST so answered still ends at once and is never sent again — no `request_id` could be read: on the transport's line for a broken answer (a raw exception before), on the not-JSON line it always had. `xai.http_json` raises `xai.TransportError` for every answer it cannot read whole, each with the line it always had; `reason` is the `http.client` exception's repr for a broken answer, `response was not valid JSON` for the other.
+- **`tests/gen/test_video.py`** answers over a real socket from a local server: `xai.http_json` raises `xai.TransportError` for a body closed 10 bytes into its 200, a 503 body cut the same way, a garbage status line (the transport's line) and a whole non-JSON `200` (the not-JSON line); cut, garbled, not JSON, cut refusal, pending, done is asked on the one request with waits 8, 16, 32, 60, 4 and published with `polls_unanswered` 4; a resumed poll cut short is asked again and published; cut until a 30 s timeout, the CLI leaves the unfinished report with `last_poll.error` the `IncompleteRead` and an empty stdout; a cut POST is sent once and is not unfinished.
- **Output change: an `openai` image report carries the response's rate-limit headers — `extra.rate_limit`.** The provider dropped the response headers, so what the account may spend per minute and has left could not be read off real traffic. `extra.rate_limit` now holds every `x-ratelimit-*` header the response carried, names lower-cased and values as sent, and `{}` when none came (docs/gen.md). `openai_provider.http_image` returns the response headers as a third value.
- **`tests/gen/test_openai_provider.py`** answers with headers: the report holds exactly the `x-ratelimit-*` ones among others of any case, and `{}` for a response without any.
diff --git a/docs/video.md b/docs/video.md
index 86d915e..81488c8 100644
--- a/docs/video.md
+++ b/docs/video.md
@@ -148,18 +148,22 @@ What happens, in order:
API's own error text.
3. `GET /v1/videos/{request_id}` every 4 s until `status` is `done`. `failed`,
`expired` or a non-2xx poll fails loudly — except a poll answered `429` or `500`,
- `502`, `503`, `504`, or one that gets no answer at all (the connection failed or
- timed out): the request was accepted, so that poll is asked again on the same
- `request_id` (a new POST would buy the clip twice; a GET buys nothing), after a wait
- that doubles from 8 s up to 60 s while the polls fail and goes back to 4 s on an
- answer. A `[video] poll N of generation <id> was refused (429); asking again in 8s`
- line — or `… got no answer (timed out); asking again in 8s` — goes to stderr each
- time. The poll timeout (`SPRITE_GEN_VIDEO_TIMEOUT_SECONDS`, default 600, counted
- from the POST's answer) bounds the whole poll: no wait runs past it, one last poll is
- asked at it, and a generation still pending, refused or unanswered then is
- *unfinished* (below). A poll answered `401` or `403` — the credential stopped
- working after the POST, typically a login that expired — is unfinished at once. No
- clip is written on any failure path.
+ `502`, `503`, `504`, or one that gets no answer it can read (the connection failed
+ or timed out; the answer broke on the way — a body shorter than its
+ `Content-Length`, a status line that is not one, a refusal's body cut the same way;
+ or a `200` body that is not JSON): the request was accepted, so that poll is asked
+ again on the same `request_id` (a new POST would buy the clip twice; a GET buys
+ nothing), after a wait that doubles from 8 s up to 60 s while the polls fail and goes
+ back to 4 s on an answer. A `[video] poll N of generation <id> was refused (429);
+ asking again in 8s` line — or `… got no answer (timed out); asking again in 8s`,
+ the reason `IncompleteRead(10 bytes read, 190 more expected)` or `response was not
+ valid JSON` for an answer it could not read — goes to stderr each time. The poll
+ timeout (`SPRITE_GEN_VIDEO_TIMEOUT_SECONDS`, default 600, counted from the POST's
+ answer) bounds the whole poll: no wait runs past it, one last poll is asked at it,
+ and a generation still pending, refused or unanswered then is *unfinished* (below).
+ A poll answered `401` or `403` — the credential stopped working after the POST,
+ typically a login that expired — is unfinished at once. No clip is written on any
+ failure path.
4. Download `video.url`, verify the bytes start with an mp4 `ftyp` box, then move
the file into `--out` atomically (`.part` staging).
@@ -236,8 +240,8 @@ the request — the unfinished report's `auth_source`; refresh an expired login
`video`), `bytes`, `host`, requested/reported duration (`input_duration` for
extend / edit), resolution, aspect ratio, audio flag, `elapsed_seconds`, and
`polls`, `polls_refused` (how many of `polls` xAI answered 429 or 5xx, each asked
-again) and `polls_unanswered` (how many got no answer, each asked again). The
-pre-modes `image` field stays. **Tokens and download URLs are never
+again) and `polls_unanswered` (how many got no answer they could read, each asked
+again). The pre-modes `image` field stays. **Tokens and download URLs are never
printed or written** — only the download host (`vidgen.x.ai`).
### An unfinished generation
@@ -267,16 +271,20 @@ same reason. With `--report` the run leaves a
"last_poll": {"http": 429, "status": null}, "elapsed_seconds": 600.0}
```
-`last_poll` gains `"error"` (what the socket said) when the last poll got no answer,
-its `http` then `null`. `auth_source` is the credential that submitted it — the one
-that can ask for it again. A report already at that path is replaced, as on success.
+`last_poll` gains `"error"` (what the transport said: `timed out`,
+`IncompleteRead(10 bytes read, 190 more expected)`, `response was not valid JSON`, …)
+when the last poll got no answer it could read, its `http` then `null`. `auth_source`
+is the credential that submitted it — the one that can ask for it again. A report
+already at that path is replaced, as on success.
Every verb that polls (`video`, `video-extend`, `video-edit`, `video-resume`) does
the same, `verb` naming it; a `video-resume` report's `model` is `null`.
-A POST that gets no answer is not unfinished: no `request_id` came back, and the line
-is the transport's own (`xai: request failed or timed out; no automatic retry (the
-server may have accepted it)`). Python callers catch `xai.TransportError`, a
-`SystemExit` with that line and the socket's words in `reason`.
+A POST that gets no answer it can read is not unfinished: no `request_id` could be
+read, so it ends at once and is never sent again. Its line is the transport's own
+(`xai: request failed or timed out; no automatic retry (the server may have accepted
+it)`), or `xai: response was not valid JSON` for a whole answer that is not JSON.
+Python callers catch `xai.TransportError`, a `SystemExit` with that line and the
+transport's words in `reason`.
## Quotas and limits
diff --git a/sprite_gen/gen/video.py b/sprite_gen/gen/video.py
index 2329d46..ac5fd43 100644
--- a/sprite_gen/gen/video.py
+++ b/sprite_gen/gen/video.py
@@ -101,8 +101,9 @@ MODE_RESUME = "resume"
REQUEST_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*")
POLL_INTERVAL_SECONDS = 4.0
POLL_TIMEOUT_SECONDS = int(os.environ.get("SPRITE_GEN_VIDEO_TIMEOUT_SECONDS", "600"))
-# A poll xAI throttles (429) or fails on its own side (5xx), or one that gets no answer at all (the
-# connection failed or timed out), is asked again, on the same request: the POST was accepted, so a
+# A poll xAI throttles (429) or fails on its own side (5xx), or one that gets no answer it can read (the
+# connection failed or timed out, the answer broke on the way or was not JSON — xai.TransportError), is
+# asked again, on the same request: the POST was accepted, so a
# new submission would buy the clip twice, and a GET buys nothing. The wait doubles from
# POLL_INTERVAL_SECONDS while the polls fail, up to the ceiling; no wait runs past the poll timeout,
# which bounds the whole poll however its polls are answered.
@@ -138,7 +139,7 @@ class GenerationUnfinished(SystemExit):
self.polls_unanswered = polls_unanswered
self.last_http = last_http
self.last_status = last_status
- self.last_error = last_error # what the socket said when the last poll got no answer
+ self.last_error = last_error # what the transport said when the last poll got no answer it could read
self.elapsed_seconds = elapsed_seconds
def to_dict(self) -> dict[str, Any]:
@@ -316,7 +317,7 @@ class VideoResult:
duration_requested: int | None # video-resume: None, this run requested nothing
duration_reported: float | None = None
polls_refused: int = 0 # polls xAI answered 429 or 5xx, each asked again
- polls_unanswered: int = 0 # polls that got no answer (connection failed or timed out), each asked again
+ polls_unanswered: int = 0 # polls that got no answer they could read (xai.TransportError), each asked again
host: str | None = None
image: Path | None = None
prompt: str = ""
diff --git a/sprite_gen/gen/xai.py b/sprite_gen/gen/xai.py
index a745cba..f8b87c4 100644
--- a/sprite_gen/gen/xai.py
+++ b/sprite_gen/gen/xai.py
@@ -7,6 +7,7 @@ that login. Never spawn an agent or switch sources after a login failure.
"""
from __future__ import annotations
+import http.client
import json
import os
import urllib.error
@@ -131,18 +132,36 @@ def resolve_credential(*, env: dict[str, str] | None = None, now: datetime | Non
HttpCall = Callable[[str, str, str, dict | None], tuple[int, Any]]
TRANSPORT_FAILED = "xai: request failed or timed out; no automatic retry (the server may have accepted it)"
+NOT_JSON = "xai: response was not valid JSON"
class TransportError(SystemExit):
- """No HTTP answer came back: the connection failed or timed out, and the server may have taken the
- request. Still the SystemExit and line it always was; a caller that may safely ask again (a poll of
- an accepted request) catches this type. `reason` is what the socket said."""
-
- def __init__(self, reason: str) -> None:
- super().__init__(TRANSPORT_FAILED)
+ """No answer that could be read came back: the connection failed or timed out, the answer broke
+ after it began (a body shorter than its Content-Length, a status line that is not one), or it came
+ whole and was not JSON — and the server may have taken the request. Still the SystemExit and line
+ it always was (`NOT_JSON` for the last); a caller that may safely ask again (a poll of an accepted
+ request) catches this type. `reason` is what the transport said."""
+
+ def __init__(self, reason: str, line: str = TRANSPORT_FAILED) -> None:
+ super().__init__(line)
self.reason = reason
+# Everything that ends a request before its answer is read whole. http.client raises its own
+# HTTPException family (IncompleteRead, BadStatusLine, LineTooLong, …) past urllib once the answer
+# has begun; the socket's errors are OSError, inside a URLError while the request is still sent.
+_NO_ANSWER = (OSError, http.client.HTTPException)
+
+
+def _no_answer(exc: BaseException) -> TransportError:
+ said = exc.reason if isinstance(exc, urllib.error.URLError) else exc
+ # An http.client exception carries what the server sent (BadStatusLine its line, CRLF included);
+ # its repr names the kind and keeps that on one line. RemoteDisconnected is a reset too: its words.
+ if isinstance(said, http.client.HTTPException) and not isinstance(said, OSError):
+ return TransportError(repr(said)[:200])
+ return TransportError((str(said) or type(said).__name__)[:200])
+
+
def http_json(method: str, url: str, token: str, body: dict | None = None, *, timeout: float = HTTP_TIMEOUT_SECONDS) -> tuple[int, Any]:
data = None if body is None else json.dumps(body).encode("utf-8")
request = urllib.request.Request(url, data=data, method=method)
@@ -155,13 +174,16 @@ def http_json(method: str, url: str, token: str, body: dict | None = None, *, ti
raw = response.read().decode("utf-8")
return response.status, (json.loads(raw) if raw else {})
except (json.JSONDecodeError, UnicodeDecodeError) as exc:
- raise SystemExit("xai: response was not valid JSON") from exc
+ raise TransportError("response was not valid JSON", NOT_JSON) from exc
except urllib.error.HTTPError as exc:
- raw = exc.read().decode("utf-8", "replace")
+ # A refusal's status alone is not an answer read whole: its body broken on the way is no answer too.
+ try:
+ raw = exc.read().decode("utf-8", "replace")
+ except _NO_ANSWER as cut:
+ raise _no_answer(cut) from cut
try:
return exc.code, json.loads(raw)
except json.JSONDecodeError:
return exc.code, refusal.RawBody(raw)
- except (urllib.error.URLError, TimeoutError, OSError) as exc:
- said = exc.reason if isinstance(exc, urllib.error.URLError) else exc
- raise TransportError((str(said) or type(said).__name__)[:200]) from exc
+ except _NO_ANSWER as exc:
+ raise _no_answer(exc) from exc