Skip to content

Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.

fix(video): a clip that is done and whose download fails is downloaded again, then unfinished on its request_id; bytes that are not an mp4 are unfinished, not refused

Soohong Kim7b8e3bd0da50
diff --git a/CHANGELOG.md b/CHANGELOG.md
index eb4e7b3..c3e6332 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,10 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the
- **`tests/gen/test_video.py`** asks again by id and loses polls: `xai.http_json` raises `xai.TransportError` with the old line for a DNS failure, a timeout and a reset; lost, pending, refused, lost, done is asked on the one request with waits 8, 4, 8, 16 and published with `polls_unanswered` 2; lost until a 30 s timeout waits 8, 16, 6 and ends unfinished with `last_poll.error`; a lost POST is sent once and is not unfinished; a 401 or 403 poll, on the login or on the key, ends unfinished at once naming its fix; `video-resume` polls with no POST and announces no charge, ends unfinished on the same id when refused, lost or pending past the timeout or when its credential is rejected, ends with `(HTTP 404)` for an unknown id, refuses eight ids that cannot be one before any call, and its CLI writes the resume report, the unfinished report and an empty stdout; the unified CLI routes it under `gen`.
- **Fix: a poll whose answer is cut short, garbled or not JSON is asked again on the same request — the run no longer ends with no `request_id`.** A poll answer that broke after it began — a body that closed before its `Content-Length` (`http.client.IncompleteRead`), a status line that is not one (`BadStatusLine`), a refusal's body cut the same way — escaped `video`, `video-extend`, `video-edit` and `video-resume` as a raw exception, with no `request_id`, no unfinished report and no resume command, so a caller could only submit the clip again and buy it twice; a `200` poll whose body is not JSON ended the run on `xai: response was not valid JSON`, also with no id. Both are now a poll that got no answer it could read, as a lost one is: asked again on the same `request_id` with the same backoff and stderr note (`got no answer (IncompleteRead(10 bytes read, 190 more expected))`, `got no answer (response was not valid JSON)`), counted in `polls_unanswered`, and unfinished at the poll timeout with that reason in `last_poll.error`. A POST so answered still ends at once and is never sent again — no `request_id` could be read: on the transport's line for a broken answer (a raw exception before), on the not-JSON line it always had. `xai.http_json` raises `xai.TransportError` for every answer it cannot read whole, each with the line it always had; `reason` is the `http.client` exception's repr for a broken answer, `response was not valid JSON` for the other.
- **`tests/gen/test_video.py`** answers over a real socket from a local server: `xai.http_json` raises `xai.TransportError` for a body closed 10 bytes into its 200, a 503 body cut the same way, a garbage status line (the transport's line) and a whole non-JSON `200` (the not-JSON line); cut, garbled, not JSON, cut refusal, pending, done is asked on the one request with waits 8, 16, 32, 60, 4 and published with `polls_unanswered` 4; a resumed poll cut short is asked again and published; cut until a 30 s timeout, the CLI leaves the unfinished report with `last_poll.error` the `IncompleteRead` and an empty stdout; a cut POST is sent once and is not unfinished.
+- **Fix: a clip that is done and whose download fails leaves the run unfinished on its `request_id` — never a second purchase.** Once the poll said done the clip is made and may be billed, yet a failed download ended `video`, `video-extend`, `video-edit` and `video-resume` with no `request_id`, no unfinished report and no resume command: a refused connection on `video: cannot download the clip: …`, a body cut short of its `Content-Length` (`http.client.IncompleteRead`) or a read that timed out as a raw exception, a refused download on `video: download failed with HTTP <n>`, and bytes that are not an mp4 on `… refusing to publish` — so a caller could only submit the clip again and buy it twice. A download that gets no answer it can read, or that the clip's host refuses with `429`, `500`, `502`, `503` or `504`, is now downloaded again after 8 s and then 16 s, three downloads in all, with a stderr note before each (`download N of generation <id> got no answer (<reason>); downloading again in 8s`, `… was refused (503); …`); one still failing, one refused with another status (a `401` or `403` still asked once more with the bearer token first), or bytes that are not an mp4 end the run unfinished on that id: ``… is done, but its download got no answer (last: <reason>) after 3 attempts; the clip is made and may be billed — ask for request_id=<id> again with `sprite-gen video-resume --request-id <id> --out <out>` …``, no `(HTTP <n>)` on it. `video-resume` polls afresh, so it downloads from the URL the new poll names. `video.http_download` raises `xai.TransportError` (its line still `video: cannot download the clip: <reason>`) for a download that got no answer and `video.DownloadRefused`, a `SystemExit` carrying `status`, for a refused one; `xai.NO_ANSWER` and `xai.no_answer` are the transport's classifier, now shared by both.
+- **Output change: bytes that came whole and are not an mp4 are unfinished, not a refusal.** xAI's one content-policy signal is `respect_moderation: false` on the done poll, read before any download; anything else at the clip's URL — an expired link's page, a proxy's error page served `200` — is a fault in delivering a clip that is made, and the `request_id` is the only way back to it. `video-resume` costs a poll and asks for a fresh URL; a refusal left only a new submission. The same URL brings the same bytes, so the run does not download it again itself (docs/video.md, An unfinished generation). `video.verify_mp4` is gone.
+- **Output change: reports count the clip's downloads.** The success report gains `downloads`, how many downloads the clip took, the last one whole (1 unless one failed). An unfinished report of a clip that could not be downloaded gains `downloads` and `last_download` — `http` the status the clip's host refused it with, else `null`; `error` what the transport said, or `not an mp4 (no ftyp box)` with the `bytes` that came — and its `last_poll` is the done poll; an unfinished report of a poll is unchanged.
+- **`tests/gen/test_video.py`** downloads the clip over a real socket from the local server: a clip body closed 10 bytes into its 76, a read held open past a 1 s timeout and a connection to a port nothing holds are each downloaded three times with waits 8 and 16, and the CLI leaves the unfinished report naming the request with `downloads` 3, `last_download.error` the transport's words and an empty stdout; a cut download, then a `503`, then the whole clip publishes with `downloads` 3; a `503` three times and a `404` once end unfinished with `last_download.http`, no `(HTTP n)`; 34 whole bytes of HTML end unfinished at once with `last_download.bytes`; and a run left unfinished by its downloads is published by `video-resume` on the id its report names, from the fresh URL of a new poll and with no POST. The fake transport's non-mp4 refusal test is that HTML test now. A port held bound and not listening is no refused port on macOS (its SYN is dropped and the connect times out), so the test takes a free port and lets it go.
- **Output change: an `openai` image report carries the response's rate-limit headers — `extra.rate_limit`.** The provider dropped the response headers, so what the account may spend per minute and has left could not be read off real traffic. `extra.rate_limit` now holds every `x-ratelimit-*` header the response carried, names lower-cased and values as sent, and `{}` when none came (docs/gen.md). `openai_provider.http_image` returns the response headers as a third value.
- **`tests/gen/test_openai_provider.py`** answers with headers: the report holds exactly the `x-ratelimit-*` ones among others of any case, and `{}` for a response without any.
diff --git a/docs/video.md b/docs/video.md
index 81488c8..076fe06 100644
--- a/docs/video.md
+++ b/docs/video.md
@@ -165,7 +165,17 @@ What happens, in order:
typically a login that expired — is unfinished at once. No clip is written on any
failure path.
4. Download `video.url`, verify the bytes start with an mp4 `ftyp` box, then move
- the file into `--out` atomically (`.part` staging).
+ the file into `--out` atomically (`.part` staging). The clip is made by now and may
+ be billed, so a download that fails never ends the run without its `request_id`. One
+ that gets no answer it can read (the connection refused or timed out, the body cut
+ short of its `Content-Length`) or that the clip's host refuses with `429`, `500`,
+ `502`, `503` or `504` is downloaded again, after 8 s and then 16 s — three downloads
+ in all, each with its own 120 s read timeout and none bounded by the poll timeout —
+ and a `[video] download N of generation <id> got no answer (<reason>); downloading
+ again in 8s` (or `… was refused (503); …`) line goes to stderr before each. One still
+ failing, one refused with another status (a `401` or `403` is asked once more with
+ the bearer token first), or bytes that are not an mp4 end the run *unfinished*
+ (below).
Reference-to-video tends to lock the opening composition to a referenced scene
image (probe m3: the clip opened on the third reference's framing), so put
@@ -224,8 +234,10 @@ the request — the unfinished report's `auth_source`; refresh an expired login
- The report is a `sprite-gen-video-report` with `mode: "resume"`, the same
`request_id`, `inputs: {}` and `duration_requested: null`; `model` and
`duration_reported` are what the finished poll says.
-- Cut short again — the timeout, or a rejected credential — it is unfinished on the
- same id, `verb: "video-resume"` (below), and can be resumed again.
+- Cut short again — the timeout, a rejected credential, or a download that failed —
+ it is unfinished on the same id, `verb: "video-resume"` (below), and can be resumed
+ again. Each resume polls afresh, so a clip whose download failed is fetched from the
+ URL the new poll names.
- A request xAI does not know (`404`), or one that failed or expired, ends with its
status like any failed poll.
- The clip is checked as an mp4 (`ftyp` box) only: `video-extend`'s length rule needs
@@ -240,22 +252,29 @@ the request — the unfinished report's `auth_source`; refresh an expired login
`video`), `bytes`, `host`, requested/reported duration (`input_duration` for
extend / edit), resolution, aspect ratio, audio flag, `elapsed_seconds`, and
`polls`, `polls_refused` (how many of `polls` xAI answered 429 or 5xx, each asked
-again) and `polls_unanswered` (how many got no answer they could read, each asked
-again). The pre-modes `image` field stays. **Tokens and download URLs are never
+again), `polls_unanswered` (how many got no answer they could read, each asked
+again) and `downloads` (how many downloads the clip took, the last one whole: 1 unless
+one failed). The pre-modes `image` field stays. **Tokens and download URLs are never
printed or written** — only the download host (`vidgen.x.ai`).
### An unfinished generation
xAI accepted the request and the run ended before it did — the poll timeout came with
-its polls still pending, refused or unanswered, or xAI rejected the credential on a
-poll. It may finish and be billed, so the run ends naming the request and the
-`video-resume` command that asks for it again, never as a refusal to send again:
+its polls still pending, refused or unanswered, xAI rejected the credential on a
+poll, or the poll said done and the clip could not be downloaded (step 4). It may
+finish and be billed — a clip that is done is made — so the run ends naming the
+request and the `video-resume` command that asks for it again, never as a refusal to
+send again:
```
video: generation <id> was accepted, but its polls were refused (last answer 429) until the poll timeout (600s, 14 polls); it may still finish and be billed — ask for request_id=<id> again with `sprite-gen video-resume --request-id <id> --out /abs/clip.mp4` instead of submitting a new request; nothing was written
video: generation <id> was accepted, but its polls got no answer (last: timed out) until the poll timeout (600s, 14 polls); it may still finish and be billed — ask for request_id=<id> again with `…` instead of submitting a new request; nothing was written
video: generation <id> still 'pending' after 151 polls — poll timeout; it may still finish and be billed — ask for request_id=<id> again with `…` instead of submitting a new request; nothing was written
video: generation <id> was accepted, but xAI rejected the credential (grok-login) on poll 30 (answer 401); it may still finish and be billed — refresh the login with `grok models` run from an empty directory (e.g. `cd "$(mktemp -d)"`) or `grok login`, then ask for request_id=<id> again with `…` instead of submitting a new request; nothing was written
+video: generation <id> is done, but its download got no answer (last: IncompleteRead(1048576 bytes read, 3195904 more expected)) after 3 attempts; the clip is made and may be billed — ask for request_id=<id> again with `…` instead of submitting a new request; nothing was written
+video: generation <id> is done, but its download was refused (last answer 503) after 3 attempts; the clip is made and may be billed — ask for request_id=<id> again with `…` instead of submitting a new request; nothing was written
+video: generation <id> is done, but its download was refused (answer 404); the clip is made and may be billed — ask for request_id=<id> again with `…` instead of submitting a new request; nothing was written
+video: generation <id> is done, but its download brought 34 bytes that are not an mp4 (no ftyp box); the clip is made and may be billed — ask for request_id=<id> again with `…` instead of submitting a new request; nothing was written
```
The line carries no `(HTTP <n>)`, the status of a refusal line (docs/gen.md): a caller
@@ -273,9 +292,39 @@ same reason. With `--report` the run leaves a
`last_poll` gains `"error"` (what the transport said: `timed out`,
`IncompleteRead(10 bytes read, 190 more expected)`, `response was not valid JSON`, …)
-when the last poll got no answer it could read, its `http` then `null`. `auth_source`
-is the credential that submitted it — the one that can ask for it again. A report
-already at that path is replaced, as on success.
+when the last poll got no answer it could read, its `http` then `null`. A run whose
+clip could not be downloaded adds `downloads` (how many were tried) and
+`last_download`, how the last one ended: `http` the status the clip's host refused it
+with, else `null`; `error` what the transport said when it got no answer it could read,
+or `not an mp4 (no ftyp box)` with the `bytes` that came; its `last_poll` is the done
+poll (`{"http": 200, "status": "done"}`):
+
+```json
+{"kind": "sprite-gen-video-unfinished", "provider": "grok-imagine", "verb": "video",
+ "auth_source": "grok-login", "model": "grok-imagine-video-1.5", "request_id": "<id>",
+ "out": "/abs/clip.mp4", "polls": 31, "polls_refused": 0, "polls_unanswered": 0,
+ "last_poll": {"http": 200, "status": "done"}, "downloads": 3,
+ "last_download": {"http": null, "error": "IncompleteRead(1048576 bytes read, 3195904 more expected)"},
+ "elapsed_seconds": 144.0}
+```
+
+Bytes that came whole and are not an mp4 make the run unfinished, not refused, and
+are not downloaded again:
+
+- They are no verdict of xAI's on the request. Its one content-policy signal is
+ `respect_moderation: false` on the done poll, read before any download and refused
+ as a content-policy block. Anything else at the clip's URL — an expired link's page,
+ a proxy's or captive portal's error page served `200` — is a fault in delivering a
+ clip that is made.
+- The `request_id` is the only way back to a clip that may be billed. A refusal would
+ leave one path, a new submission, which buys it again; `video-resume` costs a poll
+ and polls afresh, so it downloads from the URL the new poll names. If those bytes
+ are not an mp4 either, it is unfinished again — never a second purchase.
+- The same URL would bring the same bytes, so the run does not download it again
+ itself.
+
+`auth_source` is the credential that submitted it — the one that can ask for it again.
+A report already at that path is replaced, as on success.
Every verb that polls (`video`, `video-extend`, `video-edit`, `video-resume`) does
the same, `verb` naming it; a `video-resume` report's `model` is `null`.
diff --git a/sprite_gen/gen/video.py b/sprite_gen/gen/video.py
index ac5fd43..19e1435 100644
--- a/sprite_gen/gen/video.py
+++ b/sprite_gen/gen/video.py
@@ -66,7 +66,7 @@ from sprite_gen.video import facing as facing_mod
from .xai import (
API_BASE, AUTH_ENV, AUTH_SOURCE_API_KEY, AUTH_SOURCE_GROK_LOGIN,
HTTP_TIMEOUT_SECONDS, GROK_REFRESH_COMMAND, GROK_REFRESH_WHERE, GROK_LOGIN_COMMAND,
- Credential, HttpCall, TransportError, grok_home, resolve_credential, http_json,
+ NO_ANSWER, Credential, HttpCall, TransportError, grok_home, resolve_credential, http_json, no_answer,
)
DEFAULT_MODEL = "grok-imagine-video-1.5"
@@ -109,6 +109,13 @@ POLL_TIMEOUT_SECONDS = int(os.environ.get("SPRITE_GEN_VIDEO_TIMEOUT_SECONDS", "6
# which bounds the whole poll however its polls are answered.
POLL_RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})
POLL_BACKOFF_MAX_SECONDS = 60.0
+# A clip the poll says is done is made, and may be billed: its request_id is the one way back to it. A
+# download that gets no answer it can read, or that the clip's host refuses with one of
+# POLL_RETRY_STATUSES, is downloaded again after the polls' doubling wait (8 s, then 16 s), up to
+# DOWNLOAD_ATTEMPTS downloads in all. One still failing, one refused with another status, or bytes
+# that are not an mp4, end the run unfinished on that request_id: `video-resume` polls it again for
+# a fresh URL, where a new submission would buy the clip twice.
+DOWNLOAD_ATTEMPTS = 3
# Every mp4/mov starts with a size-prefixed `ftyp` box: bytes 4..8 spell it.
MP4_FTYP_OFFSET = 4
MP4_FTYP = b"ftyp"
@@ -118,7 +125,7 @@ _FAILED_STATUSES = ("failed", "error", "expired")
class GenerationUnfinished(SystemExit):
"""xAI accepted the request (it may bill it) and the run ended before it did: the poll timeout
- came, or xAI rejected the credential on a poll.
+ came, xAI rejected the credential on a poll, or the clip it made could not be downloaded.
The line names `request_id=<id>` and the `video-resume` command that asks for it again, and
carries no `(HTTP n)`, a refusal line's status: a caller that reads `(HTTP 429)` there sends the
@@ -127,7 +134,8 @@ class GenerationUnfinished(SystemExit):
def __init__(self, message: str, *, verb: str, request_id: str, auth_source: str, model: Any, out: Path,
polls: int, polls_refused: int, polls_unanswered: int, last_http: int | None, last_status: Any,
- last_error: str | None, elapsed_seconds: float) -> None:
+ last_error: str | None, elapsed_seconds: float, downloads: int = 0,
+ last_download: dict[str, Any] | None = None) -> None:
super().__init__(message)
self.verb = verb
self.request_id = request_id
@@ -141,6 +149,8 @@ class GenerationUnfinished(SystemExit):
self.last_status = last_status
self.last_error = last_error # what the transport said when the last poll got no answer it could read
self.elapsed_seconds = elapsed_seconds
+ self.downloads = downloads # downloads of a clip the poll said is done; 0 when none was
+ self.last_download = last_download # how the last of them failed
def to_dict(self) -> dict[str, Any]:
return {
@@ -156,26 +166,42 @@ class GenerationUnfinished(SystemExit):
"polls_unanswered": self.polls_unanswered,
"last_poll": {"http": self.last_http, "status": self.last_status,
**({"error": self.last_error} if self.last_error is not None else {})},
+ **({"downloads": self.downloads, "last_download": self.last_download} if self.downloads else {}),
"elapsed_seconds": round(self.elapsed_seconds, 3),
}
+class DownloadRefused(SystemExit):
+ """The clip's host answered its download with an error `status` (a 401 or 403 once more with the bearer token)."""
+
+ def __init__(self, status: int, line: str) -> None:
+ super().__init__(line)
+ self.status = status
+
+
def http_download(url: str, token: str) -> bytes:
+ """The clip's bytes, read whole. An error status raises DownloadRefused; a download that gets no answer
+ it can read whole — the connection failed or timed out, the body broke on the way — raises
+ xai.TransportError, its words in `reason`."""
request = urllib.request.Request(url)
try:
- with urllib.request.urlopen(request, timeout=HTTP_TIMEOUT_SECONDS) as response:
- return response.read()
- except urllib.error.HTTPError as first:
- if first.code not in (401, 403):
- raise SystemExit(f"video: download failed with HTTP {first.code}") from first
- request.add_header("Authorization", f"Bearer {token}")
try:
with urllib.request.urlopen(request, timeout=HTTP_TIMEOUT_SECONDS) as response:
return response.read()
- except urllib.error.HTTPError as second:
- raise SystemExit(f"video: download failed with HTTP {second.code} even with the bearer token") from second
- except urllib.error.URLError as exc:
- raise SystemExit(f"video: cannot download the clip: {exc.reason}") from exc
+ except urllib.error.HTTPError as first:
+ if first.code not in (401, 403):
+ raise DownloadRefused(first.code, f"video: download failed with HTTP {first.code}") from first
+ request.add_header("Authorization", f"Bearer {token}")
+ try:
+ with urllib.request.urlopen(request, timeout=HTTP_TIMEOUT_SECONDS) as response:
+ return response.read()
+ except urllib.error.HTTPError as second:
+ raise DownloadRefused(
+ second.code, f"video: download failed with HTTP {second.code} even with the bearer token"
+ ) from second
+ except NO_ANSWER as exc:
+ reason = no_answer(exc).reason
+ raise TransportError(reason, f"video: cannot download the clip: {reason}") from exc
def _data_url(path: Path) -> str:
@@ -202,11 +228,8 @@ def _redact_host(url: str) -> str | None:
return parts[2] if len(parts) > 2 else None
-def verify_mp4(data: bytes) -> None:
- if len(data) < MP4_FTYP_OFFSET + len(MP4_FTYP) or data[MP4_FTYP_OFFSET : MP4_FTYP_OFFSET + len(MP4_FTYP)] != MP4_FTYP:
- raise SystemExit(
- f"video: downloaded {len(data)} bytes but they are not an mp4 (no ftyp box) — refusing to publish"
- )
+def _is_mp4(data: bytes) -> bool:
+ return data[MP4_FTYP_OFFSET : MP4_FTYP_OFFSET + len(MP4_FTYP)] == MP4_FTYP
def _resolved(path: Path | None) -> Path | None:
@@ -243,7 +266,7 @@ def _require_mp4_clip(clip: Path, *, verb: str) -> None:
raise SystemExit(f"{verb}: input clip not found: {clip}")
with clip.open("rb") as handle:
head = handle.read(MP4_FTYP_OFFSET + len(MP4_FTYP))
- if len(head) < MP4_FTYP_OFFSET + len(MP4_FTYP) or head[MP4_FTYP_OFFSET:] != MP4_FTYP:
+ if not _is_mp4(head):
raise SystemExit(f"{verb}: {clip} is not an mp4 (no ftyp box); refusing to upload it")
@@ -318,6 +341,7 @@ class VideoResult:
duration_reported: float | None = None
polls_refused: int = 0 # polls xAI answered 429 or 5xx, each asked again
polls_unanswered: int = 0 # polls that got no answer they could read (xai.TransportError), each asked again
+ downloads: int = 1 # downloads of the clip, the last one whole; more than one when one failed
host: str | None = None
image: Path | None = None
prompt: str = ""
@@ -356,6 +380,7 @@ class VideoResult:
"polls": self.polls,
"polls_refused": self.polls_refused,
"polls_unanswered": self.polls_unanswered,
+ "downloads": self.downloads,
**({"extra": self.extra} if self.extra else {}),
**({"facing": self.facing} if self.facing is not None else {}),
}
@@ -442,6 +467,7 @@ class _Published:
polls: int
polls_refused: int
polls_unanswered: int
+ downloads: int
elapsed_seconds: float
@@ -524,21 +550,23 @@ def _poll_publish(
staged file is removed and `out` is never written.
The poll timeout counts from here. A run that ends before the request does — the timeout
- came, or xAI rejected the credential — raises GenerationUnfinished naming the request."""
+ came, xAI rejected the credential, or the clip it made could not be downloaded (DOWNLOAD_ATTEMPTS)
+ — raises GenerationUnfinished naming the request."""
timeout = POLL_TIMEOUT_SECONDS if poll_timeout is None else poll_timeout
deadline = time.monotonic() + timeout
- polls = polls_refused = polls_unanswered = failed_in_a_row = 0
+ polls = polls_refused = polls_unanswered = failed_in_a_row = downloads = 0
video_url: str | None = None
duration_reported: float | None = None
model_reported: str | None = None
final: dict[str, Any] = {}
- def unfinished(line: str, http: int | None, poll_status: Any, lost: str | None) -> GenerationUnfinished:
+ def unfinished(line: str, http: int | None, poll_status: Any, lost: str | None,
+ last_download: dict[str, Any] | None = None) -> GenerationUnfinished:
return GenerationUnfinished(
f"{verb}: generation {request_id} {line}", verb=verb, request_id=request_id,
auth_source=credential.source, model=model, out=out, polls=polls, polls_refused=polls_refused,
polls_unanswered=polls_unanswered, last_http=http, last_status=poll_status, last_error=lost,
- elapsed_seconds=time.monotonic() - started,
+ elapsed_seconds=time.monotonic() - started, downloads=downloads, last_download=last_download,
)
while True:
@@ -612,8 +640,42 @@ def _poll_publish(
if not video_url:
raise SystemExit(f"{verb}: generation {request_id} is done but reported no video url")
- data = download(video_url, credential.token)
- verify_mp4(data)
+ # The clip is made: whatever stops its download, the run ends naming the request, never a second purchase.
+ while True:
+ downloads += 1
+ said: str | None = None # what the transport said when the download got no answer it could read
+ refused: int | None = None # the status the clip's host refused it with
+ try:
+ data = download(video_url, credential.token)
+ except TransportError as failed:
+ said = failed.reason
+ except DownloadRefused as failed:
+ refused = failed.status
+ else:
+ if _is_mp4(data):
+ break
+ # Bytes read whole that are not an mp4 are no verdict of xAI's on the request (that is
+ # respect_moderation, above) and the same URL brings the same bytes: video-resume asks for a fresh one.
+ raise unfinished(f"is done, but its download brought {len(data)} bytes that are not an mp4 (no ftyp box); "
+ f"the clip is made and may be billed — {_ask_again(request_id, out)}", http, poll_status, None,
+ {"http": None, "error": "not an mp4 (no ftyp box)", "bytes": len(data)})
+ again = said is not None or refused in POLL_RETRY_STATUSES
+ if not again or downloads == DOWNLOAD_ATTEMPTS:
+ if said is not None:
+ failure = f"got no answer (last: {said}) after {downloads} attempts"
+ elif again:
+ failure = f"was refused (last answer {refused}) after {downloads} attempts"
+ else:
+ failure = f"was refused (answer {refused})"
+ raise unfinished(f"is done, but its download {failure}; the clip is made and may be billed — "
+ f"{_ask_again(request_id, out)}", http, poll_status, None,
+ {"http": refused, **({"error": said} if said is not None else {})})
+ wait = POLL_INTERVAL_SECONDS * 2 ** downloads
+ # The status in plain digits, as the poll notes say it.
+ failure = f"got no answer ({said})" if said is not None else f"was refused ({refused})"
+ print(f"[{verb}] download {downloads} of generation {request_id} {failure}; downloading again in {wait:g}s",
+ file=sys.stderr)
+ sleep(wait)
elapsed = time.monotonic() - started
out.parent.mkdir(parents=True, exist_ok=True)
@@ -629,7 +691,7 @@ def _poll_publish(
return _Published(
data=data, request_id=request_id, model=model_reported, duration_reported=duration_reported,
host=_redact_host(video_url), polls=polls, polls_refused=polls_refused, polls_unanswered=polls_unanswered,
- elapsed_seconds=elapsed,
+ downloads=downloads, elapsed_seconds=elapsed,
)
@@ -695,6 +757,7 @@ def generate_video(
polls=published.polls,
polls_refused=published.polls_refused,
polls_unanswered=published.polls_unanswered,
+ downloads=published.downloads,
duration_requested=request.duration,
duration_reported=published.duration_reported,
host=published.host,
@@ -776,7 +839,7 @@ def extend_video(
out=out, bytes=len(published.data), model=published.model or request.model, request_id=published.request_id,
auth_source=credential.source, elapsed_seconds=published.elapsed_seconds, polls=published.polls,
polls_refused=published.polls_refused, polls_unanswered=published.polls_unanswered,
- duration_requested=request.duration, duration_reported=published.duration_reported, host=published.host,
+ downloads=published.downloads, duration_requested=request.duration, duration_reported=published.duration_reported, host=published.host,
prompt=request.prompt, mode=MODE_EXTEND, inputs={"video": str(clip)}, input_duration=input_seconds,
output_duration=measured.get("output"),
)
@@ -837,7 +900,7 @@ def edit_video(
out=out, bytes=len(published.data), model=published.model or request.model, request_id=published.request_id,
auth_source=credential.source, elapsed_seconds=published.elapsed_seconds, polls=published.polls,
polls_refused=published.polls_refused, polls_unanswered=published.polls_unanswered,
- duration_requested=0, duration_reported=published.duration_reported, host=published.host,
+ downloads=published.downloads, duration_requested=0, duration_reported=published.duration_reported, host=published.host,
prompt=request.prompt, mode=MODE_EDIT, inputs={"video": str(clip)}, input_duration=input_seconds,
output_duration=measured.get("output"),
)
@@ -876,7 +939,7 @@ def resume_video(
out=out, bytes=len(published.data), model=published.model, request_id=published.request_id,
auth_source=credential.source, elapsed_seconds=published.elapsed_seconds, polls=published.polls,
polls_refused=published.polls_refused, polls_unanswered=published.polls_unanswered,
- duration_requested=None, duration_reported=published.duration_reported, host=published.host, mode=MODE_RESUME,
+ downloads=published.downloads, duration_requested=None, duration_reported=published.duration_reported, host=published.host, mode=MODE_RESUME,
)
diff --git a/sprite_gen/gen/xai.py b/sprite_gen/gen/xai.py
index f8b87c4..010625d 100644
--- a/sprite_gen/gen/xai.py
+++ b/sprite_gen/gen/xai.py
@@ -140,7 +140,7 @@ class TransportError(SystemExit):
after it began (a body shorter than its Content-Length, a status line that is not one), or it came
whole and was not JSON — and the server may have taken the request. Still the SystemExit and line
it always was (`NOT_JSON` for the last); a caller that may safely ask again (a poll of an accepted
- request) catches this type. `reason` is what the transport said."""
+ request, the download of a clip that is done) catches this type. `reason` is what the transport said."""
def __init__(self, reason: str, line: str = TRANSPORT_FAILED) -> None:
super().__init__(line)
@@ -150,10 +150,10 @@ class TransportError(SystemExit):
# Everything that ends a request before its answer is read whole. http.client raises its own
# HTTPException family (IncompleteRead, BadStatusLine, LineTooLong, …) past urllib once the answer
# has begun; the socket's errors are OSError, inside a URLError while the request is still sent.
-_NO_ANSWER = (OSError, http.client.HTTPException)
+NO_ANSWER = (OSError, http.client.HTTPException)
-def _no_answer(exc: BaseException) -> TransportError:
+def no_answer(exc: BaseException) -> TransportError:
said = exc.reason if isinstance(exc, urllib.error.URLError) else exc
# An http.client exception carries what the server sent (BadStatusLine its line, CRLF included);
# its repr names the kind and keeps that on one line. RemoteDisconnected is a reset too: its words.
@@ -179,11 +179,11 @@ def http_json(method: str, url: str, token: str, body: dict | None = None, *, ti
# A refusal's status alone is not an answer read whole: its body broken on the way is no answer too.
try:
raw = exc.read().decode("utf-8", "replace")
- except _NO_ANSWER as cut:
- raise _no_answer(cut) from cut
+ except NO_ANSWER as cut:
+ raise no_answer(cut) from cut
try:
return exc.code, json.loads(raw)
except json.JSONDecodeError:
return exc.code, refusal.RawBody(raw)
- except _NO_ANSWER as exc:
- raise _no_answer(exc) from exc
+ except NO_ANSWER as exc:
+ raise no_answer(exc) from exc
diff --git a/tests/gen/test_video.py b/tests/gen/test_video.py
index 991d945..bb4226c 100644
--- a/tests/gen/test_video.py
+++ b/tests/gen/test_video.py
@@ -1110,8 +1110,7 @@ def test_a_submission_cut_short_is_not_unfinished_and_is_never_sent_again(tmp_pa
# that timed out) or that the clip's host refuses with 429 or 5xx is downloaded again, up to three
# downloads; one still failing — or refused with another status, or bytes that are not an mp4 —
# leaves the run unfinished on that request_id, never an exception that names no request, and
-# `video-resume` polls it again for a fresh URL. Before, a refused connection ended on
-# `video: cannot download the clip: …` and a cut or timed-out read escaped raw.
+# `video-resume` polls it again for a fresh URL.
CLIP = _answer("200 OK", MP4, content_type="video/mp4")
CLIP_CUT = _answer("200 OK", MP4[:10], length=len(MP4), content_type="video/mp4") # closes 10 bytes in