Skip to content

AI turns documents or topics into real, native PowerPoint decks—with native shapes, transitions and animations, data-backed charts and tables on demand, audio narration from speaker notes, and support for your own .pptx templates. · by Hugo He

feat(image_gen): switch to JSON manifest with adaptive concurrency

hugohe37a998d22c434
diff --git a/.env.example b/.env.example
index 23c69672..aafc71ee 100644
--- a/.env.example
+++ b/.env.example
@@ -21,6 +21,11 @@
# 推荐核心后端:openai / gemini / qwen / zhipu / volcengine
# IMAGE_BACKEND=openai
+# Max concurrent requests in --manifest batch mode (default 3).
+# Auto-halves on rate-limit; 1 is the serial fallback.
+# --manifest 批量模式下的最大并发数(默认 3)。命中限流自动减半,最低 1(=串行)。
+# IMAGE_CONCURRENCY=3
+
# You may also provide the same variables directly via the current process environment.
# 也可以不写 .env,而是直接通过当前运行进程的环境变量提供同样的配置。
diff --git a/skills/ppt-master/SKILL.md b/skills/ppt-master/SKILL.md
index 229efb06..18b14d18 100644
--- a/skills/ppt-master/SKILL.md
+++ b/skills/ppt-master/SKILL.md
@@ -243,7 +243,7 @@ Then **lazy-load the path-specific reference** for each row that actually needs
| `web` | `references/image-searcher.md` | `python3 ${SKILL_DIR}/scripts/image_search.py ...` |
| `user` / `placeholder` | (skip) | (skip) |
-A deck with only `ai` rows never loads `image-searcher.md`; a deck with only `web` rows never loads `image-generator.md`. A mixed deck loads both, processes each row through its own path, and writes both `image_prompts.md` and `image_sources.json`.
+A deck with only `ai` rows never loads `image-searcher.md`; a deck with only `web` rows never loads `image-generator.md`. A mixed deck loads both, processes each row through its own path, and writes both `image_prompts.json` and `image_sources.json`.
Workflow:
@@ -254,7 +254,7 @@ Workflow:
**✅ Checkpoint — Confirm acquisition attempted for every row**:
```markdown
## ✅ Image Acquisition Phase Complete
-- [x] image_prompts.md created (when any ai rows processed)
+- [x] image_prompts.json created (when any ai rows processed)
- [x] image_sources.json created (when any web rows processed)
- [x] Each row: status is `Generated` / `Sourced` / `Needs-Manual` (no `Pending` remaining)
```
@@ -337,7 +337,7 @@ python3 ${SKILL_DIR}/scripts/svg_quality_checker.py <project_path>
🚧 **Image readiness GATE** (when Step 5 left ai rows in `Needs-Manual`): every expected file must exist at `project/images/<filename>` before running 7.1.
-> If files are missing: PAUSE, list the missing filenames, point the user to `images/image_prompts.md` (each `### Image N:` block is paste-ready for ChatGPT / Gemini / Midjourney) and the required placement `project/images/<filename>`. Resume Step 7.1 only after all expected files are in place. `finalize_svg.py` and `svg_to_pptx.py` do not detect missing files at this layer — proceeding with gaps produces a deck with broken image references.
+> If files are missing: PAUSE, list the missing filenames, point the user to `images/image_prompts.json` (the `prompt` field of each `Needs-Manual` item is paste-ready for ChatGPT / Gemini / Midjourney) and the required placement `project/images/<filename>`. Resume Step 7.1 only after all expected files are in place. `finalize_svg.py` and `svg_to_pptx.py` do not detect missing files at this layer — proceeding with gaps produces a deck with broken image references.
> ⚠️ Run the three sub-steps **one at a time** — each must complete successfully before the next.
> ❌ **NEVER** combine them into a single code block or shell invocation.
diff --git a/skills/ppt-master/references/image-base.md b/skills/ppt-master/references/image-base.md
index ee4104c4..4944d703 100644
--- a/skills/ppt-master/references/image-base.md
+++ b/skills/ppt-master/references/image-base.md
@@ -61,7 +61,7 @@ After all rows reach terminal status:
- Every non-skipped row has a file at `project/images/<filename>`, or is marked `Needs-Manual`
- No `Pending` rows remain
-- `image_prompts.md` exists when ≥1 ai row processed
+- `image_prompts.json` exists when ≥1 ai row processed; every entry has `status ∈ {Generated, Failed, Needs-Manual}` (no `Pending` remaining)
- `image_sources.json` exists when ≥1 web row processed; every entry has `license_tier ∈ {no-attribution, attribution-required}`
> `Needs-Manual` is a legitimate terminal state for ai rows — Step 7 entry waits for the user to place the file. See [`image-generator.md`](./image-generator.md) §3.2 Offline Manual Mode.
@@ -75,7 +75,7 @@ After all rows reach terminal status:
1. Try once
2. On recoverable failure (network, no candidates, license rejection, rate limit), retry once with broadened parameters
3. On second failure, set `Status: Needs-Manual`, log the reason in conversation, continue
-4. After the phase completes, summarize all `Needs-Manual` rows for the user — list filenames, where prompts live (`images/image_prompts.md` for ai rows), and where to place generated files (`project/images/<filename>`)
+4. After the phase completes, summarize all `Needs-Manual` rows for the user — list filenames, where prompts live (`images/image_prompts.json` `items[].prompt` for ai rows), and where to place generated files (`project/images/<filename>`)
`Needs-Manual` is also the entry status for **Offline Manual Mode** (no `IMAGE_BACKEND` configured, no host-native image tool in use). Affected ai rows are marked `Needs-Manual` from the start without a failed attempt — see [`image-generator.md`](./image-generator.md) §3.2.
@@ -128,6 +128,6 @@ Executor does NOT invoke `image_gen.py` / `image_search.py`.
## ✅ Image Acquisition Phase Complete
- [x] {N} rows processed (`ai`: {a} / `web`: {b})
- [x] {a} `Generated`, {b} `Sourced`, {c} `Needs-Manual`
-- [x] image_prompts.md / image_sources.json written
+- [x] image_prompts.json / image_sources.json written
- [ ] **Next**: Auto-proceed to Executor phase
```
diff --git a/skills/ppt-master/references/image-generator.md b/skills/ppt-master/references/image-generator.md
index fc2402c6..38540906 100644
--- a/skills/ppt-master/references/image-generator.md
+++ b/skills/ppt-master/references/image-generator.md
@@ -12,24 +12,20 @@ Role definition for the **AI image generation path**: convert each `Acquire Via:
### 1.1 Standard Output Format
-Every image must be emitted into `image_prompts.md` in the following block format:
+Every image becomes one entry under `items[]` in `project/images/image_prompts.json`. See §4 for the full schema and a paste-ready template.
-```markdown
-### Image N: {filename}
-
-| Attribute | Value |
-| --------- | ----- |
-| Purpose | {which page / what function} |
-| Type | {Background / Illustration / Photography / Diagram / Decorative} |
-| Dimensions | {width}x{height} ({aspect ratio}) |
-| Original description | {Reference field from the resource list} |
-
-**Prompt**:
-{subject description}, {style directive}, {color directive}, {composition directive}, {quality directive}
-
-**Alt Text**:
-> {Description for accessibility and image captions}
-```
+| Field | Required | Description |
+|---|---|---|
+| `filename` | yes | Output filename with extension, e.g. `cover_bg.png` |
+| `prompt` | yes | Subject + style + color + composition + quality, assembled per §1.2 |
+| `aspect_ratio` | yes | One of §1.5; passed to `image_gen.py --aspect_ratio` |
+| `status` | yes | `Pending` for new entries — CLI flips to `Generated` / `Failed` |
+| `purpose` | no | Which slide / what function (audit only) |
+| `type` | no | `Background` / `Photography` / `Illustration` / `Diagram` / `Decorative` |
+| `image_size` | no | `512px` / `1K` / `2K` / `4K`; overrides CLI `--image_size` |
+| `negative_prompt` | no | Audit only; not all backends consume it |
+| `alt_text` | no | Caption for accessibility |
+| `model` | no | Per-entry model override |
### 1.2 Prompt Components
@@ -97,7 +93,7 @@ Image 3 prompt: [Deck Style Anchor], growth chart with upward trending line...
**Exception**: Background images may replace style keywords with `background`, `backdrop`, `negative space for text overlay` while keeping the same color directive. This ensures color consistency without compromising background functionality.
-**Rule**: Define the Deck Style Anchor once in the prompt document header (Section 4), then reference it in every individual prompt.
+**Rule**: Set `deck_style_anchor` once at the manifest's top level (§4), then prepend its text to every `items[].prompt` value.
---
@@ -188,24 +184,24 @@ For each image with `Acquire Via: ai` and `Status: Pending`:
2. **Understand purpose** → Which page? What function?
3. **Analyze the Reference field** → User's intent description
4. **Apply type-specific key points** → Reference §2's table for that type
-5. **Generate optimized prompt** → Use the §1.1 standard output format
-6. **Save prompt document** → **Must** write to `project/images/image_prompts.md`
+5. **Generate optimized prompt** → §1.1 schema; prepend the §1.6 Deck Style Anchor
+6. **Save manifest** → **Must** write to `project/images/image_prompts.json` with `status: "Pending"` for every new entry
-> `image_prompts.md` is human-readable; each `### Image N:` block is paste-ready for ChatGPT / Gemini / Midjourney. See §3.2 Offline Manual Mode for the handoff.
+> The manifest is machine-consumed by `image_gen.py --manifest`. The `prompt` field of each entry doubles as a paste-ready string for the Offline Manual handoff (§3.2).
### 3.2 Image Generation Phase
-> Prerequisite: §3.1 must be complete; `images/image_prompts.md` must exist.
+> Prerequisite: §3.1 must be complete; `images/image_prompts.json` must exist and validate.
#### Path Selection (Deterministic)
-C (AI-generated) supports three implementation modes sharing one `image_prompts.md` source:
+C (AI-generated) supports three implementation modes sharing one `image_prompts.json` source:
| Trigger | Mode | Mechanism |
|---|---|---|
-| **Default** — `IMAGE_BACKEND` configured | **Path A**: `image_gen.py` CLI | Agent runs the script; outputs land at `project/images/<filename>` |
+| **Default** — `IMAGE_BACKEND` configured | **Path A**: `image_gen.py --manifest` | One command runs the whole manifest with concurrency; status writes back per item |
| **Path A unavailable/fails OR User explicitly names host tool** | **Path B**: Host-native tool | Agent invokes the host's image capability; outputs land at `project/images/<filename>` |
-| **Both Path A and Path B fail/unavailable** | **Offline Manual Mode** | Agent writes prompts to `image_prompts.md`; user generates externally and places files at `project/images/<filename>` |
+| **Both Path A and Path B fail/unavailable** | **Offline Manual Mode** | Manifest stays on disk; user generates externally from `items[].prompt` and places files at `project/images/<filename>` |
**Selection logic** (automatic, no user prompting):
@@ -219,26 +215,29 @@ C (AI-generated) supports three implementation modes sharing one `image_prompts.
> All three modes share one output contract: file at `project/images/<filename>`. Step 6 SVG references are mode-agnostic.
-#### Path A — `image_gen.py` CLI (Default)
+#### Path A — `image_gen.py --manifest` (Default)
```bash
-python3 scripts/image_gen.py "your prompt" \
- --aspect_ratio 16:9 --image_size 1K \
- --output project/images --filename cover_bg
+python3 scripts/image_gen.py \
+ --manifest project/images/image_prompts.json \
+ --output project/images
```
+The CLI iterates `items[]` with adaptive concurrency, writes `status` back per item, and is **idempotent**: re-running only re-processes entries whose status is `Pending` or `Failed`.
+
**Parameters**:
| Parameter | Short | Description | Default |
-|-----------|-------|-------------|---------|
-| `prompt` | - | Prompt (positional arg) | - |
-| `--aspect_ratio` | - | Image aspect ratio | `1:1` |
-| `--image_size` | - | Size (`1K`/`2K`/`4K`) | `1K` |
-| `--output` | `-o` | Output directory | Current directory |
-| `--filename` | `-f` | Output filename (no extension) | Auto-named |
-| `--backend` | `-b` | Override backend (see `--list-backends` for options) | None |
-| `--model` | `-m` | Model name | Backend default |
-| `--list-backends` | - | Print support tiers and exit | `false` |
+|---|---|---|---|
+| `--manifest` | - | Path to `image_prompts.json` | — |
+| `--concurrency` | - | Max concurrent requests; halves on rate-limit, min 1 | `IMAGE_CONCURRENCY` env or `3` |
+| `--image_size` | - | Default size (`512px`/`1K`/`2K`/`4K`); per-item `image_size` wins | `1K` |
+| `--output` | `-o` | Output directory | Manifest's parent dir |
+| `--backend` | `-b` | Override `IMAGE_BACKEND` for this run | env |
+| `--model` | `-m` | Default model; per-item `model` wins | Backend default |
+| `--list-backends` | - | Print support tiers and exit | — |
+
+> The single-image form `image_gen.py "prompt" --filename ...` is preserved for ad-hoc one-offs (re-rolling a single image) but is no longer the primary path.
**Configuration sources**:
- Current process environment variables
@@ -251,6 +250,7 @@ Precedence:
| Variable | Required | Description |
|----------|----------|-------------|
| `IMAGE_BACKEND` | Required | Backend identifier; run `image_gen.py --list-backends` for the current set |
+| `IMAGE_CONCURRENCY` | Optional | Manifest-mode default concurrency (CLI `--concurrency` wins) |
| `{PROVIDER}_API_KEY` | Required | Provider-specific API key, e.g. `GEMINI_API_KEY`, `ZHIPU_API_KEY` |
| `{PROVIDER}_BASE_URL` | Optional | Provider-specific custom endpoint |
| `{PROVIDER}_MODEL` | Optional | Provider-specific model override |
@@ -263,16 +263,18 @@ Precedence:
**Support tiers (recommended usage)**: Core / Extended / Experimental. Run `image_gen.py --list-backends` for the current assignments.
-**Generation pacing (mandatory)**:
-- Execute only one generation command at a time; wait for file confirmation before the next
-- Recommend 2-5 second intervals between images to avoid concurrency failures
+**Concurrency (manifest mode)**:
+- Default 3 concurrent requests, halves on the first rate-limit response, minimum 1 (= serial fallback)
+- Rate-limited items requeue automatically; per-item failures are recorded with `last_error` and skipped
+- Interrupting mid-run is safe — completed items keep `status: Generated` and are skipped on re-run
#### Path B — Host-Native Image Tool (On Explicit User Request)
Triggered only when the user explicitly asks the skill to use the host's built-in image generation (e.g. Codex, Antigravity, or any other host that provides a native image tool).
-- Agent invokes the host's native image tool directly; prompts come from the same `image_prompts.md`
+- Agent invokes the host's native image tool directly; prompts come from `items[].prompt`
- Outputs **must** land at `project/images/<filename-from-resource-list>` with dimensions matching the Image Resource List
+- After each placement, set the corresponding item's `status` to `Generated` in the manifest
- Executor downstream is path-agnostic — no spec change required between Path A and Path B
#### Offline Manual Mode (C's third implementation mode)
@@ -281,12 +283,12 @@ Triggered only when the user explicitly asks the skill to use the host's built-i
**Workflow** (no user prompting; system enters this mode automatically):
-1. Verify `images/image_prompts.md` was generated in §3.1
-2. Set `Status: Needs-Manual` on every affected ai row per [`image-base.md`](./image-base.md) §6
+1. Verify `images/image_prompts.json` was written in §3.1
+2. Set `status: "Needs-Manual"` on every affected item per [`image-base.md`](./image-base.md) §6
3. Continue to Step 6 — SVG references `images/<filename>` optimistically; Step 7 entry verifies presence
4. Print one consolidated handoff to the user:
- Filenames awaiting manual generation
- - Pointer to `images/image_prompts.md`: each `### Image N:` block is a paste-ready prompt for ChatGPT / Gemini / Midjourney
+ - Pointer to `images/image_prompts.json`: each `items[].prompt` is paste-ready for ChatGPT / Gemini / Midjourney
- Target placement: `project/images/<filename>` matching the resource list exactly
- Resume command: re-run Step 7 once all expected files exist
@@ -315,54 +317,39 @@ If Path A's backend fails twice in a row:
---
-## 4. Prompt Document Template
-
-Use the following structure when creating `project/images/image_prompts.md`:
-
-```markdown
-# Image Generation Prompts
-
-> Project: {project_name}
-> Generated: {date}
-> Color scheme: Primary {#HEX} | Secondary {#HEX} | Accent {#HEX}
-> Deck Style Anchor: {15–25 word prefix per §1.6}
-
----
-
-## Image List Overview
-
-| # | Filename | Type | Dimensions | Status |
-|---|----------|------|-----------|--------|
-| 1 | cover_bg.png | Background | 1920x1080 | Pending |
-
----
-
-## Detailed Prompts
-
-### Image 1: cover_bg.png
-
-| Attribute | Value |
-|-----------|-------|
-| Purpose | Cover background |
-| Type | Background |
-| Dimensions | 1920x1080 (16:9) |
-| Original description | Modern tech abstract background, deep blue gradient |
-
-**Prompt**:
-[Deck Style Anchor], Abstract futuristic background with flowing digital waves...
-
-**Alt Text**:
-> Modern tech abstract background with deep blue gradient, digital waves, and particle effects
-
----
+## 4. Manifest Template
+
+Write `project/images/image_prompts.json` with this shape. Top-level fields are audit-only; `items[]` is the machine contract.
+
+```json
+{
+ "project": "{project_name}",
+ "generated_at": "{ISO-8601 date}",
+ "color_scheme": {
+ "primary": "#1A3A5C",
+ "secondary": "#F8F9FA",
+ "accent": "#E8A838"
+ },
+ "deck_style_anchor": "{15–25 word prefix per §1.6}",
+ "items": [
+ {
+ "filename": "cover_bg.png",
+ "purpose": "Cover background (Slide 01)",
+ "type": "Background",
+ "aspect_ratio": "16:9",
+ "image_size": "2K",
+ "prompt": "{deck_style_anchor}, abstract futuristic background with flowing digital waves, deep navy gradient #1A3A5C to midnight, soft particle accents, clean center for text overlay, cinematic, 4K",
+ "negative_prompt": "text, watermark, faces, busy patterns",
+ "alt_text": "Modern tech abstract background with deep blue gradient and digital waves",
+ "status": "Pending"
+ }
+ ]
+}
+```
-## Usage Instructions
+**Field reference**: see §1.1 for required vs optional. The CLI rewrites each item's `status` (and adds `last_error` on failure) — do not hand-edit those while a run is in flight.
-1. Copy the "Prompt" above into an AI image generation tool
-2. Recommended platforms: gpt-image-2 / Midjourney / DALL-E 3 / Gemini / Stable Diffusion
-3. Rename generated images to the corresponding filenames
-4. Place in the `images/` directory
-```
+**Paste-ready for manual mode**: each `items[].prompt` is a complete, self-contained prompt; copy it directly into ChatGPT / Gemini / Midjourney when running Offline Manual Mode (§3.2).
---
@@ -391,10 +378,9 @@ Diagnose the problem category and apply a targeted prompt fix:
| Low quality | Image is blurry, has artifacts, or lacks detail | Add `highly detailed, sharp focus, professional quality, 8K resolution` |
**Variant workflow**:
-1. Keep the original prompt as "Variant A" in `image_prompts.md`
-2. Create modified prompt as "Variant B" with targeted fixes from the table above
-3. If needed, create "Variant C" with a different stylistic approach
-4. Label all variants clearly so the user can compare results
+1. Set the unsatisfactory item's `status` back to `Pending` and update its `prompt` in place
+2. Re-run `image_gen.py --manifest` — only that item is re-processed
+3. If trying multiple stylistic approaches, append additional items with distinct filenames (e.g. `cover_bg_v2.png`) rather than overwriting
---
@@ -403,4 +389,4 @@ Diagnose the problem category and apply a targeted prompt fix:
- Generating prompts for `web` rows — those go through [`image-searcher.md`](./image-searcher.md)
- Brand names or HEX codes inside the subject description (degrades output)
- Mixed Deck Style Anchors across images in the same deck (breaks coherence)
-- Placing an image without updating `image_prompts.md` and the resource list status
+- Placing an image without updating its `image_prompts.json` `status` and the resource list status
diff --git a/skills/ppt-master/references/strategist.md b/skills/ppt-master/references/strategist.md
index cd48ab88..b3dbb612 100644
--- a/skills/ppt-master/references/strategist.md
+++ b/skills/ppt-master/references/strategist.md
@@ -235,7 +235,7 @@ Baseline choice follows **content density**, not style. Common: `18px` (dense) /
|---|---|---|
| **Path A** | `IMAGE_BACKEND` configured (default) | `image_gen.py` runs in Step 5 |
| **Path B** | User explicitly names host's image tool (Codex / Antigravity) | Host-native generation |
-| **Offline Manual** | Path A unavailable AND Path B not in use | Prompts written to `images/image_prompts.md`; user generates externally and places files in `project/images/` |
+| **Offline Manual** | Path A unavailable AND Path B not in use | Prompts written to `images/image_prompts.json`; user generates externally and places files in `project/images/` |
Selection is automatic in Step 5 (A → B → Manual). Detailed contract: [`image-generator.md`](./image-generator.md) §3.2.
diff --git a/skills/ppt-master/scripts/image_gen.py b/skills/ppt-master/scripts/image_gen.py
index 3eccb983..4e407f59 100644
--- a/skills/ppt-master/scripts/image_gen.py
+++ b/skills/ppt-master/scripts/image_gen.py
@@ -38,12 +38,19 @@ Supported keys:
Usage:
python3 image_gen.py "prompt" --aspect_ratio 16:9 --image_size 1K -o images/
+ python3 image_gen.py --manifest project/images/image_prompts.json -o project/images/
python3 image_gen.py --list-backends
"""
+import concurrent.futures
+import json
import os
import sys
import argparse
+import tempfile
+import threading
+import time
+from pathlib import Path
from config import load_prefixed_env_file, resolve_env_path
@@ -327,6 +334,213 @@ def _resolve_backend() -> tuple[object, str]:
sys.exit(1)
+DEFAULT_MANIFEST_CONCURRENCY = 3
+
+STATUS_PENDING = "Pending"
+STATUS_GENERATED = "Generated"
+STATUS_FAILED = "Failed"
+STATUS_NEEDS_MANUAL = "Needs-Manual"
+VALID_STATUSES = {STATUS_PENDING, STATUS_GENERATED, STATUS_FAILED, STATUS_NEEDS_MANUAL}
+RETRYABLE_STATUSES = {STATUS_PENDING, STATUS_FAILED}
+REQUIRED_ITEM_FIELDS = ("filename", "prompt", "aspect_ratio", "status")
+
+
+def load_manifest(path: str) -> dict:
+ """Load and validate an `image_prompts.json` manifest.
+
+ Schema (top level): {"items": [ ... ]}, optionally with
+ `deck_style_anchor`, `color_scheme`, `generated_at`.
+
+ Each item requires: `filename`, `prompt`, `aspect_ratio`, `status`.
+ Optional: `image_size`, `model`, `negative_prompt`, `alt_text`,
+ `purpose`, `type`, `last_error`.
+ """
+ try:
+ data = json.loads(Path(path).read_text(encoding="utf-8"))
+ except json.JSONDecodeError as exc:
+ raise ValueError(
+ f"Invalid JSON in {path}: {exc.msg} "
+ f"(line {exc.lineno}, col {exc.colno})"
+ ) from exc
+
+ if not isinstance(data, dict):
+ raise ValueError(
+ f"{path}: top level must be a JSON object, "
+ f"got {type(data).__name__}"
+ )
+
+ items = data.get("items")
+ if not isinstance(items, list) or not items:
+ raise ValueError(f"{path}: 'items' must be a non-empty array")
+
+ seen_filenames: set[str] = set()
+ for i, item in enumerate(items):
+ prefix = f"{path}: items[{i}]"
+ if not isinstance(item, dict):
+ raise ValueError(f"{prefix} must be an object")
+ for field in REQUIRED_ITEM_FIELDS:
+ if field not in item:
+ raise ValueError(f"{prefix} missing required field '{field}'")
+ if not isinstance(item[field], str) or not item[field].strip():
+ raise ValueError(
+ f"{prefix} field '{field}' must be a non-empty string"
+ )
+ if item["status"] not in VALID_STATUSES:
+ raise ValueError(
+ f"{prefix} status '{item['status']}' is invalid. "
+ f"Valid: {sorted(VALID_STATUSES)}"
+ )
+ fname = item["filename"]
+ if fname in seen_filenames:
+ raise ValueError(f"{prefix} duplicate filename '{fname}'")
+ seen_filenames.add(fname)
+
+ return data
+
+
+def save_manifest(path: str, data: dict) -> None:
+ """Atomically write manifest back to disk (tmp file + rename)."""
+ target = Path(path)
+ fd, tmp_path = tempfile.mkstemp(
+ prefix=target.stem + ".",
+ suffix=".tmp",
+ dir=str(target.parent),
+ )
+ try:
+ with os.fdopen(fd, "w", encoding="utf-8") as f:
+ json.dump(data, f, ensure_ascii=False, indent=2)
+ f.write("\n")
+ os.replace(tmp_path, target)
+ except Exception:
+ try:
+ os.unlink(tmp_path)
+ except OSError:
+ pass
+ raise
+
+
+def _run_manifest(manifest: dict, manifest_path: str, backend_module, *,
+ initial_concurrency: int,
+ image_size: str,
+ output_dir: str,
+ model: str | None) -> tuple[int, int, int]:
+ """Run Pending/Failed items through the backend with adaptive concurrency.
+
+ Strategy:
+ - Start at `initial_concurrency` workers per batch.
+ - On any rate-limit error in a batch, halve concurrency (min 1) and
+ requeue the rate-limited items.
+ - Per-item failures are recorded as `status: Failed` + `last_error`
+ and not retried within this run.
+ - Status is written back to the manifest file after each completion;
+ a Ctrl-C in the middle still preserves done items.
+ - `Needs-Manual` items are skipped (user processes them externally).
+
+ Returns (ok_count, failed_count, skipped_count).
+ """
+ from image_backends.backend_common import is_rate_limit_error
+
+ items = manifest["items"]
+ pending_idx = [
+ i for i, it in enumerate(items) if it["status"] in RETRYABLE_STATUSES
+ ]
+ total = len(pending_idx)
+ skipped = len(items) - total
+
+ if total == 0:
+ print(
+ f"[Manifest] Nothing to do — all {len(items)} items already in "
+ "a terminal state (Generated / Needs-Manual)."
+ )
+ return 0, 0, skipped
+
+ print(
+ f"\n[Manifest] {total} item(s) to generate, "
+ f"{skipped} already done. concurrency={initial_concurrency}\n"
+ )
+
+ queue: list[int] = list(pending_idx)
+ ok_count = 0
+ fail_count = 0
+ current = max(1, initial_concurrency)
+ state_lock = threading.Lock()
+
+ def _one(idx: int):
+ item = items[idx]
+ try:
+ saved_path = backend_module.generate(
+ prompt=item["prompt"],
+ aspect_ratio=item["aspect_ratio"],
+ image_size=item.get("image_size", image_size),
+ output_dir=output_dir,
+ filename=Path(item["filename"]).stem,
+ model=item.get("model", model),
+ )
+ return idx, saved_path, None
+ except Exception as exc: # noqa: BLE001 — backend raises arbitrary types
+ return idx, None, exc
+
+ while queue:
+ batch_size = min(current, len(queue))
+ batch_idx = queue[:batch_size]
+ queue = queue[batch_size:]
+
+ print(
+ f"--- Batch of {batch_size} (concurrency={current}, "
+ f"remaining_after={len(queue)}) ---"
+ )
+
+ rate_limited = False
+ with concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as ex:
+ futures = [ex.submit(_one, i) for i in batch_idx]
+ for fut in concurrent.futures.as_completed(futures):
+ idx, saved_path, exc = fut.result()
+ item = items[idx]
+ with state_lock:
+ if exc is None:
+ item["status"] = STATUS_GENERATED
+ item.pop("last_error", None)
+ ok_count += 1
+ print(f" [OK] {item['filename']}")
+ elif is_rate_limit_error(exc):
+ rate_limited = True
+ queue.append(idx)
+ print(f" [RATE] {item['filename']} — requeued")
+ else:
+ item["status"] = STATUS_FAILED
+ item["last_error"] = str(exc)[:500]
+ fail_count += 1
+ print(f" [FAIL] {item['filename']}: {exc}")
+ save_manifest(manifest_path, manifest)
+
+ if rate_limited and current > 1:
+ new_current = max(1, current // 2)
+ print(
+ f"\n ⚠ Rate-limit hit — concurrency {current} → {new_current}, "
+ "pausing 10s before next batch\n"
+ )
+ current = new_current
+ time.sleep(10)
+ elif queue:
+ time.sleep(2)
+
+ print(
+ f"\n[Manifest] Done: {ok_count} ok / {fail_count} failed "
+ f"({skipped} pre-skipped). Manifest written to {manifest_path}"
+ )
+ return ok_count, fail_count, skipped
+
+
+def _resolve_concurrency(cli_value: int | None) -> int:
+ """CLI value wins over IMAGE_CONCURRENCY env; default 3."""
+ if cli_value is not None:
+ return max(1, cli_value)
+ env_val = os.environ.get("IMAGE_CONCURRENCY", "").strip()
+ if env_val.isdigit():
+ return max(1, int(env_val))
+ return DEFAULT_MANIFEST_CONCURRENCY
+
+
def main() -> None:
"""Run the CLI entry point."""
parser = argparse.ArgumentParser(
@@ -364,6 +578,21 @@ def main() -> None:
"--list-backends", action="store_true",
help="List available backends grouped by support tier and exit."
)
+ parser.add_argument(
+ "--manifest", default=None, metavar="IMAGE_PROMPTS_JSON",
+ help=(
+ "Path to image_prompts.json. Runs every Pending/Failed item in "
+ "parallel; writes status back to the manifest as each completes."
+ ),
+ )
+ parser.add_argument(
+ "--concurrency", type=int, default=None,
+ help=(
+ "Max concurrent requests in --manifest mode. Defaults to "
+ f"IMAGE_CONCURRENCY env or {DEFAULT_MANIFEST_CONCURRENCY}. "
+ "Auto-halves on rate-limit; 1 is the serial fallback."
+ ),
+ )
args = parser.parse_args()
@@ -385,6 +614,30 @@ def main() -> None:
backend, backend_name = _resolve_backend()
print(f"Using backend: {backend_name}\n")
+ if args.manifest:
+ if not os.path.isfile(args.manifest):
+ print(f"Error: manifest file not found: {args.manifest}")
+ sys.exit(1)
+ try:
+ manifest = load_manifest(args.manifest)
+ except ValueError as e:
+ print(f"Error: {e}")
+ sys.exit(1)
+
+ concurrency = _resolve_concurrency(args.concurrency)
+ try:
+ _, failed, _ = _run_manifest(
+ manifest, args.manifest, backend,
+ initial_concurrency=concurrency,
+ image_size=args.image_size,
+ output_dir=args.output or str(Path(args.manifest).parent),
+ model=args.model,
+ )
+ except KeyboardInterrupt:
+ print("\n\nInterrupted by user. Partial progress preserved in manifest.")
+ sys.exit(130)
+ sys.exit(1 if failed else 0)
+
try:
backend.generate(
prompt=args.prompt,