AI turns documents or topics into real, native PowerPoint decks—with native shapes, transitions and animations, data-backed charts and tables on demand, audio narration from speaker notes, and support for your own .pptx templates. · by Hugo He
feat(image_search): add concurrent --batch mode for web image rows
hugohe35ec1e95631ee
diff --git a/skills/ppt-master/SKILL.md b/skills/ppt-master/SKILL.mdindex 2fb75a29..032a97bd 100644--- a/skills/ppt-master/SKILL.md+++ b/skills/ppt-master/SKILL.md@@ -432,13 +432,15 @@ Then **lazy-load the path-specific reference** for each row that actually needs| Acquire Via | Load reference (only if any such row exists) | Run ||---|---|---|| `ai` | `references/image-generator.md` | `python3 ${SKILL_DIR}/scripts/image_gen.py --manifest <project_path>/images/image_prompts.json` |-| `web` | `references/image-searcher.md` | `python3 ${SKILL_DIR}/scripts/image_search.py ...` |+| `web` | `references/image-searcher.md` | `python3 ${SKILL_DIR}/scripts/image_search.py ...` (≥2 web rows → `--batch images/image_queries.json`) || `user` / `placeholder` | (skip) | (skip) |A deck with only `ai` rows never loads `image-searcher.md`; a deck with only `web` rows never loads `image-generator.md`. A mixed deck loads both, processes each row through its own path, and writes both `image_prompts.json` and `image_sources.json`.> ⚠️ **In-pipeline ai path MUST use manifest mode** — even when only 1 ai row exists. Write `images/image_prompts.json` first, then run `image_gen.py --manifest`, then `image_gen.py --render-md` to produce the `image_prompts.md` sidecar. The positional form (`image_gen.py "prompt" ...`) is reserved for **out-of-pipeline one-off testing / single-image fixups** — it skips manifest + sidecar, leaving no audit trail.+> ⚠️ **web path — batch multiple rows**: when ≥2 rows are `Acquire Via: web`, write all queries into `images/image_queries.json` and run `image_search.py --batch` once (concurrent acquisition, status written back), instead of one CLI call per row. A single web row may use the positional single-query form. See [image-searcher.md](references/image-searcher.md) §5.+> ⚠️ **Honor the confirmed image source**: the `ai` generation path (Path A = `image_gen.py` API / Path B = host-native tool / Offline Manual) is **not** auto-only — a confirmed choice other than `auto` wins, whether it came from chat (canonical) or, when the page was used, `result.json.image_ai_path`. `host-native` forces Path B even when `IMAGE_BACKEND` is configured; `api` forces Path A; `manual` forces offline. The `--manifest` command above is Path A. Full selection rule: [image-generator.md](references/image-generator.md) §7 Path Selection.Workflow:diff --git a/skills/ppt-master/references/image-searcher.md b/skills/ppt-master/references/image-searcher.mdindex 5721dd2c..0e802884 100644--- a/skills/ppt-master/references/image-searcher.md+++ b/skills/ppt-master/references/image-searcher.md@@ -119,7 +119,37 @@ python3 scripts/image_search.py "<query>" \| `--strict-no-attribution` | no | off | Restrict to no-attribution licenses; refuse CC BY / CC BY-SA || `--manifest` | no | (default) | Override manifest path |-**Pacing (mandatory)**: one search at a time. Wikimedia/Openverse expect identifying User-Agent and reasonable rate (~1 req/sec). Default pacing is fine.+### Batch mode (≥ 2 web rows) — preferred++When more than one row is `Acquire Via: web`, do **not** call the CLI once per row. Write all rows into one `image_queries.json` and run a single concurrent batch — the web sister of `image_gen.py --manifest`:++```bash+python3 scripts/image_search.py --batch <project_path>/images/image_queries.json \+ -o <project_path>/images+```++`image_queries.json` schema (one item per web row):++```json+{+ "items": [+ {+ "filename": "team.jpg",+ "query": "executive boardroom meeting",+ "slide": "03_team",+ "purpose": "background",+ "orientation": "landscape",+ "status": "Pending"+ }+ ]+}+```++Required per item: `filename`, `query`, `status` (`Pending`). Optional per-item overrides: `slide`, `purpose`, `orientation`, `provider`, `strict_no_attribution`, `min_width`, `min_height`.++The runner searches all `Pending` / `Failed` rows concurrently, appends each success to `image_sources.json` (the credit source of truth, idempotent on `filename`), and writes status back into `image_queries.json` — `Sourced` on success, `Needs-Manual` when the full provider/stage chain is exhausted. Status is saved after each completion, so an interrupted run preserves finished rows; re-running skips terminal rows. A single `web` row may still use single-query mode above.++**Pacing**: free providers (Wikimedia/Openverse) are rate-sensitive, so batch concurrency defaults to a modest **3** (`--concurrency N`, or `IMAGE_SEARCH_CONCURRENCY` env). Use `--concurrency 1` to restore strict one-at-a-time pacing. Single-query mode is one request at a time by nature.---diff --git a/skills/ppt-master/scripts/docs/image.md b/skills/ppt-master/scripts/docs/image.mdindex 20acfd66..7927a1f0 100644--- a/skills/ppt-master/scripts/docs/image.md+++ b/skills/ppt-master/scripts/docs/image.md@@ -167,6 +167,8 @@ python3 scripts/image_search.py "offshore wind farm" \--orientation landscape -o projects/demo/images```+For multiple web rows, `--batch images/image_queries.json` searches them concurrently (modest default, `--concurrency N` / `IMAGE_SEARCH_CONCURRENCY` to tune) instead of one call per row — the web sister of `image_gen.py --manifest`. Schema and status semantics: [`image-searcher.md`](../../references/image-searcher.md) §5.+Providers (Openverse and Wikimedia work with no key; configure Pexels / Pixabay for better stock-photo quality):| Provider | Config | Strength |diff --git a/skills/ppt-master/scripts/image_search.py b/skills/ppt-master/scripts/image_search.pyindex 8e07f395..29217d8e 100644--- a/skills/ppt-master/scripts/image_search.py+++ b/skills/ppt-master/scripts/image_search.py@@ -39,10 +39,13 @@ Examples:from __future__ import annotationsimport argparse+import concurrent.futuresimport importlibimport jsonimport osimport sys+import tempfile+import threadingfrom datetime import datetime, timezonefrom pathlib import Pathfrom typing import Optional@@ -85,6 +88,29 @@ ALL_PROVIDERS: tuple[str, ...] = ZERO_CONFIG_PROVIDERS + KEYED_PROVIDERSORIENTATION_CHOICES = ("any", "landscape", "portrait", "square")+# --- Batch mode (`--batch image_queries.json`) -----------------------------+# Web providers are politeness-sensitive (Wikimedia/Openverse expect a modest+# rate), so the default concurrency is deliberately low. Sister-tool+# `image_gen.py` hits a paid API and defaults higher; here 3 keeps several+# rows in flight without hammering any single free provider. Set to 1 to+# restore strict one-at-a-time pacing.+DEFAULT_SEARCH_CONCURRENCY = 3++SEARCH_STATUS_PENDING = "Pending"+SEARCH_STATUS_SOURCED = "Sourced"+SEARCH_STATUS_FAILED = "Failed"+SEARCH_STATUS_NEEDS_MANUAL = "Needs-Manual"+SEARCH_VALID_STATUSES = {+ SEARCH_STATUS_PENDING,+ SEARCH_STATUS_SOURCED,+ SEARCH_STATUS_FAILED,+ SEARCH_STATUS_NEEDS_MANUAL,+}+# A row reaching `Needs-Manual` after the full provider/stage chain is terminal+# (see image-searcher.md §8); only Pending/Failed rows are retried on re-run.+SEARCH_RETRYABLE_STATUSES = {SEARCH_STATUS_PENDING, SEARCH_STATUS_FAILED}+SEARCH_REQUIRED_ITEM_FIELDS = ("filename", "query", "status")+# ---------------------------------------------------------------------------# .env loading@@ -531,6 +557,241 @@ def promote_candidate(return 0+# ---------------------------------------------------------------------------+# Batch mode (`--batch image_queries.json`)+# ---------------------------------------------------------------------------+++def load_search_manifest(path: str) -> dict:+ """Load and validate an ``image_queries.json`` batch manifest.++ Schema (top level): ``{"items": [ ... ]}``. Each item requires+ ``filename``, ``query``, ``status``. Optional per-item overrides:+ ``slide``, ``purpose``, ``orientation``, ``provider``,+ ``strict_no_attribution``, ``min_width``, ``min_height``, ``last_error``.+ """+ try:+ data = json.loads(Path(path).read_text(encoding="utf-8"))+ except json.JSONDecodeError as exc:+ raise ValueError(+ f"Invalid JSON in {path}: {exc.msg} "+ f"(line {exc.lineno}, col {exc.colno})"+ ) from exc++ if not isinstance(data, dict):+ raise ValueError(+ f"{path}: top level must be a JSON object, got {type(data).__name__}"+ )++ items = data.get("items")+ if not isinstance(items, list) or not items:+ raise ValueError(f"{path}: 'items' must be a non-empty array")++ seen_filenames: set[str] = set()+ for i, item in enumerate(items):+ prefix = f"{path}: items[{i}]"+ if not isinstance(item, dict):+ raise ValueError(f"{prefix} must be an object")+ for field in SEARCH_REQUIRED_ITEM_FIELDS:+ if field not in item:+ raise ValueError(f"{prefix} missing required field '{field}'")+ if not isinstance(item[field], str) or not item[field].strip():+ raise ValueError(+ f"{prefix} field '{field}' must be a non-empty string"+ )+ if item["status"] not in SEARCH_VALID_STATUSES:+ raise ValueError(+ f"{prefix} status '{item['status']}' is invalid. "+ f"Valid: {sorted(SEARCH_VALID_STATUSES)}"+ )+ fname = item["filename"]+ if fname in seen_filenames:+ raise ValueError(f"{prefix} duplicate filename '{fname}'")+ seen_filenames.add(fname)++ return data+++def save_search_manifest(path: str, data: dict) -> None:+ """Atomically write the batch manifest back (tmp file + rename)."""+ target = Path(path)+ fd, tmp_path = tempfile.mkstemp(+ prefix=target.stem + ".", suffix=".tmp", dir=str(target.parent)+ )+ try:+ with os.fdopen(fd, "w", encoding="utf-8") as f:+ json.dump(data, f, ensure_ascii=False, indent=2)+ f.write("\n")+ os.replace(tmp_path, target)+ except Exception:+ try:+ os.unlink(tmp_path)+ except OSError:+ pass+ raise+++def _resolve_search_concurrency(cli_value: Optional[int]) -> int:+ """CLI value wins over IMAGE_SEARCH_CONCURRENCY env; default 3."""+ if cli_value is not None:+ return max(1, cli_value)+ env_val = os.environ.get("IMAGE_SEARCH_CONCURRENCY", "").strip()+ if env_val.isdigit():+ return max(1, int(env_val))+ return DEFAULT_SEARCH_CONCURRENCY+++def _search_one_item(+ item: dict,+ *,+ output_dir: Path,+ save_candidates: bool,+ max_candidates: int,+ default_provider: Optional[str],+ default_strict: bool,+ default_min_width: int,+ default_min_height: int,+) -> tuple[Optional[dict], Optional[str]]:+ """Run the full search + download for one batch item (thread worker).++ Returns ``(manifest_item, error)``. Only the network/disk work happens+ here; all manifest writes are serialized by the caller.+ """+ orientation = item.get("orientation", "any") or "any"+ strict = bool(item.get("strict_no_attribution", default_strict))+ request = ImageSearchRequest(+ query=item["query"],+ purpose=item.get("purpose", ""),+ orientation="" if orientation == "any" else orientation,+ filename=item["filename"],+ slide=item.get("slide", ""),+ min_width=int(item.get("min_width", default_min_width)),+ min_height=int(item.get("min_height", default_min_height)),+ )++ pinned = item.get("provider") or default_provider+ providers = [pinned] if pinned else _default_provider_chain()+ output_path = output_dir / item["filename"]++ candidate, provider_name, stage = search_and_download(+ providers,+ request,+ output_path=output_path,+ strict_no_attribution=strict,+ save_candidates=save_candidates,+ max_candidates=max_candidates,+ )+ if candidate is None:+ return None, "no acceptable candidate across all providers/stages"++ actual_dimensions = _measure_actual_image(output_path)+ item_args = argparse.Namespace(+ filename=item["filename"],+ slide=item.get("slide", ""),+ purpose=item.get("purpose", ""),+ query=item["query"],+ orientation=orientation,+ )+ manifest_item = _candidate_to_manifest_item(+ candidate,+ item_args,+ provider_name=provider_name,+ stage=stage,+ actual_dimensions=actual_dimensions,+ )+ return manifest_item, None+++def run_search_manifest(+ manifest: dict,+ manifest_path: str,+ *,+ output_dir: Path,+ sources_manifest_path: Path,+ concurrency: int,+ save_candidates: bool,+ max_candidates: int,+ default_provider: Optional[str],+ default_strict: bool,+ default_min_width: int,+ default_min_height: int,+) -> tuple[int, int, int]:+ """Process all Pending/Failed rows concurrently with a bounded pool.++ On success the rich provenance entry is appended to ``image_sources.json``+ (the credit source of truth) and the row's status flips to ``Sourced``.+ A row that exhausts the provider/stage chain becomes ``Needs-Manual``+ (terminal). Status is written back after each completion, so an interrupt+ preserves finished rows. Returns ``(sourced, needs_manual, skipped)``.+ """+ items = manifest["items"]+ pending_idx = [+ i for i, it in enumerate(items)+ if it["status"] in SEARCH_RETRYABLE_STATUSES+ ]+ total = len(pending_idx)+ skipped = len(items) - total++ if total == 0:+ print(+ f"[Batch] Nothing to do — all {len(items)} row(s) already in a "+ "terminal state (Sourced / Needs-Manual)."+ )+ return 0, 0, skipped++ print(+ f"\n[Batch] {total} row(s) to search, {skipped} already done. "+ f"concurrency={concurrency}\n"+ )++ sourced_count = 0+ needs_manual_count = 0+ write_lock = threading.Lock()++ def _one(idx: int):+ try:+ manifest_item, error = _search_one_item(+ items[idx],+ output_dir=output_dir,+ save_candidates=save_candidates,+ max_candidates=max_candidates,+ default_provider=default_provider,+ default_strict=default_strict,+ default_min_width=default_min_width,+ default_min_height=default_min_height,+ )+ return idx, manifest_item, error+ except Exception as exc: # noqa: BLE001 — provider code raises freely+ return idx, None, str(exc)[:500]++ with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as ex:+ futures = [ex.submit(_one, i) for i in pending_idx]+ for fut in concurrent.futures.as_completed(futures):+ idx, manifest_item, error = fut.result()+ item = items[idx]+ with write_lock:+ if manifest_item is not None:+ write_sources_manifest(sources_manifest_path, manifest_item)+ item["status"] = SEARCH_STATUS_SOURCED+ item["provider"] = manifest_item.get("provider", "")+ item["license_tier"] = manifest_item.get("license_tier", "")+ item.pop("last_error", None)+ sourced_count += 1+ print(f" [OK] {item['filename']} ({item['provider']})")+ else:+ item["status"] = SEARCH_STATUS_NEEDS_MANUAL+ item["last_error"] = error or "search failed"+ needs_manual_count += 1+ print(f" [MANUAL] {item['filename']} — {item['last_error']}")+ save_search_manifest(manifest_path, manifest)++ print(+ f"\n[Batch] Done: {sourced_count} sourced / {needs_manual_count} "+ f"needs-manual ({skipped} pre-skipped). Manifest: {manifest_path}"+ )+ return sourced_count, needs_manual_count, skipped++# ---------------------------------------------------------------------------# CLI# ---------------------------------------------------------------------------@@ -544,11 +805,19 @@ def build_parser() -> argparse.ArgumentParser:),formatter_class=argparse.RawDescriptionHelpFormatter,)- parser.add_argument("query", help="Search query (2-5 keywords work best).")+ parser.add_argument(+ "query",+ nargs="?",+ default=None,+ help="Search query (2-5 keywords work best). Omit in --batch mode.",+ )parser.add_argument("--filename",- required=True,- help="Local filename for the chosen image (e.g. cover_bg.jpg).",+ default=None,+ help=(+ "Local filename for the chosen image (e.g. cover_bg.jpg). "+ "Required for single-query and --promote modes; ignored in --batch."+ ),)parser.add_argument("-o",@@ -606,6 +875,27 @@ def build_parser() -> argparse.ArgumentParser:default=None,help="Override manifest path. Defaults to <output>/image_sources.json.",)+ parser.add_argument(+ "--batch",+ default=None,+ metavar="QUERIES_JSON",+ help=(+ "Process a batch of search requests from an image_queries.json "+ "manifest concurrently, writing provenance into image_sources.json "+ "and status back into the queries manifest."+ ),+ )+ parser.add_argument(+ "--concurrency",+ type=int,+ default=None,+ help=(+ "Max concurrent searches in --batch mode. Defaults to "+ f"IMAGE_SEARCH_CONCURRENCY env or {DEFAULT_SEARCH_CONCURRENCY}. "+ "Keep modest — free providers are rate-sensitive; use 1 for "+ "strict one-at-a-time pacing."+ ),+ )parser.add_argument("--no-candidates",action="store_true",@@ -651,6 +941,8 @@ def main(argv: Optional[list[str]] = None) -> int:# --- Promote mode ---if args.promote:+ if not args.filename:+ parser.error("--filename is required in --promote mode")return promote_candidate(output_dir,args.filename,@@ -658,7 +950,52 @@ def main(argv: Optional[list[str]] = None) -> int:manifest_path=Path(args.manifest) if args.manifest else None,)- # --- Search mode ---+ # --- Batch mode ---+ if args.batch:+ if not os.path.isfile(args.batch):+ print(f"Error: queries manifest not found: {args.batch}", file=sys.stderr)+ return 1+ try:+ manifest = load_search_manifest(args.batch)+ except ValueError as exc:+ print(f"Error: {exc}", file=sys.stderr)+ return 1+ batch_output_dir = (+ output_dir if args.output != "." else Path(args.batch).parent+ )+ batch_output_dir.mkdir(parents=True, exist_ok=True)+ sources_manifest_path = (+ Path(args.manifest) if args.manifest+ else default_manifest_path(str(batch_output_dir))+ )+ try:+ _, needs_manual, _ = run_search_manifest(+ manifest,+ args.batch,+ output_dir=batch_output_dir,+ sources_manifest_path=sources_manifest_path,+ concurrency=_resolve_search_concurrency(args.concurrency),+ save_candidates=not args.no_candidates,+ max_candidates=args.max_candidates,+ default_provider=args.provider,+ default_strict=args.strict_no_attribution,+ default_min_width=args.min_width,+ default_min_height=args.min_height,+ )+ except KeyboardInterrupt:+ print("\n\nInterrupted by user. Partial progress preserved in manifest.")+ return 130+ # Mirror image_gen.py: a non-zero code flags rows that need manual+ # attention. It is a signal, not a halt — the workflow (image-base.md+ # §6) surfaces Needs-Manual rows and continues regardless.+ return 1 if needs_manual else 0++ # --- Single-query search mode ---+ if not args.query:+ parser.error("query is required unless --batch or --promote is used")+ if not args.filename:+ parser.error("--filename is required in single-query mode")+request = ImageSearchRequest(query=args.query,purpose=args.purpose,