Skip to content

利用 AI 大模型和自动化工作流,根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.

feat: add local SDXL image server (#1668)

Mohd Kashifea3b8eee66fc
diff --git a/config.example.toml b/config.example.toml
index 5e7be56e..86d4c011 100644
--- a/config.example.toml
+++ b/config.example.toml
@@ -189,6 +189,7 @@ metaso_minimax_run_timeout = 1800
# Images are generated on demand (one per script keyword, billed per image by
# your provider) and rendered into short zoom-effect video clips.
# Example: "https://api.openai.com/v1" or "http://127.0.0.1:8080/v1"
+# For a free local SDXL server, see docs/local-image-server/README.md.
openai_image_base_url = ""
# Optional for local gateways that need no auth. Leave empty to send requests
# without an Authorization header; provide one or more keys for cloud relays.
diff --git a/docs/local-image-server/README.md b/docs/local-image-server/README.md
new file mode 100644
index 00000000..d1d896d0
--- /dev/null
+++ b/docs/local-image-server/README.md
@@ -0,0 +1,85 @@
+# Local image server for `openai_image`
+
+Stock footage often has little to do with the script. The `openai_image` source fixes that by generating one image per script segment. This folder has a small server that does it on your own GPU, so there are no per-image costs.
+
+It exposes the OpenAI `/v1/images/generations` API and runs [RealVisXL V5.0 Lightning](https://huggingface.co/SG161222/RealVisXL_V5.0_Lightning), a photoreal SDXL model that needs about 6 steps per image.
+
+Tested on an RTX 5070 Ti (16 GB):
+
+- about 3.9 s per 832x1216 image over HTTP once warm (about 2 s of that is the model itself)
+- 5 to 7 minutes for a 50 s video with 8 scenes, including voice, Whisper subtitles and rendering; the online voiceover accounts for most of the variation
+
+## Setup
+
+```bash
+cd docs/local-image-server
+python -m venv .venv
+.venv/Scripts/pip install torch --index-url https://download.pytorch.org/whl/cu128
+.venv/Scripts/pip install -r requirements.txt
+.venv/Scripts/python server.py
+```
+
+`uvicorn server:app --port 7861` works the same way: the idle worker starts with the app either way.
+
+The first request downloads the fp16 weights, about 6.6 GB. The server listens on `http://127.0.0.1:7861`. A `size` that isn't `WIDTHxHEIGHT` with positive numbers gets a 422 response.
+
+Tests, from the repo root, with no GPU needed: `python -m pytest test/services/test_local_image_policy.py test/services/test_local_image_server.py`. The server tests run the real HTTP app with a stand-in pipeline and a fake Ollama.
+
+## config.toml
+
+```toml
+[app]
+video_source = "openai_image"
+openai_image_base_url = "http://127.0.0.1:7861/v1"
+openai_image_model = "realvisxl-v5-lightning"
+match_materials_to_script = true
+video_clip_duration = 6
+```
+
+`match_materials_to_script` asks the LLM for up to 8 keywords in narration order, which gives one scene per part of the script. Each image becomes one clip, so 8 keywords x 6 s covers about 48 s of narration. With shorter clips, scenes repeat to fill the gap.
+
+## Sharing the GPU with Ollama
+
+Measured on the RTX 5070 Ti: SDXL's weights take 6.7 GB, and painting one 832x1216 image peaks at 10.6 GB allocated. qwen3:14b in Ollama takes 9.0 GB. They don't fit on a 16 GB card together, so the server keeps one of them on the GPU at a time:
+
+- Before every image, it asks Ollama which models are loaded, unloads them, and waits until Ollama reports them gone. This runs on every request, not just the first, because MPT may load the LLM again between images or between back-to-back videos.
+- After each request it releases PyTorch's cached buffers. Without that, the process keeps about 14 GB reserved between images.
+- After `IMAGE_IDLE_SECONDS` (90 by default) without a request, a background worker moves SDXL to the CPU, so Ollama gets the whole GPU for the next script.
+
+The server can't stop Ollama from loading while an image is painting. Running LLM and image work at the same moment isn't supported: the next image request reclaims the GPU, but in the meantime both models share it. On Windows the driver spills the overflow into system memory, which is slow but didn't crash in testing. We haven't tested this case on Linux.
+
+## Options
+
+| Variable | Default | |
+|---|---|---|
+| `IMAGE_MODEL` | `SG161222/RealVisXL_V5.0_Lightning` | any SDXL repo with fp16 weights |
+| `IMAGE_STEPS` | `6` | |
+| `IMAGE_GUIDANCE` | `1.5` | |
+| `IMAGE_IDLE_SECONDS` | `90` | idle time before SDXL moves to the CPU |
+| `IMAGE_IDLE_CHECK_SECONDS` | `10` | how often the idle worker checks |
+| `IMAGE_PORT` | `7861` | used by `python server.py` |
+| `OLLAMA_URL` | `http://localhost:11434` | |
+| `OLLAMA_UNLOAD_TIMEOUT` | `30` | how long to wait for Ollama to release the GPU |
+| `CONTENT_POLICY` | `policy.toml` | path to a policy file, or `off` |
+
+## Content policy
+
+Scene keywords come from an LLM, so a video about religion or history can ask the image model for things that shouldn't be generated. Before each prompt reaches the model, `policy.toml` checks it. The policy is on by default, and a rule only acts when its keywords appear.
+
+| Rule | What it does |
+|---|---|
+| `god` | God is never shown as a figure. The scene becomes light breaking through clouds. |
+| `prophet_muhammad`, `islamic_prophets`, `companions_and_family_of_the_prophet` | No depiction, following Islamic practice. The scene becomes an empty landscape of the place mentioned (Mecca, Medina, Hira, Sinai, Jerusalem), or a desert at sunrise otherwise. |
+| `bahai_central_figures`, `sikh_gurus` | No depiction. The scene becomes a fitting place without people. |
+| `atrocities` | No fabricated "historical photos" of real atrocities. The scene becomes a candle memorial. |
+| `children_in_distress` | The scene becomes a symbolic image without people. |
+| `living_public_figures` | No photoreal faces of named office holders. The scene becomes a government building. |
+| `sacred_script` | Image models write fake Arabic, Hebrew, Sanskrit or Gurmukhi, so calligraphy becomes geometric patterns. |
+| `holy_books` | Holy books are shown closed, with no lettering. |
+| `revered_figures` | Jesus, Mary, the Buddha, Hindu deities and others are depicted in these traditions, so they're kept, framed reverently. |
+
+`islamic_prophets_english_names` is off by default. For Islamic content, switch it on so Jesus, Moses, Abraham and Noah are also shown as places rather than people.
+
+Swapping in a whole scene is deliberate. A negative prompt alone doesn't keep people out reliably: in testing it still produced figures in 2 of 6 prophet scenes. Raising the guidance strength to compensate burned the Lightning model's images.
+
+Known limits: keyword rules miss phrasings they don't list and can misfire on common names (Muhammad Ali, a person called Isa). `test/services/test_local_image_policy.py` covers the expected cases. Edit `policy.toml` to fit your audience.
diff --git a/docs/local-image-server/policy.py b/docs/local-image-server/policy.py
new file mode 100644
index 00000000..f0c30343
--- /dev/null
+++ b/docs/local-image-server/policy.py
@@ -0,0 +1,64 @@
+"""Content policy for scene prompts, loaded from policy.toml."""
+
+import re
+import tomllib
+from dataclasses import dataclass, field
+from pathlib import Path
+
+DEFAULT_PATH = Path(__file__).with_name("policy.toml")
+
+
+@dataclass
+class Result:
+ prompt: str
+ negative: str = ""
+ no_people: bool = False
+ rules: list = field(default_factory=list)
+
+
+class Policy:
+ def __init__(self, data):
+ self.negative = data.get("defaults", {}).get("negative", "")
+ self.places = [(re.compile(p["match"]), p["scene"]) for p in data.get("places", [])]
+ self.rules = []
+ for r in data.get("rules", []):
+ if not r.get("enabled", True):
+ continue
+ r = dict(r)
+ r["patterns"] = [re.compile(m) for m in r["match"]]
+ self.rules.append(r)
+
+ def apply(self, prompt):
+ result = Result(prompt=prompt, negative=self.negative)
+ for rule in self.rules:
+ if not any(p.search(result.prompt) for p in rule["patterns"]):
+ continue
+ result.rules.append(rule["name"])
+ if rule.get("negative"):
+ result.negative = ", ".join(filter(None, [result.negative, rule["negative"]]))
+ result.no_people = result.no_people or rule.get("no_people", False)
+ action = rule["action"]
+ if action == "replace":
+ scene = rule["scene"]
+ if rule.get("use_places"):
+ scene = next((s for p, s in self.places if p.search(result.prompt)), scene)
+ result.prompt = f"deserted, nobody present, {scene}" if rule.get("no_people") else scene
+ break
+ if action == "substitute":
+ for p in rule["patterns"]:
+ result.prompt = p.sub(rule["replacement"], result.prompt)
+ elif action == "append":
+ result.prompt = f"{result.prompt}, {rule['text']}"
+ if result.no_people:
+ result.negative = ", ".join(
+ filter(None, [result.negative, "person, people, human, man, woman, crowd, face, figure, silhouette"])
+ )
+ result.prompt = re.sub(r"\s+", " ", result.prompt).strip(" ,")
+ return result
+
+
+def load(path=None):
+ """Load a policy file. Returns None when path is "off"."""
+ if str(path).lower() == "off":
+ return None
+ return Policy(tomllib.loads(Path(path or DEFAULT_PATH).read_text(encoding="utf-8")))
diff --git a/docs/local-image-server/policy.toml b/docs/local-image-server/policy.toml
new file mode 100644
index 00000000..931b090a
--- /dev/null
+++ b/docs/local-image-server/policy.toml
@@ -0,0 +1,155 @@
+# Content policy for generated scenes.
+#
+# Patterns are Python regex and case-sensitive unless they start with (?i).
+# Rules run in order. The first matching "replace" rule ends the run.
+# replace swap the whole prompt for a safe scene
+# substitute replace only the matched words
+# append add guidance to the end of the prompt
+# Set enabled = false to switch a rule off.
+
+[defaults]
+negative = "blood, gore, corpses, wounds, nudity"
+
+# scenes for "replace" rules with use_places = true, first match wins
+[[places]]
+match = '(?i)\b(hira|cave)\b'
+scene = "rocky mountain cave entrance at dawn, soft light"
+
+[[places]]
+match = '(?i)\b(medina|madinah?|yathrib)\b'
+scene = "palm oasis with mud brick houses at golden hour"
+
+[[places]]
+match = '(?i)\b(mecca|makkah|kaaba|ka.bah)\b'
+scene = "arabian desert valley between rocky mountains at dawn"
+
+[[places]]
+match = '(?i)\b(jerusalem|al-?aqsa)\b'
+scene = "ancient stone city walls at sunrise"
+
+[[places]]
+match = '(?i)\b(sinai)\b'
+scene = "granite desert mountains at sunrise"
+
+# --- never depicted ---
+
+[[rules]]
+name = "god"
+# God is not shown as a figure in Islam, Judaism or most of Christianity
+action = "replace"
+match = ['\bGod\b', '(?i)\b(allah|yahweh|jehovah|the almighty)\b']
+scene = "rays of soft light breaking through clouds over a vast quiet landscape"
+no_people = true
+
+[[rules]]
+name = "prophet_muhammad"
+action = "replace"
+match = [
+ '''(?i)\b(muhammad|mohammed|mohammad|mohamed)(['’]?s)?\b''',
+ '\b[Tt]he (Holy )?Prophet\b',
+ '(?i)\b(messenger of (god|allah)|rasul(ullah)?|rasool)\b',
+]
+use_places = true
+scene = "vast desert landscape with distant mountains at sunrise, soft light"
+no_people = true
+
+[[rules]]
+name = "islamic_prophets"
+# arabic names imply islamic framing, where prophets are not depicted
+action = "replace"
+match = [
+ '(?i)\bprophets?\s+(isa|nuh|ibrahim|musa|yusuf|dawud|sulayman|yunus|ismail|yahya|zakariy+a|adam|idris|hud|salih|shu.?ayb)\b',
+ # bare names are common muslim names, so only with islamic context
+ '(?i)(?<!mansa )\b(isa|nuh|ibrahim|musa|yusuf|dawud|sulayman|yunus|yahya|zakariy+a|idris)\b(?=.*\b(islam\w*|qur.?an\w*|muslims?)\b)',
+]
+use_places = true
+scene = "vast desert landscape with distant mountains at sunrise, soft light"
+no_people = true
+
+[[rules]]
+name = "islamic_prophets_english_names"
+# for islamic content: Jesus, Moses, Abraham and Noah are prophets in Islam too.
+# off by default because Christian and Jewish art depicts them
+enabled = false
+action = "replace"
+match = ['(?i)\b(jesus|christ|moses|abraham|noah|joseph|david|solomon|jonah|john the baptist)\b']
+use_places = true
+scene = "vast desert landscape with distant mountains at sunrise, soft light"
+no_people = true
+
+[[rules]]
+name = "companions_and_family_of_the_prophet"
+# many muslims also avoid depicting the companions and the prophet's family
+action = "replace"
+match = [
+ '(?i)\b(sahaba|companions of the prophet|rightly guided caliphs?)\b',
+ '(?i)\b(abu bakr|umar ibn al.?khattab|caliph umar|uthman ibn affan|caliph uthman|ali ibn abi talib|imam ali|imam hu?s[as]+[ae]?in|khadijah?|fatimah? (al.?)?zahra|aisha bint abi bakr)\b',
+]
+use_places = true
+scene = "vast desert landscape with distant mountains at sunrise, soft light"
+no_people = true
+
+[[rules]]
+name = "bahai_central_figures"
+action = "replace"
+match = ['''(?i)\b(the b[aá]b|bah[aá]['’]?u['’]?ll[aá]h)\b''']
+scene = "terraced gardens descending a mountainside at sunrise"
+no_people = true
+
+[[rules]]
+name = "sikh_gurus"
+# images of the gurus exist, but many sikhs object to them
+action = "replace"
+match = ['(?i)\b(sikh gurus?|guru (nanak|angad|amar das|ram das|arjan|hargobind|har rai|har krishan|tegh bahadur|gobind singh))\b']
+scene = "golden temple reflected in the still pool at dawn"
+no_people = true
+
+# --- real suffering ---
+
+[[rules]]
+name = "atrocities"
+# a photoreal "historical photo" of a real atrocity is a fabrication
+action = "replace"
+match = ['(?i)\b(holocaust|shoah|auschwitz|concentration camps?|death camps?|genocides?|massacres?|ethnic cleansing|lynching|slave ships?|terrorist attacks?|9/11|september 11|hiroshima bombing|nakba)\b']
+scene = "memorial with candles and flowers at dusk, solemn and quiet"
+no_people = true
+
+[[rules]]
+name = "children_in_distress"
+action = "replace"
+match = ['(?i)\b(child|children|kids?|orphans?|bab(y|ies)|toddlers?)\b.{0,40}\b(starving|crying|wounded|injured|dying|dead|abused|suffering|malnourished)\b', '(?i)\b(starving|crying|wounded|injured|dying|dead|abused|suffering|malnourished)\b.{0,40}\b(child|children|kids?|orphans?|bab(y|ies)|toddlers?)\b']
+scene = "a small worn toy on an empty dusty road at dusk"
+no_people = true
+
+# --- softer fixes ---
+
+[[rules]]
+name = "living_public_figures"
+# no photoreal faces of real office holders
+action = "replace"
+match = ['''\b(?i:president|prime minister|chancellor|senator|governor|minister|pope|ceo|secretary.general)\s+[A-Z][\w'-]+(\s+[A-Z][\w'-]+)?''']
+scene = "government building exterior with flags at dusk"
+no_people = true
+
+[[rules]]
+name = "sacred_script"
+# image models write fake arabic, hebrew, sanskrit or gurmukhi
+action = "substitute"
+match = ['(?i)\b((arabic|hebrew|sanskrit|gurmukhi|tibetan|devanagari) )?(calligraph\w*|inscriptions?|scripture text|holy verses?|verses)\b']
+replacement = "ornate geometric patterns"
+negative = "calligraphy, script, lettering"
+
+[[rules]]
+name = "holy_books"
+action = "append"
+match = ['''(?i)\b(qur.?an|koran|torah|bible|gospels?|guru granth sahib|vedas?|bhagavad gita|sutras?|tripitaka|holy books?|scriptures?)\b''']
+text = "the holy book shown closed with a plain cover decorated only with geometric ornament, resting on an ornate stand"
+negative = "open book, pages, writing, lettering on the cover, emblem with text"
+
+[[rules]]
+name = "revered_figures"
+# depicted in these traditions, so keep them but frame them with respect
+action = "append"
+match = ['(?i)\b(jesus|christ|virgin mary|madonna|saints?|moses|abraham|buddha|bodhisattvas?|shiva|vishnu|krishna|rama|ganesh[a]?|durga|kali|lakshmi|hanuman|saraswati|parvati|mahavira)\b']
+text = "depicted reverently in a traditional devotional setting, dignified"
+negative = "cartoon, comic, sexualized, revealing clothing, mocking, toy, on shoes"
diff --git a/docs/local-image-server/requirements.txt b/docs/local-image-server/requirements.txt
new file mode 100644
index 00000000..27e035bf
--- /dev/null
+++ b/docs/local-image-server/requirements.txt
@@ -0,0 +1,10 @@
+# install torch for your gpu first, e.g. CUDA 12.8:
+# pip install torch --index-url https://download.pytorch.org/whl/cu128
+diffusers
+transformers
+accelerate
+safetensors
+fastapi
+uvicorn
+pillow
+requests
diff --git a/docs/local-image-server/server.py b/docs/local-image-server/server.py
new file mode 100644
index 00000000..13a75cf8
--- /dev/null
+++ b/docs/local-image-server/server.py
@@ -0,0 +1,243 @@
+"""OpenAI-compatible /v1/images/generations server running SDXL Lightning locally."""
+
+import base64
+import io
+import os
+import random
+import threading
+import time
+from contextlib import asynccontextmanager
+
+import requests
+import uvicorn
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel, field_validator
+
+import policy
+
+MODEL = os.environ.get("IMAGE_MODEL", "SG161222/RealVisXL_V5.0_Lightning")
+OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434")
+STEPS = int(os.environ.get("IMAGE_STEPS", "6"))
+GUIDANCE = float(os.environ.get("IMAGE_GUIDANCE", "1.5"))
+IDLE_SECONDS = float(os.environ.get("IMAGE_IDLE_SECONDS", "90"))
+IDLE_CHECK_SECONDS = float(os.environ.get("IMAGE_IDLE_CHECK_SECONDS", "10"))
+OLLAMA_UNLOAD_TIMEOUT = float(os.environ.get("OLLAMA_UNLOAD_TIMEOUT", "30"))
+HOST = os.environ.get("IMAGE_HOST", "127.0.0.1")
+PORT = int(os.environ.get("IMAGE_PORT", "7861"))
+POLICY = policy.load(os.environ.get("CONTENT_POLICY"))
+
+STYLE = (
+ "candid cinematic photograph, natural light, highly detailed, sharp focus, "
+ "documentary style, 35mm film"
+)
+NEGATIVE = (
+ "text, letters, words, watermark, logo, signature, caption, deformed, "
+ "distorted, bad anatomy, extra fingers, blurry, lowres, cartoon, 3d render, "
+ "eyeglasses, modern clothing, map, looking at camera, posed portrait"
+)
+
+# sdxl sizes near 1 megapixel
+BUCKETS = [
+ (1024, 1024), (896, 1152), (832, 1216), (768, 1344), (640, 1536),
+ (1152, 896), (1216, 832), (1344, 768), (1536, 640),
+]
+DEFAULT_SIZE = (832, 1216)
+
+# one lock guards the pipeline and the gpu state below
+lock = threading.Lock()
+pipe = None
+on_gpu = False
+last_used = 0.0
+stop_idle = threading.Event()
+idle_thread = None
+
+
+class ImageRequest(BaseModel):
+ prompt: str
+ model: str | None = None
+ n: int = 1
+ size: str | None = "1024x1536"
+ response_format: str | None = "b64_json"
+
+ @field_validator("size")
+ @classmethod
+ def check_size(cls, v):
+ if v is None or v.strip().lower() in ("", "auto"):
+ return None
+ w, sep, h = v.strip().lower().partition("x")
+ if not (sep and w.isdigit() and h.isdigit() and int(w) > 0 and int(h) > 0):
+ raise ValueError("size must be WIDTHxHEIGHT with positive integers, e.g. 1024x1536")
+ return v.strip().lower()
+
+
+def pick_size(size):
+ if size is None:
+ return DEFAULT_SIZE
+ w, h = (int(v) for v in size.split("x"))
+ return min(BUCKETS, key=lambda b: abs(b[0] / b[1] - w / h))
+
+
+def build_prompts(prompt):
+ negative = NEGATIVE
+ if POLICY:
+ r = POLICY.apply(prompt)
+ prompt, negative = r.prompt, f"{NEGATIVE}, {r.negative}"
+ if r.rules:
+ print(f"policy {','.join(r.rules)}: {prompt}")
+ return f"{prompt}, {STYLE}", negative
+
+
+def ollama_models():
+ try:
+ r = requests.get(f"{OLLAMA_URL}/api/ps", timeout=5)
+ return [m["name"] for m in r.json().get("models", [])]
+ except (requests.RequestException, ValueError):
+ return []
+
+
+def unload_ollama():
+ # ollama can load again between requests, so this runs before every image
+ names = ollama_models()
+ for name in names:
+ try:
+ requests.post(
+ f"{OLLAMA_URL}/api/generate",
+ json={"model": name, "keep_alive": 0},
+ timeout=30,
+ )
+ except requests.RequestException:
+ pass
+ if not names:
+ return []
+ # wait until ollama reports the models gone, so their vram is free
+ deadline = time.time() + OLLAMA_UNLOAD_TIMEOUT
+ while (left := ollama_models()) and time.time() < deadline:
+ time.sleep(0.5)
+ if left:
+ print(f"ollama still holds {', '.join(left)} after {OLLAMA_UNLOAD_TIMEOUT:g}s, painting anyway")
+ else:
+ print(f"unloaded ollama: {', '.join(names)}")
+ return names
+
+
+def load_pipeline():
+ import torch
+ from diffusers import DPMSolverMultistepScheduler, StableDiffusionXLPipeline
+
+ p = StableDiffusionXLPipeline.from_pretrained(MODEL, torch_dtype=torch.float16, variant="fp16")
+ p.scheduler = DPMSolverMultistepScheduler.from_config(
+ p.scheduler.config,
+ algorithm_type="sde-dpmsolver++",
+ use_karras_sigmas=True,
+ )
+ return p
+
+
+def empty_cuda_cache():
+ try:
+ import torch
+ except ImportError:
+ return
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+
+
+def claim_gpu():
+ """Make SDXL the only large model on the gpu. Call with the lock held."""
+ global pipe, on_gpu
+ unload_ollama()
+ if pipe is None:
+ pipe = load_pipeline()
+ if not on_gpu:
+ pipe.to("cuda")
+ on_gpu = True
+ return pipe
+
+
+def paint(p, prompt, negative, width, height):
+ import torch
+
+ return p(
+ prompt=prompt,
+ negative_prompt=negative,
+ width=width,
+ height=height,
+ num_inference_steps=STEPS,
+ guidance_scale=GUIDANCE,
+ generator=torch.Generator("cuda").manual_seed(random.randint(0, 2**31 - 1)),
+ ).images[0]
+
+
+def offload_if_idle():
+ """Move SDXL back to the cpu once it has been idle long enough."""
+ global on_gpu
+ with lock:
+ if not on_gpu or time.time() - last_used < IDLE_SECONDS:
+ return False
+ pipe.to("cpu")
+ empty_cuda_cache()
+ on_gpu = False
+ print("idle: moved pipeline to cpu")
+ return True
+
+
+def idle_worker():
+ while not stop_idle.wait(IDLE_CHECK_SECONDS):
+ offload_if_idle()
+
+
+@asynccontextmanager
+async def lifespan(app):
+ # runs under both `python server.py` and `uvicorn server:app`
+ global idle_thread
+ stop_idle.clear()
+ idle_thread = threading.Thread(target=idle_worker, name="idle-offload", daemon=True)
+ idle_thread.start()
+ try:
+ yield
+ finally:
+ stop_idle.set()
+ idle_thread.join(timeout=5)
+
+
+app = FastAPI(lifespan=lifespan)
+
+
+def health():
+ return {
+ "ok": True,
+ "on_gpu": on_gpu,
+ "idle_worker": bool(idle_thread and idle_thread.is_alive()),
+ }
+
+
[email protected]("/v1/models")
+def models():
+ return {"data": [{"id": MODEL, "object": "model"}]}
+
+
[email protected]("/v1/images/generations")
+def generate(req: ImageRequest):
+ global last_used
+ if not req.prompt.strip():
+ raise HTTPException(400, "prompt is empty")
+ width, height = pick_size(req.size)
+ prompt, negative = build_prompts(req.prompt)
+ data = []
+ with lock:
+ for _ in range(max(1, min(req.n, 4))):
+ # reclaim per image: ollama may have loaded while the last one painted
+ p = claim_gpu()
+ image = paint(p, prompt, negative, width, height)
+ buf = io.BytesIO()
+ image.save(buf, format="PNG")
+ data.append({"b64_json": base64.b64encode(buf.getvalue()).decode()})
+ # torch keeps ~7 GB of decode buffers cached; give it back so ollama fits
+ empty_cuda_cache()
+ last_used = time.time()
+ return {"created": int(time.time()), "data": data}
+
+
+if __name__ == "__main__":
+ uvicorn.run(app, host=HOST, port=PORT)
diff --git a/test/services/test_local_image_policy.py b/test/services/test_local_image_policy.py
new file mode 100644
index 00000000..c02eec02
--- /dev/null
+++ b/test/services/test_local_image_policy.py
@@ -0,0 +1,84 @@
+import importlib.util
+import sys
+from pathlib import Path
+
+import pytest
+
+# policy.py lives with the example server in docs/
+_path = Path(__file__).resolve().parents[2] / "docs" / "local-image-server" / "policy.py"
+_spec = importlib.util.spec_from_file_location("local_image_policy", _path)
+policy = importlib.util.module_from_spec(_spec)
+sys.modules[_spec.name] = policy
+_spec.loader.exec_module(policy)
+
+P = policy.load()
+
+
+ "prompt, rule, scene",
+ [
+ ("Prophet Muhammad in Mecca", "prophet_muhammad", "desert valley"),
+ ("the Prophet's mosque in Medina", "prophet_muhammad", "palm oasis"),
+ ("Muhammad's teachings", "prophet_muhammad", "desert landscape"),
+ ("Prophet Muhammad receiving revelation in cave Hira", "prophet_muhammad", "cave entrance"),
+ ("God creating the world", "god", "rays of soft light"),
+ ("Allah's mercy", "god", "rays of soft light"),
+ ("Prophet Musa at Mount Sinai", "islamic_prophets", "granite desert mountains"),
+ ("Yusuf in the Quran", "islamic_prophets", "desert landscape"),
+ ("Abu Bakr leading the community", "companions_and_family_of_the_prophet", "desert landscape"),
+ ("Baha'u'llah in exile", "bahai_central_figures", "terraced gardens"),
+ ("Guru Nanak travelling", "sikh_gurus", "golden temple"),
+ ("Holocaust survivors", "atrocities", "memorial with candles"),
+ ("starving children in a refugee camp", "children_in_distress", "worn toy"),
+ ("President Biden giving a speech", "living_public_figures", "government building"),
+ ],
+)
+def test_replaced_with_people_free_scene(prompt, rule, scene):
+ r = P.apply(prompt)
+ assert rule in r.rules
+ assert scene in r.prompt
+ assert r.no_people
+ assert "person" in r.negative
+
+
+ "prompt, rule, expected",
+ [
+ ("Jesus teaching on a hill", "revered_figures", "depicted reverently"),
+ ("Moses parting the Red Sea", "revered_figures", "depicted reverently"),
+ ("Lord Shiva meditating", "revered_figures", "depicted reverently"),
+ ("Buddha statue in a temple", "revered_figures", "depicted reverently"),
+ ("Quran on a wooden stand", "holy_books", "shown closed"),
+ ("Arabic calligraphy in a mosque", "sacred_script", "ornate geometric patterns in a mosque"),
+ ],
+)
+def test_kept_but_adjusted(prompt, rule, expected):
+ r = P.apply(prompt)
+ assert rule in r.rules
+ assert expected in r.prompt
+ assert not r.no_people
+
+
+ "prompt",
+ [
+ "Mansa Musa pilgrimage to Mecca",
+ "statue of the Greek god Zeus",
+ "Isaac Newton under an apple tree",
+ "Battle of Badr cavalry charge",
+ "spice market in medieval Baghdad",
+ ],
+)
+def test_unrelated_prompts_unchanged(prompt):
+ r = P.apply(prompt)
+ assert r.rules == []
+ assert r.prompt == prompt
+ assert "gore" in r.negative
+
+
+def test_shared_prophets_off_by_default():
+ assert "islamic_prophets_english_names" not in [r["name"] for r in P.rules]
+
+
+def test_policy_can_be_switched_off():
+ assert policy.load("off") is None
diff --git a/test/services/test_local_image_server.py b/test/services/test_local_image_server.py
new file mode 100644
index 00000000..d9cd0af7
--- /dev/null
+++ b/test/services/test_local_image_server.py
@@ -0,0 +1,216 @@
+import importlib.util
+import os
+import socket
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+import pytest
+import requests
+from fastapi.testclient import TestClient
+from PIL import Image
+
+# the example server lives in docs/ and imports policy.py from its own folder
+SERVER_DIR = Path(__file__).resolve().parents[2] / "docs" / "local-image-server"
+sys.path.insert(0, str(SERVER_DIR))
+_spec = importlib.util.spec_from_file_location("local_image_server", SERVER_DIR / "server.py")
+srv = importlib.util.module_from_spec(_spec)
+sys.modules[_spec.name] = srv
+_spec.loader.exec_module(srv)
+
+
+class FakePipe:
+ def __init__(self):
+ self.devices = []
+
+ def to(self, device):
+ self.devices.append(device)
+ return self
+
+
+class FakeOllama:
+ """Stands in for ollama's /api/ps and /api/generate."""
+
+ def __init__(self, models=()):
+ self.models = list(models)
+ self.unloaded = []
+
+ def get(self, url, timeout=None):
+ assert url.endswith("/api/ps")
+ models = [{"name": m} for m in self.models]
+ return type("Resp", (), {"json": lambda _self: {"models": models}})()
+
+ def post(self, url, json=None, timeout=None):
+ assert url.endswith("/api/generate") and json["keep_alive"] == 0
+ self.models.remove(json["model"])
+ self.unloaded.append(json["model"])
+
+
+def server(monkeypatch):
+ ollama = FakeOllama()
+ pipe = FakePipe()
+ painted = []
+
+ def fake_paint(p, prompt, negative, width, height):
+ # ollama must be gone before every image
+ painted.append({"size": (width, height), "ollama": list(ollama.models)})
+ return Image.new("RGB", (8, 8))
+
+ monkeypatch.setattr(srv, "pipe", None)
+ monkeypatch.setattr(srv, "on_gpu", False)
+ monkeypatch.setattr(srv, "last_used", 0.0)
+ monkeypatch.setattr(srv, "IDLE_SECONDS", 3600)
+ monkeypatch.setattr(srv, "IDLE_CHECK_SECONDS", 0.05)
+ monkeypatch.setattr(srv, "OLLAMA_UNLOAD_TIMEOUT", 1)
+ monkeypatch.setattr(srv, "load_pipeline", lambda: pipe)
+ monkeypatch.setattr(srv, "paint", fake_paint)
+ monkeypatch.setattr(srv.requests, "get", ollama.get)
+ monkeypatch.setattr(srv.requests, "post", ollama.post)
+ with TestClient(srv.app) as client:
+ yield type("Server", (), {"client": client, "ollama": ollama, "pipe": pipe, "painted": painted})
+
+
+def post_image(client, **body):
+ return client.post("/v1/images/generations", json={"prompt": "mountain", **body})
+
+
[email protected]("size", ["1024x0", "0x1024", "-1x1024", "1024", "axb", "1024x1536x2", "x"])
+def test_invalid_size_is_rejected(server, size):
+ r = post_image(server.client, size=size)
+ assert r.status_code == 422
+ assert server.painted == []
+
+
+ "size, expected",
+ [("1024x1536", (832, 1216)), ("1920x1080", (1344, 768)), ("1024x1024", (1024, 1024)), ("auto", (832, 1216)), (None, (832, 1216))],
+)
+def test_valid_size_picks_sdxl_bucket(server, size, expected):
+ r = post_image(server.client, size=size)
+ assert r.status_code == 200
+ assert r.json()["data"][0]["b64_json"]
+ assert server.painted[0]["size"] == expected
+
+
+def test_cuda_cache_is_released_after_each_request(server, monkeypatch):
+ calls = []
+ monkeypatch.setattr(srv, "empty_cuda_cache", lambda: calls.append(1))
+ assert post_image(server.client).status_code == 200
+ assert post_image(server.client).status_code == 200
+ assert len(calls) == 2
+
+
+def test_ollama_is_unloaded_before_every_image(server):
+ server.ollama.models = ["qwen3:14b"]
+ assert post_image(server.client).status_code == 200
+
+ # ollama loads again while sdxl is still on the gpu
+ server.ollama.models = ["qwen3:14b"]
+ assert post_image(server.client).status_code == 200
+
+ assert server.ollama.unloaded == ["qwen3:14b", "qwen3:14b"]
+ assert [p["ollama"] for p in server.painted] == [[], []]
+ assert server.pipe.devices == ["cuda"]
+
+
+def test_ollama_unloaded_between_images_in_one_request(server, monkeypatch):
+ painted = []
+
+ def paint_while_ollama_loads(p, prompt, negative, width, height):
+ painted.append(list(server.ollama.models))
+ server.ollama.models = ["qwen3:14b"] # ollama loads during this image
+ return Image.new("RGB", (8, 8))
+
+ monkeypatch.setattr(srv, "paint", paint_while_ollama_loads)
+ r = post_image(server.client, n=3)
+ assert r.status_code == 200 and len(r.json()["data"]) == 3
+ assert painted == [[], [], []]
+ assert server.ollama.unloaded == ["qwen3:14b", "qwen3:14b"]
+
+
+def test_image_painted_when_ollama_is_not_running(server, monkeypatch):
+ def refuse(url, timeout=None):
+ raise requests.ConnectionError("connection refused")
+
+ monkeypatch.setattr(srv.requests, "get", refuse)
+ assert post_image(server.client).status_code == 200
+ assert len(server.painted) == 1
+
+
+def test_image_painted_when_ollama_will_not_unload(server, monkeypatch):
+ server.ollama.models = ["qwen3:14b"]
+ monkeypatch.setattr(server.ollama, "post", lambda url, json=None, timeout=None: None)
+ monkeypatch.setattr(srv.requests, "post", server.ollama.post)
+ monkeypatch.setattr(srv, "OLLAMA_UNLOAD_TIMEOUT", 0.3)
+ start = time.time()
+ assert post_image(server.client).status_code == 200
+ assert time.time() - start < 5
+ assert server.painted[0]["ollama"] == ["qwen3:14b"]
+
+
+def test_ollama_reloaded_after_idle_offload(server, monkeypatch):
+ assert post_image(server.client).status_code == 200
+ monkeypatch.setattr(srv, "IDLE_SECONDS", 0)
+ assert srv.offload_if_idle()
+
+ server.ollama.models = ["qwen3:14b"]
+ monkeypatch.setattr(srv, "IDLE_SECONDS", 3600)
+ assert post_image(server.client).status_code == 200
+
+ assert server.ollama.unloaded == ["qwen3:14b"]
+ assert server.painted[-1]["ollama"] == []
+ assert server.pipe.devices == ["cuda", "cpu", "cuda"]
+
+
+def test_idle_worker_offloads_under_lifespan(server, monkeypatch):
+ assert server.client.get("/health").json()["idle_worker"] is True
+ assert post_image(server.client).status_code == 200
+ assert srv.on_gpu
+
+ monkeypatch.setattr(srv, "IDLE_SECONDS", 0)
+ deadline = time.time() + 5
+ while srv.on_gpu and time.time() < deadline:
+ time.sleep(0.05)
+ assert not srv.on_gpu
+ assert server.pipe.devices == ["cuda", "cpu"]
+
+
+def test_idle_worker_stops_on_shutdown(monkeypatch):
+ monkeypatch.setattr(srv, "IDLE_CHECK_SECONDS", 0.05)
+ with TestClient(srv.app):
+ thread = srv.idle_thread
+ assert thread.is_alive()
+ assert not thread.is_alive()
+
+
+def free_port():
+ with socket.socket() as s:
+ s.bind(("127.0.0.1", 0))
+ return s.getsockname()[1]
+
+
[email protected]("launch", ["script", "asgi"])
+def test_idle_worker_starts_in_both_launch_modes(launch):
+ port = free_port()
+ if launch == "script":
+ cmd = [sys.executable, "server.py"]
+ else:
+ cmd = [sys.executable, "-m", "uvicorn", "server:app", "--host", "127.0.0.1", "--port", str(port)]
+ env = {**os.environ, "IMAGE_PORT": str(port), "OLLAMA_URL": "http://127.0.0.1:9"}
+ proc = subprocess.Popen(cmd, cwd=SERVER_DIR, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ try:
+ health = None
+ deadline = time.time() + 30
+ while time.time() < deadline:
+ try:
+ health = requests.get(f"http://127.0.0.1:{port}/health", timeout=1).json()
+ break
+ except requests.RequestException:
+ time.sleep(0.2)
+ assert health is not None, "server did not start"
+ assert health["idle_worker"] is True
+ finally:
+ proc.terminate()
+ proc.wait(timeout=10)