GLSL and Shadertoy shader skills for Claude Code, Codex, Cursor, Copilot and Gemini CLI. 186 functions, SDF raymarching, procedural terrain, water, clouds and generative art, with a bundler, checker and 11 examples.
Evals
Does the shader-art-coding skill make Claude's shaders better? This suite measures it with
claude plugin eval. Each case runs with this plugin loaded and
again with no plugin, and the report shows both scores and their difference, Δ.
Running it costs money. Every agent run and every judge call is a real model call on your account: billed to your API key, or counted against your plan. Validating the files is free; see Validate.
Cases
Six positive cases should trigger the skill. Three near-miss negatives sound shader-adjacent but should not.
| Case | Kind | The prompt asks for | Case-specific graders |
|---|---|---|---|
sdf-still-life |
positive | a raymarched bowl and fruit on a table | |
terrain-flyover |
positive | a camera flight over eroded desert mountains at sunset | |
cloud-layer |
positive | sunlit cumulus clouds drifting over a blue sky | |
poster-2d |
positive | a flat 2D poster: sun disc, mountain silhouettes, frame, halftone | |
fix-twist-holes |
positive, repair | a fix for a twisted column with holes. The prompt includes the shader, whose map() twists a box at 3 rad/unit with no Lipschitz correction |
twist-corrected (judge) |
port-webgl1 |
positive, repair | a port of a WebGL1 shader that uses #version 100, uniforms, texture2D, gl_FragColor and a sine hash |
samples-ichannel0, no-webgl1-io |
neg-css-gradient |
negative | a "shader demo" animated gradient in plain CSS | css-gradient, css-keyframes |
neg-hlsl-compute |
negative | a Unity HLSL compute shader for a separable Gaussian blur | hlsl-kernels, no-glsl |
neg-matplotlib-noise |
negative | a Perlin-noise heatmap with matplotlib | uses-matplotlib, saves-figure |
Each positive prompt asks for the finished shader in final.glsl. Every positive case carries these five graders:
| Grader | Check | Counts toward the score |
|---|---|---|
mainimage |
final.glsl exists and has void mainImage(out vec4 …, in vec2 …) |
yes, both arms |
no-banned-constructs |
final.glsl has no gl_FragColor, texture2D(, #version or fract(sin( outside // comments |
yes, both arms |
ran-check |
the agent ran glsl_tools.py check on a .glsl file |
no: with-plugin indicator (arm: with-only) |
check-clean |
the last checker output in the transcript was 0 error(s) |
no: with-plugin indicator (arm: with-only) |
skill-fired |
Claude invoked the shader-art-coding skill |
no: excluded automatically, like every tool_used: Skill grader |
Each negative case has skill-not-fired (tool_used: Skill, min: 0, max: 0, arm: both), which counts in both
arms. Its other graders check that the agent did the non-shader task it was given.
Why three positive graders do not count toward the score
ran-check, check-clean and skill-fired can never pass without the plugin, because without it the script and the
skill don't exist. Counting them would push the no-plugin score toward zero and inflate Δ. So Δ on positives
comes only from the output graders, and the indicators show in the with-plugin runs as the skill-trigger rate and the
clean-check rate. Under --ablation none all graders count.
What "compile rate" means here
claude plugin eval has no custom-code graders, so a grader cannot compile the shader. The scored graders catch the
constructs that break Shadertoy and WebGL2 in both arms. A real glslang compile shows up only in the with-plugin
arm, through check-clean, and only when glslangValidator is on PATH; without it, check only lints. To get a
compile rate for both arms, run with --keep-temp and run
python3 skills/shader-art-coding/scripts/glsl_tools.py check <run dir>/final.glsl on each kept run.
Requirements
- Claude Code v2.1.269 or later (
claude --version) and git 2.31 or later. - Credentials: export
ANTHROPIC_API_KEY, or use the account or cloud-provider variables your normal sessions use. - Recommended:
glslangValidator(brew install glslangorapt install glslang-tools), so the skill's checker compiles as well as lints. - Linux:
bubblewrapandsocatfor the Bash sandbox. On Ubuntu 24.04 and later, bubblewrap also needs the AppArmor profile from the sandboxing docs. The CI workflow installs it.
Validate (free)
python3 tests/check_evals.py
This checks every prompt.md and grader against the documented fields and this suite's conventions. It also runs
the grader regexes on real inputs: the gallery shaders, the fixture shaders in the prompts, and real
glsl_tools.py check output. It makes no model calls. CI runs it in checks.yml.
Run
Run from the repository root, which is the plugin root. Put the target . first and --allow-tools last, since it
takes a list. The cases need Write and Edit to save their output and Bash for the skill's checker. Without the
grant those tools are removed from the session and the positives fail. Granted Bash runs under the OS sandbox.
Smoke test: one case, one run, no baseline.
claude plugin eval . --case sdf-still-life --runs 1 --ablation none --max-cost-usd 2 --allow-tools Write Edit Bash
For scale: on the default model, one run of port-webgl1 took 23 s and cost $0.15, and one run of
neg-css-gradient took 30 s and cost $0.15.
Full comparison: 9 cases × 3 runs × 2 arms = 54 agent sessions. Positive runs allow up to 40 turns. Add 3 judge calls
for each of the 6 fix-twist-holes runs. Set a ceiling you're comfortable with:
claude plugin eval . --model claude-sonnet-5-5 --judge-model claude-haiku-5-5 --threshold 0.8 --max-cost-usd 30 --allow-tools Write Edit Bash
Filters: --tag positive, --tag negative, --tag repair, or --case 'neg-*'. The first run in a terminal asks
Trust this plugin directory?. Answer y, or pass --trust-plugin.
Results go to evals/results/<timestamp>/, which is gitignored: report.html plus aggregate-result.json. When you're
signed in with a claude.ai subscription that has artifacts, the report is also published privately and the command
prints a Published: link. Pass
--no-publish to keep it local.
Reading the result
- Positives:
WITHis the share of output graders passed with the skill, andW/OUTis the same without it. A positiveΔmeans the skill improved output-rule adherence. Open the report to see the with-pluginskill-fired,ran-checkandcheck-cleanindicators per run. - Negatives:
Δbelow 0 means the skill fired, or derailed the task, where it shouldn't have. - Exit code: the default
--thresholdis1.0, so the command exits 1 when any case scores below perfect.Δnever changes the exit code.
CI
.github/workflows/evals.yml runs the suite on manual dispatch only. Add an
ANTHROPIC_API_KEY repository secret, then go to Actions → evals → Run workflow. The inputs set the case glob, runs,
models, threshold and cost ceiling. The job uploads eval-results (report and JSON) and writes a per-case table to
the run summary.
Adding a case
- Create
evals/<name>/prompt.md. Use only the documented frontmatter fields (description,tags,runs,expected_outcome,model,max_turns,timeout_seconds,allowed_tools,append_system_prompt,env, …). Double-quote any value that contains:or#. Write the body the way a user would type it, and name the output file in backticks. - Tag the case
positiveornegative. Copy the five shared graders into a positive case unchanged; a negative case needsskill-not-fired.md. Add graders for the case's own outcome. Prefer regex graders over long files and keepllmgraders for short outputs. - Run
python3 tests/check_evals.py, then a smoke run of the new case.
Troubleshooting
not grantedin the progress output: add--allow-tools Write Edit Bash, after the target.- No
W/OUTcolumn, or "ablation requested but no plugin resolved": addplugins: ["../.."]to the case'sprompt.md. ran-checkpasses, butcheck-cleanfails and the transcript shows a permission error onglsl_tools.py: eval runs make your home directory unreadable to shell commands, so the sandbox may block the plugin's script when the checkout lives under your home directory. Copy the repository outside it (for example to/tmp/shader-skills) and run from there. The CI workflow does the same. On macOS with v2.1.299, a checkout under/Usersworked: the plugin directory stayed readable to sandboxed Bash.couldn't create cache file '…/xcrun_db-…' (errno=Operation not permitted)in a macOS transcript: the/usr/bin/python3shim tries to write an Xcode cache that the sandbox blocks. The checker still runs and prints itsN error(s)line, so the noise does not affect any grader.check-cleanfails although the checker passed: the regex reads the whole transcript. If Claude's own text after the last clean check quotes an earlier non-zero count like2 error(s), the grader fails.