Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

docs: v0.0.302.18 release notes measured with graff logged in

justrach7464d2c1e804
diff --git a/docs/releases/v0.0.302.18.md b/docs/releases/v0.0.302.18.md
index aa526ebb..7f6654fb 100644
--- a/docs/releases/v0.0.302.18.md
+++ b/docs/releases/v0.0.302.18.md
@@ -8,27 +8,31 @@ against Pi 1.0 with its codemode:
## Evals
On a ChatGPT plan, interleaved over 11 tasks (five MCP tasks, a git digest and
-five solo coding tasks), 3 runs each, 8 at a time. Cost is the list price of
-each run's tokens. Every arm passed every run.
+five solo coding tasks), 3 runs each, 8 at a time, with graff logged in to
+Codegraff as it is for its users. Cost is the list price of each run's
+tokens. Every arm passed every run.
### gpt-6.1-sol: ahead of Pi with codemode on time, cost and cache hits
| | Pass | Wall per task | Median | List cost per task | Cache hit | Output tokens per task |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| v0.0.302.17 | 33/33 | 38.8s | 35.0s | $0.0272 | 84.0% | 855 |
-| **v0.0.302.18** | 33/33 | **31.7s** | **25.3s** | **$0.0212** | **86.5%** | 687 |
-| Pi 1.0 + codemode | 33/33 | 33.7s | 27.9s | $0.0277 | 47.8% | 613 |
+| v0.0.302.17 | 33/33 | 40.8s | 37.5s | $0.0259 | 87.0% | 874 |
+| **v0.0.302.18** | 33/33 | **33.0s** | **26.8s** | **$0.0192** | **89.8%** | 701 |
+| Pi 1.0 + codemode | 33/33 | 34.2s | 29.0s | $0.0281 | 48.3% | 612 |
-### gpt-6-astra: 21% less time than the previous release, the lowest cost of the three
+### gpt-6-astra: 14% less time than the previous release, the lowest cost of the three
| | Pass | Wall per task | Median | List cost per task | Cache hit | Output tokens per task |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| v0.0.302.17 | 33/33 | 50.7s | 41.8s | $0.120 | 90.3% | 800 |
-| **v0.0.302.18** | 33/33 | **40.0s** | 31.8s | **$0.113** | 89.5% | 725 |
-| Pi 1.0 + codemode | 33/33 | 40.8s | 31.5s | $0.138 | 54.8% | 670 |
-
-On gpt-6-astra the two are level on time; graff's cost stays lower because most
-of its input is read from the cache.
+| v0.0.302.17 | 33/33 | 48.2s | 38.2s | $0.125 | 88.3% | 767 |
+| **v0.0.302.18** | 33/33 | 41.4s | **31.7s** | **$0.106** | **91.2%** | 693 |
+| Pi 1.0 + codemode | 33/33 | **40.7s** | 32.9s | $0.135 | 56.1% | 696 |
+
+On gpt-6-astra Pi is 0.7 seconds faster on the mean and graff's median run is
+faster; graff costs 21% less because most of its input is read from the
+cache. Logged in, graff also offers the model `jev_effort`, which asks the
+hosted Jev selector to choose the reasoning effort. Across 132 graff runs on
+the two models it was never called, so these are graff's default efforts.
## Shorter turns