Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(tuning): add tuning foundations guide (#1199)

sabhatinascf92c11354d7
diff --git a/docs/assets/tuning/configuration-pareto.svg b/docs/assets/tuning/configuration-pareto.svg
new file mode 100644
index 000000000..bb5ed830e
--- /dev/null
+++ b/docs/assets/tuning/configuration-pareto.svg
@@ -0,0 +1,328 @@
+<?xml version="1.0" encoding="utf-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="633.6pt" height="345.6pt" viewBox="0 0 633.6 345.6" xmlns="http://www.w3.org/2000/svg" version="1.1">
+ <metadata>
+ <rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
+ <cc:Work>
+ <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
+ <dc:format>image/svg+xml</dc:format>
+ <dc:creator>
+ <cc:Agent>
+ <dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>
+ </cc:Agent>
+ </dc:creator>
+ </cc:Work>
+ </rdf:RDF>
+ </metadata>
+ <defs>
+ <style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>
+ </defs>
+ <g id="figure_1">
+ <g id="patch_1">
+ <path d="M 0 345.6
+L 633.6 345.6
+L 633.6 0
+L 0 0
+z
+" style="fill: #ffffff"/>
+ </g>
+ <g id="axes_1">
+ <g id="patch_2">
+ <path d="M 38.998599 309.598041
+L 630.59976 309.598041
+L 630.59976 30.118365
+L 38.998599 30.118365
+z
+" style="fill: #ffffff"/>
+ </g>
+ <g id="PathCollection_1">
+ <defs>
+ <path id="m2d3404be39" d="M -4.609772 4.609772
+L 4.609772 -4.609772
+M -4.609772 -4.609772
+L 4.609772 4.609772
+" style="stroke: #5a6873; stroke-width: 2"/>
+ </defs>
+ <g clip-path="url(#p79675e61a4)">
+ <use xlink:href="#m2d3404be39" x="393.959296" y="123.278257" style="fill: #5a6873; stroke: #5a6873; stroke-width: 2"/>
+ </g>
+ </g>
+ <g id="PathCollection_2">
+ <g clip-path="url(#p79675e61a4)">
+ <use xlink:href="#m2d3404be39" x="224.930393" y="207.969068" style="fill: #5a6873; stroke: #5a6873; stroke-width: 2"/>
+ </g>
+ </g>
+ <g id="PathCollection_3"/>
+ <g id="PathCollection_4">
+ <path d="M 292.541954 129.797459
+C 294.270867 129.797459 295.9292 129.110555 297.151726 127.888029
+C 298.374252 126.665503 299.061156 125.00717 299.061156 123.278257
+C 299.061156 121.549344 298.374252 119.891011 297.151726 118.668485
+C 295.9292 117.445959 294.270867 116.759055 292.541954 116.759055
+C 290.813041 116.759055 289.154708 117.445959 287.932182 118.668485
+C 286.709656 119.891011 286.022752 121.549344 286.022752 123.278257
+C 286.022752 125.00717 286.709656 126.665503 287.932182 127.888029
+C 289.154708 129.110555 290.813041 129.797459 292.541954 129.797459
+L 292.541954 129.797459
+z
+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #087f8c; stroke-width: 1.5"/>
+ </g>
+ <g id="matplotlib.axis_1">
+ <g id="xtick_1">
+ <g id="line2d_1">
+ <defs>
+ <path id="m6c093b1dcb" d="M 0 0
+L 0 3.5
+" style="stroke: #000000; stroke-width: 0.8"/>
+ </defs>
+ <g>
+ <use xlink:href="#m6c093b1dcb" x="55.90149" y="309.598041" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_1">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="55.90149" y="324.956322" transform="rotate(-0 55.90149 324.956322)">$0.010</text>
+ </g>
+ </g>
+ <g id="xtick_2">
+ <g id="line2d_2">
+ <g>
+ <use xlink:href="#m6c093b1dcb" x="224.930393" y="309.598041" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_2">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="224.930393" y="324.956322" transform="rotate(-0 224.930393 324.956322)">$0.020</text>
+ </g>
+ </g>
+ <g id="xtick_3">
+ <g id="line2d_3">
+ <g>
+ <use xlink:href="#m6c093b1dcb" x="393.959296" y="309.598041" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_3">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="393.959296" y="324.956322" transform="rotate(-0 393.959296 324.956322)">$0.030</text>
+ </g>
+ </g>
+ <g id="xtick_4">
+ <g id="line2d_4">
+ <g>
+ <use xlink:href="#m6c093b1dcb" x="562.988199" y="309.598041" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_4">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="562.988199" y="324.956322" transform="rotate(-0 562.988199 324.956322)">$0.040</text>
+ </g>
+ </g>
+ <g id="text_5">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="334.79918" y="339.957182" transform="rotate(-0 334.79918 339.957182)">Mean cost per task (lower is better)</text>
+ </g>
+ </g>
+ <g id="matplotlib.axis_2">
+ <g id="ytick_1">
+ <g id="line2d_5">
+ <path d="M 38.998599 275.721717
+L 630.59976 275.721717
+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>
+ </g>
+ <g id="line2d_6">
+ <defs>
+ <path id="m5fdf44b481" d="M 0 0
+L -3.5 0
+" style="stroke: #000000; stroke-width: 0.8"/>
+ </defs>
+ <g>
+ <use xlink:href="#m5fdf44b481" x="38.998599" y="275.721717" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_6">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="279.900428" transform="rotate(-0 31.998599 279.900428)">70</text>
+ </g>
+ </g>
+ <g id="ytick_2">
+ <g id="line2d_7">
+ <path d="M 38.998599 233.376311
+L 630.59976 233.376311
+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>
+ </g>
+ <g id="line2d_8">
+ <g>
+ <use xlink:href="#m5fdf44b481" x="38.998599" y="233.376311" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_7">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="237.555022" transform="rotate(-0 31.998599 237.555022)">75</text>
+ </g>
+ </g>
+ <g id="ytick_3">
+ <g id="line2d_9">
+ <path d="M 38.998599 191.030906
+L 630.59976 191.030906
+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>
+ </g>
+ <g id="line2d_10">
+ <g>
+ <use xlink:href="#m5fdf44b481" x="38.998599" y="191.030906" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_8">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="195.209617" transform="rotate(-0 31.998599 195.209617)">80</text>
+ </g>
+ </g>
+ <g id="ytick_4">
+ <g id="line2d_11">
+ <path d="M 38.998599 148.6855
+L 630.59976 148.6855
+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>
+ </g>
+ <g id="line2d_12">
+ <g>
+ <use xlink:href="#m5fdf44b481" x="38.998599" y="148.6855" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_9">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="152.864211" transform="rotate(-0 31.998599 152.864211)">85</text>
+ </g>
+ </g>
+ <g id="ytick_5">
+ <g id="line2d_13">
+ <path d="M 38.998599 106.340095
+L 630.59976 106.340095
+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>
+ </g>
+ <g id="line2d_14">
+ <g>
+ <use xlink:href="#m5fdf44b481" x="38.998599" y="106.340095" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_10">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="110.518806" transform="rotate(-0 31.998599 110.518806)">90</text>
+ </g>
+ </g>
+ <g id="ytick_6">
+ <g id="line2d_15">
+ <path d="M 38.998599 63.994689
+L 630.59976 63.994689
+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>
+ </g>
+ <g id="line2d_16">
+ <g>
+ <use xlink:href="#m5fdf44b481" x="38.998599" y="63.994689" style="stroke: #000000; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_11">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="68.1734" transform="rotate(-0 31.998599 68.1734)">95</text>
+ </g>
+ </g>
+ <g id="text_12">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="11.358521" y="169.858203" transform="rotate(-90 11.358521 169.858203)">Tasks passed out of 100 (higher is better)</text>
+ </g>
+ </g>
+ <g id="line2d_17">
+ <path d="M 38.998599 148.6855
+L 630.59976 148.6855
+" clip-path="url(#p79675e61a4)" style="fill: none; stroke-dasharray: 5.18,2.24; stroke-dashoffset: 0; stroke: #7961a8; stroke-width: 1.4"/>
+ </g>
+ <g id="line2d_18">
+ <path d="M 89.70727 275.721717
+L 191.124612 191.030906
+L 292.541954 123.278257
+L 529.182418 72.46377
+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #087f8c; stroke-width: 1.4; stroke-linecap: square"/>
+ <defs>
+ <path id="m5c9d91a73f" d="M 0 3.5
+C 0.928211 3.5 1.81853 3.131218 2.474874 2.474874
+C 3.131218 1.81853 3.5 0.928211 3.5 0
+C 3.5 -0.928211 3.131218 -1.81853 2.474874 -2.474874
+C 1.81853 -3.131218 0.928211 -3.5 0 -3.5
+C -0.928211 -3.5 -1.81853 -3.131218 -2.474874 -2.474874
+C -3.131218 -1.81853 -3.5 -0.928211 -3.5 0
+C -3.5 0.928211 -3.131218 1.81853 -2.474874 2.474874
+C -1.81853 3.131218 -0.928211 3.5 0 3.5
+z
+" style="stroke: #087f8c"/>
+ </defs>
+ <g clip-path="url(#p79675e61a4)">
+ <use xlink:href="#m5c9d91a73f" x="89.70727" y="275.721717" style="fill: #087f8c; stroke: #087f8c"/>
+ <use xlink:href="#m5c9d91a73f" x="191.124612" y="191.030906" style="fill: #087f8c; stroke: #087f8c"/>
+ <use xlink:href="#m5c9d91a73f" x="292.541954" y="123.278257" style="fill: #087f8c; stroke: #087f8c"/>
+ <use xlink:href="#m5c9d91a73f" x="529.182418" y="72.46377" style="fill: #087f8c; stroke: #087f8c"/>
+ </g>
+ </g>
+ <g id="patch_3">
+ <path d="M 38.998599 309.598041
+L 38.998599 30.118365
+" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
+ </g>
+ <g id="patch_4">
+ <path d="M 38.998599 309.598041
+L 630.59976 309.598041
+" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
+ </g>
+ <g id="text_13">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="96.70727" y="266.721717" transform="rotate(-0 96.70727 266.721717)">A</text>
+ </g>
+ <g id="text_14">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="198.124612" y="182.030906" transform="rotate(-0 198.124612 182.030906)">B</text>
+ </g>
+ <g id="text_15">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="400.959296" y="114.278257" transform="rotate(-0 400.959296 114.278257)">C · baseline</text>
+ </g>
+ <g id="text_16">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="231.930393" y="225.969068" transform="rotate(-0 231.930393 225.969068)">D</text>
+ </g>
+ <g id="text_17">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="299.541954" y="103.278257" transform="rotate(-0 299.541954 103.278257)">E · selected</text>
+ </g>
+ <g id="text_18">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="536.182418" y="63.46377" transform="rotate(-0 536.182418 63.46377)">F</text>
+ </g>
+ <g id="text_19">
+ <text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start" x="38.998599" y="12.118365" transform="rotate(-0 38.998599 12.118365)">Fixed algorithm and model pair; six routing configurations</text>
+ </g>
+ <g id="legend_1">
+ <g id="line2d_19">
+ <path d="M 509.005541 269.184526
+L 518.005541 269.184526
+L 527.005541 269.184526
+" style="fill: none; stroke-dasharray: 5.18,2.24; stroke-dashoffset: 0; stroke: #7961a8; stroke-width: 1.4"/>
+ </g>
+ <g id="text_20">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start" x="534.205541" y="272.334526" transform="rotate(-0 534.205541 272.334526)">85% success target</text>
+ </g>
+ <g id="line2d_20">
+ <path d="M 509.005541 282.685229
+L 518.005541 282.685229
+L 527.005541 282.685229
+" style="fill: none; stroke: #087f8c; stroke-width: 1.4; stroke-linecap: square"/>
+ <g>
+ <use xlink:href="#m5c9d91a73f" x="518.005541" y="282.685229" style="fill: #087f8c; stroke: #087f8c"/>
+ </g>
+ </g>
+ <g id="text_21">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start" x="534.205541" y="285.835229" transform="rotate(-0 534.205541 285.835229)">Pareto frontier</text>
+ </g>
+ <g id="PathCollection_5">
+ <defs>
+ <path id="m07558401bf" d="M -4.1833 4.1833
+L 4.1833 -4.1833
+M -4.1833 -4.1833
+L 4.1833 4.1833
+" style="stroke: #5a6873; stroke-width: 1.5"/>
+ </defs>
+ <g>
+ <use xlink:href="#m07558401bf" x="518.005541" y="296.973432" style="fill: #5a6873; stroke: #5a6873; stroke-width: 1.5"/>
+ </g>
+ </g>
+ <g id="text_22">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start" x="534.205541" y="299.335932" transform="rotate(-0 534.205541 299.335932)">Dominated option</text>
+ </g>
+ </g>
+ </g>
+ </g>
+ <defs>
+ <clipPath id="p79675e61a4">
+ <rect x="38.998599" y="30.118365" width="591.601161" height="279.479676"/>
+ </clipPath>
+ </defs>
+</svg>
diff --git a/docs/assets/tuning/model-pair-loss.svg b/docs/assets/tuning/model-pair-loss.svg
new file mode 100644
index 000000000..3a4bc1651
--- /dev/null
+++ b/docs/assets/tuning/model-pair-loss.svg
@@ -0,0 +1,352 @@
+<?xml version="1.0" encoding="utf-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="590.4pt" height="194.4pt" viewBox="0 0 590.4 194.4" xmlns="http://www.w3.org/2000/svg" version="1.1">
+ <metadata>
+ <rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
+ <cc:Work>
+ <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
+ <dc:format>image/svg+xml</dc:format>
+ <dc:creator>
+ <cc:Agent>
+ <dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>
+ </cc:Agent>
+ </dc:creator>
+ </cc:Work>
+ </rdf:RDF>
+ </metadata>
+ <defs>
+ <style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>
+ </defs>
+ <g id="figure_1">
+ <g id="patch_1">
+ <path d="M 0 194.4
+L 590.4 194.4
+L 590.4 0
+L 0 0
+z
+" style="fill: #ffffff"/>
+ </g>
+ <g id="axes_1">
+ <g id="patch_2">
+ <path d="M 118.08 151.632
+L 316.993964 151.632
+L 316.993964 52.488
+L 118.08 52.488
+z
+" style="fill: #ffffff"/>
+ </g>
+ <g id="patch_3">
+ <path d="M 118.08 77.43391
+L 171.840531 77.43391
+L 171.840531 62.722219
+L 118.08 62.722219
+z
+" clip-path="url(#p9188dad553)" style="fill: #087f8c"/>
+ </g>
+ <g id="patch_4">
+ <path d="M 118.08 109.415845
+L 193.344743 109.415845
+L 193.344743 94.704155
+L 118.08 94.704155
+z
+" clip-path="url(#p9188dad553)" style="fill: #087f8c"/>
+ </g>
+ <g id="patch_5">
+ <path d="M 118.08 141.397781
+L 221.300219 141.397781
+L 221.300219 126.68609
+L 118.08 126.68609
+z
+" clip-path="url(#p9188dad553)" style="fill: #087f8c"/>
+ </g>
+ <g id="patch_6">
+ <path d="M 171.840531 77.43391
+L 279.361593 77.43391
+L 279.361593 62.722219
+L 171.840531 62.722219
+z
+" clip-path="url(#p9188dad553)" style="fill: url(#h107803ec0a); stroke: #7961a8; stroke-linejoin: miter"/>
+ </g>
+ <g id="patch_7">
+ <path d="M 193.344743 109.415845
+L 193.344743 109.415845
+L 193.344743 94.704155
+L 193.344743 94.704155
+z
+" clip-path="url(#p9188dad553)" style="fill: url(#h107803ec0a); stroke: #7961a8; stroke-linejoin: miter"/>
+ </g>
+ <g id="patch_8">
+ <path d="M 221.300219 141.397781
+L 221.300219 141.397781
+L 221.300219 126.68609
+L 221.300219 126.68609
+z
+" clip-path="url(#p9188dad553)" style="fill: url(#h107803ec0a); stroke: #7961a8; stroke-linejoin: miter"/>
+ </g>
+ <g id="matplotlib.axis_1">
+ <g id="xtick_1">
+ <g id="line2d_1">
+ <defs>
+ <path id="m59e2246da5" d="M 0 0
+L 0 3.5
+" style="stroke: #5a6873; stroke-width: 0.8"/>
+ </defs>
+ <g>
+ <use xlink:href="#m59e2246da5" x="118.08" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_1">
+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="118.08" y="164.71075" transform="rotate(-0 118.08 164.71075)">$0.00</text>
+ </g>
+ </g>
+ <g id="xtick_2">
+ <g id="line2d_2">
+ <g>
+ <use xlink:href="#m59e2246da5" x="171.840531" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_2">
+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="171.840531" y="164.71075" transform="rotate(-0 171.840531 164.71075)">$0.10</text>
+ </g>
+ </g>
+ <g id="xtick_3">
+ <g id="line2d_3">
+ <g>
+ <use xlink:href="#m59e2246da5" x="225.601062" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_3">
+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="225.601062" y="164.71075" transform="rotate(-0 225.601062 164.71075)">$0.20</text>
+ </g>
+ </g>
+ <g id="xtick_4">
+ <g id="line2d_4">
+ <g>
+ <use xlink:href="#m59e2246da5" x="279.361593" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_4">
+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="279.361593" y="164.71075" transform="rotate(-0 279.361593 164.71075)">$0.30</text>
+ </g>
+ </g>
+ </g>
+ <g id="matplotlib.axis_2">
+ <g id="ytick_1">
+ <g id="line2d_5"/>
+ <g id="text_5">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="114.58" y="73.497361" transform="rotate(-0 114.58 73.497361)">Pair 1 · E1/C1</text>
+ </g>
+ </g>
+ <g id="ytick_2">
+ <g id="line2d_6"/>
+ <g id="text_6">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="114.58" y="105.479297" transform="rotate(-0 114.58 105.479297)">Pair 2 · E2/C1</text>
+ </g>
+ </g>
+ <g id="ytick_3">
+ <g id="line2d_7"/>
+ <g id="text_7">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="114.58" y="137.461232" transform="rotate(-0 114.58 137.461232)">Pair 3 · E2/C2</text>
+ </g>
+ </g>
+ </g>
+ <g id="patch_9">
+ <path d="M 118.08 151.632
+L 316.993964 151.632
+" style="fill: none; stroke: #d8dde1; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
+ </g>
+ <g id="text_8">
+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="283.12483" y="72.546209" transform="rotate(-0 283.12483 72.546209)">$0.300</text>
+ </g>
+ <g id="text_9">
+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="197.10798" y="104.528145" transform="rotate(-0 197.10798 104.528145)">$0.140</text>
+ </g>
+ <g id="text_10">
+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="225.063456" y="136.51008" transform="rotate(-0 225.063456 136.51008)">$0.192</text>
+ </g>
+ <g id="text_11">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="118.08" y="37.488" transform="rotate(-0 118.08 37.488)">Total operating loss</text>
+ </g>
+ <g id="legend_1">
+ <g id="patch_10">
+ <path d="M 114.245162 10.499811
+L 132.245162 10.499811
+L 132.245162 4.199811
+L 114.245162 4.199811
+z
+" style="fill: #087f8c"/>
+ </g>
+ <g id="text_12">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="139.445162" y="10.499811" transform="rotate(-0 139.445162 10.499811)">Execution cost</text>
+ </g>
+ <g id="patch_11">
+ <path d="M 223.143756 10.499811
+L 241.143756 10.499811
+L 241.143756 4.199811
+L 223.143756 4.199811
+z
+" style="fill: url(#h107803ec0a); stroke: #7961a8; stroke-linejoin: miter"/>
+ </g>
+ <g id="text_13">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="248.343756" y="10.499811" transform="rotate(-0 248.343756 10.499811)">Failure penalty</text>
+ </g>
+ </g>
+ </g>
+ <g id="axes_2">
+ <g id="patch_12">
+ <path d="M 413.773258 151.632
+L 566.784 151.632
+L 566.784 52.488
+L 413.773258 52.488
+z
+" style="fill: #ffffff"/>
+ </g>
+ <g id="patch_13">
+ <path d="M 413.773258 77.43391
+L 523.066645 77.43391
+L 523.066645 62.722219
+L 413.773258 62.722219
+z
+" clip-path="url(#p9b4b3d3bd8)" style="fill: #5a6873"/>
+ </g>
+ <g id="patch_14">
+ <path d="M 413.773258 109.415845
+L 464.776839 109.415845
+L 464.776839 94.704155
+L 413.773258 94.704155
+z
+" clip-path="url(#p9b4b3d3bd8)" style="fill: #087f8c"/>
+ </g>
+ <g id="patch_15">
+ <path d="M 413.773258 141.397781
+L 483.721026 141.397781
+L 483.721026 126.68609
+L 413.773258 126.68609
+z
+" clip-path="url(#p9b4b3d3bd8)" style="fill: #7961a8"/>
+ </g>
+ <g id="matplotlib.axis_3">
+ <g id="xtick_5">
+ <g id="line2d_8">
+ <g>
+ <use xlink:href="#m59e2246da5" x="413.773258" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_14">
+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="413.773258" y="164.71075" transform="rotate(-0 413.773258 164.71075)">$0.000</text>
+ </g>
+ </g>
+ <g id="xtick_6">
+ <g id="line2d_9">
+ <g>
+ <use xlink:href="#m59e2246da5" x="468.419952" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_15">
+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="468.419952" y="164.71075" transform="rotate(-0 468.419952 164.71075)">$0.025</text>
+ </g>
+ </g>
+ <g id="xtick_7">
+ <g id="line2d_10">
+ <g>
+ <use xlink:href="#m59e2246da5" x="523.066645" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>
+ </g>
+ </g>
+ <g id="text_16">
+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="523.066645" y="164.71075" transform="rotate(-0 523.066645 164.71075)">$0.050</text>
+ </g>
+ </g>
+ </g>
+ <g id="matplotlib.axis_4">
+ <g id="ytick_4">
+ <g id="line2d_11"/>
+ <g id="text_17">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="410.273258" y="73.49701" transform="rotate(-0 410.273258 73.49701)">4/6 pass</text>
+ </g>
+ </g>
+ <g id="ytick_5">
+ <g id="line2d_12"/>
+ <g id="text_18">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="410.273258" y="105.478945" transform="rotate(-0 410.273258 105.478945)">6/6 pass</text>
+ </g>
+ </g>
+ <g id="ytick_6">
+ <g id="line2d_13"/>
+ <g id="text_19">
+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="410.273258" y="137.460881" transform="rotate(-0 410.273258 137.460881)">6/6 pass</text>
+ </g>
+ </g>
+ </g>
+ <g id="patch_16">
+ <path d="M 413.773258 151.632
+L 566.784 151.632
+" style="fill: none; stroke: #d8dde1; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
+ </g>
+ <g id="text_20">
+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="527.438381" y="72.546209" transform="rotate(-0 527.438381 72.546209)">$0.0500</text>
+ </g>
+ <g id="text_21">
+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="469.148574" y="104.528145" transform="rotate(-0 469.148574 104.528145)">$0.0233</text>
+ </g>
+ <g id="text_22">
+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="488.092761" y="136.51008" transform="rotate(-0 488.092761 136.51008)">$0.0320</text>
+ </g>
+ <g id="text_23">
+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="413.773258" y="37.488" transform="rotate(-0 413.773258 37.488)">Mean loss per task</text>
+ </g>
+ </g>
+ </g>
+ <defs>
+ <clipPath id="p9188dad553">
+ <rect x="118.08" y="52.488" width="198.913964" height="99.144"/>
+ </clipPath>
+ <clipPath id="p9b4b3d3bd8">
+ <rect x="413.773258" y="52.488" width="153.010742" height="99.144"/>
+ </clipPath>
+ </defs>
+ <defs>
+ <pattern id="h107803ec0a" patternUnits="userSpaceOnUse" x="0" y="0" width="72" height="72">
+ <rect x="0" y="0" width="73" height="73" fill="#e4ddea"/>
+ <path d="M -36 36
+L 36 -36
+M -32 40
+L 40 -32
+M -28 44
+L 44 -28
+M -24 48
+L 48 -24
+M -20 52
+L 52 -20
+M -16 56
+L 56 -16
+M -12 60
+L 60 -12
+M -8 64
+L 64 -8
+M -4 68
+L 68 -4
+M 0 72
+L 72 0
+M 4 76
+L 76 4
+M 8 80
+L 80 8
+M 12 84
+L 84 12
+M 16 88
+L 88 16
+M 20 92
+L 92 20
+M 24 96
+L 96 24
+M 28 100
+L 100 28
+M 32 104
+L 104 32
+M 36 108
+L 108 36
+" style="fill: #7961a8; stroke: #7961a8; stroke-width: 1.0; stroke-linecap: butt; stroke-linejoin: miter"/>
+ </pattern>
+ </defs>
+</svg>
diff --git a/docs/index.md b/docs/index.md
index a385f379d..c714b2b01 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -32,7 +32,7 @@ Use it with [Claude Code or Codex](recipes/single_provider_coding_agents.md),
- [Host Integration](reference/host_integration.md): connect your clients and application lifecycle
- [Rust API](reference/rust_api.md): browse the core library, protocols, translation, clients, and runner
- [Context-Window Handling](operations/context_window.md): configure eviction and retry behavior
-- [Tuning](tuning/index.md): the loop from verifying routing to recording a result
+- [Tuning](tuning/index.md): foundations and worked configuration comparisons
- [Troubleshooting](operations/troubleshooting.md): symptoms, silent misconfigurations, and fixes
## Reference
diff --git a/docs/routing_algorithms/model_selection.md b/docs/routing_algorithms/model_selection.md
index 057a8a374..9a08752d3 100644
--- a/docs/routing_algorithms/model_selection.md
+++ b/docs/routing_algorithms/model_selection.md
@@ -5,6 +5,9 @@ you put in each role change results more than any threshold does. This page says
what each role needs, how to check a model before you route to it, which pairs
have measured results, and the mistakes we have seen in real runs.
+For a worked comparison of cost and task success, see the
+[model-pair selection example](../tuning/examples.md#model-pair-selection-with-fixed-algorithm-settings).
+
## Roles
Each strategy names its models by role. The TOML keys differ by route type:
diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.md
index f2e8a0a6a..bbda63f3e 100644
--- a/docs/routing_algorithms/overview.md
+++ b/docs/routing_algorithms/overview.md
@@ -27,7 +27,7 @@ can optionally call an LLM judge. The TOML configuration keys are unchanged.
Whichever strategy you choose, the two models you put behind it matter more than
any threshold. See [Model Selection](model_selection.md). To tune cost against
-quality, follow the [tuning loop](../tuning/index.md).
+quality, see [Examples](../tuning/examples.md).
### What the names mean
diff --git a/docs/tuning/examples.md b/docs/tuning/examples.md
new file mode 100644
index 000000000..4fc030a2b
--- /dev/null
+++ b/docs/tuning/examples.md
@@ -0,0 +1,199 @@
+---
+title: Tuning Examples
+---
+
+# Tuning Examples
+
+These two comparisons show how to choose routing settings or a model pair for
+a fixed objective. Start with [Tuning Foundations](foundations.md) to define
+that objective and design candidates. All numbers are illustrative, not measured
+benchmark results. The final section outlines planned GEPA support.
+
+## Tune configuration options for cost
+
+Suppose the goal is to **minimize mean cost per task while passing at least 85%
+of tasks**. Keep the routing algorithm and model pair fixed, then compare its
+configuration options.
+
+### Choose the stage-router controls to tune
+
+This example focuses on the stage router. The same process applies to other
+routing algorithms: choose controls that affect your objective, compare
+candidates, and validate the selected configuration.
+
+Stage router has many fields. With the model pair fixed, start with the controls
+that most directly change the routing split and cost: **profile, threshold, and
+hold turns**. The profile is selected by the `picker` field:
+
+| Profile (`picker`) | Default behavior | When it may fit better |
+|---|---|---|
+| `efficient_first` | Use efficient on ambiguous turns; escalate when signals call for capable. | The efficient model handles most work, and occasional escalation can recover difficult tasks. A useful starting point for reducing cost. |
+| `capable_first` | Use capable on ambiguous turns; offload when signals support efficient. | Ambiguous or difficult work needs the capable model, with clear routine stretches to offload. Experimental: calibrate on your workload before relying on it. |
+
+The threshold changes which signals trigger a switch, and therefore the share
+of requests each tier serves. Its direction depends on the profile:
+
+| Control | Profile | Lower value | Higher value |
+|---|---|---|---|
+| `confidence_threshold` | `efficient_first` | Easier escalation; more capable traffic, tending toward higher model cost. | Harder escalation; more efficient traffic, tending toward lower model cost. |
+| `confidence_threshold` | `capable_first` | Easier offloading; more efficient traffic, tending toward lower model cost. | Harder offloading; more capable traffic, tending toward higher model cost. |
+| `capable_hold_turns` | Both | Fewer forced capable requests; routing can return to efficient sooner. | More requests stay capable after switching, allowing longer recovery at greater capable usage. |
+| `efficient_hold_turns` | Both | Fewer forced efficient requests; routing can escalate sooner. | More requests stay efficient after switching, which can save cost but delay escalation even when errors appear. |
+
+These threshold effects describe signal-based routing without a classifier;
+active holds and hard overrides take precedence. For holds, `0` disables the
+hold; positive values count requests, while `-1` pins the tier after a switch.
+Longer holds can reduce switching and preserve cached context, so measure total
+cost, retries, and task success together. See the
+[stage-router guide](../routing_algorithms/stage_router_routing.md#pickers) for
+profile and threshold details.
+
+**Handoff notes can break cache reuse across turns.** They are added only to the forwarded
+request, so later requests can omit the note and break reuse of the cached
+continuation. A fix is planned for a future release. Leave `handoff_notes`
+unconfigured for this cost comparison; see
+[handoff notes](../routing_algorithms/stage_router_routing.md#optional-handoff-notes).
+
+### Split the benchmark and collect baselines
+
+Use **20% for calibration and 80% for held-out validation**, chosen before tuning.
+For this example's 500-task benchmark, tune on 100 tasks and reserve 400 for the
+frozen configuration. Keep related tasks together and both splits representative;
+see [split guidance](foundations.md#4-check-the-final-choice).
+
+First run efficient-only, capable-only, and the current routed configuration on
+the same 100 calibration tasks. These illustrative baselines show what routing
+must improve on:
+
+| Baseline | Tasks passed | Mean cost per task |
+|---|---|---|
+| Efficient-only | 64/100 | $0.010 |
+| Capable-only | 94/100 | $0.040 |
+| Current routing (C below) | 88/100 | $0.030 |
+
+Use the same agent, task limits, serving settings, and initial cache setup for
+every run. Include all billed calls, retries, and cache charges.
+
+### Compare six configurations
+
+For this experiment, choose the efficient-first profile and vary only the
+threshold and capable hold. This isolates their effect before trying another
+profile.
+
+Use `stage_router` with `picker = "efficient_first"`, no classifier,
+`recent_turn_window = 5`, and `efficient_hold_turns = 0`. Keep the model pair,
+serving settings, agent, task limits, initial cache setup, and all other options
+fixed. Compare all six candidates on the same 100 calibration tasks.
+
+Run each candidate end to end: changed routing can alter later
+prompts, cache state, and costs, so reusing one run's outcomes can misrank them.
+
+Here are results from one run per calibration task, with C as the baseline.
+Compare mean cost among candidates meeting the 85% target. Include failed tasks,
+retries, and cache charges in that cost.
+
+| Candidate | Threshold | Hold (requests) | Tasks passed | Mean cost per task |
+|---|---|---|---|---|
+| A | 0.7 | 0 | 70/100 | $0.012 |
+| B | 0.5 | 0 | 80/100 | $0.018 |
+| C (baseline) | 0.3 | 0 | 88/100 | $0.030 |
+| D | 0.7 | 2 | 78/100 | $0.020 |
+| E | 0.5 | 2 | 88/100 | $0.024 |
+| F | 0.3 | 2 | 94/100 | $0.038 |
+
+E improves on C: it passes the same number of tasks and costs less. B improves
+on D in both cost and task success. Remove C and D from consideration.
+
+**A, B, E, and F form the Pareto frontier among these six routed candidates.**
+No other candidate in this sweep matches or improves both measures while
+improving at least one. These options offer the remaining cost/success tradeoffs:
+
+![A, B, E, and F form the cost/success Pareto frontier. C is more expensive than E at the same success rate; D costs more and passes fewer tasks than B. E is the cheapest frontier option above the 85% success target.](../assets/tuning/configuration-pareto.svg)
+
+*Figure 1. The circles form the observed Pareto frontier. Crosses mark candidates
+that another option improves on. The dashed line is the 85% task-success target.*
+
+**Select E under this objective.** A and B miss the success target. E and F both
+meet it, but E costs less: $0.024 per task, or $2.40 across 100 tasks. E reduces
+cost by 20% from baseline C while preserving its observed 88% success rate.
+It also costs 40% less than capable-only, with six fewer tasks passed, while
+still meeting the 85% target. F buys a higher success rate at a higher cost.
+
+The frontier describes the calibration sample. Repeat promising runs, then
+freeze the selection. Run E and all three baselines on the untouched 400-task
+validation split to check whether the cost advantage and success target hold.
+
+## Model-pair selection with fixed algorithm settings
+
+This independent example fixes the algorithm settings and changes the
+efficient/capable model pair. Score each task as **execution cost plus $0.10 if
+it fails**, then minimize the mean score. Choose that penalty before comparing
+results; it expresses how much failure matters for this example.
+
+Use `stage_router` with `picker = "efficient_first"`, `confidence_threshold = 0.5`,
+`recent_turn_window = 5`, `capable_hold_turns = 2`, `efficient_hold_turns = 0`, and
+no classifier or handoff notes. Keep this profile, all other settings, and the
+evaluation conditions fixed while changing models. Check
+[model compatibility](../routing_algorithms/model_selection.md) before running.
+
+Use the same **20% calibration / 80% held-out validation** split. To keep the
+arithmetic small, this separate example uses 30 tasks: six for calibration and
+24 reserved for validation. Run each candidate end to end with the same agent,
+task limits, serving settings, and initial cache setup.
+
+First measure each model alone on the six calibration tasks. These fictional
+single-model baselines use the same $0.10 failure penalty:
+
+| Single-model baseline | Tasks passed | Total execution cost | Mean loss per task |
+|---|---|---|---|
+| E1 only | 3/6 | $0.048 | $0.0580 |
+| E2 only | 4/6 | $0.072 | $0.0453 |
+| C1 only | 6/6 | $0.240 | $0.0400 |
+| C2 only | 6/6 | $0.300 | $0.0500 |
+
+Now compare three routed pairs on those same tasks. Pair 1 is the current routed
+baseline. Pair 2 changes only the efficient model from Pair 1; Pair 3 changes
+only the capable model from Pair 2. Fixed routing settings can still produce
+different decisions because model outputs affect later routing signals.
+
+| Pair | Efficient / capable | Tasks passed | Total execution cost | Failure penalty | Mean loss per task |
+|---|---|---|---|---|---|
+| 1 | E1 / C1 | 4/6 | $0.100 | $0.200 | $0.0500 |
+| 2 | E2 / C1 | 6/6 | $0.140 | $0.000 | $0.0233 |
+| 3 | E2 / C2 | 6/6 | $0.192 | $0.000 | $0.0320 |
+
+Include every billed call, retry, and cache charge. For Pair 1, add $0.100 in
+execution costs and $0.200 for two failures, then divide by six tasks: $0.0500
+per task. Failure penalties are part of the score, not billed charges.
+
+![Pair 1 has $0.300 total operating loss including failure penalties. Pair 2 has $0.140 and Pair 3 has $0.192. Pair 2 has the lowest mean loss.](../assets/tuning/model-pair-loss.svg)
+
+*Figure 2. Model-pair comparison under fixed algorithm settings.*
+
+**Select Pair 2.** Pair 1 has the lowest billed cost but loses its advantage when
+failures count. Pair 3 costs more without improving success on this sample.
+Pair 2 also passes all six tasks at lower cost than C1 alone, the best
+single-model baseline under this scoring rule.
+
+Freeze Pair 2 and run it, the current Pair 1, and the single-model baselines on
+the untouched 24-task validation split. Compare mean loss and task success before
+making a deployment choice; follow the
+[final validation guidance](foundations.md#4-check-the-final-choice).
+
+## Upcoming: GEPA tuning for the unified router
+
+GEPA-based prompt tuning for the unified router is planned for a future release,
+with tooling and concrete APIs to run the workflow. GEPA proposes instruction
+changes from feedback, evaluates them, and repeats while model weights stay
+fixed. See the [GEPA paper](https://arxiv.org/abs/2507.19457) for the approach.
+
+The planned example will keep the model pool and objective fixed while changing
+the router's instructions:
+
+- Use calibration-task outcomes to show where routing choices helped or hurt the
+ objective. A reflector model uses that feedback to propose revised instructions.
+- Compare proposed prompts within the calibration split using the same cost
+ and success measures, including the cost of routing calls.
+- Freeze the chosen prompt and evaluate it on the held-out validation split.
+
+The API walkthrough and worked results will accompany the tooling release.
diff --git a/docs/tuning/foundations.md b/docs/tuning/foundations.md
new file mode 100644
index 000000000..9cdc938ce
--- /dev/null
+++ b/docs/tuning/foundations.md
@@ -0,0 +1,84 @@
+# Tuning Foundations
+
+This guide helps you find a **golden configuration for your objective and
+workload**: define the outcome you want, compare candidate settings on
+calibration tasks, and validate the selected configuration on held-out tasks
+before adopting it as your baseline.
+
+See [Examples](examples.md) for configuration and model-pair comparisons
+with concrete data.
+
+## 1. Define the objective
+
+Define what the user wants to improve, on which workload, and what makes a result
+acceptable. Keep this objective fixed throughout the comparison.
+
+- **Reduce cost while preserving accuracy.** Minimize total cost per task while
+ meeting a defined accuracy requirement. Include decision calls, retries, and
+ cache charges in the cost.
+- **Maximize task success.** Maximize the fraction of tasks completed correctly,
+ subject to any cost or latency limits.
+- **Reach a desired routing split.** For example, target 70% of calls on the
+ efficient tier and 30% on the capable tier. Specify the allowed deviation and
+ quality requirement.
+- **Reach a capable-tier cache-usage target.** For example, require at least 80%
+ of capable-tier input tokens to be served from cache. Measure cache-read input
+ tokens divided by all input tokens on that tier, with a defined quality floor.
+
+Once the objective is measurable, the next step is to choose configurations that
+might achieve it. This separates the outcome we want from the settings we will
+test.
+
+## 2. Design configuration candidates
+
+A candidate is a complete configuration to test under the objective. Start with
+a baseline and a small set of changes you expect to help:
+
+1. **Record baselines.** Measure efficient-only, capable-only, and the current
+ routed configuration on the same tasks. Save their settings and outcomes.
+2. **Choose what varies.** For example, change model targets, thresholds, or hold
+ durations. Keep other settings fixed and state why each change might help.
+3. **Test interactions.** A threshold and hold duration can affect the same
+ routing decisions, so include combinations when needed.
+4. **Keep runs comparable.** Use the same tasks, evaluation procedure, and initial
+ cache setup. Pin model and provider versions, and save each configuration.
+
+Run the same tasks for every candidate. The following examples show how to use
+the results to choose between them.
+
+## 3. Choose an example
+
+Start with the comparison that matches what you want to change:
+
+- **[Tune configuration options](examples.md#tune-configuration-options-for-cost):**
+ keep the algorithm and model pair fixed, vary routing settings, and find the
+ lowest-cost option that meets the task-success target. A Pareto chart shows
+ the cost and success tradeoffs.
+- **[Select a model pair](examples.md#model-pair-selection-with-fixed-algorithm-settings):**
+ keep algorithm settings fixed, swap the efficient/capable models, and compare
+ execution cost plus a penalty for failed tasks.
+- **[Tune the unified router with GEPA](examples.md#upcoming-gepa-tuning-for-the-unified-router):**
+ planned for a future release. Use benchmark feedback to revise routing
+ instructions and select a prompt against a fixed objective.
+
+## 4. Check the final choice
+
+Split **100% of the evaluation benchmark before tuning**. A practical starting
+allocation is:
+
+| Split | Share | Use |
+|---|---|---|
+| Calibration | 20% | Collect baselines, try settings, and select the candidate. |
+| Held-out validation | 80% | Evaluate the frozen candidate against the baselines. Do not use these results to tune it. |
+
+For 500 tasks, this means 100 calibration tasks and 400 validation tasks. Adjust
+the percentages before running if the calibration set is too small to cover the
+workload. Here, validation is the final held-out check.
+
+- Keep related examples in the same split. If predicting future data, respect
+ time when splitting. Use representative workloads and repeated trials when
+ execution is stochastic.
+- Record the objective, scoring rule, configurations, and evaluation conditions.
+ Changing the models or settings requires new measurements.
+- If validation results guide another change, those tasks have become
+ calibration data. Use fresh held-out tasks for the next independent check.
diff --git a/docs/tuning/index.md b/docs/tuning/index.md
index e3c0480a0..89b4b62a2 100644
--- a/docs/tuning/index.md
+++ b/docs/tuning/index.md
@@ -1,95 +1,11 @@
# Tuning
-Routing has two outcomes you can measure: how many tasks the agent completes,
-and what a run costs. Tuning is the loop below. Every step links the page that
-explains it, names what it produces, and says what it costs to run.
+Tuning helps you find a configuration that meets your goals on your workload.
+Start with Foundations to plan the comparison, then use Examples to
+see how to evaluate candidates and choose a configuration.
-Start where your goal is:
-
-| Your goal | Start at |
-|---|---|
-| Lower cost at the same quality | Step 1, then step 4 for the baseline numbers |
-| Add a cheaper model safely | Step 2 |
-| Routing is on, and results got worse | Step 5 |
-
-## The loop
-
-1. **Verify routing is active.** Follow
- [Verify routing is active](../operations/local_evaluation.md#verify-routing-is-active).
- Produces: the selected-model header from two requests and a `/v1/stats`
- capture. Costs: two requests, no agent run.
-
-2. **Choose and check the models.** Run the
- [five checks](../routing_algorithms/model_selection.md#check-the-efficient-model-before-you-route)
- on the efficient model, then the
- [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair).
- Produces: a pass or fail per check, and solved tasks plus cost for each
- fixed arm. Costs: single requests for most checks, one short agent run for
- the tool-calling check, and one agent run per arm on at least 20 tasks.
-
-3. **Start from a measured profile.** Pick the closest row in
- [Measured pairs](../routing_algorithms/model_selection.md#measured-pairs)
- and copy its profile from `benchmark/routing-profiles/`. The
- [strategy comparison](../routing_algorithms/overview.md#compare-the-strategies)
- says which route type fits your traffic. Produces: a route file that passes
- `switchyard-server --config routes.toml --dry-run`. Costs: none.
-
-4. **Run the routed arm against the fixed arms.** Use the same tasks, agent,
- and provider for all three arms. The [benchmark guide](../../benchmark/README.md)
- shows the harness the checked-in results used, and the
- [Benchmarks](../benchmarks.md) page lists the published results with their
- configs. Produces: solved tasks and
- cost per run for each arm. Costs: `k` runs per arm. The plan/execute and
- advisor profiles use three runs and report the mean with the sample standard
- deviation, as in the
- [plan/execute profile header](../../benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml).
- The stage profile records a single run. A single run cannot separate a small
- change from noise.
-
-5. **Read the decision mix.** For stage routes, read
- [decision sources](../routing_algorithms/stage_router_routing.md#decision-sources)
- from `/v1/stats`. For escalation routes, read the
- [observability](../routing_algorithms/escalation_router_routing.md#observability)
- section. For advisor routes, `/v1/stats` reports verdicts and consult
- failures under `advisor_gate`. Start the server with `--routing-log-file`
- to keep one record per model call: the served call, and each judge call
- under the judge's model ID. Filter by `model` before adding up usage.
- Produces: the `routing_decisions` capture and the routing log. Costs: none
- beyond the runs in step 4.
-
-6. **Move one dial.** Change one setting, then repeat step 4.
-
- | Strategy | First dial | Guidance |
- |---|---|---|
- | Execution, Auto | `confidence_threshold` | [Tuning `confidence_threshold`](../routing_algorithms/stage_router_routing.md#tuning-confidence_threshold) and [Calibrating the threshold from run data](../routing_algorithms/stage_router_routing.md#calibrating-the-threshold-from-run-data) |
- | Task | `base_threshold`, then `classify_trigger` | [Tuning options](../routing_algorithms/llm_classifier_routing.md#tuning-options) and [When the judge runs](../routing_algorithms/llm_classifier_routing.md#when-the-judge-runs) |
- | Escalation | `confirmations` | [Tuning options](../routing_algorithms/escalation_router_routing.md#tuning-options) |
- | Advisor Gate | `max_reviews` with `gate_min_tool_results` | [Tuning](../routing_algorithms/advisor_gate_routing.md#tuning) |
- | Composite | `efficient_first_threshold` and `capable_first_threshold` | [What exists today](../routing_algorithms/composite_routing.md#what-exists-today) |
-
- Produces: a before-and-after table with the arm results from step 4.
- Costs: `k` runs per setting you try.
-
-7. **Record the result.** Write down what you ran, in the form the profile
- headers use:
-
- - the Switchyard commit;
- - each model with its provider, wire format, and reasoning effort;
- - the agent and its version;
- - the benchmark, the task count, and `k`;
- - the result band next to the fixed arms;
- - the cost per run.
-
- The
- [plan/execute profile header](../../benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml)
- records all of these. Produces: the header. Costs: none.
-
-8. **Recalibrate on change.** A new model version, provider, wire format,
- reasoning effort, agent version, or workload moves the operating point, and
- the thresholds do not carry over. See
- [Pin the provider and endpoint](../routing_algorithms/model_selection.md#pin-the-provider-and-endpoint).
- Return to step 4 with the recorded header as your baseline. Produces: a new
- header next to the old one. Costs: the runs of step 4.
+- [Tuning Foundations](foundations.md): define your objective, candidates, and evaluation splits.
+- [Examples](examples.md): compare routing settings and model pairs with worked examples.
## Related
diff --git a/mkdocs.yml b/mkdocs.yml
index 33d7d3fac..d50e697c9 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -44,7 +44,9 @@ nav:
- Choose Models: routing_algorithms/model_selection.md
- Evaluate Routing:
- Benchmarks: benchmarks.md
- - Compare and Tune: tuning/index.md
+ - Tuning: tuning/index.md
+ - Tuning Foundations: tuning/foundations.md
+ - Examples: tuning/examples.md
- Local Demo:
- Run the Reference Server: operations/local_evaluation.md
- Local Agents: