Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(tuning): add tuning foundations guide (#1199)
sabhatinascf92c11354d7
diff --git a/docs/assets/tuning/configuration-pareto.svg b/docs/assets/tuning/configuration-pareto.svgnew file mode 100644index 000000000..bb5ed830e--- /dev/null+++ b/docs/assets/tuning/configuration-pareto.svg@@ -0,0 +1,328 @@+<?xml version="1.0" encoding="utf-8" standalone="no"?>+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">+<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="633.6pt" height="345.6pt" viewBox="0 0 633.6 345.6" xmlns="http://www.w3.org/2000/svg" version="1.1">+ <metadata>+ <rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">+ <cc:Work>+ <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>+ <dc:format>image/svg+xml</dc:format>+ <dc:creator>+ <cc:Agent>+ <dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>+ </cc:Agent>+ </dc:creator>+ </cc:Work>+ </rdf:RDF>+ </metadata>+ <defs>+ <style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>+ </defs>+ <g id="figure_1">+ <g id="patch_1">+ <path d="M 0 345.6+L 633.6 345.6+L 633.6 0+L 0 0+z+" style="fill: #ffffff"/>+ </g>+ <g id="axes_1">+ <g id="patch_2">+ <path d="M 38.998599 309.598041+L 630.59976 309.598041+L 630.59976 30.118365+L 38.998599 30.118365+z+" style="fill: #ffffff"/>+ </g>+ <g id="PathCollection_1">+ <defs>+ <path id="m2d3404be39" d="M -4.609772 4.609772+L 4.609772 -4.609772+M -4.609772 -4.609772+L 4.609772 4.609772+" style="stroke: #5a6873; stroke-width: 2"/>+ </defs>+ <g clip-path="url(#p79675e61a4)">+ <use xlink:href="#m2d3404be39" x="393.959296" y="123.278257" style="fill: #5a6873; stroke: #5a6873; stroke-width: 2"/>+ </g>+ </g>+ <g id="PathCollection_2">+ <g clip-path="url(#p79675e61a4)">+ <use xlink:href="#m2d3404be39" x="224.930393" y="207.969068" style="fill: #5a6873; stroke: #5a6873; stroke-width: 2"/>+ </g>+ </g>+ <g id="PathCollection_3"/>+ <g id="PathCollection_4">+ <path d="M 292.541954 129.797459+C 294.270867 129.797459 295.9292 129.110555 297.151726 127.888029+C 298.374252 126.665503 299.061156 125.00717 299.061156 123.278257+C 299.061156 121.549344 298.374252 119.891011 297.151726 118.668485+C 295.9292 117.445959 294.270867 116.759055 292.541954 116.759055+C 290.813041 116.759055 289.154708 117.445959 287.932182 118.668485+C 286.709656 119.891011 286.022752 121.549344 286.022752 123.278257+C 286.022752 125.00717 286.709656 126.665503 287.932182 127.888029+C 289.154708 129.110555 290.813041 129.797459 292.541954 129.797459+L 292.541954 129.797459+z+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #087f8c; stroke-width: 1.5"/>+ </g>+ <g id="matplotlib.axis_1">+ <g id="xtick_1">+ <g id="line2d_1">+ <defs>+ <path id="m6c093b1dcb" d="M 0 0+L 0 3.5+" style="stroke: #000000; stroke-width: 0.8"/>+ </defs>+ <g>+ <use xlink:href="#m6c093b1dcb" x="55.90149" y="309.598041" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_1">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="55.90149" y="324.956322" transform="rotate(-0 55.90149 324.956322)">$0.010</text>+ </g>+ </g>+ <g id="xtick_2">+ <g id="line2d_2">+ <g>+ <use xlink:href="#m6c093b1dcb" x="224.930393" y="309.598041" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_2">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="224.930393" y="324.956322" transform="rotate(-0 224.930393 324.956322)">$0.020</text>+ </g>+ </g>+ <g id="xtick_3">+ <g id="line2d_3">+ <g>+ <use xlink:href="#m6c093b1dcb" x="393.959296" y="309.598041" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_3">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="393.959296" y="324.956322" transform="rotate(-0 393.959296 324.956322)">$0.030</text>+ </g>+ </g>+ <g id="xtick_4">+ <g id="line2d_4">+ <g>+ <use xlink:href="#m6c093b1dcb" x="562.988199" y="309.598041" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_4">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="562.988199" y="324.956322" transform="rotate(-0 562.988199 324.956322)">$0.040</text>+ </g>+ </g>+ <g id="text_5">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="334.79918" y="339.957182" transform="rotate(-0 334.79918 339.957182)">Mean cost per task (lower is better)</text>+ </g>+ </g>+ <g id="matplotlib.axis_2">+ <g id="ytick_1">+ <g id="line2d_5">+ <path d="M 38.998599 275.721717+L 630.59976 275.721717+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>+ </g>+ <g id="line2d_6">+ <defs>+ <path id="m5fdf44b481" d="M 0 0+L -3.5 0+" style="stroke: #000000; stroke-width: 0.8"/>+ </defs>+ <g>+ <use xlink:href="#m5fdf44b481" x="38.998599" y="275.721717" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_6">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="279.900428" transform="rotate(-0 31.998599 279.900428)">70</text>+ </g>+ </g>+ <g id="ytick_2">+ <g id="line2d_7">+ <path d="M 38.998599 233.376311+L 630.59976 233.376311+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>+ </g>+ <g id="line2d_8">+ <g>+ <use xlink:href="#m5fdf44b481" x="38.998599" y="233.376311" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_7">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="237.555022" transform="rotate(-0 31.998599 237.555022)">75</text>+ </g>+ </g>+ <g id="ytick_3">+ <g id="line2d_9">+ <path d="M 38.998599 191.030906+L 630.59976 191.030906+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>+ </g>+ <g id="line2d_10">+ <g>+ <use xlink:href="#m5fdf44b481" x="38.998599" y="191.030906" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_8">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="195.209617" transform="rotate(-0 31.998599 195.209617)">80</text>+ </g>+ </g>+ <g id="ytick_4">+ <g id="line2d_11">+ <path d="M 38.998599 148.6855+L 630.59976 148.6855+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>+ </g>+ <g id="line2d_12">+ <g>+ <use xlink:href="#m5fdf44b481" x="38.998599" y="148.6855" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_9">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="152.864211" transform="rotate(-0 31.998599 152.864211)">85</text>+ </g>+ </g>+ <g id="ytick_5">+ <g id="line2d_13">+ <path d="M 38.998599 106.340095+L 630.59976 106.340095+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>+ </g>+ <g id="line2d_14">+ <g>+ <use xlink:href="#m5fdf44b481" x="38.998599" y="106.340095" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_10">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="110.518806" transform="rotate(-0 31.998599 110.518806)">90</text>+ </g>+ </g>+ <g id="ytick_6">+ <g id="line2d_15">+ <path d="M 38.998599 63.994689+L 630.59976 63.994689+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #b0b0b0; stroke-opacity: 0.15; stroke-width: 0.8; stroke-linecap: square"/>+ </g>+ <g id="line2d_16">+ <g>+ <use xlink:href="#m5fdf44b481" x="38.998599" y="63.994689" style="stroke: #000000; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_11">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end" x="31.998599" y="68.1734" transform="rotate(-0 31.998599 68.1734)">95</text>+ </g>+ </g>+ <g id="text_12">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle" x="11.358521" y="169.858203" transform="rotate(-90 11.358521 169.858203)">Tasks passed out of 100 (higher is better)</text>+ </g>+ </g>+ <g id="line2d_17">+ <path d="M 38.998599 148.6855+L 630.59976 148.6855+" clip-path="url(#p79675e61a4)" style="fill: none; stroke-dasharray: 5.18,2.24; stroke-dashoffset: 0; stroke: #7961a8; stroke-width: 1.4"/>+ </g>+ <g id="line2d_18">+ <path d="M 89.70727 275.721717+L 191.124612 191.030906+L 292.541954 123.278257+L 529.182418 72.46377+" clip-path="url(#p79675e61a4)" style="fill: none; stroke: #087f8c; stroke-width: 1.4; stroke-linecap: square"/>+ <defs>+ <path id="m5c9d91a73f" d="M 0 3.5+C 0.928211 3.5 1.81853 3.131218 2.474874 2.474874+C 3.131218 1.81853 3.5 0.928211 3.5 0+C 3.5 -0.928211 3.131218 -1.81853 2.474874 -2.474874+C 1.81853 -3.131218 0.928211 -3.5 0 -3.5+C -0.928211 -3.5 -1.81853 -3.131218 -2.474874 -2.474874+C -3.131218 -1.81853 -3.5 -0.928211 -3.5 0+C -3.5 0.928211 -3.131218 1.81853 -2.474874 2.474874+C -1.81853 3.131218 -0.928211 3.5 0 3.5+z+" style="stroke: #087f8c"/>+ </defs>+ <g clip-path="url(#p79675e61a4)">+ <use xlink:href="#m5c9d91a73f" x="89.70727" y="275.721717" style="fill: #087f8c; stroke: #087f8c"/>+ <use xlink:href="#m5c9d91a73f" x="191.124612" y="191.030906" style="fill: #087f8c; stroke: #087f8c"/>+ <use xlink:href="#m5c9d91a73f" x="292.541954" y="123.278257" style="fill: #087f8c; stroke: #087f8c"/>+ <use xlink:href="#m5c9d91a73f" x="529.182418" y="72.46377" style="fill: #087f8c; stroke: #087f8c"/>+ </g>+ </g>+ <g id="patch_3">+ <path d="M 38.998599 309.598041+L 38.998599 30.118365+" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>+ </g>+ <g id="patch_4">+ <path d="M 38.998599 309.598041+L 630.59976 309.598041+" style="fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>+ </g>+ <g id="text_13">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="96.70727" y="266.721717" transform="rotate(-0 96.70727 266.721717)">A</text>+ </g>+ <g id="text_14">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="198.124612" y="182.030906" transform="rotate(-0 198.124612 182.030906)">B</text>+ </g>+ <g id="text_15">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="400.959296" y="114.278257" transform="rotate(-0 400.959296 114.278257)">C · baseline</text>+ </g>+ <g id="text_16">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="231.930393" y="225.969068" transform="rotate(-0 231.930393 225.969068)">D</text>+ </g>+ <g id="text_17">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="299.541954" y="103.278257" transform="rotate(-0 299.541954 103.278257)">E · selected</text>+ </g>+ <g id="text_18">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="536.182418" y="63.46377" transform="rotate(-0 536.182418 63.46377)">F</text>+ </g>+ <g id="text_19">+ <text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start" x="38.998599" y="12.118365" transform="rotate(-0 38.998599 12.118365)">Fixed algorithm and model pair; six routing configurations</text>+ </g>+ <g id="legend_1">+ <g id="line2d_19">+ <path d="M 509.005541 269.184526+L 518.005541 269.184526+L 527.005541 269.184526+" style="fill: none; stroke-dasharray: 5.18,2.24; stroke-dashoffset: 0; stroke: #7961a8; stroke-width: 1.4"/>+ </g>+ <g id="text_20">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start" x="534.205541" y="272.334526" transform="rotate(-0 534.205541 272.334526)">85% success target</text>+ </g>+ <g id="line2d_20">+ <path d="M 509.005541 282.685229+L 518.005541 282.685229+L 527.005541 282.685229+" style="fill: none; stroke: #087f8c; stroke-width: 1.4; stroke-linecap: square"/>+ <g>+ <use xlink:href="#m5c9d91a73f" x="518.005541" y="282.685229" style="fill: #087f8c; stroke: #087f8c"/>+ </g>+ </g>+ <g id="text_21">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start" x="534.205541" y="285.835229" transform="rotate(-0 534.205541 285.835229)">Pareto frontier</text>+ </g>+ <g id="PathCollection_5">+ <defs>+ <path id="m07558401bf" d="M -4.1833 4.1833+L 4.1833 -4.1833+M -4.1833 -4.1833+L 4.1833 4.1833+" style="stroke: #5a6873; stroke-width: 1.5"/>+ </defs>+ <g>+ <use xlink:href="#m07558401bf" x="518.005541" y="296.973432" style="fill: #5a6873; stroke: #5a6873; stroke-width: 1.5"/>+ </g>+ </g>+ <g id="text_22">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start" x="534.205541" y="299.335932" transform="rotate(-0 534.205541 299.335932)">Dominated option</text>+ </g>+ </g>+ </g>+ </g>+ <defs>+ <clipPath id="p79675e61a4">+ <rect x="38.998599" y="30.118365" width="591.601161" height="279.479676"/>+ </clipPath>+ </defs>+</svg>diff --git a/docs/assets/tuning/model-pair-loss.svg b/docs/assets/tuning/model-pair-loss.svgnew file mode 100644index 000000000..3a4bc1651--- /dev/null+++ b/docs/assets/tuning/model-pair-loss.svg@@ -0,0 +1,352 @@+<?xml version="1.0" encoding="utf-8" standalone="no"?>+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">+<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="590.4pt" height="194.4pt" viewBox="0 0 590.4 194.4" xmlns="http://www.w3.org/2000/svg" version="1.1">+ <metadata>+ <rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">+ <cc:Work>+ <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>+ <dc:format>image/svg+xml</dc:format>+ <dc:creator>+ <cc:Agent>+ <dc:title>Matplotlib v3.11.2, https://matplotlib.org/</dc:title>+ </cc:Agent>+ </dc:creator>+ </cc:Work>+ </rdf:RDF>+ </metadata>+ <defs>+ <style type="text/css">*{stroke-linejoin: round; stroke-linecap: butt}</style>+ </defs>+ <g id="figure_1">+ <g id="patch_1">+ <path d="M 0 194.4+L 590.4 194.4+L 590.4 0+L 0 0+z+" style="fill: #ffffff"/>+ </g>+ <g id="axes_1">+ <g id="patch_2">+ <path d="M 118.08 151.632+L 316.993964 151.632+L 316.993964 52.488+L 118.08 52.488+z+" style="fill: #ffffff"/>+ </g>+ <g id="patch_3">+ <path d="M 118.08 77.43391+L 171.840531 77.43391+L 171.840531 62.722219+L 118.08 62.722219+z+" clip-path="url(#p9188dad553)" style="fill: #087f8c"/>+ </g>+ <g id="patch_4">+ <path d="M 118.08 109.415845+L 193.344743 109.415845+L 193.344743 94.704155+L 118.08 94.704155+z+" clip-path="url(#p9188dad553)" style="fill: #087f8c"/>+ </g>+ <g id="patch_5">+ <path d="M 118.08 141.397781+L 221.300219 141.397781+L 221.300219 126.68609+L 118.08 126.68609+z+" clip-path="url(#p9188dad553)" style="fill: #087f8c"/>+ </g>+ <g id="patch_6">+ <path d="M 171.840531 77.43391+L 279.361593 77.43391+L 279.361593 62.722219+L 171.840531 62.722219+z+" clip-path="url(#p9188dad553)" style="fill: url(#h107803ec0a); stroke: #7961a8; stroke-linejoin: miter"/>+ </g>+ <g id="patch_7">+ <path d="M 193.344743 109.415845+L 193.344743 109.415845+L 193.344743 94.704155+L 193.344743 94.704155+z+" clip-path="url(#p9188dad553)" style="fill: url(#h107803ec0a); stroke: #7961a8; stroke-linejoin: miter"/>+ </g>+ <g id="patch_8">+ <path d="M 221.300219 141.397781+L 221.300219 141.397781+L 221.300219 126.68609+L 221.300219 126.68609+z+" clip-path="url(#p9188dad553)" style="fill: url(#h107803ec0a); stroke: #7961a8; stroke-linejoin: miter"/>+ </g>+ <g id="matplotlib.axis_1">+ <g id="xtick_1">+ <g id="line2d_1">+ <defs>+ <path id="m59e2246da5" d="M 0 0+L 0 3.5+" style="stroke: #5a6873; stroke-width: 0.8"/>+ </defs>+ <g>+ <use xlink:href="#m59e2246da5" x="118.08" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_1">+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="118.08" y="164.71075" transform="rotate(-0 118.08 164.71075)">$0.00</text>+ </g>+ </g>+ <g id="xtick_2">+ <g id="line2d_2">+ <g>+ <use xlink:href="#m59e2246da5" x="171.840531" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_2">+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="171.840531" y="164.71075" transform="rotate(-0 171.840531 164.71075)">$0.10</text>+ </g>+ </g>+ <g id="xtick_3">+ <g id="line2d_3">+ <g>+ <use xlink:href="#m59e2246da5" x="225.601062" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_3">+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="225.601062" y="164.71075" transform="rotate(-0 225.601062 164.71075)">$0.20</text>+ </g>+ </g>+ <g id="xtick_4">+ <g id="line2d_4">+ <g>+ <use xlink:href="#m59e2246da5" x="279.361593" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_4">+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="279.361593" y="164.71075" transform="rotate(-0 279.361593 164.71075)">$0.30</text>+ </g>+ </g>+ </g>+ <g id="matplotlib.axis_2">+ <g id="ytick_1">+ <g id="line2d_5"/>+ <g id="text_5">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="114.58" y="73.497361" transform="rotate(-0 114.58 73.497361)">Pair 1 · E1/C1</text>+ </g>+ </g>+ <g id="ytick_2">+ <g id="line2d_6"/>+ <g id="text_6">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="114.58" y="105.479297" transform="rotate(-0 114.58 105.479297)">Pair 2 · E2/C1</text>+ </g>+ </g>+ <g id="ytick_3">+ <g id="line2d_7"/>+ <g id="text_7">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="114.58" y="137.461232" transform="rotate(-0 114.58 137.461232)">Pair 3 · E2/C2</text>+ </g>+ </g>+ </g>+ <g id="patch_9">+ <path d="M 118.08 151.632+L 316.993964 151.632+" style="fill: none; stroke: #d8dde1; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>+ </g>+ <g id="text_8">+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="283.12483" y="72.546209" transform="rotate(-0 283.12483 72.546209)">$0.300</text>+ </g>+ <g id="text_9">+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="197.10798" y="104.528145" transform="rotate(-0 197.10798 104.528145)">$0.140</text>+ </g>+ <g id="text_10">+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="225.063456" y="136.51008" transform="rotate(-0 225.063456 136.51008)">$0.192</text>+ </g>+ <g id="text_11">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="118.08" y="37.488" transform="rotate(-0 118.08 37.488)">Total operating loss</text>+ </g>+ <g id="legend_1">+ <g id="patch_10">+ <path d="M 114.245162 10.499811+L 132.245162 10.499811+L 132.245162 4.199811+L 114.245162 4.199811+z+" style="fill: #087f8c"/>+ </g>+ <g id="text_12">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="139.445162" y="10.499811" transform="rotate(-0 139.445162 10.499811)">Execution cost</text>+ </g>+ <g id="patch_11">+ <path d="M 223.143756 10.499811+L 241.143756 10.499811+L 241.143756 4.199811+L 223.143756 4.199811+z+" style="fill: url(#h107803ec0a); stroke: #7961a8; stroke-linejoin: miter"/>+ </g>+ <g id="text_13">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="248.343756" y="10.499811" transform="rotate(-0 248.343756 10.499811)">Failure penalty</text>+ </g>+ </g>+ </g>+ <g id="axes_2">+ <g id="patch_12">+ <path d="M 413.773258 151.632+L 566.784 151.632+L 566.784 52.488+L 413.773258 52.488+z+" style="fill: #ffffff"/>+ </g>+ <g id="patch_13">+ <path d="M 413.773258 77.43391+L 523.066645 77.43391+L 523.066645 62.722219+L 413.773258 62.722219+z+" clip-path="url(#p9b4b3d3bd8)" style="fill: #5a6873"/>+ </g>+ <g id="patch_14">+ <path d="M 413.773258 109.415845+L 464.776839 109.415845+L 464.776839 94.704155+L 413.773258 94.704155+z+" clip-path="url(#p9b4b3d3bd8)" style="fill: #087f8c"/>+ </g>+ <g id="patch_15">+ <path d="M 413.773258 141.397781+L 483.721026 141.397781+L 483.721026 126.68609+L 413.773258 126.68609+z+" clip-path="url(#p9b4b3d3bd8)" style="fill: #7961a8"/>+ </g>+ <g id="matplotlib.axis_3">+ <g id="xtick_5">+ <g id="line2d_8">+ <g>+ <use xlink:href="#m59e2246da5" x="413.773258" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_14">+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="413.773258" y="164.71075" transform="rotate(-0 413.773258 164.71075)">$0.000</text>+ </g>+ </g>+ <g id="xtick_6">+ <g id="line2d_9">+ <g>+ <use xlink:href="#m59e2246da5" x="468.419952" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_15">+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="468.419952" y="164.71075" transform="rotate(-0 468.419952 164.71075)">$0.025</text>+ </g>+ </g>+ <g id="xtick_7">+ <g id="line2d_10">+ <g>+ <use xlink:href="#m59e2246da5" x="523.066645" y="151.632" style="fill: #5a6873; stroke: #5a6873; stroke-width: 0.8"/>+ </g>+ </g>+ <g id="text_16">+ <text style="font-size: 8px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #5a6873" x="523.066645" y="164.71075" transform="rotate(-0 523.066645 164.71075)">$0.050</text>+ </g>+ </g>+ </g>+ <g id="matplotlib.axis_4">+ <g id="ytick_4">+ <g id="line2d_11"/>+ <g id="text_17">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="410.273258" y="73.49701" transform="rotate(-0 410.273258 73.49701)">4/6 pass</text>+ </g>+ </g>+ <g id="ytick_5">+ <g id="line2d_12"/>+ <g id="text_18">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="410.273258" y="105.478945" transform="rotate(-0 410.273258 105.478945)">6/6 pass</text>+ </g>+ </g>+ <g id="ytick_6">+ <g id="line2d_13"/>+ <g id="text_19">+ <text style="font-size: 9px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #181818" x="410.273258" y="137.460881" transform="rotate(-0 410.273258 137.460881)">6/6 pass</text>+ </g>+ </g>+ </g>+ <g id="patch_16">+ <path d="M 413.773258 151.632+L 566.784 151.632+" style="fill: none; stroke: #d8dde1; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>+ </g>+ <g id="text_20">+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="527.438381" y="72.546209" transform="rotate(-0 527.438381 72.546209)">$0.0500</text>+ </g>+ <g id="text_21">+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="469.148574" y="104.528145" transform="rotate(-0 469.148574 104.528145)">$0.0233</text>+ </g>+ <g id="text_22">+ <text style="font-size: 9.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="488.092761" y="136.51008" transform="rotate(-0 488.092761 136.51008)">$0.0320</text>+ </g>+ <g id="text_23">+ <text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #181818" x="413.773258" y="37.488" transform="rotate(-0 413.773258 37.488)">Mean loss per task</text>+ </g>+ </g>+ </g>+ <defs>+ <clipPath id="p9188dad553">+ <rect x="118.08" y="52.488" width="198.913964" height="99.144"/>+ </clipPath>+ <clipPath id="p9b4b3d3bd8">+ <rect x="413.773258" y="52.488" width="153.010742" height="99.144"/>+ </clipPath>+ </defs>+ <defs>+ <pattern id="h107803ec0a" patternUnits="userSpaceOnUse" x="0" y="0" width="72" height="72">+ <rect x="0" y="0" width="73" height="73" fill="#e4ddea"/>+ <path d="M -36 36+L 36 -36+M -32 40+L 40 -32+M -28 44+L 44 -28+M -24 48+L 48 -24+M -20 52+L 52 -20+M -16 56+L 56 -16+M -12 60+L 60 -12+M -8 64+L 64 -8+M -4 68+L 68 -4+M 0 72+L 72 0+M 4 76+L 76 4+M 8 80+L 80 8+M 12 84+L 84 12+M 16 88+L 88 16+M 20 92+L 92 20+M 24 96+L 96 24+M 28 100+L 100 28+M 32 104+L 104 32+M 36 108+L 108 36+" style="fill: #7961a8; stroke: #7961a8; stroke-width: 1.0; stroke-linecap: butt; stroke-linejoin: miter"/>+ </pattern>+ </defs>+</svg>diff --git a/docs/index.md b/docs/index.mdindex a385f379d..c714b2b01 100644--- a/docs/index.md+++ b/docs/index.md@@ -32,7 +32,7 @@ Use it with [Claude Code or Codex](recipes/single_provider_coding_agents.md),- [Host Integration](reference/host_integration.md): connect your clients and application lifecycle- [Rust API](reference/rust_api.md): browse the core library, protocols, translation, clients, and runner- [Context-Window Handling](operations/context_window.md): configure eviction and retry behavior-- [Tuning](tuning/index.md): the loop from verifying routing to recording a result+- [Tuning](tuning/index.md): foundations and worked configuration comparisons- [Troubleshooting](operations/troubleshooting.md): symptoms, silent misconfigurations, and fixes## Referencediff --git a/docs/routing_algorithms/model_selection.md b/docs/routing_algorithms/model_selection.mdindex 057a8a374..9a08752d3 100644--- a/docs/routing_algorithms/model_selection.md+++ b/docs/routing_algorithms/model_selection.md@@ -5,6 +5,9 @@ you put in each role change results more than any threshold does. This page sayswhat each role needs, how to check a model before you route to it, which pairshave measured results, and the mistakes we have seen in real runs.+For a worked comparison of cost and task success, see the+[model-pair selection example](../tuning/examples.md#model-pair-selection-with-fixed-algorithm-settings).+## RolesEach strategy names its models by role. The TOML keys differ by route type:diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.mdindex f2e8a0a6a..bbda63f3e 100644--- a/docs/routing_algorithms/overview.md+++ b/docs/routing_algorithms/overview.md@@ -27,7 +27,7 @@ can optionally call an LLM judge. The TOML configuration keys are unchanged.Whichever strategy you choose, the two models you put behind it matter more thanany threshold. See [Model Selection](model_selection.md). To tune cost against-quality, follow the [tuning loop](../tuning/index.md).+quality, see [Examples](../tuning/examples.md).### What the names meandiff --git a/docs/tuning/examples.md b/docs/tuning/examples.mdnew file mode 100644index 000000000..4fc030a2b--- /dev/null+++ b/docs/tuning/examples.md@@ -0,0 +1,199 @@+---+title: Tuning Examples+---++# Tuning Examples++These two comparisons show how to choose routing settings or a model pair for+a fixed objective. Start with [Tuning Foundations](foundations.md) to define+that objective and design candidates. All numbers are illustrative, not measured+benchmark results. The final section outlines planned GEPA support.++## Tune configuration options for cost++Suppose the goal is to **minimize mean cost per task while passing at least 85%+of tasks**. Keep the routing algorithm and model pair fixed, then compare its+configuration options.++### Choose the stage-router controls to tune++This example focuses on the stage router. The same process applies to other+routing algorithms: choose controls that affect your objective, compare+candidates, and validate the selected configuration.++Stage router has many fields. With the model pair fixed, start with the controls+that most directly change the routing split and cost: **profile, threshold, and+hold turns**. The profile is selected by the `picker` field:++| Profile (`picker`) | Default behavior | When it may fit better |+|---|---|---|+| `efficient_first` | Use efficient on ambiguous turns; escalate when signals call for capable. | The efficient model handles most work, and occasional escalation can recover difficult tasks. A useful starting point for reducing cost. |+| `capable_first` | Use capable on ambiguous turns; offload when signals support efficient. | Ambiguous or difficult work needs the capable model, with clear routine stretches to offload. Experimental: calibrate on your workload before relying on it. |++The threshold changes which signals trigger a switch, and therefore the share+of requests each tier serves. Its direction depends on the profile:++| Control | Profile | Lower value | Higher value |+|---|---|---|---|+| `confidence_threshold` | `efficient_first` | Easier escalation; more capable traffic, tending toward higher model cost. | Harder escalation; more efficient traffic, tending toward lower model cost. |+| `confidence_threshold` | `capable_first` | Easier offloading; more efficient traffic, tending toward lower model cost. | Harder offloading; more capable traffic, tending toward higher model cost. |+| `capable_hold_turns` | Both | Fewer forced capable requests; routing can return to efficient sooner. | More requests stay capable after switching, allowing longer recovery at greater capable usage. |+| `efficient_hold_turns` | Both | Fewer forced efficient requests; routing can escalate sooner. | More requests stay efficient after switching, which can save cost but delay escalation even when errors appear. |++These threshold effects describe signal-based routing without a classifier;+active holds and hard overrides take precedence. For holds, `0` disables the+hold; positive values count requests, while `-1` pins the tier after a switch.+Longer holds can reduce switching and preserve cached context, so measure total+cost, retries, and task success together. See the+[stage-router guide](../routing_algorithms/stage_router_routing.md#pickers) for+profile and threshold details.++**Handoff notes can break cache reuse across turns.** They are added only to the forwarded+request, so later requests can omit the note and break reuse of the cached+continuation. A fix is planned for a future release. Leave `handoff_notes`+unconfigured for this cost comparison; see+[handoff notes](../routing_algorithms/stage_router_routing.md#optional-handoff-notes).++### Split the benchmark and collect baselines++Use **20% for calibration and 80% for held-out validation**, chosen before tuning.+For this example's 500-task benchmark, tune on 100 tasks and reserve 400 for the+frozen configuration. Keep related tasks together and both splits representative;+see [split guidance](foundations.md#4-check-the-final-choice).++First run efficient-only, capable-only, and the current routed configuration on+the same 100 calibration tasks. These illustrative baselines show what routing+must improve on:++| Baseline | Tasks passed | Mean cost per task |+|---|---|---|+| Efficient-only | 64/100 | $0.010 |+| Capable-only | 94/100 | $0.040 |+| Current routing (C below) | 88/100 | $0.030 |++Use the same agent, task limits, serving settings, and initial cache setup for+every run. Include all billed calls, retries, and cache charges.++### Compare six configurations++For this experiment, choose the efficient-first profile and vary only the+threshold and capable hold. This isolates their effect before trying another+profile.++Use `stage_router` with `picker = "efficient_first"`, no classifier,+`recent_turn_window = 5`, and `efficient_hold_turns = 0`. Keep the model pair,+serving settings, agent, task limits, initial cache setup, and all other options+fixed. Compare all six candidates on the same 100 calibration tasks.++Run each candidate end to end: changed routing can alter later+prompts, cache state, and costs, so reusing one run's outcomes can misrank them.++Here are results from one run per calibration task, with C as the baseline.+Compare mean cost among candidates meeting the 85% target. Include failed tasks,+retries, and cache charges in that cost.++| Candidate | Threshold | Hold (requests) | Tasks passed | Mean cost per task |+|---|---|---|---|---|+| A | 0.7 | 0 | 70/100 | $0.012 |+| B | 0.5 | 0 | 80/100 | $0.018 |+| C (baseline) | 0.3 | 0 | 88/100 | $0.030 |+| D | 0.7 | 2 | 78/100 | $0.020 |+| E | 0.5 | 2 | 88/100 | $0.024 |+| F | 0.3 | 2 | 94/100 | $0.038 |++E improves on C: it passes the same number of tasks and costs less. B improves+on D in both cost and task success. Remove C and D from consideration.++**A, B, E, and F form the Pareto frontier among these six routed candidates.**+No other candidate in this sweep matches or improves both measures while+improving at least one. These options offer the remaining cost/success tradeoffs:++++*Figure 1. The circles form the observed Pareto frontier. Crosses mark candidates+that another option improves on. The dashed line is the 85% task-success target.*++**Select E under this objective.** A and B miss the success target. E and F both+meet it, but E costs less: $0.024 per task, or $2.40 across 100 tasks. E reduces+cost by 20% from baseline C while preserving its observed 88% success rate.+It also costs 40% less than capable-only, with six fewer tasks passed, while+still meeting the 85% target. F buys a higher success rate at a higher cost.++The frontier describes the calibration sample. Repeat promising runs, then+freeze the selection. Run E and all three baselines on the untouched 400-task+validation split to check whether the cost advantage and success target hold.++## Model-pair selection with fixed algorithm settings++This independent example fixes the algorithm settings and changes the+efficient/capable model pair. Score each task as **execution cost plus $0.10 if+it fails**, then minimize the mean score. Choose that penalty before comparing+results; it expresses how much failure matters for this example.++Use `stage_router` with `picker = "efficient_first"`, `confidence_threshold = 0.5`,+`recent_turn_window = 5`, `capable_hold_turns = 2`, `efficient_hold_turns = 0`, and+no classifier or handoff notes. Keep this profile, all other settings, and the+evaluation conditions fixed while changing models. Check+[model compatibility](../routing_algorithms/model_selection.md) before running.++Use the same **20% calibration / 80% held-out validation** split. To keep the+arithmetic small, this separate example uses 30 tasks: six for calibration and+24 reserved for validation. Run each candidate end to end with the same agent,+task limits, serving settings, and initial cache setup.++First measure each model alone on the six calibration tasks. These fictional+single-model baselines use the same $0.10 failure penalty:++| Single-model baseline | Tasks passed | Total execution cost | Mean loss per task |+|---|---|---|---|+| E1 only | 3/6 | $0.048 | $0.0580 |+| E2 only | 4/6 | $0.072 | $0.0453 |+| C1 only | 6/6 | $0.240 | $0.0400 |+| C2 only | 6/6 | $0.300 | $0.0500 |++Now compare three routed pairs on those same tasks. Pair 1 is the current routed+baseline. Pair 2 changes only the efficient model from Pair 1; Pair 3 changes+only the capable model from Pair 2. Fixed routing settings can still produce+different decisions because model outputs affect later routing signals.++| Pair | Efficient / capable | Tasks passed | Total execution cost | Failure penalty | Mean loss per task |+|---|---|---|---|---|---|+| 1 | E1 / C1 | 4/6 | $0.100 | $0.200 | $0.0500 |+| 2 | E2 / C1 | 6/6 | $0.140 | $0.000 | $0.0233 |+| 3 | E2 / C2 | 6/6 | $0.192 | $0.000 | $0.0320 |++Include every billed call, retry, and cache charge. For Pair 1, add $0.100 in+execution costs and $0.200 for two failures, then divide by six tasks: $0.0500+per task. Failure penalties are part of the score, not billed charges.++++*Figure 2. Model-pair comparison under fixed algorithm settings.*++**Select Pair 2.** Pair 1 has the lowest billed cost but loses its advantage when+failures count. Pair 3 costs more without improving success on this sample.+Pair 2 also passes all six tasks at lower cost than C1 alone, the best+single-model baseline under this scoring rule.++Freeze Pair 2 and run it, the current Pair 1, and the single-model baselines on+the untouched 24-task validation split. Compare mean loss and task success before+making a deployment choice; follow the+[final validation guidance](foundations.md#4-check-the-final-choice).++## Upcoming: GEPA tuning for the unified router++GEPA-based prompt tuning for the unified router is planned for a future release,+with tooling and concrete APIs to run the workflow. GEPA proposes instruction+changes from feedback, evaluates them, and repeats while model weights stay+fixed. See the [GEPA paper](https://arxiv.org/abs/2507.19457) for the approach.++The planned example will keep the model pool and objective fixed while changing+the router's instructions:++- Use calibration-task outcomes to show where routing choices helped or hurt the+ objective. A reflector model uses that feedback to propose revised instructions.+- Compare proposed prompts within the calibration split using the same cost+ and success measures, including the cost of routing calls.+- Freeze the chosen prompt and evaluate it on the held-out validation split.++The API walkthrough and worked results will accompany the tooling release.diff --git a/docs/tuning/foundations.md b/docs/tuning/foundations.mdnew file mode 100644index 000000000..9cdc938ce--- /dev/null+++ b/docs/tuning/foundations.md@@ -0,0 +1,84 @@+# Tuning Foundations++This guide helps you find a **golden configuration for your objective and+workload**: define the outcome you want, compare candidate settings on+calibration tasks, and validate the selected configuration on held-out tasks+before adopting it as your baseline.++See [Examples](examples.md) for configuration and model-pair comparisons+with concrete data.++## 1. Define the objective++Define what the user wants to improve, on which workload, and what makes a result+acceptable. Keep this objective fixed throughout the comparison.++- **Reduce cost while preserving accuracy.** Minimize total cost per task while+ meeting a defined accuracy requirement. Include decision calls, retries, and+ cache charges in the cost.+- **Maximize task success.** Maximize the fraction of tasks completed correctly,+ subject to any cost or latency limits.+- **Reach a desired routing split.** For example, target 70% of calls on the+ efficient tier and 30% on the capable tier. Specify the allowed deviation and+ quality requirement.+- **Reach a capable-tier cache-usage target.** For example, require at least 80%+ of capable-tier input tokens to be served from cache. Measure cache-read input+ tokens divided by all input tokens on that tier, with a defined quality floor.++Once the objective is measurable, the next step is to choose configurations that+might achieve it. This separates the outcome we want from the settings we will+test.++## 2. Design configuration candidates++A candidate is a complete configuration to test under the objective. Start with+a baseline and a small set of changes you expect to help:++1. **Record baselines.** Measure efficient-only, capable-only, and the current+ routed configuration on the same tasks. Save their settings and outcomes.+2. **Choose what varies.** For example, change model targets, thresholds, or hold+ durations. Keep other settings fixed and state why each change might help.+3. **Test interactions.** A threshold and hold duration can affect the same+ routing decisions, so include combinations when needed.+4. **Keep runs comparable.** Use the same tasks, evaluation procedure, and initial+ cache setup. Pin model and provider versions, and save each configuration.++Run the same tasks for every candidate. The following examples show how to use+the results to choose between them.++## 3. Choose an example++Start with the comparison that matches what you want to change:++- **[Tune configuration options](examples.md#tune-configuration-options-for-cost):**+ keep the algorithm and model pair fixed, vary routing settings, and find the+ lowest-cost option that meets the task-success target. A Pareto chart shows+ the cost and success tradeoffs.+- **[Select a model pair](examples.md#model-pair-selection-with-fixed-algorithm-settings):**+ keep algorithm settings fixed, swap the efficient/capable models, and compare+ execution cost plus a penalty for failed tasks.+- **[Tune the unified router with GEPA](examples.md#upcoming-gepa-tuning-for-the-unified-router):**+ planned for a future release. Use benchmark feedback to revise routing+ instructions and select a prompt against a fixed objective.++## 4. Check the final choice++Split **100% of the evaluation benchmark before tuning**. A practical starting+allocation is:++| Split | Share | Use |+|---|---|---|+| Calibration | 20% | Collect baselines, try settings, and select the candidate. |+| Held-out validation | 80% | Evaluate the frozen candidate against the baselines. Do not use these results to tune it. |++For 500 tasks, this means 100 calibration tasks and 400 validation tasks. Adjust+the percentages before running if the calibration set is too small to cover the+workload. Here, validation is the final held-out check.++- Keep related examples in the same split. If predicting future data, respect+ time when splitting. Use representative workloads and repeated trials when+ execution is stochastic.+- Record the objective, scoring rule, configurations, and evaluation conditions.+ Changing the models or settings requires new measurements.+- If validation results guide another change, those tasks have become+ calibration data. Use fresh held-out tasks for the next independent check.diff --git a/docs/tuning/index.md b/docs/tuning/index.mdindex e3c0480a0..89b4b62a2 100644--- a/docs/tuning/index.md+++ b/docs/tuning/index.md@@ -1,95 +1,11 @@# Tuning-Routing has two outcomes you can measure: how many tasks the agent completes,-and what a run costs. Tuning is the loop below. Every step links the page that-explains it, names what it produces, and says what it costs to run.+Tuning helps you find a configuration that meets your goals on your workload.+Start with Foundations to plan the comparison, then use Examples to+see how to evaluate candidates and choose a configuration.-Start where your goal is:--| Your goal | Start at |-|---|---|-| Lower cost at the same quality | Step 1, then step 4 for the baseline numbers |-| Add a cheaper model safely | Step 2 |-| Routing is on, and results got worse | Step 5 |--## The loop--1. **Verify routing is active.** Follow- [Verify routing is active](../operations/local_evaluation.md#verify-routing-is-active).- Produces: the selected-model header from two requests and a `/v1/stats`- capture. Costs: two requests, no agent run.--2. **Choose and check the models.** Run the- [five checks](../routing_algorithms/model_selection.md#check-the-efficient-model-before-you-route)- on the efficient model, then the- [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair).- Produces: a pass or fail per check, and solved tasks plus cost for each- fixed arm. Costs: single requests for most checks, one short agent run for- the tool-calling check, and one agent run per arm on at least 20 tasks.--3. **Start from a measured profile.** Pick the closest row in- [Measured pairs](../routing_algorithms/model_selection.md#measured-pairs)- and copy its profile from `benchmark/routing-profiles/`. The- [strategy comparison](../routing_algorithms/overview.md#compare-the-strategies)- says which route type fits your traffic. Produces: a route file that passes- `switchyard-server --config routes.toml --dry-run`. Costs: none.--4. **Run the routed arm against the fixed arms.** Use the same tasks, agent,- and provider for all three arms. The [benchmark guide](../../benchmark/README.md)- shows the harness the checked-in results used, and the- [Benchmarks](../benchmarks.md) page lists the published results with their- configs. Produces: solved tasks and- cost per run for each arm. Costs: `k` runs per arm. The plan/execute and- advisor profiles use three runs and report the mean with the sample standard- deviation, as in the- [plan/execute profile header](../../benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml).- The stage profile records a single run. A single run cannot separate a small- change from noise.--5. **Read the decision mix.** For stage routes, read- [decision sources](../routing_algorithms/stage_router_routing.md#decision-sources)- from `/v1/stats`. For escalation routes, read the- [observability](../routing_algorithms/escalation_router_routing.md#observability)- section. For advisor routes, `/v1/stats` reports verdicts and consult- failures under `advisor_gate`. Start the server with `--routing-log-file`- to keep one record per model call: the served call, and each judge call- under the judge's model ID. Filter by `model` before adding up usage.- Produces: the `routing_decisions` capture and the routing log. Costs: none- beyond the runs in step 4.--6. **Move one dial.** Change one setting, then repeat step 4.-- | Strategy | First dial | Guidance |- |---|---|---|- | Execution, Auto | `confidence_threshold` | [Tuning `confidence_threshold`](../routing_algorithms/stage_router_routing.md#tuning-confidence_threshold) and [Calibrating the threshold from run data](../routing_algorithms/stage_router_routing.md#calibrating-the-threshold-from-run-data) |- | Task | `base_threshold`, then `classify_trigger` | [Tuning options](../routing_algorithms/llm_classifier_routing.md#tuning-options) and [When the judge runs](../routing_algorithms/llm_classifier_routing.md#when-the-judge-runs) |- | Escalation | `confirmations` | [Tuning options](../routing_algorithms/escalation_router_routing.md#tuning-options) |- | Advisor Gate | `max_reviews` with `gate_min_tool_results` | [Tuning](../routing_algorithms/advisor_gate_routing.md#tuning) |- | Composite | `efficient_first_threshold` and `capable_first_threshold` | [What exists today](../routing_algorithms/composite_routing.md#what-exists-today) |-- Produces: a before-and-after table with the arm results from step 4.- Costs: `k` runs per setting you try.--7. **Record the result.** Write down what you ran, in the form the profile- headers use:-- - the Switchyard commit;- - each model with its provider, wire format, and reasoning effort;- - the agent and its version;- - the benchmark, the task count, and `k`;- - the result band next to the fixed arms;- - the cost per run.-- The- [plan/execute profile header](../../benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml)- records all of these. Produces: the header. Costs: none.--8. **Recalibrate on change.** A new model version, provider, wire format,- reasoning effort, agent version, or workload moves the operating point, and- the thresholds do not carry over. See- [Pin the provider and endpoint](../routing_algorithms/model_selection.md#pin-the-provider-and-endpoint).- Return to step 4 with the recorded header as your baseline. Produces: a new- header next to the old one. Costs: the runs of step 4.+- [Tuning Foundations](foundations.md): define your objective, candidates, and evaluation splits.+- [Examples](examples.md): compare routing settings and model pairs with worked examples.## Relateddiff --git a/mkdocs.yml b/mkdocs.ymlindex 33d7d3fac..d50e697c9 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -44,7 +44,9 @@ nav:- Choose Models: routing_algorithms/model_selection.md- Evaluate Routing:- Benchmarks: benchmarks.md- - Compare and Tune: tuning/index.md+ - Tuning: tuning/index.md+ - Tuning Foundations: tuning/foundations.md+ - Examples: tuning/examples.md- Local Demo:- Run the Reference Server: operations/local_evaluation.md- Local Agents: