Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

E2E: run every throwaway-stack lane on each release tag

Petr Bauch02fe9c683fdc
diff --git a/.github/workflows/e2e-scheduled.yml b/.github/workflows/e2e-scheduled.yml
index 55a1dcf92..2bd491e05 100644
--- a/.github/workflows/e2e-scheduled.yml
+++ b/.github/workflows/e2e-scheduled.yml
@@ -6,6 +6,13 @@ name: E2E scheduled
# after a merge to main the provider lane on a throwaway stack. Real model
# calls, so not on every PR, and never a required check
# nightly the provider lane again, then a notification
+# a release tag (switch-v*, candidates included) every lane that runs
+# on a throwaway stack: fast, provider and agentic, on
+# the tagged commit, always reported. A candidate may
+# be cut from a release branch, whose commits no main
+# run has seen, and it is the code about to be
+# promoted; the deployed environments test what they
+# run, not what is about to reach them
#
# A failed run after a merge is reported too, since the lane is not a required
# check and nothing else would surface it before the night; a passing one is
@@ -33,6 +40,9 @@ on:
- '!docs/**'
- '!**/*.md'
- 'console/packages/plugins/src/switch-skill/SKILL.md'
+ # Path filters do not apply to tags: every release tag runs.
+ tags:
+ - 'switch-v*'
workflow_dispatch:
permissions:
@@ -46,10 +56,25 @@ jobs:
lane: provider
agent_type: claude-code
+ fast:
+ name: Fast lane
+ if: ${{ startsWith(github.ref, 'refs/tags/') }}
+ uses: ./.github/workflows/e2e.yml
+ with:
+ lane: fast
+ agent_type: claude-code
+
+ agentic:
+ name: Agentic lane
+ if: ${{ startsWith(github.ref, 'refs/tags/') }}
+ uses: ./.github/workflows/e2e.yml
+ with:
+ lane: agentic
+ agent_type: claude-code
notify:
name: Report
- needs: [provider]
+ needs: [provider, fast, agentic]
if: ${{ always() && github.event_name != 'workflow_dispatch' }}
runs-on: ubuntu-latest
steps:
@@ -59,8 +84,14 @@ jobs:
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
PROVIDER_RESULT: ${{ needs.provider.result }}
PROVIDER_SUMMARY: ${{ needs.provider.outputs.summary }}
+ FAST_RESULT: ${{ needs.fast.result }}
+ FAST_SUMMARY: ${{ needs.fast.outputs.summary }}
+ AGENTIC_RESULT: ${{ needs.agentic.result }}
+ AGENTIC_SUMMARY: ${{ needs.agentic.outputs.summary }}
run: |
- if [ "$GITHUB_EVENT_NAME" = push ] && [ "$PROVIDER_RESULT" = success ]; then
+ tagged=false
+ [ "$GITHUB_REF_TYPE" = tag ] && tagged=true
+ if [ "$GITHUB_EVENT_NAME" = push ] && [ "$tagged" = false ] && [ "$PROVIDER_RESULT" = success ]; then
echo "The provider lane passed after the merge; nothing to report."
exit 0
fi
@@ -69,14 +100,21 @@ jobs:
[ "$2" = success ] && mark=✅
printf '%s %s — %s\n' "$mark" "$1" "${3:-$2, no scenario summary}"
}
- if [ "$GITHUB_EVENT_NAME" = push ]; then
+ if [ "$tagged" = true ]; then
+ title="Switch E2E on $GITHUB_REF_NAME"
+ elif [ "$GITHUB_EVENT_NAME" = push ]; then
title="Switch E2E after merging ${GITHUB_SHA:0:9}"
else
title='Switch E2E nightly'
fi
- text="$(printf '%s\n%s\n%s' "$title" \
- "$(line 'Provider lane (ephemeral)' "$PROVIDER_RESULT" "$PROVIDER_SUMMARY")" \
- "$RUN_URL")"
+ lines="$(line 'Provider lane (ephemeral)' "$PROVIDER_RESULT" "$PROVIDER_SUMMARY")"
+ if [ "$tagged" = true ]; then
+ lines="$(printf '%s\n%s\n%s' \
+ "$(line 'Fast lane (ephemeral)' "$FAST_RESULT" "$FAST_SUMMARY")" \
+ "$lines" \
+ "$(line 'Agentic lane (ephemeral)' "$AGENTIC_RESULT" "$AGENTIC_SUMMARY")")"
+ fi
+ text="$(printf '%s\n%s\n%s' "$title" "$lines" "$RUN_URL")"
echo "$text"
if [ -z "$WEBHOOK" ]; then
echo "::warning::E2E_NOTIFY_WEBHOOK_URL is not set; this report was not sent anywhere."
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
index 6326f05ff..7e64267eb 100644
--- a/.github/workflows/e2e.yml
+++ b/.github/workflows/e2e.yml
@@ -22,6 +22,8 @@ name: E2E
# GitHub Environment, whose protection rules decide which refs may run it:
#
# e2e-provider secret ANTHROPIC_API_KEY, for the controller's Claude Code
+# (and the agentic lane's driver and judge); open to main and
+# to the release tags, switch-v*
#
# Runs are strict (SWITCH_E2E_STRICT=1): a scenario that cannot run fails
# rather than skips.
diff --git a/console/tooling-e2e/README.md b/console/tooling-e2e/README.md
index 0e6c34751..79c8adf94 100644
--- a/console/tooling-e2e/README.md
+++ b/console/tooling-e2e/README.md
@@ -417,12 +417,14 @@ hand.
| Every PR, and every push to main (`pr-ci.yml`) | fast | ephemeral | yes |
| After a merge to main (`e2e-scheduled.yml`) | provider | ephemeral | no |
| Nightly (`e2e-scheduled.yml`) | provider | ephemeral | no |
-| By hand only, for now (`e2e.yml`, dispatch) | agentic | ephemeral | no |
+| A release tag, candidates included (`e2e-scheduled.yml`) | fast; provider; agentic | ephemeral | no |
+| By hand (`e2e.yml`, dispatch) | any | ephemeral | no |
- **Secrets.** An ephemeral fast run generates every secret it uses on the
runner, so it works the same on a fork PR. Every other run reads its secrets
from the GitHub Environment `e2e-provider`, which holds `ANTHROPIC_API_KEY`
- and whose protection rules decide which branches may use it.
+ and whose protection rules decide which refs may use it: `main`, and the
+ release tags (`switch-v*`).
- **Artifacts.** The output directory is uploaded only after it passes two
scans. The first (`scan-artifacts.sh`) looks for the exact secret values the
run used: the ones it was configured with, plus every credential the server
@@ -434,7 +436,8 @@ hand.
- **Reporting.** Each run writes a scenario table to the job summary. Reports
go to the webhook in the repository secret `E2E_NOTIFY_WEBHOOK_URL`:
- one summary each night;
- - a report when the provider lane fails after a merge.
+ - a report when the provider lane fails after a merge;
+ - a report for every release tag, pass or fail.
## Against a deployed environment