Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

ci: skip live evals without credentials

JerrettDavis2fd48260f6b9
diff --git a/.github/workflows/eval.yml b/.github/workflows/eval.yml
index 640904fc7..4fc3ad331 100644
--- a/.github/workflows/eval.yml
+++ b/.github/workflows/eval.yml
@@ -86,7 +86,17 @@ jobs:
pip install -e ".[all]"
python -c "from headroom._core import SmartCrusher; print('headroom._core OK')"
- name: Run Tier 1 evaluation suite
- run: python -m headroom.evals suite --tier 1 --ci -o eval_results/
+ run: |
+ if [ -z "${OPENAI_API_KEY}" ]; then
+ echo "::warning title=Weekly eval skipped::OPENAI_API_KEY is not configured for this repo; skipping the live Tier 1 suite."
+ mkdir -p eval_results
+ printf '%s\n\n%s\n' \
+ '# Weekly Evaluation Skipped' \
+ 'OPENAI_API_KEY is not configured for this repository, so the live Tier 1 evaluation suite was skipped.' \
+ > eval_results/skipped.md
+ exit 0
+ fi
+ python -m headroom.evals suite --tier 1 --ci -o eval_results/
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
diff --git a/pyproject.toml b/pyproject.toml
index f05a02450..0ac0beafb 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -193,7 +193,7 @@ html = [
]
# Comprehensive LLM benchmarks
benchmark = [
- "lm-eval>=0.4.0",
+ "lm-eval[api]>=0.4.0",
"openai>=1.0.0",
"anthropic>=0.18.0",
]