Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

ci(hermes-plugin): run external provider regression tests (#5418)

z1gond2e585b128d2
diff --git a/.github/workflows/hermes-plugin-tests.yml b/.github/workflows/hermes-plugin-tests.yml
new file mode 100644
index 000000000..5b54bf106
--- /dev/null
+++ b/.github/workflows/hermes-plugin-tests.yml
@@ -0,0 +1,73 @@
+name: Hermes Plugin Tests
+
+on:
+ pull_request:
+ branches: [main, develop]
+ paths:
+ - 'examples/hermes-plugin/**'
+ - '.github/workflows/hermes-plugin-tests.yml'
+ push:
+ branches: [main, develop]
+ paths:
+ - 'examples/hermes-plugin/**'
+ - '.github/workflows/hermes-plugin-tests.yml'
+ workflow_dispatch:
+
+permissions:
+ contents: read
+
+concurrency:
+ group: hermes-plugin-tests-${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ hermes-plugin-tests:
+ runs-on: ubuntu-24.04
+ timeout-minutes: 15
+ steps:
+ - uses: actions/checkout@v6
+ with:
+ persist-credentials: false
+
+ - name: Check out the reviewed Hermes test host
+ uses: actions/checkout@v6
+ with:
+ repository: NousResearch/hermes-agent
+ ref: 9a0a1625367242596d338ae2da541c4a1fc785a2
+ path: .ci/hermes-agent
+ persist-credentials: false
+
+ - uses: actions/setup-python@v6
+ with:
+ python-version: '3.14'
+
+ - uses: astral-sh/setup-uv@v7
+ with:
+ version: '0.12.3'
+ enable-cache: true
+ cache-dependency-glob: |
+ .ci/hermes-agent/uv.lock
+ examples/hermes-plugin/pyproject.toml
+
+ - name: Install the host and plugin test dependencies
+ working-directory: .ci/hermes-agent
+ run: |
+ uv sync --frozen --group dev --no-install-project
+ uv export --frozen --group dev --no-emit-project --no-hashes \
+ --output-file "$RUNNER_TEMP/hermes-plugin-constraints.txt"
+ # Keep plugin dependencies within the reviewed host pins. PyYAML
+ # is used by the test fixtures; its version matches Hermes's lock.
+ uv pip install --python .venv/bin/python \
+ --constraint "$RUNNER_TEMP/hermes-plugin-constraints.txt" \
+ --requirement "$GITHUB_WORKSPACE/examples/hermes-plugin/pyproject.toml" \
+ 'PyYAML==6.0.3'
+
+ - name: Run the external memory provider tests
+ working-directory: .ci/hermes-agent
+ env:
+ HERMES_TEST_FILE_RETRIES: '0'
+ HERMES_TEST_WORKERS: '2'
+ run: |
+ HERMES_PYTHON="$PWD/.venv/bin/python" scripts/run_tests.sh \
+ "$GITHUB_WORKSPACE/examples/hermes-plugin/tests" \
+ --confcutdir="$GITHUB_WORKSPACE/examples/hermes-plugin/tests" -q
diff --git a/examples/hermes-plugin/DEVELOPMENT.md b/examples/hermes-plugin/DEVELOPMENT.md
index c9d18962e..d93648602 100644
--- a/examples/hermes-plugin/DEVELOPMENT.md
+++ b/examples/hermes-plugin/DEVELOPMENT.md
@@ -68,6 +68,14 @@ install or package the OpenViking server.
## Validation
+The `Hermes Plugin Tests` workflow runs this directory's complete external-provider
+suite on plugin changes, pushes to `main`/`develop`, and manual dispatch.
+It uses Python 3.14 and a reviewed Hermes commit, with test retries disabled.
+When updating the host SHA in `.github/workflows/hermes-plugin-tests.yml`, check
+the host dependency pins and run the suite before submitting the change.
+These regression tests use mock responses and local test servers; live-service
+validation remains part of release testing.
+
Use a Hermes checkout with its development dependencies installed.
The complete mirror tests require the committed-entry event contract introduced