Skip to content

Agent Mesh

agents: a flaky test is a bug report

Antonio Ojead48b1dc643e2
diff --git a/.gemini/styleguide.md b/.gemini/styleguide.md
index ddda31a1..a59ec105 100644
--- a/.gemini/styleguide.md
+++ b/.gemini/styleguide.md
@@ -263,10 +263,14 @@ instrumentation as much as in the logic.
ID; a line about a lease names the addresses; a line about a router
names the router. Flag lines that say what happened without saying to
whom.
-5. **A flaky-test fix adds instrumentation.** A PR that makes a test
- deterministic by waiting on a condition must expose that condition in
- the binary (a gauge, a `/debug` view, a log line) and wait on it there,
- not on a `time.Sleep`. Flag a fix whose only change is in the test.
+5. **A flaky-test fix adds instrumentation.** A flake is a race the
+ production code has too. A PR that makes a test deterministic by
+ waiting on a condition must expose that condition in the binary (a
+ gauge, a `/debug` view, a log line) and wait on it there, not on a
+ `time.Sleep`. Flag a fix whose only change is in the test, and flag a
+ retry loop, a widened timeout or a sleep added to a test that failed in
+ CI: ask for the reproduction (`stress` with `-race`, see `AGENTS.md`
+ §3) and the code change it led to.
6. **Safe defaults, zero configuration.** A new flag or option needs a
default that is secure without the operator reading the docs: deny, verify,
short-lived, loopback-only, bounded. Flag a default that opens something
diff --git a/AGENTS.md b/AGENTS.md
index a3bab059..854e4fff 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -31,6 +31,7 @@ Enforce strict modularity in testing. The repository uses a defined testing pyra
* **Integration Tests (`tests/integration/`):** Verify module interactions and API compliance in Go and those are time bounded, no more than 10 seconds per execution.
* **E2E Tests (`tests/e2e/*.bats`):** Use Bats (Bash Automated Testing System) exclusively for high-level, black-box testing of core CUJs.
* **Idiomatic Go, no test frameworks:** Go tests use the standard `testing` package and nothing else: `t.Run` for cases, table-driven tests, `t.Fatalf`/`t.Errorf` with a message that states what was got and what was wanted, `t.Cleanup` for teardown, `t.TempDir` for files, `httptest` and `net.Listen("tcp", "127.0.0.1:0")` for servers. Ginkgo, Gomega, Testify and any other assertion or matcher library are not used and must not be added, directly or through a dependency. Test code is production code: the people who write and read the implementation write and read its tests, so a test is plain Go that compiles under the same linter and reads top to bottom without a DSL to learn. A helper is justified by repetition across tests, not by making one test shorter; do not build fixture hierarchies, suite objects or matcher abstractions. A failing test must say in its own output what happened, and must not swallow an error to keep going.
+* **A flaky test is a bug report:** a test that fails once in CI and passes on a rerun has found a race or an ordering the code did not account for, and that race exists in production too, where it has no rerun. Do not retry it, widen its timeout or add a `time.Sleep`. Reproduce it first: compile the test binary with the race detector and run it in parallel under load, `go test -c -race -o /tmp/pkg.test ./internal/pkg && stress -p 8 -timeout 2m /tmp/pkg.test -test.run 'TestName'` (`golang.org/x/tools/cmd/stress`), and starve it of CPU with `taskset -c 0-1` when the CI runner is smaller than the workstation. Read the failure from the binaries' own output; when the logs and metrics in the failure do not explain it, that is the gap to close first, in the production code: the log line, counter or gauge for the state the test could not see. Then fix the code, make the test wait on the exported condition rather than on time, and run the stress loop again until it is clean for hundreds of runs. Tests in `tests/integration/` reserve ports from one fixed base per process, so several processes of that suite collide on ports; run it with `-p 1` under `taskset` and reserve `-p N` for package unit tests.
## 4. Code Quality & Modularity
* Ensure all new code is highly modular, prioritizing small, single-responsibility functions that are easy to unit test.