Skip to content

Building 70 Projects ranging from beginner to advanced so anyone can — learn from, build upon, use as a reference, or even copy directly. Owned by my cybersecurity company 👇

feat(khaos): validate recovery and package standalone installation

CarterPerez-dev25d496ddf2ca
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/.dockerignore b/PROJECTS/intermediate/chaos-engineering-tool/.dockerignore
index 19816542..39a89af4 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/.dockerignore
+++ b/PROJECTS/intermediate/chaos-engineering-tool/.dockerignore
@@ -4,5 +4,7 @@
*
!go.mod
!go.sum
+!bundle.go
!cmd/
!internal/
+!range/
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/.github/workflows/ci.yml b/PROJECTS/intermediate/chaos-engineering-tool/.github/workflows/ci.yml
index 49a60938..0e00675a 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/.github/workflows/ci.yml
+++ b/PROJECTS/intermediate/chaos-engineering-tool/.github/workflows/ci.yml
@@ -7,6 +7,7 @@ on:
push:
branches: ["**"]
pull_request:
+ workflow_call:
permissions:
contents: read
@@ -15,17 +16,18 @@ jobs:
check:
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@v4
+ - uses: actions/checkout@v7
- - uses: actions/setup-go@v5
+ - uses: actions/setup-go@v7
with:
go-version-file: go.mod
- - name: gofmt
- run: test -z "$(gofmt -l .)"
+ - uses: golangci/golangci-lint-action@v9
+ with:
+ version: v2.10.1
- - name: vet
- run: go vet ./...
+ - name: installer syntax and shell checks
+ run: bash -n install.sh && shellcheck install.sh
- name: module tidiness
run: go mod tidy -diff
@@ -34,7 +36,7 @@ jobs:
run: go test -race ./...
- name: build
- run: go build -o khaos ./cmd/khaos
+ run: CGO_ENABLED=0 go build -trimpath -o khaos ./cmd/khaos
- name: color degradation gate
run: ./scripts/colors.sh
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/.github/workflows/release.yml b/PROJECTS/intermediate/chaos-engineering-tool/.github/workflows/release.yml
new file mode 100644
index 00000000..b9968339
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/.github/workflows/release.yml
@@ -0,0 +1,36 @@
+# ©AngelaMos | 2026
+# release.yml
+
+name: release
+
+on:
+ push:
+ tags: ["v*"]
+
+permissions:
+ contents: read
+
+jobs:
+ check:
+ uses: ./.github/workflows/ci.yml
+
+ publish:
+ needs: check
+ runs-on: ubuntu-latest
+ permissions:
+ contents: write
+ steps:
+ - uses: actions/checkout@v7
+ with:
+ fetch-depth: 0
+
+ - uses: actions/setup-go@v7
+ with:
+ go-version-file: go.mod
+
+ - uses: goreleaser/goreleaser-action@v7
+ with:
+ version: v2.19.0
+ args: release --clean
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/.golangci.yml b/PROJECTS/intermediate/chaos-engineering-tool/.golangci.yml
index fe39eaed..805c3aaf 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/.golangci.yml
+++ b/PROJECTS/intermediate/chaos-engineering-tool/.golangci.yml
@@ -1,4 +1,4 @@
-# AngelaMos | 2026
+# ©AngelaMos | 2026
# .golangci.yml
version: "2"
@@ -27,23 +27,6 @@ linters:
exclusions:
rules:
- # errcheck is excluded in tests, and only in tests, because a test's
- # error-handling policy IS its assertion: a discarded return that mattered
- # shows up as a failing test, not as a silent wrong answer in the field.
- # Every one of the 13 test sites that discards a meaningful return was read
- # by hand first; each asserts on the value it keeps, and several discard an
- # error the measurement requires (a complete answer through the fault proxy
- # always ends in one). The exclusion is not standing permission to stop
- # reading them.
- #
- # gocognit joins funlen and dupl for the same reason those two are already
- # here. Cognitive complexity counts branches, and the eight tests it flags
- # are the CENSUS tests: one function that walks Kinds() and asserts every
- # property khaos promises for each. Their score is assertion density, not
- # tangle -- each `if wrong { t.Fatalf }` is independent and reads on its
- # own -- and density is the property that makes a census worth having. The
- # limit stays ON for source, where a high score means nested branches that
- # actually interact.
- path: _test\.go
linters:
- funlen
@@ -59,53 +42,13 @@ linters:
settings:
errcheck:
check-type-assertions: true
-
- # check-blank is off, and this is a decision rather than a default. It
- # treats `_ = f()` as unchecked, which means it distrusts the exact thing
- # this codebase uses to SAY a discard is deliberate. All 191 hits it
- # produced were read: 103 in source and 88 in tests, and not one was a
- # bug. Several were load-bearing in the other direction -- the journal
- # append on an already-failing revert must stay best-effort so the fault
- # stays outstanding and reclaim retries it, and the proxy's settle() read
- # exists precisely to throw its result away.
- #
- # Satisfying it would mean 58 permanent //nolint:errcheck directives
- # acknowledging a second time what `_ =` already acknowledges once, in a
- # comment the compiler never checks, suppressing the whole line rather
- # than one call. This file already carries about twenty nolint directives
- # that each record a real decision; burying those among 58 ceremonial ones
- # is how a reader learns to skim all of them.
- #
- # check-type-assertions stays ON: `x := y.(T)` has no `_ =` form that
- # states intent, and getting it wrong is a panic rather than a dropped
- # error. errcheck also still catches the genuine mistake this setting is
- # confused with -- a bare `f()` whose error is not assigned anywhere at
- # all, which is verified by a negative control, not assumed.
check-blank: false
-
- # Writes to khaos's own output stream are not error-checked, declared once
- # here rather than acknowledged at 46 call sites. A CLI that cannot print
- # has already lost the operator it would report the failure to, and the
- # exit status carries the verdict regardless. Close is deliberately NOT in
- # this list: closing a file khaos WROTE is what flushes it, so the journal
- # and its compaction temp file check Close and must keep checking it.
exclude-functions:
- fmt.Fprint
- fmt.Fprintf
- fmt.Fprintln
exhaustive:
- # A switch that deliberately handles a subset and sends the rest to a
- # default is not a bug here, and spelling the rest out as no-op cases that
- # fall through to the same body would add about a hundred lines that say
- # nothing. The constants this would guard are already guarded harder: every
- # Kind dispatch is covered by a census test that walks Kinds() and fails
- # the moment a new constant exists without a case of its own, and by a
- # differential that fails when a trap behaves like its own control. That is
- # exhaustiveness in BEHAVIOUR; the linter only checks that a constant is
- # named. What this setting leaves switched on is the case the linter is
- # genuinely better at than a test -- a switch with NO default that misses a
- # constant and falls out to a zero value.
default-signifies-exhaustive: true
funlen:
@@ -119,82 +62,14 @@ linters:
enable-all: true
disable:
- fieldalignment
-
- # shadow is not in go vet's default suite, and this is why: all 32 hits
- # here are `if err := f(); err != nil`, the idiom every Go reader parses
- # without thinking, where the scoped err is checked on the next line and
- # then gone. Both non-test hits were read in full; in one the outer err
- # is already checked and dead, and the other is the bare idiom. Renaming
- # 32 of them to applyErr and listenErr would make correct code read
- # worse to silence a check with a 32-of-32 false positive rate on this
- # tree. The bug shadow is meant to catch -- assigning the inner err and
- # then testing the outer one -- needs the outer err to be live after the
- # block, which none of these have.
- shadow
- revive:
- rules:
- - name: blank-imports
- - name: context-as-argument
- - name: context-keys-type
- - name: error-return
- - name: error-strings
- - name: error-naming
- - name: exported
- - name: increment-decrement
- - name: var-declaration
- - name: package-comments
- disabled: true
- - name: range
- - name: receiver-naming
- - name: time-naming
- - name: unexported-return
- - name: indent-error-flow
- - name: errorf
- - name: empty-block
- - name: superfluous-else
- - name: unreachable-code
-
- staticcheck:
- checks:
- - all
-
gosec:
- # G104 unhandled errors: errcheck owns that question, with a stricter
- # setting than gosec applies.
- #
- # G101 is a NAME heuristic. It fires on an identifier containing token,
- # credential or secret, and this tool's entire vocabulary is those three
- # words. All nine hits were read: eight are message, summary or format
- # strings, and the ninth is DefaultInvalidToken, which is REQUIRED to be a
- # known-bad constant because it is khaos's negative control.
- #
- # G704 flags a request whose URL came from outside the function, which is
- # every request khaos makes, by construction. Probing a URL the operator
- # names IS the product, and there is no privilege boundary to cross: khaos
- # runs as the operator, against a target the operator chose.
- #
- # G304 and G404 are deliberately NOT excluded. A future unsafe path open,
- # or a future security-relevant use of math/rand in a tool that mints its
- # own tokens, has to stay catchable, so those are acknowledged one site at
- # a time instead of waived here.
excludes:
- G104
- G101
- G704
- sloglint:
- no-mixed-args: true
- kv-only: true
- context: all
-
-# Both caps are off because the defaults silently truncate the report. golangci
-# stops at 50 issues per linter and 3 per repeated message, so this project read
-# 152 issues while it actually had 311 -- errcheck alone was pinned at exactly
-# 50 of its 191. A gate that hides more than half of what it found cannot be
-# used to decide whether the tree is clean, and a cap that bites is invisible in
-# the output. These were previously set under linters.exclusions, which is not a
-# section golangci reads them from, so neither one did anything.
issues:
max-issues-per-linter: 0
max-same-issues: 0
@@ -208,9 +83,6 @@ formatters:
golines:
max-len: 80
reformat-tags: true
- goimports:
- local-prefixes:
- - github.com/carterperez-dev/khaos
gci:
sections:
- standard
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/.goreleaser.yaml b/PROJECTS/intermediate/chaos-engineering-tool/.goreleaser.yaml
new file mode 100644
index 00000000..af877535
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/.goreleaser.yaml
@@ -0,0 +1,41 @@
+# ©AngelaMos | 2026
+# .goreleaser.yaml
+
+version: 2
+
+project_name: khaos
+
+snapshot:
+ version_template: "0.0.0-dev-{{ .ShortCommit }}"
+
+builds:
+ - id: khaos
+ main: ./cmd/khaos
+ binary: khaos
+ env:
+ - CGO_ENABLED=0
+ goos: [linux]
+ goarch: [amd64, arm64]
+ flags:
+ - -trimpath
+ ldflags:
+ - -s -w -X github.com/CarterPerez-dev/khaos/internal/version.ldflagsVersion={{ .Version }}
+
+archives:
+ - id: khaos
+ name_template: "{{ .ProjectName }}_{{ .Os }}_{{ .Arch }}"
+ formats: [tar.gz]
+ files:
+ - README.md
+ - LICENSE
+
+checksum:
+ name_template: checksums.txt
+
+changelog:
+ sort: asc
+
+release:
+ github:
+ owner: CarterPerez-dev
+ name: khaos
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/README.md b/PROJECTS/intermediate/chaos-engineering-tool/README.md
new file mode 100644
index 00000000..726870a9
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/README.md
@@ -0,0 +1,175 @@
+<!-- ©AngelaMos | 2026 -->
+<!-- README.md -->
+
+```text
+ _ ___ _ _ ___ ___
+ | |/ / | | | /_\ / _ \/ __|
+ | ' <| |_| | / _ \| (_) \__ \
+ |_|\_\\___/ /_/ \_\\___/|___/
+```
+
+[![Cybersecurity Projects](https://img.shields.io/badge/Cybersecurity--Projects-Intermediate-red?style=flat&logo=github)](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/intermediate/chaos-engineering-tool)
+[![Go](https://img.shields.io/badge/Go-1.26+-00ADD8?style=flat&logo=go&logoColor=white)](https://go.dev)
+[![Linux](https://img.shields.io/badge/Linux-amd64%20%7C%20arm64-333333?style=flat&logo=linux&logoColor=white)](https://github.com/CarterPerez-dev/khaos/releases)
+[![License: AGPLv3](https://img.shields.io/badge/License-AGPL_v3-purple.svg)](LICENSE)
+
+> Inject security failures into a Docker stack and measure whether its controls still enforce their decisions. Khaos tests legitimate and invalid credentials before, during, and after each fault, then records what changed and whether recovery succeeded.
+
+Security concepts, architecture, and walkthroughs are in the [learn modules](#learn).
+
+## What It Does
+
+- 90 experiments across network, identity, TLS, DNS, credential, and security-control failures
+- A bundled Docker range with correct and deliberately defective gateways, plus an optional PostgreSQL, MySQL, MariaDB, and Redis tier
+- Terminal reports, canonical JSON records, JUnit XML, and Prometheus textfile metrics
+- Saved run history, YAML experiment files, replay with target identity checks, and an optional health stop
+- Consent labels, mutation journals, renewable leases, bounded rollback, and recovery after an interrupted run
+- An interactive console with completion, alongside ordinary commands for scripts and CI
+
+| Family | Examples |
+|--------|----------|
+| `network` | Dependency outages, resets, latency, incomplete responses, and stalled connections |
+| `identity` | Revoked tokens, key rotation, unavailable key sets, malformed introspection, and JWT verification defects |
+| `tls` | Expired certificates, trust-chain errors, revoked identities, weak keys, and session resumption |
+| `dns` | Failed resolution, forged answers, truncated responses, and changes hidden by connection reuse |
+| `credential` | Database password rotation, account locking, session termination, missing secrets, and authorization denials |
+| `control` | Unavailable rate-limit counters, policy stores, and rule backends |
+
+## Install
+
+```bash
+curl -fsSL https://angelamos.com/khaos/install.sh | bash
+```
+
+The installer verifies the release checksum, installs `khaos` onto PATH, and sets up Docker Engine and Compose when needed. If a release is unavailable, it builds a static binary from source and downloads a verified Go toolchain when necessary. Run it again to update.
+
+Linux AMD64 and ARM64 are supported. The bundled range requires a local Docker Engine with host access to its bridge network. Use a Linux VM on macOS or Windows. Docker setup may require sudo.
+
+To install only the executable:
+
+```bash
+curl -fsSL https://angelamos.com/khaos/install.sh | bash -s -- --no-docker
+```
+
+Or use Go 1.26+:
+
+```bash
+CGO_ENABLED=0 go install github.com/CarterPerez-dev/khaos/cmd/khaos@latest
+```
+
+Go installs into `GOBIN`, or `$(go env GOPATH)/bin` when unset. That directory must be on PATH. `CGO_ENABLED=0` is required because Khaos uses its own executable inside resolver and secret-file helper containers. Range build files are embedded, so a source checkout is unnecessary.
+
+## Quick Start
+
+Start the range with its authentication gateway configured to refuse requests when the identity provider is unavailable:
+
+```bash
+khaos range up --fail-mode closed
+khaos doctor
+khaos plan network.dependency_outage
+khaos run network.dependency_outage
+```
+
+Now run the same experiment against a gateway that permits requests when that dependency fails:
+
+```bash
+khaos range up --fail-mode open
+khaos run network.dependency_outage
+khaos range down
+```
+
+The second run should report `PERMITTED` and exit with code `2`: the gateway accepted an invalid credential during the fault. A closed gateway may also deny legitimate traffic during an outage; that availability deviation has its own exit code.
+
+Run `khaos` without arguments to open the interactive console. Use `khaos catalog` to list experiments and `khaos plan <experiment>` to inspect one before applying it.
+
+> [!TIP]
+> This project uses [`just`](https://github.com/casey/just) as a command runner. Type `just` in a checkout to see its recipes.
+>
+> Install: `curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin`
+
+## Commands
+
+| Command | Purpose |
+|---------|---------|
+| `khaos doctor` | Check Docker access, consenting targets, and helper-binary compatibility |
+| `khaos targets` | List eligible Docker targets |
+| `khaos catalog` | List the experiment catalog |
+| `khaos plan <experiment>` | Describe an experiment without injecting its fault |
+| `khaos run <experiment>` | Measure baseline, fault, and recovery |
+| `khaos run-file experiment.yaml` | Run a saved experiment configuration |
+| `khaos runs` | List saved runs |
+| `khaos show <run-id>` | Read a run and verify its event hashes |
+| `khaos replay path/to/run.json` | Repeat a recorded configuration after checking target identity |
+| `khaos reclaim` | Retry outstanding recovery obligations |
+| `khaos reap` | Watch leases and recover abandoned mutations |
+| `khaos range up --with-db` | Start the range with its database tier |
+| `khaos range status` | Show range endpoints and status |
+| `khaos range down` | Remove the disposable range and its volumes |
+
+Each command has `--help`. The range automatically supplies targets, addresses, and test credentials. For your own stack, add `khaos.target: "true"` to consenting services and provide the parameters shown by the experiment's plan.
+
+## Reports and CI
+
+```bash
+khaos run network.dependency_outage \
+ --record ./results/outage \
+ --output json \
+ --junit ./results/outage.xml \
+ --metrics-out ./results/outage.prom
+```
+
+`run.json` is the canonical result. `events.jsonl` contains the individual observations and is referenced by its SHA-256 hash. Terminal output, JUnit, and metrics derive from that record. Saved configurations use environment-variable references for supplied secrets.
+
+| Exit code | Meaning |
+|-----------|---------|
+| `0` | Accepted outcome under the selected failure policy |
+| `1` | Usage, configuration, or artifact-publication error |
+| `2` | Measured security defect |
+| `3` | Measured deviation outside the experiment's defect direction |
+| `4` | Inconclusive evidence |
+| `5` | Probe failure |
+| `6` | Rollback failed |
+| `7` | Experiment validity checks failed |
+| `8` | Baseline requirements were not met |
+| `9` | Experiment unsupported for this target |
+| `10` | Interrupted |
+| `70` | Internal error |
+
+The default preserves nonzero outcomes. `--fail-on` selects which measured outcomes fail a CI job; usage, rollback, and internal errors cannot be suppressed. `--abort-after N` stops after consecutive unhealthy primary probe pairs and reverts the fault while preserving any bypass already measured.
+
+Run history and recovery state live in `${XDG_STATE_HOME:-~/.local/state}/khaos`. Set `KHAOS_STATE_DIR` to override that location. Bundled range sources are cached under `${XDG_CACHE_HOME:-~/.cache}/khaos/range`, with their contents verified before reuse.
+
+## Scope and Recovery
+
+Use Khaos in disposable development, test, and CI environments. Docker access is powerful enough to affect the host; consent checks and recovery controls bound the operations Khaos performs.
+
+Mutations require the exact label `khaos.target=true`. Khaos refuses unsafe target configurations and unresolved recovery obligations. Keep `khaos reap` running when automatic recovery after a killed process is required, and use `khaos reclaim` to inspect and retry abandoned work.
+
+Network-disconnect recovery verifies endpoint settings and addresses. Docker 28 may allocate a new MAC when an endpoint is recreated, which the fault description discloses before injection. A target requiring an unchanged MAC should use another fault mechanism.
+
+## Build and Test
+
+```bash
+just build
+just check
+just race
+just gate
+```
+
+`just build` produces a static binary. `just check` runs the configured linters, module checks, unit tests, and terminal-color checks. `just gate` also runs the Docker integration suites sequentially; they create and remove the disposable range. Additional ownership and restart tests require `KHAOS_DOCKER_TESTS=1`.
+
+The standalone repository at [CarterPerez-dev/khaos](https://github.com/CarterPerez-dev/khaos) provides Go installation and release binaries. The project also lives in the [Cybersecurity Projects collection](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/intermediate/chaos-engineering-tool).
+
+## Learn
+
+| Module | Topic |
+|--------|-------|
+| [00 - Overview](learn/00-OVERVIEW.md) | Prerequisites, installation, and a first experiment |
+| [01 - Concepts](learn/01-CONCEPTS.md) | Security chaos engineering, failure direction, and evidence |
+| [02 - Architecture](learn/02-ARCHITECTURE.md) | Injection tiers, probes, journaling, and recovery |
+| [03 - Implementation](learn/03-IMPLEMENTATION.md) | Walkthrough from experiment selection to a verified report |
+| [04 - Challenges](learn/04-CHALLENGES.md) | Extension ideas and exercises |
+
+## License
+
+[AGPL 3.0](LICENSE).
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/bundle.go b/PROJECTS/intermediate/chaos-engineering-tool/bundle.go
new file mode 100644
index 00000000..df4ddd89
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/bundle.go
@@ -0,0 +1,144 @@
+// ©AngelaMos | 2026
+// bundle.go
+
+package khaos
+
+import (
+ "crypto/sha256"
+ "embed"
+ "errors"
+ "fmt"
+ "io"
+ "io/fs"
+ "os"
+ "path/filepath"
+)
+
+//go:embed go.mod go.sum bundle.go cmd internal range
+var rangeSource embed.FS
+
+func RangeDirectory() (string, error) {
+ cache, err := os.UserCacheDir()
+ if err != nil {
+ return "", fmt.Errorf("locate the range cache: %w", err)
+ }
+
+ return cacheRange(filepath.Join(cache, "khaos", "range"))
+}
+
+func cacheRange(parent string) (string, error) {
+ digest, err := sourceDigest(rangeSource)
+ if err != nil {
+ return "", fmt.Errorf("read the bundled range: %w", err)
+ }
+ destination := filepath.Join(parent, digest)
+ if _, err := os.Lstat(destination); err == nil {
+ return verifiedRange(destination, digest)
+ } else if !errors.Is(err, fs.ErrNotExist) {
+ return "", err
+ }
+ if err := os.MkdirAll(parent, 0o700); err != nil {
+ return "", err
+ }
+ staging, err := os.MkdirTemp(parent, ".unpack-")
+ if err != nil {
+ return "", err
+ }
+ defer func() { _ = os.RemoveAll(staging) }()
+ if err := unpackRange(staging); err != nil {
+ return "", fmt.Errorf("unpack the bundled range: %w", err)
+ }
+ if err := os.Rename(staging, destination); err != nil {
+ if dir, verifyErr := verifiedRange(
+ destination,
+ digest,
+ ); verifyErr == nil {
+ return dir, nil
+ }
+ return "", fmt.Errorf("publish the range cache: %w", err)
+ }
+ return verifiedRange(destination, digest)
+}
+
+func verifiedRange(directory, want string) (string, error) {
+ info, err := os.Lstat(directory)
+ if err != nil {
+ return "", err
+ }
+ if !info.IsDir() {
+ return "", fmt.Errorf("range cache %s is not a directory", directory)
+ }
+ root, err := os.OpenRoot(directory)
+ if err != nil {
+ return "", err
+ }
+ defer func() { _ = root.Close() }()
+ digest, err := sourceDigest(root.FS())
+ if err != nil {
+ return "", fmt.Errorf("verify the range cache %s: %w", directory, err)
+ }
+ if digest != want {
+ return "", fmt.Errorf(
+ "range cache %s was modified; move it aside and retry", directory,
+ )
+ }
+ return filepath.Join(directory, "range"), nil
+}
+
+func sourceDigest(source fs.FS) (string, error) {
+ digest := sha256.New()
+ err := fs.WalkDir(source, ".", func(
+ path string, entry fs.DirEntry, walkErr error,
+ ) error {
+ if walkErr != nil {
+ return walkErr
+ }
+ if entry.IsDir() {
+ return nil
+ }
+ if !entry.Type().IsRegular() {
+ return fmt.Errorf("unexpected non-regular range file: %s", path)
+ }
+ file, err := source.Open(path)
+ if err != nil {
+ return err
+ }
+ defer func() { _ = file.Close() }()
+ fmt.Fprintf(digest, "%s\x00", path)
+ if _, err := io.Copy(digest, file); err != nil {
+ return err
+ }
+ fmt.Fprint(digest, "\x00")
+ return nil
+ })
+ if err != nil {
+ return "", err
+ }
+ return fmt.Sprintf("%x", digest.Sum(nil)), nil
+}
+
+func unpackRange(directory string) error {
+ root, err := os.OpenRoot(directory)
+ if err != nil {
+ return err
+ }
+ defer func() { _ = root.Close() }()
+ return fs.WalkDir(rangeSource, ".", func(
+ path string, entry fs.DirEntry, walkErr error,
+ ) error {
+ if walkErr != nil {
+ return walkErr
+ }
+ if path == "." {
+ return nil
+ }
+ if entry.IsDir() {
+ return root.Mkdir(path, 0o700)
+ }
+ contents, err := rangeSource.ReadFile(path)
+ if err != nil {
+ return err
+ }
+ return root.WriteFile(path, contents, 0o600)
+ })
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/bundle_test.go b/PROJECTS/intermediate/chaos-engineering-tool/bundle_test.go
new file mode 100644
index 00000000..39efaf3a
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/bundle_test.go
@@ -0,0 +1,112 @@
+// ©AngelaMos | 2026
+// bundle_test.go
+
+package khaos
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "sync"
+ "testing"
+)
+
+func TestBundledRangeWorksWithoutACheckout(t *testing.T) {
+ parent := t.TempDir()
+ directory, err := cacheRange(parent)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, name := range []string{
+ "range/compose.yml", "range/Dockerfile", "go.mod", "go.sum",
+ "bundle.go", "cmd/khaos-range/main.go", "internal/dnsrv/resolver.go",
+ } {
+ if _, err := os.Stat(
+ filepath.Join(filepath.Dir(directory), name),
+ ); err != nil {
+ t.Fatalf("missing runtime source %s: %v", name, err)
+ }
+ }
+ again, err := cacheRange(parent)
+ if err != nil || again != directory {
+ t.Fatalf("reusing range: %q, %v", again, err)
+ }
+ for _, name := range []string{"docs", "learn", ".git", "scripts"} {
+ if _, err := os.Stat(
+ filepath.Join(filepath.Dir(directory), name),
+ ); !os.IsNotExist(
+ err,
+ ) {
+ t.Fatalf("unexpected development content %s: %v", name, err)
+ }
+ }
+}
+
+func TestModifiedRangeCacheIsRefused(t *testing.T) {
+ for _, change := range []string{"modified file", "extra file", "symlink"} {
+ t.Run(change, func(t *testing.T) {
+ parent := t.TempDir()
+ directory, err := cacheRange(parent)
+ if err != nil {
+ t.Fatal(err)
+ }
+ path := filepath.Join(directory, "compose.yml")
+ switch change {
+ case "modified file":
+ err = os.WriteFile(path, []byte("name: somebody-else\n"), 0o600)
+ case "extra file":
+ err = os.WriteFile(
+ filepath.Join(directory, "extra"),
+ nil,
+ 0o600,
+ )
+ case "symlink":
+ if err = os.Remove(path); err == nil {
+ err = os.Symlink(
+ filepath.Join(t.TempDir(), "foreign"),
+ path,
+ )
+ }
+ }
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := cacheRange(parent); err == nil {
+ t.Fatal("modified range source was accepted")
+ }
+ })
+ }
+}
+
+func TestConcurrentRangeExtractionIsAtomic(t *testing.T) {
+ const count = 6
+ parent := t.TempDir()
+ directories := make([]string, count)
+ errors := make([]error, count)
+ var workers sync.WaitGroup
+ for index := range count {
+ workers.Go(func() {
+ directories[index], errors[index] = cacheRange(parent)
+ })
+ }
+ workers.Wait()
+ for index := range count {
+ if errors[index] != nil || directories[index] != directories[0] {
+ t.Fatalf(
+ "worker %d: %q, %v",
+ index,
+ directories[index],
+ errors[index],
+ )
+ }
+ }
+ entries, err := os.ReadDir(parent)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, entry := range entries {
+ if strings.HasPrefix(entry.Name(), ".unpack-") {
+ t.Fatalf("unfinished extraction survived: %s", entry.Name())
+ }
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/authzgateway.go b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/authzgateway.go
index edcc0c92..d9f0030e 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/authzgateway.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/authzgateway.go
@@ -17,21 +17,8 @@ const (
// fallback policy.
AuthzDistinguishes = "distinguishes"
- // AuthzCollapses is research 07 section 6.2's exact shape:
- //
- // if err != nil || resp.StatusCode != 200 { return Allow }
- //
- // The dependency said no, loudly and correctly, and the caller read it as
- // "no answer". It is the single most common fail-open in service-to-service
- // code because error handling is usually written as "did it work" rather
- // than "what did it say".
AuthzCollapses = "collapses"
- // AuthzRetries reaches the right verdict the wrong way: generic retry
- // middleware that retries any non-2xx, so a definitive refusal is asked
- // again and again. It fails CLOSED, which is why it is a third mode rather
- // than a variant of the second, and it is the shape research 07 section 6.2
- // step 4 names as the retry storm.
AuthzRetries = "retries"
// AuthzCached is also CORRECT, and it exists because khaos's own verdict
@@ -77,16 +64,6 @@ const (
authzBackoff = 50 * time.Millisecond
)
-// authzGateway is the CALLER, and the only range fixture whose dependency is
-// not a container: khaos serves that itself, because no correct authorization
-// service answers a definitive deny on request.
-//
-// AUTHENTICATION is this service's own job and it is correct in every mode.
-// That separation is what makes the experiment measurable: a caller that allows
-// on any non-200 from its authorization dependency has no authorization check
-// at all, so khaos's baseline guard would correctly void a run against one that
-// did nothing else. Research 07's row is about a service whose other checks
-// work and whose authorization check collapses.
type authzGateway struct {
dependency string
mode string
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/authzgateway_test.go b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/authzgateway_test.go
index 5317153a..8d8a3eaa 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/authzgateway_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/authzgateway_test.go
@@ -129,8 +129,6 @@ func TestEveryModeAuthenticatesLocally(t *testing.T) {
}
}
-// The three callers, which are three cells of research 07 section 6.2's
-// matrix. The deny is the row that separates them.
func TestTheThreeCallersReadADenyDifferently(t *testing.T) {
for _, row := range []struct {
mode string
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/dbgateway.go b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/dbgateway.go
index 0614b089..810fa89f 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/dbgateway.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/dbgateway.go
@@ -37,9 +37,6 @@ const (
// attempt, so the pool redials and the stale secret surfaces.
PoolBounded = "bounded"
- // PoolHeld is the sharpest shape and the one research 07 section 5.3 did
- // not have: a connection CHECKED OUT and never returned is age-unbounded,
- // because the cleaner only ever walks the idle list.
PoolHeld = "held"
boundedLifetime = 2 * time.Second
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/secretgateway.go b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/secretgateway.go
index 7b591d65..18db3b3d 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/secretgateway.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/secretgateway.go
@@ -14,10 +14,6 @@ import (
)
const (
- // SecretValidated is the correct handling research 07 part 8 prescribes:
- // explicit length validation after every read. An empty secret is not "no
- // secret", it is "a secret equal to the empty string", and HMAC will
- // happily compute with it.
SecretValidated = rangelab.SecretValidated
// SecretUnchecked is the CVE-2019-20933 shape: the secret is read, no
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/tlsgateway.go b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/tlsgateway.go
index 64cd90aa..7bba344c 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/tlsgateway.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/tlsgateway.go
@@ -93,13 +93,17 @@ func dialAddress(raw string) string {
// client is built per request, because a trust store or certificate held from
// boot could never experience a fault armed after startup.
func (g *tlsGateway) client(ctx context.Context) *tlsclient.Client {
+ var sessions tls.ClientSessionCache
+ if g.mode == tlsclient.PolicyInPeerCallback {
+ sessions = g.sessions
+ }
return tlsclient.New(tlsclient.Config{
Mode: g.mode,
Host: g.certHost,
Roots: g.trustStore(ctx),
Timeout: dependencyTimeout,
Identity: g.identity(ctx),
- Sessions: g.sessions,
+ Sessions: sessions,
})
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/tlsgateway_test.go b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/tlsgateway_test.go
index 1d4e68f0..9f3acda6 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/tlsgateway_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos-range/tlsgateway_test.go
@@ -25,6 +25,12 @@ func TestEveryRoleBuildsItsRoutesWithoutPanicking(t *testing.T) {
roleGateway,
roleVerifier,
roleTLS,
+ roleRogue,
+ roleDB,
+ roleSecret,
+ roleAuthz,
+ roleControl,
+ roleControld,
} {
t.Run(role, func(t *testing.T) {
handler, err := roleHandler(role)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos/main.go b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos/main.go
index 9f2f4ace..fca14e26 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos/main.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/cmd/khaos/main.go
@@ -41,7 +41,7 @@ const (
)
var internalFailure = errors.New(
- "khaos hit a bug and stopped; every fault it had applied was reverted on the way out",
+ "khaos hit a bug and stopped; run `khaos reclaim` to verify recovery and retry any unfinished rollback",
)
var errInterrupted = errors.New(
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/go.mod b/PROJECTS/intermediate/chaos-engineering-tool/go.mod
index edcf9023..54cc0138 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/go.mod
+++ b/PROJECTS/intermediate/chaos-engineering-tool/go.mod
@@ -17,6 +17,7 @@ require (
golang.org/x/crypto v0.57.0
golang.org/x/sys v0.48.0
golang.org/x/term v0.46.0
+ gopkg.in/yaml.v3 v3.0.1
)
require (
@@ -56,6 +57,5 @@ require (
go.opentelemetry.io/otel/trace v1.35.0 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/sync v0.22.0 // indirect
- gopkg.in/yaml.v3 v3.0.1 // indirect
mvdan.cc/sh/v3 v3.13.1 // indirect
)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/install.sh b/PROJECTS/intermediate/chaos-engineering-tool/install.sh
new file mode 100755
index 00000000..b88c1c68
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/install.sh
@@ -0,0 +1,508 @@
+#!/usr/bin/env bash
+# ©AngelaMos | 2026
+# install.sh
+
+set -euo pipefail
+
+BINARY=khaos
+REPO_URL="${KHAOS_REPO_URL:-https://github.com/CarterPerez-dev/khaos.git}"
+RELEASE_URL="${KHAOS_RELEASE_URL:-https://github.com/CarterPerez-dev/khaos/releases}"
+INSTALL_DIR="${KHAOS_INSTALL_DIR:-${HOME:?HOME must be set}/.local/bin}"
+CACHE_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/khaos/install"
+VERSION="${KHAOS_VERSION:-latest}"
+SOURCE_REF="${KHAOS_REF:-main}"
+SOURCE_DIR="${KHAOS_SOURCE_DIR:-}"
+GO_VERSION="${KHAOS_GO_VERSION:-}"
+ORIGINAL_PATH="$PATH"
+BUILD_SOURCE=0
+SETUP_DOCKER=1
+TMP_DIR=""
+LINK_DIR=""
+OS=""
+ARCH=""
+GO_BIN=""
+ROOT=()
+
+if [ -t 2 ] && [ -z "${NO_COLOR:-}" ]; then
+ BOLD=$'\033[1m'
+ CYAN=$'\033[36m'
+ GREEN=$'\033[32m'
+ YELLOW=$'\033[33m'
+ RED=$'\033[31m'
+ RESET=$'\033[0m'
+else
+ BOLD="" CYAN="" GREEN="" YELLOW="" RED="" RESET=""
+fi
+
+info() { printf '%s\n' " ${CYAN}+${RESET} $*" >&2; }
+ok() { printf '%s\n' " ${GREEN}+${RESET} $*" >&2; }
+warn() { printf '%s\n' " ${YELLOW}!${RESET} $*" >&2; }
+die() { printf '%s\n' " ${RED}x${RESET} $*" >&2; exit 1; }
+have() { command -v "$1" >/dev/null 2>&1; }
+
+cleanup() {
+ if [ -n "$TMP_DIR" ] && [ -d "$TMP_DIR" ]; then
+ rm -rf -- "$TMP_DIR"
+ fi
+ return 0
+}
+
+trap 'printf "%s\n" "${RED}Installation failed at line ${LINENO}.${RESET}" >&2' ERR
+trap cleanup EXIT
+
+usage() {
+ cat >&2 <<USAGE
+khaos installer
+
+ curl -fsSL https://angelamos.com/khaos/install.sh | bash
+ bash install.sh [options]
+
+Options:
+ --prefix DIR Binary directory (default: ~/.local/bin)
+ --version TAG Install a particular release, such as v0.1.0
+ --source Build from source instead of downloading a binary
+ --no-docker Install only Khaos; leave Docker setup to the operator
+ -h, --help Show this help
+
+Supports Linux AMD64 and ARM64 with native Docker Engine networking.
+Installs Docker and Compose when absent. Administrative setup may use sudo.
+Existing Docker endpoints and running services are preserved.
+
+Environment: KHAOS_INSTALL_DIR, KHAOS_VERSION, KHAOS_SOURCE_DIR,
+KHAOS_REF, KHAOS_REPO_URL, KHAOS_RELEASE_URL, KHAOS_GO_VERSION,
+XDG_CACHE_HOME, NO_COLOR.
+USAGE
+}
+
+parse_args() {
+ while [ "$#" -gt 0 ]; do
+ case "$1" in
+ --prefix|--version)
+ if [ "$#" -lt 2 ] || [ -z "$2" ]; then die "$1 needs a value"; fi
+ case "$2" in --*) die "$1 needs a value" ;; esac
+ if [ "$1" = --prefix ]; then INSTALL_DIR="$2"; else VERSION="$2"; fi
+ shift 2
+ ;;
+ --prefix=*) INSTALL_DIR="${1#*=}"; shift ;;
+ --version=*) VERSION="${1#*=}"; shift ;;
+ --source) BUILD_SOURCE=1; shift ;;
+ --no-docker) SETUP_DOCKER=0; shift ;;
+ -h|--help) usage; exit 0 ;;
+ *) die "Unknown option: $1 (try --help)" ;;
+ esac
+ done
+ case "$INSTALL_DIR" in
+ /*) ;;
+ *) die "--prefix must be an absolute directory" ;;
+ esac
+ case "$INSTALL_DIR" in
+ *:*|*$'\n'*|*$'\r'*) die "The install directory cannot contain colons or newlines" ;;
+ esac
+ [[ "$VERSION" == latest || "$VERSION" =~ ^v[0-9]+\.[0-9]+\.[0-9]+([-+][A-Za-z0-9.-]+)?$ ]] ||
+ die "--version must be latest or a release tag such as v0.1.0"
+}
+
+banner() {
+ printf '\n%s%s' "$BOLD" "$CYAN" >&2
+ cat >&2 <<'ART'
+ _ ___ _ _ ___ ___
+ | |/ / | | | /_\ / _ \/ __|
+ | ' <| |_| | / _ \| (_) \__ \
+ |_|\_\\___/ /_/ \_\\___/|___/
+ART
+ printf '%s\n Security chaos engineering\n\n' "$RESET" >&2
+}
+
+platform() {
+ case "$(uname -s)" in
+ Linux) OS=linux ;;
+ Darwin) die "Khaos requires Linux Docker networking. Run this installer inside a Linux VM." ;;
+ MINGW*|MSYS*|CYGWIN*) die "Run this installer inside a Linux VM or WSL2 with a native Docker Engine." ;;
+ *) die "Unsupported operating system: $(uname -s)" ;;
+ esac
+ case "$(uname -m)" in
+ x86_64|amd64) ARCH=amd64 ;;
+ aarch64|arm64) ARCH=arm64 ;;
+ *) die "Unsupported architecture: $(uname -m). Supported: AMD64, ARM64." ;;
+ esac
+}
+
+root_access() {
+ if [ "$(id -u)" -eq 0 ]; then
+ ROOT=()
+ elif have sudo; then
+ ROOT=(sudo)
+ sudo -v || die "Administrative access is required for this installation step"
+ else
+ die "This step needs root or sudo. A writable PATH directory and --no-docker avoid system setup."
+ fi
+}
+
+pkg_install() {
+ root_access
+ if have apt-get; then
+ "${ROOT[@]}" apt-get update -q
+ "${ROOT[@]}" env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends "$@"
+ elif have dnf; then
+ "${ROOT[@]}" dnf install -y "$@"
+ elif have pacman; then
+ "${ROOT[@]}" pacman -S --needed --noconfirm "$@"
+ elif have zypper; then
+ "${ROOT[@]}" zypper --non-interactive install "$@"
+ elif have apk; then
+ "${ROOT[@]}" apk add --no-cache "$@"
+ elif have brew; then
+ brew install "$@"
+ else
+ die "No supported package manager found; required packages: $*"
+ fi
+}
+
+download() {
+ if have curl; then
+ curl --fail --silent --show-error --location --retry 3 \
+ --connect-timeout 15 --max-time 300 "$1" --output "$2"
+ elif have wget; then
+ wget --quiet --timeout=30 --tries=3 --output-document="$2" "$1"
+ else
+ die "curl or wget is required to download installation files"
+ fi
+}
+
+digest() {
+ if have sha256sum; then
+ sha256sum "$1" | awk '{print $1}'
+ elif have shasum; then
+ shasum -a 256 "$1" | awk '{print $1}'
+ else
+ die "No SHA-256 verifier found; install coreutils or perl-Digest-SHA"
+ fi
+}
+
+verify_checksum() {
+ local expected="$1" actual
+ [[ "$expected" =~ ^[a-fA-F0-9]{64}$ ]] || die "Missing or invalid SHA-256 checksum for $2"
+ actual="$(digest "$2")"
+ [ "${actual,,}" = "${expected,,}" ] || die "Checksum mismatch for $2; refusing this download"
+}
+
+prebuilt() {
+ local base asset expected
+ asset="${BINARY}_${OS}_${ARCH}.tar.gz"
+ if [ "$VERSION" = latest ]; then
+ base="$RELEASE_URL/latest/download"
+ else
+ base="$RELEASE_URL/download/$VERSION"
+ fi
+ info "Looking for a verified $OS/$ARCH release"
+ if ! download "$base/$asset" "$TMP_DIR/$asset"; then
+ warn "Release download unavailable; building from source"
+ return 1
+ fi
+ download "$base/checksums.txt" "$TMP_DIR/checksums.txt" ||
+ die "The release is missing checksums.txt; refusing an unchecked binary"
+ expected="$(awk -v file="$asset" '$2 == file || $2 == "*" file {print $1}' "$TMP_DIR/checksums.txt")"
+ verify_checksum "$expected" "$TMP_DIR/$asset"
+ tar -xzf "$TMP_DIR/$asset" -C "$TMP_DIR" --no-same-owner "$BINARY" ||
+ die "Could not extract the release executable"
+ if [ ! -f "$TMP_DIR/$BINARY" ] || [ -L "$TMP_DIR/$BINARY" ]; then
+ die "The release does not contain a regular $BINARY executable"
+ fi
+ chmod 0755 "$TMP_DIR/$BINARY"
+ "$TMP_DIR/$BINARY" version >&2 || die "The release executable failed its version check"
+ return 0
+}
+
+is_checkout() {
+ [ -f "$1/go.mod" ] && [ -f "$1/cmd/khaos/main.go" ] &&
+ grep -qx 'module github.com/CarterPerez-dev/khaos' "$1/go.mod"
+}
+
+resolve_source() {
+ local self cache origin
+ if [ -n "$SOURCE_DIR" ]; then
+ is_checkout "$SOURCE_DIR" || die "KHAOS_SOURCE_DIR is not a Khaos checkout: $SOURCE_DIR"
+ SOURCE_DIR="$(cd "$SOURCE_DIR" && pwd -P)"
+ return
+ fi
+ if [ "$VERSION" = latest ] && is_checkout "$PWD"; then
+ SOURCE_DIR="$PWD"
+ return
+ fi
+ self="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
+ if [ "$VERSION" = latest ] && is_checkout "$self"; then
+ SOURCE_DIR="$self"
+ return
+ fi
+ if ! have git; then pkg_install git ca-certificates; fi
+ have git || die "Git installation did not provide git"
+ if [ "$VERSION" != latest ]; then SOURCE_REF="$VERSION"; fi
+ [[ "$SOURCE_REF" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] || die "Invalid KHAOS_REF: $SOURCE_REF"
+ cache="$CACHE_DIR/source/$SOURCE_REF"
+ if [ -e "$cache" ]; then
+ is_checkout "$cache" || die "The source cache is not a Khaos checkout: $cache"
+ origin="$(git -C "$cache" remote get-url origin)"
+ [ "$origin" = "$REPO_URL" ] || die "The source cache has a different origin: $cache"
+ [ -z "$(git -C "$cache" status --porcelain)" ] || die "The source cache contains edits: $cache"
+ if [ "$VERSION" = latest ]; then
+ [ "$(git -C "$cache" branch --show-current)" = "$SOURCE_REF" ] ||
+ die "The source cache is on an unexpected branch: $cache"
+ git -C "$cache" pull --ff-only >&2
+ fi
+ else
+ mkdir -p "$(dirname "$cache")"
+ git clone --depth 1 --branch "$SOURCE_REF" "$REPO_URL" "$cache" >&2
+ fi
+ SOURCE_DIR="$cache"
+}
+
+version_at_least() {
+ [ "$(printf '%s\n' "$1" "$2" | sort -V | head -n 1)" = "$2" ]
+}
+
+toolchain() {
+ local current archive expected destination
+ if have go; then
+ current="$(GOTOOLCHAIN=local go env GOVERSION)"
+ if version_at_least "${current#go}" 1.26.0; then
+ GO_BIN="$(command -v go)"
+ return
+ fi
+ fi
+ if [ -z "$GO_VERSION" ]; then
+ download 'https://go.dev/VERSION?m=text' "$TMP_DIR/go-version"
+ IFS= read -r GO_VERSION < "$TMP_DIR/go-version"
+ fi
+ GO_VERSION="${GO_VERSION#go}"
+ [[ "$GO_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "Invalid Go version: $GO_VERSION"
+ version_at_least "$GO_VERSION" 1.26.0 || die "Khaos requires Go 1.26 or newer"
+ destination="$CACHE_DIR/toolchains/go$GO_VERSION-$OS-$ARCH"
+ GO_BIN="$destination/go/bin/go"
+ if [ -x "$GO_BIN" ]; then return; fi
+ archive="go${GO_VERSION}.${OS}-${ARCH}.tar.gz"
+ info "Installing Go $GO_VERSION into the private build cache"
+ download 'https://go.dev/dl/?mode=json&include=all' "$TMP_DIR/go-releases.json"
+ expected="$(awk -v archive="$archive" '
+ /"filename":/ { wanted = index($0, "\"" archive "\"") > 0 }
+ wanted && /"sha256":/ {gsub(/[",[:space:]]/, ""); sub(/^sha256:/, ""); print; exit}
+ ' "$TMP_DIR/go-releases.json")"
+ download "https://go.dev/dl/$archive" "$TMP_DIR/$archive"
+ verify_checksum "$expected" "$TMP_DIR/$archive"
+ mkdir -p "$TMP_DIR/toolchain" "$(dirname "$destination")"
+ tar -xzf "$TMP_DIR/$archive" -C "$TMP_DIR/toolchain" --no-same-owner
+ [ -x "$TMP_DIR/toolchain/go/bin/go" ] || die "Go download did not contain a working toolchain"
+ [ ! -e "$destination" ] || die "Incomplete Go cache already exists: $destination"
+ mv "$TMP_DIR/toolchain" "$destination"
+ "$GO_BIN" version >&2
+}
+
+build_source() {
+ resolve_source
+ toolchain
+ info "Building a static Khaos binary"
+ (
+ cd "$SOURCE_DIR"
+ unset GOROOT
+ CGO_ENABLED=0 GOTOOLCHAIN=auto "$GO_BIN" build -trimpath \
+ -ldflags='-s -w' -o "$TMP_DIR/$BINARY" ./cmd/khaos
+ )
+ if have strip; then strip "$TMP_DIR/$BINARY"; fi
+ "$TMP_DIR/$BINARY" version >&2
+}
+
+install_compose() {
+ if docker compose version >/dev/null 2>&1; then return; fi
+ local machine base name expected directory
+ info "Installing Docker Compose"
+ machine=x86_64
+ if [ "$ARCH" = arm64 ]; then machine=aarch64; fi
+ name="docker-compose-linux-$machine"
+ base=https://github.com/docker/compose/releases/download/v2.40.0
+ download "$base/$name" "$TMP_DIR/docker-compose"
+ download "$base/$name.sha256" "$TMP_DIR/compose.sha256"
+ expected="$(awk 'NR == 1 {print $1}' "$TMP_DIR/compose.sha256")"
+ verify_checksum "$expected" "$TMP_DIR/docker-compose"
+ directory="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins"
+ mkdir -p "$directory"
+ install -m 0755 "$TMP_DIR/docker-compose" "$directory/docker-compose"
+ docker compose version >/dev/null 2>&1 || die "Docker Compose installation failed"
+}
+
+install_docker() {
+ if have docker; then return; fi
+ info "Installing Docker Engine"
+ if have apt-get || have dnf; then
+ download https://get.docker.com "$TMP_DIR/docker-install.sh"
+ root_access
+ "${ROOT[@]}" sh "$TMP_DIR/docker-install.sh" >&2
+ elif have pacman; then
+ pkg_install docker docker-compose
+ elif have zypper; then
+ pkg_install docker docker-compose
+ elif have apk; then
+ pkg_install docker docker-cli-compose
+ else
+ die "Docker Engine needs apt, dnf, pacman, zypper, or apk on this Linux host"
+ fi
+ have docker || die "Docker installation did not provide the docker command"
+}
+
+start_docker() {
+ if "${ROOT[@]}" docker info >/dev/null 2>&1; then return; fi
+ info "Starting the local Docker daemon"
+ if have systemctl; then
+ "${ROOT[@]}" systemctl enable --now docker
+ elif have rc-service; then
+ "${ROOT[@]}" rc-update add docker default
+ "${ROOT[@]}" rc-service docker start
+ else
+ die "No supported service manager found for Docker; use a running Linux Docker Engine"
+ fi
+ local attempt
+ for attempt in {1..30}; do
+ if "${ROOT[@]}" docker info >/dev/null 2>&1; then return; fi
+ if [ "$attempt" -eq 30 ]; then break; fi
+ sleep 1
+ done
+ die "Docker did not become ready within 30 seconds"
+}
+
+docker_access() {
+ local endpoint user
+ if docker info >/dev/null 2>&1; then return; fi
+ endpoint="$(docker context inspect --format '{{.Endpoints.docker.Host}}' 2>/dev/null || true)"
+ if [ -n "${DOCKER_HOST:-}${DOCKER_CONTEXT:-}" ] ||
+ { [ -n "$endpoint" ] && [ "$endpoint" != unix:///var/run/docker.sock ]; }; then
+ die "The selected Docker endpoint is unavailable. Its configuration was preserved: ${DOCKER_HOST:-$endpoint}"
+ fi
+ root_access
+ start_docker
+ if [ "$(id -u)" -ne 0 ]; then
+ user="$(id -un)"
+ warn "Granting $user access to the local Docker socket, which permits administrative container operations"
+ if ! have setfacl; then pkg_install acl; fi
+ "${ROOT[@]}" usermod -aG docker "$user"
+ "${ROOT[@]}" setfacl -m "u:$user:rw" /var/run/docker.sock
+ fi
+ docker info >/dev/null 2>&1 || die "The current user still cannot reach Docker"
+}
+
+path_contains() {
+ case ":$ORIGINAL_PATH:" in *":$1:"*) return 0 ;; *) return 1 ;; esac
+}
+
+choose_link_dir() {
+ local directory
+ if path_contains "$INSTALL_DIR"; then LINK_DIR="$INSTALL_DIR"; return; fi
+ local -a candidates
+ IFS=: read -r -a candidates <<< "$ORIGINAL_PATH"
+ for directory in "${candidates[@]}"; do
+ case "$directory" in
+ /*)
+ if [ -d "$directory" ] && [ -w "$directory" ]; then
+ LINK_DIR="$directory"
+ return
+ fi
+ ;;
+ esac
+ done
+ for directory in /usr/local/bin /usr/bin; do
+ if path_contains "$directory"; then
+ root_access
+ LINK_DIR="$directory"
+ return
+ fi
+ done
+ die "No usable directory exists on PATH; choose --prefix pointing to a directory already on PATH"
+}
+
+write_path_config() {
+ if path_contains "$INSTALL_DIR"; then return; fi
+ local rc line shell_name quoted
+ shell_name="${SHELL:-/bin/sh}"
+ shell_name="${shell_name##*/}"
+ printf -v quoted '%q' "$INSTALL_DIR"
+ case "$shell_name" in
+ zsh) rc="${ZDOTDIR:-$HOME}/.zshrc"; line="export PATH=$quoted:\$PATH" ;;
+ bash) rc="$HOME/.bashrc"; line="export PATH=$quoted:\$PATH" ;;
+ fish)
+ rc="${XDG_CONFIG_HOME:-$HOME/.config}/fish/conf.d/khaos-path.fish"
+ line="fish_add_path -- $quoted"
+ ;;
+ *) rc="$HOME/.profile"; line="export PATH=$quoted:\$PATH" ;;
+ esac
+ mkdir -p "$(dirname "$rc")"
+ if ! grep -Fqx -- "$line" "$rc" 2>/dev/null; then
+ printf '\n%s\n' "$line" >> "$rc"
+ fi
+}
+
+install_binary() {
+ local staging
+ choose_link_dir
+ if [ ! -d "$INSTALL_DIR" ]; then
+ if ! mkdir -p "$INSTALL_DIR" 2>/dev/null; then
+ root_access
+ "${ROOT[@]}" mkdir -p "$INSTALL_DIR"
+ fi
+ fi
+ if [ -w "$INSTALL_DIR" ]; then
+ staging="$(mktemp "$INSTALL_DIR/.khaos-install.XXXXXX")"
+ install -m 0755 "$TMP_DIR/$BINARY" "$staging"
+ mv -f "$staging" "$INSTALL_DIR/$BINARY"
+ else
+ root_access
+ staging="$("${ROOT[@]}" mktemp "$INSTALL_DIR/.khaos-install.XXXXXX")"
+ "${ROOT[@]}" install -m 0755 "$TMP_DIR/$BINARY" "$staging"
+ "${ROOT[@]}" mv -f "$staging" "$INSTALL_DIR/$BINARY"
+ fi
+ if [ "$LINK_DIR" != "$INSTALL_DIR" ]; then
+ if [ -e "$LINK_DIR/$BINARY" ] && [ ! -L "$LINK_DIR/$BINARY" ]; then
+ die "A different executable already occupies $LINK_DIR/$BINARY; choose that directory with --prefix to update it"
+ fi
+ if [ -w "$LINK_DIR" ]; then
+ ln -sfn "$INSTALL_DIR/$BINARY" "$LINK_DIR/$BINARY"
+ else
+ root_access
+ "${ROOT[@]}" ln -sfn "$INSTALL_DIR/$BINARY" "$LINK_DIR/$BINARY"
+ fi
+ write_path_config
+ fi
+ hash -r
+ have "$BINARY" || die "The installed command is not discoverable on the original PATH"
+ [ "$(readlink -f "$(command -v "$BINARY")")" = "$(readlink -f "$INSTALL_DIR/$BINARY")" ] ||
+ die "Another khaos executable shadows the installation on PATH"
+ "$BINARY" version >&2
+ ok "Installed $BINARY in $INSTALL_DIR"
+}
+
+footer() {
+ printf '\n%s\n' ' Ready:' ' khaos range up' ' khaos catalog' ' khaos' >&2
+ printf '\n%s\n' ' Docs: https://github.com/CarterPerez-dev/khaos#readme' >&2
+ if have just && [ -n "$SOURCE_DIR" ]; then
+ printf '%s\n' " Checkout: $SOURCE_DIR" ' Recipes: just build, just check, just gate' >&2
+ fi
+ if [ "$SETUP_DOCKER" -eq 0 ]; then
+ warn "Docker setup was skipped by request; experiments need a reachable Linux Docker Engine"
+ fi
+}
+
+main() {
+ parse_args "$@"
+ platform
+ banner
+ TMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/khaos-install.XXXXXX")"
+ have tar || pkg_install tar
+ if ! have sha256sum && ! have shasum; then pkg_install coreutils; fi
+ if [ "$BUILD_SOURCE" -eq 1 ] || ! prebuilt; then build_source; fi
+ if [ "$SETUP_DOCKER" -eq 1 ]; then
+ install_docker
+ install_compose
+ docker_access
+ ok "Docker Engine and Compose are ready"
+ fi
+ install_binary
+ footer
+}
+
+main "$@" </dev/null
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/install_test.go b/PROJECTS/intermediate/chaos-engineering-tool/install_test.go
new file mode 100644
index 00000000..57d7bd61
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/install_test.go
@@ -0,0 +1,321 @@
+// ©AngelaMos | 2026
+// install_test.go
+
+package khaos
+
+import (
+ "archive/tar"
+ "compress/gzip"
+ "crypto/sha256"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+type installerFixture struct {
+ root string
+ bin string
+ prefix string
+ script string
+ env []string
+}
+
+func newInstallerFixture(t *testing.T) installerFixture {
+ t.Helper()
+ root := t.TempDir()
+ bin := filepath.Join(root, "bin")
+ home := filepath.Join(root, "home")
+ for _, path := range []string{bin, home} {
+ if err := os.Mkdir(path, 0o700); err != nil {
+ t.Fatal(err)
+ }
+ }
+ for _, name := range []string{
+ "awk", "bash", "cat", "chmod", "cp", "dirname", "grep", "head",
+ "id", "install", "ln", "mkdir", "mktemp", "mv", "readlink", "rm",
+ "sha256sum", "sort", "tar", "gzip",
+ } {
+ path, err := exec.LookPath(name)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink(path, filepath.Join(bin, name)); err != nil {
+ t.Fatal(err)
+ }
+ }
+ script, err := filepath.Abs("install.sh")
+ if err != nil {
+ t.Fatal(err)
+ }
+ fixture := installerFixture{
+ root: root, bin: bin, prefix: filepath.Join(home, "custom bin"),
+ script: script,
+ env: []string{
+ "PATH=" + bin, "HOME=" + home, "SHELL=/bin/bash", "NO_COLOR=1",
+ "XDG_CACHE_HOME=" + filepath.Join(root, "cache"),
+ "KHAOS_TEST_ROOT=" + root,
+ },
+ }
+ fixture.command(t, "uname", `case "$1" in
+-s) printf '%s\n' "${KHAOS_TEST_SYSTEM:-Linux}" ;;
+-m) printf '%s\n' "${KHAOS_TEST_ARCH:-x86_64}" ;;
+esac
+`)
+ fixture.command(t, "curl", `cat >/dev/null
+destination=
+url=
+while [ "$#" -gt 0 ]; do
+ case "$1" in
+ --output) destination="$2"; shift 2 ;;
+ http*) url="$1"; shift ;;
+ *) shift ;;
+ esac
+done
+case "$url" in
+ */checksums.txt) cp "$KHAOS_TEST_ROOT/checksums.txt" "$destination" ;;
+ */khaos_linux_amd64.tar.gz)
+ [ "${KHAOS_TEST_MISSING_RELEASE:-0}" = 0 ] || exit 22
+ cp "$KHAOS_TEST_ROOT/release.tar.gz" "$destination"
+ ;;
+ *) exit 22 ;;
+esac
+`)
+ fixture.command(
+ t,
+ "docker",
+ `printf '%s\n' "$*" >> "$KHAOS_TEST_ROOT/docker-calls"
+case "$*" in
+ "compose version") exit 0 ;;
+ "info") exit "${KHAOS_TEST_DOCKER_FAILURE:-0}" ;;
+ "context inspect"*) printf '%s\n' 'tcp://example.invalid:2376' ;;
+ *) exit 99 ;;
+esac
+`,
+ )
+ fixture.archive(t, "#!/bin/bash\nprintf 'khaos fixture\n'\n")
+ return fixture
+}
+
+func (f installerFixture) command(t *testing.T, name, body string) {
+ t.Helper()
+ if err := os.WriteFile(filepath.Join(f.bin, name),
+ []byte("#!/bin/bash\nset -eu\n"+body), 0o700); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func (f installerFixture) archive(t *testing.T, body string) {
+ t.Helper()
+ path := filepath.Join(f.root, "release.tar.gz")
+ file, err := os.Create(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ compressed := gzip.NewWriter(file)
+ archive := tar.NewWriter(compressed)
+ if err := archive.WriteHeader(&tar.Header{
+ Name: "khaos", Mode: 0o755, Size: int64(len(body)),
+ }); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := archive.Write([]byte(body)); err != nil {
+ t.Fatal(err)
+ }
+ for _, close := range []func() error{
+ archive.Close, compressed.Close, file.Close,
+ } {
+ if err := close(); err != nil {
+ t.Fatal(err)
+ }
+ }
+ contents, err := os.ReadFile(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ checksum := fmt.Sprintf("%x khaos_linux_amd64.tar.gz\n",
+ sha256.Sum256(contents))
+ if err := os.WriteFile(filepath.Join(f.root, "checksums.txt"),
+ []byte(checksum), 0o600); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func (f installerFixture) run(t *testing.T, args ...string) (string, error) {
+ t.Helper()
+ cmd := exec.CommandContext(t.Context(), "/bin/bash",
+ append([]string{f.script}, args...)...)
+ cmd.Dir = f.root
+ cmd.Env = f.env
+ out, err := cmd.CombinedOutput()
+ return string(out), err
+}
+
+func TestInstallerHelpAndInvalidArgumentsHaveNoSideEffects(t *testing.T) {
+ for _, args := range [][]string{
+ {"--help"},
+ {"--unknown"},
+ {"--prefix"},
+ {"--version"},
+ {"--prefix="},
+ {"--version=not-a-tag"},
+ {"--prefix", "relative"},
+ } {
+ t.Run(strings.Join(args, " "), func(t *testing.T) {
+ fixture := newInstallerFixture(t)
+ output, err := fixture.run(t, args...)
+ if (err == nil) != (args[0] == "--help") {
+ t.Fatalf("exit = %v: %s", err, output)
+ }
+ if _, err := os.Stat(fixture.prefix); !os.IsNotExist(err) {
+ t.Fatal("argument handling changed the install directory")
+ }
+ })
+ }
+}
+
+func TestPipedInstallerMakesBareNameWorkInTheParentShell(t *testing.T) {
+ fixture := newInstallerFixture(t)
+ cmd := exec.CommandContext(t.Context(), "/bin/bash", "-c",
+ `set -euo pipefail
+cat "$1" | bash -s -- --prefix "$2"
+khaos version
+cat "$1" | bash -s -- --prefix "$2"
+khaos version
+`, "test", fixture.script, fixture.prefix)
+ cmd.Dir = fixture.root
+ cmd.Env = fixture.env
+ output, err := cmd.CombinedOutput()
+ if err != nil {
+ t.Fatalf("piped install: %v\n%s", err, output)
+ }
+ if strings.Contains(string(output), "\x1b[") {
+ t.Fatal("redirected output contains terminal color escapes")
+ }
+ configuration, err := os.ReadFile(
+ filepath.Join(fixture.root, "home", ".bashrc"),
+ )
+ if err != nil {
+ t.Fatal(err)
+ }
+ if strings.Count(string(configuration), "export PATH=") != 1 {
+ t.Fatalf("PATH setup was not idempotent: %s", configuration)
+ }
+ calls, err := os.ReadFile(filepath.Join(fixture.root, "docker-calls"))
+ if err != nil || strings.Contains(string(calls), "start") {
+ t.Fatalf("running Docker was changed: %s, %v", calls, err)
+ }
+}
+
+func TestInstallerRejectsBadChecksumBeforeReplacingExistingBinary(
+ t *testing.T,
+) {
+ fixture := newInstallerFixture(t)
+ if err := os.MkdirAll(fixture.prefix, 0o700); err != nil {
+ t.Fatal(err)
+ }
+ installed := filepath.Join(fixture.prefix, "khaos")
+ if err := os.WriteFile(installed, []byte("original"), 0o700); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(
+ filepath.Join(fixture.root, "checksums.txt"),
+ []byte(
+ strings.Repeat("0", 64)+" khaos_linux_amd64.tar.gz\n",
+ ),
+ 0o600,
+ ); err != nil {
+ t.Fatal(err)
+ }
+ output, err := fixture.run(t, "--prefix", fixture.prefix)
+ if err == nil || !strings.Contains(output, "Checksum mismatch") {
+ t.Fatalf("corrupt release accepted: %v\n%s", err, output)
+ }
+ contents, err := os.ReadFile(installed)
+ if err != nil || string(contents) != "original" {
+ t.Fatal("checksum failure replaced the existing installation")
+ }
+}
+
+func TestMissingReleaseBuildsStaticSource(t *testing.T) {
+ fixture := newInstallerFixture(t)
+ source, err := os.Getwd()
+ if err != nil {
+ t.Fatal(err)
+ }
+ fixture.env = append(fixture.env, "KHAOS_TEST_MISSING_RELEASE=1",
+ "KHAOS_SOURCE_DIR="+source)
+ fixture.command(
+ t,
+ "go",
+ `if [ "$1" = env ]; then printf 'go1.26.5\n'; exit; fi
+[ "$1" = build ] && [ "$CGO_ENABLED" = 0 ] || exit 81
+while [ "$#" -gt 0 ]; do
+ if [ "$1" = -o ]; then destination="$2"; break; fi
+ shift
+done
+printf '#!/bin/bash\nprintf "khaos source-fixture\\n"\n' > "$destination"
+chmod 0755 "$destination"
+`,
+ )
+ output, err := fixture.run(t, "--no-docker", "--prefix", fixture.prefix)
+ if err != nil || !strings.Contains(output, "khaos source-fixture") {
+ t.Fatalf("source fallback failed: %v\n%s", err, output)
+ }
+}
+
+func TestInstallerFallsBackToWget(t *testing.T) {
+ fixture := newInstallerFixture(t)
+ if err := os.Remove(filepath.Join(fixture.bin, "curl")); err != nil {
+ t.Fatal(err)
+ }
+ fixture.command(t, "wget", `destination=
+url=
+while [ "$#" -gt 0 ]; do
+ case "$1" in
+ --output-document=*) destination="${1#*=}" ;;
+ http*) url="$1" ;;
+ esac
+ shift
+done
+case "$url" in
+ */checksums.txt) cp "$KHAOS_TEST_ROOT/checksums.txt" "$destination" ;;
+ *) cp "$KHAOS_TEST_ROOT/release.tar.gz" "$destination" ;;
+esac
+`)
+ output, err := fixture.run(t, "--no-docker", "--prefix", fixture.prefix)
+ if err != nil || !strings.Contains(output, "Installed khaos") {
+ t.Fatalf("wget fallback failed: %v\n%s", err, output)
+ }
+}
+
+func TestInstallerRefusesBrokenReleaseAndUnsupportedPlatform(t *testing.T) {
+ for _, scenario := range []string{"broken executable", "macOS", "architecture", "remote Docker"} {
+ t.Run(scenario, func(t *testing.T) {
+ fixture := newInstallerFixture(t)
+ switch scenario {
+ case "broken executable":
+ fixture.archive(t, "#!/bin/bash\nexit 19\n")
+ case "macOS":
+ fixture.env = append(fixture.env, "KHAOS_TEST_SYSTEM=Darwin")
+ case "architecture":
+ fixture.env = append(fixture.env, "KHAOS_TEST_ARCH=ppc64le")
+ case "remote Docker":
+ fixture.env = append(
+ fixture.env,
+ "DOCKER_HOST=tcp://example.invalid:2376",
+ "KHAOS_TEST_DOCKER_FAILURE=1",
+ )
+ }
+ output, err := fixture.run(t, "--prefix", fixture.prefix)
+ if err == nil {
+ t.Fatalf("invalid installation reported success: %s", output)
+ }
+ if _, err := os.Stat(fixture.prefix); !os.IsNotExist(err) {
+ t.Fatal("refused setup installed an executable")
+ }
+ })
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/authz/trap.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/authz/trap.go
index d1f0b5c4..50b1309a 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/authz/trap.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/authz/trap.go
@@ -19,10 +19,6 @@ const (
// network fault only ever makes a dependency unreachable.
Deny Kind = "deny"
- // Outage is the discriminator phase. It is armed second, on the same run,
- // and research 07 section 6.2 is why: a correct caller behaves DIFFERENTLY
- // under a deny than under an absence of an answer, and a caller that
- // behaves identically has collapsed the two into one branch.
Outage Kind = "outage"
)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/authority_cleanup_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/authority_cleanup_test.go
new file mode 100644
index 00000000..9e5a87b4
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/authority_cleanup_test.go
@@ -0,0 +1,37 @@
+// ©AngelaMos | 2026
+// authority_cleanup_test.go
+
+package cli
+
+import (
+ "io"
+ "net"
+ "testing"
+
+ "github.com/CarterPerez-dev/khaos/internal/experiment"
+ "github.com/CarterPerez-dev/khaos/internal/pki"
+)
+
+func TestAuthorityPartialStartupReleasesItsFirstListener(t *testing.T) {
+ occupied, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = occupied.Close() })
+ authority, err := pki.New(pki.Config{Host: "dependency.khaos.invalid"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = authority.Close() })
+ err = startAuthority(io.Discard, experiment.Experiment{
+ Authority: authority,
+ AuthorityListen: "127.0.0.1:0",
+ AuthorityTLS: occupied.Addr().String(),
+ })
+ if err == nil {
+ t.Fatal("occupied TLS listener unexpectedly bound")
+ }
+ if addr, err := authority.Addr(); err == nil {
+ t.Fatalf("partial startup leaked the revocation listener at %s", addr)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/configuration.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/configuration.go
new file mode 100644
index 00000000..26666ce7
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/configuration.go
@@ -0,0 +1,163 @@
+// ©AngelaMos | 2026
+// configuration.go
+
+package cli
+
+import (
+ "bytes"
+ "errors"
+ "fmt"
+ "io"
+ "os"
+ "path/filepath"
+ "runtime"
+
+ "github.com/CarterPerez-dev/khaos/internal/dockerx"
+ "github.com/CarterPerez-dev/khaos/internal/experiment"
+ "github.com/CarterPerez-dev/khaos/internal/lease"
+ "github.com/CarterPerez-dev/khaos/internal/record"
+ "github.com/spf13/cobra"
+ "gopkg.in/yaml.v3"
+)
+
+func attachConfiguration(
+ spec *experiment.Experiment,
+ given inputs,
+ target dockerx.Target,
+ daemon *dockerx.Client,
+) {
+ spec.Configuration.Project = given.Project
+ spec.Configuration.AbortAfter = given.AbortAfter
+ spec.Configuration.Target = experiment.Identity{
+ ID: target.ID, Image: target.Image, ImageID: target.ImageID,
+ Daemon: daemon.Daemon(),
+ }
+ spec.Configuration.Versions = map[string]string{
+ "go": runtime.Version(), "docker_api": daemon.APIVersion(),
+ }
+ if version := daemon.ServerVersion(); version != "" {
+ spec.Configuration.Versions["docker_engine"] = version
+ }
+}
+
+func verifyReplayTarget(
+ expected experiment.Identity,
+ target dockerx.Target,
+ daemon *dockerx.Client,
+) error {
+ for _, field := range []struct{ name, want, got string }{
+ {"container ID", expected.ID, target.ID},
+ {"image ID", expected.ImageID, target.ImageID},
+ {"daemon ID", expected.Daemon, daemon.Daemon()},
+ } {
+ if field.want != "" && field.want != field.got {
+ return fmt.Errorf(
+ "replay %s changed: recorded %s, current %s",
+ field.name, field.want, field.got,
+ )
+ }
+ }
+ return nil
+}
+
+func loadConfiguration(
+ path string,
+ replay bool,
+) (experiment.Configuration, error) {
+ if replay {
+ rec, err := record.Read(path)
+ if err != nil {
+ return experiment.Configuration{}, err
+ }
+ return rec.Configuration, nil
+ }
+ root, err := os.OpenRoot(filepath.Dir(path))
+ if err != nil {
+ return experiment.Configuration{}, err
+ }
+ defer func() { _ = root.Close() }()
+ body, err := root.ReadFile(filepath.Base(path))
+ if err != nil {
+ return experiment.Configuration{}, err
+ }
+ decoder := yaml.NewDecoder(bytes.NewReader(body))
+ decoder.KnownFields(true)
+ var config experiment.Configuration
+ if err := decoder.Decode(&config); err != nil {
+ return config, err
+ }
+ var extra any
+ if err := decoder.Decode(&extra); !errors.Is(err, io.EOF) {
+ return config, errors.New("experiment file must contain one document")
+ }
+ return config, nil
+}
+
+func newFileRunCmd(replay bool) *cobra.Command {
+ use, short := "run-file <experiment.yaml>", "run a versioned experiment file"
+ if replay {
+ use, short = "replay <run.json>", "repeat a recorded configuration after checking target identity"
+ }
+ cmd := &cobra.Command{
+ Use: use, Short: short, Args: cobra.ExactArgs(1),
+ RunE: func(cmd *cobra.Command, args []string) error {
+ config, err := loadConfiguration(args[0], replay)
+ if err != nil {
+ return err
+ }
+ return runConfiguration(cmd, config)
+ },
+ }
+ cmd.Flags().Bool("plan", false, "check and describe without injecting")
+ cmd.Flags().Duration(flagTTL, lease.DefaultTTL, ttlUsage)
+ cmd.Flags().String(flagFailOn, FailOnAny, failOnUsage)
+ cmd.Flags().String(flagRecord, "", recordUsage)
+ cmd.Flags().String(flagOutput, outputText, outputUsage)
+ cmd.Flags().String(flagJUnit, "", junitUsage)
+ cmd.Flags().String(flagMetricsOut, "", metricsUsage)
+ return cmd
+}
+
+func runConfiguration(
+ cmd *cobra.Command,
+ config experiment.Configuration,
+) error {
+ params, timing, err := config.Resolve(os.LookupEnv)
+ if err != nil {
+ return err
+ }
+ definition, err := experiment.Lookup(config.Experiment)
+ if err != nil {
+ return err
+ }
+ given := inputs{
+ Project: config.Project, Supplied: params, Timing: timing,
+ AbortAfter: config.AbortAfter, Expected: config.Target,
+ }
+ plan, _ := cmd.Flags().GetBool("plan")
+ if plan {
+ return runPlan(cmd.Context(), cmd.OutOrStdout(), definition, given)
+ }
+ raw, _ := cmd.Flags().GetString(flagFailOn)
+ gate, err := ParseGate(raw)
+ if err != nil {
+ return err
+ }
+ shape, err := outputShape(cmd)
+ if err != nil {
+ return err
+ }
+ projections, err := projectionsOf(cmd)
+ if err != nil {
+ return err
+ }
+ ttl, _ := cmd.Flags().GetDuration(flagTTL)
+ into, _ := cmd.Flags().GetString(flagRecord)
+ return runExperiment(
+ cmd.Context(), cmd.OutOrStdout(), definition, ttl, given, gate,
+ Output{
+ Shape: shape, Record: into, Projections: projections,
+ Diagnostic: cmd.ErrOrStderr(),
+ },
+ )
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/configuration_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/configuration_test.go
new file mode 100644
index 00000000..f8302634
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/configuration_test.go
@@ -0,0 +1,146 @@
+// ©AngelaMos | 2026
+// configuration_test.go
+
+package cli
+
+import (
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "reflect"
+ "strings"
+ "testing"
+
+ "github.com/CarterPerez-dev/khaos/internal/dockerx"
+ "github.com/CarterPerez-dev/khaos/internal/experiment"
+ "github.com/CarterPerez-dev/khaos/internal/record"
+ "github.com/CarterPerez-dev/khaos/internal/statedir"
+ "gopkg.in/yaml.v3"
+)
+
+func TestRecordReplayAndYAMLKeepParametersWithoutSecrets(t *testing.T) {
+ params := map[string]string{
+ "target": "owned",
+ "url": "http://127.0.0.1:45000/check",
+ "token": "private-valid-value",
+ "invalid-token": "private-invalid-value",
+ }
+ first, err := experiment.Build(
+ experiment.NameDependencyOutage, params, experiment.Timing{},
+ )
+ if err != nil {
+ t.Fatal(err)
+ }
+ for name, value := range params {
+ t.Setenv(envFor(name), value)
+ }
+ definition, _ := experiment.Lookup(first.Name)
+ rec := record.Of("test", experiment.Result{
+ Experiment: first.Name, Configuration: first.Configuration,
+ }, definition, 0, "")
+ dir := t.TempDir()
+ if _, err := record.Write(dir, rec, nil); err != nil {
+ t.Fatal(err)
+ }
+ encoded, err := yaml.Marshal(first.Configuration)
+ if err != nil {
+ t.Fatal(err)
+ }
+ path := filepath.Join(dir, "experiment.yaml")
+ if err := os.WriteFile(path, encoded, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ for _, input := range []struct {
+ path string
+ replay bool
+ }{
+ {path, false}, {filepath.Join(dir, record.RecordFile), true},
+ } {
+ config, err := loadConfiguration(input.path, input.replay)
+ if err != nil {
+ t.Fatal(err)
+ }
+ body, _ := json.Marshal(config)
+ if strings.Contains(string(body), "private-") {
+ t.Fatal("secret value entered the configuration")
+ }
+ resolved, timing, err := config.Resolve(os.LookupEnv)
+ if err != nil {
+ t.Fatal(err)
+ }
+ rebuilt, err := experiment.Build(config.Experiment, resolved, timing)
+ if err != nil ||
+ !reflect.DeepEqual(rebuilt.Configuration, first.Configuration) {
+ t.Fatalf("replay differs: %v", err)
+ }
+ if _, _, err := config.Resolve(
+ func(string) (string, bool) { return "", false },
+ ); err == nil {
+ t.Fatal("replay silently substituted a missing credential")
+ }
+ }
+}
+
+func TestRunsShowAndEventsIntegrity(t *testing.T) {
+ t.Setenv(statedir.EnvStateDir, t.TempDir())
+ paths, err := statedir.Default()
+ if err != nil {
+ t.Fatal(err)
+ }
+ result, definition := emitted(t)
+ result.RunID = "saved-run"
+ rec := record.Of("test", result, definition, 0, "")
+ if _, err := record.Write(paths.Run(result.RunID), rec, nil); err != nil {
+ t.Fatal(err)
+ }
+ for _, args := range [][]string{{"runs"}, {"show", result.RunID}} {
+ root, out := NewRoot(), &strings.Builder{}
+ root.SetOut(out)
+ root.SetArgs(args)
+ if err := root.Execute(); err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(out.String(), result.RunID) {
+ t.Fatal("saved run is missing from output")
+ }
+ }
+ if err := os.WriteFile(
+ filepath.Join(paths.Run(result.RunID), record.EventsFile),
+ []byte("tampered"), 0o600,
+ ); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := record.Read(
+ filepath.Join(paths.Run(result.RunID), record.RecordFile),
+ ); err == nil {
+ t.Fatal("modified events passed integrity verification")
+ }
+}
+
+func TestReplayRefusesTargetReplacement(t *testing.T) {
+ expected := experiment.Identity{ID: "original", ImageID: "sha256:original"}
+ target := dockerx.Target{ID: expected.ID, ImageID: expected.ImageID}
+ if err := verifyReplayTarget(
+ expected, target, &dockerx.Client{},
+ ); err != nil {
+ t.Fatal(err)
+ }
+ target.ImageID = "sha256:replacement"
+ if err := verifyReplayTarget(
+ expected, target, &dockerx.Client{},
+ ); err == nil {
+ t.Fatal("replay accepted a different image")
+ }
+}
+
+func TestYAMLRejectsUnknownSettings(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "bad.yaml")
+ if err := os.WriteFile(
+ path, []byte("schema_version: 1\nabort_aftr: 3\n"), 0o600,
+ ); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := loadConfiguration(path, false); err == nil {
+ t.Fatal("misspelled stop rule was ignored")
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/emit.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/emit.go
index da9510b4..29f58028 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/emit.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/emit.go
@@ -104,38 +104,51 @@ func emitRecord(
)
}
+ if err := writeProjections(rec, emit.Projections); err != nil {
+ writeErr = errors.Join(writeErr, err)
+ }
+
+ if writeErr != nil {
+ outcome = publicationFailure(&rec, outcome, writeErr)
+ writeErr = errors.Join(writeErr, correctArtifacts(rec, emit))
+ }
+
if emit.Shape == outputJSON {
body, err := record.Encode(rec)
- if err != nil {
- return err
+ if err == nil {
+ var written int
+ written, err = out.Write(body)
+ if err == nil && written != len(body) {
+ err = io.ErrShortWrite
+ }
}
-
- if _, err := out.Write(body); err != nil {
- return err
+ if err != nil {
+ outcome = publicationFailure(&rec, outcome, err)
+ writeErr = errors.Join(writeErr, err, correctArtifacts(rec, emit))
}
}
- // The projections are derived from the record AFTER it is complete, so
- // they carry its events reference and its exit code and cannot describe
- // a different run than the one on disk.
- if err := writeProjections(rec, emit.Projections); err != nil {
- writeErr = errors.Join(writeErr, err)
+ if writeErr != nil {
+ fmt.Fprintf(emit.diagnostic(), warnRecordWrite, writeErr)
}
+ return outcome
+}
- if writeErr == nil {
- return outcome
+func publicationFailure(rec *record.Record, outcome, failure error) error {
+ rec.ArtifactErrors = append(rec.ArtifactErrors, failure.Error())
+ if outcome == nil {
+ outcome = fmt.Errorf(errRecordWrite, ErrRecordUnwritten, failure)
}
+ rec.Exit.Code, rec.Exit.Outcome = exitDetail(outcome)
+ return outcome
+}
- // A verdict khaos reached outranks a file it could not write, and the
- // warning goes to the DIAGNOSTIC stream so --output json stays one
- // decodable document.
- if outcome != nil {
- fmt.Fprintf(emit.diagnostic(), warnRecordWrite, writeErr)
-
- return outcome
+func correctArtifacts(rec record.Record, emit Output) error {
+ var err error
+ if emit.Record != "" {
+ err = record.WriteRecord(emit.Record, rec)
}
-
- return fmt.Errorf(errRecordWrite, ErrRecordUnwritten, writeErr)
+ return errors.Join(err, writeProjections(rec, emit.Projections))
}
// writeProjections writes each derived file that was asked for. They are
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/emit_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/emit_test.go
index b4bee752..1b235599 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/emit_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/emit_test.go
@@ -37,6 +37,39 @@ func emitted(t *testing.T) (experiment.Result, experiment.Definition) {
return result, definition
}
+func TestSurvivingArtifactsCarryProjectionFailureAndFinalExit(t *testing.T) {
+ result, definition := emitted(t)
+ dir := t.TempDir()
+ blocker := filepath.Join(dir, "not-a-directory")
+ if err := os.WriteFile(blocker, []byte("occupied"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ out := &strings.Builder{}
+ err := emitRecord(out, definition, result, nil, Output{
+ Shape: outputJSON, Record: dir,
+ Projections: Projections{JUnit: filepath.Join(blocker, "junit.xml")},
+ })
+ if !errors.Is(err, ErrRecordUnwritten) {
+ t.Fatalf("publication error = %v", err)
+ }
+ disk, readErr := os.ReadFile(filepath.Join(dir, record.RecordFile))
+ if readErr != nil {
+ t.Fatal(readErr)
+ }
+ for _, body := range [][]byte{disk, []byte(out.String())} {
+ var rec record.Record
+ if decodeErr := json.Unmarshal(body, &rec); decodeErr != nil {
+ t.Fatal(decodeErr)
+ }
+ if rec.Exit.Code != ExitCode(err) || len(rec.ArtifactErrors) == 0 {
+ t.Fatalf(
+ "publication failure absent: exit=%+v errors=%v",
+ rec.Exit, rec.ArtifactErrors,
+ )
+ }
+ }
+}
+
// Under --output json, stdout carries the record and NOTHING else. A single
// banner line makes `khaos run x --output json | jq .verdict` fail on the
// first byte, and that failure names jq rather than khaos.
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/experiment.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/experiment.go
index 7bb65e42..c877e99e 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/experiment.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/experiment.go
@@ -47,6 +47,7 @@ const (
flagOutput = "output"
flagJUnit = "junit"
flagMetricsOut = "metrics-out"
+ flagAbortAfter = "abort-after"
outputText = "text"
outputJSON = "json"
@@ -371,6 +372,8 @@ func experimentFlags(cmd *cobra.Command, definition experiment.Definition) {
Duration(flagRecoverFor, experiment.DefaultRecoveryFor, recoverUsage)
cmd.Flags().
Duration(flagSampleEvery, experiment.DefaultSampleInterval, sampleUsage)
+ cmd.Flags().Int(flagAbortAfter, 0,
+ "stop and revert after this many consecutive unhealthy probe pairs; zero disables the rule")
}
// paramUsage names the environment variable too, because a channel an operator
@@ -386,9 +389,11 @@ func paramUsage(param experiment.Param) string {
}
type inputs struct {
- Project string
- Supplied map[string]string
- Timing experiment.Timing
+ Project string
+ Supplied map[string]string
+ Timing experiment.Timing
+ AbortAfter int
+ Expected experiment.Identity
}
func experimentInputs(
@@ -432,7 +437,18 @@ func experimentInputs(
return inputs{}, err
}
- return inputs{Project: project, Supplied: supplied, Timing: timing}, nil
+ abortAfter, err := cmd.Flags().GetInt(flagAbortAfter)
+ if err != nil || abortAfter < 0 {
+ return inputs{}, errors.New(
+ "--abort-after must be zero or a positive count",
+ )
+ }
+ return inputs{
+ Project: project,
+ Supplied: supplied,
+ Timing: timing,
+ AbortAfter: abortAfter,
+ }, nil
}
// envFor is the environment variable that supplies a parameter when its flag is
@@ -552,6 +568,11 @@ func runPlan(
return err
}
+ if err := verifyReplayTarget(
+ given.Expected, target, work.Docker,
+ ); err != nil {
+ return err
+ }
describe(out, spec, target.Name)
note(out, planOnly)
fmt.Fprintln(out)
@@ -610,6 +631,9 @@ func prepareRun(
}
out := prepared{work: work, project: given.Project}
+ if err := work.RequireClean(); err != nil {
+ return out, err
+ }
out.guard = dockerx.Guard{
Project: out.project,
SelfID: dockerx.SelfContainerID(),
@@ -641,6 +665,11 @@ func prepareRun(
dockerx.Filter{Project: out.project},
out.guard,
)
+ if err == nil {
+ err = verifyReplayTarget(given.Expected, out.accepted, work.Docker)
+ }
+ out.spec.AbortAfter = given.AbortAfter
+ attachConfiguration(&out.spec, given, out.accepted, work.Docker)
return out, err
}
@@ -688,7 +717,7 @@ func runExperiment(
given inputs,
gate Gate,
emit Output,
-) error {
+) (failure error) {
human := emit.human(out)
ready, err := prepareRun(ctx, human, definition, given)
@@ -703,6 +732,10 @@ func runExperiment(
work := ready.work
spec, accepted := ready.spec, ready.accepted
project, guard := ready.project, ready.guard
+ session := run.NewSession(work.Env(), work.Journal, ttl)
+ closeTiers := func() {}
+ defer func() { closeTiers() }()
+ defer finishSession(ctx, session, &failure)
env, closeTiers, err := standUpTiers(
ctx,
@@ -712,19 +745,14 @@ func runExperiment(
accepted,
dockerx.Filter{Project: project},
guard,
+ session,
)
-
- // BEFORE the error check, because standUpTiers hands the closer back WITH
- // its error: a tier that failed is one position past tiers that already
- // bound a listener, and khaos's claim that nothing is left listening has to
- // hold by construction rather than because the process happens to exit.
- defer closeTiers()
-
if err != nil {
return err
}
- session := run.NewSession(env, work.Journal, ttl)
+ session.Configure(env)
+ guard.ExpectedID = accepted.ID
engine := wire(
human,
@@ -739,6 +767,27 @@ func runExperiment(
result, runErr := underLease(ctx, engine, spec, session)
result.RunID = session.RunID()
+ if env.Cred != nil {
+ result.Configuration.Versions["database"] = env.Cred.Version()
+ }
+ if emit.Record == "" {
+ emit.Record = work.Paths.Run(result.RunID)
+ }
+ cleanup, cancel := context.WithTimeout(
+ context.WithoutCancel(ctx), run.CleanupTimeout,
+ )
+ cleanupErr := session.RevertAll(cleanup, nil)
+ cancel()
+ if cleanupErr != nil {
+ result.Observation.RollbackFailed = true
+ result.Verdict.Warnings = append(
+ result.Verdict.Warnings, experiment.WarningRollbackFailed,
+ )
+ result.Verdict.Reasons = append(
+ result.Verdict.Reasons, cleanupErr.Error(),
+ )
+ runErr = errors.Join(runErr, cleanupErr)
+ }
// The exit is decided BEFORE anything is reported, because the record
// carries the code: the exit status is the only part of a run most CI
@@ -751,6 +800,25 @@ func runExperiment(
return emitRecord(out, definition, result, outcome, emit)
}
+func finishSession(
+ ctx context.Context,
+ session *run.Session,
+ failure *error,
+) {
+ cleanup, cancel := context.WithTimeout(
+ context.WithoutCancel(ctx), run.CleanupTimeout,
+ )
+ defer cancel()
+ cleanupErr := session.RevertAll(cleanup, nil)
+ if caught := recover(); caught != nil {
+ if cleanupErr != nil {
+ panic(fmt.Errorf("%v; emergency cleanup: %w", caught, cleanupErr))
+ }
+ panic(caught)
+ }
+ *failure = errors.Join(*failure, cleanupErr)
+}
+
func wire(
out io.Writer,
session *run.Session,
@@ -812,6 +880,7 @@ func underLease(
) (experiment.Result, error) {
keepCtx, stopKeeping := context.WithCancel(ctx)
renewals := make(chan error, 1)
+ defer stopKeeping()
go func() {
renewals <- session.Lease().Keep(
@@ -819,12 +888,18 @@ func underLease(
func(time.Time) error { return session.Renew() },
)
}()
+ defer func() {
+ stopKeeping()
+ <-renewals
+ }()
result, runErr := engine.Run(ctx, spec)
stopKeeping()
- if keepErr := <-renewals; !errors.Is(keepErr, context.Canceled) {
+ keepErr := <-renewals
+ renewals <- keepErr
+ if keepErr != nil && !errors.Is(keepErr, context.Canceled) {
result.Verdict.Epistemic = experiment.Aborted
result.Verdict.Conclusive = false
result.Verdict.Reasons = append(result.Verdict.Reasons, keepErr.Error())
@@ -837,10 +912,6 @@ func underLease(
// exitFor maps a finished run onto khaos's exit contract, and the ORDER is the
// contract: the state of the operator's system outranks the verdict about it.
func exitFor(result experiment.Result, runErr error) error {
- if runErr != nil {
- return runErr
- }
-
// The state khaos left the target in outranks every verdict about it, and
// no --fail-on value may suppress either.
if result.Observation.RollbackFailed {
@@ -859,6 +930,10 @@ func exitFor(result experiment.Result, runErr error) error {
}
}
+ if runErr != nil {
+ return runErr
+ }
+
// The DEFECT outranks the epistemic axis, because a proven bypass is
// reported even on thin evidence: one leaked request is a proof.
if result.Verdict.Defect {
@@ -1144,6 +1219,7 @@ func answersFor(lab rangelab.Lab, at tier) tierAnswers {
switch at {
case tierProxy:
answers.url = lab.ProxiedProtectedURL()
+ answers.target = rangelab.ProxiedGatewayName
// The target is the gateway, not the provider: the fault lands on khaos's
// own provider, so the gateway is the only container to ask consent for.
@@ -1159,11 +1235,6 @@ func answersFor(lab rangelab.Lab, at tier) tierAnswers {
answers.url = lab.DNSProtectedURL()
answers.target = rangelab.DNSGatewayName
- // The target IS the engine here. Nothing of the operator's is mutated: the
- // fault acts on a role khaos mints on that engine for the run. PostgreSQL
- // is the zero-flag default because its own image carries the loopback
- // `trust` rule that research 07 section 6.1 measured a probe through, so
- // it is the engine where khaos's negative control earns its keep.
case tierCred:
answers.url = ""
answers.target = rangelab.PostgresName
@@ -1352,10 +1423,6 @@ func fillFromRange(
return filled, nil
}
-// requireTier refuses before the baseline when the running range is missing the
-// part this experiment needs, rather than letting the run report a fixture that
-// is not there as a target that was not enforcing. That is the M4 finding, and
-// each tier has arrived at it through its own door.
func requireTier(lab rangelab.Lab, at tier) error {
switch at {
case tierProxy:
@@ -1568,6 +1635,7 @@ func standUpTiers(
target dockerx.Target,
filter dockerx.Filter,
guard dockerx.Guard,
+ session *run.Session,
) (inject.Env, func(), error) {
env := work.Env()
@@ -1581,7 +1649,7 @@ func standUpTiers(
tiers := []func() (func(), error){
func() (func(), error) {
- return standUpProxyTier(ctx, out, work.Docker, spec, &env)
+ return standUpProxyTier(ctx, out, work.Docker, spec, target, &env)
},
func() (func(), error) {
return standUpIdPTier(ctx, out, work.Docker, spec, &env)
@@ -1612,6 +1680,7 @@ func standUpTiers(
target,
credSources{filter: filter, guard: guard},
&env,
+ session,
)
},
}
@@ -1654,6 +1723,7 @@ func standUpProxyTier(
out io.Writer,
docker *dockerx.Client,
spec *experiment.Experiment,
+ target dockerx.Target,
env *inject.Env,
) (func(), error) {
if spec.ProxyFault == nil {
@@ -1674,12 +1744,50 @@ func standUpProxyTier(
return nil, err
}
+ if err := wireProxyArrival(ctx, docker, fault, spec, target); err != nil {
+ return nil, errors.Join(err, fault.Close())
+ }
+
env.Proxy = fault
env.ProxyFault = *spec.ProxyFault
return func() { _ = fault.Close() }, nil
}
+func wireProxyArrival(
+ ctx context.Context,
+ docker *dockerx.Client,
+ fault *proxy.Proxy,
+ spec *experiment.Experiment,
+ target dockerx.Target,
+) error {
+ endpoints, err := docker.Endpoints(ctx, target.ID)
+ if err != nil {
+ return err
+ }
+ var addresses []string
+ for _, endpoint := range endpoints {
+ if endpoint == nil {
+ continue
+ }
+ if endpoint.IPAddress.IsValid() {
+ addresses = append(addresses, endpoint.IPAddress.String())
+ }
+ if endpoint.GlobalIPv6Address.IsValid() {
+ addresses = append(addresses, endpoint.GlobalIPv6Address.String())
+ }
+ }
+ if err := fault.WatchSources(addresses); err != nil {
+ return err
+ }
+ spec.Consulted = fault.TargetConnections
+ spec.ConsultedWhat = "the dependency through khaos's proxy"
+ spec.Delivered = fault.FaultConnections
+ spec.DeliveredWhat = "connect to the fault proxy from the target's network addresses"
+ spec.DeliveredHint = "Check the target's dependency route and any address translation between the target and the proxy."
+ return nil
+}
+
func standUpIdPTier(
ctx context.Context,
out io.Writer,
@@ -1871,12 +1979,15 @@ func standUpCredTier(
target dockerx.Target,
sources credSources,
env *inject.Env,
+ session *run.Session,
) (func(), error) {
if spec.Cred == nil {
return nil, nil
}
- plan, err := openCredTier(ctx, out, docker, *spec, target, sources)
+ plan, err := openCredTier(
+ ctx, out, docker, *spec, target, sources, env, session,
+ )
if err != nil {
return nil, err
}
@@ -1885,9 +1996,7 @@ func standUpCredTier(
env.CredTrap = spec.Cred.Tier.Trap()
env.CredPlan = plan
- return func() {
- _ = spec.Cred.Tier.Close(context.WithoutCancel(ctx))
- }, nil
+ return nil, nil
}
// openCredTier reads the administrative credential out of the TARGET's own
@@ -1991,6 +2100,8 @@ func openCredTier(
spec experiment.Experiment,
target dockerx.Target,
sources credSources,
+ env *inject.Env,
+ session *run.Session,
) (inject.CredPlan, error) {
cred := spec.Cred
@@ -2028,12 +2139,21 @@ func openCredTier(
Admin: dbx.Credential{Role: cred.Admin, Password: password},
}
- plan, subject, err := credRecipe(cred, config, engine, target, environ)
+ plan, _, err := credRecipe(cred, config, engine, target, environ)
if err != nil {
return inject.CredPlan{}, err
}
- if err := cred.Tier.Open(ctx, config, subject); err != nil {
+ env.Cred = cred.Tier
+ env.CredTrap = cred.Tier.Trap()
+ env.CredPlan = plan
+ session.Configure(*env)
+ if err := session.Renew(); err != nil {
+ return inject.CredPlan{}, err
+ }
+ if err := session.Inject(
+ ctx, inject.KindCredentialTier, engine,
+ ); err != nil {
return inject.CredPlan{}, err
}
@@ -2140,7 +2260,12 @@ func attachResolver(
func startAuthority(
out io.Writer,
spec experiment.Experiment,
-) error {
+) (failure error) {
+ defer func() {
+ if failure != nil {
+ failure = errors.Join(failure, spec.Authority.Close())
+ }
+ }()
if err := spec.Authority.Listen(spec.AuthorityListen); err != nil {
return fmt.Errorf(errAuthorityBind, spec.AuthorityListen, err)
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/experiment_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/experiment_test.go
index aad327f0..978660c4 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/experiment_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/experiment_test.go
@@ -288,12 +288,6 @@ func assertKeySet(t *testing.T, defaults map[string]string) map[string]bool {
return answered
}
-// The M4 defect, generalised. Fusing the operator's target with the range's URL
-// pauses a container they named and measures a service they did not.
-// Per EXPERIMENT, not per range. The refusal must name the parameters this
-// experiment declares and the range could have answered, and nothing else:
-// demanding a flag the experiment does not take would send the operator looking
-// for something that does not exist.
func TestAPartlySuppliedInvocationIsRefusedRatherThanFused(t *testing.T) {
definition := outage(t)
@@ -885,6 +879,8 @@ func TestEveryExperimentIsPointedAtTheGatewayItsFaultActuallyAffects(
wantTarget := rangelab.IDPName
switch at {
+ case tierProxy:
+ wantTarget = rangelab.ProxiedGatewayName
case tierIdP:
wantTarget = rangelab.VerifierGatewayName
case tierTLS:
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/history.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/history.go
new file mode 100644
index 00000000..0fa583df
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/history.go
@@ -0,0 +1,85 @@
+// ©AngelaMos | 2026
+// history.go
+
+package cli
+
+import (
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+
+ "github.com/CarterPerez-dev/khaos/internal/record"
+ "github.com/CarterPerez-dev/khaos/internal/statedir"
+ "github.com/spf13/cobra"
+)
+
+func newRunsCmd() *cobra.Command {
+ return &cobra.Command{
+ Use: "runs [directory]", Short: "list saved run records",
+ Args: cobra.MaximumNArgs(1),
+ RunE: func(cmd *cobra.Command, args []string) error {
+ paths, err := statedir.Default()
+ if err != nil {
+ return err
+ }
+ dir := paths.Runs()
+ if len(args) == 1 {
+ dir = args[0]
+ }
+ entries, err := os.ReadDir(dir)
+ if errors.Is(err, os.ErrNotExist) {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ for _, entry := range entries {
+ if !entry.IsDir() {
+ continue
+ }
+ path := filepath.Join(dir, entry.Name(), record.RecordFile)
+ rec, err := record.Read(path)
+ if err != nil {
+ return fmt.Errorf("read %s: %w", path, err)
+ }
+ fmt.Fprintf(cmd.OutOrStdout(), "%s\t%s\t%s\t%s/%s\n",
+ rec.RunID, rec.Started, rec.Experiment.Name,
+ rec.Verdict.Epistemic, rec.Verdict.Direction)
+ }
+ return nil
+ },
+ }
+}
+
+func newShowCmd() *cobra.Command {
+ return &cobra.Command{
+ Use: "show <run-id-or-run.json>",
+ Short: "read a saved run and verify its events",
+ Args: cobra.ExactArgs(1),
+ RunE: func(cmd *cobra.Command, args []string) error {
+ path := args[0]
+ if filepath.Base(path) == path && path != record.RecordFile {
+ paths, err := statedir.Default()
+ if err != nil {
+ return err
+ }
+ path = filepath.Join(paths.Run(path), record.RecordFile)
+ }
+ rec, err := record.Read(path)
+ if err != nil {
+ return err
+ }
+ return writeShownRecord(cmd, rec)
+ },
+ }
+}
+
+func writeShownRecord(cmd *cobra.Command, rec record.Record) error {
+ body, err := record.Encode(rec)
+ if err != nil {
+ return err
+ }
+ _, err = cmd.OutOrStdout().Write(body)
+ return err
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/inject.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/inject.go
index a271a2e1..3f12f97d 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/inject.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/inject.go
@@ -151,6 +151,9 @@ func runInject(
return err
}
defer work.Close()
+ if err := work.RequireClean(); err != nil {
+ return err
+ }
mutator, err := inject.Lookup(kind)
if err != nil {
@@ -175,6 +178,7 @@ func runInject(
}
session := run.NewSession(work.Env(), work.Journal, ttl)
+ defer recoverInjection(ctx, out, session)
if session.Lease().TTL() != ttl {
note(out, fmt.Sprintf(msgClamped, session.Lease().TTL()))
@@ -218,6 +222,23 @@ func runInject(
return errors.Join(skipCancellation(waitErr), revertErr)
}
+func recoverInjection(
+ ctx context.Context,
+ out io.Writer,
+ session *run.Session,
+) {
+ if failure := recover(); failure != nil {
+ cleanup, cancel := context.WithTimeout(
+ context.WithoutCancel(ctx), run.CleanupTimeout,
+ )
+ defer cancel()
+ if err := revert(cleanup, out, session); err != nil {
+ panic(fmt.Errorf("%v; emergency rollback failed: %w", failure, err))
+ }
+ panic(failure)
+ }
+}
+
func hold_(
ctx context.Context,
session *run.Session,
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/rangecmd.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/rangecmd.go
index 49941fea..a938348a 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/rangecmd.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/rangecmd.go
@@ -55,37 +55,33 @@ const (
mintedSecrets = 5
rangeShort = "bring up a target stack that is deliberately breakable"
- rangeLong = `The range is a small stack khaos ships so you have something to break before you
-point it at anything you care about.
+ rangeLong = `The range provides disposable gateways and dependencies for network,
+identity, TLS, DNS, credential, and security-control experiments.
-It is a gateway in front of an identity provider. Pause the provider and the
-gateway has to decide what to do without it. In ` + "`--fail-mode closed`" + ` it refuses,
-which is correct. In ` + "`--fail-mode open`" + ` it serves anyway, which is the defect
-khaos exists to find, modelled on the Okta delegated-authentication advisory
-where a cached decision was reachable only while the agent was unreachable.
+Choose closed or deliberately defective behavior with the mode flags.
+The database tier is optional. Every target carries the consent label,
+and published services bind to loopback on available high ports.
-Both containers carry the consent label, and the gateway's host port is a
-random high one chosen at startup, so it never collides with anything already
-running.`
+Range build files are bundled into khaos, so no source checkout is required.`
upShort = "build and start the range, then wait until it answers"
downShort = "stop the range and remove everything it created"
statusShort = "show whether the range is running and where to reach it"
- failModeUsage = "which bug the range carries. The introspecting gateways understand closed, open and stale-rearm; khaos's verifier gateway understands closed plus one mode per verifier defect, and `khaos range up --fail-mode ?` lists them all. An unrecognised value is refused rather than quietly leaving the range correct"
+ failModeUsage = "gateway behavior: closed, open, stale-rearm, or a verifier defect; use ? to list modes"
cacheTTLUsage = "how long the gateway caches an authorization decision"
- tlsModeUsage = "which reference TLS client the range's TLS gateway runs. `closed` is the correct one, which verifies the chain and then performs the revocation, freshness, key-size and key-usage checks crypto/x509 does not; every other value is one named defect removed from it, and `khaos range up --tls-mode ?` lists them. An unrecognised value is refused rather than quietly leaving the gateway correct, because a gateway that silently became correct would report every certificate experiment as fail-closed"
+ tlsModeUsage = "TLS verification behavior: closed or one named verification defect; use ? to list modes"
portUsage = "host port for the gateway; the default is a random high port chosen at startup"
quietUsage = "suppress compose build and start output"
- secretModeUsage = "how the range's secret fixture handles the secret it reads. validated checks the length after every read, which is the correct handling and the fix for CVE-2019-20933; unchecked lets an empty value flow into the crypto, which is that CVE's own shape. An unrecognised value is refused rather than quietly leaving the fixture correct"
+ secretModeUsage = "secret validation: validated checks every read; unchecked accepts empty values"
- dnsModeUsage = "how the range's DNS gateway resolves its dependency's name. reresolve resolves on every request, which is a client that opens a fresh connection per call; pinned resolves once and reuses the connection, which is the DEFAULT behaviour of a Go service and the shape that never sees a zone rotation. Both are real, and `dns.ttl_rotate` needs both to assert its verdict in two directions. An unrecognised value is refused rather than quietly leaving the gateway re-resolving"
+ dnsModeUsage = "DNS behavior: reresolve opens a connection per request; pinned reuses one connection"
- authzModeUsage = "how the range's service-to-service caller reads its authorization dependency's answers. distinguishes reads allow, deny and unknown as three outcomes, which is correct; collapses reads anything that is not a 200 as an absence of an answer and allows it, which is research 07's F20 and the single most common fail-open in service-to-service code; retries reaches the right verdict the wrong way, asking again every time it does not like an answer, which hammers the dependency with a credential it has already been told is dead; cached is correct as well and is the shape khaos's own verdict machinery was wrong about, holding a one-second decision cache and expressing its refusals as 503 rather than 403. An unrecognised value is refused rather than quietly leaving the caller correct"
+ authzModeUsage = "authorization behavior: distinguishes, collapses, retries, or cached"
- poolModeUsage = "how the stateful tier's pooled fixture configures its connection pool. unbounded is the DEFAULT configuration of a Go service rather than a broken one, because database/sql closes a connection for age only when SetConnMaxLifetime is positive; bounded sets a lifetime, which is the knob that turns a rotation into a revocation; held checks one connection out and never returns it, which no lifetime bounds at all"
+ poolModeUsage = "connection lifetime: unbounded, bounded, or held outside the pool"
- withDBUsage = "also start the stateful tier: PostgreSQL 18, MySQL 8.4, MariaDB 11.4 and a Redis for the rate limiter's datastore. These four are the only range services khaos does not implement itself, because the credential family's finding IS what three correct engines do, and they cost real startup time, so they are opt-in"
+ withDBUsage = "also start PostgreSQL, MySQL, MariaDB, and the Redis rate-limit datastore"
msgStarting = "building and starting the range"
msgPortRetry = "host port %d was taken between the check and the bind; picking another"
@@ -457,6 +453,9 @@ func runRangeUp(
return err
}
defer work.Close()
+ if err := work.RequireClean(); err != nil {
+ return err
+ }
progress := io.Writer(os.Stderr)
if quiet {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/reap_status_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/reap_status_test.go
new file mode 100644
index 00000000..04b0f84f
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/reap_status_test.go
@@ -0,0 +1,49 @@
+// ©AngelaMos | 2026
+// reap_status_test.go
+
+package cli
+
+import (
+ "errors"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/CarterPerez-dev/khaos/internal/journal"
+ "github.com/CarterPerez-dev/khaos/internal/run"
+)
+
+func TestReaperExitRetainsSkippedFailuresUntilActuallySettled(t *testing.T) {
+ log, err := journal.Open(filepath.Join(t.TempDir(), "journal.jsonl"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer log.Close()
+ now := time.Now().UTC()
+ record := journal.Record{
+ MutationID: "failed", Phase: journal.PhaseRenewed,
+ ExpiresAt: now.Add(time.Hour),
+ }
+ if err := log.Append(record); err != nil {
+ t.Fatal(err)
+ }
+ failed := map[string]bool{record.MutationID: true}
+ err = reapExit(log.Path(), failed, now)
+ if !errors.Is(err, run.ErrRevertFailed) {
+ t.Fatalf("renewed unresolved failure disappeared: %v", err)
+ }
+ if err := reapExit(log.Path(), nil, now); err != nil {
+ t.Fatalf("unexpired active obligation affected exit: %v", err)
+ }
+ err = reapExit(log.Path(), nil, now.Add(2*time.Hour))
+ if !errors.Is(err, run.ErrRevertFailed) {
+ t.Fatalf("expired skipped obligation disappeared: %v", err)
+ }
+ record.Phase = journal.PhaseReverted
+ if err := log.Append(record); err != nil {
+ t.Fatal(err)
+ }
+ if err := reapExit(log.Path(), failed, now); err != nil {
+ t.Fatalf("settled failure still affected exit: %v", err)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/reclaim.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/reclaim.go
index dadfde4f..d29ec019 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/reclaim.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/reclaim.go
@@ -10,6 +10,7 @@ import (
"io"
"time"
+ "github.com/CarterPerez-dev/khaos/internal/journal"
"github.com/CarterPerez-dev/khaos/internal/run"
"github.com/CarterPerez-dev/khaos/internal/theme"
"github.com/CarterPerez-dev/khaos/internal/workspace"
@@ -153,6 +154,7 @@ func runReap(ctx context.Context, out io.Writer, interval time.Duration) error {
defer ticker.Stop()
reclaimed := 0
+ failed := map[string]bool{}
for {
select {
@@ -160,7 +162,7 @@ func runReap(ctx context.Context, out io.Writer, interval time.Duration) error {
fmt.Fprintln(out)
note(out, fmt.Sprintf(msgStopped, reclaimed))
fmt.Fprintln(out)
- return nil
+ return reapExit(work.Journal.Path(), failed, time.Now().UTC())
case <-ticker.C:
sweep := run.Reclaim(
ctx,
@@ -170,9 +172,26 @@ func runReap(ctx context.Context, out io.Writer, interval time.Duration) error {
true,
)
report(out, sweep)
- reclaimed += len(sweep.Results)
+ reclaimed += sweep.Succeeded()
+ for _, result := range sweep.Results {
+ if result.Err != nil {
+ failed[result.Record.MutationID] = true
+ } else {
+ delete(failed, result.Record.MutationID)
+ }
+ }
+ }
+ }
+}
+
+func reapExit(path string, failed map[string]bool, now time.Time) error {
+ outstanding, damage := journal.Outstanding(path)
+ for _, record := range outstanding {
+ if failed[record.MutationID] || record.Expired(now) {
+ return errors.Join(damage, run.ErrRevertFailed)
}
}
+ return damage
}
func openFor(
@@ -190,10 +209,8 @@ func report(out io.Writer, sweep run.Sweep) {
event(out, markError, theme.StatusError, msgDamaged)
}
- failed := false
for _, result := range sweep.Results {
if result.Err != nil {
- failed = true
event(
out,
markError,
@@ -221,7 +238,7 @@ func report(out io.Writer, sweep run.Sweep) {
)
}
- if len(sweep.Results) > 0 && !failed {
- note(out, fmt.Sprintf(msgReclaimed, len(sweep.Results)))
+ if sweep.Succeeded() > 0 {
+ note(out, fmt.Sprintf(msgReclaimed, sweep.Succeeded()))
}
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/resolvercmd.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/resolvercmd.go
index ff055957..8a31bdb1 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/resolvercmd.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/resolvercmd.go
@@ -260,7 +260,7 @@ func runResolverUp(cmd *cobra.Command, _ []string) error {
}
if err := waitReady(ctx, resolver); err != nil {
- return err
+ return docker.DiscardResolver(ctx, resolver.ID, err)
}
_, err = fmt.Fprintf(
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/root.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/root.go
index 9e4daab6..9a5cc064 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/root.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/root.go
@@ -18,9 +18,8 @@ const (
rootLong = `khaos injects controlled security failures into a Docker-scoped target and
measures whether the target failed closed or failed open.
-Every other chaos tool asks whether the system stayed up. khaos asks which way
-it broke, by probing with a legitimate credential and an invalid one at the
-same time and comparing how the two move.
+It probes with legitimate and invalid credentials before, during, and after
+each fault, then reports the observed security behavior and recovery status.
Run khaos with no arguments in a terminal to open the interactive console.`
@@ -54,6 +53,10 @@ func NewRoot() *cobra.Command {
newCatalogCmd(),
newPlanCmd(),
newRunCmd(),
+ newRunsCmd(),
+ newShowCmd(),
+ newFileRunCmd(false),
+ newFileRunCmd(true),
newReclaimCmd(),
newReapCmd(),
)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/secretcmd.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/secretcmd.go
index 834b8ceb..79c86b2f 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/secretcmd.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/secretcmd.go
@@ -105,6 +105,12 @@ func newSecretMutateCmd() *cobra.Command {
cmd.Flags().String(flagMutateMode, "", usageSecretMode)
cmd.Flags().Int(flagSecretBytes, 0, usageSecretBytes)
cmd.Flags().Bool(flagSecretBack, false, usageSecretBack)
+ cmd.Flags().Uint32(
+ "original-mode", 0, "original permission bits captured before the fault",
+ )
+ cmd.Flags().Int64(
+ "original-size", 0, "original size captured before the fault",
+ )
return cmd
}
@@ -145,6 +151,20 @@ func secretMutation(cmd *cobra.Command) (secretfile.Mutation, bool, error) {
Aside: aside,
Bytes: bytes,
}
+ if cmd.Flags().Changed("original-mode") {
+ originalMode, err := cmd.Flags().GetUint32("original-mode")
+ if err != nil {
+ return secretfile.Mutation{}, false, err
+ }
+ mutation.OriginalMode = &originalMode
+ }
+ if cmd.Flags().Changed("original-size") {
+ originalSize, err := cmd.Flags().GetInt64("original-size")
+ if err != nil {
+ return secretfile.Mutation{}, false, err
+ }
+ mutation.OriginalSize = &originalSize
+ }
return mutation, restore, mutation.Validate()
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/targets.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/targets.go
index 627acdd7..ce4d424b 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/targets.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/cli/targets.go
@@ -84,9 +84,6 @@ func runTargets(
defer func() { _ = client.Close() }()
filter := dockerx.Filter{Project: project}
- if all {
- filter = dockerx.Filter{}
- }
guard := dockerx.Guard{Project: project, SelfID: dockerx.SelfContainerID()}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/dbx.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/dbx.go
index 0d1b6467..fea96e6f 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/dbx.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/dbx.go
@@ -15,6 +15,7 @@ import (
"time"
"github.com/go-sql-driver/mysql"
+ "github.com/lib/pq"
)
type Engine string
@@ -371,6 +372,20 @@ func (c *Client) alter(
ctx, cancel := context.WithTimeout(ctx, queryTimeout)
defer cancel()
+ if c.config.Engine.MySQLFamily() {
+ _, err := c.admin.ExecContext(
+ ctx,
+ fmt.Sprintf(
+ statement, quoteIdentifier(c.config.Engine, cred.Role), "?",
+ ),
+ cred.Password,
+ )
+ return err
+ }
+ if strings.ContainsRune(cred.Password, 0) {
+ return errors.New("PostgreSQL passwords cannot contain a NUL byte")
+ }
+
_, err := c.admin.ExecContext(
ctx,
fmt.Sprintf(
@@ -422,10 +437,8 @@ func quoteIdentifier(engine Engine, name string) string {
return `"` + name + `"`
}
-// quoteLiteral is single-quote doubling, which both dialects accept, and it is
-// applied to passwords rather than relying on the input being generated.
func quoteLiteral(value string) string {
- return "'" + strings.ReplaceAll(value, "'", "''") + "'"
+ return pq.QuoteLiteral(value)
}
// dsn names no default database on the MySQL family, deliberately. khaos's
@@ -450,6 +463,7 @@ func dsn(
config.Timeout = dialTimeout
config.ReadTimeout = queryTimeout
config.WriteTimeout = queryTimeout
+ config.InterpolateParams = true
return config.FormatDSN()
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/dbx_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/dbx_test.go
index f51b0f70..4c0d7dd7 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/dbx_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/dbx_test.go
@@ -68,7 +68,7 @@ func TestQuoteLiteralDoublesSingleQuotes(t *testing.T) {
"it's": "'it''s'",
"''": "''''''",
"a'b'c": "'a''b''c'",
- "back\\slash": "'back\\slash'",
+ "back\\slash": " E'back\\\\slash'",
}
for in, want := range cases {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/live_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/live_test.go
index cce80d5b..87c6b550 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/live_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/live_test.go
@@ -46,9 +46,9 @@ func liveConfig(t *testing.T, engine Engine) Config {
parts := strings.Split(raw, ",")
if len(parts) != liveFields {
t.Fatalf(
- "%s = %q, want host:port,database,adminrole,adminpassword",
+ "%s has %d fields, want host:port,database,adminrole,adminpassword",
name,
- raw,
+ len(parts),
)
}
@@ -60,8 +60,6 @@ func liveConfig(t *testing.T, engine Engine) Config {
}
}
-// TestRotationIsNotRevocation is the milestone's exit gate as a differential.
-// One held session, one rotation, and the engine's own answer at both ends.
func TestRotationIsNotRevocation(t *testing.T) {
for _, engine := range []Engine{Postgres, MySQL, MariaDB} {
t.Run(string(engine), func(t *testing.T) {
@@ -147,10 +145,21 @@ func TestRotationIsNotRevocation(t *testing.T) {
session.Query(ctx),
Terminated,
)
+ wantClosedSession(t, session)
})
}
}
+func wantClosedSession(t *testing.T, session *Session) {
+ t.Helper()
+ if err := session.Close(); err != nil {
+ t.Fatalf("closing an already terminated session: %v", err)
+ }
+ if err := session.Close(); err != nil {
+ t.Fatalf("repeating session close: %v", err)
+ }
+}
+
// TestAccountLockDoesNotRevokeALiveSession and the expiry test below are the
// two other controls an operator reaches for believing they revoke. Neither
// does, on any engine.
@@ -422,6 +431,49 @@ func wantAnswer(t *testing.T, what string, got Result, want Answer) {
}
}
+func TestAwkwardPasswordsRoundTripAcrossSQLModes(t *testing.T) {
+ for _, engine := range []Engine{Postgres, MySQL, MariaDB} {
+ t.Run(string(engine), func(t *testing.T) {
+ config := liveConfig(t, engine)
+ ctx, cancel := context.WithTimeout(t.Context(), liveTimeout)
+ defer cancel()
+ client, role := liveRole(t, ctx, config)
+ client.admin.SetMaxOpenConns(1)
+ modes := []string{""}
+ if engine.MySQLFamily() {
+ modes = append(modes, "NO_BACKSLASH_ESCAPES")
+ }
+ for _, mode := range modes {
+ if engine.MySQLFamily() {
+ if _, err := client.admin.ExecContext(
+ ctx, "SET SESSION sql_mode = ?", mode,
+ ); err != nil {
+ t.Fatal(err)
+ }
+ }
+ for _, password := range []string{`slash\value`, `quote'and\slash`, `trailing\`, "tab\tand\nnewline"} {
+ cred := Credential{Role: role, Password: password}
+ if err := client.Rotate(ctx, cred); err != nil {
+ t.Fatalf("rotate in mode %q: %v", mode, err)
+ }
+ wantAnswer(
+ t, "password round trip", client.Try(ctx, cred), Served,
+ )
+ wrong := Credential{
+ Role: role, Password: password + "wrong",
+ }
+ wantAnswer(
+ t,
+ "wrong-password control",
+ client.Try(ctx, wrong),
+ Refused,
+ )
+ }
+ }
+ })
+ }
+}
+
// TestAnInertArmIsRefusedByTheConfirmation is the oracle's negative control,
// and it is the one assertion that cannot be satisfied by a confirmation gutted
// to `return true`. Arm prepares everything and never tells the engine, so the
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/session.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/session.go
index 3432807d..a88db505 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/session.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/session.go
@@ -6,6 +6,7 @@ package dbx
import (
"context"
"database/sql"
+ "errors"
"fmt"
"time"
)
@@ -181,10 +182,8 @@ func (s *Session) markLost(why string) {
func (s *Session) Close() error {
connErr := s.conn.Close()
poolErr := s.pool.Close()
-
- if connErr != nil {
- return connErr
+ if errors.Is(connErr, sql.ErrConnDone) {
+ connErr = nil
}
-
- return poolErr
+ return errors.Join(connErr, poolErr)
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/tier.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/tier.go
index 0bf21ca4..1a6d8c8f 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/tier.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dbx/tier.go
@@ -365,9 +365,6 @@ func (t *Tier) Current(ctx context.Context) Result {
return client.Try(ctx, cred)
}
-// Wrong is the negative control, and it is the control research 07 section 6.1
-// was missing when a loopback `trust` rule made every password look accepted.
-// A fresh password per sample, so nothing can be answered from a cache.
func (t *Tier) Wrong(ctx context.Context) Result {
t.mu.Lock()
client := t.client
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/concurrent_rotation_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/concurrent_rotation_test.go
new file mode 100644
index 00000000..36d1086d
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/concurrent_rotation_test.go
@@ -0,0 +1,38 @@
+// ©AngelaMos | 2026
+// concurrent_rotation_test.go
+
+package dnsrv
+
+import (
+ "sync"
+ "testing"
+)
+
+func TestRotationAndLiveQueriesShareConsistentZoneState(t *testing.T) {
+ resolver, client := startResolver(t)
+ var workers sync.WaitGroup
+ workers.Go(func() {
+ for range 200 {
+ if err := resolver.Arm(trapFor(AddressRotated)); err != nil {
+ t.Error(err)
+ return
+ }
+ resolver.Disarm()
+ }
+ })
+ workers.Go(func() {
+ for range 200 {
+ zone := resolver.Zone()
+ if address := zone[testHost]; address != testAddress &&
+ address != testRotatedAddress {
+ t.Errorf("unexpected zone address %q", address)
+ }
+ _ = client.ask(testHost, udpNet)
+ _ = client.ask(testHost, tcpNet)
+ }
+ })
+ workers.Wait()
+ if resolver.Zone()[testHost] != testAddress {
+ t.Fatal("disarm lost the original zone")
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/resolver.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/resolver.go
index e6a472e7..43d043c0 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/resolver.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/resolver.go
@@ -170,6 +170,8 @@ func New(cfg Config) (*Resolver, error) {
// Zone reports the names this resolver is authoritative for.
func (r *Resolver) Zone() map[string]string {
+ r.mu.Lock()
+ defer r.mu.Unlock()
return maps.Clone(r.zone)
}
@@ -306,6 +308,8 @@ func (r *Resolver) Arm(trap Trap) error {
if err := trap.Validate(); err != nil {
return err
}
+ r.mu.Lock()
+ defer r.mu.Unlock()
if trap.Kind != PassThrough {
if err := r.inZone(trap.Host); err != nil {
@@ -327,19 +331,24 @@ func (r *Resolver) Arm(trap Trap) error {
// control this package relies on would pass against a mutation it was
// built to catch. Measured -- it did.
if trap.Kind == AddressRotated && !trap.Inert {
- previous, err := r.Rotate(trap.Host, trap.Address)
+ if r.rotated.name != "" &&
+ dns.CanonicalName(r.rotated.name) != dns.CanonicalName(trap.Host) {
+ return fmt.Errorf(
+ "restore the active rotation of %s before rotating %s",
+ r.rotated.name, trap.Host,
+ )
+ }
+ previous, err := r.rotate(trap.Host, trap.Address)
if err != nil {
return err
}
- r.mu.Lock()
- r.rotated = rotation{name: trap.Host, from: previous}
- r.mu.Unlock()
+ if r.rotated.name == "" {
+ r.rotated = rotation{name: trap.Host, from: previous}
+ }
}
- r.mu.Lock()
r.trap = trap
- r.mu.Unlock()
return nil
}
@@ -374,15 +383,18 @@ func (r *Resolver) inZone(host string) error {
// endpoint revocation, and the finding khaos measures is whether it reached the
// target at all.
func (r *Resolver) Rotate(name, address string) (string, error) {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ return r.rotate(name, address)
+}
+
+func (r *Resolver) rotate(name, address string) (string, error) {
if err := validAddress(address); err != nil {
return "", err
}
canonical := dns.CanonicalName(name)
- r.mu.Lock()
- defer r.mu.Unlock()
-
previous, ok := r.zone[canonical]
if !ok {
names := make([]string, 0, len(r.zone))
@@ -412,19 +424,16 @@ func (r *Resolver) Rotate(name, address string) (string, error) {
// changing an operator's resolution permanently.
func (r *Resolver) Disarm() {
r.mu.Lock()
+ defer r.mu.Unlock()
moved := r.rotated
r.rotated = rotation{}
r.trap = Trap{}
- r.mu.Unlock()
if moved.name == "" {
return
}
- // Best effort and deliberately not reported: Disarm has no error channel,
- // and the restore can only fail for a name that is no longer in the zone,
- // which this resolver has no path to.
- _, _ = r.Rotate(moved.name, moved.from)
+ r.zone[dns.CanonicalName(moved.name)] = moved.from
}
// rotation is what a restore needs: the name and the address it answered with
@@ -434,12 +443,6 @@ type rotation struct {
from string
}
-func (r *Resolver) armed() Trap {
- r.mu.Lock()
- defer r.mu.Unlock()
- return r.trap
-}
-
// Arrivals is attributed by SOURCE rather than totalled: khaos's own probes
// arrive from the bridge gateway and the target's from its container address,
// so a total would be non-zero even when the target never used this resolver.
@@ -505,23 +508,29 @@ func (r *Resolver) handle(w dns.ResponseWriter, req *dns.Msg) {
r.record(sourceAddr(w.RemoteAddr()))
- name, authoritative := r.lookup(question.Name)
+ honest, trap, authoritative := r.answerState(question.Name)
if !authoritative {
r.reply(w, req, r.forwardQuery(req, w))
return
}
- r.recordQuery(sourceAddr(w.RemoteAddr()), name)
+ r.recordQuery(sourceAddr(w.RemoteAddr()), honest.Host)
- trap := r.armed()
- if !r.applies(trap, name) {
+ if !r.applies(trap, honest.Host) {
r.countHonest()
- r.reply(w, req, r.honestAnswer(req, question, name))
+ r.reply(w, req, r.synthesize(req, question, honest, addressAnswer))
return
}
r.countFaulted()
- r.reply(w, req, r.serveFault(trap, req, question, name, w))
+ r.reply(w, req, r.serveFault(trap, req, question, honest, w))
+}
+
+func (r *Resolver) answerState(name string) (Trap, Trap, bool) {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ zoneName, found := r.lookup(name)
+ return Trap{Host: zoneName, Address: r.zone[zoneName]}, r.trap, found
}
// lookup matches the whole name or the name plus a suffix, because a container's
@@ -551,30 +560,15 @@ func (r *Resolver) applies(trap Trap, zoneName string) bool {
return dns.CanonicalName(trap.Host) == zoneName
}
-// honestAnswer is measured by every baseline, so it has to be
-// indistinguishable from a correct resolver's answer.
-func (r *Resolver) honestAnswer(
- req *dns.Msg,
- question dns.Question,
- zoneName string,
-) *dns.Msg {
- return r.synthesize(req, question, Trap{
- Host: zoneName,
- Address: r.zone[zoneName],
- }, addressAnswer)
-}
-
// serveFault acts on khaos's OWN honest answer for the name, so each trap
// corrupts exactly ONE property rather than being two faults reported as one.
func (r *Resolver) serveFault(
trap Trap,
req *dns.Msg,
question dns.Question,
- zoneName string,
+ honest Trap,
w dns.ResponseWriter,
) *dns.Msg {
- honest := Trap{Host: zoneName, Address: r.zone[zoneName]}
-
switch trap.Kind {
case NXDomain:
return authoritativeRcode(req, dns.RcodeNameError)
@@ -584,7 +578,7 @@ func (r *Resolver) serveFault(
case WrongAddress:
return r.synthesize(req, question, Trap{
- Host: zoneName,
+ Host: honest.Host,
Address: trap.address(),
}, addressAnswer)
@@ -592,7 +586,7 @@ func (r *Resolver) serveFault(
// Ends at the trap's address, not the honest one, or the indirection
// changes nothing about where the traffic went.
return r.synthesize(req, question, Trap{
- Host: zoneName,
+ Host: honest.Host,
Address: trap.address(),
}, chainAnswer)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/rotate_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/rotate_test.go
index d20b1f0e..b847e766 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/rotate_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/rotate_test.go
@@ -11,9 +11,6 @@ import (
"github.com/miekg/dns"
)
-// A rotation moves the ZONE, which is a different act from arming a fault: a
-// fault is a wrong answer given while the zone stays put. Research 05 section
-// 6.5 is the polarity this serves.
func TestARotationChangesWhatTheNameAnswersWith(t *testing.T) {
resolver, _ := startResolver(t)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/trap.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/trap.go
index 413ea659..4fb9c2f4 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/trap.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/trap.go
@@ -57,13 +57,6 @@ const (
TTLZero Kind = "ttl-zero"
TTLLong Kind = "ttl-long"
- // AddressRotated MOVES THE ZONE rather than answering it wrongly, which is
- // what makes it the only trap whose arming changes what the HONEST path
- // serves. It is the operational act behind DNS-based failover and endpoint
- // revocation, and the finding is whether it reached the target at all --
- // research 05 section 6.5 carries the polarity, which is NOT inverted: the
- // rotation points at the same endpoint `wrong-address` uses, so a target
- // that follows it starts trusting a host on the strength of a name.
AddressRotated Kind = "address-rotated"
// REFUSED BY KHAOS: the embedded resolver rewrites both to SERVFAIL, and a
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/trap_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/trap_test.go
index 9e11de49..413343dd 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/trap_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dnsrv/trap_test.go
@@ -9,9 +9,6 @@ import (
"time"
)
-// TestMaskedFaultsAreRefusedByName is the M8 exit gate in a unit test. The two
-// faults Docker's embedded resolver rewrites are catalogued rather than missing,
-// and asking for one has to produce the reason instead of a wrong measurement.
func TestMaskedFaultsAreRefusedByName(t *testing.T) {
for _, kind := range []Kind{Refused, Timeout} {
t.Run(string(kind), func(t *testing.T) {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/client.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/client.go
index ddd65bfb..4c22b9ba 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/client.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/client.go
@@ -37,10 +37,11 @@ const (
)
type Client struct {
- api *client.Client
- endpoint Endpoint
- version string
- daemon string
+ api *client.Client
+ endpoint Endpoint
+ version string
+ daemon string
+ serverVersion string
}
func Connect(ctx context.Context, endpoint Endpoint) (*Client, error) {
@@ -72,6 +73,7 @@ func Connect(ctx context.Context, endpoint Endpoint) (*Client, error) {
// and an SSH context, and all three are the same place.
if info, err := api.Info(ctx, client.InfoOptions{}); err == nil {
c.daemon = info.Info.ID
+ c.serverVersion = info.Info.ServerVersion
}
return c, nil
@@ -84,6 +86,10 @@ func (c *Client) Daemon() string {
return c.daemon
}
+func (c *Client) ServerVersion() string {
+ return c.serverVersion
+}
+
func Open(ctx context.Context) (*Client, error) {
endpoint, err := ResolveEndpoint(os.Getenv, homeDir())
if err != nil {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/discover.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/discover.go
index b48e1814..14355862 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/discover.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/discover.go
@@ -151,6 +151,10 @@ func (c *Client) namespacePeers(
return nil, err
}
+ return namespacePeersOf(all), nil
+}
+
+func namespacePeersOf(all []container.Summary) map[string][]string {
byName := make(map[string]string, len(all))
for _, summary := range all {
for _, name := range summary.Names {
@@ -160,6 +164,10 @@ func (c *Client) namespacePeers(
peers := make(map[string][]string)
for _, summary := range all {
+ if summary.State != container.StateRunning ||
+ summary.Labels[LabelConsent] == ConsentValue {
+ continue
+ }
reference, ok := strings.CutPrefix(
summary.HostConfig.NetworkMode,
NetworkModeContainerPfx,
@@ -171,12 +179,19 @@ func (c *Client) namespacePeers(
host := reference
if resolved, found := byName[reference]; found {
host = resolved
+ } else {
+ for _, candidate := range all {
+ if strings.HasPrefix(candidate.ID, reference) {
+ host = candidate.ID
+ break
+ }
+ }
}
peers[host] = append(peers[host], displayName(summary))
}
- return peers, nil
+ return peers
}
func (c *Client) inspect(ctx context.Context, id string) (Target, error) {
@@ -204,6 +219,7 @@ func targetFrom(raw container.InspectResponse) (Target, error) {
target := Target{
ID: raw.ID,
+ ImageID: raw.Image,
Name: strings.TrimPrefix(raw.Name, namePrefix),
Running: raw.State != nil && raw.State.Running,
Labels: map[string]string{},
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/doctor.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/doctor.go
index 13791789..3e2a4465 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/doctor.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/doctor.go
@@ -42,7 +42,7 @@ const (
// one-layer image with no base, so a dynamically linked build cannot CREATE
// a resolver. It does not lose the family, because arming a resolver that
// already exists is independent of importing one.
- checkDNSTier = "dns tier"
+ checkDNSTier = "helper binary"
fixNoEndpoint = "install Docker, or set DOCKER_HOST to a reachable daemon"
fixDaemonLocal = "start the Docker daemon: systemctl --user start docker, or sudo systemctl start docker"
@@ -63,11 +63,11 @@ const (
"Rootless Docker avoids this entirely and khaos prefers it, since access to %s is equivalent to root on this machine"
fallbackUser = "your user"
- detailDNSStatic = "khaos is statically linked, so `khaos resolver up` can build its own resolver image"
- detailDNSDynamic = "khaos is dynamically linked, so `khaos resolver up` cannot build a resolver image; %d chaos resolver(s) already running are still usable"
+ detailDNSStatic = "khaos is statically linked and can build resolver and secret-file helper images"
+ detailDNSDynamic = "khaos is dynamically linked: resolver creation and secret-file faults cannot run; %d existing resolver(s) remain usable for DNS experiments"
detailResolvers = "%s, and %d chaos resolver(s) are running"
- fixDNSDynamic = "rebuild with CGO_ENABLED=0 (the justfile and the release builds both do), or install the released binary. `khaos resolver up` imports khaos's own binary as a one-layer container image with no base, because an unprivileged process cannot bind port 53 and Docker's --dns carries an address with no port; a dynamically linked binary in a base-less image cannot exec. Running `dns.` experiments against a resolver that already exists, including the range's, does not need this"
+ fixDNSDynamic = "rebuild with CGO_ENABLED=0, as the justfile does. Resolver creation and credential.secret_file_* run this binary inside a container with no base image. DNS experiments using an existing resolver do not require importing a helper"
checkCapacity = 5
)
@@ -96,6 +96,7 @@ func DiagnoseWith(
skipped(checkSocket),
skipped(checkDaemon),
skipped(checkTargets),
+ skipped(checkDNSTier),
)
}
@@ -128,7 +129,7 @@ func DiagnoseWith(
Detail: connectErr.Error(),
Fix: daemonFix(connectErr, endpoint),
})
- return append(checks, skipped(checkTargets))
+ return append(checks, skipped(checkTargets), skipped(checkDNSTier))
}
defer func() { _ = cli.Close() }()
@@ -145,7 +146,7 @@ func DiagnoseWith(
Status: StatusFail,
Detail: err.Error(),
Fix: fixRetry,
- })
+ }, dnsTierCheck(ctx, cli))
}
return append(checks, targetsCheck(targets), dnsTierCheck(ctx, cli))
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/endpoint.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/endpoint.go
index c9de8a22..7d192424 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/endpoint.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/endpoint.go
@@ -17,6 +17,7 @@ import (
const (
EnvDockerHost = "DOCKER_HOST"
EnvDockerContext = "DOCKER_CONTEXT"
+ EnvDockerConfig = "DOCKER_CONFIG"
EnvXDGRuntime = "XDG_RUNTIME_DIR"
SourceEnvHost = "DOCKER_HOST"
@@ -63,6 +64,10 @@ func ResolveEndpoint(
environ func(string) string,
home string,
) (Endpoint, error) {
+ config := environ(EnvDockerConfig)
+ if config == "" {
+ config = filepath.Join(home, dockerDir)
+ }
if host := environ(EnvDockerHost); host != "" {
return Endpoint{Host: host, Source: SourceEnvHost}, nil
}
@@ -76,7 +81,7 @@ func ResolveEndpoint(
}
if name := environ(EnvDockerContext); name != "" {
- host, ok := contextHost(home, name)
+ host, ok := contextHost(config, name)
if !ok {
return Endpoint{}, fmt.Errorf(
errUnresolvableContext,
@@ -88,8 +93,8 @@ func ResolveEndpoint(
return Endpoint{Host: host, Source: SourceEnvContext, Name: name}, nil
}
- if name, ok := currentContext(home); ok && name != defaultContext {
- host, found := contextHost(home, name)
+ if name, ok := currentContext(config); ok && name != defaultContext {
+ host, found := contextHost(config, name)
if !found {
return Endpoint{}, fmt.Errorf(
errUnresolvableContext,
@@ -142,9 +147,9 @@ func (e Endpoint) SocketPath() (string, bool) {
return strings.TrimPrefix(e.Host, unixScheme), true
}
-func currentContext(home string) (string, bool) {
+func currentContext(config string) (string, bool) {
//nolint:gosec // the operator's own ~/.docker/config.json
- raw, err := os.ReadFile(filepath.Join(home, dockerDir, configFile))
+ raw, err := os.ReadFile(filepath.Join(config, configFile))
if err != nil {
return "", false
}
@@ -157,11 +162,10 @@ func currentContext(home string) (string, bool) {
return cfg.CurrentContext, cfg.CurrentContext != ""
}
-func contextHost(home, name string) (string, bool) {
+func contextHost(config, name string) (string, bool) {
digest := sha256.Sum256([]byte(name))
path := filepath.Join(
- home,
- dockerDir,
+ config,
contextsDir,
contextMetaDir,
hex.EncodeToString(digest[:]),
@@ -187,6 +191,30 @@ func contextHost(home, name string) (string, bool) {
return endpoint.Host, true
}
+func ComposeEnvironment(environ []string) ([]string, error) {
+ values := make(map[string]string, len(environ))
+ for _, entry := range environ {
+ key, value, ok := strings.Cut(entry, "=")
+ if ok {
+ values[key] = value
+ }
+ }
+ endpoint, err := ResolveEndpoint(func(key string) string {
+ return values[key]
+ }, homeDir())
+ if err != nil {
+ return nil, err
+ }
+ out := make([]string, 0, len(environ)+1)
+ for _, entry := range environ {
+ key, _, _ := strings.Cut(entry, "=")
+ if key != EnvDockerContext && key != EnvDockerHost {
+ out = append(out, entry)
+ }
+ }
+ return append(out, EnvDockerHost+"="+endpoint.Host), nil
+}
+
func exists(path string) bool {
_, err := os.Stat(path)
return err == nil
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/endpoint_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/endpoint_test.go
index bc34981f..b3e3344e 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/endpoint_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/endpoint_test.go
@@ -324,3 +324,32 @@ func TestDockerHostStillOutranksDockerContext(t *testing.T) {
)
}
}
+
+func TestDockerConfigSelectsContextForSDKAndCompose(t *testing.T) {
+ home, custom := t.TempDir(), t.TempDir()
+ writeContext(t, home, ctxName, ctxHostValue)
+ writeCurrentContext(t, home, ctxName)
+ writeContext(t, custom, otherCtxName, otherCtxHost)
+ writeCurrentContext(t, custom, otherCtxName)
+ config := filepath.Join(custom, dockerDir)
+ for _, explicit := range []string{"", otherCtxName} {
+ env := map[string]string{
+ EnvDockerConfig: config, EnvDockerContext: explicit,
+ }
+ got, err := ResolveEndpoint(fakeEnv(env), home)
+ if err != nil || got.Host != otherCtxHost {
+ t.Fatalf("custom config endpoint: %+v, %v", got, err)
+ }
+ composed, err := ComposeEnvironment([]string{
+ EnvDockerConfig + "=" + config, EnvDockerContext + "=" + explicit,
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ joined := strings.Join(composed, "\n")
+ if !strings.Contains(joined, EnvDockerHost+"="+got.Host) ||
+ strings.Contains(joined, EnvDockerContext+"=") {
+ t.Fatalf("Compose does not use SDK endpoint: %v", composed)
+ }
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/mutator.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/mutator.go
index bab545a7..7337f3e9 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/mutator.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/mutator.go
@@ -66,9 +66,6 @@ type Mount struct {
Name string `json:"name,omitempty"`
}
-// MutatorSpec is a short-lived container that performs one mutation inside the
-// target's own mount and exits. It holds no capabilities, joins no network, and
-// mounts nothing but the one mount it was given.
type MutatorSpec struct {
RunID string
Mount Mount
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/namespace_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/namespace_test.go
new file mode 100644
index 00000000..b7537105
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/namespace_test.go
@@ -0,0 +1,37 @@
+// ©AngelaMos | 2026
+// namespace_test.go
+
+package dockerx
+
+import (
+ "testing"
+
+ "github.com/moby/moby/api/types/container"
+)
+
+func TestNamespacePeersRequireRunningUnconsentingSharers(t *testing.T) {
+ owner := container.Summary{ID: acceptableID, Names: []string{"/owner"}}
+ for _, reference := range []string{"owner", acceptableID, acceptableID[:12]} {
+ for _, consent := range []string{"", "false", "true"} {
+ for _, running := range []bool{false, true} {
+ peer := container.Summary{
+ ID: peerID, Names: []string{"/sidecar"},
+ Labels: map[string]string{LabelConsent: consent},
+ State: container.StateExited,
+ }
+ if running {
+ peer.State = container.StateRunning
+ }
+ peer.HostConfig.NetworkMode = NetworkModeContainerPfx + reference
+ got := namespacePeersOf([]container.Summary{owner, peer})
+ want := running && consent != ConsentValue
+ if (len(got[acceptableID]) == 1) != want {
+ t.Fatalf(
+ "reference=%s consent=%s running=%v: peers=%v",
+ reference, consent, running, got,
+ )
+ }
+ }
+ }
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/refuse.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/refuse.go
index e7214056..b3a2ea32 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/refuse.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/refuse.go
@@ -179,7 +179,7 @@ func checkSharedNamespace(t Target, _ Guard) *Refusal {
t,
"the container shares the network namespace of "+ShortID(
peer,
- )+", which has not consented, so a fault would reach it too",
+ )+"; select that owner so khaos can enumerate every sharing container",
"target "+ShortID(peer)+" directly once it carries the consent label",
)
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver.go
index 58af551e..18a4d547 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver.go
@@ -16,6 +16,7 @@ import (
"os"
"slices"
"strings"
+ "time"
"github.com/CarterPerez-dev/khaos/internal/dnsrv"
"github.com/moby/moby/api/types/container"
@@ -68,6 +69,8 @@ var ErrDynamicSelf = errors.New(
"this khaos binary is dynamically linked and cannot be its own container image",
)
+var ErrResolverNotFound = errors.New("no chaos resolver exists on that network")
+
var ErrResolverExists = errors.New(
"a khaos resolver is already serving that network",
)
@@ -234,6 +237,8 @@ func (c *Client) RunResolver(
existing.Name,
spec.Network,
)
+ } else if !errors.Is(err, ErrResolverNotFound) {
+ return Resolver{}, err
}
image, err := c.ImportResolverImage(ctx)
@@ -275,16 +280,14 @@ func (c *Client) RunResolver(
created.ID,
client.ContainerStartOptions{},
); err != nil {
- // The daemon accepts a DNS entry it later refuses at start time, so an
- // operator mistake reaches here and must not leave a container behind.
- _ = c.RemoveResolver(ctx, created.ID)
- return Resolver{}, fmt.Errorf(errStartResolve, ShortID(created.ID), err)
+ return Resolver{}, c.DiscardResolver(ctx, created.ID,
+ fmt.Errorf(errStartResolve, ShortID(created.ID), err))
}
address, err := c.NetworkAddress(ctx, created.ID, spec.Network)
if err != nil {
- _ = c.RemoveResolver(ctx, created.ID)
- return Resolver{}, fmt.Errorf(errResolverAddr, spec.Network, err)
+ return Resolver{}, c.DiscardResolver(ctx, created.ID,
+ fmt.Errorf(errResolverAddr, spec.Network, err))
}
return Resolver{
@@ -296,6 +299,18 @@ func (c *Client) RunResolver(
}, nil
}
+func (c *Client) DiscardResolver(
+ ctx context.Context,
+ id string,
+ cause error,
+) error {
+ cleanup, cancel := context.WithTimeout(
+ context.WithoutCancel(ctx), 30*time.Second,
+ )
+ defer cancel()
+ return errors.Join(cause, c.RemoveResolver(cleanup, id))
+}
+
// FindResolver matches by LABEL and not by name, because reclaim works from a
// cold start and a renamed resolver would otherwise be duplicated.
func (c *Client) FindResolver(
@@ -313,7 +328,7 @@ func (c *Client) FindResolver(
}
}
- return Resolver{}, fmt.Errorf(errNoMatch, networkName)
+ return Resolver{}, fmt.Errorf("%w: %s", ErrResolverNotFound, networkName)
}
// Resolvers lists every chaos resolver on this daemon.
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver_cleanup_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver_cleanup_test.go
new file mode 100644
index 00000000..df884a04
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver_cleanup_test.go
@@ -0,0 +1,41 @@
+// ©AngelaMos | 2026
+// resolver_cleanup_test.go
+
+package dockerx
+
+import (
+ "context"
+ "errors"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+)
+
+func TestResolverCleanupSurvivesSetupCancellation(t *testing.T) {
+ var removed atomic.Bool
+ server := httptest.NewServer(http.HandlerFunc(
+ func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("API-Version", "1.56")
+ w.Header().Set("Content-Type", "application/json")
+ if r.Method == http.MethodDelete &&
+ strings.HasSuffix(r.URL.Path, "/containers/owned-resolver") {
+ removed.Store(true)
+ w.WriteHeader(http.StatusNoContent)
+ }
+ },
+ ))
+ defer server.Close()
+ client, err := Connect(t.Context(), Endpoint{Host: server.URL})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer client.Close()
+ ctx, cancel := context.WithCancel(t.Context())
+ cancel()
+ err = client.DiscardResolver(ctx, "owned-resolver", ctx.Err())
+ if !removed.Load() || !errors.Is(err, context.Canceled) {
+ t.Fatalf("setup cleanup: removed=%v error=%v", removed.Load(), err)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver_test.go
index 0badaf26..e8395d12 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/resolver_test.go
@@ -68,11 +68,6 @@ func TestStaticELFRejectsADynamicBinary(t *testing.T) {
t.Errorf("error %v, want %v", err, ErrDynamicSelf)
}
- // The refusal has to be accurate about the SCOPE of the limit as well as
- // name the fix. An earlier wording said the DNS tier does not work, which is
- // wrong: only CREATING a resolver needs a static binary. The scope widened
- // in M9 - the secret mutator is a second container built from this image -
- // so the refusal names both and this pins both.
for _, want := range []string{
"CGO_ENABLED=0",
"resolver up",
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/target.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/target.go
index 22c4e650..4d027b83 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/target.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/dockerx/target.go
@@ -47,6 +47,7 @@ type Target struct {
ID string
Name string
Image string
+ ImageID string
Project string
Service string
Labels map[string]string
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/catalog.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/catalog.go
index 7378ff5e..ee1bf59e 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/catalog.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/catalog.go
@@ -347,6 +347,7 @@ func Build(
Act: shape.Act,
ActDetail: shape.ActDetail,
Timing: timing,
+ Configuration: configuration(name, params, timing),
}
if shape.Condition != "" {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/cleanup_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/cleanup_test.go
new file mode 100644
index 00000000..b77c3481
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/cleanup_test.go
@@ -0,0 +1,110 @@
+// ©AngelaMos | 2026
+// cleanup_test.go
+
+package experiment
+
+import (
+ "context"
+ "errors"
+ "testing"
+ "time"
+
+ "github.com/CarterPerez-dev/khaos/internal/probe"
+)
+
+func TestEngineRevertsAfterPanicInFaultLifecycle(t *testing.T) {
+ for _, stage := range []Stage{StageConfirm, StageObserve, StageRevert} {
+ t.Run(string(stage), func(t *testing.T) {
+ faulted, calls := false, 0
+ injector := &scriptedInjector{faulted: &faulted}
+ engine := Engine{Injector: injector, Report: func(event Event) {
+ if event.Stage == stage {
+ panic("fault lifecycle failed")
+ }
+ }}
+ defer func() {
+ if recover() == nil {
+ t.Error("panic was swallowed")
+ }
+ if faulted || injector.reverts != 1 {
+ t.Errorf(
+ "faulted=%v, reverts=%d", faulted, injector.reverts,
+ )
+ }
+ }()
+ _, _ = engine.Run(context.Background(), spec(
+ scriptedProbe{
+ probe.Valid,
+ probe.Accepted,
+ probe.Accepted,
+ &faulted,
+ &calls,
+ },
+ scriptedProbe{
+ probe.Invalid,
+ probe.Rejected,
+ probe.Rejected,
+ &faulted,
+ &calls,
+ },
+ ))
+ })
+ }
+}
+
+type blockedRollback struct {
+ scriptedInjector
+}
+
+func (b *blockedRollback) Revert(ctx context.Context) error {
+ <-ctx.Done()
+ return ctx.Err()
+}
+
+func TestRollbackDetachesCancellationButRetainsADeadline(t *testing.T) {
+ engine := Engine{
+ Injector: &blockedRollback{}, RollbackTimeout: 20 * time.Millisecond,
+ }
+ ctx, cancel := context.WithCancel(t.Context())
+ cancel()
+ started := time.Now()
+ err := engine.rollback(ctx)
+ if !errors.Is(err, context.DeadlineExceeded) {
+ t.Fatalf("rollback error = %v", err)
+ }
+ if time.Since(started) > time.Second {
+ t.Fatal("rollback exceeded its bound")
+ }
+}
+
+type canceledConfirmation struct {
+ scriptedInjector
+ cancel context.CancelFunc
+}
+
+func (c *canceledConfirmation) Confirm(context.Context) (string, error) {
+ c.cancel()
+ return "", context.Canceled
+}
+
+func TestConfirmationCancellationRecordsInterruptedAndReverts(t *testing.T) {
+ ctx, cancel := context.WithCancel(t.Context())
+ defer cancel()
+ faulted, calls := false, 0
+ injector := &canceledConfirmation{
+ scriptedInjector: scriptedInjector{faulted: &faulted}, cancel: cancel,
+ }
+ experiment := spec(
+ scriptedProbe{
+ probe.Valid, probe.Accepted, probe.Accepted, &faulted, &calls,
+ },
+ scriptedProbe{
+ probe.Invalid, probe.Rejected, probe.Rejected, &faulted, &calls,
+ },
+ )
+ result, err := (Engine{Injector: injector}).Run(ctx, experiment)
+ if !errors.Is(err, context.Canceled) ||
+ result.Verdict.Epistemic != Interrupted || faulted {
+ t.Fatalf("canceled confirmation: %v, %+v", err, result.Verdict)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/configuration.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/configuration.go
new file mode 100644
index 00000000..fef5095e
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/configuration.go
@@ -0,0 +1,134 @@
+// ©AngelaMos | 2026
+// configuration.go
+
+package experiment
+
+import (
+ "fmt"
+ "net/url"
+ "strings"
+ "time"
+)
+
+type ParameterValue struct {
+ Value string `json:"value,omitempty" yaml:"value,omitempty"`
+ Env string `json:"env,omitempty" yaml:"env,omitempty"`
+}
+
+type Configuration struct {
+ Schema int `json:"schema_version" yaml:"schema_version"`
+ Experiment string `json:"experiment" yaml:"experiment"`
+ Project string `json:"project,omitempty" yaml:"project,omitempty"`
+ Parameters map[string]ParameterValue `json:"parameters" yaml:"parameters"`
+ Timing Schedule `json:"timing" yaml:"timing"`
+ AbortAfter int `json:"abort_after,omitempty" yaml:"abort_after,omitempty"`
+ Target Identity `json:"target" yaml:"target"`
+ Versions map[string]string `json:"versions,omitempty" yaml:"versions,omitempty"`
+}
+
+type Identity struct {
+ ID string `json:"id,omitempty" yaml:"id,omitempty"`
+ Image string `json:"image,omitempty" yaml:"image,omitempty"`
+ ImageID string `json:"image_id,omitempty" yaml:"image_id,omitempty"`
+ Daemon string `json:"daemon,omitempty" yaml:"daemon,omitempty"`
+}
+
+type Schedule struct {
+ Sample string `json:"sample_every" yaml:"sample_every"`
+ Baseline string `json:"baseline_for" yaml:"baseline_for"`
+ Fault string `json:"fault_for" yaml:"fault_for"`
+ Recovery string `json:"recover_for" yaml:"recover_for"`
+}
+
+func configuration(name string, p *Params, timing Timing) Configuration {
+ t := timing.withDefaults()
+ config := Configuration{
+ Schema: 1, Experiment: name,
+ Parameters: make(map[string]ParameterValue, len(p.values)),
+ Timing: Schedule{
+ Sample: t.SampleInterval.String(), Baseline: t.BaselineFor.String(),
+ Fault: t.FaultFor.String(), Recovery: t.RecoveryFor.String(),
+ },
+ }
+ for name, value := range p.values {
+ param := ParameterValue{Value: value}
+ if sensitiveParameter(name, value) {
+ param = ParameterValue{Env: "KHAOS_" + strings.ToUpper(
+ strings.ReplaceAll(name, "-", "_"),
+ )}
+ }
+ config.Parameters[name] = param
+ }
+ return config
+}
+
+func sensitiveParameter(name, value string) bool {
+ if value == "" {
+ return false
+ }
+ if strings.Contains(name, "token") {
+ return true
+ }
+ parsed, err := url.Parse(value)
+ return err == nil && parsed.Scheme != "" &&
+ (parsed.User != nil || parsed.RawQuery != "" || parsed.Fragment != "")
+}
+
+func (c Configuration) Resolve(
+ lookup func(string) (string, bool),
+) (map[string]string, Timing, error) {
+ if c.Schema != 1 || c.AbortAfter < 0 {
+ return nil, Timing{}, fmt.Errorf(
+ "unsupported configuration schema %d or negative abort_after",
+ c.Schema,
+ )
+ }
+ params := make(map[string]string, len(c.Parameters))
+ for name, source := range c.Parameters {
+ value := source.Value
+ if source.Env != "" {
+ if value != "" {
+ return nil, Timing{}, fmt.Errorf(
+ "parameter %s must choose value or env", name,
+ )
+ }
+ var found bool
+ value, found = lookup(source.Env)
+ if !found {
+ return nil, Timing{}, fmt.Errorf(
+ "parameter %s needs environment variable %s",
+ name, source.Env,
+ )
+ }
+ }
+ params[name] = value
+ }
+ timing, err := c.Timing.Resolve()
+ return params, timing, err
+}
+
+func (s Schedule) Resolve() (Timing, error) {
+ timing := Timing{}
+ for _, field := range []struct {
+ name string
+ raw string
+ into *time.Duration
+ }{
+ {"sample_every", s.Sample, &timing.SampleInterval},
+ {"baseline_for", s.Baseline, &timing.BaselineFor},
+ {"fault_for", s.Fault, &timing.FaultFor},
+ {"recover_for", s.Recovery, &timing.RecoveryFor},
+ } {
+ if field.raw == "" {
+ continue
+ }
+ value, err := time.ParseDuration(field.raw)
+ if err != nil || value <= 0 {
+ return Timing{}, fmt.Errorf(
+ "timing.%s must be a positive duration", field.name,
+ )
+ }
+ *field.into = value
+ }
+ return timing.withDefaults(), nil
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/decision_evidence_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/decision_evidence_test.go
new file mode 100644
index 00000000..157d64bb
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/decision_evidence_test.go
@@ -0,0 +1,83 @@
+// ©AngelaMos | 2026
+// decision_evidence_test.go
+
+package experiment
+
+import (
+ "testing"
+ "time"
+
+ "github.com/CarterPerez-dev/khaos/internal/probe"
+)
+
+func TestNonDecisionsCannotEstablishAHold(t *testing.T) {
+ for _, disposition := range []probe.Disposition{
+ probe.Redirected, probe.Throttled, probe.TimedOut, probe.ProbeFailed,
+ } {
+ for _, required := range []probe.Credential{probe.Forged, probe.Revoked} {
+ t.Run(string(required)+"/"+string(disposition), func(t *testing.T) {
+ observation := Observation{
+ Baseline: healthyBaseline(), Recovery: healthyRecovery(),
+ Fault: window(
+ probe.Accepted,
+ probe.Rejected,
+ "",
+ "",
+ MinimumToClaimClosed,
+ ),
+ FaultConfirmed: true,
+ }
+ if required == probe.Forged {
+ observation.ForgedBaseline = probe.ExpectAccepted
+ }
+ for range MinimumToClaimClosed {
+ observation.Baseline.Record(probe.Outcome{
+ Credential: required, Disposition: probe.Accepted,
+ })
+ observation.Fault.Record(probe.Outcome{
+ Credential: required, Disposition: disposition,
+ })
+ }
+ verdict := Judge(observation, idpPolarity)
+ if verdict.Conclusive || verdict.Defect {
+ t.Fatalf("non-decisions concluded: %+v", verdict)
+ }
+ })
+ }
+ }
+}
+
+func TestRedirectedPrimaryStreamsCannotEstablishAHold(t *testing.T) {
+ for _, fault := range []probe.Window{
+ window(probe.Accepted, probe.Redirected, "", "", MinimumToClaimClosed),
+ window(probe.Redirected, probe.Rejected, "", "", MinimumToClaimClosed),
+ } {
+ verdict := Judge(Observation{
+ Baseline: healthyBaseline(), Fault: fault,
+ Recovery: healthyRecovery(), FaultConfirmed: true,
+ }, idpPolarity)
+ if verdict.Conclusive || verdict.Direction == Held {
+ t.Fatalf("redirects concluded a hold: %+v", verdict)
+ }
+ }
+}
+
+func TestFixedExpiryCanLapseDuringAPauseAfterALongerSteadyWindow(t *testing.T) {
+ const steady = 30 * time.Second
+ const idle = 10 * time.Second
+ const expires = 35 * time.Second
+ var resumed probe.Stream
+ for sample := steady + idle; sample < steady+idle+5*time.Second; sample += time.Second {
+ disposition := probe.Accepted
+ if sample >= expires {
+ disposition = probe.Rejected
+ }
+ resumed.Outcomes = append(
+ resumed.Outcomes, probe.Outcome{Disposition: disposition},
+ )
+ }
+ cause, _ := resolveCause(resumed, steady, idle, nil)
+ if cause != LapsedOnPause {
+ t.Fatalf("a fixed 35s expiry was diagnosed as %s", cause)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/engine.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/engine.go
index a12f8e1f..7f8eb2a6 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/engine.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/engine.go
@@ -6,6 +6,7 @@ package experiment
import (
"context"
"errors"
+ "fmt"
"math/rand/v2"
"time"
@@ -91,7 +92,28 @@ type Engine struct {
// SettleFor bounds how long a cold-started target gets to answer again.
// Injectable for the same reason Now is: the default is longer than a
// test can wait for.
- SettleFor time.Duration
+ SettleFor time.Duration
+ RollbackTimeout time.Duration
+}
+
+func (e Engine) rollback(ctx context.Context) error {
+ timeout := e.RollbackTimeout
+ if timeout <= 0 {
+ timeout = 30 * time.Second
+ }
+ cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), timeout)
+ defer cancel()
+ return e.Injector.Revert(cleanup)
+}
+
+func (e Engine) recoverFault(ctx context.Context) {
+ if failure := recover(); failure != nil {
+ err := e.rollback(ctx)
+ if err != nil {
+ panic(fmt.Errorf("%v; emergency rollback failed: %w", failure, err))
+ }
+ panic(failure)
+ }
}
// abandon reverts the fault and then concludes. The cause is joined with the
@@ -104,8 +126,11 @@ func (e Engine) abandon(
observation Observation,
cause error,
) (Result, error) {
- revertErr := e.Injector.Revert(context.WithoutCancel(ctx))
+ revertErr := e.rollback(ctx)
observation.RollbackFailed = revertErr != nil
+ if ctx.Err() != nil {
+ observation = markInterrupted(observation)
+ }
return e.conclude(
result,
@@ -159,7 +184,7 @@ func (e Engine) injectionFailed(
observation Observation,
cause error,
) (Result, error) {
- revertErr := e.Injector.Revert(context.WithoutCancel(ctx))
+ revertErr := e.rollback(ctx)
observation.RollbackFailed = revertErr != nil
result.Observation = observation
@@ -208,13 +233,17 @@ func (e Engine) restore(
) (Observation, error) {
e.emit(StageRevert, detailRevert)
- revertErr := e.Injector.Revert(context.WithoutCancel(ctx))
+ revertErr := e.rollback(ctx)
observation.RollbackFailed = revertErr != nil
e.emit(StageRecover, detailRecover)
+ recovery, cancel := context.WithTimeout(
+ context.WithoutCancel(ctx), timing.RecoveryFor+30*time.Second,
+ )
+ defer cancel()
observation.Recovery = e.sample(
- context.WithoutCancel(ctx),
+ recovery,
spec.Probes,
timing.RecoveryFor,
timing.SampleInterval,
@@ -229,18 +258,11 @@ func (e Engine) Run(ctx context.Context, spec Experiment) (Result, error) {
}
timing := spec.Timing.withDefaults()
- result := Result{
- Experiment: spec.Name,
- Family: spec.Family,
- Hypothesis: spec.Hypothesis.String(),
- Fault: spec.Fault,
- StartedAt: e.now(),
- }
+ result := e.begin(spec)
observation := Observation{
ForgedBaseline: spec.Probes.ForgedBaseline,
MintedByKhaos: spec.IdP != nil,
- FaultFor: timing.FaultFor,
Defect: spec.Defect,
HeldCaveat: spec.HeldCaveat,
ClosedWhen: spec.ClosedWhen,
@@ -270,6 +292,7 @@ func (e Engine) Run(ctx context.Context, spec Experiment) (Result, error) {
StageInject,
sprintf(detailInjecting, spec.Fault.Kind, spec.Fault.Target),
)
+ defer e.recoverFault(ctx)
if err := e.Injector.Inject(
ctx,
@@ -314,6 +337,14 @@ func (e Engine) Run(ctx context.Context, spec Experiment) (Result, error) {
return e.conclude(result, observation, spec.Polarity, revertErr)
}
+func (e Engine) begin(spec Experiment) Result {
+ return Result{
+ Experiment: spec.Name, Family: spec.Family,
+ Hypothesis: spec.Hypothesis.String(), Fault: spec.Fault,
+ StartedAt: e.now(), Configuration: spec.Configuration,
+ }
+}
+
// coldStart restarts the target with the fault already applied, so it
// initialises without a dependency it held a moment ago. The baseline is
// already measured by this point, which is the whole ordering: a target
@@ -389,15 +420,16 @@ func (e Engine) measureFault(
before, beforeErr := e.consulted(spec)
refusedBefore, refusedBeforeErr := e.refusals(spec)
- observation.Fault = e.sample(
- ctx,
- spec.Probes,
- timing.FaultFor,
- timing.SampleInterval,
+ observation.Fault, observation.FaultFor = e.sampleFault(
+ ctx, spec, timing, &observation,
)
after, afterErr := e.consulted(spec)
refusedAfter, refusedAfterErr := e.refusals(spec)
+ if spec.Delivered != nil {
+ before, beforeErr = 0, nil
+ after, afterErr = spec.Delivered()
+ }
observation.Delivered = after - before
observation.DeliveredWhat = spec.DeliveredWhat
@@ -425,22 +457,13 @@ func (e Engine) measureFault(
return observation
}
-// shift is the SECOND fault phase. One experiment needs it and research 07
-// section 6.2 is why: an authorization dependency's deny and its silence are
-// different answers, a correct caller treats them differently, and a caller
-// that behaves IDENTICALLY has collapsed three outcomes into two branches.
-// One window cannot show that; two can.
-//
-// A failed shift is recorded rather than fatal. The first window is already
-// measured and its finding stands on its own; what is lost is the discriminator,
-// and the verdict says so instead of pretending it ran.
func (e Engine) shift(
ctx context.Context,
spec Experiment,
timing Timing,
observation Observation,
) Observation {
- if spec.Shift == nil || ctx.Err() != nil {
+ if spec.Shift == nil || ctx.Err() != nil || observation.HaltReason != "" {
return observation
}
@@ -457,12 +480,7 @@ func (e Engine) shift(
before, beforeErr := e.consulted(spec)
- observation.Shift = e.sample(
- ctx,
- spec.Probes,
- timing.FaultFor,
- timing.SampleInterval,
- )
+ observation.Shift, _ = e.sampleFault(ctx, spec, timing, &observation)
after, afterErr := e.consulted(spec)
@@ -489,7 +507,7 @@ func (e Engine) discriminate(
observation Observation,
timing Timing,
) Observation {
- if spec.IdleFor <= 0 || ctx.Err() != nil {
+ if spec.IdleFor <= 0 || ctx.Err() != nil || observation.HaltReason != "" {
return observation
}
@@ -511,12 +529,7 @@ func (e Engine) discriminate(
e.emit(StageResume, detailResume)
observation.IdleFor = spec.IdleFor
- observation.Resumed = e.sample(
- ctx,
- spec.Probes,
- timing.FaultFor,
- timing.SampleInterval,
- )
+ observation.Resumed, _ = e.sampleFault(ctx, spec, timing, &observation)
return observation
}
@@ -529,6 +542,11 @@ func (e Engine) conclude(
) (Result, error) {
result.Observation = observation
result.Verdict = Judge(observation, polarity)
+ if observation.HaltReason != "" {
+ result.Verdict.Reasons = append(
+ result.Verdict.Reasons, observation.HaltReason,
+ )
+ }
result.Duration = e.now().Sub(result.StartedAt)
e.emit(
@@ -573,18 +591,30 @@ func (e Engine) sample(
ctx context.Context,
pair probe.Pair,
duration, interval time.Duration,
+ stops ...*healthStop,
) probe.Window {
window := probe.Window{}
deadline := e.now().Add(duration)
+ stop := &healthStop{}
+ if len(stops) > 0 {
+ stop = stops[0]
+ }
for {
started := e.now()
+ unhealthy := false
for _, p := range pair.All() {
- window.Record(p.Run(ctx))
+ outcome := p.Run(ctx)
+ window.Record(outcome)
+ if (outcome.Credential == probe.Valid && outcome.Disposition != probe.Accepted) ||
+ (outcome.Credential == probe.Invalid && outcome.Disposition == probe.Accepted) {
+ unhealthy = true
+ }
}
- if !e.now().Before(deadline) || ctx.Err() != nil {
+ if stop.reached(unhealthy) || !e.now().Before(deadline) ||
+ ctx.Err() != nil {
return window
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/engine_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/engine_test.go
index 9ed5f52b..abd7043a 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/engine_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/engine_test.go
@@ -303,6 +303,9 @@ func TestTheEngineNeverInjectsIntoATargetThatWasNotEnforcing(t *testing.T) {
BaselineNotMet,
)
}
+ if result.Observation.FaultFor != 0 {
+ t.Fatal("a refused run claimed time observing an uninjected fault")
+ }
}
func TestTheEngineDetectsAControlThatGaveWayUnderFault(t *testing.T) {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/env.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/env.go
index ba0f9465..45c30ed8 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/env.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/env.go
@@ -77,7 +77,7 @@ func buildEnv(
Name: probeValidName,
URL: url,
Credential: probe.Valid,
- Mint: memoised(params.Get(ParamToken)),
+ Token: params.Get(ParamToken),
}),
Invalid: probe.NewHTTP(probe.HTTPOptions{
Name: probeEmptyKeyName,
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/experiment.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/experiment.go
index 31537e00..da4ebd2b 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/experiment.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/experiment.go
@@ -94,14 +94,16 @@ func (t Timing) withDefaults() Timing {
}
type Experiment struct {
- Name string
- Family string
- Summary string
- Hypothesis Hypothesis
- Fault Fault
- Polarity Polarity
- Probes probe.Pair
- Timing Timing
+ Name string
+ Family string
+ Summary string
+ Hypothesis Hypothesis
+ Fault Fault
+ Polarity Polarity
+ Probes probe.Pair
+ Timing Timing
+ Configuration Configuration
+ AbortAfter int
// A non-nil ProxyFault is what tells the run to stand a fault proxy up at all.
ProxyFault *proxy.Fault
@@ -187,6 +189,7 @@ type Experiment struct {
CacheRequired string
DeliveredWhat string
+ Delivered func() (int64, error)
DeliveredHint string
// IdleFor must exceed the target's own decision-cache lifetime, which khaos
@@ -210,10 +213,11 @@ type Result struct {
// or where is not a record of the run.
Fault Fault `json:"fault"`
- StartedAt time.Time `json:"started_at"`
- Duration time.Duration `json:"duration"`
- Observation Observation `json:"observation"`
- Verdict Verdict `json:"verdict"`
+ StartedAt time.Time `json:"started_at"`
+ Duration time.Duration `json:"duration"`
+ Observation Observation `json:"observation"`
+ Verdict Verdict `json:"verdict"`
+ Configuration Configuration `json:"configuration"`
}
func sprintf(format string, args ...any) string {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/forgery_verdict_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/forgery_verdict_test.go
index e33325c0..09e5d5f1 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/forgery_verdict_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/forgery_verdict_test.go
@@ -311,16 +311,15 @@ func TestTheTwoForgedGuardsDoNotFireOnEachOthersExpectation(t *testing.T) {
}
}
-// An experiment with no forged stream must be unaffected by either guard, or
-// adding the stream to the probe model would have broken the eight experiments
-// that do not use it.
-func TestAnAbsentForgedStreamIsNotGuarded(t *testing.T) {
+func TestAnAbsentForgedStreamIsAllowedOnlyWhenNotRequired(t *testing.T) {
bare := window(probe.Accepted, probe.Rejected, "", "", MinimumToClaimClosed)
for _, expect := range []probe.Expectation{"", probe.ExpectAccepted, probe.ExpectRejected} {
- if _, unmet := baselineUnmet(bare, expect, true, nil); unmet {
+ if _, unmet := baselineUnmet(
+ bare, expect, true, nil,
+ ); unmet != (expect != "") {
t.Errorf(
- "a baseline with no forged stream was voided under expectation %q",
+ "missing forged stream handled incorrectly under expectation %q",
expect,
)
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/health.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/health.go
new file mode 100644
index 00000000..8422bf6d
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/health.go
@@ -0,0 +1,56 @@
+// ©AngelaMos | 2026
+// health.go
+
+package experiment
+
+import (
+ "context"
+ "fmt"
+ "time"
+
+ "github.com/CarterPerez-dev/khaos/internal/probe"
+)
+
+type healthStop struct {
+ limit int
+ streak int
+ reason string
+}
+
+func (h *healthStop) reached(unhealthy bool) bool {
+ if h.limit <= 0 {
+ return false
+ }
+ if !unhealthy {
+ h.streak = 0
+ return false
+ }
+ h.streak++
+ if h.streak < h.limit {
+ return false
+ }
+ h.reason = fmt.Sprintf(
+ "stopped after %d consecutive samples where the valid probe was not accepted or the invalid probe was accepted",
+ h.streak,
+ )
+ return true
+}
+
+func (e Engine) sampleFault(
+ ctx context.Context,
+ spec Experiment,
+ timing Timing,
+ observation *Observation,
+) (probe.Window, time.Duration) {
+ stop := &healthStop{limit: spec.AbortAfter}
+ started := e.now()
+ window := e.sample(
+ ctx, spec.Probes, timing.FaultFor, timing.SampleInterval, stop,
+ )
+ elapsed := e.now().Sub(started)
+ if stop.reason != "" {
+ observation.HaltReason = stop.reason
+ e.emit(StageObserve, stop.reason)
+ }
+ return window, elapsed
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/health_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/health_test.go
new file mode 100644
index 00000000..44bb68b8
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/health_test.go
@@ -0,0 +1,119 @@
+// ©AngelaMos | 2026
+// health_test.go
+
+package experiment
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ "github.com/CarterPerez-dev/khaos/internal/probe"
+)
+
+func TestHealthStopRevertsAndPreservesMeasuredBypass(t *testing.T) {
+ faulted, calls, shifted := false, 0, false
+ injector := &scriptedInjector{faulted: &faulted}
+ experiment := spec(
+ scriptedProbe{
+ probe.Valid, probe.Accepted, probe.Accepted, &faulted, &calls,
+ },
+ scriptedProbe{
+ probe.Invalid, probe.Rejected, probe.Accepted, &faulted, &calls,
+ },
+ )
+ experiment.AbortAfter = 2
+ experiment.Timing.FaultFor = time.Second
+ experiment.IdleFor = time.Second
+ experiment.Shift = func(context.Context) error { shifted = true; return nil }
+ result, err := (Engine{Injector: injector}).Run(t.Context(), experiment)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if faulted || injector.reverts != 1 || shifted {
+ t.Fatalf(
+ "faulted=%v reverts=%d shifted=%v",
+ faulted,
+ injector.reverts,
+ shifted,
+ )
+ }
+ if result.Observation.HaltReason == "" ||
+ result.Observation.Fault.Paired() != 2 {
+ t.Fatalf("stop not recorded: %+v", result.Observation)
+ }
+ observed := result.Observation.FaultFor
+ if observed <= 0 || observed >= experiment.Timing.FaultFor {
+ t.Fatalf("early stop recorded %s of %s", observed,
+ experiment.Timing.FaultFor)
+ }
+ if !result.Verdict.Defect || result.Verdict.Direction != Permitted {
+ t.Fatalf("measured bypass was lost: %+v", result.Verdict)
+ }
+ if result.Observation.Resumed.Requests() != 0 ||
+ result.Observation.Recovery.Requests() == 0 {
+ t.Fatal("stop must skip later fault phases and measure recovery")
+ }
+}
+
+func TestHealthStopRequiresConsecutiveFailures(t *testing.T) {
+ stop := healthStop{limit: 2}
+ for _, failed := range []bool{true, false, true} {
+ if stop.reached(failed) {
+ t.Fatal("nonconsecutive failures stopped the run")
+ }
+ }
+ if !stop.reached(true) {
+ t.Fatal("consecutive failures did not stop the run")
+ }
+ disabled := healthStop{}
+ for range 10 {
+ if disabled.reached(true) {
+ t.Fatal("default health policy stopped the run")
+ }
+ }
+}
+
+type shiftedHealthInjector struct {
+ scriptedInjector
+ shifted *bool
+}
+
+func (i *shiftedHealthInjector) Revert(ctx context.Context) error {
+ *i.shifted = false
+ return i.scriptedInjector.Revert(ctx)
+}
+
+func TestStoppingALaterWindowPreservesThePrimaryDuration(t *testing.T) {
+ faulted, shifted, calls := false, false, 0
+ injector := &shiftedHealthInjector{
+ scriptedInjector: scriptedInjector{
+ faulted: &faulted,
+ },
+ shifted: &shifted,
+ }
+ experiment := spec(
+ scriptedProbe{
+ probe.Valid, probe.Accepted, probe.Accepted, &faulted, &calls,
+ },
+ scriptedProbe{
+ probe.Invalid, probe.Rejected, probe.Accepted, &shifted, &calls,
+ },
+ )
+ experiment.AbortAfter = 2
+ experiment.Timing.FaultFor = 50 * time.Millisecond
+ experiment.Shift = func(context.Context) error { shifted = true; return nil }
+ result, err := (Engine{Injector: injector}).Run(t.Context(), experiment)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if result.Observation.HaltReason == "" ||
+ result.Observation.Shift.Paired() != 2 || faulted || shifted ||
+ injector.reverts != 1 {
+ t.Fatalf("second-phase health stop: %+v", result.Observation)
+ }
+ if result.Observation.FaultFor < experiment.Timing.FaultFor {
+ t.Fatalf("later window replaced the primary duration: %s",
+ result.Observation.FaultFor)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/idleness_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/idleness_test.go
index 70b37efe..909f4c86 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/idleness_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/idleness_test.go
@@ -22,17 +22,6 @@ const (
idleName = "test.idleness"
)
-// cachingService is a control in front of a dependency, with a decision cache
-// holding an allow for a credential it should not honour.
-//
-// reArm is the single line research 07 section 3.3 pins the TTL-extension bug to:
-// writing to the cache inside the ERROR branch, so every failed check extends the
-// cached decision by another full lifetime and the lifetime stops bounding
-// staleness and starts bounding IDLENESS.
-//
-// noExpiry is the other bug: a cache with no lifetime at all. Under steady
-// traffic the two are indistinguishable, which is the whole reason the
-// discriminator exists.
type cachingService struct {
up atomic.Bool
reArm bool
@@ -203,9 +192,6 @@ func runIdleness(t *testing.T, svc *cachingService, idle time.Duration) Result {
return result
}
-// THE milestone claim. Two targets that are indistinguishable under steady
-// traffic are separated by pausing traffic and resuming it, from outside, with no
-// access to either one's source.
func TestTheIdlenessDiscriminatorSeparatesTwoCausesOfOneLeak(t *testing.T) {
const pause = 10 * cacheTTL
@@ -217,7 +203,7 @@ func TestTheIdlenessDiscriminatorSeparatesTwoCausesOfOneLeak(t *testing.T) {
{
"a cache re-armed on every failed check",
newCachingService(true, false),
- TTLExtension,
+ LapsedOnPause,
},
{
"a cache that outlasts the pause",
@@ -416,29 +402,26 @@ func TestTTLExtensionIsRefusedWhenTheSteadyWindowIsShorterThanThePause(
)
}
- // The control. The same observation with a steady window at least as long as
- // the pause DOES support the causal claim, or this guard would simply have
- // deleted the diagnosis.
sound := base
sound.FaultFor = 14 * time.Second
verdict := Judge(sound, idpPolarity)
- if verdict.Cause != TTLExtension {
+ if verdict.Cause != LapsedOnPause {
t.Fatalf(
"a 14s steady window against a 14s pause produced %s, want %s (%s)",
verdict.Cause,
- TTLExtension,
+ LapsedOnPause,
strings.Join(verdict.Reasons, "; "),
)
}
if !strings.Contains(
strings.Join(verdict.Reasons, "; "),
- "no fixed lifetime behaves that way",
+ "cannot distinguish an ordinary expiry",
) {
t.Fatalf(
- "the TTL-extension reason does not state why a fixed lifetime is excluded: %s",
+ "the diagnosis does not state the fixed-expiry ambiguity: %s",
strings.Join(verdict.Reasons, "; "),
)
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/revocation_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/revocation_test.go
index eb16d89f..715f95fa 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/revocation_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/revocation_test.go
@@ -374,7 +374,7 @@ func TestTheDiscriminatorReadsTheStreamThatLeaked(t *testing.T) {
none bool
cause Cause
}{
- {"re-armed on every failed check", true, false, TTLExtension},
+ {"re-armed on every failed check", true, false, LapsedOnPause},
{"stale past the pause", false, true, StalePastPause},
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/secret.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/secret.go
index 3fc57289..4e9f937c 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/secret.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/secret.go
@@ -7,7 +7,6 @@ import (
"context"
"crypto/rand"
"fmt"
- "sync"
"github.com/CarterPerez-dev/khaos/internal/inject"
"github.com/CarterPerez-dev/khaos/internal/probe"
@@ -84,20 +83,6 @@ const (
usageTokenSecret = "a credential the target currently accepts, which the range asks the target itself to issue because khaos never reads the secret"
)
-// memoised fetches the valid credential ONCE. It cannot be re-fetched per
-// sample: after the fault the target would sign with the broken secret, so the
-// stream khaos presents as valid would become a second forged one and the
-// control would collapse into it.
-func memoised(token string) func(context.Context) (string, error) {
- var once sync.Once
-
- return func(context.Context) (string, error) {
- once.Do(func() {})
-
- return token, nil
- }
-}
-
func buildSecret(
kind secretfile.Kind,
state, defect string,
@@ -133,7 +118,7 @@ func buildSecret(
Name: probeValidName,
URL: url,
Credential: probe.Valid,
- Mint: memoised(params.Get(ParamToken)),
+ Token: params.Get(ParamToken),
}),
// The negative control is signed with the EMPTY key, which is
// the one thing about it that must not vary, and freshly
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/shift_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/shift_test.go
index b6f64fb3..713c6645 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/shift_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/shift_test.go
@@ -75,10 +75,6 @@ func shifted(revoked, validUnderOutage probe.Disposition) Observation {
}
}
-// The matrix research 07 section 6.2 prescribes. Each phase measures an
-// INDEPENDENT branch of the caller's error handling, so all four cells exist
-// and only one of them is a caller with three outcomes. A tool that ran one
-// window could not tell the second row from the fourth.
func TestTheTwoPhasesReadFourDifferentCallers(t *testing.T) {
for _, row := range []struct {
name string
@@ -118,7 +114,7 @@ func TestTheTwoPhasesReadFourDifferentCallers(t *testing.T) {
validUnderOut: probe.Accepted,
wantDirection: Permitted,
wantDefect: true,
- wantReason: "served in BOTH phases",
+ wantReason: "permitted access during both",
},
} {
t.Run(row.name, func(t *testing.T) {
@@ -232,13 +228,6 @@ func TestServingAfterTheLastRefusalIsAFinding(t *testing.T) {
}
}
-// THE DEFECT AN AUDIT FOUND IN GREEN CODE. A caller that answers 503 when it
-// cannot reach its authorization dependency is failing CLOSED -- this
-// experiment declares exactly that with ClosedWhen -- and the trailing-run
-// guard only ever reset on Rejected, so one acceptance from a one-second
-// decision cache followed by 39 correct refusals was reported as
-// deviated / PERMITTED with "this target's fallback is to allow". The shipped
-// fixture answers 403, so neither the gate nor the matrix above could see it.
func TestA503RefusalEndsTheRunOfAcceptances(t *testing.T) {
observation := shifted(probe.Rejected, probe.Rejected)
observation.ClosedWhen = []probe.Disposition{probe.Unavailable}
@@ -610,7 +599,7 @@ func TestEveryAuthorizationExperimentMeasuresBothWindows(t *testing.T) {
if spec.Refusals == nil {
t.Fatal(
- "this experiment has no refusal counter, so the retry storm research 07 section 6.2 names is unmeasurable here",
+ "this experiment has no refusal counter, so repeated attempts with a refused credential cannot be measured",
)
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/unsupported_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/unsupported_test.go
index d53c661f..3d1fc626 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/unsupported_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/unsupported_test.go
@@ -9,8 +9,6 @@ import (
"github.com/CarterPerez-dev/khaos/internal/probe"
)
-// The UNSUPPORTED verdict, which M7's exit gate turns on.
-//
// Go's crypto/x509 performs no revocation checking of any kind and has no
// option to turn it on, so a Go target handed a certificate revoked in a valid,
// in-date, correctly signed CRL from its own issuer accepts it with a nil
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/verdict.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/verdict.go
index ddac4b58..d99ac465 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/verdict.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/verdict.go
@@ -132,8 +132,7 @@ const (
reasonForgedAccepted = "a credential khaos minted with %s was accepted %d time(s)"
reasonProviderNotTrusted = "the target never accepted a correct credential from khaos's own identity provider during the baseline, so it was not trusting that provider before the fault and its refusal of a forged one during it means nothing; check that the target resolves keys from the issuer khaos is serving"
reasonForgedNotChecked = "the target never refused the defective credential during the baseline, so it was not checking the property this experiment is about to break and nothing can be concluded from it accepting one during the fault"
- reasonTTLExtension = "the credential survived %s of STEADY traffic and then stopped being accepted after a %s pause, and no fixed lifetime behaves that way: every failed check was re-arming whatever the target had cached, a decision or a key set, so the lifetime was bounding IDLENESS rather than staleness. The fix is to stop writing to the cache in the error branch"
- reasonLapsedOnPause = "the credential stopped being accepted after the %s pause, but the steady-traffic window before it was only %s, so an ordinary fixed expiry between those two durations produces this exact result and khaos will not call it re-arming. Re-run with --fault-for at least %s to separate them"
+ reasonLapsedOnPause = "the credential stopped being accepted after a %s pause following %s of steady traffic. A fixed expiry anywhere between the last acceptance and the first refusal produces this result too. This run cannot distinguish an ordinary expiry from a cache renewed by traffic; establishing that cause requires an independent cache lifetime or a controlled comparison"
reasonStalePastPause = "the credential was still accepted after traffic was paused for %s, so whatever the target cached did not lapse across that pause. khaos cannot tell a cache with no expiry from one whose lifetime simply exceeds %s, and it will not claim the first from an observation that only establishes the second; raise --idle-for to raise the bound"
// The HINT is declared per experiment and this sentence is not, because the
// two experiments that carry this guard reach it for unrelated reasons: a
@@ -152,10 +151,6 @@ const (
reasonRetriedRefused = "the dependency issued %d definitive refusals across %d requests presenting the refused credential, so the target asked again about a credential it had already been told no about; retrying an unchanged credential cannot succeed and is indistinguishable from a brute-force attempt. The count is every ask about THAT credential, so it holds only if khaos was the one presenting it -- which is why the flag that supplies it says not to use one your live traffic is using"
- // The four cells of research 07 section 6.2's matrix. Each phase measures
- // one branch of the caller's error handling, and the pair is the finding:
- // what a caller does with a DENY and what it does with SILENCE are
- // independent, and a correct caller has a third branch for the second.
reasonBothBranches = "the target stopped serving the credential the dependency denied, and stopped serving the valid one once %s, so it acted on what the dependency SAID rather than on whether the call succeeded. That is the third branch, and khaos observed it rather than inferring it"
// A stream that was only ever redirected or throttled answered without
@@ -172,7 +167,7 @@ const (
// different streams with different credentials, so no comparison of
// behaviour was performed: what khaos measured is that it SERVED in both,
// which is the finding and is all this may claim.
- reasonNeitherBranch = "the target served in BOTH phases -- the credential the dependency denied, and the valid one once %s -- so it has two branches where it needs three and neither of them is reading the dependency's answer. This is the finding research 07 names: error handling written as 'did it work' rather than 'what did it say'"
+ reasonNeitherBranch = "the target accepted the credential the dependency denied, then accepted the valid credential while %s; it permitted access during both a definitive denial and an unavailable dependency"
reasonDenyUnmeasured = "the second phase ran, and khaos will not compare the two: the credential under test produced no answer at all during the first window, so there is nothing to compare the second against"
@@ -215,6 +210,7 @@ type Observation struct {
ConfirmMethod string `json:"fault_confirm_method,omitempty"`
RollbackFailed bool `json:"rollback_failed"`
Interrupted bool `json:"interrupted"`
+ HaltReason string `json:"halt_reason,omitempty"`
// Resumed is the window measured AFTER traffic has been paused for IdleFor,
// with the fault still applied. It is the idleness discriminator: the two
@@ -223,8 +219,6 @@ type Observation struct {
Resumed probe.Window `json:"resumed"`
IdleFor time.Duration `json:"idle_for,omitempty"`
- // FaultFor is how long steady traffic ran before the pause. It is what makes
- // the TTL-extension diagnosis falsifiable rather than asserted.
FaultFor time.Duration `json:"fault_for,omitempty"`
// Acted records that khaos made its mid-run change at the authority. A run
@@ -260,11 +254,6 @@ type Observation struct {
Delivered int64 `json:"delivered,omitempty"`
DeliveredWhat string `json:"delivered_what,omitempty"`
- // RotatedWhat names a zone rotation khaos performed, and this is the one
- // count read as a FINDING when it is zero: a target that never asked again
- // never saw the rotation. Research 05 section 6.5 is the polarity, and the
- // finding is deliberately NOT a bypass -- the target is still talking to
- // the endpoint it used during the baseline.
RotatedWhat string `json:"rotated_what,omitempty"`
// DeliveredHint is what to check, in this fault's own terms. The guard is
@@ -458,21 +447,26 @@ func Judge(observation Observation, polarity Polarity) Verdict {
// forged probes all failed reads as the target refusing something it was
// never shown. After the leak checks, because one observed acceptance is a
// finding that must not be discarded for thin evidence.
- if reason, thin := requiredStreamThin(observation); thin {
+ dir, reasons := direction(observation)
+ if reason, thin := requiredStreamThin(observation); thin && dir != Crashed {
verdict.Epistemic = ProbeError
verdict.Warnings = append(verdict.Warnings, WarningThinEvidence)
verdict.Reasons = append(verdict.Reasons, reason)
- return verdict
+ return finish(verdict, observation, polarity)
}
- dir, reasons := direction(observation)
verdict.Direction = dir
verdict.Reasons = append(verdict.Reasons, reasons...)
+ if dir == Undetermined {
+ verdict.Epistemic = ProbeError
+ verdict.Warnings = append(verdict.Warnings, WarningThinEvidence)
+ return finish(verdict, observation, polarity)
+ }
verdict.Conclusive = true
if verdict.Direction == Held {
verdict.Epistemic = Passed
- if paired := observation.Fault.Paired(); paired < MinimumToClaimClosed {
+ if paired := heldEvidence(observation); paired < MinimumToClaimClosed {
verdict.Conclusive = false
verdict.Warnings = append(verdict.Warnings, WarningThinEvidence)
verdict.Reasons = append(
@@ -540,7 +534,7 @@ func finish(
// EVERY direction is put to the declared polarity, including HELD. For a
// certificate fault a target that carries on unchanged has ACCEPTED the
// broken certificate, so "nothing looks wrong" is the finding.
- verdict.Defect = polarity.IsDefect(verdict.Direction)
+ verdict.Defect = verdict.Conclusive && polarity.IsDefect(verdict.Direction)
verdict.Reasons = append(
verdict.Reasons,
polarity.Explain(verdict.Direction),
@@ -772,17 +766,49 @@ func staleWindow(observation Observation) time.Duration {
// hypothesis is about that credential, so an absence of findings on it is only
// meaningful if the target actually decided about it.
func requiredStreamThin(observation Observation) (string, bool) {
- if observation.ForgedBaseline == "" {
- return "", false
+ for _, required := range []struct {
+ name string
+ needed bool
+ stream probe.Stream
+ }{
+ {"forged", observation.ForgedBaseline != "", observation.Fault.Forged},
+ {"revoked", observation.Baseline.Revoked.Len() > 0 || observation.Acted, observation.Fault.Revoked},
+ } {
+ if required.needed &&
+ decisionCount(required.stream, observation) < MinimumPaired {
+ return sprintf(
+ "the %s stream produced fewer than %d decisions about its credential",
+ required.name,
+ MinimumPaired,
+ ), true
+ }
}
- // Answered(), not Countable(): a timeout is a transport observation and
- // not a decision about the credential.
- if decided := observation.Fault.Forged.Answered(); decided < MinimumPaired {
- return sprintf(reasonThinForged, decided, MinimumPaired), true
+ return "", false
+}
+
+func decisionCount(stream probe.Stream, observation Observation) int {
+ count := stream.Count(probe.Accepted)
+ for _, disposition := range closedWhen(observation) {
+ count += stream.Count(disposition)
}
+ return count
+}
- return "", false
+func heldEvidence(observation Observation) int {
+ evidence := min(observation.Fault.Valid.Count(probe.Accepted),
+ decisionCount(observation.Fault.Invalid, observation))
+ if observation.ForgedBaseline != "" {
+ evidence = min(
+ evidence, decisionCount(observation.Fault.Forged, observation),
+ )
+ }
+ if observation.Baseline.Revoked.Len() > 0 || observation.Acted {
+ evidence = min(
+ evidence, decisionCount(observation.Fault.Revoked, observation),
+ )
+ }
+ return evidence
}
func forgedAccepted(observation Observation) int {
@@ -904,19 +930,9 @@ func resolveCause(
resumed.Countable(), idle))
}
- // The soundness condition for the causal claim: a fixed, never-re-armed
- // cache whose lifetime sits between the fault window and the pause produces
- // this same observation.
- if steady < idle {
- return LapsedOnPause, append(
- reasons,
- sprintf(reasonLapsedOnPause, idle, steady, idle),
- )
- }
-
- return TTLExtension, append(
+ return LapsedOnPause, append(
reasons,
- sprintf(reasonTTLExtension, steady, idle),
+ sprintf(reasonLapsedOnPause, idle, steady),
)
}
@@ -957,7 +973,7 @@ func baselineUnmet(
// A mid-run revocation needs the target to have been HONOURING the
// credential first, or there is nothing to take away.
- if baseline.Revoked.Countable() > 0 &&
+ if baseline.Revoked.Len() > 0 &&
!baseline.Revoked.Any(probe.Accepted) {
return reasonNothingToRevoke, true
}
@@ -1044,7 +1060,7 @@ func forgedBaselineUnmet(
baseline probe.Window,
expect probe.Expectation,
) (string, bool) {
- if baseline.Forged.Countable() == 0 {
+ if expect == "" {
return "", false
}
@@ -1081,6 +1097,12 @@ func direction(observation Observation) (Direction, []string) {
return Denied, []string{sprintf(reasonDenied, denied)}
}
+ if heldEvidence(observation) < MinimumPaired {
+ return Undetermined, []string{
+ "the fault window did not contain enough credential decisions to establish a hold",
+ }
+ }
+
return Held, []string{reasonHeld}
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/verdict_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/verdict_test.go
index ade4d37b..56f54959 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/verdict_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/experiment/verdict_test.go
@@ -485,7 +485,7 @@ func TestFailingClosedIsDeclaredPerControl(t *testing.T) {
}{
{
name: "nothing declared, so only a refusal counts",
- want: Held,
+ want: Undetermined,
},
{
name: "unavailability declared, so losing access counts",
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/idp/bind_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/idp/bind_test.go
index c6990023..c076b9f6 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/idp/bind_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/idp/bind_test.go
@@ -10,11 +10,6 @@ import (
"github.com/CarterPerez-dev/khaos/internal/bindguard"
)
-// This provider mints credentials that are deliberately defective -- `alg:
-// none`, a signature over a zero-length key, a token carrying its own verifying
-// key -- and serves the key set a target resolves them against. It had no bind
-// guard at all while the fault proxy had carried one since M5, so the rule was
-// a property of one tier rather than of khaos. An audit found it.
func TestAnExposedBindIsRefused(t *testing.T) {
for _, addr := range []string{"0.0.0.0:45999", ":45999", "[::]:45999"} {
provider, err := New(Config{})
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/idp/observe_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/idp/observe_test.go
index 5ca27ace..49c64a33 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/idp/observe_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/idp/observe_test.go
@@ -552,10 +552,6 @@ func TestAnUnpinnedProviderNamesItselfAfterBinding(t *testing.T) {
}
}
-// The key-set fetch count is the measurement research 07 requirement R5 exists
-// for: how often a verifier refetches, which no third-party provider will tell
-// you. khaos's own read-back goes over the same HTTP surface, so without the
-// correction khaos's instrumentation would appear in the number khaos reports.
func TestKhaosOwnConfirmationDoesNotInflateTheFetchCount(t *testing.T) {
for _, kind := range []Kind{JWKSUnreachable, KidRollover, KeyRotation} {
t.Run(string(kind), func(t *testing.T) {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/attachment_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/attachment_test.go
new file mode 100644
index 00000000..23f1c375
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/attachment_test.go
@@ -0,0 +1,63 @@
+// ©AngelaMos | 2026
+// attachment_test.go
+
+package inject
+
+import (
+ "net/netip"
+ "strings"
+ "testing"
+
+ "github.com/moby/moby/api/types/network"
+)
+
+func TestAttachedNetworkMustMatchTheRecordedEndpoint(t *testing.T) {
+ before := attachment{
+ Network: "fixture", Aliases: []string{"api", "worker"},
+ EndpointID: "original", MacAddress: "02:42:ac:16:00:08",
+ Address: netip.MustParseAddr("172.22.0.8"), GwPriority: 5,
+ DriverOpts: map[string]string{"test": "original"},
+ }
+ if before.settings().IPAMConfig != nil {
+ t.Fatal("a dynamically allocated address became an explicit IP request")
+ }
+ original := func() *network.EndpointSettings {
+ current := before.settings()
+ current.EndpointID = before.EndpointID
+ current.IPAddress = before.Address
+ return current
+ }
+ for _, test := range []struct {
+ field string
+ change func(*network.EndpointSettings)
+ }{
+ {"aliases", func(e *network.EndpointSettings) { e.Aliases = []string{"other"} }},
+ {"IPv4 address", func(e *network.EndpointSettings) { e.IPAddress = netip.MustParseAddr("172.22.0.9") }},
+ {"gateway priority", func(e *network.EndpointSettings) { e.GwPriority = 0 }},
+ {"driver options", func(e *network.EndpointSettings) { e.DriverOpts = nil }},
+ {"MAC address", func(e *network.EndpointSettings) { e.MacAddress = network.HardwareAddr{} }},
+ } {
+ t.Run(test.field, func(t *testing.T) {
+ current := original()
+ test.change(current)
+ err := before.verify(current)
+ if err == nil || !strings.Contains(err.Error(), test.field) {
+ t.Fatalf("endpoint drift was not named: %v", err)
+ }
+ })
+ }
+ current := original()
+ current.Aliases = []string{"worker", "api"}
+ if err := before.verify(current); err != nil {
+ t.Fatalf("matching endpoint refused: %v", err)
+ }
+ current.EndpointID = "recreated"
+ if err := current.MacAddress.UnmarshalText(
+ []byte("02:42:ac:16:00:09"),
+ ); err != nil {
+ t.Fatal(err)
+ }
+ if err := before.verify(current); err != nil {
+ t.Fatalf("allocated MAC on a recreated endpoint refused: %v", err)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/credlifecycle.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/credlifecycle.go
new file mode 100644
index 00000000..cb99c018
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/credlifecycle.go
@@ -0,0 +1,105 @@
+// ©AngelaMos | 2026
+// credlifecycle.go
+
+package inject
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+
+ "github.com/CarterPerez-dev/khaos/internal/dbx"
+ "github.com/CarterPerez-dev/khaos/internal/dockerx"
+ "github.com/CarterPerez-dev/khaos/internal/journal"
+)
+
+const KindCredentialTier = "credential-tier-lifetime"
+
+type credentialTier struct{}
+
+func init() {
+ registry[KindCredentialTier] = credentialTier{}
+}
+
+func (credentialTier) Obligation() {}
+
+func (credentialTier) Describe(_ Env, target dockerx.Target) string {
+ return "open the temporary credential tier for " + target.Name
+}
+
+func (credentialTier) Plan(
+ _ context.Context,
+ env Env,
+ _ dockerx.Target,
+) (json.RawMessage, error) {
+ if env.Cred == nil || env.CredPlan.Address == "" {
+ return nil, ErrNoCredRecipe
+ }
+ trap := env.Cred.Trap()
+ if err := trap.Validate(); err != nil {
+ return nil, err
+ }
+ return journal.EncodePayload(credPayload{Trap: trap, Plan: env.CredPlan})
+}
+
+func (credentialTier) Apply(
+ ctx context.Context,
+ env Env,
+ _ dockerx.Target,
+ raw json.RawMessage,
+) error {
+ if env.Cred == nil || env.Docker == nil {
+ return ErrNoCredTier
+ }
+ var payload credPayload
+ if err := json.Unmarshal(raw, &payload); err != nil {
+ return err
+ }
+ admin, err := readSource(ctx, env.Docker, payload.Plan.Admin)
+ if err != nil {
+ return err
+ }
+ var subject dbx.Credential
+ if payload.Plan.Subject != nil {
+ subject, err = readSource(ctx, env.Docker, *payload.Plan.Subject)
+ if err != nil {
+ return err
+ }
+ }
+ return env.Cred.Open(ctx, dbx.Config{
+ Engine: payload.Plan.Engine,
+ Address: payload.Plan.Address,
+ Database: payload.Plan.Database,
+ Admin: admin,
+ }, subject)
+}
+
+func (credentialTier) Confirm(
+ _ context.Context,
+ env Env,
+ _ dockerx.Target,
+) (string, bool, error) {
+ if env.Cred == nil {
+ return "", false, ErrNoCredTier
+ }
+ return "the credential tier has an open administrative connection", env.Cred.Version() != "", nil
+}
+
+func (credentialTier) Revert(
+ ctx context.Context,
+ env Env,
+ record journal.Record,
+) error {
+ var payload credPayload
+ if err := record.DecodePayload(&payload); err != nil {
+ return err
+ }
+ var closeErr error
+ if env.Cred != nil {
+ closeErr = env.Cred.Close(ctx)
+ }
+ if !payload.Trap.Minted {
+ return closeErr
+ }
+ return errors.Join(closeErr, revertFromRecipe(ctx, env, payload))
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/dns_recovery_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/dns_recovery_test.go
new file mode 100644
index 00000000..68c623af
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/dns_recovery_test.go
@@ -0,0 +1,61 @@
+// ©AngelaMos | 2026
+// dns_recovery_test.go
+
+package inject
+
+import (
+ "context"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+
+ "github.com/CarterPerez-dev/khaos/internal/dnsrv"
+ "github.com/CarterPerez-dev/khaos/internal/dockerx"
+ "github.com/CarterPerez-dev/khaos/internal/journal"
+)
+
+func TestDNSRecoveryDistinguishesAbsenceFromDiscoveryFailure(t *testing.T) {
+ for _, failed := range []bool{false, true} {
+ t.Run(fmt.Sprint(failed), func(t *testing.T) {
+ handler := func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("API-Version", "1.56")
+ w.Header().Set("Content-Type", "application/json")
+ switch {
+ case strings.HasSuffix(r.URL.Path, "/containers/json"):
+ if failed {
+ w.WriteHeader(http.StatusInternalServerError)
+ fmt.Fprint(w, `{"message":"daemon unavailable"}`)
+ } else {
+ fmt.Fprint(w, `[]`)
+ }
+ case strings.HasSuffix(r.URL.Path, "/info"):
+ fmt.Fprint(w, `{"ID":"test-daemon"}`)
+ }
+ }
+ server := httptest.NewServer(http.HandlerFunc(handler))
+ defer server.Close()
+ docker, err := dockerx.Connect(
+ context.Background(), dockerx.Endpoint{Host: server.URL},
+ )
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer docker.Close()
+ payload, err := journal.EncodePayload(
+ dnsPayload{Network: "fixture"},
+ )
+ if err != nil {
+ t.Fatal(err)
+ }
+ err = (dnsMutator{kind: dnsrv.NXDomain}).Revert(
+ context.Background(), Env{Docker: docker},
+ journal.Record{Payload: payload},
+ )
+ if (err != nil) != failed {
+ t.Fatalf("discovery failure=%v, recovery error=%v", failed, err)
+ }
+ })
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/dnsfault.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/dnsfault.go
index a5527e65..3b5cbc72 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/dnsfault.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/dnsfault.go
@@ -128,13 +128,15 @@ func (m dnsMutator) Revert(
if client == nil {
var payload dnsPayload
- if len(record.Payload) > 0 {
- if err := record.DecodePayload(&payload); err != nil {
- return err
- }
+ if err := record.DecodePayload(&payload); err != nil {
+ return err
}
- client = reconnectResolver(ctx, env, payload)
+ var err error
+ client, err = reconnectResolver(ctx, env, payload)
+ if err != nil {
+ return err
+ }
}
// Nothing left to disarm is SUCCESS: the resolver named is gone, so it took
@@ -148,21 +150,21 @@ func (m dnsMutator) Revert(
return err
}
-// reconnectResolver returns no error ON PURPOSE: every way it can fail to
-// produce a client means the resolver that held the fault is not there, so the
-// caller treats nil as a revert with nothing left to do.
func reconnectResolver(
ctx context.Context,
env Env,
payload dnsPayload,
-) *dnsrv.Client {
+) (*dnsrv.Client, error) {
if env.Docker == nil || payload.Network == "" {
- return nil
+ return nil, ErrNoResolver
}
resolver, err := env.Docker.FindResolver(ctx, payload.Network)
if err != nil {
- return nil
+ if errors.Is(err, dockerx.ErrResolverNotFound) {
+ return nil, nil
+ }
+ return nil, err
}
return dnsrv.NewClient(
@@ -170,7 +172,7 @@ func reconnectResolver(
dockerx.ResolverControlURL(resolver.Address),
resolver.Nonce,
payload.Control,
- )
+ ), nil
}
// Confirm asks the resolver the same question the target asks, never reading
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/envfault.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/envfault.go
index 98557856..3a19d389 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/envfault.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/envfault.go
@@ -8,6 +8,7 @@ import (
"encoding/json"
"errors"
"fmt"
+ "maps"
"slices"
"strings"
@@ -28,7 +29,8 @@ const (
// required-field check written with the wrong one does not fire.
EnvEmpty EnvKind = "empty"
- KindEnvPrefix = "env-"
+ KindEnvPrefix = "env-"
+ ReplacementLabel = "khaos.replacement"
errNoEnvPlan = "%w: %s needs to be told which variable to take away"
@@ -81,6 +83,7 @@ type EnvPlan struct {
// left for the rollback to guess.
OriginalID string `json:"original_id"`
ReplacedID string `json:"replaced_id,omitempty"`
+ Owner string `json:"owner,omitempty"`
}
// envMutator is the only tier that REPLACES the target. Everything else mutates
@@ -99,7 +102,10 @@ func init() {
}
func (m envMutator) Describe(env Env, target dockerx.Target) string {
- variable := env.EnvPlan.Variable
+ variable := ""
+ if env.EnvPlan != nil {
+ variable = env.EnvPlan.Variable
+ }
if variable == "" {
variable = string(m.kind)
}
@@ -174,6 +180,7 @@ func (m envMutator) Plan(
plan.Held = dockerx.HeldName(spec.Name)
plan.OriginalID = target.ID
+ plan.Owner = journal.NewID()
return journal.EncodePayload(*plan)
}
@@ -198,6 +205,11 @@ func (m envMutator) Apply(
}
environ := m.replacement(spec.Config.Env, plan.Variable)
+ spec.Config.Labels = maps.Clone(spec.Config.Labels)
+ if spec.Config.Labels == nil {
+ spec.Config.Labels = map[string]string{}
+ }
+ spec.Config.Labels[ReplacementLabel] = plan.Owner
// Stopped before the rename, because two containers cannot both hold the
// network's static address and the service's aliases.
@@ -213,10 +225,6 @@ func (m envMutator) Apply(
if err != nil {
// The original is still there under its held name, so the revert can
// put it back whatever happened here.
- if replaced != "" {
- _ = env.Docker.Remove(context.WithoutCancel(ctx), replaced)
- }
-
return err
}
@@ -332,12 +340,10 @@ func (m envMutator) Revert(
plan.ReplacedID = env.EnvPlan.ReplacedID
}
- if env.Docker == nil || plan.Held == "" {
+ if env.Docker == nil || plan.Held == "" || plan.OriginalID == "" {
return fmt.Errorf(errNoEnvPlan, ErrNoEnvPlan, EnvFaultKind(m.kind))
}
- var failures []error
-
original := strings.TrimSuffix(plan.Held, dockerx.HeldSuffix)
// A cold start has only the NAME, and that name is the original's.
@@ -347,39 +353,28 @@ func (m envMutator) Revert(
}
if err := m.removeReplacement(ctx, env, plan, replaced); err != nil {
- failures = append(failures, err)
- }
-
- // A revert that completed and failed only to append REVERTED is replayed,
- // and replay has to be a no-op rather than a second attempt.
- if restored, err := m.alreadyRestored(
- ctx,
- env,
- plan,
- original,
- ); err != nil {
- failures = append(failures, err)
- } else if restored {
- if err := env.Docker.Start(ctx, plan.OriginalID); err != nil {
- failures = append(failures, err)
- }
-
- return errors.Join(failures...)
+ return err
}
- if err := env.Docker.Rename(ctx, plan.Held, original); err != nil {
- return errors.Join(append(failures, err)...)
+ spec, err := env.Docker.Capture(ctx, plan.OriginalID)
+ if err != nil {
+ return err
}
-
- // Started by the name it now has, not by the one it had a line ago. The
- // first version started plan.Held, which the rename had just taken away,
- // so the original came back STOPPED and the next run reported the range as
- // missing the fixture.
- if err := env.Docker.Start(ctx, original); err != nil {
- failures = append(failures, err)
+ if spec.Name != original {
+ if spec.Name != plan.Held {
+ return fmt.Errorf(
+ "original container %s was renamed to %q outside this mutation",
+ plan.OriginalID, spec.Name,
+ )
+ }
+ if err := env.Docker.Rename(
+ ctx, plan.OriginalID, original,
+ ); err != nil {
+ return err
+ }
}
- return errors.Join(failures...)
+ return env.Docker.Start(ctx, plan.OriginalID)
}
// NeedsEnv tells the run this mutation replaces the target, derived from the
@@ -412,11 +407,6 @@ func ValidEnvKind(kind EnvKind) error {
// Prefix declares this tier's kinds to the registry.
func (envMutator) Prefix() string { return KindEnvPrefix }
-// removeReplacement deletes the container khaos created, and NOTHING else.
-// On a cold start the only handle is a NAME, and that name is the original's
-// own, so the id behind it decides: it is resolved first, compared against
-// OriginalID, and the removal is addressed to the id rather than the name,
-// which can move between the two calls.
func (m envMutator) removeReplacement(
ctx context.Context,
env Env,
@@ -433,6 +423,17 @@ func (m envMutator) removeReplacement(
case id == plan.OriginalID:
return nil
}
+ spec, err := env.Docker.Capture(ctx, id)
+ if err != nil {
+ return err
+ }
+ if plan.Owner == "" || spec.Config.Labels[ReplacementLabel] != plan.Owner {
+ return fmt.Errorf(
+ "container %s at %q is not owned by this mutation; refusing removal",
+ id,
+ ref,
+ )
+ }
if err := env.Docker.Remove(ctx, id); err != nil &&
!errors.Is(err, dockerx.ErrContainerGone) {
@@ -441,33 +442,3 @@ func (m envMutator) removeReplacement(
return nil
}
-
-// alreadyRestored reports that the original is back under its own name, by
-// IDENTITY: a replacement still holding that name reads the same otherwise.
-func (m envMutator) alreadyRestored(
- ctx context.Context,
- env Env,
- plan EnvPlan,
- original string,
-) (bool, error) {
- if plan.OriginalID == "" {
- return false, nil
- }
-
- if _, err := env.Docker.Identify(ctx, plan.Held); err == nil {
- return false, nil
- } else if !errors.Is(err, dockerx.ErrContainerGone) {
- return false, err
- }
-
- id, err := env.Docker.Identify(ctx, original)
-
- switch {
- case errors.Is(err, dockerx.ErrContainerGone):
- return false, nil
- case err != nil:
- return false, err
- }
-
- return id == plan.OriginalID, nil
-}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/envfault_recovery_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/envfault_recovery_test.go
index 83a1667c..31abe4b6 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/envfault_recovery_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/envfault_recovery_test.go
@@ -6,6 +6,7 @@ package inject
import (
"context"
"encoding/json"
+ "os"
"os/exec"
"strings"
"testing"
@@ -17,7 +18,6 @@ import (
const (
fixtureImage = "alpine:3.22"
- fixtureName = "khaos-envrevert-original"
)
// dockerOrSkip connects to the daemon, or skips: this package's other tiers
@@ -25,6 +25,9 @@ const (
// without a real container to rename.
func dockerOrSkip(t *testing.T) *dockerx.Client {
t.Helper()
+ if os.Getenv("KHAOS_DOCKER_TESTS") != "1" {
+ t.Skip("set KHAOS_DOCKER_TESTS=1 to run Docker integration tests")
+ }
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
@@ -41,25 +44,55 @@ func dockerOrSkip(t *testing.T) *dockerx.Client {
// returns its immutable id. Created through the CLI rather than through
// dockerx, because dockerx creates only from a captured spec and the point
// here is to own a container khaos did not make.
-func seedOriginal(t *testing.T, name string) string {
+func seedOriginal(t *testing.T, name string, labels ...string) string {
t.Helper()
- _ = exec.Command("docker", "rm", "-f", name).Run()
-
- out, err := exec.Command(
- "docker", "create", "--name", name, fixtureImage, "sleep", "300",
- ).Output()
+ args := []string{"create", "--name", name}
+ for _, label := range labels {
+ args = append(args, "--label", label)
+ }
+ args = append(args, fixtureImage, "sleep", "300")
+ out, err := exec.Command("docker", args...).Output()
if err != nil {
- t.Skipf("could not create the fixture container: %v", err)
+ t.Fatalf("could not create the fixture container: %v", err)
}
id := strings.TrimSpace(string(out))
- t.Cleanup(func() { _ = exec.Command("docker", "rm", "-f", name).Run() })
+ t.Cleanup(func() { _ = exec.Command("docker", "rm", "-f", id).Run() })
return id
}
+func TestRecoveryRefusesAnUnownedContainerReusingTheOriginalName(t *testing.T) {
+ client := dockerOrSkip(t)
+ defer client.Close()
+ name := "khaos-env-unowned-" + strings.ToLower(journal.NewID())
+ id := seedOriginal(t, name)
+ held := dockerx.HeldName(name)
+ if err := exec.Command("docker", "rename", id, held).Run(); err != nil {
+ t.Fatal(err)
+ }
+ unrelated := seedOriginal(t, name)
+ payload, err := journal.EncodePayload(EnvPlan{
+ Kind: EnvUnset, Variable: "SECRET", Held: held, OriginalID: id,
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
+ defer cancel()
+ err = (envMutator{kind: EnvUnset}).Revert(
+ ctx, Env{Docker: client}, journal.Record{Payload: payload},
+ )
+ if err == nil || !alive(unrelated) || !alive(id) {
+ t.Fatalf(
+ "unowned container recovery: error=%v original=%v unrelated=%v",
+ err, alive(id), alive(unrelated),
+ )
+ }
+}
+
func alive(name string) bool {
return exec.Command("docker", "inspect", name).Run() == nil
}
@@ -89,6 +122,7 @@ func TestARevertNeverRemovesTheOriginalContainer(t *testing.T) {
"the revert already ran": {mustSucceed: true},
} {
t.Run(name, func(t *testing.T) {
+ fixtureName := "khaos-envrevert-" + strings.ToLower(journal.NewID())
originalID := seedOriginal(t, fixtureName)
payload, err := json.Marshal(EnvPlan{
@@ -138,6 +172,7 @@ func TestARevertStillRemovesAGenuineReplacement(t *testing.T) {
client := dockerOrSkip(t)
defer func() { _ = client.Close() }()
+ fixtureName := "khaos-envrevert-" + strings.ToLower(journal.NewID())
held := fixtureName + dockerx.HeldSuffix
originalID := seedOriginal(t, fixtureName)
@@ -149,9 +184,8 @@ func TestARevertStillRemovesAGenuineReplacement(t *testing.T) {
t.Skipf("could not rename the fixture aside: %v", err)
}
- t.Cleanup(func() { _ = exec.Command("docker", "rm", "-f", held).Run() })
-
- replacementID := seedOriginal(t, fixtureName)
+ owner := journal.NewID()
+ replacementID := seedOriginal(t, fixtureName, ReplacementLabel+"="+owner)
if replacementID == originalID {
t.Fatal("the fixture did not create a second container")
@@ -162,6 +196,7 @@ func TestARevertStillRemovesAGenuineReplacement(t *testing.T) {
Variable: "A_SECRET",
Held: held,
OriginalID: originalID,
+ Owner: owner,
// ReplacedID deliberately empty: the cold-start shape, which is the
// dangerous path.
})
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/deadlock_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/deadlock_test.go
index a26c3c75..53af101b 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/deadlock_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/deadlock_test.go
@@ -7,9 +7,6 @@ import (
"time"
)
-// A dependency that accepts and never answers. This is what research 06 section
-// 6.1 measured for both `docker pause` and `docker network disconnect`: a silent
-// hang, no RST, no FIN, no error.
func TestCloseDoesNotDeadlockAgainstAWedgedDependency(t *testing.T) {
wedged, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/delivery.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/delivery.go
new file mode 100644
index 00000000..4d1aefb7
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/delivery.go
@@ -0,0 +1,65 @@
+// ©AngelaMos | 2026
+// delivery.go
+
+package proxy
+
+import (
+ "fmt"
+ "net"
+)
+
+func (p *Proxy) WatchSources(addresses []string) error {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ if len(addresses) == 0 {
+ return fmt.Errorf(
+ "the proxy has no target network addresses to observe",
+ )
+ }
+ for _, address := range addresses {
+ ip := net.ParseIP(address)
+ if ip == nil {
+ return fmt.Errorf("invalid target address %q", address)
+ }
+ p.sources[ip.String()] = 0
+ p.faultSources[ip.String()] = 0
+ }
+ return nil
+}
+
+func (p *Proxy) countSource(address net.Addr) {
+ peer, ok := address.(*net.TCPAddr)
+ if !ok {
+ return
+ }
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ key := peer.IP.String()
+ if _, watched := p.sources[key]; !watched {
+ return
+ }
+ p.sources[key]++
+ if p.state.Load().fault.Kind != PassThrough {
+ p.faultSources[key]++
+ }
+}
+
+func (p *Proxy) TargetConnections() (int64, error) {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ var total int64
+ for _, count := range p.sources {
+ total += count
+ }
+ return total, nil
+}
+
+func (p *Proxy) FaultConnections() (int64, error) {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ var total int64
+ for _, count := range p.faultSources {
+ total += count
+ }
+ return total, nil
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/delivery_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/delivery_test.go
new file mode 100644
index 00000000..c5819225
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/delivery_test.go
@@ -0,0 +1,51 @@
+// ©AngelaMos | 2026
+// delivery_test.go
+
+package proxy
+
+import (
+ "io"
+ "net"
+ "testing"
+ "time"
+)
+
+func TestDeliveryCountsOnlyTheWatchedTarget(t *testing.T) {
+ p := through(t)
+ if err := p.WatchSources([]string{"127.0.0.2"}); err != nil {
+ t.Fatal(err)
+ }
+ if err := p.Arm(Fault{Kind: EOF}); err != nil {
+ t.Fatal(err)
+ }
+ _, confirmed, err := p.Observe(t.Context(), EOF)
+ if err != nil || !confirmed {
+ t.Fatalf("confirmation: %v, %v", confirmed, err)
+ }
+ if count, _ := p.FaultConnections(); count != 0 {
+ t.Fatal("confirmation was attributed to the target")
+ }
+ dialer := net.Dialer{
+ LocalAddr: &net.TCPAddr{IP: net.ParseIP("127.0.0.2")},
+ Timeout: time.Second,
+ }
+ conn, err := dialer.DialContext(t.Context(), "tcp", p.Addr())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := conn.SetDeadline(time.Now().Add(time.Second)); err != nil {
+ t.Fatal(err)
+ }
+ _, _ = io.ReadAll(conn)
+ _ = conn.Close()
+ if count, _ := p.FaultConnections(); count != 1 {
+ t.Fatalf("target count = %d, want 1", count)
+ }
+ p.Disarm()
+ if count, _ := p.FaultConnections(); count != 0 {
+ t.Fatal("fault delivery survived disarm")
+ }
+ if count, _ := p.TargetConnections(); count != 1 {
+ t.Fatal("baseline counter was reset with the fault")
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/eof_stream_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/eof_stream_test.go
new file mode 100644
index 00000000..42192a74
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/eof_stream_test.go
@@ -0,0 +1,39 @@
+// ©AngelaMos | 2026
+// eof_stream_test.go
+
+package proxy
+
+import (
+ "errors"
+ "io"
+ "net"
+ "testing"
+ "time"
+)
+
+func TestEOFAcceptsWritesBeyondTheOldDrainWindow(t *testing.T) {
+ p := through(t)
+ if err := p.Arm(Fault{Kind: EOF}); err != nil {
+ t.Fatal(err)
+ }
+ conn, err := net.DialTimeout("tcp", p.Addr(), time.Second)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer conn.Close()
+ if err := conn.SetDeadline(time.Now().Add(2 * time.Second)); err != nil {
+ t.Fatal(err)
+ }
+ for range 4 {
+ if _, err := io.WriteString(
+ conn, "request body continues\n",
+ ); err != nil {
+ t.Fatalf("clean EOF reset a continuing request: %v", err)
+ }
+ time.Sleep(150 * time.Millisecond)
+ }
+ var body [1]byte
+ if _, err := conn.Read(body[:]); !errors.Is(err, io.EOF) {
+ t.Fatalf("clean close = %v, want EOF", err)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/proxy.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/proxy.go
index 5e141470..4dbd729f 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/proxy.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/proxy.go
@@ -72,8 +72,10 @@ type Proxy struct {
probe string
// Both ends are held; closing the client cannot unblock the dependency.
- mu sync.Mutex
- held map[net.Conn]struct{}
+ mu sync.Mutex
+ held map[net.Conn]struct{}
+ sources map[string]int64
+ faultSources map[string]int64
// inert forwards normally with a fault armed: a negative control.
inert atomic.Bool
@@ -105,11 +107,13 @@ func Open(listenAddr, upstream, probe string) (*Proxy, error) {
}
p := &Proxy{
- upstream: upstream,
- listener: listener,
- probe: probe,
- held: map[net.Conn]struct{}{},
- closed: make(chan struct{}),
+ upstream: upstream,
+ listener: listener,
+ probe: probe,
+ held: map[net.Conn]struct{}{},
+ closed: make(chan struct{}),
+ sources: map[string]int64{},
+ faultSources: map[string]int64{},
}
p.state.Store(&armed{fault: Fault{Kind: PassThrough}})
@@ -155,6 +159,9 @@ func (p *Proxy) swap(fault Fault) {
previous := p.state.Load()
p.state.Store(&armed{fault: fault, generation: previous.generation + 1})
+ for address := range p.faultSources {
+ p.faultSources[address] = 0
+ }
conns := make([]net.Conn, 0, len(p.held))
for conn := range p.held {
@@ -220,6 +227,7 @@ func (p *Proxy) serve() {
}
p.accepted.Add(1)
+ p.countSource(conn.RemoteAddr())
p.wg.Add(1)
go func() {
@@ -279,11 +287,7 @@ func (p *Proxy) sever(client net.Conn, fault Fault, generation uint64) bool {
p.severed.Add(1)
return true
case EOF:
- // The kernel sends FIN only on an empty receive queue, RST otherwise.
- drain(client)
- _ = client.Close()
- p.severed.Add(1)
- return true
+ return p.eof(client, generation)
default:
return p.hold(client, generation)
}
@@ -301,19 +305,21 @@ func settle(client net.Conn) {
_, _ = client.Read(buffer)
}
-// drain runs to the deadline: a short read is not an empty queue.
-func drain(client net.Conn) {
- if err := client.SetReadDeadline(time.Now().Add(drainWindow)); err != nil {
- return
+func (p *Proxy) eof(client net.Conn, generation uint64) bool {
+ if !p.track(generation, client) {
+ return false
}
-
- buffer := make([]byte, copyBuffer)
-
- for {
- if _, err := client.Read(buffer); err != nil {
- return
+ defer p.untrack(client)
+ p.severed.Add(1)
+ if tcp, ok := client.(*net.TCPConn); ok {
+ if err := tcp.CloseWrite(); err != nil {
+ return true
}
}
+ if err := client.SetReadDeadline(time.Now().Add(hangCeiling)); err == nil {
+ _, _ = io.Copy(io.Discard, client)
+ }
+ return true
}
// hold implements Hang: keep the connection open and answer nothing.
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/proxy_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/proxy_test.go
index 97a02008..26603458 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/proxy_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxy/proxy_test.go
@@ -554,8 +554,6 @@ func TestSeveringAndForwardingFaultsAreSeparated(t *testing.T) {
}
}
-// Audit findings, reproduced as controls before being fixed.
-
func TestSlowCloseHoldsTheConnectionOpenAfterTheAnswer(t *testing.T) {
const delay = 600 * time.Millisecond
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxyfault_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxyfault_test.go
index 5142187c..13bbda1d 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxyfault_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/proxyfault_test.go
@@ -279,14 +279,6 @@ func TestRevertingWithoutAProxyIsResolvedRatherThanFailed(t *testing.T) {
// that something is excluded lets one tier's exclusion alibi the other's: this
// test passed the whole time the identity tier was unreachable from DirectKinds
// by accident rather than by the registry knowing about it.
-// The prefix list is now DERIVED from the registry rather than written here,
-// and that change has a defect behind it. The hand-written list named the proxy
-// and identity tiers; M7 added the certificate authority and extended neither
-// the type switch nor this list, so `khaos inject` advertised thirty-six
-// `pki-*` kinds that each failed with "needs khaos's certificate authority, and
-// this run did not start one" — and this test stayed green for the whole
-// milestone. A per-prefix assertion over a list nobody maintains is an
-// aggregate assertion wearing better clothes.
func TestInProcessKindsAreNotOfferedForDirectInjection(t *testing.T) {
prefixes := PerRunPrefixes()
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/registry.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/registry.go
index 7ed8f285..12d2c6c2 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/registry.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/registry.go
@@ -8,6 +8,9 @@ import (
"encoding/json"
"errors"
"fmt"
+ "maps"
+ "net/netip"
+ "reflect"
"slices"
"strings"
@@ -37,7 +40,7 @@ const (
describePause = "freeze every process in %s so it stops answering without closing its connections"
describeRestart = "stop %s and start it again, so it initialises without a dependency it already had"
confirmRestart = "%s is stopped, read back off the daemon"
- describeDisconnect = "detach %s from every network it is on, then reattach it with the same aliases, addresses and driver options"
+ describeDisconnect = "detach %s from every network, then restore its addresses, aliases, links, gateway priority and driver options; Docker may allocate a new MAC when recreating an endpoint"
)
var ErrAlreadyPaused = errors.New("that container is already paused")
@@ -318,12 +321,15 @@ func (pauseMutator) Revert(
type attachment struct {
Network string `json:"network"`
+ EndpointID string `json:"endpoint_id,omitempty"`
IPAMConfig *network.EndpointIPAMConfig `json:"ipam_config,omitempty"`
Aliases []string `json:"aliases,omitempty"`
Links []string `json:"links,omitempty"`
DriverOpts map[string]string `json:"driver_opts,omitempty"`
MacAddress string `json:"mac_address,omitempty"`
GwPriority int `json:"gw_priority,omitempty"`
+ Address netip.Addr `json:"address,omitzero"`
+ IPv6 netip.Addr `json:"ipv6,omitzero"`
}
type disconnectPayload struct {
@@ -338,7 +344,6 @@ func (a attachment) settings() *network.EndpointSettings {
DriverOpts: a.DriverOpts,
GwPriority: a.GwPriority,
}
-
if a.MacAddress != "" {
var mac network.HardwareAddr
if err := mac.UnmarshalText([]byte(a.MacAddress)); err == nil {
@@ -380,11 +385,18 @@ func (disconnectMutator) Plan(
settings := endpoints[name]
entry := attachment{Network: name}
if settings != nil {
+ entry.EndpointID = settings.EndpointID
entry.IPAMConfig = settings.IPAMConfig
entry.Aliases = settings.Aliases
entry.Links = settings.Links
entry.DriverOpts = settings.DriverOpts
entry.GwPriority = settings.GwPriority
+ if settings.IPAddress.IsValid() {
+ entry.Address = settings.IPAddress
+ }
+ if settings.GlobalIPv6Address.IsValid() {
+ entry.IPv6 = settings.GlobalIPv6Address
+ }
if mac := settings.MacAddress.String(); mac != "" {
entry.MacAddress = mac
}
@@ -435,7 +447,8 @@ func (disconnectMutator) Revert(
var failures []error
for _, entry := range payload.Attachments {
- if _, attached := endpoints[entry.Network]; attached {
+ if current, attached := endpoints[entry.Network]; attached {
+ failures = append(failures, entry.verify(current))
continue
}
if err := env.Docker.Connect(
@@ -445,12 +458,62 @@ func (disconnectMutator) Revert(
entry.settings(),
); err != nil {
failures = append(failures, err)
+ continue
}
+ current, err := env.Docker.Endpoints(ctx, record.TargetID)
+ if err != nil {
+ failures = append(failures, err)
+ continue
+ }
+ failures = append(failures, entry.verify(current[entry.Network]))
}
return errors.Join(failures...)
}
+func (a attachment) verify(current *network.EndpointSettings) error {
+ if current == nil {
+ return fmt.Errorf("network %s has no restored endpoint", a.Network)
+ }
+ fields := map[string]bool{
+ "aliases": sameNames(a.Aliases, current.Aliases),
+ "links": sameNames(a.Links, current.Links),
+ "driver options": maps.Equal(a.DriverOpts, current.DriverOpts),
+ "gateway priority": a.GwPriority == current.GwPriority,
+ "MAC address": a.MacAddress == "" || a.recreated(current) ||
+ strings.EqualFold(a.MacAddress, current.MacAddress.String()),
+ "IPv4 address": !a.Address.IsValid() ||
+ a.Address == current.IPAddress,
+ "IPv6 address": !a.IPv6.IsValid() ||
+ a.IPv6 == current.GlobalIPv6Address,
+ "IPAM": a.IPAMConfig == nil ||
+ reflect.DeepEqual(a.IPAMConfig, current.IPAMConfig),
+ }
+ var failures []error
+ for field, matches := range fields {
+ if !matches {
+ failures = append(failures, fmt.Errorf(
+ "network %s restored with different %s; reconcile the endpoint before reclaiming",
+ a.Network,
+ field,
+ ))
+ }
+ }
+ return errors.Join(failures...)
+}
+
+func (a attachment) recreated(current *network.EndpointSettings) bool {
+ return a.EndpointID != "" && current.EndpointID != "" &&
+ a.EndpointID != current.EndpointID && current.MacAddress.String() != ""
+}
+
+func sameNames(a, b []string) bool {
+ a, b = slices.Clone(a), slices.Clone(b)
+ slices.Sort(a)
+ slices.Sort(b)
+ return slices.Equal(slices.Compact(a), slices.Compact(b))
+}
+
func (pauseMutator) Confirm(
ctx context.Context,
env Env,
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/registry_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/registry_test.go
index d0b87016..8e17ab4c 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/registry_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/registry_test.go
@@ -6,8 +6,21 @@ package inject
import (
"slices"
"testing"
+
+ "github.com/CarterPerez-dev/khaos/internal/dockerx"
)
+func TestEveryMutatorCanDescribeWithoutAPlan(t *testing.T) {
+ for kind, mutator := range registry {
+ t.Run(kind, func(t *testing.T) {
+ got := mutator.Describe(Env{}, dockerx.Target{Name: "target"})
+ if got == "" {
+ t.Fatal("mutator has no description before planning")
+ }
+ })
+ }
+}
+
// An obligation kind is khaos's own bookkeeping: a reclaim has to be able to
// resolve it, and `khaos inject` must never offer it. Both halves matter --
// unresolvable means a target stays down after a failed restart, and offered
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/secretfault.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/secretfault.go
index 3507265d..66fb2235 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/secretfault.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/inject/secretfault.go
@@ -21,7 +21,7 @@ const (
errMissingOnMountPoint = "%w: %s IS the mount point rather than a file inside one, and a mount point cannot be renamed, so khaos cannot make it vanish. The in-place faults still work here; a secret delivered as a whole directory can also be made to vanish"
- describeSecret = "%s, performed in place by a short-lived container of khaos's own that mounts only %s, holds no capabilities and joins no network"
+ describeSecret = "%s, performed in place by a short-lived container of khaos's own that mounts only %s, drops all capabilities except DAC_OVERRIDE for reading and writing the secret and FOWNER for restoring its permissions, and joins no network"
confirmEmpty = "the target's own copy of %s is now zero bytes, and os.ReadFile returns no error on one"
confirmTruncated = "the target's own copy of %s is %d bytes where it was %d"
@@ -164,6 +164,9 @@ func (m secretMutator) Plan(
// Written back through the pointer as well as journaled: the confirmation
// is a DIFFERENCE and has to know what the target had.
plan.Before = stat.Size
+ mode := stat.Mode & modeBits
+ plan.Mutation.OriginalMode = &mode
+ plan.Mutation.OriginalSize = &stat.Size
return journal.EncodePayload(*plan)
}
@@ -309,6 +312,16 @@ func mutateArgs(plan SecretPlan, restore bool) []string {
fmt.Sprintf(bytesFormat, plan.Mutation.Bytes),
)
}
+ if plan.Mutation.OriginalMode != nil {
+ args = append(
+ args, "--original-mode", fmt.Sprint(*plan.Mutation.OriginalMode),
+ )
+ }
+ if plan.Mutation.OriginalSize != nil {
+ args = append(
+ args, "--original-size", fmt.Sprint(*plan.Mutation.OriginalSize),
+ )
+ }
if restore {
args = append(args, applyBack)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/journal/compaction_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/journal/compaction_test.go
new file mode 100644
index 00000000..7d509861
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/journal/compaction_test.go
@@ -0,0 +1,33 @@
+// ©AngelaMos | 2026
+// compaction_test.go
+
+package journal
+
+import (
+ "path/filepath"
+ "testing"
+)
+
+func TestCompactionPreservesOpenWriters(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "journal.jsonl")
+ writer, err := Open(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer writer.Close()
+ record := Record{MutationID: "active", Phase: PhaseApplied}
+ if err := writer.Append(record); err != nil {
+ t.Fatal(err)
+ }
+ if err := Compact(path); err != nil {
+ t.Fatal(err)
+ }
+ record.Phase = PhaseReverted
+ if err := writer.Append(record); err != nil {
+ t.Fatal(err)
+ }
+ live, err := Outstanding(path)
+ if err != nil || len(live) != 0 {
+ t.Fatalf("reversion was lost after compaction: %v, %v", live, err)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/journal/journal.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/journal/journal.go
index 6ca08f30..84402647 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/journal/journal.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/journal/journal.go
@@ -5,13 +5,17 @@ package journal
import (
"bufio"
+ "context"
"crypto/rand"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
+ "sync"
"time"
+
+ "github.com/CarterPerez-dev/khaos/internal/lock"
)
type Phase string
@@ -109,6 +113,7 @@ func (r Record) Outstanding() bool {
}
type Journal struct {
+ mu sync.Mutex
path string
file *os.File
}
@@ -136,6 +141,18 @@ func (j *Journal) Path() string {
}
func (j *Journal) Append(record Record) error {
+ j.mu.Lock()
+ defer j.mu.Unlock()
+ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+ defer cancel()
+ held, err := lock.AcquireContext(ctx, j.path+".write")
+ if err != nil {
+ return fmt.Errorf(errAppend, err)
+ }
+ defer func() { _ = held.Release() }()
+ if err := j.refresh(); err != nil {
+ return err
+ }
if record.At.IsZero() {
record.At = time.Now().UTC()
}
@@ -153,10 +170,47 @@ func (j *Journal) Append(record Record) error {
}
func (j *Journal) Close() error {
+ j.mu.Lock()
+ defer j.mu.Unlock()
return j.file.Close()
}
+func (j *Journal) refresh() error {
+ opened, err := j.file.Stat()
+ if err != nil {
+ return err
+ }
+ current, err := os.Stat(j.path)
+ if err == nil && os.SameFile(opened, current) {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ reopened, err := Open(j.path)
+ if err != nil {
+ return err
+ }
+ _ = j.file.Close()
+ j.file = reopened.file
+ return nil
+}
+
func Replay(path string) ([]Record, error) {
+ if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) {
+ return nil, nil
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+ defer cancel()
+ held, err := lock.AcquireContext(ctx, path+".write")
+ if err != nil {
+ return nil, err
+ }
+ defer func() { _ = held.Release() }()
+ return replay(path)
+}
+
+func replay(path string) ([]Record, error) {
//nolint:gosec // khaos's own journal, from statedir.Paths
file, err := os.Open(path)
if errors.Is(err, os.ErrNotExist) {
@@ -201,7 +255,10 @@ func Replay(path string) ([]Record, error) {
func Outstanding(path string) ([]Record, error) {
records, damage := Replay(path)
+ return outstanding(records), damage
+}
+func outstanding(records []Record) []Record {
latest := make(map[string]Record, len(records))
order := make([]string, 0, len(records))
settled := make(map[string]bool, len(records))
@@ -219,9 +276,9 @@ func Outstanding(path string) ([]Record, error) {
switch record.Phase {
case PhaseReverted:
settled[record.MutationID] = true
- case PhaseApplied:
+ case PhaseApplied, PhaseFailed:
delete(settled, record.MutationID)
- case PhasePending, PhaseFailed, PhaseRenewed:
+ case PhasePending, PhaseRenewed:
}
latest[record.MutationID] = record
@@ -237,14 +294,22 @@ func Outstanding(path string) ([]Record, error) {
}
}
- return out, damage
+ return out
}
func Compact(path string) error {
- outstanding, err := Outstanding(path)
+ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+ defer cancel()
+ held, err := lock.AcquireContext(ctx, path+".write")
+ if err != nil {
+ return err
+ }
+ defer func() { _ = held.Release() }()
+ records, err := replay(path)
if err != nil {
return err
}
+ live := outstanding(records)
temp := path + tempSuffix
@@ -255,7 +320,7 @@ func Compact(path string) error {
}
encoder := json.NewEncoder(file)
- for _, record := range outstanding {
+ for _, record := range live {
if err := encoder.Encode(record); err != nil {
_ = file.Close()
_ = os.Remove(temp)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/lock/lock_unix.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/lock/lock_unix.go
index 31e73b33..fd9553c3 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/lock/lock_unix.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/lock/lock_unix.go
@@ -4,12 +4,14 @@
package lock
import (
+ "context"
"errors"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
+ "time"
"golang.org/x/sys/unix"
)
@@ -42,6 +44,23 @@ type Lock struct {
file *os.File
}
+func AcquireContext(ctx context.Context, path string) (*Lock, error) {
+ for {
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
+ held, err := Acquire(path)
+ if !errors.Is(err, ErrHeld) {
+ return held, err
+ }
+ select {
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ case <-time.After(20 * time.Millisecond):
+ }
+ }
+}
+
func Acquire(path string) (*Lock, error) {
//nolint:gosec // khaos's own state dir, from statedir.Paths
if err := os.MkdirAll(filepath.Dir(path), dirMode); err != nil {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/bind_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/bind_test.go
index 2492d4a3..8ca4602c 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/bind_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/bind_test.go
@@ -10,12 +10,6 @@ import (
"github.com/CarterPerez-dev/khaos/internal/bindguard"
)
-// BOTH listeners, because they are different exposures and each had none. The
-// TLS one presents certificates khaos deliberately broke -- expired, revoked,
-// wrongly named, signed with a weak key -- and the HTTP one serves the CRL and
-// OCSP responses that decide whether a certificate is still good. The fault
-// proxy had carried this guard since M5 and neither of these did, so the rule
-// was a property of one tier rather than of khaos. An audit found it.
func TestBothListenersRefuseAnExposedBind(t *testing.T) {
for _, addr := range []string{"0.0.0.0:45999", ":45999", "[::]:45999"} {
t.Run(addr, func(t *testing.T) {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/revocation.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/revocation.go
index 2765e844..35237abd 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/revocation.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/revocation.go
@@ -35,6 +35,17 @@ func (a *Authority) buildRevocation(
host string,
now time.Time,
) (*Chain, error) {
+ if trap.Kind == RevokedViaCRL || trap.Kind == OCSPRevoked {
+ a.mu.RLock()
+ if a.armed != nil && (a.trap.Kind == "" || a.trap.Kind == PassThrough ||
+ a.trap.Kind == RevokedViaCRL || a.trap.Kind == OCSPRevoked) {
+ chain := *a.armed
+ a.mu.RUnlock()
+ return &chain, nil
+ }
+ a.mu.RUnlock()
+ }
+
template, err := a.leaf(host, now, DefaultLeafDays)
if err != nil {
return nil, err
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/verify_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/verify_test.go
index c287f046..b86ece51 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/verify_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/pki/verify_test.go
@@ -67,9 +67,6 @@ func classify(err error) (outcome, x509.InvalidReason) {
return -1, 0
}
-// expectation is one row of research 04's fault table, restated as something
-// executable. detail is a substring of CertificateInvalidError.Detail, needed
-// only where two traps share a reason code and nothing else separates them.
type expectation struct {
kind Kind
want outcome
@@ -123,17 +120,6 @@ func faultTable() []expectation {
// itself the finding: a caller that does not unwrap blames the CA.
{SHA1Signature, authority, 0, "", ""},
- // BOTH EKU faults are Reason 4, and research 04's F15 row saying the
- // CA-level one is Reason 9 is wrong for go1.26.5. Measured four ways
- // (CA clientAuth or codeSigning, leaf serverAuth or empty) and all four
- // returned 4. Traced to the source: CANotAuthorizedForExtKeyUsage
- // appears in crypto/x509 ONLY in its own declaration and in the Error()
- // string switch, and is never constructed anywhere, so Go cannot return
- // it. The constant is formattable dead code.
- //
- // This makes the two EKU faults a SECOND pair of byte-identical
- // outcomes, alongside the three authority ones, and the family treats
- // them the same way: distinguished by what khaos knows it served.
{EKUClientAuthOnly, invalid, x509.IncompatibleUsage, "", ""},
{EKUConstrainedIntermediate, invalid, x509.IncompatibleUsage, "", ""},
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/build_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/build_test.go
new file mode 100644
index 00000000..8af3abe1
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/build_test.go
@@ -0,0 +1,119 @@
+// ©AngelaMos | 2026
+// build_test.go
+
+package rangelab
+
+import (
+ "io"
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "reflect"
+ "strings"
+ "testing"
+
+ "gopkg.in/yaml.v3"
+)
+
+func TestRangeBuildMustSucceedBeforeCreatingServices(t *testing.T) {
+ for _, fail := range []bool{false, true} {
+ name := "successful build"
+ if fail {
+ name = "failed build"
+ }
+ t.Run(name, func(t *testing.T) {
+ dir := t.TempDir()
+ calls := filepath.Join(dir, "commands")
+ binary := `#!/bin/sh
+printf '%s\n' "$*" >> "$KHAOS_TEST_COMMANDS"
+case " $* " in
+ *" build "*)
+ if [ "$KHAOS_TEST_BUILD_FAIL" = 1 ]; then
+ printf '%s\n' 'fixture build failure' >&2
+ exit 37
+ fi
+ ;;
+esac
+`
+ if err := os.WriteFile(
+ filepath.Join(dir, dockerBinary), []byte(binary), 0o700,
+ ); err != nil {
+ t.Fatal(err)
+ }
+ t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
+ t.Setenv("DOCKER_HOST", "unix:///unused-test-daemon.sock")
+ t.Setenv("KHAOS_TEST_COMMANDS", calls)
+ buildFail := "0"
+ if fail {
+ buildFail = "1"
+ }
+ t.Setenv("KHAOS_TEST_BUILD_FAIL", buildFail)
+
+ server := httptest.NewServer(http.HandlerFunc(
+ func(w http.ResponseWriter, _ *http.Request) {
+ w.WriteHeader(http.StatusOK)
+ },
+ ))
+ defer server.Close()
+ address, ok := server.Listener.Addr().(*net.TCPAddr)
+ if !ok {
+ t.Fatal("fixture health listener is not TCP")
+ }
+ lab := Lab{Dir: dir, Port: address.Port}
+ err := lab.Up(t.Context(), io.Discard, "closed", "2s",
+ "", "", "", "", "", false)
+ if fail {
+ if err == nil ||
+ !strings.Contains(err.Error(), "fixture build failure") {
+ t.Fatalf("build failure was lost: %v", err)
+ }
+ } else if err != nil {
+ t.Fatal(err)
+ }
+ data, err := os.ReadFile(calls)
+ if err != nil {
+ t.Fatal(err)
+ }
+ prefix := "compose -p " + ProjectName + " -f " +
+ filepath.Join(dir, composeFile) + " "
+ want := prefix + "build idp\n"
+ if !fail {
+ want += prefix + "up --no-build --detach --wait\n"
+ }
+ if string(data) != want {
+ t.Fatalf("commands = %q, want %q", data, want)
+ }
+ })
+ }
+}
+
+func TestEveryBuiltRangeServiceUsesTheSharedImage(t *testing.T) {
+ data, err := os.ReadFile(filepath.Join("..", "..", "range", composeFile))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var document struct {
+ Services map[string]struct {
+ Image string `yaml:"image"`
+ Build map[string]any `yaml:"build"`
+ } `yaml:"services"`
+ }
+ if err := yaml.Unmarshal(data, &document); err != nil {
+ t.Fatal(err)
+ }
+ shared, ok := document.Services["idp"]
+ if !ok || shared.Image == "" || len(shared.Build) == 0 {
+ t.Fatal("range has no shared image build target")
+ }
+ for name, service := range document.Services {
+ if len(service.Build) == 0 {
+ continue
+ }
+ if service.Image != shared.Image ||
+ !reflect.DeepEqual(service.Build, shared.Build) {
+ t.Fatalf("%s needs a separate build before range startup", name)
+ }
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/bundle_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/bundle_test.go
new file mode 100644
index 00000000..b34b9b53
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/bundle_test.go
@@ -0,0 +1,37 @@
+// ©AngelaMos | 2026
+// bundle_test.go
+
+package rangelab
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+func TestLocateExtractsRangeOutsideACheckout(t *testing.T) {
+ working := t.TempDir()
+ cache := t.TempDir()
+ t.Chdir(working)
+ t.Setenv(EnvComposeDir, "")
+ t.Setenv("XDG_CACHE_HOME", cache)
+ lab, err := Locate(Ports{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.HasPrefix(lab.Dir, cache+string(os.PathSeparator)) {
+ t.Fatalf("range did not use the private cache: %s", lab.Dir)
+ }
+ if err := VerifyOwned(filepath.Join(lab.Dir, composeFile)); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestExplicitForeignRangeNeverFallsBackToBundle(t *testing.T) {
+ t.Chdir(t.TempDir())
+ t.Setenv(EnvComposeDir, filepath.Dir(writeCompose(t, "name: foreign\n")))
+ if _, err := Locate(Ports{}); err == nil {
+ t.Fatal("a foreign explicit range was silently replaced")
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/discovery_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/discovery_test.go
index 1b831053..b89e0c84 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/discovery_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/discovery_test.go
@@ -152,6 +152,41 @@ func TestDiscoverFindsAllThreeTiers(t *testing.T) {
}
}
+func TestDiscoveryDoesNotDependOnEarlierOptionalTiers(t *testing.T) {
+ for _, missing := range []string{ProxiedGatewayName, VerifierGatewayName} {
+ t.Run(missing, func(t *testing.T) {
+ reader := fullRange()
+ var targets []dockerx.Target
+ for _, target := range reader.targets {
+ if target.Name != missing {
+ targets = append(targets, target)
+ }
+ }
+ targets = append(targets,
+ container("secret-id", SecretGatewayName),
+ container("control-id", ControlGatewayName))
+ reader.targets = targets
+ reader.ports["secret-id"] = 42001
+ reader.ports["control-id"] = 42002
+ reader.environ["control-id"] = map[string]string{
+ EnvLimitStore: "redis:6379",
+ }
+ lab, err := Discover(context.Background(), reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if lab.SecretPort != 42001 || lab.ControlPort != 42002 ||
+ lab.ControlStoreAddr != "redis:6379" {
+ t.Fatalf("independent tiers missing: %+v", lab)
+ }
+ if missing == ProxiedGatewayName &&
+ lab.VerifierPort != verifierPort {
+ t.Fatal("verifier hidden by missing proxy")
+ }
+ })
+ }
+}
+
// Each tier is probed at its own URL, and they must all differ. Pointing an
// experiment at the wrong one measures a service with no causal connection to
// its fault and reports a clean survival over an armed fault.
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/freeport_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/freeport_test.go
index d23d47d7..8344475e 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/freeport_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/freeport_test.go
@@ -8,9 +8,6 @@ import (
"testing"
)
-// Testing loopback alone was weaker than it read. The published container ports
-// go to 127.0.0.1, but four tiers bind the Docker bridge gateway, and a port
-// free on one is not evidence it is free on the other. An audit found it.
func TestAPortTakenOnAnyRequiredAddressIsNotFree(t *testing.T) {
// A second loopback address stands in for the bridge gateway: it is a
// different address on the same host, which is the whole property under
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/rangelab.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/rangelab.go
index a1336762..af9a7406 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/rangelab.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/rangelab/rangelab.go
@@ -21,6 +21,7 @@ import (
"strings"
"time"
+ "github.com/CarterPerez-dev/khaos"
"github.com/CarterPerez-dev/khaos/internal/dbx"
"github.com/CarterPerez-dev/khaos/internal/dockerx"
)
@@ -140,14 +141,6 @@ const (
EnvControlSecret = "KHAOS_RANGE_CONTROL_SECRET"
EnvAuthzMode = "KHAOS_RANGE_AUTHZ_MODE"
- // AuthzDistinguishes reads the dependency's three answers as three.
- // AuthzCollapses is research 07 section 6.2's shape: anything that is not
- // a 200 is read as an absence of an answer, and allowed. AuthzRetries
- // reaches the right verdict the wrong way, asking again every time it does
- // not like an answer, which is that section's retry storm.
- // AuthzCached is correct too, and it is the input khaos's own verdict
- // machinery was wrong about: a one-second decision cache plus a refusal
- // expressed as 503 rather than 403.
AuthzDistinguishes = "distinguishes"
AuthzCollapses = "collapses"
AuthzRetries = "retries"
@@ -180,9 +173,6 @@ const (
// khaos never reads the file: it asks the fixture for a credential.
SecretPath = "/run/khaos-secrets/hmac"
- // SecretValidated checks the secret's length after every read, which is
- // what research 07 part 8 prescribes. SecretUnchecked is the
- // CVE-2019-20933 shape: the empty value flows straight into the crypto.
SecretValidated = "validated"
SecretUnchecked = "unchecked"
@@ -232,7 +222,6 @@ const (
GovernedPath = "/governed"
FilteredPath = "/filtered"
- errNoCompose = "cannot find the range compose file; run khaos from the project directory, or set %s to the directory holding khaos's own range/compose.yml"
errUnreadable = "%w: %s could not be read: %v"
errForeign = "%w: %s declares the compose project %s, and khaos only ever acts on %s"
@@ -397,12 +386,6 @@ type Ports struct {
// namespace, which is why that tier needs no capability.
DNS int
- // The M9 fixtures. These have to be CHOSEN here like every other host
- // port: leaving them unset published `127.0.0.1:0:8080`, which hands the
- // choice to the kernel's ephemeral range. khaos then neither collision
- // checks them nor keeps them stable - and the environment faults recreate
- // the container, so the port moved under khaos's own probe mid-run and the
- // valid stream reported the fixture as unreachable.
DBGateway int
Secret int
Env int
@@ -500,7 +483,7 @@ func composePath() (string, error) {
}
}
- return "", fmt.Errorf(errNoCompose, EnvComposeDir)
+ return khaos.RangeDirectory()
}
// VerifyOwned refuses any compose file that is not khaos's own, or pointing
@@ -649,7 +632,11 @@ func (l Lab) Up(
EnvReresolve+"="+reresolve,
)
- args := []string{"up", "--build", "--detach", "--wait"}
+ if err := l.compose(ctx, out, environ, "build", "idp"); err != nil {
+ return err
+ }
+
+ args := []string{"up", "--no-build", "--detach", "--wait"}
if withDB {
args = append([]string{profileFlag, dbProfile}, args...)
}
@@ -771,6 +758,10 @@ func (l Lab) compose(
) error {
// Passed explicitly as well as declared, so no edit to the file can point
// these commands at another project.
+ resolvedEnv, err := dockerx.ComposeEnvironment(environ)
+ if err != nil {
+ return err
+ }
full := append([]string{
"compose",
projectFlag, ProjectName,
@@ -782,7 +773,7 @@ func (l Lab) compose(
//nolint:gosec // khaos driving its own compose file as the operator
cmd := exec.CommandContext(ctx, dockerBinary, full...)
cmd.Dir = l.Dir
- cmd.Env = environ
+ cmd.Env = resolvedEnv
cmd.Stdout = out
cmd.Stderr = io.MultiWriter(out, &stderr)
@@ -921,71 +912,76 @@ func Discover(ctx context.Context, reader PortReader) (Lab, error) {
lab := Lab{Port: port}
- // Each further gateway is discovered separately and its absence is not
- // fatal, because the experiments that do not need it must still work.
+ if err := lab.readProxyTier(ctx, reader, running); err != nil {
+ return Lab{}, err
+ }
+ if err := lab.readIdentityTier(ctx, reader, running); err != nil {
+ return Lab{}, err
+ }
+ if _, err := lab.readTLSTier(ctx, reader, running); err != nil {
+ return Lab{}, err
+ }
+ if err := lab.readDNSTier(ctx, reader, running); err != nil {
+ return Lab{}, err
+ }
+ lab.readStatefulTier(running)
+ if err := lab.readCredentialGateways(ctx, reader, running); err != nil {
+ return Lab{}, err
+ }
+ return lab, nil
+}
+
+func (l *Lab) readProxyTier(
+ ctx context.Context,
+ reader PortReader,
+ running map[string]dockerx.Target,
+) error {
proxied, ok := running[ProxiedGatewayName]
if !ok {
- return lab, nil
+ return nil
}
- if lab.ProxiedPort, err = reader.PublishedPort(
+ var err error
+ if l.ProxiedPort, err = reader.PublishedPort(
ctx,
proxied.ID,
GatewayInternalPort,
); err != nil {
- return Lab{}, err
+ return err
}
- if lab.ProxyPort, err = proxyPortOf(ctx, reader, proxied.ID); err != nil {
- return Lab{}, err
- }
+ l.ProxyPort, err = proxyPortOf(ctx, reader, proxied.ID)
+ return err
+}
+func (l *Lab) readIdentityTier(
+ ctx context.Context,
+ reader PortReader,
+ running map[string]dockerx.Target,
+) error {
verifierGateway, ok := running[VerifierGatewayName]
if !ok {
- return lab, nil
+ return nil
}
- if lab.VerifierPort, err = reader.PublishedPort(
+ var err error
+ if l.VerifierPort, err = reader.PublishedPort(
ctx,
verifierGateway.ID,
GatewayInternalPort,
); err != nil {
- return Lab{}, err
+ return err
}
- if lab.Issuer, lab.IdPPort, err = issuerOf(
+ l.Issuer, l.IdPPort, err = issuerOf(
ctx,
reader,
verifierGateway.ID,
- ); err != nil {
- return Lab{}, err
- }
-
- complete, err := lab.readTLSTier(ctx, reader, running)
- if err != nil {
- return Lab{}, err
- }
- if !complete {
- return lab, nil
- }
-
- if err = lab.readDNSTier(ctx, reader, running); err != nil {
- return Lab{}, err
- }
-
- lab.readStatefulTier(running)
-
- if err = lab.readM9Fixtures(ctx, reader, running); err != nil {
- return Lab{}, err
- }
-
- return lab, nil
+ )
+ return err
}
-// readM9Fixtures reads the four gateways M9 added. Each one's absence is not
-// fatal, for the same reason as every other gateway: the experiments that do
-// not need it must still work.
-func (l *Lab) readM9Fixtures(
+func (l *Lab) readCredentialGateways(
ctx context.Context,
reader PortReader,
running map[string]dockerx.Target,
@@ -1012,7 +1008,7 @@ func (l *Lab) readM9Fixtures(
caller, ok := running[AuthzGatewayName]
if !ok {
- return nil
+ return l.readControlTier(ctx, reader, running)
}
environ, err := reader.ContainerEnv(ctx, caller.ID)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/junit.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/junit.go
index fec604de..a5fbe5bd 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/junit.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/junit.go
@@ -6,6 +6,7 @@ package record
import (
"encoding/xml"
"fmt"
+ "slices"
"strconv"
"strings"
)
@@ -220,7 +221,14 @@ func validityCase(rec Record) (testcase, bool) {
Body: strings.Join(rec.Verdict.Reasons, "\n"),
}
- case !concluded(rec) && !rec.Verdict.Defect:
+ case rec.Verdict.Recovery == "degraded":
+ out.Error = &detail{
+ Message: "the target did not recover after rollback",
+ Type: "recovery-degraded",
+ Body: strings.Join(rec.Verdict.Reasons, "\n"),
+ }
+
+ case runInvalid(rec):
out.Error = &detail{
Message: fmt.Sprintf(messageVoid, rec.Verdict.Epistemic),
Type: typeVoid,
@@ -234,6 +242,17 @@ func validityCase(rec Record) (testcase, bool) {
return out, true
}
+func runInvalid(rec Record) bool {
+ switch rec.Verdict.Epistemic {
+ case "aborted", "interrupted", "baseline-not-met", "error":
+ return true
+ case "probe-error":
+ return !slices.Contains(rec.Verdict.Warnings, "thin-evidence")
+ default:
+ return false
+ }
+}
+
// concluded reads the record rather than the exit code, because the exit code
// can be narrowed by --fail-on and the XML must describe the run.
func concluded(rec Record) bool {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/metrics.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/metrics.go
index 0b8207a8..bb23cd78 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/metrics.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/metrics.go
@@ -13,9 +13,6 @@ import (
// Prometheus text exposition, written to a FILE for node_exporter's textfile
// collector. No client library, no server, no port, no lifecycle.
//
-// Two rules from the format's own reference, both load-bearing and both
-// omitted by the research that specified this:
-//
// the last line must end with \n, or the file is a parse failure and a
// parse failure drops the WHOLE file;
//
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/projection_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/projection_test.go
index efbbd5de..74739d14 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/projection_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/projection_test.go
@@ -181,8 +181,11 @@ func TestThinEvidenceIsSkippedRatherThanFailed(t *testing.T) {
root := parseSuites(t, body)
- if root.Failures != 0 {
- t.Fatalf("a thin-evidence run reports %d failures", root.Failures)
+ if root.Failures != 0 || root.Errors != 0 {
+ t.Fatalf(
+ "thin evidence reports failures=%d errors=%d",
+ root.Failures, root.Errors,
+ )
}
if root.Skipped != 1 {
@@ -190,6 +193,23 @@ func TestThinEvidenceIsSkippedRatherThanFailed(t *testing.T) {
}
}
+func TestDegradedRecoveryIsAValidityErrorEvenAfterAHold(t *testing.T) {
+ rec := shaped("deviated", "HELD", false, true)
+ rec.Verdict.Recovery = "degraded"
+ body, err := JUnit(rec)
+ if err != nil {
+ t.Fatal(err)
+ }
+ root := parseSuites(t, body)
+ if root.Errors != 1 || root.Failures != 0 {
+ t.Fatalf(
+ "degraded recovery: errors=%d failures=%d",
+ root.Errors,
+ root.Failures,
+ )
+ }
+}
+
// A rollback failure is an <error>: visible as a problem without being
// reported as a security regression, which it is not.
func TestARollbackFailureIsAnErrorRatherThanAFailure(t *testing.T) {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/read.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/read.go
new file mode 100644
index 00000000..1aaeff71
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/read.go
@@ -0,0 +1,45 @@
+// ©AngelaMos | 2026
+// read.go
+
+package record
+
+import (
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+)
+
+func Read(path string) (Record, error) {
+ root, err := os.OpenRoot(filepath.Dir(path))
+ if err != nil {
+ return Record{}, err
+ }
+ defer func() { _ = root.Close() }()
+ body, err := root.ReadFile(filepath.Base(path))
+ if err != nil {
+ return Record{}, err
+ }
+ var rec Record
+ if err := json.Unmarshal(body, &rec); err != nil {
+ return rec, err
+ }
+ if rec.Schema < 1 || rec.Schema > Version {
+ return rec, fmt.Errorf("unsupported record schema %d", rec.Schema)
+ }
+ if rec.Events == nil {
+ return rec, nil
+ }
+ events, err := root.ReadFile(rec.Events.File)
+ if err != nil {
+ return rec, err
+ }
+ sum := sha256.Sum256(events)
+ if hex.EncodeToString(sum[:]) != rec.Events.SHA256 {
+ return rec, errors.New("events hash does not match the run record")
+ }
+ return rec, nil
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/record.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/record.go
index 93794a00..6e672be7 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/record.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/record.go
@@ -14,7 +14,7 @@ const (
// Version is the record's own schema version, independent of khaos's. A
// consumer that parses this file needs to know when its shape changed,
// and the binary's version moves for reasons that do not touch it.
- Version = 1
+ Version = 2
// EventsFile sits beside the record rather than inside it. A sixty-second
// run at 300ms over four streams is roughly 800 observations: embedded
@@ -28,15 +28,6 @@ const (
timeLayout = time.RFC3339Nano
)
-// Record is the canonical artifact. Every other output khaos emits is a
-// projection of it, which is asserted rather than asserted-about: if the
-// terminal can show something this struct does not carry, the record is
-// incomplete and a test fails.
-//
-// It carries ONLY what khaos measured. Research 08's schema also lists host
-// details, compose-file hashes, image digests, confidence intervals and a
-// probe footprint; khaos measures none of those, and an empty key is a claim
-// that something was looked at. The gap is documented, not papered over.
type Record struct {
Schema int `json:"schema_version"`
Khaos string `json:"khaos_version"`
@@ -46,8 +37,10 @@ type Record struct {
DurationMS int64 `json:"duration_ms"`
- Experiment Experiment `json:"experiment"`
- Fault Fault `json:"fault"`
+ Experiment Experiment `json:"experiment"`
+ Fault Fault `json:"fault"`
+ Configuration experiment.Configuration `json:"configuration"`
+ HaltReason string `json:"halt_reason,omitempty"`
Windows map[string]Window `json:"windows"`
@@ -56,7 +49,8 @@ type Record struct {
// Exit is what khaos returned to the shell, in the record, because the
// exit code is the only part of a run most CI ever sees and a record that
// cannot explain it is not a record of the run.
- Exit Exit `json:"exit"`
+ Exit Exit `json:"exit"`
+ ArtifactErrors []string `json:"artifact_errors,omitempty"`
// Events names the file beside this one and the hash of its bytes, so a
// record cannot be silently separated from the observations it summarises.
@@ -138,12 +132,14 @@ func Of(
ended := result.StartedAt.Add(result.Duration)
return Record{
- Schema: Version,
- Khaos: version,
- RunID: result.RunID,
- Started: result.StartedAt.UTC().Format(time.RFC3339Nano),
- Ended: ended.UTC().Format(time.RFC3339Nano),
- DurationMS: result.Duration.Milliseconds(),
+ Schema: Version,
+ Khaos: version,
+ RunID: result.RunID,
+ Started: result.StartedAt.UTC().Format(time.RFC3339Nano),
+ Ended: ended.UTC().Format(time.RFC3339Nano),
+ DurationMS: result.Duration.Milliseconds(),
+ Configuration: result.Configuration,
+ HaltReason: result.Observation.HaltReason,
Experiment: Experiment{
Name: result.Experiment,
Family: result.Family,
@@ -217,7 +213,8 @@ func windows(observation experiment.Observation) map[string]Window {
out := map[string]Window{}
for _, phase := range Phases(observation) {
- if phase.Window.Len() == 0 {
+ if phase.Window.Valid.Len()+phase.Window.Invalid.Len()+
+ phase.Window.Revoked.Len()+phase.Window.Forged.Len() == 0 {
continue
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/unpaired_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/unpaired_test.go
new file mode 100644
index 00000000..ad5fb365
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/unpaired_test.go
@@ -0,0 +1,30 @@
+// ©AngelaMos | 2026
+// unpaired_test.go
+
+package record
+
+import (
+ "testing"
+
+ "github.com/CarterPerez-dev/khaos/internal/experiment"
+ "github.com/CarterPerez-dev/khaos/internal/probe"
+)
+
+func TestUnpairedAndUncountableObservationsRemainInTheRecord(t *testing.T) {
+ observation := experiment.Observation{}
+ observation.Fault.Record(probe.Outcome{
+ Credential: probe.Revoked,
+ Disposition: probe.Accepted,
+ })
+ observation.Recovery.Record(probe.Outcome{
+ Credential: probe.Valid,
+ Disposition: probe.ProbeFailed,
+ })
+ got := windows(observation)
+ if got["fault"].Streams["revoked"].Distribution["accepted"] != 1 {
+ t.Fatal("unpaired revoked acceptance omitted")
+ }
+ if _, ok := got["recovery"]; !ok {
+ t.Fatal("uncountable observations omitted")
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/write.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/write.go
index c66ce763..5b27d2e5 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/record/write.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/record/write.go
@@ -72,19 +72,23 @@ func Write(dir string, rec Record, events []Event) (Record, error) {
Count: len(events),
}
+ return rec, WriteRecord(dir, rec)
+}
+
+func WriteRecord(dir string, rec Record) error {
body, err := Encode(rec)
if err != nil {
- return rec, err
+ return err
}
if err := writeAtomic(
filepath.Join(dir, RecordFile),
body,
); err != nil {
- return rec, fmt.Errorf(errWriteFile, RecordFile, err)
+ return fmt.Errorf(errWriteFile, RecordFile, err)
}
- return rec, nil
+ return nil
}
// Encode is the one JSON form, used for the files and for stdout alike, so
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/audit_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/audit_test.go
new file mode 100644
index 00000000..74984b11
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/audit_test.go
@@ -0,0 +1,157 @@
+// ©AngelaMos | 2026
+// audit_test.go
+
+package run
+
+import (
+ "context"
+ "errors"
+ "testing"
+ "time"
+
+ "github.com/CarterPerez-dev/khaos/internal/dockerx"
+ "github.com/CarterPerez-dev/khaos/internal/inject"
+ "github.com/CarterPerez-dev/khaos/internal/journal"
+)
+
+func TestMissingContainerDoesNotSettlePersistentResources(t *testing.T) {
+ for _, kind := range []string{inject.KindPause, inject.KindCredentialTier, "secret-unreadable"} {
+ t.Run(kind, func(t *testing.T) {
+ log := tempJournal(t)
+ mutator := &fake{revertRe: dockerx.ErrContainerGone}
+ registry := registryOf(map[string]inject.Mutator{kind: mutator})
+ session := newSessionWith(inject.Env{}, log, time.Minute, registry)
+ if err := session.Inject(t.Context(), kind, target()); err != nil {
+ t.Fatal(err)
+ }
+ err := session.RevertAll(t.Context(), nil)
+ if (err == nil) != (kind == inject.KindPause) {
+ t.Fatalf("revert %s: %v", kind, err)
+ }
+ live, err := journal.Outstanding(log.Path())
+ if err != nil || (len(live) == 0) != (kind == inject.KindPause) {
+ t.Fatalf("outstanding %s: %v, %v", kind, live, err)
+ }
+ })
+ }
+}
+
+func TestCompletedApplyKeepsRenewedExpiry(t *testing.T) {
+ for _, fails := range []bool{false, true} {
+ name := map[bool]string{false: "success", true: "partial failure"}[fails]
+ t.Run(name, func(t *testing.T) {
+ log := tempJournal(t)
+ mutator := &fake{
+ entered: make(chan struct{}), release: make(chan struct{}),
+ }
+ if fails {
+ mutator.applyErr = errors.New("partial application")
+ }
+ registry := registryOf(map[string]inject.Mutator{fakeKind: mutator})
+ session := newSessionWith(inject.Env{}, log, time.Hour, registry)
+ done := make(chan error, 1)
+ go func() {
+ done <- session.Inject(context.Background(), fakeKind, target())
+ }()
+ <-mutator.entered
+ if err := session.Renew(); err != nil {
+ t.Fatal(err)
+ }
+ renewed := session.Lease().ExpiresAt()
+ close(mutator.release)
+ if err := <-done; (err != nil) != fails {
+ t.Fatalf("apply: %v", err)
+ }
+ live, err := journal.Outstanding(log.Path())
+ if err != nil || len(live) != 1 {
+ t.Fatalf("outstanding: %v, %v", live, err)
+ }
+ if live[0].ExpiresAt.Before(renewed) {
+ t.Fatalf(
+ "application lost renewal: %s < %s",
+ live[0].ExpiresAt,
+ renewed,
+ )
+ }
+ })
+ }
+}
+
+func TestReaperCannotSettleAnInFlightPartialApplication(t *testing.T) {
+ log := tempJournal(t)
+ mutator := &fake{
+ entered: make(chan struct{}), release: make(chan struct{}),
+ applyErr: errors.New("changed state before failing"),
+ }
+ registry := registryOf(map[string]inject.Mutator{fakeKind: mutator})
+ session := newSessionWith(inject.Env{}, log, time.Minute, registry)
+ done := make(chan error, 1)
+ go func() {
+ done <- session.Inject(context.Background(), fakeKind, target())
+ }()
+ <-mutator.entered
+ sweep := reclaimWith(context.Background(), inject.Env{}, log,
+ time.Now().Add(time.Hour), true, registry)
+ close(mutator.release)
+ if err := <-done; err == nil {
+ t.Fatal("partial application unexpectedly succeeded")
+ }
+ if len(sweep.Results) != 0 || mutator.reverted != 0 {
+ t.Fatal("recovery entered the mutator while its application was active")
+ }
+ live, err := journal.Outstanding(log.Path())
+ if err != nil || len(live) != 1 {
+ t.Fatalf("partial application disappeared: %v, %v", live, err)
+ }
+ if err := session.RevertAll(context.Background(), nil); err != nil {
+ t.Fatal(err)
+ }
+ if mutator.reverted != 1 {
+ t.Fatal("completed application was not recoverable")
+ }
+}
+
+func TestCredentialLifetimeRemainsJournaledThroughRecovery(t *testing.T) {
+ log := tempJournal(t)
+ resource, fault := &fake{}, &fake{}
+ registry := registryOf(map[string]inject.Mutator{
+ inject.KindCredentialTier: resource,
+ fakeKind: fault,
+ })
+ session := newSessionWith(inject.Env{}, log, time.Hour, registry)
+ for _, kind := range []string{inject.KindCredentialTier, fakeKind} {
+ if err := session.Inject(t.Context(), kind, target()); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := session.RevertFaults(t.Context(), nil); err != nil {
+ t.Fatal(err)
+ }
+ live, err := journal.Outstanding(log.Path())
+ if err != nil || len(live) != 1 ||
+ live[0].Kind != inject.KindCredentialTier || resource.reverted != 0 {
+ t.Fatalf(
+ "credential lifetime settled before recovery: %v, %v", live, err,
+ )
+ }
+ if err := session.Renew(); err != nil {
+ t.Fatal(err)
+ }
+ resource.revertRe = errors.New("role removal failed")
+ revertErr := session.RevertAll(t.Context(), nil)
+ if !errors.Is(revertErr, ErrRevertFailed) {
+ t.Fatalf("cleanup failure suppressed: %v", revertErr)
+ }
+ live, err = journal.Outstanding(log.Path())
+ if err != nil || len(live) != 1 {
+ t.Fatalf("failed role cleanup lost its obligation: %v, %v", live, err)
+ }
+ resource.revertRe = nil
+ if err := session.RevertAll(t.Context(), nil); err != nil {
+ t.Fatal(err)
+ }
+ live, err = journal.Outstanding(log.Path())
+ if err != nil || len(live) != 0 {
+ t.Fatalf("finished cleanup remains outstanding: %v, %v", live, err)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/injector.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/injector.go
index ca7bf2f9..b09292ff 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/injector.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/injector.go
@@ -87,7 +87,7 @@ func (i *Injector) Confirm(ctx context.Context) (string, error) {
}
func (i *Injector) Revert(ctx context.Context) error {
- return i.Session.RevertAll(ctx, i.Progress)
+ return i.Session.RevertFaults(ctx, i.Progress)
}
// Restart restarts the container this run's fault was applied to, which is the
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/injector_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/injector_test.go
index d5a401c0..2df8f35f 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/injector_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/injector_test.go
@@ -7,8 +7,41 @@ import (
"context"
"errors"
"testing"
+
+ "github.com/CarterPerez-dev/khaos/internal/dockerx"
)
+type replacementResolver struct{}
+
+func (replacementResolver) Accept(
+ _ context.Context,
+ _ string,
+ _ dockerx.Filter,
+ guard dockerx.Guard,
+) (dockerx.Target, error) {
+ target := dockerx.Target{
+ ID: "replacement", Name: "same-name", Inspected: true, Running: true,
+ Labels: map[string]string{dockerx.LabelConsent: dockerx.ConsentValue},
+ }
+ if refusal := dockerx.Refuse(target, guard); refusal != nil {
+ return dockerx.Target{}, refusal
+ }
+ return target, nil
+}
+
+func TestInjectorPreservesTheBaselineTargetIdentity(t *testing.T) {
+ injector := &Injector{
+ Resolver: replacementResolver{},
+ Guard: dockerx.Guard{ExpectedID: "baseline-container"},
+ }
+ err := injector.Inject(t.Context(), "container-pause", "same-name")
+ var refusal *dockerx.Refusal
+ if !errors.As(err, &refusal) ||
+ refusal.Predicate != dockerx.PredicateIdentityChanged {
+ t.Fatalf("replacement was not refused before mutation: %v", err)
+ }
+}
+
// The cold-start experiment restarts the container the fault was applied to.
// Asked before anything is applied, khaos has no container it can be SURE was
// meant, and guessing would restart something the operator never consented
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/reclaim.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/reclaim.go
index 81ce1aaa..581ac42b 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/reclaim.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/reclaim.go
@@ -5,6 +5,7 @@ package run
import (
"context"
+ "crypto/sha256"
"errors"
"fmt"
"time"
@@ -17,36 +18,20 @@ import (
// settleSuffix mirrors statedir.Paths.Settle.
const settleSuffix = ".settle"
-const (
- // Contention on the settle lock is two short critical sections
- // colliding, so it is worth waiting out rather than walking past.
- settleAttempts = 25
- settleBackoff = 20 * time.Millisecond
-)
-
-// holdSettle serialises read-then-append between a run and the reaper. The
-// returned release is always safe to call.
-//
-// A filesystem that cannot flock is a different answer from a lock somebody
-// holds: retrying it would stall every recovery there forever, so it
-// proceeds unlocked.
-func holdSettle(path string) func() {
- for attempt := range settleAttempts {
- settle, err := lock.Acquire(path + settleSuffix)
- if err == nil {
- return func() { _ = settle.Release() }
- }
-
- if !errors.Is(err, lock.ErrHeld) {
- return func() {}
- }
+const CleanupTimeout = 30 * time.Second
- if attempt < settleAttempts-1 {
- time.Sleep(settleBackoff)
- }
+func holdSettle(ctx context.Context, path string) (func(), error) {
+ bounded, cancel := context.WithTimeout(ctx, CleanupTimeout)
+ defer cancel()
+ held, err := lock.AcquireContext(bounded, path+settleSuffix)
+ if err != nil {
+ return nil, err
}
+ return func() { _ = held.Release() }, nil
+}
- return func() {}
+func mutationPath(path, id string) string {
+ return fmt.Sprintf("%s.%x.active", path, sha256.Sum256([]byte(id)))
}
type Reclaimed struct {
@@ -74,10 +59,16 @@ func (s Sweep) Failed() bool {
return false
}
-// Reclaim settles outstanding mutations, holding the settle lock for the whole
-// pass so a renewal and a sweep cannot undo each other's work. The lock is
-// best-effort: a dead man's switch that will not fire is worse than one that
-// races.
+func (s Sweep) Succeeded() int {
+ succeeded := 0
+ for _, result := range s.Results {
+ if result.Err == nil {
+ succeeded++
+ }
+ }
+ return succeeded
+}
+
func Reclaim(
ctx context.Context,
env inject.Env,
@@ -85,7 +76,11 @@ func Reclaim(
now time.Time,
expiredOnly bool,
) Sweep {
- defer holdSettle(log.Path())()
+ release, err := holdSettle(ctx, log.Path())
+ if err != nil {
+ return Sweep{Damage: err}
+ }
+ defer release()
return reclaimWith(ctx, env, log, now, expiredOnly, inject.Lookup)
}
@@ -112,6 +107,18 @@ func reclaimWith(
continue
}
+ active, err := lock.Acquire(mutationPath(log.Path(), record.MutationID))
+ if errors.Is(err, lock.ErrHeld) {
+ continue
+ }
+ if err != nil {
+ sweep.Results = append(
+ sweep.Results, Reclaimed{Record: record, Err: err},
+ )
+ continue
+ }
+ _ = active.Release()
+
// A fault on another daemon is left outstanding. Reverting it here
// would not fail cleanly: the ID is NotFound on this daemon, which
// khaos would read as the target being gone and write REVERTED.
@@ -126,9 +133,12 @@ func reclaimWith(
continue
}
+ cleanup, cancel := context.WithTimeout(ctx, CleanupTimeout)
+ err = revertWith(cleanup, env, log, record, lookup)
+ cancel()
sweep.Results = append(sweep.Results, Reclaimed{
Record: record,
- Err: revertWith(ctx, env, log, record, lookup),
+ Err: err,
})
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/reclaim_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/reclaim_test.go
index dbf62919..de65a313 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/reclaim_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/reclaim_test.go
@@ -4,6 +4,7 @@
package run
import (
+ "context"
"errors"
"path/filepath"
"strings"
@@ -91,7 +92,10 @@ func TestTheSettleLockIsWaitedForRatherThanWalkedPast(t *testing.T) {
entered := make(chan time.Time, 1)
go func() {
- release := holdSettle(path)
+ release, err := holdSettle(context.Background(), path)
+ if err != nil {
+ return
+ }
entered <- time.Now()
release()
}()
@@ -103,7 +107,7 @@ func TestTheSettleLockIsWaitedForRatherThanWalkedPast(t *testing.T) {
"holdSettle returned at %s with the lock still held by someone else, so the critical section is not serialised",
at.Format(time.RFC3339Nano),
)
- case <-time.After(60 * time.Millisecond):
+ case <-time.After(600 * time.Millisecond):
}
if err := held.Release(); err != nil {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/restart_obligation_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/restart_obligation_test.go
index 61089430..dbb92aad 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/restart_obligation_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/restart_obligation_test.go
@@ -7,6 +7,7 @@ import (
"context"
"fmt"
"net"
+ "os"
"os/exec"
"strings"
"testing"
@@ -21,6 +22,9 @@ const restartImage = "alpine:3.22"
func dockerOrSkip(t *testing.T) *dockerx.Client {
t.Helper()
+ if os.Getenv("KHAOS_DOCKER_TESTS") != "1" {
+ t.Skip("set KHAOS_DOCKER_TESTS=1 to run Docker integration tests")
+ }
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
@@ -158,20 +162,18 @@ func freeHighPort(t *testing.T) int {
func seedPublished(t *testing.T, name string, port int) string {
t.Helper()
- _ = exec.Command("docker", "rm", "-f", name).Run()
-
out, err := exec.Command(
"docker", "run", "-d", "--name", name,
"-p", fmt.Sprintf("127.0.0.1:%d:7", port),
restartImage, "sleep", "600",
).Output()
if err != nil {
- t.Skipf("could not create the fixture: %v", err)
+ t.Fatalf("could not create the fixture: %v", err)
}
- t.Cleanup(func() { _ = exec.Command("docker", "rm", "-f", name).Run() })
-
- return strings.TrimSpace(string(out))
+ id := strings.TrimSpace(string(out))
+ t.Cleanup(func() { _ = exec.Command("docker", "rm", "-f", id).Run() })
+ return id
}
func running(name string) bool {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/session.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/session.go
index 4cec7fdb..74677492 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/session.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/session.go
@@ -14,6 +14,7 @@ import (
"github.com/CarterPerez-dev/khaos/internal/inject"
"github.com/CarterPerez-dev/khaos/internal/journal"
"github.com/CarterPerez-dev/khaos/internal/lease"
+ "github.com/CarterPerez-dev/khaos/internal/lock"
)
const (
@@ -92,6 +93,10 @@ func (s *Session) Lease() *lease.Lease {
return s.lease
}
+func (s *Session) Configure(env inject.Env) {
+ s.env = env
+}
+
func (s *Session) Inject(
ctx context.Context,
kind string,
@@ -120,14 +125,15 @@ func (s *Session) Inject(
ExpiresAt: s.lease.ExpiresAt(),
}
- if err := s.log.Append(record); err != nil {
+ active, err := lock.Acquire(mutationPath(s.log.Path(), record.MutationID))
+ if err != nil {
return err
}
+ defer func() { _ = active.Release() }()
- // Remembered BEFORE the apply: an Apply that stops, renames and creates
- // containers is not instant, and a record outside the renewable set
- // cannot have its expiry extended while that work is in flight.
- s.remember(record)
+ if err := s.persist(record); err != nil {
+ return err
+ }
applyErr := mutator.Apply(ctx, s.env, target, payload)
record.At = time.Now().UTC()
@@ -135,25 +141,35 @@ func (s *Session) Inject(
if applyErr != nil {
record.Phase = journal.PhaseFailed
record.Detail = applyErr.Error()
- s.remember(record)
- if err := s.log.Append(record); err != nil {
+ if err := s.persist(record); err != nil {
return err
}
return fmt.Errorf(errApply, kind, target.Name, applyErr)
}
record.Phase = journal.PhaseApplied
- s.remember(record)
+ return s.persist(record)
+}
+func (s *Session) persist(record journal.Record) error {
+ release, err := holdSettle(context.Background(), s.log.Path())
+ if err != nil {
+ return err
+ }
+ defer release()
+ record.ExpiresAt = s.lease.ExpiresAt()
+ s.remember(record)
return s.log.Append(record)
}
// Renew re-reads the journal first, so a mutation something else already
// reverted is dropped rather than resurrected.
func (s *Session) Renew() error {
- // The same settle lock Reclaim takes: read-then-append is two operations
- // and the reaper runs beside this.
- defer holdSettle(s.log.Path())()
+ release, err := holdSettle(context.Background(), s.log.Path())
+ if err != nil {
+ return err
+ }
+ defer release()
live, err := journal.Outstanding(s.log.Path())
if err != nil {
@@ -182,10 +198,30 @@ func (s *Session) Renew() error {
}
func (s *Session) RevertAll(ctx context.Context, progress Progress) error {
+ return s.revert(ctx, progress, true)
+}
+
+func (s *Session) RevertFaults(ctx context.Context, progress Progress) error {
+ return s.revert(ctx, progress, false)
+}
+
+func (s *Session) revert(
+ ctx context.Context,
+ progress Progress,
+ final bool,
+) error {
+ release, err := holdSettle(ctx, s.log.Path())
+ if err != nil {
+ return errors.Join(ErrRevertFailed, err)
+ }
+ defer release()
var firstErr error
for _, record := range s.outstanding() {
- err := revertWith(ctx, s.env, s.log, record, s.lookup)
+ if !final && record.Kind == inject.KindCredentialTier {
+ continue
+ }
+ err := s.revertReady(ctx, record)
s.settle(record.MutationID, err)
if progress != nil {
@@ -205,6 +241,20 @@ func (s *Session) RevertAll(ctx context.Context, progress Progress) error {
return nil
}
+func (s *Session) revertReady(
+ ctx context.Context,
+ record journal.Record,
+) error {
+ active, err := lock.Acquire(mutationPath(s.log.Path(), record.MutationID))
+ if err != nil {
+ return err
+ }
+ defer func() { _ = active.Release() }()
+ cleanup, cancel := context.WithTimeout(ctx, CleanupTimeout)
+ defer cancel()
+ return revertWith(cleanup, s.env, s.log, record, s.lookup)
+}
+
func (s *Session) settle(mutationID string, failure error) {
s.mu.Lock()
defer s.mu.Unlock()
@@ -254,8 +304,7 @@ func (s *Session) renewable(
for i := range s.mutations {
// PENDING is renewable too: it is a mutation still inside its Apply.
- if s.mutations[i].Phase != journal.PhaseApplied &&
- s.mutations[i].Phase != journal.PhasePending {
+ if !s.mutations[i].Outstanding() {
continue
}
@@ -321,7 +370,7 @@ func revertWith(
case err == nil:
record.Phase = journal.PhaseReverted
record.Detail = ""
- case errors.Is(err, dockerx.ErrContainerGone):
+ case errors.Is(err, dockerx.ErrContainerGone) && containerBound(record.Kind):
record.Phase = journal.PhaseReverted
record.Detail = detailTargetGone
default:
@@ -334,6 +383,11 @@ func revertWith(
return log.Append(record)
}
+func containerBound(kind string) bool {
+ return kind == inject.KindPause || kind == inject.KindDisconnect ||
+ kind == inject.KindRestart
+}
+
// Restart stops and starts the target in place, for the one experiment whose
// fault is the target INITIALISING without a dependency it already had.
//
@@ -378,11 +432,15 @@ func (s *Session) Restart(
ExpiresAt: s.lease.ExpiresAt(),
}
- if err := s.log.Append(record); err != nil {
+ active, err := lock.Acquire(mutationPath(s.log.Path(), record.MutationID))
+ if err != nil {
return err
}
+ defer func() { _ = active.Release() }()
- s.remember(record)
+ if err := s.persist(record); err != nil {
+ return err
+ }
if err := s.settleRestart(ctx, record, target); err != nil {
return err
@@ -405,8 +463,7 @@ func (s *Session) settleRestart(
// reclaim decides by reading the container back.
record.Phase = journal.PhaseFailed
record.Detail = err.Error()
- s.remember(record)
- _ = s.log.Append(record)
+ _ = s.persist(record)
return err
}
@@ -414,16 +471,13 @@ func (s *Session) settleRestart(
if err := s.env.Docker.Start(ctx, target.ID); err != nil {
record.Phase = journal.PhaseApplied
record.Detail = err.Error()
- s.remember(record)
- _ = s.log.Append(record)
+ _ = s.persist(record)
return err
}
record.Phase = journal.PhaseReverted
- s.remember(record)
-
- return s.log.Append(record)
+ return s.persist(record)
}
func (s *Session) Confirm(
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/session_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/session_test.go
index 65586043..9959878c 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/run/session_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/run/session_test.go
@@ -297,7 +297,7 @@ func TestRevertAllContinuesPastAFailureAndReportsIt(t *testing.T) {
if !errors.Is(err, ErrRevertFailed) {
t.Fatalf(
- "got %v, want an error wrapping ErrRevertFailed so M10 can map it to an exit code",
+ "got %v, want an error wrapping ErrRevertFailed for the rollback exit code",
err,
)
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/replay_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/replay_test.go
new file mode 100644
index 00000000..3b895479
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/replay_test.go
@@ -0,0 +1,71 @@
+// ©AngelaMos | 2026
+// replay_test.go
+
+package secretfile
+
+import (
+ "os"
+ "testing"
+)
+
+func TestSecretReplayVerifiesCapturedMetadata(t *testing.T) {
+ for _, name := range Kinds() {
+ t.Run(name, func(t *testing.T) {
+ path, aside := seeded(t)
+ mode, size := uint32(secretPerm), int64(len(original))
+ m := Mutation{
+ Kind: Kind(name), Path: path, Aside: aside,
+ OriginalMode: &mode, OriginalSize: &size,
+ }
+ if err := m.Restore(); err != nil {
+ t.Fatalf("before application: %v", err)
+ }
+ if err := m.Apply(); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.Restore(); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.Restore(); err != nil {
+ t.Fatalf("replay: %v", err)
+ }
+ if err := os.Chmod(path, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.Restore(); err == nil {
+ t.Fatal(
+ "mismatched original permissions accepted without a backup",
+ )
+ }
+ })
+ }
+}
+
+func TestZeroPermissionBackupAndReplayKeepOriginalMode(t *testing.T) {
+ path, aside := seeded(t)
+ if err := publishCopy(path, aside, 0); err != nil {
+ t.Fatal(err)
+ }
+ info, err := os.Stat(aside)
+ if err != nil || info.Mode().Perm() != 0 {
+ t.Fatalf("zero mode backup: %v, %v", info, err)
+ }
+ if err := os.Remove(aside); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Chmod(path, 0); err != nil {
+ t.Fatal(err)
+ }
+ mode, size := uint32(0), int64(len(original))
+ mutation := Mutation{
+ Kind: Unreadable, Path: path, Aside: aside,
+ OriginalMode: &mode, OriginalSize: &size,
+ }
+ if err := mutation.Restore(); err != nil {
+ t.Fatal(err)
+ }
+ info, err = os.Stat(path)
+ if err != nil || info.Mode().Perm() != 0 {
+ t.Fatalf("zero mode replay: %v, %v", info, err)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/secretfile.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/secretfile.go
index 8d2f53c9..abc3c7ef 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/secretfile.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/secretfile.go
@@ -41,7 +41,6 @@ const (
AsideSuffix = ".khaos-held"
unreadableMode = 0o000
- restoredMode = 0o600
// A secret is conventionally delivered read-only, and the OWNER of a 0400
// file cannot write it either. Making it writable first means the
@@ -81,10 +80,12 @@ var ErrBadPath = errors.New(
// Mutation is one secret-file fault, performed INSIDE the target's own mount by
// a short-lived container. Every field is a path in the container's namespace.
type Mutation struct {
- Kind Kind `json:"kind"`
- Path string `json:"path"`
- Aside string `json:"aside"`
- Bytes int `json:"bytes,omitempty"`
+ Kind Kind `json:"kind"`
+ Path string `json:"path"`
+ Aside string `json:"aside"`
+ Bytes int `json:"bytes,omitempty"`
+ OriginalMode *uint32 `json:"original_mode,omitempty"`
+ OriginalSize *int64 `json:"original_size,omitempty"`
}
func Kinds() []string {
@@ -239,8 +240,10 @@ func (m Mutation) Restore() error {
held, err := os.Stat(m.Aside)
if err != nil {
- // Already restored is success: a revert runs again after a reclaim.
- if _, err := os.Stat(m.Path); err == nil && m.Kind != Unreadable {
+ if !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ if m.alreadyRestored() {
return nil
}
@@ -268,6 +271,20 @@ func (m Mutation) Restore() error {
return os.Remove(m.Aside)
}
+func (m Mutation) alreadyRestored() bool {
+ info, err := os.Stat(m.Path)
+ if err != nil {
+ return false
+ }
+ if m.OriginalSize != nil && info.Size() != *m.OriginalSize {
+ return false
+ }
+ if m.OriginalMode != nil {
+ return uint32(info.Mode().Perm()) == *m.OriginalMode
+ }
+ return m.Kind != Unreadable || info.Mode().Perm() != unreadableMode
+}
+
// publishCopy writes to a temporary file in the destination's own directory,
// flushes it, and renames it into place, so a failed copy leaves NOTHING a
// later Restore could mistake for a complete original.
@@ -283,10 +300,6 @@ func publishCopy(from, to string, mode os.FileMode) error {
}
defer func() { _ = source.Close() }()
- if mode.Perm() == 0 {
- mode = restoredMode
- }
-
temp, err := os.CreateTemp(filepath.Dir(to), tempPrefix)
if err != nil {
return err
@@ -341,10 +354,6 @@ func copyInto(from, to string, mode os.FileMode) error {
}
defer func() { _ = source.Close() }()
- if mode.Perm() == 0 {
- mode = restoredMode
- }
-
// Truncated rather than appended, so a restore over a longer file leaves
// no tail of the fault behind.
//nolint:gosec // same: this command exists to write the path it is given
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/secretfile_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/secretfile_test.go
index 20496549..8dbae15b 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/secretfile_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/secretfile/secretfile_test.go
@@ -49,6 +49,9 @@ func TestEveryFaultIsReversible(t *testing.T) {
if err := mutation.Restore(); err != nil {
t.Fatalf("restore: %v", err)
}
+ if err := mutation.Restore(); err != nil {
+ t.Fatalf("replayed restore: %v", err)
+ }
restored, err := os.ReadFile(path)
if err != nil {
@@ -86,10 +89,6 @@ func TestEveryFaultIsReversible(t *testing.T) {
}
}
-// TestEachFaultIsTheSHAPE research 07 part 8 describes, measured rather than
-// asserted. The four shapes are what make them distinguishable from outside: an
-// empty read is not an error, a truncated one is not either, a missing one is
-// ErrNotExist and an unreadable one is ErrPermission.
func TestEachFaultIsTheShapeTheResearchDescribes(t *testing.T) {
t.Run("empty reads as success with an empty slice", func(t *testing.T) {
path, aside := seeded(t)
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/shell/shell.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/shell/shell.go
index bc9fc393..a0ce5234 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/shell/shell.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/shell/shell.go
@@ -38,7 +38,16 @@ var ErrBusy = errors.New(
)
func Run(ctx context.Context, out io.Writer) error {
+ app := newConsole(out)
+ if path, ok := historyPath(); ok {
+ app.ActiveMenu().AddHistorySourceFile(historyName, path)
+ }
+ return app.StartContext(ctx)
+}
+
+func newConsole(out io.Writer) *console.Console {
app := console.New(version.Name)
+ app.SetEscapeMode(console.EscapeLiteral)
app.NewlineBefore = true
app.NewlineAfter = true
@@ -59,11 +68,7 @@ func Run(ctx context.Context, out io.Writer) error {
leaveOrSayWhyNot(out)
})
- if path, ok := historyPath(); ok {
- menu.AddHistorySourceFile(historyName, path)
- }
-
- return app.StartContext(ctx)
+ return app
}
func newExitCmd(out io.Writer) *cobra.Command {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/shell/shell_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/shell/shell_test.go
index 0b30606c..2e02ad05 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/shell/shell_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/shell/shell_test.go
@@ -5,13 +5,41 @@ package shell
import (
"bytes"
+ "context"
+ "io"
+ "slices"
"strings"
"testing"
"github.com/CarterPerez-dev/khaos/internal/lock"
"github.com/CarterPerez-dev/khaos/internal/statedir"
+ "github.com/spf13/cobra"
)
+func TestConsolePreservesCredentialBackslashes(t *testing.T) {
+ app := newConsole(io.Discard)
+ menu := app.ActiveMenu()
+ var got []string
+ menu.SetCommands(func() *cobra.Command {
+ root := &cobra.Command{Use: "khaos"}
+ root.AddCommand(&cobra.Command{
+ Use: "capture",
+ Run: func(_ *cobra.Command, args []string) { got = args },
+ })
+ return root
+ })
+ err := menu.RunCommandLine(
+ context.Background(), `capture C:\secrets\ "a b" password\`,
+ )
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []string{`C:\secrets\`, "a b", `password\`}
+ if !slices.Equal(got, want) {
+ t.Fatalf("arguments = %q, want %q", got, want)
+ }
+}
+
// busy holds the instance lock for the duration of a test, which is what a live
// run holds while it owns a fault. The state dir is redirected first so the
// test cannot take the lock out from under a khaos the operator is running.
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/statedir/statedir.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/statedir/statedir.go
index b5154287..52ef1e79 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/statedir/statedir.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/statedir/statedir.go
@@ -56,6 +56,14 @@ func (p Paths) Journal() string {
return filepath.Join(p.Root, journalName)
}
+func (p Paths) Runs() string {
+ return filepath.Join(p.Root, "runs")
+}
+
+func (p Paths) Run(id string) string {
+ return filepath.Join(p.Runs(), id)
+}
+
// Settle guards a read-decide-append on the journal. It is separate from Lock
// because the reaper runs beside a live run and cannot take that one.
func (p Paths) Settle() string {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/matrix_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/matrix_test.go
index aecab41e..2a02a542 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/matrix_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/matrix_test.go
@@ -86,15 +86,6 @@ const (
// a hard-failing client treats as a refusal.
byAvailability
- // byReading is the mutual TLS group, and it is the only guard that is not
- // about a check the client performs at all.
- //
- // khaos's server rejects these handshakes itself, correctly, whatever the
- // client verifies. The only question is whether the client FINDS OUT, and
- // under TLS 1.3 that depends entirely on whether it reads: research 04
- // measured every row of the ClientAuth matrix returning a successful
- // tls.Dial even where the server served nothing. So every mode catches
- // these except the one that never reads a byte.
byReading
// byNothing is a trap no client-side check can catch. CASwap is the only
@@ -202,9 +193,6 @@ func asks(mode tlsclient.Mode) bool {
mode != tlsclient.NoRevocationCheck
}
-// guardFor maps each trap to the single check that catches it. A trap mapped to
-// the wrong guard makes a whole row of the matrix wrong, so these are grouped
-// to read against research 04's fault table.
func guardFor(kind pki.Kind) guard {
switch kind {
// In the leaf, so nothing hides them.
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/mode.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/mode.go
index 8a4d2a74..0180e6e4 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/mode.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/mode.go
@@ -153,7 +153,7 @@ func (m Mode) reads() bool {
}
func (m Mode) resumes() bool {
- return m == PolicyInPeerCallback
+ return m == PolicyInPeerCallback || m == Correct
}
func (m Mode) rechecksPerConnection() bool {
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/resumption_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/resumption_test.go
index f08dded1..2ba35436 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/resumption_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/resumption_test.go
@@ -21,11 +21,6 @@ import (
// the FIRST handshake and skipped on every resumption afterwards, for as long
// as the ticket lives -- which Go caps at seven days (common.go:983-985).
//
-// Expiry cannot survive it. Go's client re-verifies the cached chain for
-// NotAfter and chain validity before offering a ticket and drops the entry if
-// it fails, which research 04 section 7.4 measured. REVOCATION survives,
-// because nothing anywhere checks it.
-//
// The sequence this asserts is the one khaos's own phases produce: establish a
// session while the certificate is good, then revoke it, then connect again.
@@ -93,6 +88,10 @@ func TestASessionIsEstablishedAndThenResumed(t *testing.T) {
// being used.
func TestARevokedCertificateSurvivesResumption(t *testing.T) {
a, addr, cache := session(t)
+ before, err := a.Served()
+ if err != nil {
+ t.Fatal(err)
+ }
client := resuming(t, a, cache, tlsclient.PolicyInPeerCallback)
@@ -111,6 +110,15 @@ func TestARevokedCertificateSurvivesResumption(t *testing.T) {
}
after := client.Decide(t.Context(), addr)
+ current, err := a.Served()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if current.Leaf.SerialNumber.Cmp(before.Leaf.SerialNumber) != 0 {
+ t.Fatal(
+ "the revoked identity differs from the cached baseline identity",
+ )
+ }
if !after.Resumed {
t.Fatal(
@@ -146,6 +154,9 @@ func TestACorrectClientCatchesRevocationDespiteResumption(t *testing.T) {
after := client.Decide(t.Context(), addr)
+ if !after.Resumed {
+ t.Fatal("the correct client did not resume the baseline session")
+ }
if after.Accepted {
t.Fatal(
"the correct client used a revoked certificate, so putting the policy in the peer callback is not what made the other client vulnerable",
@@ -153,6 +164,40 @@ func TestACorrectClientCatchesRevocationDespiteResumption(t *testing.T) {
}
}
+func TestOCSPRevokesTheCachedLeafOnResumption(t *testing.T) {
+ for _, mode := range []tlsclient.Mode{
+ tlsclient.Correct, tlsclient.PolicyInPeerCallback,
+ } {
+ t.Run(string(mode), func(t *testing.T) {
+ a, addr, cache := session(t)
+ before, err := a.Served()
+ if err != nil {
+ t.Fatal(err)
+ }
+ client := resuming(t, a, cache, mode)
+ if first := client.Decide(t.Context(), addr); !first.Accepted {
+ t.Fatal(first.Reason)
+ }
+ if err := a.Arm(pki.Trap{
+ Kind: pki.OCSPRevoked, Host: host,
+ }); err != nil {
+ t.Fatal(err)
+ }
+ current, err := a.Served()
+ if err != nil || current.Leaf.SerialNumber.Cmp(
+ before.Leaf.SerialNumber,
+ ) != 0 {
+ t.Fatalf("revocation changed the cached identity: %v", err)
+ }
+ after := client.Decide(t.Context(), addr)
+ if !after.Resumed ||
+ after.Accepted != (mode == tlsclient.PolicyInPeerCallback) {
+ t.Fatalf("resumed OCSP decision: %+v", after)
+ }
+ })
+ }
+}
+
// The hole is WIDER than revocation, and this measures how wide.
//
// A resumed handshake does not re-send the certificate, so the chain the server
@@ -160,13 +205,6 @@ func TestACorrectClientCatchesRevocationDespiteResumption(t *testing.T) {
// VerifyPeerCertificate that means every certificate fault is invisible for the
// ticket's lifetime, not only the ones nothing checks: an expired leaf, which
// crypto/x509 refuses outright on a full handshake, is accepted here.
-//
-// This is NOT the case research 04 section 7.4 measured, and the first version
-// of this test confused the two. There, the CACHED certificate itself expired
-// while time passed, and Go's client correctly re-verified the cached chain for
-// NotAfter, dropped the entry and fell back to a full handshake that failed
-// closed. That behaviour is real and is not what happens when the server swaps
-// to a different, expired certificate underneath a live ticket.
func TestAResumedSessionIgnoresWhatTheServerNowServes(t *testing.T) {
a, addr, cache := session(t)
@@ -201,7 +239,7 @@ func TestAResumedSessionIgnoresWhatTheServerNowServes(t *testing.T) {
// And the contrast: a client that does NOT reuse sessions sees the expired
// certificate and refuses it, so the acceptance above is the resumption and
// not something about the chain khaos minted.
- fresh := resuming(t, a, cache, tlsclient.Correct)
+ fresh := resuming(t, a, nil, tlsclient.Correct)
if direct := fresh.Decide(t.Context(), addr); direct.Accepted {
t.Fatal(
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/revocation.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/revocation.go
index 5cb70c54..f7cec5c8 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/revocation.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/tlsclient/revocation.go
@@ -37,14 +37,21 @@ func (c *Client) revocation(
}
decision := Decision{Accepted: true, Resumed: state.DidResume}
+ if state.DidResume {
+ state.OCSPResponse = nil
+ }
if reason, ok := c.fromStaple(state, leaf, issuer, &decision); !ok {
- return Decision{Reason: reason, AskedOCSP: decision.AskedOCSP}
+ return Decision{
+ Reason: reason, AskedOCSP: decision.AskedOCSP,
+ Resumed: state.DidResume,
+ }
}
if reason, ok := c.fromOCSP(ctx, leaf, issuer, &decision); !ok {
return Decision{
Reason: reason,
+ Resumed: state.DidResume,
AskedOCSP: decision.AskedOCSP,
AskedCRL: decision.AskedCRL,
}
@@ -53,6 +60,7 @@ func (c *Client) revocation(
if reason, ok := c.fromCRL(ctx, leaf, issuer, &decision); !ok {
return Decision{
Reason: reason,
+ Resumed: state.DidResume,
AskedOCSP: decision.AskedOCSP,
AskedCRL: decision.AskedCRL,
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/verifier/matrix_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/verifier/matrix_test.go
index 2d2059f8..2847b1d1 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/verifier/matrix_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/verifier/matrix_test.go
@@ -125,17 +125,6 @@ func subjects() []subject {
beforeFault: true,
},
- // A token that WAS legitimate. The rotation withdraws its key, so a
- // verifier that refetches cannot find the kid and refuses.
- //
- // TWO modes still honour it, and the second is the one that matters. A
- // forever cache obviously never notices. But a verifier that refetches
- // only on an unknown `kid` never notices either, because NOTHING in this
- // cell presents a new-key token: research 07 section 2.3's point is that
- // the eviction of the retired key is a side effect of the refetch a
- // new-key token triggers. That is the realistic shape for a low-traffic
- // service, and khaos's own experiment was hiding it by supplying the
- // new-key traffic itself.
{
trap: idp.KidRollover,
present: func(p *idp.Provider) (string, error) { return p.HonestToken() },
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/verifier/verifier_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/verifier/verifier_test.go
index 0cff069d..e1eb5eda 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/verifier/verifier_test.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/verifier/verifier_test.go
@@ -182,10 +182,6 @@ func TestTheDecisionCacheExpiresInCorrectMode(t *testing.T) {
}
}
-// An unknown kid has to trigger a refetch, or an overlapping rotation breaks
-// every verifier rather than none of them. Research 07 established that this
-// behaviour is convention and not specification, which is exactly why it is
-// implemented inconsistently and worth measuring.
func TestAnUnknownKidTriggersARefetch(t *testing.T) {
provider, v := pair(t, verifier.Correct)
@@ -325,11 +321,6 @@ func TestATokenOfTheWrongTypeIsRefused(t *testing.T) {
}
}
-// The amplifier. `kid` is an attacker-controlled header field, and research 07
-// section 1.4 calls an unbounded refetch "a denial-of-service amplifier".
-//
-// Measured before the fix: 200 requests carrying 200 attacker-chosen kids
-// produced 200 upstream fetches, a ratio of 1.00.
func TestUnknownKidsDoNotAmplifyIntoOneFetchEach(t *testing.T) {
provider, v := pair(t, verifier.Correct)
@@ -368,16 +359,6 @@ func TestUnknownKidsDoNotAmplifyIntoOneFetchEach(t *testing.T) {
)
}
- // The control. Bounding the refetch must not stop a legitimate rotation being
- // picked up, which is the availability failure research 07 section 2.2 counts
- // as `wrongly_rejected`.
- //
- // BOUNDED pickup, not instant. The burst above is an attack, and the cooldown
- // it trips is the whole point: research 07 section 1.4 lists a refetch
- // cooldown as a mitigation precisely because the unknown-kid path is
- // attacker-driven, and every mitigation there trades a little rotation
- // latency for it. Demanding instant pickup immediately after two hundred
- // forged kids would be asserting that the brake does not work.
if err := provider.Arm(idp.Trap{Kind: idp.KeyRotation}); err != nil {
t.Fatalf("rotate: %v", err)
}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/workspace/reconciliation_test.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/workspace/reconciliation_test.go
new file mode 100644
index 00000000..ade5f812
--- /dev/null
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/workspace/reconciliation_test.go
@@ -0,0 +1,46 @@
+// ©AngelaMos | 2026
+// reconciliation_test.go
+
+package workspace
+
+import (
+ "errors"
+ "path/filepath"
+ "testing"
+
+ "github.com/CarterPerez-dev/khaos/internal/journal"
+ "github.com/CarterPerez-dev/khaos/internal/run"
+)
+
+func TestMutationsRefuseOutstandingRecoveryAndDamagedReplay(t *testing.T) {
+ log, err := journal.Open(filepath.Join(t.TempDir(), "journal.jsonl"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = log.Close() })
+ work := &Workspace{Journal: log}
+ if err := work.RequireClean(); err != nil {
+ t.Fatalf("clean workspace refused: %v", err)
+ }
+ work.Reconciled.Damage = errors.New("damaged journal tail")
+ if err := work.RequireClean(); !errors.Is(err, run.ErrRevertFailed) {
+ t.Fatalf("damaged reconciliation accepted: %v", err)
+ }
+ work.Reconciled = run.Sweep{}
+ record := journal.Record{
+ MutationID: journal.NewID(), Kind: "fake", Phase: journal.PhasePending,
+ }
+ if err := log.Append(record); err != nil {
+ t.Fatal(err)
+ }
+ if err := work.RequireClean(); !errors.Is(err, run.ErrRevertFailed) {
+ t.Fatalf("outstanding mutation accepted: %v", err)
+ }
+ record.Phase = journal.PhaseReverted
+ if err := log.Append(record); err != nil {
+ t.Fatal(err)
+ }
+ if err := work.RequireClean(); err != nil {
+ t.Fatalf("reconciled workspace refused: %v", err)
+ }
+}
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/internal/workspace/workspace.go b/PROJECTS/intermediate/chaos-engineering-tool/internal/workspace/workspace.go
index 5693f71f..12e8fb37 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/internal/workspace/workspace.go
+++ b/PROJECTS/intermediate/chaos-engineering-tool/internal/workspace/workspace.go
@@ -5,6 +5,7 @@ package workspace
import (
"context"
+ "errors"
"fmt"
"time"
@@ -85,6 +86,33 @@ func (w *Workspace) Env() inject.Env {
return inject.Env{Docker: w.Docker}
}
+func (w *Workspace) RequireClean() error {
+ failures := []error{w.Reconciled.Damage}
+ for _, result := range w.Reconciled.Results {
+ if result.Err != nil {
+ failures = append(failures, result.Err)
+ }
+ }
+ if err := errors.Join(failures...); err != nil {
+ return errors.Join(run.ErrRevertFailed, err)
+ }
+ live, err := journal.Outstanding(w.Journal.Path())
+ if err != nil {
+ return errors.Join(run.ErrRevertFailed, err)
+ }
+ for _, record := range live {
+ if w.Docker != nil && record.Daemon != "" &&
+ record.Daemon != w.Docker.Daemon() {
+ continue
+ }
+ return fmt.Errorf(
+ "%w: %s on %s remains outstanding",
+ run.ErrRevertFailed, record.Kind, record.TargetName,
+ )
+ }
+ return nil
+}
+
func (w *Workspace) Close() {
_ = w.Docker.Close()
_ = w.Journal.Close()
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/justfile b/PROJECTS/intermediate/chaos-engineering-tool/justfile
index 0bc1fbc4..f6009bc0 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/justfile
+++ b/PROJECTS/intermediate/chaos-engineering-tool/justfile
@@ -8,20 +8,11 @@ binary := "khaos"
default:
@just --list
-# CGO_ENABLED=0 is REQUIRED, not a preference. khaos's DNS tier imports khaos's
-# own binary as a one-layer container image with no base, because an
-# unprivileged process cannot bind port 53 and Docker's --dns carries an address
-# with no port. A dynamically linked binary in a base-less image cannot exec:
-# there is no /lib64/ld-linux-x86-64.so.2 for it to find. khaos refuses the DNS
-# tier by name when it detects a PT_INTERP segment in itself, so a build without
-# this flag loses one whole fault family.
-
-# build the khaos binary, statically linked because two tiers require it
build:
- CGO_ENABLED=0 go build -o {{binary}} ./cmd/{{binary}}
+ CGO_ENABLED=0 go build -trimpath -o {{binary}} ./cmd/{{binary}}
run *args:
- go run ./cmd/{{binary}} {{args}}
+ CGO_ENABLED=0 go run ./cmd/{{binary}} {{args}}
banner: build
./{{binary}} banner
@@ -32,50 +23,22 @@ test:
race:
go test -race ./...
-# golangci-lint is the authority on this tree and `.golangci.yml` is what it
-# reads: `linters.default: none` plus an explicit list of nineteen, and three
-# FORMATTERS that plain gofmt does not satisfy -- gofumpt with extra rules,
-# gci's three import groups, and golines at 80 columns. Function length is a
-# separate number, funlen at 88 lines and 50 statements.
-#
-# These recipes exist because the justfile used to check `gofmt -l` and nothing
-# else, so `just check` and `just gate` could both pass on a tree golangci-lint
-# had findings on. The pre-commit hook runs `golangci-lint run --fix`, so the
-# first time anyone saw those findings was AT COMMIT TIME, with the fixer
-# rewriting files mid-commit and leaving them unstaged.
linter := "golangci-lint"
-# Same version CI pins, so a green local run means a green CI run.
linter-version := "v2.10.1"
vet:
go vet ./...
-# Applies all three formatters. `gofmt -w .` is NOT equivalent and never was.
-
-# format: gofumpt, gci and golines, per .golangci.yml
fmt:
{{linter}} fmt ./...
-# The same formatters as a CHECK, which is what a gate wants: it reports what
-# would change and rewrites nothing.
-
-# report formatting that would change, rewriting nothing
fmt-check:
{{linter}} fmt --diff ./...
-# No `vet` dependency on purpose: this config runs govet with `enable-all`,
-# which is a superset of bare `go vet`, and the formatters report through it
-# too. The `vet` recipe stays for a fast standalone signal.
-
-# lint and formatting, the nineteen linters .golangci.yml enables
lint:
{{linter}} run ./...
-# Prints what is installed against what CI pins, because a local lint that
-# passes on a different version is not evidence about CI.
-
-# print the installed linter version against the one CI pins
lint-version:
@{{linter}} --version
@echo "CI pins {{linter-version}}"
@@ -86,16 +49,20 @@ tidy:
tidy-check:
go mod tidy -diff
-# The one-command "make it right": format, apply every fix the linters can
-# apply, tidy the module. Leaves whatever needs a human.
-
-# format, apply every automatic lint fix, and tidy the module
fix:
{{linter}} fmt ./...
{{linter}} run --fix ./... || true
go mod tidy
-check: lint tidy-check test colors
+installer-check:
+ bash -n install.sh
+ shellcheck install.sh
+
+release-check:
+ goreleaser check
+ goreleaser release --snapshot --clean
+
+check: lint tidy-check installer-check test colors
gate: check race smoke survive range-gate proxy-gate identity-gate family-gate tls-gate dns-gate credential-gate control-gate report-gate
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/range/Dockerfile b/PROJECTS/intermediate/chaos-engineering-tool/range/Dockerfile
index 69a0bce4..2dae9b3b 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/range/Dockerfile
+++ b/PROJECTS/intermediate/chaos-engineering-tool/range/Dockerfile
@@ -9,6 +9,8 @@ RUN go mod download
COPY cmd ./cmd
COPY internal ./internal
+COPY bundle.go ./
+COPY range ./range
RUN CGO_ENABLED=0 go build -trimpath -o /out/khaos-range ./cmd/khaos-range
FROM alpine:3.22
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/range/compose.yml b/PROJECTS/intermediate/chaos-engineering-tool/range/compose.yml
index adf7c2c5..c64a1f37 100644
--- a/PROJECTS/intermediate/chaos-engineering-tool/range/compose.yml
+++ b/PROJECTS/intermediate/chaos-engineering-tool/range/compose.yml
@@ -183,12 +183,6 @@ services:
restart: "no"
- # The service a DNS redirect fault points the target at, and the reason
- # `dns.wrong_address` is a fail-open demonstration rather than an outage.
- # It approves every credential, including ones khaos never issued, which is
- # the shape of an attacker who has won a name. Research 05 section 10: a
- # hostname allowlist does not merely stop working when the name moves, it
- # actively authorises the new host.
rogue:
container_name: khaos-range-rogue
build:
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/control.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/control.sh
index 85df6ec7..dae06f92 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/control.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/control.sh
@@ -4,8 +4,6 @@
set -euo pipefail
-# M9's `control.` exit gate. Five things have to be true of this family.
-#
# 1. EVERY experiment is asserted in BOTH directions on a live range. A
# fail-open target reports `deviated / PERMITTED` exit 2 and a fail-closed
# one reports `passed / HELD` exit 0 on the identical fault. One direction
@@ -80,6 +78,7 @@ start_range() {
local started
if ! started="$("$BIN" "${args[@]}" 2>&1)"; then
fail "range up --fail-mode $fail_mode (with-db=$with_db): $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
@@ -204,10 +203,6 @@ assert_denied() {
"${label} is not called a defect" \
"without calling it a defect"
- # 3, not 0. M10 gave a deviation in a direction the polarity does NOT
- # call a defect its own exit code: the control did change under the
- # fault, and reporting that as a clean pass hid a real availability
- # finding behind a green build.
assert_exit "$label" 3
}
@@ -260,9 +255,6 @@ if start_range open; then
"policy_store_cold_start does not claim a policy was dropped" \
"stops being enforced when its store is unreachable"
- # The limiter needs the stateful tier and this range does not have it, so
- # it is refused BY NAME rather than reported as a control that was not
- # enforcing. That is the M4 finding arriving through another door.
run_experiment control.ratelimit_store_down "${TIMING[@]}"
if [[ "$last_code" == 0 ]]; then
fail "ratelimit_store_down ran without the stateful tier, so it measured a datastore that is not there"
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/credential.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/credential.sh
index e4f8039a..de5f977e 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/credential.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/credential.sh
@@ -4,9 +4,6 @@
set -euo pipefail
-# M9's exit gate for the database half of the `credential.` family: rotation
-# distinguishes rotation from revocation on all three engines.
-#
# This is the first family whose authority behaves CORRECTLY. Every other tier
# exists because no correct dependency will emit the defect - no correct
# provider mints `alg: none`, no correct CA signs an expired leaf. Here the
@@ -21,14 +18,6 @@ set -euo pipefail
# on the same engine. One statement is the entire difference between a
# change of admission policy and a revocation.
#
-# 2. THE NEGATIVE CONTROL EARNS ITS KEEP HERE. Research 07 section 6.1 is a
-# write-up of a probe that reported PostgreSQL accepting a rotated-away
-# password: it connected over loopback inside the server container, where
-# `postgres:18` ships `host all all 127.0.0.1 trust` AHEAD of scram, so no
-# password was ever evaluated. khaos's invalid stream presents a password
-# that was never correct and requires the engine to refuse it, which is the
-# control that was missing. A run that cannot establish it is void.
-#
# 3. EVERY OBSERVABLE IS THE ENGINE'S OWN MESSAGE. The confirmation quotes
# what the engine said, per engine, and the four faults produce DISTINCT
# strings. A shared constant describing the technique measures nothing,
@@ -90,8 +79,14 @@ failures=0
report=""
last_code=0
ran=()
+active_pid=""
cleanup() {
+ if [[ -n "$active_pid" ]]; then
+ kill -9 "$active_pid" 2>/dev/null || true
+ wait "$active_pid" 2>/dev/null || true
+ fi
+ "$BIN" reclaim --no-color >/dev/null 2>&1 || true
"$BIN" range down --no-color >/dev/null 2>&1 || true
rm -rf "$KHAOS_STATE_DIR"
}
@@ -205,9 +200,6 @@ assert_report_lacks() {
fi
}
-# The baseline guard that research 07 section 6.1 paid for. If the engine ever
-# accepted the never-valid password, khaos voids the run by name rather than
-# reporting a finding, and this asserts the run was NOT voided that way.
assert_control_held() {
local label="$1"
# The subject is a parameter because this one helper now serves two kinds of
@@ -264,6 +256,45 @@ leftover_roles() {
esac
}
+kill_credential_at() {
+ local engine="$1" stage="$2" log attempt reached=0 had_role=0 baseline=4s
+ if [[ "$stage" == baseline ]]; then
+ baseline=60s
+ fi
+ log="${KHAOS_STATE_DIR}/kill-${engine}-${stage}.log"
+ "$BIN" run credential.db_rotate --target "${CONTAINER[$engine]}" \
+ --engine "$engine" --baseline-for "$baseline" --fault-for 2s \
+ --recover-for 60s --sample-every 300ms --no-color \
+ >"$log" 2>&1 &
+ active_pid=$!
+ for ((attempt = 0; attempt < 60; attempt++)); do
+ if grep -qE "plan[[:space:]]+${stage}[[:space:]]" "$log"; then
+ reached=1
+ break
+ fi
+ kill -0 "$active_pid" 2>/dev/null || break
+ sleep 1
+ done
+ if [[ "$reached" != 1 ]]; then
+ fail "${engine} did not reach ${stage} before interruption"
+ cat "$log"
+ elif [[ "$(leftover_roles "$engine")" != 1 ]]; then
+ fail "${engine} ${stage} did not hold exactly one temporary role"
+ else
+ had_role=1
+ fi
+ kill -9 "$active_pid" 2>/dev/null || true
+ wait "$active_pid" 2>/dev/null || true
+ active_pid=""
+ if "$BIN" reclaim --no-color >>"$log" 2>&1 && \
+ [[ "$(leftover_roles "$engine")" == 0 && "$had_role" == 1 ]]; then
+ pass "${engine} kill during ${stage} left a role that cold reclaim removed"
+ else
+ fail "${engine} cold cleanup after ${stage} left a role or failed"
+ cat "$log"
+ fi
+}
+
# The pool configuration is fixed at container creation, deliberately, so a
# gate spanning all three has to restart the range. A fixture that could be
# reconfigured mid-run would let khaos change the target it is measuring.
@@ -289,6 +320,7 @@ start_range() {
if ! started="$("$BIN" range up --with-db --pool-mode "$pool" \
--secret-mode "$secret" --no-color --quiet 2>&1)"; then
fail "range up --with-db --pool-mode $pool --secret-mode $secret: $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
@@ -393,8 +425,6 @@ echo "== the pooled fixture: the fail-open is the DEFAULT configuration =="
# which is why the experiment declares what failing closed means.
# held the same lifetime as bounded, plus one connection checked out
# and never returned. The cleaner only walks the IDLE list, so the
-# lifetime does not bound it and HELD is the defect again. This is
-# the row research 07 section 5.3 did not have.
declare -A WANT_VERDICT=(
[unbounded]="passed / HELD"
[bounded]="deviated / DENIED"
@@ -461,16 +491,6 @@ done
echo "== the secret file: one fault of four is a finding, and the other three say why =="
-# The four shapes are what make the one that matters discriminating. Research 07
-# part 8 is explicit that the empty file is the nastiest because it is the ONLY
-# one that is not an error at the syscall level: os.ReadFile returns no error
-# and an empty slice, so every error-handling path in the target is bypassed and
-# the empty value reaches the crypto. crypto/hmac then computes happily with a
-# zero-length key, which is CVE-2019-20933's exact shape.
-#
-# The other three are expected to make a correct target refuse and CANNOT do
-# anything else, which is what makes them the controls: if all four read the
-# same, khaos is not measuring the key at all.
declare -A WANT_SECRET_UNCHECKED=(
[credential.secret_file_empty]="deviated / PERMITTED"
[credential.secret_file_truncated]="deviated / DENIED"
@@ -529,9 +549,6 @@ fi
echo "== the length check is the fix, and khaos says so =="
-# The same four faults against a verifier that validates the length after every
-# read, which is what research 07 part 8 prescribes and what the upstream fix
-# for CVE-2019-20933 literally added. The one finding has to disappear.
if start_range unbounded validated; then
run_experiment credential.secret_file_empty
@@ -801,6 +818,11 @@ done
echo "== the classifier is a differential against all three engines =="
+for engine in "${ENGINES[@]}"; do
+ kill_credential_at "$engine" baseline
+ kill_credential_at "$engine" recover
+done
+
# Including the confirmation oracle's inert control, which is the one assertion
# an oracle gutted to `return true` cannot satisfy.
differential_env=()
@@ -851,33 +873,14 @@ fi
echo "== service to service: a DENY and a SILENCE are different answers =="
-# The one fault in this family whose authority is khaos itself again, and the
-# only one that needs TWO fault windows. Research 07 section 6.2: an
-# authorization dependency has three outcomes - allow, deny and unknown - and
-# only `unknown` is eligible for a fallback policy. A caller written as
-#
-# if err != nil || resp.StatusCode != 200 { return Allow }
-#
-# has two branches where it needs three, and one window cannot show that: the
-# acceptance under a deny is already a finding, but what PROVES the mechanism is
-# that the same caller behaves identically when the dependency answers nothing
-# at all. So khaos denies one credential, measures, then makes its own
-# dependency unavailable, measures again, and reads the pair.
-#
-# Each phase measures an INDEPENDENT branch, so all four cells of that matrix
-# exist and three of the modes below are three of them. `collapses` ignores the
-# deny, `distinguishes` honours both, and `retries` honours both and asks again
-# every time, which is the same section's retry storm.
-#
-# This tier needs no stateful tier: it is in the DEFAULT range, and that is
-# asserted by bringing the range up without `--with-db`.
start_authz_range() {
local mode="$1"
"$BIN" range down --no-color >/dev/null 2>&1 || true
- if ! "$BIN" range up --authz-mode "$mode" --no-color --quiet \
- >/dev/null 2>&1; then
- fail "range up --authz-mode $mode"
+ local started
+ if ! started="$("$BIN" range up --authz-mode "$mode" --no-color --quiet 2>&1)"; then
+ fail "range up --authz-mode $mode: $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
@@ -936,10 +939,10 @@ if start_authz_range collapses; then
assert_report_has \
"the collapse is named as serving in both phases" \
- "the target served in BOTH phases"
+ "permitted access during both"
assert_report_has \
- "and it is named as two branches where three are needed" \
- "two branches where it needs three"
+ "both dependency outcomes are named" \
+ "definitive denial and an unavailable dependency"
assert_report_lacks \
"a collapsing caller is not credited with the third branch" \
"That is the third branch"
@@ -953,7 +956,7 @@ if start_authz_range distinguishes; then
"That is the third branch, and khaos observed it rather than inferring it"
assert_report_lacks \
"and is not reported as having collapsed anything" \
- "served in BOTH phases"
+ "permitted access during both"
# The negative control for the retry-storm instrument. A caller that asks
# once per request must NOT be reported as retrying, or the instrument says
@@ -983,23 +986,6 @@ fi
echo "== the regression control: a correct caller that refuses with 503 =="
-# THE INPUT THIS GATE MUST REJECT, and it could not until now. An audit found
-# khaos reporting a CORRECT fail-closed caller as `deviated / PERMITTED` with
-# "this target's fallback is to allow": the order-sensitive counts that stop
-# khaos calling a short decision cache a bypass reset on a REFUSAL, and this
-# experiment declares a 503 as its refusal, so a caller that served one request
-# from a one-second cache and then correctly refused with 503s read as a bypass.
-#
-# The three modes above cannot produce it. They answer 403 and hold no cache, so
-# the sequence is unreachable against them - which is exactly why the defect
-# shipped under 204 green assertions. This mode answers 503 and caches for a
-# second, in BOTH fault windows, so it exercises both halves of the fix:
-# `AcceptedDespiteLapse` on window one's revoked stream and
-# `AcceptedSinceLastRefusal` on window two's valid stream.
-#
-# VERIFIED by reverting the fix against this fixture on a live range: the same
-# run returns `deviated / PERMITTED` exit 2 with the predicate gutted, and
-# `passed / HELD` exit 0 with it in place.
if start_authz_range cached; then
run_experiment credential.apikey_revoked
@@ -1012,7 +998,7 @@ if start_authz_range cached; then
"this target's fallback is to allow"
assert_report_lacks \
"and is not reported as having served in both phases" \
- "served in BOTH phases"
+ "permitted access during both"
assert_report_lacks \
"and no bypass is claimed against it" \
"was still accepted"
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/dns.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/dns.sh
index 65a3dda4..614e65af 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/dns.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/dns.sh
@@ -4,9 +4,6 @@
set -euo pipefail
-# M8's exit gate. Four things have to be true of the `dns.` family, and every
-# one of them is about khaos declining to say more than it knows.
-#
# 1. The MASKED faults are refused by name, not approximated. Docker's
# embedded resolver rewrites REFUSED to SERVFAIL and answers SERVFAIL
# itself at a measured 4.001s rather than letting a query go unanswered,
@@ -18,14 +15,6 @@ set -euo pipefail
# 4.000s a slow answer silently becomes the timeout fault, so khaos
# refuses the magnitude rather than serving a different experiment.
#
-# 3. TRUNCATION SHIPS, which is the correction this milestone is built on.
-# Research 05 recorded it as absorbed entirely by the embedded resolver;
-# it is not. `tc=1` reaches the client in every truncating shape and the
-# client's RFC 5966 TCP retry arrives at khaos's resolver. The original
-# measurement used `dig` without `+ignore`, so dig performed the fallback
-# itself and reported the clean answer -- the instrument absorbed the
-# fault and the daemon was blamed.
-#
# 4. A fault the target never MET is aborted, not survived. A DNS fault can
# be armed, confirmed at khaos's own resolver, and reach nothing: Go's
# HTTP transport resolves a name once per kept-alive connection, measured
@@ -79,6 +68,7 @@ start_range() {
if ! started="$("$BIN" range up --fail-mode "$1" --dns-mode "$dns_mode" \
--no-color --quiet 2>&1)"; then
fail "range up --fail-mode $1 --dns-mode $dns_mode: $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
@@ -359,9 +349,6 @@ if start_range open; then
run_experiment "dns.${fault}"
assert_verdict "${fault} vs an uncached target" "unsupported"
- # 9, not 4. M10 gave UNSUPPORTED its own code: a question that does
- # not apply to this target is not a measurement khaos failed to make,
- # and it was indistinguishable from one while both exited 4.
assert_exit "${fault} vs an uncached target" 9
# Anchored on the LIFETIME khaos served, which is the measurement the
@@ -391,11 +378,6 @@ fi
echo "== against a gateway that fails CLOSED =="
if start_range closed; then
- # DENIED is the correct outcome with an availability cost, and this family
- # declares it acceptable, so it must not be called a defect. Research 05
- # section 10 calls fail-closed bad in a DIFFERENT way: DNS becomes a single
- # point of failure and anyone who can break it can deny the service. That is
- # not a security defect and khaos will not report it as one.
# The faults that make the dependency UNFINDABLE. A gateway that cannot
# resolve its identity provider refuses everything: failed closed, correct,
# with an availability cost this family declares acceptable.
@@ -403,7 +385,6 @@ if start_range closed; then
run_experiment "dns.${fault}"
assert_verdict "${fault} vs fail-closed" "deviated / DENIED"
- # 3, not 0: M10 gave a deviation in a non-defect direction its own code.
assert_exit "${fault} vs fail-closed" 3
assert_confirmed "${fault}"
done
@@ -417,9 +398,6 @@ if start_range closed; then
# gateway explicitly configured to fail closed serves invalid credentials
# exactly as the fail-open one does, and both report PERMITTED.
#
- # Research 05 section 10 states it as the inversion: the control does not
- # merely stop working when the name moves, it grants access to whoever took
- # the name.
for fault in wrong_address cname_chain; do
run_experiment "dns.${fault}"
assert_verdict "${fault} vs fail-closed" "deviated / PERMITTED"
@@ -513,13 +491,6 @@ fi
echo "== the rotation: a zone change that does not reach the target =="
-# `dns.ttl_rotate` is the one fault that MOVES the zone rather than answering it
-# wrongly, so the honest path serves the new address afterwards. Research 05
-# section 6.5 carries the polarity and it is NOT inverted: khaos rotates the
-# name to the address `wrong_address` uses, so a target that FOLLOWS the
-# rotation starts trusting a host on the strength of a name, which is the same
-# defect from the same evidence and stays PERMITTED.
-#
# Both directions, because the finding is which one happened:
#
# reresolve the target asks again, follows the rotation, and accepts what the
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/family.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/family.sh
index 0d76202d..59870da6 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/family.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/family.sh
@@ -30,11 +30,6 @@ TIMING=(--baseline-for 3s --fault-for 6s --recover-for 2s --sample-every 150ms)
VERIFIER_GATEWAY=khaos-range-gateway-verifier
-# Each row is a fail mode followed by the experiments that mode must let through.
-# The mapping is deliberately many-to-one in places: `alg-any` is a single
-# structural defect, resolving the key and THEN dispatching on the token's
-# algorithm, and it is the root cause of both `alg: none` acceptance and
-# HS256/RS256 confusion. Research 07 frames them that way and so does the range.
CATCHING=(
"alg-any|identity.alg_none|identity.alg_confusion"
"trust-embedded-key|identity.embedded_jwk"
@@ -51,11 +46,6 @@ CATCHING=(
# container instead of the gate working around it.
"jwks-cache-forever|identity.kid_rollover"
"jwks-cache-forever|identity.key_rotation"
- # Refetch only on an unknown kid, never on a timer. Research 07 section 2.3
- # calls this "the whole finding": the eviction of a retired key is a side
- # effect of the refetch a NEW-key token triggers, so a target with no
- # new-key traffic honours the retired key for an unbounded time. khaos's own
- # valid stream used to supply that traffic and hide it.
"jwks-kid-only|identity.kid_rollover"
"introspect-fail-open|identity.introspection_error|identity.malformed_response|identity.idp_latency"
"decision-cache-forever|identity.revoked_token"
@@ -112,6 +102,7 @@ start_range() {
local started
if ! started="$("$BIN" range up --fail-mode "$1" --quiet --no-color 2>&1)"; then
fail "range up --fail-mode $1: $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
@@ -223,9 +214,6 @@ assert_states_a_finding() {
fi
}
-# The exposure as a NUMBER. A count of acceptances with no span leaves the reader
-# inferring the window from the timing flags, and research 07 section 2.2 calls
-# this the real exposure window of a leaked key.
assert_reports_the_exposure_bound() {
local label="$1"
if grep -qE "^ why .*accepted over a span of at least" <<<"$report"; then
@@ -235,9 +223,6 @@ assert_reports_the_exposure_bound() {
fi
}
-# khaos must never claim a cache has no expiry from a bounded observation.
-# Research 07's open question 5 bans exactly that: "label it as a bound, never as
-# caches forever".
assert_never_claims_an_unbounded_cache() {
local label="$1"
if grep -qiE "no expiry at all|caches forever" <<<"$report"; then
@@ -426,11 +411,6 @@ if start_range closed; then
run_experiment "$experiment"
- # DERIVED from the expected verdict rather than written beside it.
- # M10 gave a deviation in a non-defect direction its own exit
- # code, so four of these seventeen now exit 3 and the rest exit 0
- # -- and a hand-maintained second table would be one more thing
- # that can disagree with the first.
assert_exit "$label" "$last_code" "$(exit_for_verdict "$want_verdict")"
assert_verdict "$label" "$want_verdict"
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/identity.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/identity.sh
index 3aab5f86..fe60dd4d 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/identity.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/identity.sh
@@ -44,6 +44,7 @@ start_range() {
local started
if ! started="$("$BIN" range up --fail-mode "$1" --cache-ttl "$CACHE_TTL" --quiet --no-color 2>&1)"; then
fail "range up --fail-mode $1: $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
@@ -77,8 +78,6 @@ assert_exit() {
fi
}
-# THE milestone assertion. Two targets that are indistinguishable under steady
-# traffic must come back with DIFFERENT causes, because they need different fixes.
assert_cause() {
local label="$1" want="$2"
if grep -qE "^ cause .*${want}" <<<"$report"; then
@@ -160,7 +159,7 @@ assert_a_failed_revocation_voids_the_run() {
if start_range stale-rearm; then
run_experiment
assert_verdict "re-armed cache" "deviated / PERMITTED"
- assert_cause "re-armed cache" "TTL-EXTENSION"
+ assert_cause "re-armed cache" "LAPSED-ON-PAUSE"
assert_exit "re-armed cache" "$last_code" 2
assert_phase "re-armed cache" "act" "so nothing it measured had been revoked"
assert_phase "re-armed cache" "idle" "so the two causes were never separated"
@@ -185,7 +184,6 @@ if start_range closed; then
run_experiment
assert_verdict "fail-closed" "deviated / DENIED"
- # 3, not 0: M10 gave a deviation in a non-defect direction its own code.
assert_exit "fail-closed" "$last_code" 3
assert_no_cause "fail-closed"
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/proxy.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/proxy.sh
index ba4610e4..a260cd71 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/proxy.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/proxy.sh
@@ -71,6 +71,7 @@ start_range() {
local started
if ! started="$("$BIN" range up --fail-mode "$1" --quiet --no-color 2>&1)"; then
fail "range up --fail-mode $1: $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
@@ -191,11 +192,6 @@ assert_an_unconfirmable_fault_is_not_success() {
return
fi
- # The code is DERIVED from the epistemic word the run actually reported,
- # not hardcoded. M10 split the one inconclusive code into three, and this
- # check accepts three states: asserting a single code against a
- # three-state regex is a flake that fires whenever the run lands on one
- # of the other two.
local epistemic
epistemic="$(grep -E '^ verdict' <<<"$report" |
sed -E 's/^ verdict +([a-z-]+).*/\1/' | head -1 || true)"
@@ -322,10 +318,6 @@ if start_range open; then
assert_confirmed_on_the_socket "$experiment"
if [[ "$experiment" == "$EOF_DEPENDENT_FAULT" ]]; then
- # Confirmed at the socket, and correctly finding nothing: this
- # target reads a Content-Length and is not the kind of caller this
- # fault breaks. Asserting a bypass here is what hid the payload
- # delay for an entire milestone.
assert_verdict "$experiment" "passed / HELD"
assert_exit "$experiment" "$last_code" 0
continue
@@ -356,9 +348,6 @@ if start_range closed; then
assert_verdict "${experiment} fail-closed" "deviated / DENIED"
- # 3, not 0: M10 moved a deviation in a direction the polarity does
- # NOT call a defect onto its own code. The target did change under the
- # fault, and reporting that as a clean pass hid it.
assert_exit "${experiment} fail-closed" "$last_code" 3
if grep -q "acceptable for this control" <<<"$report"; then
@@ -369,10 +358,6 @@ if start_range closed; then
done
fi
-# M10's most dangerous exit code, and the one no other gate exercises. A HELD
-# verdict khaos declined to call conclusive used to exit 0: khaos asserting a
-# control held on evidence it refused to stand behind, as a green build.
-#
# Asserted as a PAIR. "exit 4 on a short window" alone is satisfied by any
# failure; the same experiment with a full window exiting 0 is what proves the
# 4 came from the sample floor and not from something else going wrong.
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/range.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/range.sh
index 304a6f2c..99f6ddf1 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/range.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/range.sh
@@ -32,6 +32,7 @@ start_range() {
local started
if ! started="$("$BIN" range up --fail-mode "$1" --cache-ttl "${2:-2s}" --quiet --no-color 2>&1)"; then
fail "range up --fail-mode $1 --cache-ttl ${2:-2s}: $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
@@ -201,7 +202,6 @@ if start_range closed; then
assert_zero_flags "fail-closed"
assert_verdict "fail-closed" "deviated / DENIED"
- # 3, not 0: M10 gave a deviation in a non-defect direction its own code.
assert_exit "fail-closed" "$last_code" 3
if grep -q "acceptable for this control" <<<"$report"; then
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/report.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/report.sh
index 7d4c9067..bcba91c4 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/report.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/report.sh
@@ -4,8 +4,6 @@
set -euo pipefail
-# M10's exit gate. Five things have to be true of khaos's outputs.
-#
# 1. The EXIT CODE TAXONOMY is exercised end to end against a live range.
# A single non-zero code would collapse "a control failed open", "the
# experiment was invalid" and "khaos could not measure" into one signal,
@@ -65,6 +63,7 @@ start_range() {
local started
if ! started="$("$BIN" range up --fail-mode "$1" --no-color --quiet 2>&1)"; then
fail "range up --fail-mode $1: $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/survive.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/survive.sh
index 5ae06fce..81d8c8c0 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/survive.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/survive.sh
@@ -216,6 +216,33 @@ else
fi
assert_no_outstanding "disconnect the journal is clean after kill -9"
+pid="$(inject_bg network-disconnect 120s)"
+if await "! attached $TARGET"; then
+ kill -9 "$pid" 2>/dev/null || true
+ wait "$pid" 2>/dev/null || true
+ docker network connect --alias changed-by-operator "$NET" "$TARGET"
+ set +e
+ mismatch="$("$BIN" reclaim --no-color 2>&1)"
+ mismatch_code=$?
+ set -e
+ if [[ "$mismatch_code" != 0 ]] && grep -q 'different aliases' <<<"$mismatch"; then
+ pass "disconnect a changed alias keeps recovery outstanding"
+ else
+ fail "disconnect altered endpoint was treated as restored: $mismatch"
+ fi
+ if aliases_of "$TARGET" | grep -q changed-by-operator; then
+ pass "disconnect recovery did not overwrite an altered endpoint"
+ else
+ fail "disconnect recovery overwrote an altered endpoint"
+ fi
+ docker network disconnect "$NET" "$TARGET"
+ docker network connect --alias "$ALIAS" "$NET" "$TARGET"
+ "$BIN" reclaim --no-color >/dev/null 2>&1 || fail "disconnect corrected endpoint did not settle"
+ assert_no_outstanding "disconnect restored configuration settles on replay"
+else
+ fail "disconnect alias drift control never detached the target"
+fi
+
docker rm -f "$DECOY" >/dev/null 2>&1 || true
pid="$(inject_bg container-pause 3s)"
if await "paused $TARGET"; then
diff --git a/PROJECTS/intermediate/chaos-engineering-tool/scripts/tls.sh b/PROJECTS/intermediate/chaos-engineering-tool/scripts/tls.sh
index e3809bdc..7442e1e4 100755
--- a/PROJECTS/intermediate/chaos-engineering-tool/scripts/tls.sh
+++ b/PROJECTS/intermediate/chaos-engineering-tool/scripts/tls.sh
@@ -4,9 +4,6 @@
set -euo pipefail
-# M7's exit gate. Two things have to be true of the `tls.` family, and both of
-# them are about khaos declining to say more than it knows.
-#
# 1. `revoked` reports UNSUPPORTED against a target that never checks
# revocation, and a DEFECT-CAPABLE verdict against one that does. Go's
# crypto/x509 performs no revocation checking of any kind and has no
@@ -56,6 +53,7 @@ start_range() {
local started
if ! started="$("$BIN" range up --tls-mode "$1" --no-color --quiet 2>&1)"; then
fail "range up --tls-mode $1: $(tail -3 <<<"$started")"
+ printf '%s\n' "$started" >&2
return 1
fi
return 0
@@ -116,7 +114,6 @@ if start_range closed; then
run_experiment tls.leaf_expired
assert_verdict "leaf-expired vs a correct client" "deviated / DENIED"
- # 3, not 0: M10 gave a deviation in a non-defect direction its own code.
assert_exit "leaf-expired vs a correct client" 3
assert_confirmed "leaf-expired"
@@ -150,18 +147,12 @@ if start_range closed; then
fi
fi
-echo "== M7 exit gate, first half: revocation is UNSUPPORTED, never a pass =="
+echo "== revocation support checks =="
if start_range no-revocation-check; then
run_experiment tls.revoked_via_crl
assert_verdict "revoked vs a client that never asks" "unsupported"
- # NOT zero. A revoked certificate in active use must never exit 0.
- #
- # 9 rather than 4 as of M10: "every inconclusive epistemic outcome exits
- # 4" was true when this was written and is not any more. UNSUPPORTED has
- # its own code, because a question that does not apply to this target is
- # a different thing from evidence khaos could not gather.
assert_exit "revoked vs a client that never asks" 9
if grep -qE '^ verdict .*passed' <<<"$report"; then
@@ -183,7 +174,7 @@ if start_range no-revocation-check; then
fi
fi
-echo "== M7 exit gate, second half: identical errors are disambiguated =="
+echo "== certificate fault attribution checks =="
if start_range closed; then
# All three produce `x509: certificate signed by unknown authority`, with
@@ -215,10 +206,6 @@ if start_range closed; then
fail "only ${#observed[@]} of the three unknown-authority faults were observable"
fi
- # The SECOND identical group, which research 04's fault table got wrong.
- # Both extended key-usage faults come back as InvalidReason 4 on go1.26.5;
- # the row claiming the CA-level one yields Reason 9 is impossible, because
- # crypto/x509 never constructs that reason at all.
declare -A eku=()
for experiment in tls.eku_client_auth_only tls.eku_constrained_intermediate; do
@@ -258,11 +245,6 @@ fi
echo "== mutual TLS: the client has to NOTICE being refused =="
-# khaos's server rejects both of these correctly, whatever the client verifies.
-# The finding is whether the client finds out, and under TLS 1.3 that comes down
-# to whether it reads: research 04 section 7.2 measured every row of the
-# ClientAuth matrix returning a successful tls.Dial even where the server served
-# nothing at all.
if start_range dial-only; then
for experiment in tls.mtls_client_expired tls.mtls_client_absent; do
run_experiment "$experiment"
diff --git a/README.md b/README.md
index c2bfba70..b74c89ad 100644
--- a/README.md
+++ b/README.md
@@ -122,7 +122,7 @@ Tools, courses, certifications, communities, and frameworks for cybersecurity pr
| **[Wireless Deauth Detector](./SYNOPSES/intermediate/Wireless.Deauth.Detector.md)**<br>Monitor WiFi deauth attacks | ![2-4d](https://img.shields.io/badge/⏱️_2--4d-blue) ![Rust](https://img.shields.io/badge/Rust-000000?logo=rust&logoColor=white) ![Intermediate](https://img.shields.io/badge/●_Intermediate-yellow) | Wireless security • Packet sniffing • Attack detection<br>[Learn More](./SYNOPSES/intermediate/Wireless.Deauth.Detector.md) |
| **[Credential Enumeration](./PROJECTS/intermediate/credential-enumeration)**<br>Post-exploitation credential collection | ![2-4d](https://img.shields.io/badge/⏱️_2--4d-blue) ![Nim](https://img.shields.io/badge/Nim-FFE953?logo=nim&logoColor=black) ![Intermediate](https://img.shields.io/badge/●_Intermediate-yellow) | Credential extraction • Browser forensics • Red team tooling • Nim language<br>[Source Code](./PROJECTS/intermediate/credential-enumeration) \| [Docs](./PROJECTS/intermediate/credential-enumeration/learn) |
| **[Binary Analysis Tool](./PROJECTS/intermediate/binary-analysis-tool)**<br>Disassemble and analyze executables | ![3-5d](https://img.shields.io/badge/⏱️_3--5d-blue) ![Rust](https://img.shields.io/badge/Rust-000000?logo=rust&logoColor=white) ![Intermediate](https://img.shields.io/badge/●_Intermediate-yellow) | Binary analysis • String extraction • Malware detection<br>[Source Code](./PROJECTS/intermediate/binary-analysis-tool) \| [Docs](./PROJECTS/intermediate/binary-analysis-tool/learn)<br>[![https://elitenetwork4life.com/](https://img.shields.io/badge/elitenetwork4life.com-E8730C?style=flat&logo=googlechrome&logoColor=white)](https://elitenetwork4life.com/) |
-| **[Chaos Engineering Security Tool](./SYNOPSES/intermediate/Chaos.Engineering.Security.Tool.md)**<br>Inject security failures to test resilience | ![3-5d](https://img.shields.io/badge/⏱️_3--5d-blue) ![Go](https://img.shields.io/badge/Go-00ADD8?logo=go&logoColor=white) ![Intermediate](https://img.shields.io/badge/●_Intermediate-yellow) | Chaos engineering • Security resilience • Credential spraying • Auth testing<br>[Learn More](./SYNOPSES/intermediate/Chaos.Engineering.Security.Tool.md) |
+| **[Chaos Engineering Security Tool](./PROJECTS/intermediate/chaos-engineering-tool)**<br>Inject security failures and measure control behavior | ![3-5d](https://img.shields.io/badge/⏱️_3--5d-blue) ![Go](https://img.shields.io/badge/Go-00ADD8?logo=go&logoColor=white) ![Intermediate](https://img.shields.io/badge/●_Intermediate-yellow) | Security chaos engineering • Authentication • TLS and DNS • Recovery<br>[Source Code](./PROJECTS/intermediate/chaos-engineering-tool) \| [Docs](./PROJECTS/intermediate/chaos-engineering-tool/learn) |
| **[Credential Rotation Enforcer](./SYNOPSES/intermediate/Credential.Rotation.Enforcer.md)**<br>Track and enforce credential rotation policies | ![2-4d](https://img.shields.io/badge/⏱️_2--4d-blue) ![Crystal](https://img.shields.io/badge/Crystal-black?logo=crystal&logoColor=white) ![Intermediate](https://img.shields.io/badge/●_Intermediate-yellow) | Credential hygiene • Secret rotation • Compliance dashboards • API integration<br>[Source Code](./PROJECTS/intermediate/credential-rotation-enforcer) \| [Docs](./PROJECTS/intermediate/credential-rotation-enforcer/learn) |
| **[Race Condition Exploiter](./SYNOPSES/intermediate/Race.Condition.Exploiter.md)**<br>TOCTOU race condition attack & defense lab | ![3-5d](https://img.shields.io/badge/⏱️_3--5d-blue) ![FastAPI](https://img.shields.io/badge/FastAPI-009688?logo=fastapi) ![React](https://img.shields.io/badge/React-61DAFB?logo=react&logoColor=black) ![Intermediate](https://img.shields.io/badge/●_Intermediate-yellow) | TOCTOU attacks • Double-spend bugs • Concurrent exploitation • Race visualization<br>[Learn More](./SYNOPSES/intermediate/Race.Condition.Exploiter.md) |
| **[JA3/JA4 TLS Fingerprinting Tool](./SYNOPSES/intermediate/JA3.JA4.TLS.Fingerprinting.Tool.md)**<br>Fingerprint TLS clients by handshake | ![2-4d](https://img.shields.io/badge/⏱️_2--4d-blue) ![Rust](https://img.shields.io/badge/Rust-000000?logo=rust&logoColor=white) ![Intermediate](https://img.shields.io/badge/●_Intermediate-yellow) | TLS handshake analysis • JA3/JA4 hashing • Bot detection • Malware C2 identification<br>[Source Code](./PROJECTS/intermediate/ja3-ja4-tls-fingerprinting) \| [Docs](./PROJECTS/intermediate/ja3-ja4-tls-fingerprinting/learn)<br>[![mkultraalumni.com](https://img.shields.io/badge/mkultraalumni.com-E8730C?style=flat&logo=googlechrome&logoColor=white)](https://mkultraalumni.com/) |