PostgreSQL backup tool with Point-In-Time-Recovery and restore verification
TypeScript backupbackupsdatabasedatabase-backupdevopsdockergolangkubernetesmariadbmongodbmysqlpgpostgrespostgresqls3self-hostedsystem-administrationtoolsweb-ui
FIX (verification): Remove the container and network an earlier attempt of the same verification left behind before each attempt, and purge orphaned agent networks at startup, so a retry stops failing with "network already exists" (issue #832)
Rostislav Dugin82bf706a06bb
diff --git a/agent/verification/e2e/scripts/run-all.sh b/agent/verification/e2e/scripts/run-all.shindex f4299ee0..a92e8a3e 100644--- a/agent/verification/e2e/scripts/run-all.sh+++ b/agent/verification/e2e/scripts/run-all.sh@@ -11,6 +11,7 @@ TESTS=("test-restore-disk-budget.sh""test-restore-timescale.sh""test-restore-include-public-schema.sh"+ "test-restore-retry-leftover.sh")PG_VERSIONS=(12 13 14 15 16 17 18)diff --git a/agent/verification/e2e/scripts/test-restore-retry-leftover.sh b/agent/verification/e2e/scripts/test-restore-retry-leftover.shnew file mode 100644index 00000000..4cbb9650--- /dev/null+++ b/agent/verification/e2e/scripts/test-restore-retry-leftover.sh@@ -0,0 +1,92 @@+#!/bin/bash+# Retry over a leftover network (issue #832): the backend requeues a failed+# attempt under the same verification ID, and the agent names the attempt's+# network after that ID. A network the earlier attempt left behind used to+# fail every retry with "network with name databasus-verif-<id> already+# exists". Seed that leftover after the startup purge has run, so only the+# cleanup before the attempt can remove it, then assert the restore completes+# and nothing leaks.+set -euo pipefail++source "$(dirname "$0")/lib.sh"++PG_VERSION=16++WORK="/tmp/agent-work-restore-retry-leftover"+AGENT_ID="77777777-7777-7777-7777-777777777777"+VERIFICATION_ID="77777777-aaaa-aaaa-aaaa-777777777777"+BACKUP_ID="77777777-bbbb-bbbb-bbbb-777777777777"+LEFTOVER_NETWORK="databasus-verif-${VERIFICATION_ID}"++cleanup() {+ docker network rm "$LEFTOVER_NETWORK" >/dev/null 2>&1 || true+}+trap cleanup EXIT++rm -rf "$WORK"+mkdir -p "$WORK"+cd "$WORK"++reset_mock_state+reset_mock_version++cleanup++cp "$ARTIFACTS/agent-v1" ./databasus-verification-agent+chmod +x ./databasus-verification-agent++start_agent "$AGENT_ID"++# The heartbeat loop starts only after the startup purge returns, so the first+# heartbeat means the purge can no longer remove the seeded network.+STARTED=0+for _ in $(seq 1 30); do+ if [ "$(hb_count)" -ge 1 ] 2>/dev/null; then+ STARTED=1+ break+ fi+ sleep 1+done++if [ "$STARTED" -ne 1 ]; then+ echo "FAIL: agent sent no heartbeat within 30s"+ dump_diagnostics+ stop_agent+ exit 1+fi++docker network create \+ --label "databasus.verification.agent_id=${AGENT_ID}" \+ --label "databasus.verification.verification_id=${VERIFICATION_ID}" \+ "$LEFTOVER_NETWORK" >/dev/null++if ! docker network ls --filter "name=${LEFTOVER_NETWORK}" --format '{{.Name}}' | grep -qx "$LEFTOVER_NETWORK"; then+ echo "FAIL: seeded leftover network $LEFTOVER_NETWORK not present"+ stop_agent+ exit 1+fi+echo "Seeded leftover network $LEFTOVER_NETWORK after the startup purge"++curl -sf -X POST "$MOCK/mock/set-backup-fixture" \+ -H 'Content-Type: application/json' \+ -d "{\"path\":\"/artifacts/good-pg${PG_VERSION}.dump\"}"++curl -sf -X POST "$MOCK/mock/set-claim" \+ -H 'Content-Type: application/json' \+ -d "{\"verificationId\":\"$VERIFICATION_ID\",\"backupId\":\"$BACKUP_ID\",\"backupSizeMb\":1,\"maxContainerDiskMb\":2048,\"database\":{\"type\":\"POSTGRES_LOGICAL\",\"postgresqlLogical\":{\"version\":\"${PG_VERSION}\"}}}"++wait_for_report '"status":"COMPLETED"' 240 '"status":"FAILED"'++assert_report "$VERIFICATION_ID" '.pgRestoreExitCode == 0'++echo "Retry report OK: COMPLETED despite the leftover network"++stop_agent++if ! leak_check "$AGENT_ID"; then+ echo "---- agent.out ----"+ cat agent.out+ exit 1+fi++echo "Verification agent restore-retry-leftover e2e passed"diff --git a/agent/verification/e2e/scripts/test-startup-purge.sh b/agent/verification/e2e/scripts/test-startup-purge.shindex 055c0832..84999461 100644--- a/agent/verification/e2e/scripts/test-startup-purge.sh+++ b/agent/verification/e2e/scripts/test-startup-purge.sh@@ -1,8 +1,10 @@#!/bin/bash-# Startup purge: at boot the agent removes every container labeled with its-# agent_id — the single-agent invariant means no other process owns containers-# with the same agent_id, so blanket removal is safe. Pre-seed a stray, start-# the agent, assert it's gone.+# Startup purge: at boot the agent removes every container and network labeled+# with its agent_id — the single-agent invariant means no other process owns+# resources with the same agent_id, so blanket removal is safe. Pre-seed a+# stray container, an orphan network with no container attached and a network+# of another agent, start the agent, assert the first two are gone and the+# foreign network remains.set -euo pipefailsource "$(dirname "$0")/lib.sh"@@ -11,6 +13,14 @@ WORK="/tmp/agent-work-purge"AGENT_ID="66666666-6666-6666-6666-666666666666"STRAY_NAME="databasus-verif-purge-stray"STRAY_LABEL="databasus.verification.agent_id=${AGENT_ID}"+ORPHAN_NETWORK="databasus-verif-purge-orphan"+FOREIGN_NETWORK="databasus-verif-purge-foreign"+FOREIGN_LABEL="databasus.verification.agent_id=67676767-6767-6767-6767-676767676767"++cleanup() {+ docker network rm "$FOREIGN_NETWORK" "$ORPHAN_NETWORK" >/dev/null 2>&1 || true+}+trap cleanup EXITrm -rf "$WORK"mkdir -p "$WORK"@@ -28,6 +38,11 @@ if ! docker ps -a --filter "label=$STRAY_LABEL" --format '{{.Names}}' | grep -qfiecho "Pre-seeded stray container $STRAY_NAME with label $STRAY_LABEL"+cleanup+docker network create --label "$STRAY_LABEL" "$ORPHAN_NETWORK" >/dev/null+docker network create --label "$FOREIGN_LABEL" "$FOREIGN_NETWORK" >/dev/null+echo "Pre-seeded orphan network $ORPHAN_NETWORK and foreign network $FOREIGN_NETWORK"+cp "$ARTIFACTS/agent-v1" ./databasus-verification-agentchmod +x ./databasus-verification-agent@@ -35,7 +50,9 @@ start_agent "$AGENT_ID"GONE=0for _ in $(seq 1 30); do- if [ -z "$(docker ps -a --filter "label=$STRAY_LABEL" --format '{{.Names}}')" ]; then+ # The purge removes containers before networks, so wait for both.+ if [ -z "$(docker ps -a --filter "label=$STRAY_LABEL" --format '{{.Names}}')" ] &&+ [ -z "$(docker network ls --filter "name=${ORPHAN_NETWORK}" --format '{{.Name}}')" ]; thenGONE=1breakfi@@ -43,14 +60,22 @@ for _ in $(seq 1 30); dodoneif [ "$GONE" -ne 1 ]; then- echo "FAIL: stray container still present after agent start; PurgeContainers did not run"+ echo "FAIL: stray container or orphan network still present after agent start"docker ps -a --filter "label=$STRAY_LABEL"+ docker network ls --filter "label=$STRAY_LABEL"echo "---- agent.out (last 30) ----"tail -30 agent.outstop_agentexit 1fi-echo "Stray container purged by agent startup"+echo "Stray container and orphan network purged by agent startup"++if ! docker network ls --filter "name=${FOREIGN_NETWORK}" --format '{{.Name}}' | grep -qx "$FOREIGN_NETWORK"; then+ echo "FAIL: foreign network $FOREIGN_NETWORK was removed by the purge"+ stop_agent+ exit 1+fi+echo "Foreign network left in place"stop_agentdiff --git a/agent/verification/go.mod b/agent/verification/go.modindex c137abf2..1a704bda 100644--- a/agent/verification/go.mod+++ b/agent/verification/go.mod@@ -3,6 +3,7 @@ module databasus-verification-agentgo 1.26.3require (+ github.com/containerd/errdefs v1.0.0github.com/go-resty/resty/v2 v2.17.2github.com/google/uuid v1.6.0github.com/jackc/pgx/v5 v5.9.2@@ -14,7 +15,6 @@ require (require (github.com/Microsoft/go-winio v0.6.2 // indirectgithub.com/cespare/xxhash/v2 v2.3.0 // indirect- github.com/containerd/errdefs v1.0.0 // indirectgithub.com/containerd/errdefs/pkg v0.3.0 // indirectgithub.com/davecgh/go-spew v1.1.1 // indirectgithub.com/distribution/reference v0.6.0 // indirectdiff --git a/agent/verification/internal/features/container/dockerengine.go b/agent/verification/internal/features/container/dockerengine.goindex 5cc2274e..f9424328 100644--- a/agent/verification/internal/features/container/dockerengine.go+++ b/agent/verification/internal/features/container/dockerengine.go@@ -24,7 +24,8 @@ import (const (// LabelVerificationID / LabelAgentID tag every container and network so the- // startup purge can find and remove this agent's leftovers by agent ID+ // startup purge can remove this agent's leftovers by agent ID, and a retried+ // attempt can remove those of its verification by both labelsLabelVerificationID = "databasus.verification.verification_id"LabelAgentID = "databasus.verification.agent_id"@@ -313,36 +314,48 @@ func (e *dockerEngine) RemoveContainer(ctx context.Context, containerID string)return err}-func (e *dockerEngine) ListManaged(- ctx context.Context, agentID string,-) ([]ManagedContainer, error) {+func (e *dockerEngine) ListManagedContainers(+ ctx context.Context, labels map[string]string,+) ([]ManagedResource, error) {listed, err := e.cli.ContainerList(ctx, client.ContainerListOptions{All: true,- Filters: make(client.Filters).Add("label", LabelAgentID+"="+agentID),+ Filters: labelFilters(labels),})if err != nil {return nil, err}- managed := make([]ManagedContainer, 0, len(listed.Items))+ managed := make([]ManagedResource, 0, len(listed.Items))for _, s := range listed.Items {- mc := ManagedContainer{- ID: s.ID,- VerificationID: s.Labels[LabelVerificationID],- CreatedUnix: s.Created,- }+ managed = append(managed, ManagedResource{ID: s.ID, VerificationID: s.Labels[LabelVerificationID]})+ }- if s.NetworkSettings != nil {- for _, endpoint := range s.NetworkSettings.Networks {- if endpoint.NetworkID != "" {- mc.NetworkID = endpoint.NetworkID- break- }- }- }+ return managed, nil+}- managed = append(managed, mc)+func (e *dockerEngine) ListManagedNetworks(+ ctx context.Context, labels map[string]string,+) ([]ManagedResource, error) {+ listed, err := e.cli.NetworkList(ctx, client.NetworkListOptions{Filters: labelFilters(labels)})+ if err != nil {+ return nil, err+ }++ managed := make([]ManagedResource, 0, len(listed.Items))+ for _, n := range listed.Items {+ managed = append(managed, ManagedResource{ID: n.ID, VerificationID: n.Labels[LabelVerificationID]})}return managed, nil}++// labelFilters passes every label as its own value of the label filter, which+// Docker combines with AND; one value holding several labels would match none.+func labelFilters(labels map[string]string) client.Filters {+ filters := make(client.Filters)+ for key, value := range labels {+ filters.Add("label", key+"="+value)+ }++ return filters+}diff --git a/agent/verification/internal/features/container/fake_engine_test.go b/agent/verification/internal/features/container/fake_engine_test.goindex abddc92d..4e03b3f5 100644--- a/agent/verification/internal/features/container/fake_engine_test.go+++ b/agent/verification/internal/features/container/fake_engine_test.go@@ -1,31 +1,144 @@package container-import "context"+import (+ "context"+ "fmt"+ "maps"+ "slices"+ "strings"+)-type fakePurgeEngine struct {- managed []ManagedContainer- removedContainer []string- removedNetwork []string- listErr error- removeContainerErr error+// fakeDockerHost stores labeled containers and networks and matches a listing+// filter with AND semantics, as Docker does. Every listing, removal and network+// creation lands in one ordered log, so tests can pin the order of operations.+type fakeDockerHost struct {+ containers []fakeResource+ networks []fakeResource+ log []string++ listContainersErr error+ listNetworksErr error+ removeErrs map[string]error+}++type fakeResource struct {+ id string+ labels map[string]string+}++func (f *fakeDockerHost) addContainer(id string, labels map[string]string) {+ f.containers = append(f.containers, fakeResource{id: id, labels: labels})+}++func (f *fakeDockerHost) addNetwork(id string, labels map[string]string) {+ f.networks = append(f.networks, fakeResource{id: id, labels: labels})+}++func (f *fakeDockerHost) failRemoval(id string, err error) {+ if f.removeErrs == nil {+ f.removeErrs = map[string]error{}+ }++ f.removeErrs[id] = err+}++func (f *fakeDockerHost) containerIDs() []string {+ return resourceIDs(f.containers)+}++func (f *fakeDockerHost) networkIDs() []string {+ return resourceIDs(f.networks)}-func (f *fakePurgeEngine) ListManaged(context.Context, string) ([]ManagedContainer, error) {- return f.managed, f.listErr+func (f *fakeDockerHost) ListManagedContainers(+ _ context.Context, labels map[string]string,+) ([]ManagedResource, error) {+ f.log = append(f.log, "list containers "+formatLabels(labels))+ if f.listContainersErr != nil {+ return nil, f.listContainersErr+ }++ return matchLabels(f.containers, labels), nil+}++func (f *fakeDockerHost) ListManagedNetworks(+ _ context.Context, labels map[string]string,+) ([]ManagedResource, error) {+ f.log = append(f.log, "list networks "+formatLabels(labels))+ if f.listNetworksErr != nil {+ return nil, f.listNetworksErr+ }++ return matchLabels(f.networks, labels), nil}-func (f *fakePurgeEngine) RemoveContainer(_ context.Context, id string) error {- if f.removeContainerErr != nil {- return f.removeContainerErr+func (f *fakeDockerHost) RemoveContainer(_ context.Context, id string) error {+ f.log = append(f.log, "remove container "+id)+ if err := f.removeErrs[id]; err != nil {+ return err}- f.removedContainer = append(f.removedContainer, id)+ f.containers = slices.DeleteFunc(f.containers, func(r fakeResource) bool { return r.id == id })return nil}-func (f *fakePurgeEngine) RemoveNetwork(_ context.Context, id string) error {- f.removedNetwork = append(f.removedNetwork, id)+func (f *fakeDockerHost) RemoveNetwork(_ context.Context, id string) error {+ f.log = append(f.log, "remove network "+id)+ if err := f.removeErrs[id]; err != nil {+ return err+ }++ f.networks = slices.DeleteFunc(f.networks, func(r fakeResource) bool { return r.id == id })return nil}++func (f *fakeDockerHost) CreateNetwork(+ _ context.Context, name string, labels map[string]string,+) (string, error) {+ f.log = append(f.log, "create network "+name)+ id := fmt.Sprintf("net-%d", len(f.networks)+1)+ f.addNetwork(id, labels)++ return id, nil+}++func matchLabels(resources []fakeResource, filter map[string]string) []ManagedResource {+ matched := []ManagedResource{}+ for _, r := range resources {+ hasAll := true+ for key, value := range filter {+ if r.labels[key] != value {+ hasAll = false++ break+ }+ }++ if hasAll {+ matched = append(matched, ManagedResource{ID: r.id, VerificationID: r.labels[LabelVerificationID]})+ }+ }++ return matched+}++// formatLabels renders a label filter in a stable order for the log.+func formatLabels(labels map[string]string) string {+ pairs := []string{}+ for _, key := range slices.Sorted(maps.Keys(labels)) {+ pairs = append(pairs, key+"="+labels[key])+ }++ return strings.Join(pairs, ",")+}++func resourceIDs(resources []fakeResource) []string {+ ids := []string{}+ for _, r := range resources {+ ids = append(ids, r.id)+ }++ return ids+}diff --git a/agent/verification/internal/features/container/leftovers.go b/agent/verification/internal/features/container/leftovers.gonew file mode 100644index 00000000..8c4bbdfe--- /dev/null+++ b/agent/verification/internal/features/container/leftovers.go@@ -0,0 +1,93 @@+package container++import (+ "context"+ "errors"+ "fmt"++ cerrdefs "github.com/containerd/errdefs"+)++// leftoverEngine is the Docker seam for removing labeled leftovers — the+// methods the unit tests fake, rather than the whole Docker surface.+type leftoverEngine interface {+ ListManagedContainers(ctx context.Context, labels map[string]string) ([]ManagedResource, error)+ ListManagedNetworks(ctx context.Context, labels map[string]string) ([]ManagedResource, error)+ RemoveContainer(ctx context.Context, containerID string) error+ RemoveNetwork(ctx context.Context, networkID string) error+}++type attemptNetworkEngine interface {+ leftoverEngine+ CreateNetwork(ctx context.Context, name string, labels map[string]string) (string, error)+}++// removeLeftovers removes the containers and then the networks that carry+// every given label. Containers go first because Docker refuses to remove a+// network with a container attached. A failure in one half does not skip the+// other, and a resource that is already gone counts as removed, so a teardown+// racing this call cannot fail it.+func removeLeftovers(+ ctx context.Context, engine leftoverEngine, labels map[string]string,+) (removed int, err error) {+ var errs []error++ containers, listErr := engine.ListManagedContainers(ctx, labels)+ if listErr != nil {+ errs = append(errs, fmt.Errorf("list containers: %w", listErr))+ }++ for _, c := range containers {+ if err := engine.RemoveContainer(ctx, c.ID); err != nil && !cerrdefs.IsNotFound(err) {+ errs = append(errs, fmt.Errorf("remove container %s of verification %s: %w", c.ID, c.VerificationID, err))++ continue+ }++ removed+++ }++ networks, listErr := engine.ListManagedNetworks(ctx, labels)+ if listErr != nil {+ errs = append(errs, fmt.Errorf("list networks: %w", listErr))+ }++ for _, n := range networks {+ if err := engine.RemoveNetwork(ctx, n.ID); err != nil && !cerrdefs.IsNotFound(err) {+ errs = append(errs, fmt.Errorf("remove network %s of verification %s: %w", n.ID, n.VerificationID, err))++ continue+ }++ removed+++ }++ return removed, errors.Join(errs...)+}++// prepareAttemptNetwork clears what an earlier attempt of the same+// verification left under these labels, then creates the attempt's network. A+// retry reuses the verification ID and so the network name, which an earlier+// attempt's leftover network would otherwise keep taken.+func prepareAttemptNetwork(+ ctx context.Context, engine attemptNetworkEngine, name string, labels map[string]string,+) (string, error) {+ if _, err := removeLeftovers(ctx, engine, labels); err != nil {+ return "", fmt.Errorf("could not remove the leftovers of an earlier attempt: %w", err)+ }++ networkID, err := engine.CreateNetwork(ctx, name, labels)+ if err != nil {+ return "", fmt.Errorf("create isolated network: %w", err)+ }++ return networkID, nil+}++func agentLabels(agentID string) map[string]string {+ return map[string]string{LabelAgentID: agentID}+}++func attemptLabels(agentID, verificationID string) map[string]string {+ return map[string]string{LabelAgentID: agentID, LabelVerificationID: verificationID}+}diff --git a/agent/verification/internal/features/container/leftovers_test.go b/agent/verification/internal/features/container/leftovers_test.gonew file mode 100644index 00000000..3ef67363--- /dev/null+++ b/agent/verification/internal/features/container/leftovers_test.go@@ -0,0 +1,142 @@+package container++import (+ "errors"+ "testing"++ cerrdefs "github.com/containerd/errdefs"+ "github.com/stretchr/testify/assert"+ "github.com/stretchr/testify/require"+)++const attemptNetworkName = "databasus-verif-v1"++// Pins issue #832: a network the previous attempt left behind used to fail+// every retry with "network with name databasus-verif-<id> already exists".+func Test_PrepareAttemptNetwork_WhenPreviousAttemptLeftNetwork_RemovesItAndCreatesNetwork(t *testing.T) {+ host := &fakeDockerHost{}+ host.addNetwork("old-network", attemptLabels("agent-1", "v1"))++ networkID, err := prepareAttemptNetwork(t.Context(), host, attemptNetworkName, attemptLabels("agent-1", "v1"))++ require.NoError(t, err)+ assert.Equal(t, []string{networkID}, host.networkIDs())+ assert.NotEqual(t, "old-network", networkID)+}++func Test_PrepareAttemptNetwork_WhenPreviousAttemptLeftContainerAndNetwork_RemovesContainerThenNetworkThenCreates(+ t *testing.T,+) {+ host := &fakeDockerHost{}+ host.addContainer("old-container", attemptLabels("agent-1", "v1"))+ host.addNetwork("old-network", attemptLabels("agent-1", "v1"))+ filter := formatLabels(attemptLabels("agent-1", "v1"))++ _, err := prepareAttemptNetwork(t.Context(), host, attemptNetworkName, attemptLabels("agent-1", "v1"))++ require.NoError(t, err)+ assert.Equal(t, []string{+ "list containers " + filter,+ "remove container old-container",+ "list networks " + filter,+ "remove network old-network",+ "create network " + attemptNetworkName,+ }, host.log)+}++func Test_PrepareAttemptNetwork_WhenLeftoverAlreadyRemoved_CreatesNetwork(t *testing.T) {+ host := &fakeDockerHost{}+ host.addContainer("old-container", attemptLabels("agent-1", "v1"))+ host.addNetwork("old-network", attemptLabels("agent-1", "v1"))+ host.failRemoval("old-container", cerrdefs.ErrNotFound)+ host.failRemoval("old-network", cerrdefs.ErrNotFound)++ _, err := prepareAttemptNetwork(t.Context(), host, attemptNetworkName, attemptLabels("agent-1", "v1"))++ require.NoError(t, err)+ assert.Contains(t, host.log, "create network "+attemptNetworkName)+}++func Test_PrepareAttemptNetwork_WhenNoLeftovers_CreatesNetworkWithoutRemovals(t *testing.T) {+ host := &fakeDockerHost{}+ filter := formatLabels(attemptLabels("agent-1", "v1"))++ _, err := prepareAttemptNetwork(t.Context(), host, attemptNetworkName, attemptLabels("agent-1", "v1"))++ require.NoError(t, err)+ assert.Equal(t, []string{+ "list containers " + filter,+ "list networks " + filter,+ "create network " + attemptNetworkName,+ }, host.log)+}++func Test_PrepareAttemptNetwork_WhenLeftoverRemovalFails_ReturnsLeftoverErrorAndCreatesNoNetwork(t *testing.T) {+ for _, refused := range []string{"old-container", "old-network"} {+ t.Run(refused, func(t *testing.T) {+ host := &fakeDockerHost{}+ host.addContainer("old-container", attemptLabels("agent-1", "v1"))+ host.addNetwork("old-network", attemptLabels("agent-1", "v1"))+ host.failRemoval(refused, errors.New("removal refused"))++ _, err := prepareAttemptNetwork(t.Context(), host, attemptNetworkName, attemptLabels("agent-1", "v1"))++ require.Error(t, err)+ assert.Contains(t, err.Error(), "could not remove the leftovers of an earlier attempt")+ assert.Contains(t, err.Error(), refused)+ assert.NotContains(t, host.log, "create network "+attemptNetworkName)+ })+ }+}++func Test_PrepareAttemptNetwork_WhenListingFails_ReturnsLeftoverErrorAndCreatesNoNetwork(t *testing.T) {+ for name, host := range map[string]*fakeDockerHost{+ "containers": {listContainersErr: errors.New("docker unreachable")},+ "networks": {listNetworksErr: errors.New("docker unreachable")},+ } {+ t.Run(name, func(t *testing.T) {+ _, err := prepareAttemptNetwork(t.Context(), host, attemptNetworkName, attemptLabels("agent-1", "v1"))++ require.Error(t, err)+ assert.Contains(t, err.Error(), "could not remove the leftovers of an earlier attempt")+ assert.NotContains(t, host.log, "create network "+attemptNetworkName)+ })+ }+}++func Test_PrepareAttemptNetwork_WhenOtherVerificationRuns_LeavesItsResourcesInPlace(t *testing.T) {+ host := &fakeDockerHost{}+ host.addContainer("running-container", attemptLabels("agent-1", "v2"))+ host.addNetwork("running-network", attemptLabels("agent-1", "v2"))++ networkID, err := prepareAttemptNetwork(t.Context(), host, attemptNetworkName, attemptLabels("agent-1", "v1"))++ require.NoError(t, err)+ assert.Equal(t, []string{"running-container"}, host.containerIDs())+ assert.Equal(t, []string{"running-network", networkID}, host.networkIDs())+}++func Test_PrepareAttemptNetwork_WhenOtherAgentHasSameVerification_LeavesItsResourcesInPlace(t *testing.T) {+ host := &fakeDockerHost{}+ host.addContainer("foreign-container", attemptLabels("agent-2", "v1"))+ host.addNetwork("foreign-network", attemptLabels("agent-2", "v1"))++ networkID, err := prepareAttemptNetwork(t.Context(), host, attemptNetworkName, attemptLabels("agent-1", "v1"))++ require.NoError(t, err)+ assert.Equal(t, []string{"foreign-container"}, host.containerIDs())+ assert.Equal(t, []string{"foreign-network", networkID}, host.networkIDs())+}++func Test_PrepareAttemptNetwork_FiltersByAgentAndVerificationLabels(t *testing.T) {+ host := &fakeDockerHost{}++ _, err := prepareAttemptNetwork(t.Context(), host, attemptNetworkName, attemptLabels("agent-1", "v1"))++ require.NoError(t, err)+ assert.Equal(t, []string{+ "list containers " + LabelAgentID + "=agent-1," + LabelVerificationID + "=v1",+ "list networks " + LabelAgentID + "=agent-1," + LabelVerificationID + "=v1",+ "create network " + attemptNetworkName,+ }, host.log)+}diff --git a/agent/verification/internal/features/container/managedcontainer.go b/agent/verification/internal/features/container/managedcontainer.godeleted file mode 100644index 83a2a6a3..00000000--- a/agent/verification/internal/features/container/managedcontainer.go+++ /dev/null@@ -1,8 +0,0 @@-package container--type ManagedContainer struct {- ID string- NetworkID string- VerificationID string- CreatedUnix int64-}diff --git a/agent/verification/internal/features/container/managedresource.go b/agent/verification/internal/features/container/managedresource.gonew file mode 100644index 00000000..3842d62b--- /dev/null+++ b/agent/verification/internal/features/container/managedresource.go@@ -0,0 +1,7 @@+package container++// ManagedResource is a container or network this agent labeled.+type ManagedResource struct {+ ID string+ VerificationID string+}diff --git a/agent/verification/internal/features/container/manager.go b/agent/verification/internal/features/container/manager.goindex f8324f13..6cab3498 100644--- a/agent/verification/internal/features/container/manager.go+++ b/agent/verification/internal/features/container/manager.go@@ -143,18 +143,16 @@ func (m *Manager) Spawn(jobCtx context.Context, req SpawnRequest) (*PostgresContreturn nil, fmt.Errorf("generate password: %w", err)}- // LabelAgentID is load-bearing for the startup purge: ListManaged filters- // solely on it, so a container spawned without this label can never be- // purged and leaks until the host is cleaned by hand.- labels := map[string]string{- LabelAgentID: m.agentID,- LabelVerificationID: req.VerificationID.String(),- }-- networkID, err := m.engine.CreateNetwork(- jobCtx, "databasus-verif-"+req.VerificationID.String(), labels)+ // Both labels are load-bearing for leftover removal: the startup purge+ // filters on LabelAgentID, and the next attempt of this verification on+ // both. A resource created without them leaks until the host is cleaned by+ // hand, and a leftover network blocks every retry by its name.+ labels := attemptLabels(m.agentID, req.VerificationID.String())++ networkID, err := prepareAttemptNetwork(+ jobCtx, m.engine, "databasus-verif-"+req.VerificationID.String(), labels)if err != nil {- return nil, fmt.Errorf("create isolated network: %w", err)+ return nil, err}spec := m.buildSpec(spawnPlan{diff --git a/agent/verification/internal/features/container/purge.go b/agent/verification/internal/features/container/purge.goindex 009a2966..b772de6a 100644--- a/agent/verification/internal/features/container/purge.go+++ b/agent/verification/internal/features/container/purge.go@@ -6,53 +6,23 @@ import ("log/slog")-// purgeEngine is the Docker seam for the startup purge — the same three methods-// the unit tests fake, rather than the whole Docker surface.-type purgeEngine interface {- ListManaged(ctx context.Context, agentID string) ([]ManagedContainer, error)- RemoveContainer(ctx context.Context, containerID string) error- RemoveNetwork(ctx context.Context, networkID string) error-}-// purgeAgentContainers runs once at startup, with no age grace and no// active-set check, so each agent process begins from a clean slate-// ("always fresh").+// ("always fresh"). It finds networks by label, not through containers, so a+// network whose container is already gone is removed too.//-// Best-effort by design: a per-item failure is logged and skipped, and a-// failed listing is logged and ignored, never blocking startup. Container and-// network names are per-UUID (databasus-verif-<uuid>), so a leftover never-// collides with a fresh run; degraded (a few stale containers until the next-// restart) beats refusing to start on a transient Docker hiccup.-func purgeAgentContainers(ctx context.Context, engine purgeEngine, agentID string, log *slog.Logger) {- managed, err := engine.ListManaged(ctx, agentID)+// Best-effort by design: a failed listing or removal is logged and never+// blocks startup. A leftover the purge misses cannot block a retry, because+// each attempt clears its own verification's leftovers first (see+// prepareAttemptNetwork); degraded beats refusing to start on a transient+// Docker hiccup.+func purgeAgentContainers(ctx context.Context, engine leftoverEngine, agentID string, log *slog.Logger) {+ removed, err := removeLeftovers(ctx, engine, agentLabels(agentID))if err != nil {- log.Warn("startup purge could not list managed containers", "error", err)-- return+ log.Warn("startup purge left some resources in place", "error", err)}- if len(managed) == 0 {- return+ if removed > 0 {+ log.Info(fmt.Sprintf("startup purge removed %d stale container(s) and network(s)", removed))}-- removed := 0- for _, mc := range managed {- if err := engine.RemoveContainer(ctx, mc.ID); err != nil {- log.Warn("startup purge failed to remove container",- "verification_id", mc.VerificationID, "error", err)-- continue- }-- if mc.NetworkID != "" {- if err := engine.RemoveNetwork(ctx, mc.NetworkID); err != nil {- log.Warn("startup purge failed to remove network",- "verification_id", mc.VerificationID, "error", err)- }- }-- removed++- }-- log.Info(fmt.Sprintf("startup purge removed %d stale container(s)", removed))}diff --git a/agent/verification/internal/features/container/purge_test.go b/agent/verification/internal/features/container/purge_test.goindex 7f394c1d..e243eb6f 100644--- a/agent/verification/internal/features/container/purge_test.go+++ b/agent/verification/internal/features/container/purge_test.go@@ -4,71 +4,151 @@ import ("errors""testing"- "github.com/google/uuid"+ cerrdefs "github.com/containerd/errdefs""github.com/stretchr/testify/assert""databasus-verification-agent/internal/testutil")func Test_PurgeAgentContainers_RemovesEveryContainerAndNetwork(t *testing.T) {- engine := &fakePurgeEngine{- managed: []ManagedContainer{- {ID: "c1", NetworkID: "n1", VerificationID: uuid.New().String()},- {ID: "c2", NetworkID: "n2", VerificationID: uuid.New().String()},- },- }+ host := &fakeDockerHost{}+ host.addContainer("c1", attemptLabels("agent-1", "v1"))+ host.addContainer("c2", attemptLabels("agent-1", "v2"))+ host.addNetwork("n1", attemptLabels("agent-1", "v1"))+ host.addNetwork("n2", attemptLabels("agent-1", "v2"))- purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger())+ purgeAgentContainers(t.Context(), host, "agent-1", testutil.DiscardLogger())- assert.ElementsMatch(t, []string{"c1", "c2"}, engine.removedContainer)- assert.ElementsMatch(t, []string{"n1", "n2"}, engine.removedNetwork)+ assert.Empty(t, host.containerIDs())+ assert.Empty(t, host.networkIDs())}-func Test_PurgeAgentContainers_WhenNetworkIDEmpty_RemovesContainerOnly(t *testing.T) {- engine := &fakePurgeEngine{- managed: []ManagedContainer{{ID: "c1", VerificationID: uuid.New().String()}},- }+func Test_PurgeAgentContainers_WhenNothingLabeled_RemovesNothing(t *testing.T) {+ host := &fakeDockerHost{}- purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger())+ purgeAgentContainers(t.Context(), host, "agent-1", testutil.DiscardLogger())- assert.Equal(t, []string{"c1"}, engine.removedContainer)- assert.Empty(t, engine.removedNetwork)+ assert.Equal(t, []string{+ "list containers " + LabelAgentID + "=agent-1",+ "list networks " + LabelAgentID + "=agent-1",+ }, host.log)}-func Test_PurgeAgentContainers_WhenListFails_RemovesNothing(t *testing.T) {- engine := &fakePurgeEngine{- managed: []ManagedContainer{{ID: "c1", NetworkID: "n1"}},- listErr: errors.New("docker unreachable"),- }+func Test_PurgeAgentContainers_WhenContainerRemovalFails_ContinuesWithRemainingResources(t *testing.T) {+ host := &fakeDockerHost{}+ host.addContainer("c1", attemptLabels("agent-1", "v1"))+ host.addContainer("c2", attemptLabels("agent-1", "v2"))+ host.addNetwork("n2", attemptLabels("agent-1", "v2"))+ host.failRemoval("c1", errors.New("container in use"))- purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger())+ assert.NotPanics(t, func() {+ purgeAgentContainers(t.Context(), host, "agent-1", testutil.DiscardLogger())+ })++ assert.Equal(t, []string{"c1"}, host.containerIDs())+ assert.Empty(t, host.networkIDs())+}++func Test_PurgeAgentContainers_WhenNetworkHasNoContainer_RemovesNetwork(t *testing.T) {+ host := &fakeDockerHost{}+ host.addNetwork("n1", attemptLabels("agent-1", "v1"))++ purgeAgentContainers(t.Context(), host, "agent-1", testutil.DiscardLogger())++ assert.Empty(t, host.networkIDs())+}++func Test_PurgeAgentContainers_WhenContainerAndNetworkExist_RemovesContainerBeforeNetwork(t *testing.T) {+ host := &fakeDockerHost{}+ host.addContainer("c1", attemptLabels("agent-1", "v1"))+ host.addNetwork("n1", attemptLabels("agent-1", "v1"))++ purgeAgentContainers(t.Context(), host, "agent-1", testutil.DiscardLogger())++ assert.Equal(t, []string{+ "list containers " + LabelAgentID + "=agent-1",+ "remove container c1",+ "list networks " + LabelAgentID + "=agent-1",+ "remove network n1",+ }, host.log)+}++func Test_PurgeAgentContainers_WhenResourcesBelongToOtherAgentOrNoAgent_LeavesThemInPlace(t *testing.T) {+ host := &fakeDockerHost{}+ host.addContainer("foreign-container", attemptLabels("agent-2", "v1"))+ host.addNetwork("foreign-network", attemptLabels("agent-2", "v1"))+ host.addNetwork("unrelated-network", map[string]string{"com.example.app": "web"})++ purgeAgentContainers(t.Context(), host, "agent-1", testutil.DiscardLogger())- assert.Empty(t, engine.removedContainer)- assert.Empty(t, engine.removedNetwork)+ assert.Equal(t, []string{"foreign-container"}, host.containerIDs())+ assert.Equal(t, []string{"foreign-network", "unrelated-network"}, host.networkIDs())}-func Test_PurgeAgentContainers_WhenListEmpty_DoesNothing(t *testing.T) {- engine := &fakePurgeEngine{}+func Test_PurgeAgentContainers_FiltersByAgentLabelOnly(t *testing.T) {+ host := &fakeDockerHost{}++ purgeAgentContainers(t.Context(), host, "agent-1", testutil.DiscardLogger())++ assert.Equal(t, []string{+ "list containers " + formatLabels(agentLabels("agent-1")),+ "list networks " + formatLabels(agentLabels("agent-1")),+ }, host.log)+}- purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger())+func Test_PurgeAgentContainers_WhenNetworkListFails_LogsAndRemovesContainers(t *testing.T) {+ host := &fakeDockerHost{listNetworksErr: errors.New("docker unreachable")}+ host.addContainer("c1", attemptLabels("agent-1", "v1"))+ host.addNetwork("n1", attemptLabels("agent-1", "v1"))+ log, logs := testutil.CaptureLogger()++ assert.NotPanics(t, func() {+ purgeAgentContainers(t.Context(), host, "agent-1", log)+ })++ assert.Empty(t, host.containerIDs())+ assert.Equal(t, []string{"n1"}, host.networkIDs())+ assert.Contains(t, logs.String(), "level=WARN msg=\"startup purge left some resources in place\"")+ assert.Contains(t, logs.String(), "list networks: docker unreachable")+}- assert.Empty(t, engine.removedContainer)- assert.Empty(t, engine.removedNetwork)+func Test_PurgeAgentContainers_WhenRemovalReturnsNotFound_ContinuesWithRemainingResources(t *testing.T) {+ host := &fakeDockerHost{}+ host.addContainer("c1", attemptLabels("agent-1", "v1"))+ host.addContainer("c2", attemptLabels("agent-1", "v2"))+ host.addNetwork("n1", attemptLabels("agent-1", "v1"))+ host.failRemoval("c1", cerrdefs.ErrNotFound)+ log, logs := testutil.CaptureLogger()++ purgeAgentContainers(t.Context(), host, "agent-1", log)++ assert.Equal(t, []string{+ "list containers " + LabelAgentID + "=agent-1",+ "remove container c1",+ "remove container c2",+ "list networks " + LabelAgentID + "=agent-1",+ "remove network n1",+ }, host.log)+ assert.Empty(t, host.networkIDs())+ assert.NotContains(t, logs.String(), "level=WARN")+ assert.Contains(t, logs.String(), "startup purge removed 3 stale container(s) and network(s)")}-func Test_PurgeAgentContainers_WhenContainerRemovalFails_SkipsNetworkAndContinues(t *testing.T) {- engine := &fakePurgeEngine{- managed: []ManagedContainer{- {ID: "c1", NetworkID: "n1", VerificationID: uuid.New().String()},- {ID: "c2", NetworkID: "n2", VerificationID: uuid.New().String()},- },- removeContainerErr: errors.New("container in use"),- }+func Test_PurgeAgentContainers_WhenContainerListFails_LogsAndRemovesNetworks(t *testing.T) {+ host := &fakeDockerHost{listContainersErr: errors.New("docker unreachable")}+ host.addContainer("c1", attemptLabels("agent-1", "v1"))+ host.addNetwork("n1", attemptLabels("agent-1", "v1"))+ host.addNetwork("n2", attemptLabels("agent-1", "v2"))+ host.failRemoval("n1", errors.New("network has active endpoints"))+ log, logs := testutil.CaptureLogger()assert.NotPanics(t, func() {- purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger())+ purgeAgentContainers(t.Context(), host, "agent-1", log)})- assert.Empty(t, engine.removedContainer)- assert.Empty(t, engine.removedNetwork)+ assert.Equal(t, []string{"c1"}, host.containerIDs())+ assert.Equal(t, []string{"n1"}, host.networkIDs())+ assert.Contains(t, logs.String(), "level=WARN msg=\"startup purge left some resources in place\"")+ assert.Contains(t, logs.String(), "list containers: docker unreachable")+ assert.Contains(t, logs.String(), "remove network n1 of verification v1: network has active endpoints")}diff --git a/agent/verification/internal/testutil/testutil.go b/agent/verification/internal/testutil/testutil.goindex 0a9b20b0..9aba0014 100644--- a/agent/verification/internal/testutil/testutil.go+++ b/agent/verification/internal/testutil/testutil.go@@ -2,10 +2,21 @@// package tests.package testutil-import "log/slog"+import (+ "bytes"+ "log/slog"+)// DiscardLogger returns a no-op logger so tests don't write to the rotating// log file or stdout.func DiscardLogger() *slog.Logger {return slog.New(slog.DiscardHandler)}++// CaptureLogger returns a logger that writes text records to the returned+// buffer, so a test can assert what was logged and at which level.+func CaptureLogger() (*slog.Logger, *bytes.Buffer) {+ buf := &bytes.Buffer{}++ return slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug})), buf+}diff --git a/openspec/changes/verification-retry-leftover-cleanup/.openspec.yaml b/openspec/changes/verification-retry-leftover-cleanup/.openspec.yamlnew file mode 100644index 00000000..7cc41bb5--- /dev/null+++ b/openspec/changes/verification-retry-leftover-cleanup/.openspec.yaml@@ -0,0 +1,2 @@+schema: spec-driven+created: 2026-10-22diff --git a/openspec/changes/verification-retry-leftover-cleanup/design.md b/openspec/changes/verification-retry-leftover-cleanup/design.mdnew file mode 100644index 00000000..a9ef9cb8--- /dev/null+++ b/openspec/changes/verification-retry-leftover-cleanup/design.md@@ -0,0 +1,78 @@+## Context++See proposal.md for the motivation. The current state that shapes the approach:++- `Spawn` names the network of every attempt `databasus-verif-<verification ID>` and creates it with no check for an existing one (`agent/verification/internal/features/container/manager.go:155`). Both the network and the container carry `LabelAgentID` and `LabelVerificationID` (`manager.go:149-152`, `dockerengine.go:26-29`).+- `Terminate` removes the container and then the network under a 30-second deadline (`container.go:95-108`, `container.go:17`). When the container removal fails or overruns, the network removal fails too, because the container is still attached. The runner only logs that error (`runner/runner.go:183-187`).+- The runner reports a failure from inside the job run, and the teardown runs later as a deferred call (`runner/runner.go:178`, `runner/runner.go:183`). The pool releases the job slot only after the run returns (`runner/pool.go:34`). With more than one slot, another slot can therefore claim the requeued attempt while the previous attempt is still tearing down. This mechanism is inferred from the code and was not reproduced.+- The startup purge lists containers by `LabelAgentID` and removes each container's first attached network (`purge.go:26-58`, `dockerengine.go:316-345`). A network with no container attached is invisible to it, which contradicts its own comment at `purge.go:21-23`.+- `Manager` holds the concrete `*dockerEngine` (`manager.go:55-61`). The purge stays unit-testable because it accepts the narrow `purgeEngine` interface (`purge.go:11-15`), faked in `fake_engine_test.go`.++The approach answers to `agent/verification/AGENTS.md`: test naming `Test_WhatWeDo_WhichConditions_WhatWeExpect`, tests beside the code, shared test helpers in `internal/testutil` (a package `testing.go` cannot see a fake defined in a `_test.go` file, and the lint skips test files, so unexported test-only helpers there would fail it), IDs as key-value pairs in logs, and the `slices` helpers instead of manual loops. No ADR in `adr/` covers the verification agent's Docker resources.++## Goals / Non-Goals++**Goals:**++- One removal routine serves both the startup purge and the cleanup before an attempt, and the two differ only in the label filter and in how they treat a failure.+- The removal is idempotent: a resource that is already gone counts as removed, so a racing teardown cannot fail the new attempt.++**Non-Goals:**++- Detecting why a previous attempt leaked. The cleanup makes the retry independent of the cause.+- Changing the teardown deadline or the order of reporting and teardown in the runner.++## Decisions++### Clear leftovers by label before every attempt++`Spawn` removes the containers and then the networks labeled with this agent's ID and the attempt's verification ID, before `CreateNetwork`.++Alternatives considered:++- Remove only an existing network with the same name. Rejected: in the suspected leak the old container is still attached, so Docker refuses to remove the network and the retry fails exactly as before.+- Reuse an existing network instead of creating one. Rejected: it keeps a half-torn-down container on the network and makes the attempt depend on the state of the previous one.+- Add the attempt number to the network name. Rejected: the agent never learns the attempt number from the assignment, and every attempt would leak another network instead of the first one blocking the rest.++### One label-filtered removal routine for both paths++A single function lists this agent's containers and networks that match a label filter, removes the containers, then removes the networks. The startup purge calls it with the agent label only. `Spawn` adds the verification label. The purge no longer derives networks from containers. `ListManaged(ctx, agentID)` (`dockerengine.go:316-345`) is replaced by a container listing that takes the label set, beside a matching network listing, so no agent-only listing is kept beside the label-filtered one. The listings return only what the routine reads, the resource ID and its verification ID for logging, so `ManagedContainer` loses `NetworkID` and `CreatedUnix`, and its network lookup goes away. The two comments that tie the labels to the startup purge alone (`dockerengine.go:26-27`, `manager.go:146-148`) are rewritten, because the cleanup before an attempt depends on both labels too.++At startup the routine lists and removes containers, then lists and removes networks, and a failure in one half does not skip the other. After a failed container listing, a network that still has a container attached fails its removal and is logged, which is the purge's existing best-effort behavior.++Alternatives considered:++- Keep the container-driven purge and add a second pass over orphaned networks. Rejected: it removes the same networks through two paths and keeps the field that hides orphaned networks.+- Write a separate cleanup for `Spawn`. Rejected: it would duplicate the list-and-remove logic and its not-found handling.++### Failure handling differs by path++The startup purge stays best-effort: it logs a failed listing or removal and moves on, so a transient Docker error never blocks startup (`purge.go:17-24`). The cleanup before an attempt returns an error, and `Spawn` fails with a message about the leftovers of an earlier attempt. That failure is still a pre-restore setup failure, so the backend requeues it as before.++Alternative considered: log the failure and let `CreateNetwork` proceed. Rejected: when the leftover survived, the attempt fails anyway with the misleading "already exists" error that this change removes.++### Treat "not found" as removed++The removal of a container or network that Docker reports as missing counts as success. Docker reports this condition as an error that `github.com/containerd/errdefs` classifies with `IsNotFound`. The removal routine applies this check, not the Docker engine, so a fake engine returning `errdefs.ErrNotFound` exercises it in unit tests. The module is already in `agent/verification/go.mod` as an indirect dependency and becomes a direct one, with no new download.++Alternative considered: match the error text. Rejected: the text is not a stable contract of the Docker API.++### Test through the narrow engine interface++The removal routine takes an interface with the listing and removal methods it uses, which extends the existing `purgeEngine` pattern. The step of `Spawn` that clears leftovers and then creates the network moves into a function that takes the same interface plus `CreateNetwork`, because `Spawn` holds the concrete engine and cannot be unit-tested. That lets a test pin that the cleanup runs before the network is created, and that no network is created when the cleanup fails.++The fake engine stores containers and networks with their labels and applies the requested label filter with AND semantics, as Docker does. A test then asserts which resources remain on the fake host, so the scenarios that leave other agents' and other verifications' resources untouched check the outcome rather than the filter arguments. One test pins the label filter values that the routine requests, since the fake's matching only proves the filter is right if the routine asks for the right labels. The Docker engine's new listing method stays thin, and the existing tests do not exercise the engine against a real daemon either.++Real Docker is covered by the agent's existing end-to-end suite rather than by unit tests. `agent/verification/e2e` runs the built agent against a mock backend with the host's `/var/run/docker.sock` mounted, the mock takes a chosen verification ID through `POST /mock/set-claim`, `test-startup-purge.sh` already seeds a labeled leftover, and `leak_check` in `e2e/scripts/lib.sh` already checks networks. Two scenarios join it: a retry against a pre-created `databasus-verif-<id>` network, and an orphaned network plus a foreign-agent network at startup. They check the label filter semantics and the not-found errors of a real daemon, which the fake only imitates.++Alternative considered: a Docker-backed test inside `just test`. Rejected: `just test` runs no Docker-backed tests today, and the e2e suite already provides that environment. The e2e checks stay out of the change verification because `just e2e` rebuilds its images without cache and runs every scenario, and the change verification section allows only the unit test command.++## Risks / Trade-offs++- [The previous attempt of the same verification is still running in another slot when the cleanup removes its container] → That attempt is stale, since the backend already requeued the verification, so ending it is the desired outcome. Its failure report may reach the backend after the retry starts, and this change leaves the backend's handling of such a report as it is.+- [Docker applies label filters with AND semantics only when each label is a separate filter value] → The listing passes the agent label and the verification label as two values of the `label` filter, which Docker combines with AND. A unit test pins the filter the routine requests.+- [A leftover labeled with a different agent ID still collides by name] → Out of scope per proposal.md. The failure then keeps today's message.++## Migration Plan++No data or config migration. The first startup after the upgrade removes networks that earlier agent versions orphaned. Rolling back restores the previous behavior with no cleanup needed.diff --git a/openspec/changes/verification-retry-leftover-cleanup/proposal.md b/openspec/changes/verification-retry-leftover-cleanup/proposal.mdnew file mode 100644index 00000000..68f97de7--- /dev/null+++ b/openspec/changes/verification-retry-leftover-cleanup/proposal.md@@ -0,0 +1,36 @@+## Why++When a verification attempt fails for an agent-side reason, the backend requeues it under the same verification ID, and the agent names the isolated Docker network of every attempt after that ID. If the previous attempt left its network behind, each retry fails with "network with name databasus-verif-<id> already exists" until the retry limit runs out, and the user sees this network error instead of the cause of the first failure (issue #832). Restarting the agent does not help, because the startup purge finds networks only through the containers attached to them.++## What Changes++- Before it creates the network and container of an attempt, the agent removes whatever an earlier attempt of the same verification left on the Docker host under this agent's ID: the container first, then the network. A resource that is already gone counts as removed.+- If such a leftover cannot be removed, the attempt fails with an error that names the leftover cleanup, instead of the misleading "network already exists".+- The startup purge also removes this agent's networks that no container is attached to, found by the agent's label rather than through containers.+- The comment on the startup purge that says a leftover never collides with a fresh run is corrected, because a retry reuses the verification ID.++No caller, config, schema or API changes, and nothing is **BREAKING**.++## Out of scope++- Retrying the network removal inside the container teardown when the container removal has not finished in time.+- Backend changes: the requeue logic, the attempt limit, the verification ID reuse and the failure reporting.+- Reordering the failure report and the container teardown inside a job run.+- Leftovers labeled with a different agent ID that collide by name on a shared Docker host.+- UI changes, including how the first failure's cause is shown.++## Capabilities++### New Capabilities++- `verification-container-cleanup`: how the verification agent clears Docker containers and networks left by earlier attempts and earlier agent processes, so that they never block a new verification attempt.++### Modified Capabilities++None.++## Impact++- Code: the container feature of the verification agent (`agent/verification/internal/features/container/`), namely the job spawn, the startup purge and the Docker engine methods that list and remove managed resources.+- Rules: the change answers to `agent/verification/AGENTS.md` and the root `AGENTS.md`.+- Operations: the first startup after the upgrade also removes networks orphaned by earlier agent versions.diff --git a/openspec/changes/verification-retry-leftover-cleanup/specs/verification-container-cleanup/spec.md b/openspec/changes/verification-retry-leftover-cleanup/specs/verification-container-cleanup/spec.mdnew file mode 100644index 00000000..a580cb4d--- /dev/null+++ b/openspec/changes/verification-retry-leftover-cleanup/specs/verification-container-cleanup/spec.md@@ -0,0 +1,108 @@+## Purpose++Defines how the verification agent clears the Docker containers and networks that earlier verification attempts and earlier agent processes left on the host, so that such leftovers never block a new verification attempt.++## ADDED Requirements++### Requirement: A retried verification is not blocked by leftovers of its earlier attempt++Before the verification agent creates the container and network of a verification attempt, it SHALL remove every container and network that it labeled for the same verification in an earlier attempt. A container or network that disappears before the agent removes it SHALL count as removed.++#### Scenario: The previous attempt left its network behind++- **WHEN** an attempt of a verification failed and left its network on the Docker host without a container attached+- **AND** the backend assigns the same verification to the agent again+- **THEN** the agent removes the leftover network+- **AND** the new attempt creates its network and proceeds to the restore++#### Scenario: The previous attempt left its container and network behind++- **WHEN** an earlier attempt of the verification left both its container and its network on the Docker host+- **AND** the agent starts a new attempt of the same verification+- **THEN** the agent removes the leftover container and then the leftover network+- **AND** the new attempt proceeds to the restore++#### Scenario: The leftover is removed concurrently++- **WHEN** the teardown of the previous attempt removes its network while the new attempt is clearing the same network+- **THEN** the new attempt treats the network as removed and proceeds to the restore++#### Scenario: No leftovers exist++- **WHEN** the agent starts the first attempt of a verification+- **THEN** the attempt proceeds to the restore without removing anything++### Requirement: A leftover that cannot be removed is reported as such++When the verification agent cannot list or cannot remove the leftovers of an earlier attempt of the same verification, it SHALL fail the new attempt before the restore and before it creates the attempt's network. The failure message SHALL say that the leftovers of an earlier attempt could not be removed, rather than that the network name is taken.++#### Scenario: Docker refuses to remove the leftover++- **WHEN** the Docker host refuses to remove a leftover container or network of an earlier attempt of the verification+- **THEN** the agent reports the attempt as failed before the restore starts+- **AND** the failure message says that the leftovers of an earlier attempt could not be removed+- **AND** the agent creates no network for the attempt++#### Scenario: Docker cannot list the leftovers++- **WHEN** the Docker host fails to list the containers or the networks of the verification before an attempt+- **THEN** the agent reports the attempt as failed before the restore starts+- **AND** the failure message says that the leftovers of an earlier attempt could not be removed+- **AND** the agent creates no network for the attempt++### Requirement: Agent startup removes orphaned networks++At startup, before it claims any verification, the verification agent SHALL remove every container and then every network labeled with its agent ID, including networks that no container is attached to. A container or network that disappears before the agent removes it SHALL count as removed. As before, a failure to list or remove a leftover SHALL be logged and SHALL NOT prevent the agent from starting. A failure with one kind of resource SHALL NOT stop the agent from clearing the other kind or the remaining resources.++#### Scenario: A network without a container remains from an earlier process++- **WHEN** the agent starts and the Docker host holds a network labeled with this agent's ID that no container is attached to+- **THEN** the agent removes that network before it claims a verification++#### Scenario: A container is still attached to the agent's network++- **WHEN** the agent starts and the Docker host holds a container labeled with this agent's ID that is attached to a network with the same label+- **THEN** the agent removes the container and then the network++#### Scenario: A leftover disappears during the purge++- **WHEN** the agent starts and a container or network labeled with this agent's ID disappears before the agent removes it+- **THEN** the agent treats it as removed and goes on with the remaining resources++#### Scenario: Docker cannot list the networks++- **WHEN** the agent starts and the Docker host fails to list networks+- **THEN** the agent logs a warning+- **AND** the agent still removes the containers labeled with its agent ID+- **AND** the agent starts normally++#### Scenario: Docker cannot list the containers++- **WHEN** the agent starts and the Docker host fails to list containers+- **THEN** the agent logs a warning+- **AND** the agent still tries to remove the networks labeled with its agent ID, logging each network that Docker refuses to remove+- **AND** the agent starts normally++### Requirement: Cleanup touches only the agent's own resources++The verification agent SHALL remove only containers and networks labeled with its own agent ID. The cleanup before an attempt SHALL additionally limit itself to resources labeled with the verification of that attempt. Containers and networks of other agents, of other verifications and of unrelated workloads on the same Docker host SHALL stay untouched.++#### Scenario: Another verification runs concurrently on the same agent++- **WHEN** the agent starts a retried attempt of one verification while a different verification runs in another slot+- **THEN** the container and network of the running verification stay untouched++#### Scenario: Another agent left resources for the same verification++- **WHEN** the agent starts an attempt of a verification and the Docker host holds a container and a network labeled with the same verification but a different agent ID+- **THEN** the agent leaves that container and network in place++#### Scenario: Another agent shares the Docker host++- **WHEN** the agent starts and the Docker host holds a network labeled with a different agent ID+- **THEN** the agent leaves that network in place++#### Scenario: An unrelated network shares the Docker host++- **WHEN** the agent starts and the Docker host holds a network without the agent's labels+- **THEN** the agent leaves that network in placediff --git a/openspec/changes/verification-retry-leftover-cleanup/tasks.md b/openspec/changes/verification-retry-leftover-cleanup/tasks.mdnew file mode 100644index 00000000..ad3a12f3--- /dev/null+++ b/openspec/changes/verification-retry-leftover-cleanup/tasks.md@@ -0,0 +1,49 @@+## 1. Label-filtered removal routine++- [x] 1.1 Replace the engine's `ListManaged(ctx, agentID)` with a container listing that takes a label set, add a network listing with the same signature, and pass every label as a separate `label` filter value. Both return only the resource ID and its verification ID, so cut `ManagedContainer` down to those fields and delete the network lookup. Until task 1.2 replaces it, the old purge compiles against the new container listing and skips its network step. Verify with `just build` in `agent/verification`.+- [x] 1.2 In one step, replace the container-driven startup purge with one routine and the test fake with a fake Docker host, then rewrite the existing purge tests against it. The routine lists containers by labels and removes them, then lists networks by labels and removes them, so a failure in one half does not skip the other. It treats a removal error that `IsNotFound` from `github.com/containerd/errdefs` recognizes as success and accepts the narrow engine interface. Make `github.com/containerd/errdefs` a direct dependency in `agent/verification/go.mod`. The fake host in `fake_engine_test.go` stores containers and networks with their labels and applies a label filter with AND semantics. It also implements `CreateNetwork` and records listing filters, removals and network creations in one ordered log, and it can be told to fail a listing, fail a removal or return `errdefs.ErrNotFound` for a removal. The fake's setup methods stay beside it in `fake_engine_test.go`, because a `testing.go` file is compiled without the test files and cannot see the fake. The label builders the tests share are the production helpers `Spawn` and the purge use, and the log-capturing helper joins `DiscardLogger` in the module's shared `internal/testutil`, so the package gets no `testing.go`. Rewrite the comments that tie the labels to the startup purge alone: the purge comment that says a leftover never collides with a fresh run, `dockerengine.go:26-27` and `manager.go:146-148`. Verify by rewriting the existing tests in `internal/features/container/purge_test.go` against the fake host and running `just test` in `agent/verification`.++## 2. Startup purge tests++Each test lives in `internal/features/container/purge_test.go` and is verified with `just test` in `agent/verification`.++- [x] 2.1 `Test_PurgeAgentContainers_WhenNetworkHasNoContainer_RemovesNetwork` pins that a network labeled with the agent's ID and no attached container is removed.+- [x] 2.2 `Test_PurgeAgentContainers_WhenContainerAndNetworkExist_RemovesContainerBeforeNetwork` pins the removal order.+- [x] 2.3 `Test_PurgeAgentContainers_WhenResourcesBelongToOtherAgentOrNoAgent_LeavesThemInPlace` pins that a network and a container labeled with another agent ID, and a network with no agent labels, remain on the fake host.+- [x] 2.4 `Test_PurgeAgentContainers_FiltersByAgentLabelOnly` pins that both listings request exactly the agent label filter.+- [x] 2.5 `Test_PurgeAgentContainers_WhenNetworkListFails_LogsAndRemovesContainers` pins that a failed network listing does not stop the purge and does not return an error.+- [x] 2.6 `Test_PurgeAgentContainers_WhenRemovalReturnsNotFound_ContinuesWithRemainingResources` pins that a not-found removal is treated as removed and the purge goes on.+- [x] 2.7 `Test_PurgeAgentContainers_WhenContainerListFails_LogsAndRemovesNetworks` pins that a failed container listing still lets the purge remove the agent's networks.++## 3. Cleanup before an attempt++- [x] 3.1 Move the step of `Spawn` that clears leftovers and creates the network into a function that takes the narrow engine interface plus `CreateNetwork`, runs the removal routine with the agent and verification labels, and returns an error saying the leftovers of an earlier attempt could not be removed when the routine fails. Make `Spawn` call it. Verify with `just build` in `agent/verification`.++## 4. Cleanup-before-attempt tests++Each test lives in a new `internal/features/container/leftovers_test.go`, or beside the function from 3.1 if it lands in another file, and is verified with `just test` in `agent/verification`.++- [x] 4.1 `Test_PrepareAttemptNetwork_WhenPreviousAttemptLeftNetwork_RemovesItAndCreatesNetwork` pins the first scenario of the retry requirement. A comment on the test names issue #832, which the project rules allow for a test that pins an issue's behavior.+- [x] 4.2 `Test_PrepareAttemptNetwork_WhenPreviousAttemptLeftContainerAndNetwork_RemovesContainerThenNetworkThenCreates` pins the order across removal and creation.+- [x] 4.3 `Test_PrepareAttemptNetwork_WhenLeftoverAlreadyRemoved_CreatesNetwork` pins that a removal returning `errdefs.ErrNotFound` lets the attempt proceed.+- [x] 4.4 `Test_PrepareAttemptNetwork_WhenNoLeftovers_CreatesNetworkWithoutRemovals` pins the first-attempt scenario.+- [x] 4.5 `Test_PrepareAttemptNetwork_WhenLeftoverRemovalFails_ReturnsLeftoverErrorAndCreatesNoNetwork` pins the error message and that `CreateNetwork` is never called.+- [x] 4.6 `Test_PrepareAttemptNetwork_WhenListingFails_ReturnsLeftoverErrorAndCreatesNoNetwork` pins the same behavior for a failed listing.+- [x] 4.7 `Test_PrepareAttemptNetwork_WhenOtherVerificationRuns_LeavesItsResourcesInPlace` pins that the container and network of another verification of the same agent remain on the fake host.+- [x] 4.8 `Test_PrepareAttemptNetwork_WhenOtherAgentHasSameVerification_LeavesItsResourcesInPlace` pins that resources with the same verification label but another agent ID remain on the fake host.+- [x] 4.9 `Test_PrepareAttemptNetwork_FiltersByAgentAndVerificationLabels` pins that both listings request exactly the agent label and the verification label.++## 5. End-to-end checks on real Docker++- [x] 5.1 Add `agent/verification/e2e/scripts/test-restore-retry-leftover.sh`, modeled on `test-restore-success.sh`. After the agent has started and its startup purge has finished, and before the claim, it creates a network named `databasus-verif-<id>` labeled with the agent ID and that verification ID, so the startup purge cannot remove it first. It asserts with `docker network ls --filter name=databasus-verif-<id>` that the network exists right before it assigns the same verification ID through `POST /mock/set-claim`, then asserts a successful report and runs `leak_check`. A comment names issue #832. Register the script in `e2e/scripts/run-all.sh`. Verify with `just e2e` in `agent/verification`.+- [x] 5.2 Extend `agent/verification/e2e/scripts/test-startup-purge.sh` to also seed a network labeled with the agent ID and no container, and a network labeled with another agent ID. Assert that the first is gone after the agent starts and the second remains, then remove the second in the script's cleanup. Verify with `just e2e` in `agent/verification`.++## 6. Module checks++- [x] 6.1 Run `just lint` in `agent/verification` and fix every finding.++## Change verification++```sh+just --justfile agent/verification/Justfile test+```