An open-source, self-hostable PaaS alternative to Vercel, Heroku & Netlify that lets you easily deploy static sites, databases, full-stack applications and 360+ one-click services on your own servers.
PHP coolifydatabasesdeploymentdockerdocker-composeinertiajslaravelmariadbmysqlnextjsnodejsphppostgresredisself-hostedself-hostingserverstatic-sitesveltesvelte5
fix(applications): stop again while a container restarts past its limit
Andras Bacsai5a08c35e7d94
diff --git a/.ai/lessons.md b/.ai/lessons.mdindex 11ca067b78..dfe393876b 100644--- a/.ai/lessons.md+++ b/.ai/lessons.md@@ -28,7 +28,8 @@- To assert that the unsaved-changes bar shows, check `el.classList.contains('is-dirty')`. Playwright treats the hidden bar (`opacity-0`) as visible, and its class attribute always contains the text `is-dirty` from Tailwind variants.- For load tests against a dev VM, use your own instance (`COOLIFY_DEV_INSTANCE=<name> COOLIFY_DEV_SIDE_BY_SIDE=1 ./scripts/dev start`); other sessions can reseed the localhost VM of a shared instance. The 1 GiB VM drops packets when its conntrack table is full (`dmesg`: `nf_conntrack: table full`), so raise `net.netfilter.nf_conntrack_max` on it and run an idle load first to prove a clean baseline.- Do not call `sleep()` in a browser test. The app server runs in the test process, so `sleep()` stops it; use `$page->wait()`.-- `Queue::fake()` also captures `dispatchSync()` of a `ShouldQueue` job, so the job never runs. Fake only the jobs to block (`Queue::fake([SomeJob::class])`).+- `Queue::fake()` also captures `dispatchSync()` of a `ShouldQueue` job, so the job never runs. Fake only the jobs to block (`Queue::fake([SomeJob::class])`). Exception: actions (`AsAction`) are queued as `JobDecorator`, so `Queue::fake([SomeAction::class])` does not stop them; use `Queue::fake()` and `SomeAction::assertPushed()`.+- `Notification::fake()` records nothing for a team that has no enabled channel. Turn one on (for example `emailNotificationSettings()->update(['use_instance_email_settings' => true, '<event>_email_notifications' => true])`) before `assertSentTo`.- In tests, turning on `is_metrics_enabled` (and other Sentinel settings) with `save()`/`update()` runs the settings hook, which starts Sentinel over a real SSH connection on the sync queue. Use `saveQuietly()` or `Queue::fake()`.- Call `visit()` directly in each `tests/v4/Browser` test body; Pest does not mark a test that only uses helper-wrapped `visit()` as a browser test, so it fails with `sendText() on null`.- The root team (id 0) has `personal_team = true`, so `personal_team` does not identify a user's own team. To clean up test users, detach them from shared teams and then delete the user; never delete a team selected by a filter. A Team delete revokes the team's API tokens and deletes its private keys before any guard can stop it.diff --git a/app/Actions/Docker/GetContainersStatus.php b/app/Actions/Docker/GetContainersStatus.phpindex 5ce2a90f6e..7faeb37b78 100644--- a/app/Actions/Docker/GetContainersStatus.php+++ b/app/Actions/Docker/GetContainersStatus.php@@ -529,7 +529,9 @@ class GetContainersStatus implements ShouldBeUnique->where('restart_limit_reached', false)->update(['restart_limit_reached' => true]) === 1;- if ($restartLimitClaimed) {+ // Stop again while a container still runs: an earlier stop can fail or come too late.+ $stillRunning = $containerStatuses->contains(fn (string $status): bool => ! str($status)->contains('exited'));+ if ($restartLimitClaimed || $stillRunning) {$application->refresh();StopApplication::dispatch(application: $application,@@ -538,6 +540,8 @@ class GetContainersStatus implements ShouldBeUniqueresetRestartCount: false,removeContainers: false,);+ }+ if ($restartLimitClaimed) {$application->environment->project->team?->notify(new ApplicationRestartLimitReached($application));}}diff --git a/app/Jobs/PushServerUpdateJob.php b/app/Jobs/PushServerUpdateJob.phpindex 9a9fd8a56c..92df6bd8c6 100644--- a/app/Jobs/PushServerUpdateJob.php+++ b/app/Jobs/PushServerUpdateJob.php@@ -614,7 +614,9 @@ class PushServerUpdateJob implements ShouldBeEncrypted, ShouldQueue, Silenced->where('restart_limit_reached', false)->update(['restart_limit_reached' => true]) === 1;- if ($restartLimitClaimed) {+ // Stop again while a container still runs: an earlier stop can fail or come too late.+ $stillRunning = $containerStatuses->contains(fn (string $status): bool => ! str($status)->contains('exited'));+ if ($restartLimitClaimed || $stillRunning) {$application->refresh();StopApplication::dispatch(application: $application,@@ -623,6 +625,8 @@ class PushServerUpdateJob implements ShouldBeEncrypted, ShouldQueue, SilencedresetRestartCount: false,removeContainers: false,);+ }+ if ($restartLimitClaimed) {$application->environment->project->team?->notify(new ApplicationRestartLimitReached($application));}}diff --git a/tests/Feature/ApplicationRestartLimitEnforcementTest.php b/tests/Feature/ApplicationRestartLimitEnforcementTest.phpnew file mode 100644index 0000000000..412938fa0d--- /dev/null+++ b/tests/Feature/ApplicationRestartLimitEnforcementTest.php@@ -0,0 +1,103 @@+<?php++use App\Actions\Application\StopApplication;+use App\Actions\Docker\GetContainersStatus;+use App\Jobs\PushServerUpdateJob;+use App\Models\Application;+use App\Models\Environment;+use App\Models\InstanceSettings;+use App\Models\Project;+use App\Models\Server;+use App\Models\StandaloneDocker;+use App\Models\Team;+use App\Notifications\Application\RestartLimitReached;+use Illuminate\Foundation\Testing\RefreshDatabase;+use Illuminate\Support\Facades\Notification;+use Illuminate\Support\Facades\Queue;++uses(RefreshDatabase::class);++beforeEach(function () {+ Queue::fake();+ Notification::fake();+ InstanceSettings::forceCreate(['id' => 0, 'fqdn' => 'https://coolify.test']);++ $this->team = Team::factory()->create();+ $this->team->emailNotificationSettings()->update([+ 'use_instance_email_settings' => true,+ 'restart_limit_reached_email_notifications' => true,+ ]);+ $this->server = Server::factory()->create(['team_id' => $this->team->id]);+ $this->server->settings()->update(['is_reachable' => true, 'is_usable' => true]);+ $destination = StandaloneDocker::where('server_id', $this->server->id)->firstOrFail();+ $project = Project::factory()->create(['team_id' => $this->team->id]);+ $environment = $project->environments()->first()+ ?? Environment::factory()->create(['project_id' => $project->id]);+ $this->application = Application::factory()->create([+ 'environment_id' => $environment->id,+ 'destination_id' => $destination->id,+ 'destination_type' => $destination->getMorphClass(),+ 'status' => 'restarting',+ ]);+ $this->application->forceFill(['max_restart_count' => 3, 'container_present' => true])->save();+});++/**+ * Reports one container of the application with the given state and Docker restart count.+ */+function reportApplicationContainer(string $path, Application $application, Server $server, string $state, int $restartCount): void+{+ $labels = [+ 'coolify.managed' => 'true',+ 'coolify.applicationUuid' => $application->uuid,+ 'coolify.pullRequestId' => '0',+ 'com.docker.compose.project' => $application->uuid,+ 'com.docker.compose.service' => $application->uuid,+ ];++ if ($path === 'sentinel') {+ (new PushServerUpdateJob($server, ['containers' => [[+ 'name' => $application->uuid,+ 'state' => $state,+ 'health_status' => 'unhealthy',+ 'restart_count' => $restartCount,+ 'labels' => $labels,+ ]]]))->handle();++ return;+ }++ GetContainersStatus::run($server->fresh(), collect([[+ 'Name' => '/'.$application->uuid,+ 'State' => ['Status' => $state],+ 'RestartCount' => $restartCount,+ 'Config' => ['Labels' => $labels],+ ]]), collect());+}++it('stops the application and notifies once when the restart limit is reached', function (string $path) {+ reportApplicationContainer($path, $this->application, $this->server, 'restarting', 3);++ expect($this->application->fresh()->restart_limit_reached)->toBeTrue();+ StopApplication::assertPushed(1);+ Notification::assertSentToTimes($this->team, RestartLimitReached::class, 1);+})->with(['sentinel', 'status check']);++it('stops the application again when it still restarts after the restart limit was reached', function (string $path) {+ $this->application->forceFill(['restart_count' => 3, 'restart_limit_reached' => true])->save();++ reportApplicationContainer($path, $this->application, $this->server, 'restarting', 5);++ StopApplication::assertPushed(1);+ Notification::assertNothingSent();+})->with(['sentinel', 'status check']);++it('does not stop the application again after its containers exited at the restart limit', function (string $path) {+ $this->application->forceFill(['status' => 'exited', 'restart_count' => 3, 'restart_limit_reached' => true])->save();++ reportApplicationContainer($path, $this->application, $this->server, 'exited', 3);++ StopApplication::assertNotPushed();+ Notification::assertNothingSent();+ expect($this->application->fresh()->stoppedAfterRestartLimit())->toBeTrue();+})->with(['sentinel', 'status check']);