An open-source, self-hostable PaaS alternative to Vercel, Heroku & Netlify that lets you easily deploy static sites, databases, full-stack applications and 360+ one-click services on your own servers.
PHP coolifydatabasesdeploymentdockerdocker-composeinertiajslaravelmariadbmysqlnextjsnodejsphppostgresredisself-hostedself-hostingserverstatic-sitesveltesvelte5
fix(security): secure remote command secrets and clone volume data
Andras Bacsai1a2a5362ad19
diff --git a/.ai/lessons.md b/.ai/lessons.mdindex 130a87bc6b..d6db4e49b2 100644--- a/.ai/lessons.md+++ b/.ai/lessons.md@@ -21,6 +21,7 @@- Redirect browser test output to a file (`> /tmp/x.log 2>&1`); piping it (`| tail`) hangs because the Playwright server keeps the pipe open.- Browser `click()`/`assertSee()` wait forever when nothing matches, so a bad selector hangs the run instead of failing it. `click('Text')` can match a hidden element first (use `button:visible:has-text("Text")`); `:text-is()` on an element with child elements and `assertSee()` text with double quotes never match. Prefer plain CSS selectors. `click()` also hung on a button far down the long `/analytics` page; click it with `$page->script("document.querySelector('...').click()")` there.- A new worktree has no `vendor/` or `public/build/`. Run `composer install --ignore-platform-req=ext-zip` on the host and copy `public/build` from the main checkout before page-rendering tests; otherwise they fail with a Vite manifest error.+- A running dev instance mounts the checkout, and its scheduled `CheckForUpdatesJob` writes the CDN data to `base_path('versions.json')`. Check `git diff versions.json` before you commit a version change.- Never run `php artisan test` inside a dev container (`./scripts/dev exec` or `docker exec coolify-dev-*`). The container sets `DB_CONNECTION=pgsql` as a real environment variable, which wins over `phpunit.xml`, so `RefreshDatabase` runs `migrate:fresh` on the dev database and wipes it. Run tests on the host only.- The helper image runs `docker compose` for deployments and for the Coolify upgrade itself. Before you bump its Docker CLI or Compose, test them against the oldest supported Docker Engine (for example a `docker:<version>-dind` container); newer clients can reject data from older daemons.- Do not symlink `vendor` into another worktree. Composer resolves the base path from the real `vendor` path, so the tests load the app classes of the other checkout. Run `composer install --ignore-platform-req=ext-zip` in the worktree.@@ -41,6 +42,8 @@- Horizon workers in a dev instance keep the code they loaded. Run `./scripts/dev exec php artisan horizon:terminate` after a code change and before a live test of queued jobs.- To prove that a server side effect is blocked, check the server itself after the request. A model hook or job can be one of several write paths (for example `ServerStorageSaveJob` and `ServerFilesFromServerJob` both write file storages).+- When you add a gate that lets a user through only after a step (2FA, password reset, onboarding), test it with users who have no known password (OAuth, magic-link invites) and together with the other redirecting gates. Two gates that each allow only their own page make a redirect loop, and a step that needs password confirmation blocks users who cannot confirm one.+## Preserve product scope- Do not replace required SPA navigation with a full-page redirect to hide a lifecycle or ordering defect.- Do not add billing restrictions, live reconciliation, or fallback behavior unless the request includes them.diff --git a/app/Actions/CoolifyTask/RunRemoteProcess.php b/app/Actions/CoolifyTask/RunRemoteProcess.phpindex 4f525ebcc8..f2c21ca121 100644--- a/app/Actions/CoolifyTask/RunRemoteProcess.php+++ b/app/Actions/CoolifyTask/RunRemoteProcess.php@@ -113,7 +113,8 @@ class RunRemoteProcess$status = ProcessStatus::IN_PROGRESS;$timeout = config('constants.ssh.command_timeout');try {- $process = Process::timeout($timeout)->start($this->getCommand(), $this->handleOutput(...));+ [$command, $input] = $this->getCommand();+ $process = Process::timeout($timeout)->input($input)->start($command, $this->handleOutput(...));// Start from the stored properties and save now, so later output writes only change the log and// never write back an old copy over a status that Coolify set meanwhile (for example a stopped import).$stored = Activity::query()->whereKey($this->activity->getKey())->first()?->properties ?? $this->activity->properties;@@ -214,13 +215,19 @@ class RunRemoteProcessevent(new $eventClass($eventData));}- protected function getCommand(): string+ /**+ * The SSH command and its input. The command batch goes to ssh on stdin, so its secrets are+ * not in a command line on the Coolify host.+ *+ * @return array{0: string, 1: string}+ */+ protected function getCommand(): array{$server_uuid = $this->activity->getExtraProperty('server_uuid');$command = RemoteProcessCommand::read($this->activity) ?? throw new \RuntimeException('The command of this task was already removed.');$server = Server::whereUuid($server_uuid)->firstOrFail();- return SshMultiplexingHelper::generateSshCommand($server, $command);+ return [SshMultiplexingHelper::generateSshBatchCommand($server), SshMultiplexingHelper::sshBatchInput($command)];}protected function handleOutput(string $type, string $output)diff --git a/app/Actions/Database/StartDatabaseImport.php b/app/Actions/Database/StartDatabaseImport.phpindex daa74c98bd..9d8add6b8f 100644--- a/app/Actions/Database/StartDatabaseImport.php+++ b/app/Actions/Database/StartDatabaseImport.php@@ -2,6 +2,8 @@namespace App\Actions\Database;+use App\Helpers\SshMultiplexingHelper;+use App\Helpers\SshRetryHandler;use App\Models\S3Storage;use App\Models\Server;use App\Models\ServiceDatabase;@@ -157,10 +159,9 @@ class StartDatabaseImportthrow new DatabaseImportException('The S3 backup was not found or exceeds the 10 GiB limit.');}$helper = "s3-restore-{$operation}";- $this->startS3HelperWithEnv($storage, $server, $helper, $network);+ $this->startS3Helper($storage, $server, $helper, $network);$sourceArg = escapeshellarg("s3temp/{$storage->bucket}/{$key}");$commandList = [- 'docker exec '.escapeshellarg($helper).' sh -c '.escapeshellarg('mc alias set s3temp "$S3_ENDPOINT" "$S3_ACCESS_KEY" "$S3_SECRET_KEY"'),'docker exec '.escapeshellarg($helper).' mc cp '.$sourceArg.' /tmp/restore',...$this->streamFromHelper($helper, $container, $containerPath),'docker rm -f '.escapeshellarg($helper).' 2>/dev/null || true',@@ -289,7 +290,12 @@ class StartDatabaseImport];}- private function startS3HelperWithEnv(S3Storage $storage, Server $server, string $helper, string $network): void+ /**+ * Starts the S3 restore helper and sets its `s3temp` mc alias. The S3 keys go to mc through the+ * stdin of a separate SSH process, so they are not in the arguments of any process on Coolify or+ * on the server. The helper is removed when either step fails.+ */+ private function startS3Helper(S3Storage $storage, Server $server, string $helper, string $network): void{$image = escapeshellarg(coolifyHelperImage().':'.getHelperVersion());@@ -298,16 +304,33 @@ class StartDatabaseImport'docker rm -f '.escapeshellarg($helper).' 2>/dev/null || true','docker run -d --network '.escapeshellarg($network).' --name '.escapeshellarg($helper)- .' -e S3_ENDPOINT='.escapeshellarg((string) $storage->endpoint)- .' -e S3_ACCESS_KEY='.escapeshellarg((string) $storage->key)- .' -e S3_SECRET_KEY='.escapeshellarg((string) $storage->secret).' '.$storage->minioClientRegionOption().' '.$image.' sleep 86400',], $server);+ $this->setMinioClientAlias($storage, $server, $helper);} catch (Throwable) {instant_remote_process(['docker rm -f '.escapeshellarg($helper).' 2>/dev/null || true'], $server, throwError: false);throw new DatabaseImportException('Unable to start the S3 restore helper.');}}++ private function setMinioClientAlias(S3Storage $storage, Server $server, string $helper): void+ {+ $remoteCommand = ($server->isNonRoot() ? 'sudo ' : '').'docker exec -i '.escapeshellarg($helper)+ .' mc alias set s3temp '.escapeshellarg((string) $storage->endpoint).' >/dev/null';+ $timeout = (int) config('constants.ssh.command_timeout');++ SshRetryHandler::retry(+ function () use ($server, $remoteCommand, $storage, $timeout) {+ $sshCommand = SshMultiplexingHelper::generateSshStdinCommand($server, $remoteCommand, disableMultiplexing: true, commandTimeout: $timeout);+ $process = runSshProcess($sshCommand, $timeout, $storage->key."\n".$storage->secret."\n");++ if ($process->exitCode() !== 0) {+ excludeCertainErrors($process->errorOutput(), $process->exitCode());+ }+ },+ ['server' => $server->ip, 'function' => 'setMinioClientAlias'],+ );+ }}diff --git a/app/Actions/Proxy/CheckProxy.php b/app/Actions/Proxy/CheckProxy.phpindex a0419feb31..2b5a502977 100644--- a/app/Actions/Proxy/CheckProxy.php+++ b/app/Actions/Proxy/CheckProxy.php@@ -114,7 +114,7 @@ class CheckProxy$results = Process::concurrently(function ($pool) use ($server, $ports, $proxyContainerName) {foreach ($ports as $port) {$commands = $this->buildPortCheckCommands($server, $port, $proxyContainerName);- $pool->command($commands['ssh_command'])->timeout(10);+ $pool->command($commands['ssh_command'])->input($commands['input'])->timeout(10);}});@@ -152,10 +152,10 @@ class CheckProxyprivate function buildPortCheckCommands(Server $server, int $port, string $proxyContainerName): array{$portCheckScript = $this->buildPortCheckScript($port, $proxyContainerName);- $sshCommand = SshMultiplexingHelper::generateSshCommand($server, $portCheckScript);return [- 'ssh_command' => $sshCommand,+ 'ssh_command' => SshMultiplexingHelper::generateSshBatchCommand($server),+ 'input' => SshMultiplexingHelper::sshBatchInput($portCheckScript),'script' => $portCheckScript,];}diff --git a/app/Actions/Server/CleanupDocker.php b/app/Actions/Server/CleanupDocker.phpindex f132ebd17d..17b796a6df 100644--- a/app/Actions/Server/CleanupDocker.php+++ b/app/Actions/Server/CleanupDocker.php@@ -43,6 +43,11 @@ class CleanupDocker implements ShouldBeUnique*/public const STOP_CLEANUP_COOLDOWN = 3600;+ /**+ * Cleanup log command of the entry that records a skipped build cache prune.+ */+ public const BUILD_CACHE_KEPT_LOG_COMMAND = 'docker builder prune, docker buildx prune';+public int $jobTries = 1;/**@@ -87,8 +92,9 @@ class CleanupDocker implements ShouldBeUnique/*** The deployment that is queued or running on the server (as target or build server), if any.- * The cleanup prunes the build caches, which fails a build that uses them. A deployment that- * wrote no log for longer than the deployment timeout is stuck and does not count.+ * Pruning the build caches fails a build that uses them, so the cleanup keeps them while such a+ * deployment exists. A deployment that wrote no log for longer than the deployment timeout is+ * stuck and does not count.*/public static function activeDeployment(Server $server): ?ApplicationDeploymentQueue{@@ -99,10 +105,17 @@ class CleanupDocker implements ShouldBeUnique->first(['id', 'deployment_uuid']);}+ /**+ * The reason the build cache was kept, read from a cleanup log, if it was kept.+ */+ public static function buildCacheKeptReason(array $cleanupLog): ?string+ {+ return collect($cleanupLog)->firstWhere('command', self::BUILD_CACHE_KEPT_LOG_COMMAND)['output'] ?? null;+ }+/*** Cleanup after a resource stop: skipped when a cleanup ran recently, and the queued run- * only cleans when the disk usage is at or above the server cleanup threshold and no- * deployment is active on the server.+ * only cleans when the disk usage is at or above the server cleanup threshold.*/public static function dispatchAfterStop(Server $server): void{@@ -137,14 +150,15 @@ class CleanupDocker implements ShouldBeUnique];}- public function handle(Server $server, bool $deleteUnusedVolumes = false, bool $deleteUnusedNetworks = false, bool $onlyAboveThreshold = false)+ /**+ * @param bool $pruneBuildCacheDuringDeployment Also prune the build caches while a deployment is active+ * (a manual cleanup). Otherwise they stay until the next run.+ */+ public function handle(Server $server, bool $deleteUnusedVolumes = false, bool $deleteUnusedNetworks = false, bool $onlyAboveThreshold = false, bool $pruneBuildCacheDuringDeployment = false){$this->deadline = now()->getTimestamp() + self::REMOTE_COMMANDS_DEADLINE;if ($onlyAboveThreshold) {- if (self::activeDeployment($server)) {- return [];- }$diskUsage = $server->getDiskUsage();if (is_numeric($diskUsage) && (int) $diskUsage < $server->settings->docker_cleanup_threshold) {return [];@@ -177,12 +191,22 @@ class CleanupDocker implements ShouldBeUnique$commands = [$this->buildContainerPruneCommand(),$imagePruneCmd,- 'docker builder prune -af',- // The prune fails on a stopped builder, so start it first.- railpackBuilderHelperCommand(railpackBuildxMetadataVolume($server), $helperImageWithVersion, 'docker buildx inspect --bootstrap coolify-railpack >/dev/null 2>&1; docker buildx prune --builder coolify-railpack -af'),- $this->buildCoolifyImageCleanupCommand($helperImageVersion),];+ $activeDeployment = $pruneBuildCacheDuringDeployment ? null : self::activeDeployment($server);+ if (! $activeDeployment) {+ $commands[] = 'docker builder prune -af';+ // The prune fails on a stopped builder, so start it first.+ $commands[] = railpackBuilderHelperCommand(railpackBuildxMetadataVolume($server), $helperImageWithVersion, 'docker buildx inspect --bootstrap coolify-railpack >/dev/null 2>&1; docker buildx prune --builder coolify-railpack -af');+ } else {+ $cleanupLog[] = [+ 'command' => self::BUILD_CACHE_KEPT_LOG_COMMAND,+ 'output' => 'Skipped: the build cache was not pruned because deployment '.$activeDeployment->deployment_uuid.' is queued or running on this server.',+ ];+ }++ $commands[] = $this->buildCoolifyImageCleanupCommand($helperImageVersion);+if ($deleteUnusedVolumes) {$commands[] = 'docker volume prune -af';}diff --git a/app/Actions/Shared/CloneVolumeDataWithSourceStopped.php b/app/Actions/Shared/CloneVolumeDataWithSourceStopped.phpnew file mode 100644index 0000000000..b0ee519fd2--- /dev/null+++ b/app/Actions/Shared/CloneVolumeDataWithSourceStopped.php@@ -0,0 +1,333 @@+<?php++namespace App\Actions\Shared;++use App\Actions\Application\StopApplication;+use App\Actions\Database\StartDatabase;+use App\Actions\Database\StopDatabase;+use App\Actions\Service\StartService;+use App\Actions\Service\StopService;+use App\Enums\ApplicationDeploymentStatus;+use App\Jobs\VolumeCloneJob;+use App\Models\Application;+use App\Models\Service;+use App\Models\StandaloneClickhouse;+use App\Models\StandaloneDragonfly;+use App\Models\StandaloneKeydb;+use App\Models\StandaloneMariadb;+use App\Models\StandaloneMongodb;+use App\Models\StandaloneMysql;+use App\Models\StandalonePostgresql;+use App\Models\StandaloneRedis;+use App\Models\StandaloneSqlite;+use App\Notifications\Internal\GeneralNotification;+use Illuminate\Support\Facades\Bus;+use Illuminate\Support\Facades\DB;+use Illuminate\Support\Facades\Log;+use Lorisleiva\Actions\Concerns\AsAction;+use RuntimeException;+use Throwable;++/**+ * Copies the volume data of a clone in one queued chain. A running source is stopped before the+ * copy and started again afterwards, also when a step fails. A stopped source stays stopped.+ * When a step fails, the team gets a notification.+ */+class CloneVolumeDataWithSourceStopped+{+ use AsAction;++ /** Exception code of a failed stop step: no volume data was copied. */+ private const STOP_FAILED = 1001;++ /** Exception code of a failed stop of an application that took down some, not all, of its containers. */+ private const APPLICATION_PARTLY_STOPPED = 1002;++ /**+ * @param array<int, VolumeCloneJob> $volumeJobs+ */+ public function handle(+ Application|Service|StandalonePostgresql|StandaloneRedis|StandaloneMongodb|StandaloneMysql|StandaloneMariadb|StandaloneKeydb|StandaloneDragonfly|StandaloneClickhouse|StandaloneSqlite $source,+ array $volumeJobs,+ ): void {+ if ($volumeJobs === []) {+ return;+ }++ $wasRunning = filled($source->status) && ! $source->isExited();++ // One closure per line: a serialized closure is found by its line, so two closures on one line run the same code.+ $chain = $wasRunning ? [+ static fn () => self::stopSource($source),+ ...$volumeJobs,+ static fn () => self::startSource($source),+ ] : $volumeJobs;++ // The jobs load the new volume rows, so they must not run before the transaction of the caller commits.+ DB::afterCommit(fn () => Bus::chain($chain)+ ->onQueue('high')+ ->catch(static fn (Throwable $e) => self::recoverSource($source, $wasRunning, $e))+ ->dispatch());+ }++ /**+ * The policy ability that allows a user to stop and start the source.+ */+ public static function sourceAbility(+ Application|Service|StandaloneClickhouse|StandaloneDragonfly|StandaloneKeydb|StandaloneMariadb|StandaloneMongodb|StandaloneMysql|StandalonePostgresql|StandaloneRedis|StandaloneSqlite $source,+ ): string {+ return match (true) {+ $source instanceof Application => 'deploy',+ $source instanceof Service => 'stop',+ default => 'manage',+ };+ }++ /**+ * Stops the source and fails the chain, so that no copy runs, when the stop reports an error+ * or a container of the source still runs. The stop actions return their errors instead of throwing.+ */+ public static function stopSource(+ Application|Service|StandaloneClickhouse|StandaloneDragonfly|StandaloneKeydb|StandaloneMariadb|StandaloneMongodb|StandaloneMysql|StandalonePostgresql|StandaloneRedis|StandaloneSqlite $source,+ ): void {+ // A partly stopped application is redeployed, so remember which of its containers ran.+ $runningBefore = $source instanceof Application ? self::runningContainersOrNull($source) : null;+ $result = match (true) {+ $source instanceof Application => StopApplication::run($source, false, false),+ $source instanceof Service => StopService::run($source, dockerCleanup: false),+ default => StopDatabase::run($source, dockerCleanup: false),+ };+ if (filled($result) && $result !== 'Database stopped successfully') {+ throw new RuntimeException($result, self::failedStopCode($source, $runningBefore));+ }++ try {+ $runningContainers = self::runningContainers($source);+ } catch (Throwable $e) {+ throw new RuntimeException('Could not check that the containers stopped: '.$e->getMessage(), self::STOP_FAILED, $e);+ }+ if ($runningContainers !== []) {+ throw new RuntimeException('These containers still run: '.implode(', ', $runningContainers), self::failedStopCode($source, $runningBefore, $runningContainers));+ }+ }++ /**+ * An application whose stop took down some of its containers, while others still run, is+ * partly stopped. When the containers before the stop are unknown, it may be partly stopped.+ *+ * @param list<string>|null $runningBefore+ * @param list<string>|null $runningAfter+ */+ private static function failedStopCode(+ Application|Service|StandaloneClickhouse|StandaloneDragonfly|StandaloneKeydb|StandaloneMariadb|StandaloneMongodb|StandaloneMysql|StandalonePostgresql|StandaloneRedis|StandaloneSqlite $source,+ ?array $runningBefore,+ ?array $runningAfter = null,+ ): int {+ if (! $source instanceof Application) {+ return self::STOP_FAILED;+ }+ $runningAfter ??= self::runningContainersOrNull($source);+ if (blank($runningAfter)) {+ return self::STOP_FAILED;+ }++ return $runningBefore === null || array_diff($runningBefore, $runningAfter) !== []+ ? self::APPLICATION_PARTLY_STOPPED+ : self::STOP_FAILED;+ }++ /**+ * @return list<string>|null+ */+ private static function runningContainersOrNull(Application $application): ?array+ {+ try {+ return self::runningContainers($application);+ } catch (Throwable) {+ return null;+ }+ }++ /**+ * The names of the running containers of the source on its main server. A swarm stack is+ * removed asynchronously, so it is not checked.+ *+ * @return list<string>+ */+ private static function runningContainers(+ Application|Service|StandaloneClickhouse|StandaloneDragonfly|StandaloneKeydb|StandaloneMariadb|StandaloneMongodb|StandaloneMysql|StandalonePostgresql|StandaloneRedis|StandaloneSqlite $source,+ ): array {+ $server = $source->destination->server;+ if ($server->isSwarm()) {+ return [];+ }+ [$type, $filters] = match (true) {+ $source instanceof Application => ['application', ['label=coolify.pullRequestId=0']],+ $source instanceof Service => ['service', []],+ default => ['database', []],+ };+ $output = instant_remote_process(dockerPsByOwnerCommands($type, $source->uuid, $filters, '{{.Names}}', all: false), $server);++ return collect(explode("\n", (string) $output))->map(fn (string $name) => trim($name))->filter()->unique()->values()->all();+ }++ public static function startSource(+ Application|Service|StandaloneClickhouse|StandaloneDragonfly|StandaloneKeydb|StandaloneMariadb|StandaloneMongodb|StandaloneMysql|StandalonePostgresql|StandaloneRedis|StandaloneSqlite $source,+ ): void {+ match (true) {+ $source instanceof Application => queue_application_deployment(+ application: $source,+ deployment_uuid: new_public_id(),+ server: $source->destination->server,+ destination: $source->destination,+ no_questions_asked: true,+ ),+ $source instanceof Service => StartService::run($source),+ default => StartDatabase::run($source),+ };+ }++ public static function recoverSource(+ Application|Service|StandaloneClickhouse|StandaloneDragonfly|StandaloneKeydb|StandaloneMariadb|StandaloneMongodb|StandaloneMysql|StandalonePostgresql|StandaloneRedis|StandaloneSqlite $source,+ bool $wasRunning,+ Throwable $e,+ ): void {+ if (in_array($e->getCode(), [self::STOP_FAILED, self::APPLICATION_PARTLY_STOPPED], true)) {+ self::recoverFromFailedStop($source, $e);++ return;+ }++ $message = "Copying the volume data of '{$source->name}' to its clone failed. The clone may have no or partial data.";++ if ($wasRunning) {+ try {+ self::startSource($source);+ $message .= " Coolify started '{$source->name}' again.";+ } catch (Throwable $startError) {+ Log::error("Failed to start {$source->name} again after a failed volume clone: ".$startError->getMessage());+ $message .= " Coolify could not start '{$source->name}' again. Start it manually.";+ }+ }++ $source->team()?->notify(new GeneralNotification(+ $message.' Error: '.remove_iip($e->getMessage()),+ success: false,+ ));+ }++ /**+ * No data was copied. A source that the stop took down fully (or whose state is unknown) is+ * started again. When containers of a service still run, only its stopped containers are+ * started again; an application or a database that still runs is left alone.+ */+ private static function recoverFromFailedStop(+ Application|Service|StandaloneClickhouse|StandaloneDragonfly|StandaloneKeydb|StandaloneMariadb|StandaloneMongodb|StandaloneMysql|StandalonePostgresql|StandaloneRedis|StandaloneSqlite $source,+ Throwable $e,+ ): void {+ $message = "Coolify did not copy the volume data of '{$source->name}' to its clone, because stopping '{$source->name}' failed. The clone has no copied data.";++ try {+ $stillRunning = self::runningContainers($source) !== [];+ } catch (Throwable) {+ $stillRunning = false;+ }++ if ($stillRunning && $source instanceof Service) {+ try {+ self::startStoppedServiceContainers($source);+ $message .= " Some containers of '{$source->name}' were still running, so Coolify started its stopped containers again.";+ } catch (Throwable $startError) {+ Log::error("Failed to start the stopped containers of {$source->name} after a failed stop for a volume clone: ".$startError->getMessage());+ $message .= " Some containers of '{$source->name}' are still running. Coolify could not start its stopped containers again. Start '{$source->name}' manually.";+ }+ } elseif ($stillRunning && $source instanceof Application && $e->getCode() === self::APPLICATION_PARTLY_STOPPED) {+ $message .= ' '.self::redeployPartlyStoppedApplication($source);+ } elseif ($stillRunning) {+ $message .= " '{$source->name}' is still running.";+ } else {+ try {+ self::startSource($source);+ $message .= " Coolify started '{$source->name}' again.";+ } catch (Throwable $startError) {+ Log::error("Failed to start {$source->name} again after a failed stop for a volume clone: ".$startError->getMessage());+ $message .= " Coolify could not start '{$source->name}' again. Start it manually.";+ }+ }++ $source->team()?->notify(new GeneralNotification(+ $message.' Error: '.remove_iip($e->getMessage()),+ success: false,+ ));+ }++ /**+ * Queues a normal deployment (a rolling update when the application supports it) of the commit+ * of the last successful deployment, so the running containers are replaced only by the same+ * version. Without such a commit, the configured commit is deployed.+ *+ * @return string The sentence for the notification.+ */+ private static function redeployPartlyStoppedApplication(Application $application): string+ {+ $commit = $application->deployment_queue()+ ->where('status', ApplicationDeploymentStatus::FINISHED->value)+ ->where('pull_request_id', 0)+ ->whereNotNull('commit')+ ->where('commit', '!=', 'HEAD')+ ->orderByDesc('updated_at')+ ->orderByDesc('id')+ ->value('commit');++ try {+ $result = queue_application_deployment(+ application: $application,+ deployment_uuid: new_public_id(),+ commit: $commit,+ no_questions_asked: true,+ );+ if (($result['status'] ?? null) !== 'queued') {+ throw new RuntimeException($result['message'] ?? 'The deployment was not queued.');+ }+ } catch (Throwable $deployError) {+ Log::error("Failed to queue a redeploy of {$application->name} after a failed stop for a volume clone: ".$deployError->getMessage());++ return "Some containers of '{$application->name}' stopped and others are still running. Coolify could not queue a redeploy of '{$application->name}' to bring back its stopped containers. Redeploy it manually.";+ }++ $sentence = "Some containers of '{$application->name}' stopped and others were still running, so Coolify started a redeploy of '{$application->name}' to bring back its stopped containers";++ return $commit === null+ ? $sentence.'. Coolify could not find the commit that ran before, so the redeploy uses the configured commit or image tag.'+ : $sentence." with the commit that ran before ({$commit}).";+ }++ /**+ * Creates and starts the containers that the stop removed, with the compose file of the last+ * start. Running containers are not recreated, so `up --force-recreate` (StartService) is not+ * used. `docker compose start` is not enough: StopService also removes the stopped containers.+ */+ private static function startStoppedServiceContainers(Service $service): void+ {+ $server = $service->destination->server;+ $workdir = $service->workdir();+ $compose = serverDockerConfigPrefix($server)."docker compose --project-directory {$workdir} -f {$workdir}/docker-compose.yml --project-name {$service->uuid}";++ if (! data_get($service, 'connect_to_docker_network')) {+ instant_remote_process(["{$compose} up -d --no-recreate < /dev/null"], $server);++ return;+ }++ // Like StartService: connect new containers to the predefined network before they start.+ $commands = ["{$compose} up -d --no-recreate --no-start < /dev/null"];+ $composeServices = data_get(parseDockerComposeYaml(data_get($service, 'docker_compose', '')), 'services', []);+ foreach ($composeServices as $serviceName => $serviceConfig) {+ if ($command = StartService::predefinedNetworkConnectCommand($service, $serviceName, $serviceConfig)) {+ $commands[] = $command;+ }+ }+ $commands[] = "{$compose} start < /dev/null";+ instant_remote_process($commands, $server);+ }+}diff --git a/app/Helpers/SshMultiplexingHelper.php b/app/Helpers/SshMultiplexingHelper.phpindex 86034b8cbd..38e832187f 100644--- a/app/Helpers/SshMultiplexingHelper.php+++ b/app/Helpers/SshMultiplexingHelper.php@@ -253,6 +253,25 @@ class SshMultiplexingHelper.$delimiter;}+ /**+ * Build an SSH command that runs the command batch given to the process as input (see sshBatchInput()).+ * The batch is not part of any command line on the Coolify host, where `ps` would show its secrets+ * (database passwords, registry passwords, tokens), and it is not limited to 128 KiB.+ * Like the quoted heredoc of generateSshCommand(), the batch gets no local shell expansion.+ */+ public static function generateSshBatchCommand(Server $server, bool $disableMultiplexing = false, ?int $commandTimeout = null): string+ {+ return self::generateSshStdinCommand($server, self::remoteShellCommand(), $disableMultiplexing, $commandTimeout);+ }++ /**+ * The process input for generateSshBatchCommand(). A heredoc ends its content with a newline, so does this.+ */+ public static function sshBatchInput(string $command): string+ {+ return $command.PHP_EOL;+ }+/*** Build an SSH command that runs one short remote command and leaves stdin free for data.* Large content must go through stdin: Linux limits one exec argument to 128 KiB, anddiff --git a/app/Http/Controllers/Api/ApplicationsController.php b/app/Http/Controllers/Api/ApplicationsController.phpindex 608e88a4c2..50f71f46dc 100644--- a/app/Http/Controllers/Api/ApplicationsController.php+++ b/app/Http/Controllers/Api/ApplicationsController.php@@ -4070,7 +4070,9 @@ class ApplicationsController extends Controller], 404);}- $env->value = $request->value;+ if ($request->has('value')) {+ $env->value = $request->value;+ }$env->fill($request->only(['is_literal', 'is_multiline', 'is_shown_once', 'is_runtime', 'is_buildtime', 'comment']));$env->save();@@ -4226,7 +4228,9 @@ class ApplicationsController extends Controllerif ($is_preview) {$env = $application->environment_variables_preview()->where('key', $key)->first();if ($env) {- $env->value = $item->get('value');+ if ($item->has('value')) {+ $env->value = $item->get('value');+ }$env->fill($item->only(['is_literal', 'is_multiline', 'is_shown_once', 'is_runtime', 'is_buildtime', 'comment'])->all());$env->save();} else {@@ -4247,7 +4251,9 @@ class ApplicationsController extends Controller} else {$env = $application->environment_variables()->where('key', $key)->first();if ($env) {- $env->value = $item->get('value');+ if ($item->has('value')) {+ $env->value = $item->get('value');+ }$env->fill($item->only(['is_literal', 'is_multiline', 'is_shown_once', 'is_runtime', 'is_buildtime', 'comment'])->all());$env->save();} else {@@ -5177,7 +5183,7 @@ class ApplicationsController extends Controller#[OA\Patch(summary: 'Update Storage',- description: 'Update a persistent storage or file storage by application UUID.',+ description: 'Update a persistent storage or file storage by application UUID. Changed file content is saved right away and written to the server in the background, so a failed write does not change the response.',path: '/applications/{uuid}/storages',operationId: 'update-storage-by-application-uuid',security: [diff --git a/app/Http/Controllers/Api/DatabasesController.php b/app/Http/Controllers/Api/DatabasesController.phpindex 41ee9ffb7d..efa2a4a5d1 100644--- a/app/Http/Controllers/Api/DatabasesController.php+++ b/app/Http/Controllers/Api/DatabasesController.php@@ -7,6 +7,7 @@ use App\Actions\Database\StartDatabase;use App\Actions\Database\StartDatabaseProxy;use App\Actions\Database\StopDatabase;use App\Actions\Database\StopDatabaseProxy;+use App\Actions\Shared\CloneVolumeDataWithSourceStopped;use App\Enums\NewDatabaseTypes;use App\Http\Controllers\Controller;use App\Jobs\DatabaseBackupJob;@@ -36,7 +37,6 @@ use Illuminate\Http\JsonResponse;use Illuminate\Http\Request;use Illuminate\Support\Arr;use Illuminate\Support\Collection;-use Illuminate\Support\Facades\Bus;use Illuminate\Support\Facades\DB;use OpenApi\Attributes as OA;@@ -4023,7 +4023,9 @@ class DatabasesController extends Controllerreturn response()->json(['message' => 'Environment variable not found.'], 404);}- $env->value = $request->value;+ if ($request->has('value')) {+ $env->value = $request->value;+ }if ($request->has('is_literal')) {$env->is_literal = $request->is_literal;}@@ -4759,7 +4761,7 @@ class DatabasesController extends Controller#[OA\Patch(summary: 'Update Storage',- description: 'Update a persistent storage or file storage by database UUID.',+ description: 'Update a persistent storage or file storage by database UUID. Changed file content is saved right away and written to the server in the background, so a failed write does not change the response.',path: '/databases/{uuid}/storages',operationId: 'update-storage-by-database-uuid',security: [@@ -5352,33 +5354,11 @@ class DatabasesController extends Controller}}- // Stop once, clone all volumes, then start once — avoids per-volume stop/start races.- if ($pendingVolumeClones !== [] && $sourceServer && $targetServer) {- try {- $chain = [- function () use ($database) {- StopDatabase::run($database);- },- ];-- foreach ($pendingVolumeClones as $clone) {- $chain[] = new VolumeCloneJob(- $clone['source'],- $clone['target'],- $sourceServer,- $targetServer,- $clone['model'],- );- }-- $chain[] = function () use ($database) {- StartDatabase::run($database);- };-- Bus::chain($chain)->onQueue('high')->dispatch();- } catch (\Exception $e) {- \Log::error('Failed to queue database volume clone for '.$database->uuid.': '.$e->getMessage());- }+ if ($sourceServer && $targetServer) {+ CloneVolumeDataWithSourceStopped::run($database, array_map(+ fn (array $clone) => new VolumeCloneJob($clone['source'], $clone['target'], $sourceServer, $targetServer, $clone['model']),+ $pendingVolumeClones,+ ));}foreach ($database->fileStorages()->get() as $storage) {diff --git a/app/Http/Controllers/Api/ServersController.php b/app/Http/Controllers/Api/ServersController.phpindex febd83a9ac..899a1ea2f9 100644--- a/app/Http/Controllers/Api/ServersController.php+++ b/app/Http/Controllers/Api/ServersController.php@@ -776,8 +776,8 @@ class ServersController extends Controller'connection_timeout' => 'integer|min:1|max:300','is_terminal_enabled' => 'boolean|nullable','wildcard_domain' => 'nullable|url',- 'server_timezone' => ['string', function (string $attribute, mixed $value, \Closure $fail): void {- if (! validate_timezone($value)) {+ 'server_timezone' => ['bail', 'filled', 'string', function (string $attribute, mixed $value, \Closure $fail): void {+ if (! is_string($value) || ! validate_timezone($value)) {$fail('The server_timezone must be a valid timezone.');}}],diff --git a/app/Http/Controllers/Api/ServicesController.php b/app/Http/Controllers/Api/ServicesController.phpindex 1e5231400a..7eecdd30b6 100644--- a/app/Http/Controllers/Api/ServicesController.php+++ b/app/Http/Controllers/Api/ServicesController.php@@ -1573,7 +1573,9 @@ class ServicesController extends Controllerreturn response()->json(['message' => 'Environment variable not found.'], 404);}- $env->value = $request->value;+ if ($request->has('value')) {+ $env->value = $request->value;+ }if ($request->has('is_literal')) {$env->is_literal = $request->is_literal;}@@ -2760,7 +2762,7 @@ class ServicesController extends Controller#[OA\Patch(summary: 'Update Storage',- description: 'Update a persistent storage or file storage by service UUID.',+ description: 'Update a persistent storage or file storage by service UUID. Changed file content is saved right away and written to the server in the background, so a failed write does not change the response.',path: '/services/{uuid}/storages',operationId: 'update-storage-by-service-uuid',security: [diff --git a/app/Http/Middleware/RequireTeamTwoFactor.php b/app/Http/Middleware/RequireTeamTwoFactor.phpindex bb5421458e..0670d7e5c6 100644--- a/app/Http/Middleware/RequireTeamTwoFactor.php+++ b/app/Http/Middleware/RequireTeamTwoFactor.php@@ -17,7 +17,9 @@ class RequireTeamTwoFactorpublic function handle(Request $request, Closure $next): Response{$user = $request->user();- if (! $user || filled($user->two_factor_confirmed_at)) {+ // Users who must set a password first (magic link invitees) stay on the password page+ // (CheckForcePasswordReset); they need a known password to confirm the 2FA setup.+ if (! $user || filled($user->two_factor_confirmed_at) || $user->force_password_reset) {return $next($request);}diff --git a/app/Jobs/DatabaseBackupJob.php b/app/Jobs/DatabaseBackupJob.phpindex 26337ebe70..a05d713165 100644--- a/app/Jobs/DatabaseBackupJob.php+++ b/app/Jobs/DatabaseBackupJob.php@@ -3,6 +3,8 @@namespace App\Jobs;use App\Events\BackupCreated;+use App\Helpers\SshMultiplexingHelper;+use App\Helpers\SshRetryHandler;use App\Models\S3Storage;use App\Models\ScheduledDatabaseBackup;use App\Models\ScheduledDatabaseBackupExecution;@@ -921,8 +923,6 @@ class DatabaseBackupJob implements ShouldBeEncrypted, ShouldQueueprotected function upload_to_s3(S3Storage $s3): void{try {- $key = $s3->key;- $secret = $s3->secret;$bucket = $s3->bucket;$endpoint = $s3->endpoint;$s3->testConnection(shouldSave: true);@@ -940,10 +940,6 @@ class DatabaseBackupJob implements ShouldBeEncrypted, ShouldQueue$mount = escapeshellarg($this->backupMountSource().':'.$this->backup_location.':ro');$commands[] = "docker run -d --network {$safeNetwork} --name backup-of-{$this->backup_log_uuid} --rm {$s3->minioClientRegionOption()} -v {$mount} {$fullImageName}";- // Escape S3 credentials to prevent command injection- $escapedEndpoint = escapeshellarg($endpoint);- $escapedKey = escapeshellarg($key);- $escapedSecret = escapeshellarg($secret);$escapedBackupLocation = escapeshellarg($this->backup_location);$escapedS3Destination = escapeshellarg("temporary/{$bucket}{$this->backup_dir}/");$resolveOptions = collect(SafeWebhookUrl::minioClientResolveOptions($endpoint, $s3->trustedInternalHosts()))@@ -951,9 +947,12 @@ class DatabaseBackupJob implements ShouldBeEncrypted, ShouldQueue->implode(' ');$resolveOptions = $resolveOptions === '' ? '' : ' '.$resolveOptions;- $commands[] = "docker exec backup-of-{$this->backup_log_uuid} mc alias set{$resolveOptions} temporary {$escapedEndpoint} {$escapedKey} {$escapedSecret}";- $commands[] = "docker exec backup-of-{$this->backup_log_uuid} mc cp{$resolveOptions} {$escapedBackupLocation} {$escapedS3Destination}";instant_remote_process($commands, $this->server, true, false, $this->commandTimeout(), disableMultiplexing: true);+ $this->setMinioClientAlias($this->server, "backup-of-{$this->backup_log_uuid}", $s3, $resolveOptions);+ instant_remote_process(+ ["docker exec backup-of-{$this->backup_log_uuid} mc cp{$resolveOptions} {$escapedBackupLocation} {$escapedS3Destination}"],+ $this->server, true, false, $this->commandTimeout(), disableMultiplexing: true,+ );} catch (Throwable $e) {$this->add_to_error_output($e->getMessage());@@ -964,6 +963,32 @@ class DatabaseBackupJob implements ShouldBeEncrypted, ShouldQueue}}+ /**+ * Sets the `temporary` mc alias in a helper container. The S3 keys go to mc through the stdin of the+ * SSH process, so they are not in the arguments of any process on Coolify or on the server.+ */+ private function setMinioClientAlias(Server $server, string $container, S3Storage $s3, string $resolveOptions): void+ {+ $remoteCommand = ($server->isNonRoot() ? 'sudo ' : '').'docker exec -i '.escapeshellarg($container)+ .' mc alias set'.$resolveOptions.' temporary '.escapeshellarg($s3->endpoint).' >/dev/null';+ $timeout = (int) config('constants.ssh.command_timeout');++ SshRetryHandler::retry(+ function () use ($server, $remoteCommand, $s3, $timeout) {+ $sshCommand = SshMultiplexingHelper::generateSshStdinCommand($server, $remoteCommand, disableMultiplexing: true, commandTimeout: $timeout);+ $process = runSshProcess($sshCommand, $timeout, $s3->key."\n".$s3->secret."\n");++ if ($process->exitCode() !== 0) {+ excludeCertainErrors($process->errorOutput(), $process->exitCode());+ }+ },+ [+ 'server' => $server->ip,+ 'function' => 'setMinioClientAlias',+ ],+ );+ }+private function getFullImageName(): string{$helperImage = coolifyHelperImage();diff --git a/app/Jobs/DockerCleanupJob.php b/app/Jobs/DockerCleanupJob.phpindex 9daee0c82c..af1778eacf 100644--- a/app/Jobs/DockerCleanupJob.php+++ b/app/Jobs/DockerCleanupJob.php@@ -110,28 +110,19 @@ class DockerCleanupJob implements ShouldBeEncrypted, ShouldQueuereturn;}- $activeDeployment = $this->manualCleanup ? null : CleanupDocker::activeDeployment($this->server);- if ($activeDeployment) {- $this->execution_log->update([- 'status' => 'success',- 'message' => 'Docker cleanup skipped: deployment '.$activeDeployment->deployment_uuid.' is queued or running on this server. The next scheduled cleanup runs as usual.',- ]);- event(new DockerCleanupDone($this->execution_log));-- return;- }-$this->usageBefore = $this->server->getDiskUsage();if ($this->manualCleanup || $this->server->settings->force_docker_cleanup) {$cleanup_log = CleanupDocker::run(server: $this->server,deleteUnusedVolumes: $this->deleteUnusedVolumes,- deleteUnusedNetworks: $this->deleteUnusedNetworks+ deleteUnusedNetworks: $this->deleteUnusedNetworks,+ pruneBuildCacheDuringDeployment: $this->manualCleanup);$usageAfter = $this->server->getDiskUsage();$message = ($this->manualCleanup ? 'Manual' : 'Forced').' Docker cleanup job executed successfully. Disk usage before: '.$this->usageBefore.'%, Disk usage after: '.$usageAfter.'%.';+ $message = $this->withBuildCacheKeptReason($message, $cleanup_log);$this->execution_log->update(['status' => 'success','message' => $message,@@ -148,10 +139,12 @@ class DockerCleanupJob implements ShouldBeEncrypted, ShouldQueue$cleanup_log = CleanupDocker::run(server: $this->server,deleteUnusedVolumes: $this->deleteUnusedVolumes,- deleteUnusedNetworks: $this->deleteUnusedNetworks+ deleteUnusedNetworks: $this->deleteUnusedNetworks,+ pruneBuildCacheDuringDeployment: $this->manualCleanup);$message = 'Docker cleanup job executed successfully, but no disk usage could be determined.';+ $message = $this->withBuildCacheKeptReason($message, $cleanup_log);$this->execution_log->update(['status' => 'success','message' => $message,@@ -168,7 +161,8 @@ class DockerCleanupJob implements ShouldBeEncrypted, ShouldQueue$cleanup_log = CleanupDocker::run(server: $this->server,deleteUnusedVolumes: $this->deleteUnusedVolumes,- deleteUnusedNetworks: $this->deleteUnusedNetworks+ deleteUnusedNetworks: $this->deleteUnusedNetworks,+ pruneBuildCacheDuringDeployment: $this->manualCleanup);$usageAfter = $this->server->getDiskUsage();$diskSaved = $this->usageBefore - $usageAfter;@@ -179,6 +173,7 @@ class DockerCleanupJob implements ShouldBeEncrypted, ShouldQueue$message = 'Docker cleanup job executed successfully, but no disk space was saved. Disk usage before: '.$this->usageBefore.'%, Disk usage after: '.$usageAfter.'%.';}+ $message = $this->withBuildCacheKeptReason($message, $cleanup_log);$this->execution_log->update(['status' => 'success','message' => $message,@@ -227,6 +222,13 @@ class DockerCleanupJob implements ShouldBeEncrypted, ShouldQueue}}+ private function withBuildCacheKeptReason(string $message, array $cleanupLog): string+ {+ $reason = CleanupDocker::buildCacheKeptReason($cleanupLog);++ return $reason ? $message.' '.$reason : $message;+ }+/*** failed() runs on a fresh job copy built from the queue payload (for example after a worker timeout),* so the execution this run created is kept under the queue job uuid instead of on the job.diff --git a/app/Jobs/ServerConnectionCheckJob.php b/app/Jobs/ServerConnectionCheckJob.phpindex 841e915b78..a641dfdc11 100644--- a/app/Jobs/ServerConnectionCheckJob.php+++ b/app/Jobs/ServerConnectionCheckJob.php@@ -7,6 +7,7 @@ use App\Helpers\SshMultiplexingHelper;use App\Models\Server;use App\Services\ServerTransfer\ServerTransferClaimer;use Illuminate\Bus\Queueable;+use Illuminate\Contracts\Process\ProcessResult;use Illuminate\Contracts\Queue\ShouldBeEncrypted;use Illuminate\Contracts\Queue\ShouldQueue;use Illuminate\Foundation\Bus\Dispatchable;@@ -225,8 +226,7 @@ class ServerConnectionCheckJob implements ShouldBeEncrypted, ShouldQueueprivate function attemptConnection(): bool{try {- $process = Process::timeout(self::SSH_TIMEOUT_SECONDS)- ->run($this->sshCommand(self::probeCommand(), disableMultiplexing: true));+ $process = $this->runSsh(self::probeCommand(), self::SSH_TIMEOUT_SECONDS, disableMultiplexing: true);if ($process->exitCode() !== 0) {$this->connectionError = 'ssh exit '.$process->exitCode().': '.Str::limit(trim($process->errorOutput()), 300);} else {@@ -308,7 +308,7 @@ class ServerConnectionCheckJob implements ShouldBeEncrypted, ShouldQueueprivate function runDockerCommand(string $command): ?string{try {- $process = Process::timeout(self::DOCKER_TIMEOUT_SECONDS)->run($this->sshCommand($command, disableMultiplexing: $this->disableMux));+ $process = $this->runSsh($command, self::DOCKER_TIMEOUT_SECONDS, disableMultiplexing: $this->disableMux);} catch (\Throwable $e) {Log::debug('ServerConnectionCheck: Docker command failed', ['server_id' => $this->server->id,@@ -322,13 +322,15 @@ class ServerConnectionCheckJob implements ShouldBeEncrypted, ShouldQueuereturn $process->successful() ? sanitize_utf8_text(trim($process->output())) : null;}- private function sshCommand(string $command, bool $disableMultiplexing = false): string+ private function runSsh(string $command, int $timeout, bool $disableMultiplexing = false): ProcessResult{$commands = [$command];if ($this->server->isNonRoot()) {$commands = parseCommandsByLineForSudo(collect($commands), $this->server);}- return SshMultiplexingHelper::generateSshCommand($this->server, implode("\n", $commands), $disableMultiplexing);+ return Process::timeout($timeout)+ ->input(SshMultiplexingHelper::sshBatchInput(implode("\n", $commands)))+ ->run(SshMultiplexingHelper::generateSshBatchCommand($this->server, $disableMultiplexing));}}diff --git a/app/Jobs/VolumeBackupJob.php b/app/Jobs/VolumeBackupJob.phpindex d3079fe096..344cefdccc 100644--- a/app/Jobs/VolumeBackupJob.php+++ b/app/Jobs/VolumeBackupJob.php@@ -3,6 +3,8 @@namespace App\Jobs;use App\Events\BackupCreated;+use App\Helpers\SshMultiplexingHelper;+use App\Helpers\SshRetryHandler;use App\Models\LocalPersistentVolume;use App\Models\S3Storage;use App\Models\ScheduledVolumeBackup;@@ -102,7 +104,7 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue$archiveScript = self::archiveScript($compressionCpuPercentage);if ($streamToS3) {$this->execution->update(['local_storage_deleted' => true]);- $archiveCommand = $this->streamToS3Command($destinations->first(), $archiveScript, $backupLocation, $source, $containerName, $image);+ $archiveCommand = $this->streamToS3Command($destinations->first(), $archiveScript, $backupLocation, $containerName);$this->execution->update(['s3_cleanup_pending' => true]);} else {$archiveCommand = 'docker run --rm --name '.escapeshellarg($containerName)@@ -124,11 +126,20 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue}}- $archiveOutput = instant_remote_process(array_filter([- $verifySourceCommand,- $streamToS3 ? null : 'mkdir -p '.escapeshellarg($backupDirectory),- $archiveCommand,- ]), $server, timeout: $this->timeout, disableMultiplexing: true);+ try {+ if ($streamToS3) {+ $this->startStreamHelper($destinations->first(), $verifySourceCommand, $source, $containerName, $image, $server);+ }+ $archiveOutput = instant_remote_process(array_filter([+ $verifySourceCommand,+ $streamToS3 ? null : 'mkdir -p '.escapeshellarg($backupDirectory),+ $archiveCommand,+ ]), $server, timeout: $this->timeout, disableMultiplexing: true);+ } finally {+ if ($streamToS3) {+ instant_remote_process(['docker rm -f '.escapeshellarg($containerName).' >/dev/null 2>&1 || true'], $server, throwError: false, disableMultiplexing: true);+ }+ }$this->execution->update(['stop_container_ids' => null,'stop_recovery_pending' => false,@@ -429,18 +440,16 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue$s3->testConnection(shouldSave: true);$containerName = 'volume-upload-'.$this->execution->uuid;$image = coolifyHelperImage().':'.getHelperVersion();- $resolveOptions = collect(SafeWebhookUrl::minioClientResolveOptions($s3->endpoint, $s3->trustedInternalHosts()))- ->map(fn (string $option): string => '--resolve '.escapeshellarg($option))- ->implode(' ');- $resolveOptions = $resolveOptions === '' ? '' : ' '.$resolveOptions;+ $resolveOptions = $this->minioClientResolveOptions($s3);try {instant_remote_process(['docker rm -f '.escapeshellarg($containerName).' >/dev/null 2>&1 || true','docker run -d --name '.escapeshellarg($containerName).' --rm '.$s3->minioClientRegionOption().' -v '.escapeshellarg($backupLocation.':'.$backupLocation.':ro').' '.escapeshellarg($image),- 'docker exec '.escapeshellarg($containerName).' mc alias set'.$resolveOptions.' temporary '- .escapeshellarg($s3->endpoint).' '.escapeshellarg($s3->key).' '.escapeshellarg($s3->secret),+ ], $server, timeout: $this->timeout, disableMultiplexing: true);+ $this->setMinioClientAlias($server, $containerName, $s3, $resolveOptions);+ instant_remote_process(['docker exec '.escapeshellarg($containerName).' mc cp'.$resolveOptions.' '.escapeshellarg($backupLocation).' '.escapeshellarg('temporary/'.$s3->bucket.$backupDirectory.'/'),], $server, timeout: $this->timeout, disableMultiplexing: true);@@ -454,32 +463,75 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue}}- private function streamToS3Command(S3Storage $s3, string $archiveScript, string $backupLocation, string $source, string $containerName, string $image): string+ /**+ * Starts the helper container that streams the volume to S3 and sets its mc alias. The archive then+ * runs in it with `docker exec`, so the S3 keys are never part of a command line.+ */+ private function startStreamHelper(S3Storage $s3, string $verifySourceCommand, string $source, string $containerName, string $image, Server $server): void+ {+ instant_remote_process([+ $verifySourceCommand,+ 'docker rm -f '.escapeshellarg($containerName).' >/dev/null 2>&1 || true',+ 'docker run -d --rm --name '.escapeshellarg($containerName)+ .' '.$s3->minioClientRegionOption()+ .' -v '.escapeshellarg($source.':/volume:ro')+ .' '.escapeshellarg($image).' >/dev/null',+ ], $server, timeout: $this->timeout, disableMultiplexing: true);+ $this->setMinioClientAlias($server, $containerName, $s3, $this->minioClientResolveOptions($s3));+ }++ /**+ * Sets the `temporary` mc alias in a helper container. The S3 keys go to mc through the stdin of the+ * SSH process, so they are not in the arguments of any process on Coolify or on the server.+ */+ private function setMinioClientAlias(Server $server, string $container, S3Storage $s3, string $resolveOptions): void+ {+ $remoteCommand = ($server->isNonRoot() ? 'sudo ' : '').'docker exec -i '.escapeshellarg($container)+ .' mc alias set'.$resolveOptions.' temporary '.escapeshellarg($s3->endpoint).' >/dev/null';+ $timeout = (int) config('constants.ssh.command_timeout');++ SshRetryHandler::retry(+ function () use ($server, $remoteCommand, $s3, $timeout) {+ $sshCommand = SshMultiplexingHelper::generateSshStdinCommand($server, $remoteCommand, disableMultiplexing: true, commandTimeout: $timeout);+ $process = runSshProcess($sshCommand, $timeout, $s3->key."\n".$s3->secret."\n");++ if ($process->exitCode() !== 0) {+ excludeCertainErrors($process->errorOutput(), $process->exitCode());+ }+ },+ [+ 'server' => $server->ip,+ 'function' => 'setMinioClientAlias',+ ],+ );+ }++ private function minioClientResolveOptions(S3Storage $s3): string{- $s3->testConnection(shouldSave: true);$resolveOptions = collect(SafeWebhookUrl::minioClientResolveOptions($s3->endpoint, $s3->trustedInternalHosts()))->map(fn (string $option): string => '--resolve '.escapeshellarg($option))->implode(' ');- $resolveOptions = $resolveOptions === '' ? '' : ' '.$resolveOptions;++ return $resolveOptions === '' ? '' : ' '.$resolveOptions;+ }++ private function streamToS3Command(S3Storage $s3, string $archiveScript, string $backupLocation, string $containerName): string+ {+ $s3->testConnection(shouldSave: true);+ $resolveOptions = $this->minioClientResolveOptions($s3);$destination = 'temporary/'.$s3->bucket.$backupLocation;// `mc pipe --quiet` panics in recent mc releases, so keep its progress bar out of the error output instead.// Each progress frame starts with "\r", so printing the last field per line keeps only the final frame and real errors.$mcPipeLog = '/tmp/mc-pipe.log';// Object Lock buckets need Content-MD5 on each upload. `mc cp` adds it when the bucket lock is enabled, `mc pipe` only with `--checksum=MD5`.- $streamScript = 'set -o pipefail; mc alias set'.$resolveOptions.' temporary '- .escapeshellarg($s3->endpoint).' '.escapeshellarg($s3->key).' '.escapeshellarg($s3->secret)- .' >/dev/null && lock=$(mc retention info --default --json'.$resolveOptions.' '.escapeshellarg('temporary/'.$s3->bucket).' 2>/dev/null || true)'+ $streamScript = 'set -o pipefail; lock=$(mc retention info --default --json'.$resolveOptions.' '.escapeshellarg('temporary/'.$s3->bucket).' 2>/dev/null || true)'.' && case "$lock" in *\'"enabled":"Enabled"\'*) checksum=--checksum=MD5 ;; *) checksum= ;; esac'.' && ('.$archiveScript.' | mc pipe'.$resolveOptions.' $checksum '.escapeshellarg($destination).' >/dev/null 2>'.$mcPipeLog.' || { status=$?; awk -F '.escapeshellarg('\r').' '.escapeshellarg('{ print $NF }').' '.$mcPipeLog.' >&2; exit $status; })'.' && mc stat --json'.$resolveOptions.' '.escapeshellarg($destination).' | sed -n '.escapeshellarg('s/.*"size":\([0-9][0-9]*\).*/\1/p');- return 'docker run --rm --name '.escapeshellarg($containerName)- .' '.$s3->minioClientRegionOption()- .' -v '.escapeshellarg($source.':/volume:ro')- .' '.escapeshellarg($image)- .' sh -c '.escapeshellarg($streamScript);+ return 'docker exec '.escapeshellarg($containerName).' sh -c '.escapeshellarg($streamScript);}private function logCompressorInDevelopment(string $image, Server $server, int $compressionCpuPercentage): voiddiff --git a/app/Jobs/VolumeCloneJob.php b/app/Jobs/VolumeCloneJob.phpindex 20ad2ff8a8..953afc4eb1 100644--- a/app/Jobs/VolumeCloneJob.php+++ b/app/Jobs/VolumeCloneJob.php@@ -30,6 +30,22 @@ class VolumeCloneJob implements ShouldBeEncrypted, ShouldQueue$this->onQueue('high');}+ /**+ * Explicit tags, so Horizon does not compare the models of the job. In a chain, the job is restored+ * with two copies of one server for a clone on the same server, and comparing them fails on PHP 8.4+ * ("Nesting level too deep"), which stops the chain before the copy.+ *+ * @return array<int, string>+ */+ public function tags(): array+ {+ return array_values(array_unique([+ Server::class.':'.$this->sourceServer->id,+ ...($this->targetServer ? [Server::class.':'.$this->targetServer->id] : []),+ LocalPersistentVolume::class.':'.$this->persistentVolume->id,+ ]));+ }+public function handle(): void{try {diff --git a/app/Livewire/Fleet/Metrics.php b/app/Livewire/Fleet/Metrics.phpindex a8e11ee0fe..0b71d44cfa 100644--- a/app/Livewire/Fleet/Metrics.php+++ b/app/Livewire/Fleet/Metrics.php@@ -12,6 +12,7 @@ use Illuminate\Foundation\Auth\Access\AuthorizesRequests;use Illuminate\Support\Collection;use Illuminate\Support\Facades\Concurrency;use Illuminate\Support\Facades\Log;+use Illuminate\Support\Facades\RateLimiter;use Livewire\Attributes\Lazy;use Livewire\Attributes\Url;use Livewire\Component;@@ -194,14 +195,43 @@ class Metrics extends Componentpublic function setRange(string $range): void{$this->range = in_array($range, ['24h', '7d', '30d'], true) ? $range : '24h';+ // Every refresh fetches the full range from each server over SSH, so a new range starts paused.+ $this->live = false;$this->loadData();}public function updatedLive(): void{if ($this->live) {- $this->loadData();+ $this->pollData();+ }+ }++ /**+ * The poll target. The server decides the refresh rate, so a client cannot fetch faster+ * than once a minute or while live refresh is off.+ */+ public function pollData(): void+ {+ $seconds = $this->pollSeconds();+ if ($seconds === null) {+ return;+ }++ // One second of slack, so a poll that fires slightly early is not dropped.+ if (RateLimiter::attempt('metrics-poll:'.$this->getId(), 1, fn () => true, $seconds - 1) === false) {+ return;}++ $this->loadData();+ }++ /**+ * Seconds between live refreshes, or null when live refresh is off.+ */+ public function pollSeconds(): ?int+ {+ return $this->live ? 60 : null;}/**diff --git a/app/Livewire/Project/CloneMe.php b/app/Livewire/Project/CloneMe.phpindex 99cacd1a93..fce8bffd0b 100644--- a/app/Livewire/Project/CloneMe.php+++ b/app/Livewire/Project/CloneMe.php@@ -2,8 +2,7 @@namespace App\Livewire\Project;-use App\Actions\Database\StartDatabase;-use App\Actions\Database\StopDatabase;+use App\Actions\Shared\CloneVolumeDataWithSourceStopped;use App\Jobs\VolumeCloneJob;use App\Models\Environment;use App\Models\Project;@@ -103,6 +102,11 @@ class CloneMe extends Componentif (! $this->environment || (int) $this->environment->project_id !== (int) $this->project->id) {throw new \Exception('Environment not found.');}+ if ($this->cloneVolumeData) {+ foreach ([...$this->environment->applications, ...$this->environment->databases(), ...$this->environment->services] as $source) {+ $this->authorize(CloneVolumeDataWithSourceStopped::sourceAbility($source), $source);+ }+ }$selectedDestination = $this->findProjectTeamDestination($this->selectedDestination);if (! $selectedDestination) {throw new \Exception('Destination not found.');@@ -192,6 +196,7 @@ class CloneMe extends Component}$newDatabase->persistentStorages()->delete();+ $pendingVolumeClones = [];$persistentVolumes = $database->persistentStorages()->get();foreach ($persistentVolumes as $volume) {$originalName = $volume->name;@@ -235,22 +240,18 @@ class CloneMe extends Component$newPersistentVolume->save();if ($this->cloneVolumeData) {- try {- StopDatabase::dispatch($database);- $sourceVolume = $volume->name;- $targetVolume = $newPersistentVolume->name;- $sourceServer = $database->destination->server;- $targetServer = $newDatabase->destination->server;-- VolumeCloneJob::dispatch($sourceVolume, $targetVolume, $sourceServer, $targetServer, $newPersistentVolume);-- StartDatabase::dispatch($database);- } catch (\Exception $e) {- \Log::error('Failed to copy volume data for '.$volume->name.': '.$e->getMessage());- }+ $pendingVolumeClones[] = new VolumeCloneJob(+ $volume->name,+ $newPersistentVolume->name,+ $database->destination->server,+ $newDatabase->destination->server,+ $newPersistentVolume,+ );}}+ CloneVolumeDataWithSourceStopped::run($database, $pendingVolumeClones);+$fileStorages = $database->fileStorages()->get();foreach ($fileStorages as $storage) {$newStorage = $storage->replicate([diff --git a/app/Livewire/Project/Shared/GetLogs.php b/app/Livewire/Project/Shared/GetLogs.phpindex cabf0bc469..57bfbd16f3 100644--- a/app/Livewire/Project/Shared/GetLogs.php+++ b/app/Livewire/Project/Shared/GetLogs.php@@ -496,10 +496,11 @@ class GetLogs extends Component*/private function executeLogCommand(Server $server, string $command, int $maxBytes): array{- $sshCommand = SshMultiplexingHelper::generateSshCommand($server, $this->boundedLogCommand($command, $maxBytes));-// boundedLogCommand() caps the output at $maxBytes + 1, so one extra byte means truncation.- $output = Process::timeout(config('constants.ssh.command_timeout'))->run($sshCommand)->output();+ $output = Process::timeout(config('constants.ssh.command_timeout'))+ ->input(SshMultiplexingHelper::sshBatchInput($this->boundedLogCommand($command, $maxBytes)))+ ->run(SshMultiplexingHelper::generateSshBatchCommand($server))+ ->output();$truncated = strlen($output) > $maxBytes;$logs = rtrim(removeAnsiColors($truncated ? substr($output, 0, $maxBytes) : $output), "\n");diff --git a/app/Livewire/Project/Shared/Metrics.php b/app/Livewire/Project/Shared/Metrics.phpindex 8b027d7b49..a5217a7be6 100644--- a/app/Livewire/Project/Shared/Metrics.php+++ b/app/Livewire/Project/Shared/Metrics.php@@ -2,6 +2,7 @@namespace App\Livewire\Project\Shared;+use Illuminate\Support\Facades\RateLimiter;use Livewire\Component;class Metrics extends Component@@ -40,18 +41,51 @@ class Metrics extends Component$this->loadData();}+ /**+ * Ranges up to one hour refresh live by default. Longer ranges load once, because each+ * refresh fetches the full range from Sentinel over SSH; the user can still turn live on.+ */+ public function updatedInterval(): void+ {+ $this->live = $this->interval <= 60;+ }+public function updatedLive(): void{if ($this->live) {- $this->loadData();+ $this->pollData();}}/**- * Short ranges refresh every 5 seconds; longer ranges change slowly, so they refresh every minute.+ * The poll target. The server decides the refresh rate, so a client cannot fetch faster+ * than the range allows or while live refresh is off.*/- public function pollSeconds(): int+ public function pollData(): void{+ $seconds = $this->pollSeconds();+ if ($seconds === null) {+ return;+ }++ // One second of slack, so a poll that fires slightly early is not dropped.+ if (RateLimiter::attempt('metrics-poll:'.$this->getId(), 1, fn () => true, $seconds - 1) === false) {+ return;+ }++ $this->loadData();+ }++ /**+ * Seconds between live refreshes, or null when live refresh is off. Short ranges refresh+ * every 5 seconds; longer ranges change slowly, so they refresh every minute.+ */+ public function pollSeconds(): ?int+ {+ if (! $this->live) {+ return null;+ }+return $this->interval <= 60 ? 5 : 60;}diff --git a/app/Livewire/Project/Shared/ResourceOperations.php b/app/Livewire/Project/Shared/ResourceOperations.phpindex 414af27128..198972406e 100644--- a/app/Livewire/Project/Shared/ResourceOperations.php+++ b/app/Livewire/Project/Shared/ResourceOperations.php@@ -2,8 +2,7 @@namespace App\Livewire\Project\Shared;-use App\Actions\Database\StartDatabase;-use App\Actions\Database\StopDatabase;+use App\Actions\Shared\CloneVolumeDataWithSourceStopped;use App\Actions\Shared\MigrateResourceToDestination;use App\Jobs\VolumeCloneJob;use App\Models\Application;@@ -72,6 +71,9 @@ class ResourceOperations extends Component{try {$this->authorize('update', $this->resource);+ if ($this->cloneVolumeData) {+ $this->authorize(CloneVolumeDataWithSourceStopped::sourceAbility($this->resource), $this->resource);+ }$new_environment = $this->resource->environment;if ($environment_id !== null) {@@ -145,6 +147,7 @@ class ResourceOperations extends Component}$new_resource->persistentStorages()->delete();+ $pendingVolumeClones = [];$persistentVolumes = $this->resource->persistentStorages()->get();foreach ($persistentVolumes as $volume) {$originalName = $volume->name;@@ -188,22 +191,18 @@ class ResourceOperations extends Component$newPersistentVolume->save();if ($this->cloneVolumeData) {- try {- StopDatabase::dispatch($this->resource);- $sourceVolume = $volume->name;- $targetVolume = $newPersistentVolume->name;- $sourceServer = $this->resource->destination->server;- $targetServer = $new_resource->destination->server;-- VolumeCloneJob::dispatch($sourceVolume, $targetVolume, $sourceServer, $targetServer, $newPersistentVolume);-- StartDatabase::dispatch($this->resource);- } catch (\Exception $e) {- \Log::error('Failed to copy volume data for '.$volume->name.': '.$e->getMessage());- }+ $pendingVolumeClones[] = new VolumeCloneJob(+ $volume->name,+ $newPersistentVolume->name,+ $this->resource->destination->server,+ $new_resource->destination->server,+ $newPersistentVolume,+ );}}+ CloneVolumeDataWithSourceStopped::run($this->resource, $pendingVolumeClones);+$fileStorages = $this->resource->fileStorages()->get();foreach ($fileStorages as $storage) {$newStorage = $storage->replicate([diff --git a/app/Livewire/Server/Charts.php b/app/Livewire/Server/Charts.phpindex 65ddde8a7d..77a56a0073 100644--- a/app/Livewire/Server/Charts.php+++ b/app/Livewire/Server/Charts.php@@ -5,6 +5,7 @@ namespace App\Livewire\Server;use App\Actions\Server\StartSentinel;use App\Models\Server;use Illuminate\Foundation\Auth\Access\AuthorizesRequests;+use Illuminate\Support\Facades\RateLimiter;use Livewire\Attributes\Validate;use Livewire\Component;@@ -150,18 +151,51 @@ class Charts extends Component$this->loadData();}+ /**+ * Ranges up to one hour refresh live by default. Longer ranges load once, because each+ * refresh fetches the full range from Sentinel over SSH; the user can still turn live on.+ */+ public function updatedInterval(): void+ {+ $this->live = $this->interval <= 60;+ }+public function updatedLive(): void{if ($this->live) {- $this->loadData();+ $this->pollData();+ }+ }++ /**+ * The poll target. The server decides the refresh rate, so a client cannot fetch faster+ * than the range allows or while live refresh is off.+ */+ public function pollData(): void+ {+ $seconds = $this->pollSeconds();+ if ($seconds === null) {+ return;+ }++ // One second of slack, so a poll that fires slightly early is not dropped.+ if (RateLimiter::attempt('metrics-poll:'.$this->getId(), 1, fn () => true, $seconds - 1) === false) {+ return;}++ $this->loadData();}/**- * Short ranges refresh every 5 seconds; longer ranges change slowly, so they refresh every minute.+ * Seconds between live refreshes, or null when live refresh is off. Short ranges refresh+ * every 5 seconds; longer ranges change slowly, so they refresh every minute.*/- public function pollSeconds(): int+ public function pollSeconds(): ?int{+ if (! $this->live) {+ return null;+ }+return $this->interval <= 60 ? 5 : 60;}}diff --git a/app/Livewire/TwoFactorRequired.php b/app/Livewire/TwoFactorRequired.phpindex 72d4091b10..7b4a06ea2a 100644--- a/app/Livewire/TwoFactorRequired.php+++ b/app/Livewire/TwoFactorRequired.php@@ -17,6 +17,7 @@ class TwoFactorRequired extends Component{return view('livewire.two-factor-required', ['teamName' => data_get(session('currentTeam'), 'name'),+ 'teamsWithoutTwoFactor' => auth()->user()->teams()->where('is_two_factor_required', false)->get(),])->layout('layouts.simple');}}diff --git a/app/Mcp/Tools/ManageApplication.php b/app/Mcp/Tools/ManageApplication.phpindex bf857b4fda..71758d80e0 100644--- a/app/Mcp/Tools/ManageApplication.php+++ b/app/Mcp/Tools/ManageApplication.php@@ -16,7 +16,7 @@ class ManageApplication extends RestActionToolprotected function actions(): array{return [- 'update' => ['method' => 'PATCH', 'path' => 'applications/{uuid}', 'ability' => 'write', 'description' => 'update application settings (data: same fields as the REST update, e.g. name, domains, git_branch, ports_exposes, build_command; instant_deploy and manual_webhook_secret_* need deploy ability)'],+ 'update' => ['method' => 'PATCH', 'path' => 'applications/{uuid}', 'ability' => 'write', 'description' => 'update application settings (data: same fields as the REST update, e.g. name, domains, git_branch, ports_exposes, build_command, is_container_label_readonly_enabled, include_coolify_default_variables; instant_deploy and manual_webhook_secret_* need deploy ability)'],'delete' => ['method' => 'DELETE', 'path' => 'applications/{uuid}', 'ability' => 'deploy', 'confirm' => true, 'description' => 'delete the application and its containers (data: delete_volumes, delete_connected_networks, delete_configurations, docker_cleanup)'],'clone' => ['method' => 'POST', 'path' => 'applications/{uuid}/clone', 'ability' => 'write', 'description' => 'clone the application to a destination of the team (data: destination_uuid, name, clone_volumes; clone_volumes needs deploy ability)'],'move' => ['method' => 'POST', 'path' => 'applications/{uuid}/move', 'ability' => 'write', 'confirm' => true, 'description' => 'move the application to another environment of the team (data: environment_uuid)'],diff --git a/app/Mcp/Tools/ManageDatabase.php b/app/Mcp/Tools/ManageDatabase.phpindex 5e4c11308d..e879c7f091 100644--- a/app/Mcp/Tools/ManageDatabase.php+++ b/app/Mcp/Tools/ManageDatabase.php@@ -16,7 +16,7 @@ class ManageDatabase extends RestActionToolprotected function actions(): array{return [- 'update' => ['method' => 'PATCH', 'path' => 'databases/{uuid}', 'ability' => 'write', 'description' => 'update database settings (data: name, description, image, is_public, public_port, public_port_timeout, limits_*, type credentials and *_conf fields, redact_all_env_values_in_logs, instant_deploy; instant_deploy and changing is_public need deploy ability)'],+ 'update' => ['method' => 'PATCH', 'path' => 'databases/{uuid}', 'ability' => 'write', 'description' => 'update database settings (data: name, description, image, is_public, public_port, public_port_timeout, limits_*, type credentials and *_conf fields, redact_all_env_values_in_logs, is_log_drain_enabled, include_coolify_default_variables, instant_deploy; instant_deploy and changing is_public need deploy ability)'],'delete' => ['method' => 'DELETE', 'path' => 'databases/{uuid}', 'ability' => 'deploy', 'confirm' => true, 'description' => 'delete the database (data: delete_volumes, delete_connected_networks, delete_configurations, docker_cleanup; all default true)'],'clone' => ['method' => 'POST', 'path' => 'databases/{uuid}/clone', 'ability' => 'write', 'confirm' => true, 'description' => 'clone the database to a destination of the team (data: destination_uuid, name, clone_volumes; clone_volumes stops the source database while volumes are copied and needs deploy ability)'],'move' => ['method' => 'POST', 'path' => 'databases/{uuid}/move', 'ability' => 'write', 'confirm' => true, 'description' => 'move the database to another environment of the team (data: environment_uuid)'],diff --git a/app/Mcp/Tools/ManageServer.php b/app/Mcp/Tools/ManageServer.phpindex 7680d1c41b..31f8acd5d1 100644--- a/app/Mcp/Tools/ManageServer.php+++ b/app/Mcp/Tools/ManageServer.php@@ -17,7 +17,7 @@ class ManageServer extends RestActionTool{return ['create' => ['method' => 'POST', 'path' => 'servers', 'ability' => 'write', 'description' => 'add a server (data: ip, private_key_uuid, name, description, port, user, server_role, proxy_type, instant_validate)'],- 'update' => ['method' => 'PATCH', 'path' => 'servers/{uuid}', 'ability' => 'write', 'description' => 'update a server (data: name, description, ip, port, user, private_key_uuid, server_role, proxy_type, concurrent_builds, dynamic_timeout, deployment_queue_limit, connection_timeout, is_terminal_enabled, server_disk_usage_*; changing proxy_type needs deploy ability)'],+ 'update' => ['method' => 'PATCH', 'path' => 'servers/{uuid}', 'ability' => 'write', 'description' => 'update a server (data: name, description, ip, port, user, private_key_uuid, server_role, proxy_type, concurrent_builds, dynamic_timeout, deployment_queue_limit, connection_timeout, is_terminal_enabled, server_disk_usage_*, wildcard_domain, server_timezone, backup_compression_cpu_percentage; changing proxy_type needs deploy ability)'],'delete' => ['method' => 'DELETE', 'path' => 'servers/{uuid}', 'ability' => 'write', 'confirm' => true, 'description' => 'delete a server without resources (the Coolify host cannot be deleted)'],'validate' => ['method' => 'POST', 'path' => 'servers/{uuid}/validate', 'ability' => 'write', 'description' => 'start a server validation (data: install=true also installs Docker and needs deploy ability)'],];diff --git a/app/Mcp/Tools/ManageService.php b/app/Mcp/Tools/ManageService.phpindex cfd5b38c75..f7868dea4f 100644--- a/app/Mcp/Tools/ManageService.php+++ b/app/Mcp/Tools/ManageService.php@@ -16,11 +16,11 @@ class ManageService extends RestActionToolprotected function actions(): array{return [- 'update' => ['method' => 'PATCH', 'path' => 'services/{uuid}', 'ability' => 'write', 'description' => 'update the service (data: name, description, docker_compose_raw base64, connect_to_docker_network, urls, force_domain_override, is_container_label_escape_enabled, redact_all_env_values_in_logs, instant_deploy; instant_deploy needs deploy ability)'],+ 'update' => ['method' => 'PATCH', 'path' => 'services/{uuid}', 'ability' => 'write', 'description' => 'update the service (data: name, description, docker_compose_raw base64, connect_to_docker_network, urls, force_domain_override, is_container_label_escape_enabled, include_coolify_default_variables, redact_all_env_values_in_logs, instant_deploy; instant_deploy needs deploy ability)'],'delete' => ['method' => 'DELETE', 'path' => 'services/{uuid}', 'ability' => 'deploy', 'confirm' => true, 'description' => 'delete the service (data: delete_volumes, delete_connected_networks, delete_configurations, docker_cleanup)'],'clone' => ['method' => 'POST', 'path' => 'services/{uuid}/clone', 'ability' => 'write', 'confirm' => true, 'description' => 'clone the service to a destination of the team (data: destination_uuid, name, clone_volumes; clone_volumes stops the source service while volumes are copied and needs deploy ability)'],'move' => ['method' => 'POST', 'path' => 'services/{uuid}/move', 'ability' => 'write', 'confirm' => true, 'description' => 'move the service to another environment of the team (data: environment_uuid)'],- 'update_application' => ['method' => 'PATCH', 'path' => 'services/{uuid}/applications/{app_uuid}', 'ability' => 'write', 'description' => 'update a service application (data: url, noindex_domains, human_name, description, image, exclude_from_status, is_log_drain_enabled, is_gzip_enabled, traefik_certificate_resolver, is_stripprefix_enabled, is_force_https_enabled, max_restart_count)'],+ 'update_application' => ['method' => 'PATCH', 'path' => 'services/{uuid}/applications/{app_uuid}', 'ability' => 'write', 'description' => 'update a service application (data: url, redirect, noindex_domains, human_name, description, image, exclude_from_status, is_log_drain_enabled, is_gzip_enabled, traefik_certificate_resolver, is_stripprefix_enabled, is_force_https_enabled, max_restart_count)'],'update_database' => ['method' => 'PATCH', 'path' => 'services/{uuid}/databases/{database_uuid}', 'ability' => 'write', 'description' => 'update a service database (data: human_name, description, image, exclude_from_status, is_log_drain_enabled, is_public, public_port, public_port_timeout; changing is_public needs deploy ability)'],'import_database' => ['method' => 'POST', 'path' => 'services/{uuid}/databases/{database_uuid}/imports', 'ability' => 'deploy', 'confirm' => true, 'description' => 'import a backup into a service database, can overwrite data (data: source s3 | server, s3_storage_uuid, path, dump_all, replace_existing, keep_owners, restore_mysql_users); check progress with get_database_import'],];diff --git a/app/Mcp/Tools/ManageStorage.php b/app/Mcp/Tools/ManageStorage.phpindex 5fdf754964..1211833052 100644--- a/app/Mcp/Tools/ManageStorage.php+++ b/app/Mcp/Tools/ManageStorage.php@@ -17,7 +17,7 @@ class ManageStorage extends RestActionTool{return ['create' => ['method' => 'POST', 'paths' => $this->paths('storages'), 'ability' => 'write', 'description' => 'create a storage (data: type persistent|file, name, mount_path, content, is_directory, is_host_file, fs_path; services also resource_uuid; file content and an fs_path outside the resource directory need deploy ability)'],- 'update' => ['method' => 'PATCH', 'paths' => $this->paths('storages'), 'ability' => 'write', 'description' => 'update a storage (data: uuid of the storage, type, name, mount_path, content, is_preview_suffix_enabled; content of a file outside the resource directory needs deploy ability)'],+ 'update' => ['method' => 'PATCH', 'paths' => $this->paths('storages'), 'ability' => 'write', 'description' => 'update a storage (data: uuid of the storage, type, name, mount_path, content, is_preview_suffix_enabled; a content change needs deploy ability; the content is saved right away and written to the server in the background, so a failed write does not change the response)'],'delete' => ['method' => 'DELETE', 'paths' => $this->paths('storages/{storage_uuid}'), 'ability' => 'write', 'confirm' => true, 'description' => 'delete a storage'],'set_backup' => ['method' => 'PUT', 'paths' => $this->paths('storages/{storage_uuid}/backups'), 'ability' => 'write', 'description' => 'create or replace the volume backup schedule (data: frequency, enabled, save_s3, s3_storage_uuid(s), stop_during_backup, retention fields ...; stop_during_backup needs deploy ability)'],'remove_backup' => ['method' => 'DELETE', 'paths' => $this->paths('storages/{storage_uuid}/backups'), 'ability' => 'write', 'confirm' => true, 'description' => 'delete the volume backup schedule and its local and S3 archives'],diff --git a/app/Models/Application.php b/app/Models/Application.phpindex f743be45f1..1f474d4aa3 100644--- a/app/Models/Application.php+++ b/app/Models/Application.php@@ -1885,9 +1885,7 @@ class Application extends BaseModel$fullRepoUrl = "{$this->source->html_url}/{$customRepository}";$escapedRepoUrl = escapeshellarg("{$this->source->html_url}/{$customRepository}");$git_clone_command = "{$git_clone_command} {$escapedRepoUrl} {$escapedBaseDir}";- if (! $only_checkout) {- $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command, public: true, commit: $commit, baseDir: $baseDir, onlyCheckout: $only_checkout);- }+ $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command, public: true, commit: $commit, baseDir: $baseDir, onlyCheckout: $only_checkout);if ($exec_in_docker) {$commands->push($this->gitCommand(executeInDocker($deployment_uuid, $git_clone_command)));} else {@@ -1912,9 +1910,7 @@ class Application extends BaseModel$git_clone_command = "{$git_clone_command} {$escapedRepoUrl} {$escapedBaseDir}";$fullRepoUrl = $repoUrl;}- if (! $only_checkout) {- $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command, public: false, commit: $commit, gitConfigOptions: $gitConfigOption, baseDir: $baseDir, onlyCheckout: $only_checkout);- }+ $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command, public: false, commit: $commit, gitConfigOptions: $gitConfigOption, baseDir: $baseDir, onlyCheckout: $only_checkout);if ($exec_in_docker) {$commands->push($this->gitCommand(executeInDocker($deployment_uuid, $git_clone_command)));} else {@@ -1960,11 +1956,7 @@ class Application extends BaseModel$escapedRepoUrl = escapeshellarg($repoUrl);$fullRepoUrl = $repoUrl;$git_clone_command_base = "{$git_clone_command} {$escapedRepoUrl} {$escapedBaseDir}";- if ($only_checkout) {- $git_clone_command = $git_clone_command_base;- } else {- $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command_base, commit: $commit, gitConfigOptions: $gitConfigOption, baseDir: $baseDir, onlyCheckout: $only_checkout);- }+ $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command_base, commit: $commit, gitConfigOptions: $gitConfigOption, baseDir: $baseDir, onlyCheckout: $only_checkout);if ($pull_request_id !== 0) {$branch = "{$pullRequestRef}:{$pr_branch_name}";@@ -2002,11 +1994,7 @@ class Application extends BaseModel$gitlabSshCommand = gitSshCommand($gitlabPort, $customSshKeyLocation);$gitlabGitSshCommand = "GIT_SSH_COMMAND=\"{$gitlabSshCommand}\"";$git_clone_command_base = "{$gitlabGitSshCommand} {$git_clone_command} {$escapedCustomRepository} {$escapedBaseDir}";- if ($only_checkout) {- $git_clone_command = $git_clone_command_base;- } else {- $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command_base, commit: $commit, gitSshCommand: $gitlabSshCommand, baseDir: $baseDir, onlyCheckout: $only_checkout);- }+ $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command_base, commit: $commit, gitSshCommand: $gitlabSshCommand, baseDir: $baseDir, onlyCheckout: $only_checkout);$commands = $this->gitSshKeySetupCommands($deployment_uuid, $private_key, $exec_in_docker);if ($pull_request_id !== 0) {@@ -2063,11 +2051,7 @@ class Application extends BaseModel$deployKeySshCommand = gitSshCommand($customPort, $customSshKeyLocation);$deployKeyGitSshCommand = "GIT_SSH_COMMAND=\"{$deployKeySshCommand}\"";$git_clone_command_base = "{$deployKeyGitSshCommand} {$git_clone_command} {$escapedCustomRepository} {$escapedBaseDir}";- if ($only_checkout) {- $git_clone_command = $git_clone_command_base;- } else {- $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command_base, commit: $commit, gitSshCommand: $deployKeySshCommand, baseDir: $baseDir, onlyCheckout: $only_checkout);- }+ $git_clone_command = $this->setGitImportSettings($deployment_uuid, $git_clone_command_base, commit: $commit, gitSshCommand: $deployKeySshCommand, baseDir: $baseDir, onlyCheckout: $only_checkout);$commands = $this->gitSshKeySetupCommands($deployment_uuid, $private_key, $exec_in_docker);if ($pull_request_id !== 0) {if ($git_type === 'gitlab') {@@ -2270,8 +2254,9 @@ class Application extends BaseModel{['commands' => $cloneCommand, 'git' => $git] = $this->generateGitImportCommands(deployment_uuid: $uuid, only_checkout: true, exec_in_docker: false, custom_base_dir: $checkoutDir);+ // Callers read data from stdout after the checkout, so progress lines of the checkout go to stderr.return [- 'command' => str_replace(' clone ', ' clone --quiet ', $this->gitCommandsAsShellCommand($cloneCommand)),+ 'command' => 'exec >&2 && '.str_replace(' clone ', ' clone --quiet ', $this->gitCommandsAsShellCommand($cloneCommand)),'git' => $git,];}@@ -2290,9 +2275,9 @@ class Application extends BaseModel$checkoutDir = "/tmp/{$uuid}/checkout";['command' => $cloneCommand, 'git' => $git] = $this->serverCheckout($uuid, $checkoutDir);$composeFile = ltrim(rtrim($this->base_directory, '/').$this->docker_compose_location, '/');- $submoduleCommands = $this->composeFileSubmoduleCommands($git, $composeFile);+ $submoduleCommand = $this->composeFileSubmoduleCommand($git, $composeFile, $checkoutDir);// Go into the submodule that contains the Compose file, and read the file from the objects of that submodule.- $enterSubmodule = $submoduleCommands === [] ? '' : implode(' ', [+ $enterSubmodule = $submoduleCommand === null ? '' : implode(' ', ['while sub=$(git ls-tree -r HEAD | while read -r mode type object path; do','if [ "$mode" = 160000 ]; then case "$file" in "$path"/*) echo "$path"; break ;; esac; fi; done) && [ -n "$sub" ];','do cd "$sub" || exit 1; file="${file#"$sub"/}"; done;',@@ -2317,9 +2302,9 @@ class Application extends BaseModel"mkdir -p /tmp/{$uuid}","cd /tmp/{$uuid}",// One sh -c line: the non-root sudo parser would turn `&& if ...; then` into invalid `&& sudo if`.- 'sh -c '.escapeshellarg($cloneCommand),+ // The submodule step runs in the same script, because a deploy key is removed when the script exits.+ 'sh -c '.escapeshellarg($submoduleCommand === null ? $cloneCommand : "{$cloneCommand} && {$submoduleCommand}"),"cd {$checkoutDir}",- ...$submoduleCommands,'sh -c '.escapeshellarg($readComposeFile),]);}@@ -2328,23 +2313,21 @@ class Application extends BaseModel* The clone has no checkout and no submodules. If the Compose file is inside a submodule, this fills the* index (without a working tree) and clones only that submodule, with the credentials of the clone.*- * @return list<string>+ * The command is appended to the clone script, so it must not be a separate sh -c line.*/- private function composeFileSubmoduleCommands(string $git, string $composeFile): array+ private function composeFileSubmoduleCommand(string $git, string $composeFile, string $checkoutDir): ?string{if (! $this->settings?->is_git_submodules_enabled) {- return [];+ return null;}$depthFlag = $this->settings->is_git_shallow_clone_enabled ? ' --depth=1' : '';$escapedComposeFile = escapeshellarg($composeFile);- // One sh -c line: the non-root sudo parser cannot change commands inside a loop.- return ['git read-tree HEAD', 'sh -c '.escapeshellarg(- 'git ls-files -s | while read -r mode object stage path; do '+ return 'cd '.escapeshellarg($checkoutDir).' && git read-tree HEAD && '+ .'git ls-files -s | while read -r mode object stage path; do '."if [ \"\$mode\" = 160000 ]; then case {$escapedComposeFile} in \"\$path\"/*) "- ."{$git} submodule update --init --recursive{$depthFlag} -- \"\$path\" >/dev/null || exit 1 ;; esac; fi; done"- )];+ ."{$git} submodule update --init --recursive{$depthFlag} -- \"\$path\" >/dev/null || exit 1 ;; esac; fi; done";}/**diff --git a/app/Models/EnvironmentVariable.php b/app/Models/EnvironmentVariable.phpindex c36934e539..4ef77a4f16 100644--- a/app/Models/EnvironmentVariable.php+++ b/app/Models/EnvironmentVariable.php@@ -55,6 +55,17 @@ class EnvironmentVariable extends BaseModel'RAILPACK_SKIP_MIGRATIONS',];+ /**+ * Magic variable commands (SERVICE_<COMMAND>_<NAME>) whose value generateEnvValue() generates.+ */+ public const GENERATED_MAGIC_VARIABLE_COMMANDS = [+ 'PASSWORD', 'PASSWORD_64', 'PASSWORDWITHSYMBOLS', 'PASSWORDWITHSYMBOLS_64',+ 'BASE64', 'BASE64_32', 'BASE64_64', 'BASE64_128',+ 'REALBASE64', 'REALBASE64_32', 'REALBASE64_64', 'REALBASE64_128',+ 'HEX_32', 'HEX_64', 'HEX_128',+ 'USER', 'LOWERCASEUSER', 'SUPABASEANON', 'SUPABASESERVICE',+ ];+protected $attributes = ['is_runtime' => true,'is_buildtime' => true,@@ -125,7 +136,8 @@ class EnvironmentVariable extends BaseModelif (! $found) {$application = Application::find($environment_variable->resourceable_id);- if ($application) {+ // A compose preview generates its own magic secrets on the first pull request parse.+ if ($application && ! ($application->build_pack === 'dockercompose' && $environment_variable->isGeneratedMagicVariable())) {ModelsEnvironmentVariable::create(['key' => $environment_variable->key,'value' => $environment_variable->value,@@ -156,6 +168,12 @@ class EnvironmentVariable extends BaseModel});}+ public function isGeneratedMagicVariable(): bool+ {+ return str_starts_with($this->key, 'SERVICE_')+ && in_array(parseCommandFromMagicEnvVariable($this->key)->value(), self::GENERATED_MAGIC_VARIABLE_COMMANDS, true);+ }+public function preservesLegacyComposeNull(): bool{return $this->preserve_compose_null && $this->value === null;diff --git a/app/Services/ContainerFilesystemService.php b/app/Services/ContainerFilesystemService.phpindex 5392f7d483..7fe8221f03 100644--- a/app/Services/ContainerFilesystemService.php+++ b/app/Services/ContainerFilesystemService.php@@ -402,12 +402,15 @@ class ContainerFilesystemService$command = parseCommandsByLineForSudo(collect([$command]), $this->server)[0];}- $process = proc_open($this->sshCommand($command), [1 => ['pipe', 'w'], 2 => ['file', '/dev/null', 'w']], $pipes);+ $process = proc_open($this->sshCommand(), [0 => ['pipe', 'r'], 1 => ['pipe', 'w'], 2 => ['file', '/dev/null', 'w']], $pipes);if (! is_resource($process)) {throw new \RuntimeException('Could not start the download.');}try {+ // The command goes to ssh on stdin, so it is not in a command line on the Coolify host.+ fwrite($pipes[0], SshMultiplexingHelper::sshBatchInput($command));+ fclose($pipes[0]);while (! feof($pipes[1])) {$chunk = fread($pipes[1], 64 * 1024);if ($chunk === false || connection_aborted()) {@@ -450,9 +453,9 @@ class ContainerFilesystemServiceinstant_remote_process([$this->buildS3UploadCommand($path, $isDir, $url, $headers)], $this->server, timeout: 3600);}- protected function sshCommand(string $command): string+ protected function sshCommand(): string{- return SshMultiplexingHelper::generateSshCommand($this->server, $command);+ return SshMultiplexingHelper::generateSshBatchCommand($this->server);}protected function escapePath(string $path, string $label): stringdiff --git a/app/Services/DatabaseStartCommandExecutor.php b/app/Services/DatabaseStartCommandExecutor.phpindex 4fc38a9539..7f8090fc61 100644--- a/app/Services/DatabaseStartCommandExecutor.php+++ b/app/Services/DatabaseStartCommandExecutor.php@@ -23,7 +23,8 @@ class DatabaseStartCommandExecutor$secrets = method_exists($database, 'resolvedSecretManagerValuesForRedaction')? $database->resolvedSecretManagerValuesForRedaction(): [];- $remoteCommand = SshMultiplexingHelper::generateSshCommand($server, implode("\n", $commands));+ // The commands go to ssh on stdin, so the database passwords in them are not in a command line on the Coolify host.+ $remoteCommand = SshMultiplexingHelper::generateSshBatchCommand($server);$activity->properties = $activity->properties->merge(['status' => ProcessStatus::IN_PROGRESS->value]);$activity->save();@@ -34,6 +35,7 @@ class DatabaseStartCommandExecutortry {$process = Process::timeout($timeout)->idleTimeout(3600)+ ->input(SshMultiplexingHelper::sshBatchInput(implode("\n", $commands)))->start($remoteCommand, function (string $type, string $output) use ($activity, $secrets, &$pending): void {$pending[$type] = ($pending[$type] ?? '').$output;$end = max(strrpos($pending[$type], "\n"), strrpos($pending[$type], "\r"));diff --git a/app/Services/DockerRegistryLogins.php b/app/Services/DockerRegistryLogins.phpindex 8406fa9ec4..8b3d62b4fc 100644--- a/app/Services/DockerRegistryLogins.php+++ b/app/Services/DockerRegistryLogins.php@@ -216,10 +216,9 @@ class DockerRegistryLogins/*** Run docker login on the server, writing to the same config file that deployments mount.*- * The SSH wrapper sends this script to the server's shell on stdin, and the shell builtin echo passes the- * secret to docker on stdin, so it is not a process argument on the server. On the Coolify host it is still- * in the command line of the local shell that runs ssh, because SshMultiplexingHelper::generateSshCommand()- * embeds the script in a heredoc. It is kept out of SSH error logs and exception messages.+ * The script goes to the server's shell on ssh stdin, and the shell builtin echo passes the secret to docker+ * on stdin, so it is not a process argument on the Coolify host or on the server. It is kept out of SSH error+ * logs and exception messages.*/public static function login(Server $server, string $registry, string $username, string $password): void{@@ -277,7 +276,9 @@ class DockerRegistryLogins{$commands = $server->isNonRoot() ? parseCommandsByLineForSudo(collect([$command]), $server) : [$command];try {- $process = Process::timeout(60)->run(SshMultiplexingHelper::generateSshCommand($server, implode("\n", $commands)));+ $process = Process::timeout(60)+ ->input(SshMultiplexingHelper::sshBatchInput(implode("\n", $commands)))+ ->run(SshMultiplexingHelper::generateSshBatchCommand($server));} catch (ProcessTimedOutException|SymfonyProcessTimedOutException) {// The timeout message contains the full SSH command, including the registry password.throw new RuntimeException('The command on the server timed out after 60 seconds.', 124);diff --git a/app/Services/SentinelTrafficClient.php b/app/Services/SentinelTrafficClient.phpindex 1c5c28e2d8..5398456a33 100644--- a/app/Services/SentinelTrafficClient.php+++ b/app/Services/SentinelTrafficClient.php@@ -727,9 +727,9 @@ class SentinelTrafficClient$commands = $this->server->isNonRoot() ? parseCommandsByLineForSudo(collect([$command]), $this->server) : [$command];try {- $process = Process::timeout(self::REMOTE_TIMEOUT)->run(- SshMultiplexingHelper::generateSshCommand($this->server, implode("\n", $commands), commandTimeout: self::REMOTE_TIMEOUT)- );+ $process = Process::timeout(self::REMOTE_TIMEOUT)+ ->input(SshMultiplexingHelper::sshBatchInput(implode("\n", $commands)))+ ->run(SshMultiplexingHelper::generateSshBatchCommand($this->server, commandTimeout: self::REMOTE_TIMEOUT));if ($process->exitCode() !== 0) {excludeCertainErrors($process->errorOutput(), $process->exitCode());diff --git a/app/Services/ServerTransfer/ServerTransferBundle.php b/app/Services/ServerTransfer/ServerTransferBundle.phpindex dd75f61b2e..09cee9d951 100644--- a/app/Services/ServerTransfer/ServerTransferBundle.php+++ b/app/Services/ServerTransfer/ServerTransferBundle.php@@ -213,6 +213,7 @@ class ServerTransferBundle}foreach (self::items($environment['databases'] ?? null) as $d => $database) {+ self::checkDatabasePorts($errors, "{$at}.databases.{$d}", (array) ($database['attributes'] ?? []));self::checkStorages($errors, "{$at}.databases.{$d}", $database);self::checkScheduledBackups($errors, "{$at}.databases.{$d}", $database);}@@ -395,4 +396,38 @@ class ServerTransferBundleself::checkPath($errors, "{$at}.scheduled_backups.{$i}.databases_to_backup", $backup['databases_to_backup'] ?? null);}}++ /**+ * Checks the port values of a database. Older databases were saved before the UI checked them+ * (March 2026), so this accepts any public port in 1-65535, spaces around mappings, empty entries,+ * and container-only ports. It refuses values that are not ports, because they reach the compose file.+ *+ * @param list<string> $errors+ * @param array<string, mixed> $attributes+ */+ private static function checkDatabasePorts(array &$errors, string $at, array $attributes): void+ {+ if (array_key_exists('public_port', $attributes)) {+ self::check($errors, "{$at}.public_port", $attributes['public_port'], ['nullable', 'integer', 'between:1,65535']);+ }++ if (! array_key_exists('ports_mappings', $attributes) || $attributes['ports_mappings'] === null) {+ return;+ }+ $mappings = $attributes['ports_mappings'];+ if (! is_string($mappings)) {+ $errors[] = "{$at}.ports_mappings must be a string.";++ return;+ }+ foreach (explode(',', $mappings) as $mapping) {+ $mapping = trim($mapping, " \t");+ if ($mapping === '' || preg_match('/^\d+(?:-\d+)?(?:\/(?:tcp|udp|sctp))?$/', $mapping) === 1 || preg_match(ValidationPatterns::PORT_MAPPINGS_PATTERN, $mapping) === 1) {+ continue;+ }+ $errors[] = "{$at}.ports_mappings must be a comma-separated list of port mappings (e.g. 5432:5432,127.0.0.1:8080:80/udp).";++ return;+ }+ }}diff --git a/app/Support/DatabaseBackupFileValidator.php b/app/Support/DatabaseBackupFileValidator.phpindex 6e885123f3..eadf066bc6 100644--- a/app/Support/DatabaseBackupFileValidator.php+++ b/app/Support/DatabaseBackupFileValidator.php@@ -100,6 +100,8 @@ class DatabaseBackupFileValidatorreturn false;}+ $sql = self::withoutCopyData($sql);+// Token separator PostgreSQL treats as whitespace: real whitespace, a block comment// (the lazy match can extend to the outer end of a nested comment), or a line// comment. Comment forms also match inside string literals, which can only block@@ -120,6 +122,29 @@ class DatabaseBackupFileValidatorreturn preg_match('~(?:from|to)'.$sep.'+program|lo_(?:import|export)'.$sep.'*\(|alter'.$sep.'+system'.$sep.'+(?:set|reset)~is', $sql) !== 0;}+ /**+ * Removes the table data of COPY ... FROM stdin blocks: rows are data, not SQL, and ordinary+ * text such as "how to program" matched the patterns. Same rule as the restore scan in+ * DatabaseImportCommandBuilder: a block starts at the line "COPY ... FROM stdin;" as pg_dump+ * writes it and ends at the line "\.".+ */+ private static function withoutCopyData(string $sql): string+ {+ $kept = '';+ $offset = 0;++ while (preg_match('/^COPY .* FROM stdin;$/m', $sql, $match, PREG_OFFSET_CAPTURE, $offset) === 1) {+ $dataStart = $match[0][1] + strlen($match[0][0]);+ $kept .= substr($sql, $offset, $dataStart - $offset);+ $end = preg_match('/^\\\\\\.$/m', $sql, $terminator, PREG_OFFSET_CAPTURE, $dataStart) === 1+ ? $terminator[0][1] + 2+ : strlen($sql);+ $offset = $end;+ }++ return $kept.substr($sql, $offset);+ }+private static function extensionFor(string $name): ?string{$lower = strtolower($name);@@ -152,14 +177,14 @@ class DatabaseBackupFileValidator$sample = (string) file_get_contents($path, false, null, 0, 4096);return match ($extension) {- 'sql' => self::looksLikeText($sample) && ! self::containsPostgresqlProgramExecution($sample),+ 'sql' => self::looksLikeText($sample),'sql.gz', 'gz', 'tar.gz', 'tgz', 'bson.gz', 'archive.gz', 'db.gz', 'sqlite.gz', 'sqlite3.gz' => str_starts_with($sample, "\x1f\x8b"),'zip' => str_starts_with($sample, "PK\x03\x04") || str_starts_with($sample, "PK\x05\x06") || str_starts_with($sample, "PK\x07\x08"),'tar' => substr($sample, 257, 5) === 'ustar','bz2' => str_starts_with($sample, 'BZh'),'xz' => str_starts_with($sample, "\xfd7zXZ\x00"),'dump', 'bak', 'archive', 'dmp' => str_starts_with($sample, 'PGDMP')- || (self::looksLikeText($sample) && ! self::containsPostgresqlProgramExecution($sample)),+ || self::looksLikeText($sample),'bson' => self::looksLikeBson($path, $sample),'db', 'sqlite', 'sqlite3' => str_starts_with($sample, "SQLite format 3\0"),default => false,diff --git a/app/Support/DatabaseImport/DatabaseImportCommandBuilder.php b/app/Support/DatabaseImport/DatabaseImportCommandBuilder.phpindex 4535420e6f..851c47fa37 100644--- a/app/Support/DatabaseImport/DatabaseImportCommandBuilder.php+++ b/app/Support/DatabaseImport/DatabaseImportCommandBuilder.php@@ -54,6 +54,11 @@ tolower($0) ~ /^[ \t]*create[ \t]+(database|schema)[ \t]/ { if (is_system(databa!skip { print }AWK;+ /**+ * Prints a PostgreSQL SQL dump without the table data of its COPY ... FROM stdin blocks.+ */+ private const POSTGRES_COPY_DATA_FILTER = 'data { if ($0 == "\\\\.") data = 0; next } { print } /^COPY .* FROM stdin;$/ { data = 1 }';+/*** @param bool $keepOwners PostgreSQL archives: restore object owners and privileges. Off by default,* because roles from another host (for example RDS) usually do not exist here.@@ -135,13 +140,19 @@ SH;// Comments are no longer stripped: deleting the rest of a line on "--" also cut// inside string literals, which let "SELECT '--'; COPY ... PROGRAM" slip through.// A commented-out PROGRAM statement is now blocked too, which is the safer default.- $scan = static fn (string $source): string => "{$source} | LC_ALL=C grep -Eiq {$escapedPsqlPattern} || {$source} | tr -d '~' | tr '\\n\\r' '~~' | LC_ALL=C grep -Eiq {$escapedSqlPattern}";+ // Table data of COPY ... FROM stdin blocks is not scanned: it is data, not SQL, and ordinary+ // text such as "how to program" matched the patterns. pg_dump writes each block as the line+ // "COPY ... FROM stdin;", the rows, and the line "\.". When awk is missing or fails, the+ // scan receives a line that matches, so the restore is blocked.+ $withoutCopyData = static fn (string $source): string => "{ {$source} | LC_ALL=C awk ".escapeshellarg(self::POSTGRES_COPY_DATA_FILTER)." || echo '\\! from program'; }";+ $scan = static fn (string $source): string => $withoutCopyData($source)." | LC_ALL=C grep -Eiq {$escapedPsqlPattern} || ".$withoutCopyData($source)." | tr -d '~' | tr '\\n\\r' '~~' | LC_ALL=C grep -Eiq {$escapedSqlPattern}";$customScan = $scan('pg_restore -f - "$inspect" 2>/dev/null');$sqlScan = $scan($contents);$blockedProgram = 'echo \'Blocked PostgreSQL restore: the backup contains a COPY ... PROGRAM, psql shell command, lo_import/lo_export, or ALTER SYSTEM pattern. If you trust this dump, restore it from the database container terminal.\'; exit 1';$blockedInspect = 'echo \'Blocked PostgreSQL restore: unable to inspect custom archive.\'; exit 1';return <<<SH+command -v awk >/dev/null 2>&1 || { echo 'Blocked PostgreSQL restore: awk is required to scan the backup.'; exit 1; }header=\$({$contents} | head -c 5)tar_magic=\$({$contents} | head -c 262 | tail -c 5)if [ "\$header" = 'PGDMP' ] || [ "\$tar_magic" = 'ustar' ]; then@@ -270,6 +281,12 @@ SH;/*** Checks the backup format before any database is dropped, then recreates the* target database and restores archives with pg_restore and SQL with psql.+ * The restore must not stop on the first error: a pg_dumpall dump always creates the+ * superuser role (and creates the target database) that already exist here, and an+ * archive with owners references roles this server may not have. Stopping there would+ * leave the server without the databases that were just dropped.+ * pg_dumpall does not create the postgres database (it expects it to exist), so it is+ * recreated too when POSTGRES_DB names another database.*/private function postgresqlDumpAll(): string{@@ -287,10 +304,11 @@ check_superuserpsql -U "$pg_user" -d template1 -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname IS NOT NULL AND pid <> pg_backend_pid()" || exit 1psql -U "$pg_user" -d template1 -t -c "SELECT datname FROM pg_database WHERE NOT datistemplate" | xargs -I {} dropdb -U "$pg_user" --maintenance-db=template1 --if-exists {} || exit 1createdb -U "$pg_user" "$db" || exit 1+[ "$db" = postgres ] || createdb -U "$pg_user" postgres || exit 1if [ "$kind" = archive ]; then- stream | pg_restore --exit-on-error -U "$pg_user" -d "$db"+ stream | pg_restore -U "$pg_user" -d "$db"else- stream | psql -v ON_ERROR_STOP=1 -U "$pg_user" -d "$db"+ stream | psql -U "$pg_user" -d "$db"fiSH;}diff --git a/app/Traits/ExecuteRemoteCommand.php b/app/Traits/ExecuteRemoteCommand.phpindex 21912fb82f..4dffd37b5f 100644--- a/app/Traits/ExecuteRemoteCommand.php+++ b/app/Traits/ExecuteRemoteCommand.php@@ -159,9 +159,9 @@ trait ExecuteRemoteCommand}/**- * Execute the actual command with process handling. With $input, the content goes to the command over- * SSH stdin, because Linux limits one exec argument to 128 KiB and generateSshCommand() puts the whole- * script into one argument.+ * Execute the actual command with process handling. Without $input, the command goes to the remote shell+ * over SSH stdin, so its secrets are not in a command line on the Coolify host. With $input, the command+ * is the SSH argument and the content goes to it over SSH stdin.*/private function executeCommandWithProcess($command, $hidden, $customType, $append, $ignore_errors, $command_hidden = false, $skip_command_log = false, ?string $input = null){@@ -169,9 +169,12 @@ trait ExecuteRemoteCommand$this->application_deployment_queue->addLogEntry('[CMD]: '.$this->redact_sensitive_info($command), hidden: true);}- $remote_command = $input === null- ? SshMultiplexingHelper::generateSshCommand($this->server, $command)- : SshMultiplexingHelper::generateSshStdinCommand($this->server, $command);+ if ($input === null) {+ $remote_command = SshMultiplexingHelper::generateSshBatchCommand($this->server);+ $input = SshMultiplexingHelper::sshBatchInput((string) $command);+ } else {+ $remote_command = SshMultiplexingHelper::generateSshStdinCommand($this->server, $command);+ }$addLogEntry = function (string $type, string $output) use ($command, $hidden, $customType, $command_hidden, $skip_command_log): void {// Sanitize output to ensure valid UTF-8 encoding before JSON encoding$log_output = str(sanitize_utf8_text($output))->trim();diff --git a/bootstrap/helpers/applications.php b/bootstrap/helpers/applications.phpindex c6d7a20a0c..d8f2d27296 100644--- a/bootstrap/helpers/applications.php+++ b/bootstrap/helpers/applications.php@@ -1,6 +1,6 @@<?php-use App\Actions\Application\StopApplication;+use App\Actions\Shared\CloneVolumeDataWithSourceStopped;use App\Enums\ApplicationDeploymentStatus;use App\Jobs\ApplicationDeploymentJob;use App\Jobs\VolumeCloneJob;@@ -447,6 +447,7 @@ function clone_application(Application $source, $destination, array $overrides =}// Clone persistent volumes+ $pendingVolumeClones = [];$persistentVolumes = $source->persistentStorages()->get();foreach ($persistentVolumes as $volume) {$newName = '';@@ -472,28 +473,18 @@ function clone_application(Application $source, $destination, array $overrides =$newPersistentVolume->save();if ($cloneVolumeData) {- try {- StopApplication::dispatch($source, false, false);- $sourceVolume = $volume->name;- $targetVolume = $newPersistentVolume->name;- $sourceServer = $source->destination->server;- $targetServer = $newApplication->destination->server;-- VolumeCloneJob::dispatch($sourceVolume, $targetVolume, $sourceServer, $targetServer, $newPersistentVolume);-- queue_application_deployment(- deployment_uuid: new_public_id(),- application: $source,- server: $sourceServer,- destination: $source->destination,- no_questions_asked: true- );- } catch (Exception $e) {- Log::error('Failed to copy volume data for '.$volume->name.': '.$e->getMessage());- }+ $pendingVolumeClones[] = new VolumeCloneJob(+ $volume->name,+ $newPersistentVolume->name,+ $source->destination->server,+ $newApplication->destination->server,+ $newPersistentVolume,+ );}}+ CloneVolumeDataWithSourceStopped::run($source, $pendingVolumeClones);+// Clone file storages$fileStorages = $source->fileStorages()->get();foreach ($fileStorages as $storage) {diff --git a/bootstrap/helpers/auth.php b/bootstrap/helpers/auth.phpindex 564ae31062..4d7b1943ef 100644--- a/bootstrap/helpers/auth.php+++ b/bootstrap/helpers/auth.php@@ -3,19 +3,49 @@use Illuminate\Http\Request;/**- * The client IP for rate limits, allow-lists, and audit logs. Behind Cloudflare, $request->ip()- * is the Cloudflare edge, so a valid CF-Connecting-IP is used: on cloud, and on self-hosted- * instances when the request comes from a Cloudflare IP. A client that connects directly- * cannot set the client IP with the header.+ * The client IP for rate limits, allow-lists, and audit logs.+ *+ * TrustProxies trusts every peer, so a client that connects directly to the dashboard port could+ * set $request->ip() with X-Forwarded-For. The TCP peer (REMOTE_ADDR) decides what to trust:+ * - A public peer is the client. Forwarded headers are ignored, except CF-Connecting-IP when the+ * peer is a Cloudflare edge.+ * - A private or loopback peer is a local reverse proxy (coolify-proxy, cloudflared, a Docker+ * network), so the address that proxy forwarded is used. CF-Connecting-IP is used only when+ * that address is a Cloudflare edge.+ * On cloud, which runs behind Cloudflare, a valid CF-Connecting-IP is always used.*/function client_ip(Request $request): string{- $ip = (string) $request->ip();$cloudflareIp = $request->header('CF-Connecting-IP');+ $cloudflareIp = is_string($cloudflareIp) && filter_var($cloudflareIp, FILTER_VALIDATE_IP) !== false ? $cloudflareIp : null;- if (is_string($cloudflareIp) && filter_var($cloudflareIp, FILTER_VALIDATE_IP) !== false && (isCloud() || isCloudflareIp($ip))) {+ $peer = (string) $request->server('REMOTE_ADDR');+ $hop = isLocalProxyIp($peer) ? (string) $request->ip() : $peer;++ if ($cloudflareIp !== null && (isCloud() || isCloudflareIp($hop))) {return $cloudflareIp;}- return $ip;+ return $hop;+}++/**+ * Whether an address is loopback, private, or link-local (IPv4, IPv6, or IPv4-mapped IPv6): a+ * reverse proxy on the same host or Docker network rather than a client on the internet.+ */+function isLocalProxyIp(string $ip): bool+{+ if (filter_var($ip, FILTER_VALIDATE_IP) === false) {+ return false;+ }++ $packed = inet_pton($ip);+ if (strlen($packed) === 16 && str_starts_with($packed, str_repeat("\0", 10)."\xff\xff")) {+ $ip = inet_ntop(substr($packed, 12));+ }++ return ipMatch($ip, [+ '127.0.0.0/8', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16', '169.254.0.0/16',+ '::1/128', 'fc00::/7', 'fe80::/10',+ ]);}diff --git a/bootstrap/helpers/parsers.php b/bootstrap/helpers/parsers.phpindex 5e34a4dc4c..8ae596a1e1 100644--- a/bootstrap/helpers/parsers.php+++ b/bootstrap/helpers/parsers.php@@ -1151,6 +1151,8 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int$pullRequestId = $pull_request_id;$isPullRequest = $pullRequestId == 0 ? false : true;+ // A public preview can run the compose file of a fork pull request.+ $isPublicPullRequest = $isPullRequest && (bool) data_get($resource, 'settings.is_pr_deployments_public_enabled');$server = data_get($resource, 'destination.server');try {$yaml = parseDockerComposeYaml($compose);@@ -1216,12 +1218,6 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int// convert environment variables to one format$environment = convertToKeyValueCollection($environment);- // Add Coolify defined environments- $allEnvironments = $resource->environment_variables()->get(['key', 'value']);-- $allEnvironments = $allEnvironments->mapWithKeys(function ($item) {- return [$item['key'] => $item['value']];- });// filter and add magic environmentsforeach ($environment as $key => $value) {// Get all SERVICE_ variables from keys and values@@ -1269,7 +1265,8 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int$resource->save();}- if (! $parsed['has_port']) {+ // The stored variables belong to the main application; a preview deployment gets its own domains.+ if (! $parsed['has_port'] && ! $isPullRequest) {$resource->environment_variables()->updateOrCreate(['key' => $key->value(),'resourceable_type' => get_class($resource),@@ -1279,7 +1276,7 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int'is_preview' => false,]);}- if ($parsed['has_port']) {+ if ($parsed['has_port'] && ! $isPullRequest) {$newKey = str($key)->beforeLast('_');$resource->environment_variables()->updateOrCreate([@@ -1375,45 +1372,48 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int$fqdnValueForEnvWithPort = "$fqdnValueForEnv:$port";}- // ALWAYS create base SERVICE_FQDN variable (host only, no scheme)- $resource->environment_variables()->firstOrCreate([- 'key' => "SERVICE_FQDN_{$serviceNamePreserved}",- 'resourceable_type' => get_class($resource),- 'resourceable_id' => $resource->id,- ], [- 'value' => $fqdnValueForEnv,- 'is_preview' => false,- ]);-- // ALWAYS create base SERVICE_URL variable (with scheme)- $resource->environment_variables()->firstOrCreate([- 'key' => "SERVICE_URL_{$serviceNamePreserved}",- 'resourceable_type' => get_class($resource),- 'resourceable_id' => $resource->id,- ], [- 'value' => $url,- 'is_preview' => false,- ]);-- // If port-specific, ALSO create port-specific pairs- if ($parsed['has_port'] && $port) {+ // The stored variables belong to the main application; a preview deployment gets its own domains.+ if (! $isPullRequest) {+ // ALWAYS create base SERVICE_FQDN variable (host only, no scheme)$resource->environment_variables()->firstOrCreate([- 'key' => "SERVICE_FQDN_{$serviceNamePreserved}_{$port}",+ 'key' => "SERVICE_FQDN_{$serviceNamePreserved}",'resourceable_type' => get_class($resource),'resourceable_id' => $resource->id,], [- 'value' => $fqdnValueForEnvWithPort,+ 'value' => $fqdnValueForEnv,'is_preview' => false,]);+ // ALWAYS create base SERVICE_URL variable (with scheme)$resource->environment_variables()->firstOrCreate([- 'key' => "SERVICE_URL_{$serviceNamePreserved}_{$port}",+ 'key' => "SERVICE_URL_{$serviceNamePreserved}",'resourceable_type' => get_class($resource),'resourceable_id' => $resource->id,], [- 'value' => $urlWithPort,+ 'value' => $url,'is_preview' => false,]);++ // If port-specific, ALSO create port-specific pairs+ if ($parsed['has_port'] && $port) {+ $resource->environment_variables()->firstOrCreate([+ 'key' => "SERVICE_FQDN_{$serviceNamePreserved}_{$port}",+ 'resourceable_type' => get_class($resource),+ 'resourceable_id' => $resource->id,+ ], [+ 'value' => $fqdnValueForEnvWithPort,+ 'is_preview' => false,+ ]);++ $resource->environment_variables()->firstOrCreate([+ 'key' => "SERVICE_URL_{$serviceNamePreserved}_{$port}",+ 'resourceable_type' => get_class($resource),+ 'resourceable_id' => $resource->id,+ ], [+ 'value' => $urlWithPort,+ 'is_preview' => false,+ ]);+ }}if ($resource->build_pack === 'dockercompose') {@@ -1437,6 +1437,21 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int}}}+ } elseif ($isPullRequest) {+ // A preview deployment reads its preview variables, so a pull request parse generates the value there.+ $previewVariable = $resource->environment_variables_preview()->firstOrCreate([+ 'key' => $key->value(),+ 'resourceable_type' => get_class($resource),+ 'resourceable_id' => $resource->id,+ ], [+ 'value' => generateEnvValue($command, $resource),+ 'is_preview' => true,+ ]);+ // A public preview does not get the production secret; it gets its own value.+ if ($isPublicPullRequest && ! $previewVariable->wasRecentlyCreated+ && $previewVariable->value === $resource->environment_variables()->where('key', $key->value())->first()?->value) {+ $previewVariable->update(['value' => generateEnvValue($command, $resource)]);+ }} else {$value = generateEnvValue($command, $resource);$resource->environment_variables()->firstOrCreate([@@ -1452,6 +1467,9 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int}}+ // A pull request parse keeps its magic variables in the preview variables (see above), not in the production variables.+ $isPullRequestMagicVariable = fn (string $key): bool => $isPullRequest && $allMagicEnvironments->has($key);+// generate SERVICE_NAME variables for docker compose services$serviceNameEnvironments = collect([]);if ($resource->build_pack === 'dockercompose') {@@ -1863,15 +1881,17 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int$isRequired = str_contains($operator, '?');// Create the primary variable with its default (only if it doesn't exist)- $envVar = $resource->environment_variables()->firstOrCreate([- 'key' => $varName,- 'resourceable_type' => get_class($resource),- 'resourceable_id' => $resource->id,- ], [- 'value' => $isRequired ? null : $defaultValue,- 'is_preview' => false,- 'is_required' => $isRequired,- ]);+ if (! $isPullRequestMagicVariable($varName)) {+ $envVar = $resource->environment_variables()->firstOrCreate([+ 'key' => $varName,+ 'resourceable_type' => get_class($resource),+ 'resourceable_id' => $resource->id,+ ], [+ 'value' => $isRequired ? null : $defaultValue,+ 'is_preview' => false,+ 'is_required' => $isRequired,+ ]);+ }// Recursively process nested variables in default valueif (str_contains($defaultValue, '${')) {@@ -1887,7 +1907,7 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int// Skip SERVICE_URL_* and SERVICE_FQDN_* variables - they are handled by magic variable system$isMagicVariable = str_starts_with($nestedVarName, 'SERVICE_URL_') || str_starts_with($nestedVarName, 'SERVICE_FQDN_');- if (! $isMagicVariable) {+ if (! $isMagicVariable && ! $isPullRequestMagicVariable($nestedVarName)) {if ($nestedSplit !== null) {$nestedEnvVar = $resource->environment_variables()->firstOrCreate(['key' => $nestedSplit['variable'],@@ -1918,14 +1938,16 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int} else {// Simple variable reference without default$parsedKeyValue = replaceVariables($value);- $envVar = $resource->environment_variables()->firstOrCreate([- 'key' => $content,- 'resourceable_type' => get_class($resource),- 'resourceable_id' => $resource->id,- ], [- 'is_preview' => false,- 'is_required' => $isRequired,- ]);+ if (! $isPullRequestMagicVariable($content)) {+ $envVar = $resource->environment_variables()->firstOrCreate([+ 'key' => $content,+ 'resourceable_type' => get_class($resource),+ 'resourceable_id' => $resource->id,+ ], [+ 'is_preview' => false,+ 'is_required' => $isRequired,+ ]);+ }}} else {// Fallback to old behavior for malformed input (backward compatibility)@@ -2108,7 +2130,8 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?intif ($environment->count() > 0) {$environment = $environment->filter(function ($value, $key) {return ! str($key)->startsWith('SERVICE_FQDN_');- })->map(function ($value, $key) use ($resource, $isPullRequest) {+ })->map(function ($value, $key) use ($resource, $isPullRequest, $isPublicPullRequest) {+ $isValueFromDatabase = false;// Preserve empty strings and null values with correct Docker Compose semantics:// - Empty string: Variable is set to "" (e.g., HTTP_PROXY="" means "no proxy")// - Null: Variable is unset/removed from container environment (may inherit from host)@@ -2119,10 +2142,18 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int} elseif ($value === '' && ! ($isPullRequest && str($key)->startsWith('SERVICE_URL_'))) {// Empty string - allow database override for backward compatibility.// The stored service URLs belong to the main application, so a preview does not use them.- $dbEnv = $resource->environment_variables()->where('key', $key)->first();+ // A preview uses its preview variables, like ApplicationDeploymentJob does.+ $dbEnv = $isPullRequest+ ? $resource->environment_variables_preview()->where('key', $key)->first()+ : $resource->environment_variables()->where('key', $key)->first();+ // A public preview does not get preview variables that still have the production value.+ if ($dbEnv && $isPublicPullRequest && $resource->environment_variables()->where('key', $key)->first()?->value === $dbEnv->value) {+ $dbEnv = null;+ }// Only use database override if it exists AND has a non-empty valueif ($dbEnv && str($dbEnv->value)->isNotEmpty()) {$value = $dbEnv->value;+ $isValueFromDatabase = true;}// Otherwise keep empty string as-is}@@ -2130,7 +2161,8 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int// Resolve shared variable patterns like {{environment.VAR}}, {{project.VAR}}, {{team.VAR}}// Without this, literal {{...}} strings end up in the compose environment: section,// which takes precedence over the resolved values in the .env file (env_file:)- if (is_string($value) && str_contains($value, '{{')) {+ // A public preview resolves them only in its own variables, not in the compose file of the pull request.+ if (is_string($value) && str_contains($value, '{{') && ($isValueFromDatabase || ! $isPublicPullRequest)) {$value = resolveSharedEnvironmentVariables($value, $resource);}diff --git a/bootstrap/helpers/remoteProcess.php b/bootstrap/helpers/remoteProcess.phpindex e5dd73de46..0c8bb86ebd 100644--- a/bootstrap/helpers/remoteProcess.php+++ b/bootstrap/helpers/remoteProcess.php@@ -192,8 +192,8 @@ function instant_remote_process_with_timeout(Collection|array $command, Server $return SshRetryHandler::retry(function () use ($server, $command_string) {- $sshCommand = SshMultiplexingHelper::generateSshCommand($server, $command_string);- $process = runSshProcess($sshCommand, 30);+ $sshCommand = SshMultiplexingHelper::generateSshBatchCommand($server);+ $process = runSshProcess($sshCommand, 30, SshMultiplexingHelper::sshBatchInput($command_string));$output = trim($process->output());$exitCode = $process->exitCode();@@ -228,8 +228,8 @@ function instant_remote_process(Collection|array $command, Server $server, boolreturn SshRetryHandler::retry(function () use ($server, $command_string, $effectiveTimeout, $disableMultiplexing) {- $sshCommand = SshMultiplexingHelper::generateSshCommand($server, $command_string, $disableMultiplexing, (int) $effectiveTimeout);- $process = runSshProcess($sshCommand, (int) $effectiveTimeout);+ $sshCommand = SshMultiplexingHelper::generateSshBatchCommand($server, $disableMultiplexing, (int) $effectiveTimeout);+ $process = runSshProcess($sshCommand, (int) $effectiveTimeout, SshMultiplexingHelper::sshBatchInput($command_string));$output = trim($process->output());$exitCode = $process->exitCode();diff --git a/bootstrap/helpers/services.php b/bootstrap/helpers/services.phpindex 7c664649cb..bfb5f22759 100644--- a/bootstrap/helpers/services.php+++ b/bootstrap/helpers/services.php@@ -25,8 +25,7 @@ function service_logo_urls(mixed $logo): array];}-use App\Actions\Service\StartService;-use App\Actions\Service\StopService;+use App\Actions\Shared\CloneVolumeDataWithSourceStopped;use App\Jobs\VolumeCloneJob;use App\Models\Application;use App\Models\GithubApp;@@ -38,8 +37,6 @@ use App\Models\ServiceDatabase;use App\Models\StandaloneDocker;use App\Models\SwarmDocker;use Illuminate\Database\Eloquent\Model;-use Illuminate\Support\Facades\Bus;-use Illuminate\Support\Facades\DB;use Illuminate\Support\Facades\Log;use Illuminate\Support\Stringable;use Spatie\Url\Url;@@ -778,22 +775,11 @@ function clone_service(Service $source, StandaloneDocker|SwarmDocker $destinatio$sourceServer = $source->destination?->server;$targetServer = $newService->destination?->server;- if ($pendingVolumeClones !== [] && $sourceServer && $targetServer) {- $chain = [- function () use ($source) {- StopService::run($source);- },- ...array_map(- fn (array $clone) => new VolumeCloneJob($clone[0], $clone[1]->name, $sourceServer, $targetServer, $clone[1]),- $pendingVolumeClones,- ),- function () use ($source) {- StartService::run($source);- },- ];-- // The jobs load the new volume rows, so they must not run before the transaction of the caller commits.- DB::afterCommit(fn () => Bus::chain($chain)->onQueue('high')->dispatch());+ if ($sourceServer && $targetServer) {+ CloneVolumeDataWithSourceStopped::run($source, array_map(+ fn (array $clone) => new VolumeCloneJob($clone[0], $clone[1]->name, $sourceServer, $targetServer, $clone[1]),+ $pendingVolumeClones,+ ));}return $newService;diff --git a/bootstrap/helpers/subscriptions.php b/bootstrap/helpers/subscriptions.phpindex f91c165e48..951fdeab1b 100644--- a/bootstrap/helpers/subscriptions.php+++ b/bootstrap/helpers/subscriptions.php@@ -122,10 +122,13 @@ function allowedPathsForTwoFactorSetup(): array{return ['two-factor-required',+ 'two-factor-required/switch-team','two-factor-challenge','logout','user/confirm-password','user/confirmed-password-status',+ 'passkeys/confirm',+ 'passkeys/confirm/options','user/two-factor-authentication','user/confirmed-two-factor-authentication','user/two-factor-qr-code',diff --git a/docker-compose.prod.yml b/docker-compose.prod.ymlindex 3a63b279d7..8e214b385e 100644--- a/docker-compose.prod.yml+++ b/docker-compose.prod.yml@@ -3,8 +3,9 @@ services:# Realtime runs inside the coolify container now. Without this entry, such an override is an# incomplete service and every upgrade fails. Zero replicas means it never starts, and# "docker compose up" removes the old coolify-realtime container (an inactive profile keeps it).+ # It uses the coolify image, so install and upgrade do not pull another image.soketi:- image: busybox:1.37+ image: "${REGISTRY_URL:-docker.io}/coollabsio/coolify:${LATEST_IMAGE:-latest}"deploy:replicas: 0coolify:diff --git a/openapi.json b/openapi.jsonindex 4ed696c239..3d24978d9c 100644--- a/openapi.json+++ b/openapi.json@@ -5395,7 +5395,7 @@"Applications"],"summary": "Update Storage",- "description": "Update a persistent storage or file storage by application UUID.",+ "description": "Update a persistent storage or file storage by application UUID. Changed file content is saved right away and written to the server in the background, so a failed write does not change the response.","operationId": "update-storage-by-application-uuid","parameters": [{@@ -10286,7 +10286,7 @@"Databases"],"summary": "Update Storage",- "description": "Update a persistent storage or file storage by database UUID.",+ "description": "Update a persistent storage or file storage by database UUID. Changed file content is saved right away and written to the server in the background, so a failed write does not change the response.","operationId": "update-storage-by-database-uuid","parameters": [{@@ -17932,7 +17932,7 @@"Servers"],"summary": "Migrate server to another Coolify instance",- "description": "Start a transfer in the background: export this server, import and claim it on the target instance (with the given token), then disable automations here. Follow the progress with GET \/servers\/{uuid}\/transfer. The target URL must use https. Requires read:sensitive and write.",+ "description": "Start a transfer in the background: export this server, import and claim it on the target instance (with the given token), then disable automations here. All secrets of the server, also locked environment variables, are sent to the target. Follow the progress with GET \/servers\/{uuid}\/transfer. The target URL must use https. Requires read:sensitive and write, and the owner role of the server team.","operationId": "migrate-server-between-instances","parameters": [{@@ -17973,7 +17973,7 @@"description": "Transfer started"},"403": {- "description": "Missing sensitive permission"+ "description": "Missing sensitive permission or not an owner of the server team"},"404": {"$ref": "#\/components\/responses\/404"@@ -21735,7 +21735,7 @@"Services"],"summary": "Update Storage",- "description": "Update a persistent storage or file storage by service UUID.",+ "description": "Update a persistent storage or file storage by service UUID. Changed file content is saved right away and written to the server in the background, so a failed write does not change the response.","operationId": "update-storage-by-service-uuid","parameters": [{diff --git a/openapi.yaml b/openapi.yamlindex f1f4987405..4f02ba373d 100644--- a/openapi.yaml+++ b/openapi.yaml@@ -3545,7 +3545,7 @@ paths:tags:- Applicationssummary: 'Update Storage'- description: 'Update a persistent storage or file storage by application UUID.'+ description: 'Update a persistent storage or file storage by application UUID. Changed file content is saved right away and written to the server in the background, so a failed write does not change the response.'operationId: update-storage-by-application-uuidparameters:-@@ -6738,7 +6738,7 @@ paths:tags:- Databasessummary: 'Update Storage'- description: 'Update a persistent storage or file storage by database UUID.'+ description: 'Update a persistent storage or file storage by database UUID. Changed file content is saved right away and written to the server in the background, so a failed write does not change the response.'operationId: update-storage-by-database-uuidparameters:-@@ -11335,7 +11335,7 @@ paths:tags:- Serverssummary: 'Migrate server to another Coolify instance'- description: 'Start a transfer in the background: export this server, import and claim it on the target instance (with the given token), then disable automations here. Follow the progress with GET /servers/{uuid}/transfer. The target URL must use https. Requires read:sensitive and write.'+ description: 'Start a transfer in the background: export this server, import and claim it on the target instance (with the given token), then disable automations here. All secrets of the server, also locked environment variables, are sent to the target. Follow the progress with GET /servers/{uuid}/transfer. The target URL must use https. Requires read:sensitive and write, and the owner role of the server team.'operationId: migrate-server-between-instancesparameters:-@@ -11364,7 +11364,7 @@ paths:'202':description: 'Transfer started''403':- description: 'Missing sensitive permission'+ description: 'Missing sensitive permission or not an owner of the server team''404':$ref: '#/components/responses/404''422':@@ -13712,7 +13712,7 @@ paths:tags:- Servicessummary: 'Update Storage'- description: 'Update a persistent storage or file storage by service UUID.'+ description: 'Update a persistent storage or file storage by service UUID. Changed file content is saved right away and written to the server in the background, so a failed write does not change the response.'operationId: update-storage-by-service-uuidparameters:-diff --git a/other/nightly/docker-compose.prod.yml b/other/nightly/docker-compose.prod.ymlindex 3a63b279d7..8e214b385e 100644--- a/other/nightly/docker-compose.prod.yml+++ b/other/nightly/docker-compose.prod.yml@@ -3,8 +3,9 @@ services:# Realtime runs inside the coolify container now. Without this entry, such an override is an# incomplete service and every upgrade fails. Zero replicas means it never starts, and# "docker compose up" removes the old coolify-realtime container (an inactive profile keeps it).+ # It uses the coolify image, so install and upgrade do not pull another image.soketi:- image: busybox:1.37+ image: "${REGISTRY_URL:-docker.io}/coollabsio/coolify:${LATEST_IMAGE:-latest}"deploy:replicas: 0coolify:diff --git a/other/nightly/install.sh b/other/nightly/install.shindex b00f35da86..7a1783082a 100755--- a/other/nightly/install.sh+++ b/other/nightly/install.sh@@ -1028,8 +1028,8 @@ if [ "$IS_COOLIFY_VOLUME_EXISTS" -eq 0 ]; thentest -f /data/coolify/ssh/keys/[email protected] && rm -f /data/coolify/ssh/keys/[email protected]ssh-keygen -t ed25519 -a 100 -f /data/coolify/ssh/keys/[email protected] -q -N "" -C coolifychown 9999 /data/coolify/ssh/keys/[email protected]- # Remove only the key from a previous install (generated with the comment "coolify"), not other keys that contain "coolify"- sed -i -E "/[[:space:]]coolify[[:space:]]*$/d" ~/.ssh/authorized_keys+ # Remove only the key from a previous install (comment "coolify", or "root@coolify" before September 2024), not other keys that contain "coolify"+ sed -i -E "/[[:space:]](root@)?coolify[[:space:]]*$/d" ~/.ssh/authorized_keyscat /data/coolify/ssh/keys/[email protected] >>~/.ssh/authorized_keysrm -f /data/coolify/ssh/keys/[email protected]fidiff --git a/resources/views/components/live-toggle.blade.php b/resources/views/components/live-toggle.blade.phpindex 2716d7c826..56bbca6fd3 100644--- a/resources/views/components/live-toggle.blade.php+++ b/resources/views/components/live-toggle.blade.php@@ -1,11 +1,14 @@{{--Live refresh toggle. Entangles the `live` bool of the host Livewire component, so it workswith any time range. The button shows "Live" with a pulsing dot when refresh is on and- "Paused" with a play icon when it is off. `storageKey` keeps the choice per browser.+ "Paused" with a play icon when it is off. `storageKey` keeps the last choice the user clicked per+ browser; changes made by the component (for example a range that starts paused) are not stored.+ A stored "off" always applies on page load; a stored "on" applies only when `restoreLive` is true.--}}@props(['active' => false,'storageKey' => null,+ 'restoreLive' => true,'title' => 'Turn live refresh on or off',])@@ -13,10 +16,9 @@@if ($storageKey) x-init="const stored = localStorage.getItem(@js($storageKey));if (stored === '0') { live = false; }- if (stored === '1') { live = true; }- $watch('live', value => localStorage.setItem(@js($storageKey), value ? '1' : '0'));+ if (stored === '1' && @js((bool) $restoreLive)) { live = true; }" @endif>- <button type="button" @click="live = !live" :aria-pressed="live ? 'true' : 'false'"+ <button type="button" @click="live = !live; @if ($storageKey) localStorage.setItem(@js($storageKey), live ? '1' : '0'); @endif" :aria-pressed="live ? 'true' : 'false'"aria-pressed="{{ $active ? 'true' : 'false' }}" title="{{ $title }}"class="inline-flex h-7 items-center gap-1.5 rounded-md px-2.5 text-[12px] font-medium ring-1 ring-neutral-200 transition-colors dark:ring-white/[0.08]":class="livediff --git a/resources/views/livewire/fleet/metrics.blade.php b/resources/views/livewire/fleet/metrics.blade.phpindex 11bd7abe14..cf258ebc0e 100644--- a/resources/views/livewire/fleet/metrics.blade.php+++ b/resources/views/livewire/fleet/metrics.blade.php@@ -52,14 +52,14 @@ $serverListboxOptions = array_merge(<x-loading compact aria-label="Loading metrics" /></div></div>- <x-live-toggle :active="$live" storageKey="metrics-live" title="Turn live refresh on or off (updates every 60 seconds)" />+ <x-live-toggle :active="$live" storageKey="metrics-live" :restoreLive="$range === '24h'" title="Turn live refresh on or off (updates every 60 seconds)" /></div></div>@endif</div>- @if ($live)- <div wire:poll.60s="loadData" class="hidden"></div>+ @if ($this->pollSeconds())+ <div wire:poll.{{ $this->pollSeconds() }}s="pollData" class="hidden"></div>@endif@if ($servers->isEmpty())diff --git a/resources/views/livewire/project/application/previews.blade.php b/resources/views/livewire/project/application/previews.blade.phpindex 8a898cd3e8..01df79cdfe 100644--- a/resources/views/livewire/project/application/previews.blade.php+++ b/resources/views/livewire/project/application/previews.blade.php@@ -86,7 +86,7 @@<div class="w-full"><x-forms.listbox id="isPrDeploymentsPublicEnabled" label="PR deployment access" onChange="savePreviewSettings"- helper="When public, anyone can trigger PR deployments. Otherwise only PRs from branches in the same repository are deployed and fork PRs are blocked.<br><br><span class='font-bold'>Secrets:</span> the code of a fork PR can read all environment variables of its preview. When public, previews do not get preview variables that still have the production value. Give previews their own values, and do not use production secrets in them."+ helper="When public, anyone can trigger PR deployments. Otherwise only PRs from branches in the same repository are deployed and fork PRs are blocked.<br><br><span class='font-bold'>Secrets:</span> the code of a fork PR can read all environment variables of its preview. When public, previews do not get preview variables that still have the production value, and shared variables are resolved only in preview variables, not in the compose file of the PR. This is not full isolation: the preview runs on the same server as production, and with the Docker Compose build pack Coolify deploys the compose file of the PR, which can, for example, mount host paths. Give previews their own values, and do not use production secrets in them.":options="[['value' => false, 'label' => 'Repository members only'],['value' => true, 'label' => 'Public (fork PRs allowed)'],diff --git a/resources/views/livewire/project/clone-me.blade.php b/resources/views/livewire/project/clone-me.blade.phpindex 9e2b941e31..ab9452068c 100644--- a/resources/views/livewire/project/clone-me.blade.php+++ b/resources/views/livewire/project/clone-me.blade.php@@ -20,6 +20,10 @@<div class="max-w-md"><x-forms.input required id="newName" label="New name" /></div>+ <div class="mt-4 max-w-md">+ <x-forms.checkbox id="cloneVolumeData" live label="Clone volume data"+ helper="Copies the data of the persistent volumes to the clones. Running source resources are stopped while the data is copied and started again afterwards." />+ </div></div></section>@@ -138,14 +142,35 @@</div><divclass="flex flex-col gap-2 border-t border-neutral-200 p-4 sm:flex-row sm:justify-end dark:border-white/[0.06]">- <x-forms.button isHighlighted wire:click="clone('environment')"- :disabled="! filled($selectedDestination)">- Clone to environment- </x-forms.button>- <x-forms.button isHighlighted wire:click="clone('project')"- :disabled="! filled($selectedDestination)">- Clone to project- </x-forms.button>+ @if ($cloneVolumeData)+ @foreach (['environment' => 'Clone to environment', 'project' => 'Clone to project'] as $cloneType => $cloneLabel)+ <div wire:key="clone-with-volume-data-{{ $cloneType }}">+ <x-modal-confirmation title="Clone with volume data?" submitAction="clone('{{ $cloneType }}')"+ :ignoreWire="false" :confirmWithText="false" :confirmWithPassword="false"+ :step2ButtonText="$cloneLabel"+ warningMessage="Running source resources are unavailable while their volume data is copied."+ :actions="[+ 'Running source resources are stopped while their volume data is copied.',+ 'They are started again after the copy, also when the copy fails.',+ ]">+ <x-slot:trigger>+ <x-forms.button isHighlighted :disabled="! filled($selectedDestination)">+ {{ $cloneLabel }}+ </x-forms.button>+ </x-slot:trigger>+ </x-modal-confirmation>+ </div>+ @endforeach+ @else+ <x-forms.button isHighlighted wire:click="clone('environment')"+ :disabled="! filled($selectedDestination)">+ Clone to environment+ </x-forms.button>+ <x-forms.button isHighlighted wire:click="clone('project')"+ :disabled="! filled($selectedDestination)">+ Clone to project+ </x-forms.button>+ @endif</div></div></section>diff --git a/resources/views/livewire/project/shared/metrics.blade.php b/resources/views/livewire/project/shared/metrics.blade.phpindex 341c66c68c..98fea1f754 100644--- a/resources/views/livewire/project/shared/metrics.blade.php+++ b/resources/views/livewire/project/shared/metrics.blade.php@@ -38,8 +38,8 @@icon-name="dashboard" /></x-application.settings-section>@else- @if ($live)- <div wire:key="metrics-poll-{{ $this->pollSeconds() }}" wire:poll.{{ $this->pollSeconds() }}s="loadData" class="hidden"></div>+ @if ($this->pollSeconds())+ <div wire:key="metrics-poll-{{ $this->pollSeconds() }}" wire:poll.{{ $this->pollSeconds() }}s="pollData" class="hidden"></div>@endif<x-application.settings-section id="metrics-overview-section" title="Metrics"helper="Inspect recent CPU and memory usage reported by Sentinel.">@@ -58,7 +58,7 @@]" /></div><p class="mt-3 text-xs leading-5 text-neutral-500 dark:text-fg-dim">- While live, ranges up to one hour refresh every five seconds. Longer ranges refresh every minute.+ Ranges up to one hour refresh live every five seconds. Longer ranges load once; turn on live refresh to update them every minute.</p></x-application.settings-section>diff --git a/resources/views/livewire/project/shared/resource-operations.blade.php b/resources/views/livewire/project/shared/resource-operations.blade.phpindex e1670db563..848af355ae 100644--- a/resources/views/livewire/project/shared/resource-operations.blade.php+++ b/resources/views/livewire/project/shared/resource-operations.blade.php@@ -1,4 +1,5 @@-<div x-on:migrate-resource-confirmed.window="$wire.migrateTo(selectedMigrateDestination)" x-data="{+<div x-on:migrate-resource-confirmed.window="$wire.migrateTo(selectedMigrateDestination)"+ x-on:clone-resource-confirmed.window="$wire.cloneTo(selectedCloneDestination, selectedCloneEnvironment)" x-data="{selectedCloneServer: null,selectedCloneDestination: null,selectedCloneProject: null,@@ -153,9 +154,9 @@@can('update', $resource)<x-application.settings-section id="clone-resource-section" title="Clone resource"helper="Create a copy of this resource. Choose where it runs and where it belongs. The current values are selected by default.">- <x-callout type="info" title="Configuration only">+ <x-callout type="info" title="What is copied">Cloning copies settings, environment variables, and resource configuration. Stored files,- database records, and other persistent data are not copied.+ database records, and other persistent data are copied only when you select Clone volume data.</x-callout><div class="mt-4 grid gap-4 md:grid-cols-2">@@ -179,15 +180,41 @@placeholder="Choose an environment…" /></div>+ <div class="mt-4">+ <x-forms.checkbox id="cloneVolumeData" live label="Clone volume data"+ helper="Copies the data of the persistent volumes to the clone. A running source resource is stopped while the data is copied and started again afterwards." />+ </div>+<divclass="mt-4 flex flex-col gap-3 border-t border-neutral-200 pt-4 sm:flex-row sm:items-center sm:justify-between dark:border-white/[0.07]">- <p class="text-[13px] text-neutral-500 dark:text-fg-dim">- The running resource will not be changed.- </p>- <x-forms.button x-bind:disabled="!selectedCloneDestination || !selectedCloneEnvironment"- @click="$wire.cloneTo(selectedCloneDestination, selectedCloneEnvironment)">- Clone resource- </x-forms.button>+ @if ($cloneVolumeData)+ <p class="text-[13px] text-neutral-500 dark:text-fg-dim">+ A running resource is stopped while its volume data is copied.+ </p>+ <x-modal-confirmation title="Clone this resource with its volume data?" submitAction="clone-resource-confirmed"+ dispatchAction :confirmWithText="false" :confirmWithPassword="false"+ step2ButtonText="Clone resource"+ warningMessage="A running resource is unavailable while its volume data is copied."+ :actions="[+ 'If the resource is running, it is stopped while its volume data is copied.',+ 'The resource is started again after the copy, also when the copy fails.',+ ]">+ <x-slot:trigger>+ <x-forms.button x-bind:disabled="!selectedCloneDestination || !selectedCloneEnvironment">+ Clone resource+ </x-forms.button>+ </x-slot:trigger>+ </x-modal-confirmation>+ @else+ <p class="text-[13px] text-neutral-500 dark:text-fg-dim">+ The running resource will not be changed.+ </p>+ <x-forms.button wire:key="clone-resource-without-volume-data"+ x-bind:disabled="!selectedCloneDestination || !selectedCloneEnvironment"+ @click="$wire.cloneTo(selectedCloneDestination, selectedCloneEnvironment)">+ Clone resource+ </x-forms.button>+ @endif</div></x-application.settings-section>diff --git a/resources/views/livewire/server/charts.blade.php b/resources/views/livewire/server/charts.blade.phpindex 43eb732701..60841dd60d 100644--- a/resources/views/livewire/server/charts.blade.php+++ b/resources/views/livewire/server/charts.blade.php@@ -12,8 +12,8 @@<div class="application-settings-form flex w-full flex-col gap-6"@if ($server->isMetricsEnabled()) x-init="$wire.loadData()" @endif>@if ($server->isMetricsEnabled())- @if ($live)- <div wire:key="metrics-poll-{{ $this->pollSeconds() }}" wire:poll.{{ $this->pollSeconds() }}s="loadData" class="hidden"></div>+ @if ($this->pollSeconds())+ <div wire:key="metrics-poll-{{ $this->pollSeconds() }}" wire:poll.{{ $this->pollSeconds() }}s="pollData" class="hidden"></div>@endif<form wire:submit.prevent="saveMetricsSettings" class="contents"><x-unsaved-bar action="saveMetricsSettings"@@ -54,7 +54,7 @@]" /></div><p class="mt-3 text-xs leading-5 text-neutral-500 dark:text-fg-dim">- While live, ranges up to one hour refresh every five seconds. Longer ranges refresh every minute.+ Ranges up to one hour refresh live every five seconds. Longer ranges load once; turn on live refresh to update them every minute.</p></x-application.settings-section></form>diff --git a/resources/views/livewire/two-factor-required.blade.php b/resources/views/livewire/two-factor-required.blade.phpindex 84ce7ffad6..4c246f6a21 100644--- a/resources/views/livewire/two-factor-required.blade.php+++ b/resources/views/livewire/two-factor-required.blade.php@@ -46,6 +46,18 @@<x-forms.button class="w-full justify-center" type="submit" isHighlighted>Configure 2FA</x-forms.button></form>@endif+ @if (!request()->user()->two_factor_confirmed_at && $teamsWithoutTwoFactor->isNotEmpty())+ <div class="flex flex-col gap-2 border-t border-neutral-200 pt-4 dark:border-white/[0.07]">+ <p class="text-sm text-neutral-500 dark:text-fg-dim">Or switch to a team that does not require 2FA:</p>+ @foreach ($teamsWithoutTwoFactor as $teamWithoutTwoFactor)+ <form action="{{ route('two-factor.required.switch-team') }}" method="POST">+ @csrf+ <input type="hidden" name="team_id" value="{{ $teamWithoutTwoFactor->id }}">+ <x-forms.button class="w-full justify-center" type="submit">{{ $teamWithoutTwoFactor->name }}</x-forms.button>+ </form>+ @endforeach+ </div>+ @endif</div>@unless (request()->user()->two_factor_confirmed_at)<x-slot:footer>diff --git a/routes/web.php b/routes/web.phpindex 49648295be..3939286d7a 100644--- a/routes/web.php+++ b/routes/web.php@@ -193,6 +193,19 @@ Route::middleware(['auth', 'verified'])->group(function () {Route::get('/force-password-reset', ForcePasswordReset::class)->name('auth.force-password-reset');});Route::get('/two-factor-required', TwoFactorRequired::class)->name('two-factor.required');+ // A member without 2FA can leave a team that requires it, but only for a team that does not.+ Route::post('/two-factor-required/switch-team', function (Request $request) {+ $team = $request->user()->teams()+ ->whereKey($request->integer('team_id'))+ ->where('is_two_factor_required', false)+ ->first();+ if (! $team) {+ return redirect()->route('two-factor.required');+ }+ refreshSession($team);++ return redirect()->route('dashboard');+ })->name('two-factor.required.switch-team');Route::get('/', Dashboard::class)->name('dashboard');Route::get('/dashboard/canvas', DashboardCanvas::class)->name('dashboard.canvas');diff --git a/scripts/install.sh b/scripts/install.shindex 905421675e..c9801fbc0b 100755--- a/scripts/install.sh+++ b/scripts/install.sh@@ -1028,8 +1028,8 @@ if [ "$IS_COOLIFY_VOLUME_EXISTS" -eq 0 ]; thentest -f /data/coolify/ssh/keys/[email protected] && rm -f /data/coolify/ssh/keys/[email protected]ssh-keygen -t ed25519 -a 100 -f /data/coolify/ssh/keys/[email protected] -q -N "" -C coolifychown 9999 /data/coolify/ssh/keys/[email protected]- # Remove only the key from a previous install (generated with the comment "coolify"), not other keys that contain "coolify"- sed -i -E "/[[:space:]]coolify[[:space:]]*$/d" ~/.ssh/authorized_keys+ # Remove only the key from a previous install (comment "coolify", or "root@coolify" before September 2024), not other keys that contain "coolify"+ sed -i -E "/[[:space:]](root@)?coolify[[:space:]]*$/d" ~/.ssh/authorized_keyscat /data/coolify/ssh/keys/[email protected] >>~/.ssh/authorized_keysrm -f /data/coolify/ssh/keys/[email protected]fidiff --git a/tests/Feature/Api/EnvironmentVariableUpdateApiTest.php b/tests/Feature/Api/EnvironmentVariableUpdateApiTest.phpindex 6bb5b79efb..97d100cc2a 100644--- a/tests/Feature/Api/EnvironmentVariableUpdateApiTest.php+++ b/tests/Feature/Api/EnvironmentVariableUpdateApiTest.php@@ -8,6 +8,7 @@ use App\Models\Project;use App\Models\Server;use App\Models\Service;use App\Models\StandaloneDocker;+use App\Models\StandalonePostgresql;use App\Models\Team;use App\Models\User;use Illuminate\Foundation\Testing\RefreshDatabase;@@ -415,3 +416,69 @@ describe('environment variable key validation for app and service APIs', functio->assertStatus(422);});});++describe('env PATCH keeps the value when value is omitted', function () {+ beforeEach(function () {+ $this->headers = [+ 'Authorization' => 'Bearer '.$this->bearerToken,+ 'Content-Type' => 'application/json',+ ];+ });++ test('keeps or updates the value', function (string $resourceType, string $endpoint, bool $bulk) {+ $resource = match ($resourceType) {+ 'application' => Application::factory()->create([+ 'environment_id' => $this->environment->id,+ 'destination_id' => $this->destination->id,+ 'destination_type' => $this->destination->getMorphClass(),+ ]),+ 'service' => Service::factory()->create([+ 'server_id' => $this->server->id,+ 'destination_id' => $this->destination->id,+ 'destination_type' => $this->destination->getMorphClass(),+ 'environment_id' => $this->environment->id,+ ]),+ 'database' => StandalonePostgresql::create([+ 'name' => 'db',+ 'postgres_user' => 'postgres',+ 'postgres_password' => 'password',+ 'postgres_db' => 'app',+ 'image' => 'postgres:16',+ 'environment_id' => $this->environment->id,+ 'destination_id' => $this->destination->id,+ 'destination_type' => $this->destination->getMorphClass(),+ ]),+ };++ $env = EnvironmentVariable::create([+ 'key' => 'KEEP_ME',+ 'value' => 'secret-value',+ 'resourceable_type' => $resource->getMorphClass(),+ 'resourceable_id' => $resource->id,+ 'is_preview' => false,+ 'is_literal' => false,+ ]);++ $url = "/api/v1/{$endpoint}/{$resource->uuid}/envs".($bulk ? '/bulk' : '');+ $send = fn (array $payload) => $this->withHeaders($this->headers)+ ->patchJson($url, $bulk ? ['data' => [$payload]] : $payload)+ ->assertStatus(201);++ $send(['key' => 'KEEP_ME', 'is_literal' => true]);++ $env->refresh();+ expect($env->value)->toBe('secret-value')+ ->and((bool) $env->is_literal)->toBeTrue();++ $send(['key' => 'KEEP_ME', 'value' => 'new-value']);++ expect($env->fresh()->value)->toBe('new-value');+ })->with([+ 'application single' => ['application', 'applications', false],+ 'application bulk' => ['application', 'applications', true],+ 'service single' => ['service', 'services', false],+ 'service bulk' => ['service', 'services', true],+ 'database single' => ['database', 'databases', false],+ 'database bulk' => ['database', 'databases', true],+ ]);+});diff --git a/tests/Feature/Api/ServerDeployAbilityTest.php b/tests/Feature/Api/ServerDeployAbilityTest.phpindex 041796f5f6..59a3000fe1 100644--- a/tests/Feature/Api/ServerDeployAbilityTest.php+++ b/tests/Feature/Api/ServerDeployAbilityTest.php@@ -210,7 +210,7 @@ describe('destination create', function () {->assertJsonPath('errors.network.0', 'A Docker network with this name already exists on the server. Use a new network name.');expect(StandaloneDocker::query()->count())->toBe($countBefore);- Process::assertRan(fn ($process) => str_contains($process->command, 'docker network ls'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker network ls'));});test('rejects the destination when the server networks cannot be read', function () {diff --git a/tests/Feature/Api/ServerDockerRegistriesApiTest.php b/tests/Feature/Api/ServerDockerRegistriesApiTest.phpindex dde328bccc..b6182a7a6f 100644--- a/tests/Feature/Api/ServerDockerRegistriesApiTest.php+++ b/tests/Feature/Api/ServerDockerRegistriesApiTest.php@@ -75,7 +75,7 @@ function serverRegistriesApiHeaders(string $bearerToken): arrayfunction fakeServerRegistriesSsh(array &$commands, string $config = '', string $error = ''): void{Process::fake(function ($process) use (&$commands, $config, $error) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands[] = $command;if (str_contains($command, '.docker/config.json')) {diff --git a/tests/Feature/Api/UiSettingsApiParityTest.php b/tests/Feature/Api/UiSettingsApiParityTest.phpindex dd3b0649b4..1d087b84c3 100644--- a/tests/Feature/Api/UiSettingsApiParityTest.php+++ b/tests/Feature/Api/UiSettingsApiParityTest.php@@ -271,6 +271,9 @@ describe('servers', function () {})->with(['wildcard domain' => ['wildcard_domain', 'not a url'],'timezone' => ['server_timezone', 'Mars/Olympus'],+ 'null timezone' => ['server_timezone', null],+ 'empty timezone' => ['server_timezone', ''],+ 'array timezone' => ['server_timezone', ['Europe/Berlin']],'compression' => ['backup_compression_cpu_percentage', 30],]);diff --git a/tests/Feature/ApiRateLimitClientIpTest.php b/tests/Feature/ApiRateLimitClientIpTest.phpindex b2272294e3..674d8b9fc0 100644--- a/tests/Feature/ApiRateLimitClientIpTest.php+++ b/tests/Feature/ApiRateLimitClientIpTest.php@@ -15,11 +15,11 @@ beforeEach(function () {InstanceSettings::forceCreate(['id' => 0]);});-function pushSentinelFrom(string $clientIp): TestResponse+function pushSentinelFrom(string $clientIp, string $peer = '172.70.0.1'): TestResponse{return test()->withHeader('CF-Connecting-IP', $clientIp)- ->withServerVariables(['REMOTE_ADDR' => '172.70.0.1'])+ ->withServerVariables(['REMOTE_ADDR' => $peer])->postJson('/api/v1/sentinel/push');}@@ -33,13 +33,13 @@ it('gives each Cloudflare client ip its own sentinel push bucket on cloud', funcpushSentinelFrom('203.0.113.20')->assertUnauthorized();});-it('ignores the Cloudflare client ip for sentinel pushes on self-hosted instances', function () {+it('ignores the Cloudflare client ip for sentinel pushes on self-hosted instances when the peer is not Cloudflare', function () {config()->set('constants.coolify.self_hosted', true);- pushSentinelFrom('203.0.113.10')->assertUnauthorized();- pushSentinelFrom('203.0.113.10')->assertUnauthorized();+ pushSentinelFrom('203.0.113.10', peer: '198.51.100.7')->assertUnauthorized();+ pushSentinelFrom('203.0.113.10', peer: '198.51.100.7')->assertUnauthorized();- pushSentinelFrom('203.0.113.20')->assertStatus(429);+ pushSentinelFrom('203.0.113.20', peer: '198.51.100.7')->assertStatus(429);});it('keys unauthenticated limiters by the Cloudflare client ip on cloud', function (string $limiter) {diff --git a/tests/Feature/ApplicationComposeInjectionValidationTest.php b/tests/Feature/ApplicationComposeInjectionValidationTest.phpindex 86a13682a3..9f3d870ead 100644--- a/tests/Feature/ApplicationComposeInjectionValidationTest.php+++ b/tests/Feature/ApplicationComposeInjectionValidationTest.php@@ -75,7 +75,7 @@ function composeInjectionPayloads(): arrayfunction fakeRepositoryCompose(string $compose): void{Process::fake(function ($process) use ($compose) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);if (str_contains($command, 'git cat-file blob')) {return Process::result(output: $compose);}@@ -310,7 +310,7 @@ test('a deployment stops with a visible log line when the repository Compose fil->not->toContain('#0 ')->not->toContain('.php');expect($this->application->refresh()->docker_compose_raw)->toBe(SAFE_APPLICATION_COMPOSE);- Process::assertNotRan(fn ($process) => str_contains(is_array($process->command) ? implode(' ', $process->command) : $process->command, 'base64 -d'));+ Process::assertNotRan(fn ($process) => str_contains(is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process), 'base64 -d'));})->with(['production' => 0, 'pull request preview' => 42]);test('the deployment log shows the unsafe Compose file explanation once and a short failure line', function () {diff --git a/tests/Feature/ApplicationDeploymentFailedKeepsHealthyContainerTest.php b/tests/Feature/ApplicationDeploymentFailedKeepsHealthyContainerTest.phpindex 8b3fd142f5..0e8d1a5dbc 100644--- a/tests/Feature/ApplicationDeploymentFailedKeepsHealthyContainerTest.php+++ b/tests/Feature/ApplicationDeploymentFailedKeepsHealthyContainerTest.php@@ -67,8 +67,8 @@ function failQueuedCopy(string $queuedCopy): voidfunction removesContainer(string $containerName): Closure{- return fn (PendingProcess $process) => str_contains((string) $process->command, 'docker rm -f')- && str_contains((string) $process->command, $containerName);+ return fn (PendingProcess $process) => str_contains(remoteProcessScript($process), 'docker rm -f')+ && str_contains(remoteProcessScript($process), $containerName);}it('keeps the new container when a deployment fails after the new container became healthy', function () {diff --git a/tests/Feature/ApplicationPreviewApiTest.php b/tests/Feature/ApplicationPreviewApiTest.phpindex f79fd1919f..bd1939ee97 100644--- a/tests/Feature/ApplicationPreviewApiTest.php+++ b/tests/Feature/ApplicationPreviewApiTest.php@@ -142,17 +142,17 @@ describe('GET /api/v1/applications/{uuid}/previews/{pull_request_id}/logs', func$privateKey = PrivateKey::factory()->create(['team_id' => $this->team->id]);$this->server->update(['private_key_id' => $privateKey->id]);Process::fake(function ($process) {- if (str_contains($process->command, 'docker ps -a')) {+ if (str_contains(remoteProcessScript($process), 'docker ps -a')) {return Process::result(output: json_encode(['ID' => 'preview-container','Names' => "{$this->application->uuid}-pr-42",'Labels' => "coolify.applicationId={$this->application->id},coolify.pullRequestId=42",]));}- if (str_contains($process->command, 'docker inspect')) {+ if (str_contains(remoteProcessScript($process), 'docker inspect')) {return Process::result(output: json_encode(['State' => ['Status' => 'running']]));}- if (str_contains($process->command, 'docker logs')) {+ if (str_contains(remoteProcessScript($process), 'docker logs')) {return Process::result(output: 'preview runtime log');}@@ -455,7 +455,7 @@ describe('GET /api/v1/applications/{uuid}/logs service_name', function () {$privateKey = PrivateKey::factory()->create(['team_id' => $this->team->id]);$this->server->update(['private_key_id' => $privateKey->id]);Process::fake(function ($process) {- if (str_contains($process->command, 'docker ps -a')) {+ if (str_contains(remoteProcessScript($process), 'docker ps -a')) {return Process::result(output: implode(PHP_EOL, [json_encode(['ID' => 'web-container',@@ -469,11 +469,11 @@ describe('GET /api/v1/applications/{uuid}/logs service_name', function () {]),]));}- if (str_contains($process->command, 'docker inspect')) {+ if (str_contains(remoteProcessScript($process), 'docker inspect')) {return Process::result(output: json_encode(['State' => ['Status' => 'running']]));}- if (str_contains($process->command, 'docker logs')) {- return Process::result(output: str_contains($process->command, 'worker-container') ? 'worker log' : 'web log');+ if (str_contains(remoteProcessScript($process), 'docker logs')) {+ return Process::result(output: str_contains(remoteProcessScript($process), 'worker-container') ? 'worker log' : 'web log');}return Process::result();diff --git a/tests/Feature/ApplicationPreviewComposeCleanupTest.php b/tests/Feature/ApplicationPreviewComposeCleanupTest.phpindex c926b35745..087a37650f 100644--- a/tests/Feature/ApplicationPreviewComposeCleanupTest.php+++ b/tests/Feature/ApplicationPreviewComposeCleanupTest.php@@ -91,7 +91,7 @@ beforeEach(function () {function fakePreviewCleanupServer(array &$commands, string $previewNetwork, ?array $previewNetworkContainers = [], string $dockerPs = ''): void{Process::fake(function ($process) use (&$commands, $previewNetwork, $previewNetworkContainers, $dockerPs) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands[] = $command;if (str_contains($command, 'docker network inspect') && str_contains($command, "'{$previewNetwork}'")) {diff --git a/tests/Feature/ApplicationPreviewVolumeCleanupTest.php b/tests/Feature/ApplicationPreviewVolumeCleanupTest.phpindex de7731aa08..9aa30744ba 100644--- a/tests/Feature/ApplicationPreviewVolumeCleanupTest.php+++ b/tests/Feature/ApplicationPreviewVolumeCleanupTest.php@@ -67,11 +67,11 @@ it('deletes only named volumes that have the preview suffix enabled', function ($this->preview->forceDelete();- Process::assertRanTimes(fn ($process) => str_contains($process->command, 'docker volume rm'), 1);- Process::assertRan(fn ($process) => str_contains($process->command, "docker volume rm -f 'app-data-pr-42'"));- Process::assertRan(fn ($process) => str_contains($process->command, "sudo docker volume rm -f 'app-data-pr-42'"));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'shared-cache'));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'seed-data'));+ Process::assertRanTimes(fn ($process) => str_contains(remoteProcessScript($process), 'docker volume rm'), 1);+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker volume rm -f 'app-data-pr-42'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "sudo docker volume rm -f 'app-data-pr-42'"));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'shared-cache'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'seed-data'));});it('reports Docker volume removal failures and keeps the preview record', function () {@@ -103,7 +103,7 @@ it('continues removing preview volumes when an earlier volume is already absent''is_preview_suffix_enabled' => true,]);Process::fake(function ($process) {- if (str_contains($process->command, "docker volume rm -f 'already-removed-pr-42'")) {+ if (str_contains(remoteProcessScript($process), "docker volume rm -f 'already-removed-pr-42'")) {return Process::result(errorOutput: 'Error response from daemon: volume already-removed-pr-42 not found',exitCode: 1,@@ -115,8 +115,8 @@ it('continues removing preview volumes when an earlier volume is already absent'$this->preview->forceDelete();- Process::assertRan(fn ($process) => str_contains($process->command, "docker volume rm -f 'already-removed-pr-42'"));- Process::assertRan(fn ($process) => str_contains($process->command, "docker volume rm -f 'app-data-pr-42'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker volume rm -f 'already-removed-pr-42'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker volume rm -f 'app-data-pr-42'"));expect(ApplicationPreview::find($this->preview->id))->toBeNull();});@@ -156,7 +156,7 @@ YAML,$this->preview->forceDelete();- Process::assertRan(fn ($process) => str_contains($process->command, "docker volume rm -f '{$uuid}_db-pr-42'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker volume rm -f '{$uuid}_db-pr-42'"));expect($this->application->persistentStorages()->pluck('name')->sort()->values()->all())->toBe(["{$uuid}_db","{$uuid}_db-pr-43",diff --git a/tests/Feature/BackupAndNotificationRedactionTest.php b/tests/Feature/BackupAndNotificationRedactionTest.phpindex d27cb48ddc..ac94d48ad5 100644--- a/tests/Feature/BackupAndNotificationRedactionTest.php+++ b/tests/Feature/BackupAndNotificationRedactionTest.php@@ -99,7 +99,7 @@ it('redacts the database password from the stored execution and the failure noti$backup = redactionMysqlBackup($this->team, $this->environment, $this->destination);$password = $backup->database->mysql_root_password;Process::fake(function ($process) use ($password) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);if (str_contains($command, 'mysqldump')) {return Process::result(errorOutput: "mysqldump: Got error: 1045: Access denied for user 'root' with password {$password}", exitCode: 1);}diff --git a/tests/Feature/BackupDirectoryPermissionsTest.php b/tests/Feature/BackupDirectoryPermissionsTest.phpindex 956f936e70..2f83cff9ba 100644--- a/tests/Feature/BackupDirectoryPermissionsTest.php+++ b/tests/Feature/BackupDirectoryPermissionsTest.php@@ -52,9 +52,9 @@ function backupDirectoryDumpScript(string $sshUser): string]);$commands = new Collection;Process::fake(function ($process) use ($commands) {- $commands->push($process->command);+ $commands->push(remoteProcessScript($process));- return Process::result(output: str_contains($process->command, 'du -b') ? '100' : '');+ return Process::result(output: str_contains(remoteProcessScript($process), 'du -b') ? '100' : '');});(new DatabaseBackupJob($backup))->handle();diff --git a/tests/Feature/BackupFailureExitCodeTest.php b/tests/Feature/BackupFailureExitCodeTest.phpindex cd5b8d492b..d1e122dc16 100644--- a/tests/Feature/BackupFailureExitCodeTest.php+++ b/tests/Feature/BackupFailureExitCodeTest.php@@ -57,7 +57,7 @@ beforeEach(function () {function interruptedMongodump(int $exitCode): void{Process::fake(function ($process) use ($exitCode) {- if (str_contains($process->command, 'mongodump')) {+ if (str_contains(remoteProcessScript($process), 'mongodump')) {return Process::result(errorOutput: "2026-08-29T15:44:00.555+0000\tdone dumping `evolution.leaverequests` (0 documents)\n2026-08-29T15:44:00.555+0000\twriting `evolution.printagents` to `archive on stdout`",exitCode: $exitCode,@@ -86,7 +86,7 @@ it('tells why a dump stopped without an error message', function (int $exitCode,it('keeps the error of a dump that reports its own failure', function () {Process::fake(function ($process) {- if (str_contains($process->command, 'mongodump')) {+ if (str_contains(remoteProcessScript($process), 'mongodump')) {return Process::result(errorOutput: 'Failed: error connecting to db server: server returned error on SASL authentication step', exitCode: 1);}diff --git a/tests/Feature/BackupRunSafetyTest.php b/tests/Feature/BackupRunSafetyTest.phpindex 7d7464ca24..b058cbadab 100644--- a/tests/Feature/BackupRunSafetyTest.php+++ b/tests/Feature/BackupRunSafetyTest.php@@ -65,7 +65,7 @@ function backupRunSafetyFakeRemoteCommands(array $outputs = []): Collection{$commands = collect();Process::fake(function ($process) use ($commands, $outputs) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands->push($command);foreach ($outputs as $needle => $output) {if (str_contains($command, $needle)) {@@ -403,7 +403,7 @@ it('fails the running backup execution and notifies the team when the worker tim$team = $this->team;$atTimeout = null;Process::fake(function ($process) use ($queuedCopy, $team, &$atTimeout) {- if (str_contains($process->command, 'pg_dump')) {+ if (str_contains(remoteProcessScript($process), 'pg_dump')) {$queuedCopy->failed(new TimeoutExceededException('App\Jobs\DatabaseBackupJob has timed out.'));$execution = ScheduledDatabaseBackupExecution::query()->sole();$atTimeout = [diff --git a/tests/Feature/BackupS3CredentialsArgvTest.php b/tests/Feature/BackupS3CredentialsArgvTest.phpnew file mode 100644index 0000000000..0cdc63c214--- /dev/null+++ b/tests/Feature/BackupS3CredentialsArgvTest.php@@ -0,0 +1,209 @@+<?php++use App\Jobs\DatabaseBackupJob;+use App\Jobs\VolumeBackupJob;+use App\Models\Application;+use App\Models\Environment;+use App\Models\InstanceSettings;+use App\Models\LocalPersistentVolume;+use App\Models\PrivateKey;+use App\Models\Project;+use App\Models\S3Storage;+use App\Models\ScheduledDatabaseBackup;+use App\Models\ScheduledVolumeBackupExecution;+use App\Models\Server;+use App\Models\StandaloneDocker;+use App\Models\Team;+use Illuminate\Filesystem\FilesystemAdapter;+use Illuminate\Foundation\Testing\RefreshDatabase;+use Illuminate\Support\Collection;+use Illuminate\Support\Facades\Process;+use Illuminate\Support\Facades\Storage;+use Symfony\Component\Process\Process as SymfonyProcess;++uses(RefreshDatabase::class);++const S3_ARGV_TEST_KEY = 'AK/+$x"y';+const S3_ARGV_TEST_SECRET = "s3c/r+e\$t 'q\"uo\\te`d`";++beforeEach(function () {+ Server::flushIdentityMap();+ config(['broadcasting.default' => 'null']);+ defined('CURLOPT_RESOLVE') || define('CURLOPT_RESOLVE', 10203);+ InstanceSettings::unguarded(fn () => InstanceSettings::firstOrCreate(['id' => 0]));+});++afterEach(function () {+ Server::flushIdentityMap();+});++/**+ * @return array{0: Team, 1: Server, 2: S3Storage}+ */+function s3ArgvTestSetup(string $sshUser): array+{+ $team = Team::factory()->create();+ $server = Server::factory()->create([+ 'team_id' => $team->id,+ 'user' => $sshUser,+ 'ip' => '203.0.113.10',+ 'private_key_id' => PrivateKey::factory()->create(['team_id' => $team->id])->id,+ ]);+ $server->settings()->update(['is_reachable' => true, 'is_usable' => true]);+ $storage = S3Storage::create([+ 'name' => 'Destination',+ 'region' => 'ru-central-1',+ 'key' => S3_ARGV_TEST_KEY,+ 'secret' => S3_ARGV_TEST_SECRET,+ 'bucket' => 'bucket',+ 'endpoint' => 'https://8.8.8.8',+ 'team_id' => $team->id,+ 'is_usable' => true,+ ]);+ $disk = Mockery::mock(FilesystemAdapter::class);+ $disk->shouldReceive('files')->zeroOrMoreTimes()->andReturn([]);+ $disk->shouldReceive('delete')->zeroOrMoreTimes()->andReturnTrue();+ $sshDisk = Storage::fake('ssh-keys');+ Storage::shouldReceive('disk')->with('ssh-keys')->andReturn($sshDisk);+ Storage::shouldReceive('build')->zeroOrMoreTimes()->andReturn($disk);++ return [$team, $server, $storage];+}++/**+ * Records the SSH processes that a backup job runs, with their stdin.+ *+ * @return Collection<int, array{command: string, input: ?string}>+ */+function s3ArgvTestFakeSsh(): Collection+{+ $processes = collect();+ Process::fake(function ($process) use ($processes) {+ $processes->push(['command' => $process->command, 'input' => $process->input]);+ $output = match (true) {+ str_contains(remoteProcessScript($process), 'du -b') => '128',+ str_contains(remoteProcessScript($process), 'mc pipe') => '128',+ default => '',+ };++ return Process::result(output: $output);+ });++ return $processes;+}++/**+ * Runs the recorded `mc alias set` SSH process in a real shell with its stdin. Stubs for `timeout`, `ssh`+ * (runs the remote command in `sh -c`, like a login shell), `sudo`, `docker` and `mc` log their arguments.+ *+ * @param array{command: string, input: ?string} $aliasProcess+ * @return array{argv: string, mcStdin: string}+ */+function s3ArgvTestRunAliasProcess(array $aliasProcess): array+{+ $bin = sys_get_temp_dir().'/s3-argv-'.bin2hex(random_bytes(6));+ mkdir($bin);+ $log = "{$bin}/argv.log";+ $stubs = [+ 'timeout' => "shift\nexec \"\$@\"",+ 'ssh' => "for last; do :; done\nexec sh -c \"\$last\"",+ 'sudo' => 'exec "$@"',+ 'docker' => "[ \"\$1\" = exec ] || exit 0\nshift\nwhile [ \"\${1#-}\" != \"\$1\" ]; do shift; done\nshift\nexec \"\$@\"",+ 'mc' => "cat > {$bin}/mc.stdin",+ ];+ foreach ($stubs as $name => $body) {+ file_put_contents("{$bin}/{$name}", "#!/bin/sh\nprintf '%s\\n' \"{$name} \$*\" >> {$log}\n{$body}\n");+ chmod("{$bin}/{$name}", 0755);+ }++ try {+ (new SymfonyProcess(['bash', '-c', $aliasProcess['command']], env: ['PATH' => "{$bin}:/usr/bin:/bin"], input: $aliasProcess['input']))->mustRun();++ return ['argv' => (string) file_get_contents($log), 'mcStdin' => (string) file_get_contents("{$bin}/mc.stdin")];+ } finally {+ array_map('unlink', glob("{$bin}/*"));+ rmdir($bin);+ }+}++function s3ArgvTestAssertSecretOnlyOnStdin(Collection $processes, bool $nonRoot): void+{+ foreach ($processes as $process) {+ expect($process['command'])->not->toContain(S3_ARGV_TEST_SECRET)+ ->and($process['command'])->not->toContain(escapeshellarg(S3_ARGV_TEST_SECRET))+ ->and($process['command'])->not->toContain(escapeshellarg(S3_ARGV_TEST_KEY))+ ->and($process['command'])->not->toContain(base64_encode(S3_ARGV_TEST_SECRET));+ }++ $aliasProcess = $processes->sole(fn (array $process) => str_contains($process['command'], 'mc alias set'));+ $run = s3ArgvTestRunAliasProcess($aliasProcess);++ expect($run['mcStdin'])->toBe(S3_ARGV_TEST_KEY."\n".S3_ARGV_TEST_SECRET."\n")+ ->and($run['argv'])->toContain('mc alias set')+ ->and($run['argv'])->toContain('temporary https://8.8.8.8')+ ->and($run['argv'])->not->toContain(S3_ARGV_TEST_SECRET)+ ->and($run['argv'])->not->toContain(S3_ARGV_TEST_KEY);+ if ($nonRoot) {+ expect($run['argv'])->toContain('sudo docker exec -i');+ }+ expect($processes->contains(fn (array $process) => str_contains($process['command']."\n".$process['input'], "-e 'MC_REGION=ru-central-1'")))->toBeTrue();+}++it('passes the S3 keys of a database backup upload to mc on stdin, not in a command line', function (string $sshUser) {+ [$team, $server, $storage] = s3ArgvTestSetup($sshUser);+ $destination = StandaloneDocker::where('server_id', $server->id)->firstOrFail();+ $environment = Environment::factory()->create(['project_id' => Project::factory()->create(['team_id' => $team->id])->id]);+ $database = create_standalone_postgresql($environment->id, $destination, ['status' => 'running:healthy']);+ $backup = new ScheduledDatabaseBackup;+ $backup->forceFill([+ 'enabled' => true,+ 'save_s3' => true,+ 'disable_local_backup' => true,+ 's3_storage_id' => $storage->id,+ 'frequency' => 'daily',+ 'databases_to_backup' => 'app',+ 'database_id' => $database->id,+ 'database_type' => $database->getMorphClass(),+ 'team_id' => $team->id,+ ])->save();+ $backup->syncS3Storages([$storage->id]);+ $processes = s3ArgvTestFakeSsh();++ (new DatabaseBackupJob($backup->fresh()))->handle();++ expect($backup->executions()->sole()->s3_uploaded)->toBeTrue();+ s3ArgvTestAssertSecretOnlyOnStdin($processes, $sshUser !== 'root');+})->with(['root', 'deploy']);++it('passes the S3 keys of a volume backup to mc on stdin, not in a command line', function (string $sshUser, bool $streamToS3) {+ [$team, $server, $storage] = s3ArgvTestSetup($sshUser);+ $destination = StandaloneDocker::where('server_id', $server->id)->firstOrFail();+ $environment = Environment::factory()->create(['project_id' => Project::factory()->create(['team_id' => $team->id])->id]);+ $application = Application::factory()->create([+ 'environment_id' => $environment->id,+ 'destination_id' => $destination->id,+ 'destination_type' => $destination->getMorphClass(),+ ]);+ $volume = LocalPersistentVolume::create([+ 'name' => 'app-data',+ 'mount_path' => '/data',+ 'resource_id' => $application->id,+ 'resource_type' => $application->getMorphClass(),+ ]);+ $backup = $volume->scheduledBackups()->create([+ 'team_id' => $team->id,+ 'frequency' => 'daily',+ 'save_s3' => true,+ 'disable_local_backup' => $streamToS3,+ 's3_storage_id' => $storage->id,+ ]);+ $processes = s3ArgvTestFakeSsh();++ (new VolumeBackupJob($backup))->handle();++ $execution = ScheduledVolumeBackupExecution::query()->sole();+ expect($execution->status)->toBe('success')+ ->and($execution->s3_uploaded)->toBeTrue()+ ->and($processes->contains(fn (array $process) => str_contains($process['command']."\n".$process['input'], 'mc pipe')))->toBe($streamToS3);+ s3ArgvTestAssertSecretOnlyOnStdin($processes, $sshUser !== 'root');+})->with(['root', 'deploy'])->with(['local archive upload' => false, 'S3-only stream' => true]);diff --git a/tests/Feature/BackupS3DestinationRedactionTest.php b/tests/Feature/BackupS3DestinationRedactionTest.phpindex 709bc05023..92deab9ded 100644--- a/tests/Feature/BackupS3DestinationRedactionTest.php+++ b/tests/Feature/BackupS3DestinationRedactionTest.php@@ -91,7 +91,7 @@ it('redacts the credentials of every S3 destination from a database backup uploa]);$backup->syncS3Storages($this->storages->pluck('id')->all());Process::fake(function ($process) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: str_contains($command, 'du -b') ? '128' : '');});@@ -142,7 +142,7 @@ it('redacts the credentials of every S3 destination from a volume backup upload$disk->shouldReceive('delete')->zeroOrMoreTimes()->andReturnTrue();Storage::shouldReceive('disk')->with('ssh-keys')->andReturn($sshDisk);Storage::shouldReceive('build')->zeroOrMoreTimes()->andReturn($disk);- Process::fake([+ fakeRemoteProcesses(['*du -b*' => '128','*mc cp*' => Process::result(errorOutput: $this->uploadError, exitCode: 1),'*' => '',diff --git a/tests/Feature/BackupS3DownloadTest.php b/tests/Feature/BackupS3DownloadTest.phpindex a04d90950e..a43bc506ed 100644--- a/tests/Feature/BackupS3DownloadTest.php+++ b/tests/Feature/BackupS3DownloadTest.php@@ -271,10 +271,10 @@ it('uploads the backup file from the server host with curl', function () {(new PrepareFileDownloadJob('dl-host', $this->user->id, $this->server->id, null, BACKUP_S3_DOWNLOAD_FILE, false, 'pg.dmp', $this->downloadStorage->id))->handle($downloads);expect(Cache::get(PrepareFileDownloadJob::cacheKey('dl-host')))->toMatchArray(['status' => 'ready', 'url' => 'https://s3.example.com/get']);- Process::assertRan(fn ($process) => str_contains($process->command, 'curl -fsS')- && str_contains($process->command, BACKUP_S3_DOWNLOAD_FILE)- && str_contains($process->command, 'X-Amz-Signature=abc&x=1')- && ! str_contains($process->command, 'docker exec'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'curl -fsS')+ && str_contains(remoteProcessScript($process), BACKUP_S3_DOWNLOAD_FILE)+ && str_contains(remoteProcessScript($process), 'X-Amz-Signature=abc&x=1')+ && ! str_contains(remoteProcessScript($process), 'docker exec'));});it('builds host file commands that the non-root sudo parser keeps whole', function () {diff --git a/tests/Feature/BackupTimeoutSecretRedactionTest.php b/tests/Feature/BackupTimeoutSecretRedactionTest.phpindex 9cb4ec23f8..29bf6fdefc 100644--- a/tests/Feature/BackupTimeoutSecretRedactionTest.php+++ b/tests/Feature/BackupTimeoutSecretRedactionTest.php@@ -57,7 +57,7 @@ it('does not put the command into the error when a remote command times out', fuProcess::fake(function ($process) use (&$attempts) {$attempts++;- throw backupRedactionTimeoutException($process->command, 7200);+ throw backupRedactionTimeoutException(remoteProcessScript($process), 7200);});try {@@ -94,8 +94,8 @@ it('does not put database credentials into the backup execution or the failure n'timeout' => 600,]);Process::fake(function ($process) {- if (str_contains($process->command, 'mysqldump')) {- throw backupRedactionTimeoutException($process->command, 600);+ if (str_contains(remoteProcessScript($process), 'mysqldump')) {+ throw backupRedactionTimeoutException(remoteProcessScript($process), 600);}return Process::result(output: '');diff --git a/tests/Feature/ClientIpSpoofingTest.php b/tests/Feature/ClientIpSpoofingTest.phpnew file mode 100644index 0000000000..93c2a0bf3a--- /dev/null+++ b/tests/Feature/ClientIpSpoofingTest.php@@ -0,0 +1,106 @@+<?php++use App\Models\InstanceSettings;+use App\Models\Team;+use App\Models\User;+use Illuminate\Foundation\Testing\RefreshDatabase;+use Illuminate\Testing\TestResponse;++uses(RefreshDatabase::class);++beforeEach(function () {+ config()->set('app.maintenance.store', 'array');+ config()->set('constants.coolify.self_hosted', true);++ InstanceSettings::forceCreate(['id' => 0, 'is_api_enabled' => true, 'allowed_ips' => '203.0.113.50,2001:db8::50']);++ $team = Team::factory()->create();+ $user = User::factory()->create();+ $team->members()->attach($user->id, ['role' => 'owner']);+ $this->apiToken = $user->createToken('client-ip', ['read'], teamId: $team->id)->plainTextToken;+});++/**+ * An API call with a valid token: 403 when the allowed IPs block the client, 200 when they let it through.+ *+ * @param array<string, string> $headers+ */+function apiIpCheckFromPeer(string $peer, array $headers = []): TestResponse+{+ return test()+ ->withToken(test()->apiToken)+ ->withHeaders($headers)+ ->withServerVariables(['REMOTE_ADDR' => $peer])+ ->getJson('/api/v1/version');+}++it('ignores forwarded headers from a public peer for the allowed IPs', function (array $headers) {+ apiIpCheckFromPeer('198.51.100.7', $headers)->assertForbidden();+})->with([+ 'X-Forwarded-For' => [['X-Forwarded-For' => '203.0.113.50']],+ 'fake Cloudflare hop' => [['X-Forwarded-For' => '104.16.0.1', 'CF-Connecting-IP' => '203.0.113.50']],+]);++it('ignores forwarded headers from a public IPv6 peer for the allowed IPs', function (string $peer) {+ apiIpCheckFromPeer($peer, ['X-Forwarded-For' => '2001:db8::50'])->assertForbidden();+})->with(['2001:db8:ffff::7', '::ffff:198.51.100.7']);++it('uses the forwarded client from a local reverse proxy for the allowed IPs', function (string $peer, string $client) {+ apiIpCheckFromPeer($peer, ['X-Forwarded-For' => $client])->assertOk();+ apiIpCheckFromPeer($peer, ['X-Forwarded-For' => '198.51.100.7'])->assertForbidden();+})->with([+ 'Docker network' => ['10.0.1.5', '203.0.113.50'],+ 'Docker bridge' => ['172.18.0.2', '203.0.113.50'],+ 'loopback' => ['127.0.0.1', '203.0.113.50'],+ 'IPv6 loopback' => ['::1', '2001:db8::50'],+ 'IPv6 unique local' => ['fd00::2', '2001:db8::50'],+ 'IPv6 link local' => ['fe80::2', '2001:db8::50'],+ 'IPv4-mapped private' => ['::ffff:10.0.1.5', '203.0.113.50'],+]);++it('uses the Cloudflare client IP behind a local reverse proxy only when the proxy received the request from Cloudflare', function () {+ apiIpCheckFromPeer('10.0.1.5', [+ 'X-Forwarded-For' => '203.0.113.50, 162.158.79.140',+ 'CF-Connecting-IP' => '203.0.113.50',+ ])->assertOk();++ apiIpCheckFromPeer('10.0.1.5', [+ 'X-Forwarded-For' => '104.16.0.1, 198.51.100.7',+ 'CF-Connecting-IP' => '203.0.113.50',+ ])->assertForbidden();+});++it('uses the Cloudflare client IP when Cloudflare connects directly', function () {+ apiIpCheckFromPeer('162.158.79.140', ['CF-Connecting-IP' => '203.0.113.50'])->assertOk();+});++it('keys the API rate limit by the public peer, not by forwarded headers', function () {+ config()->set('api.rate_limit', 2);+ InstanceSettings::query()->update(['allowed_ips' => null]);++ $push = fn (string $peer, array $headers) => test()+ ->withHeaders($headers)+ ->withServerVariables(['REMOTE_ADDR' => $peer])+ ->postJson('/api/v1/sentinel/push');++ $push('198.51.100.7', ['X-Forwarded-For' => '203.0.113.1'])->assertUnauthorized();+ $push('198.51.100.7', ['X-Forwarded-For' => '203.0.113.2', 'CF-Connecting-IP' => '203.0.113.2'])->assertUnauthorized();+ $push('198.51.100.7', ['X-Forwarded-For' => '104.16.0.1', 'CF-Connecting-IP' => '203.0.113.3'])->assertStatus(429);++ $push('10.0.1.5', ['X-Forwarded-For' => '203.0.113.10'])->assertUnauthorized();+ $push('10.0.1.5', ['X-Forwarded-For' => '203.0.113.20'])->assertUnauthorized();+});++it('ignores X-Forwarded-For from a public peer on cloud', function () {+ config()->set('constants.coolify.self_hosted', false);+ config()->set('api.rate_limit', 2);++ $push = fn (string $forwardedFor) => test()+ ->withHeader('X-Forwarded-For', $forwardedFor)+ ->withServerVariables(['REMOTE_ADDR' => '198.51.100.7'])+ ->postJson('/api/v1/sentinel/push');++ $push('203.0.113.1')->assertUnauthorized();+ $push('203.0.113.2')->assertUnauthorized();+ $push('203.0.113.3')->assertStatus(429);+});diff --git a/tests/Feature/CloneArchiveDirectoryTest.php b/tests/Feature/CloneArchiveDirectoryTest.phpindex 92b1b2df3c..2390959188 100644--- a/tests/Feature/CloneArchiveDirectoryTest.php+++ b/tests/Feature/CloneArchiveDirectoryTest.php@@ -47,17 +47,17 @@ beforeEach(function () {function fakeCloneArchiveServers(object $test, bool $failScp = false): void{Process::fake(function ($process) use ($test, $failScp) {- if (str_contains($process->command, 'mktemp -d')) {+ if (str_contains(remoteProcessScript($process), 'mktemp -d')) {$directory = '/var/tmp/coolify-clone.'.Str::random(10);$test->mktempDirectories[] = $directory;return Process::result(output: $directory);}- if ($failScp && str_contains($process->command, ' scp ')) {+ if ($failScp && str_contains(remoteProcessScript($process), ' scp ')) {return Process::result(output: '', errorOutput: 'scp failed', exitCode: 1);}- return Process::result(output: str_contains($process->command, 'docker volume inspect') ? '[]' : '');+ return Process::result(output: str_contains(remoteProcessScript($process), 'docker volume inspect') ? '[]' : '');});}@@ -68,7 +68,7 @@ function cloneArchiveCommands(): array{$commands = [];Process::assertRan(function ($process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return true;});diff --git a/tests/Feature/CloneServiceVolumeDataTest.php b/tests/Feature/CloneServiceVolumeDataTest.phpindex 68db8a10ba..c1343134b3 100644--- a/tests/Feature/CloneServiceVolumeDataTest.php+++ b/tests/Feature/CloneServiceVolumeDataTest.php@@ -76,11 +76,10 @@ beforeEach(function () {*/function chainedServiceVolumeClones(): array{- $first = Bus::dispatched(CallQueuedClosure::class)->first();+ $first = Bus::dispatched(CallQueuedClosure::class)->first() ?? Bus::dispatched(VolumeCloneJob::class)->first();expect($first)->not->toBeNull('No volume clone chain was dispatched.');- return collect($first->chained)- ->map(fn (string $job) => unserialize($job))+ return collect([$first, ...array_map(fn (string $job) => unserialize($job), $first->chained)])->filter(fn ($job) => $job instanceof VolumeCloneJob)->map(fn (VolumeCloneJob $job) => (fn () => [$this->sourceVolume, $this->targetVolume])->call($job))->values()diff --git a/tests/Feature/CloneVolumeDataSourceSafetyTest.php b/tests/Feature/CloneVolumeDataSourceSafetyTest.phpnew file mode 100644index 0000000000..9f887e330c--- /dev/null+++ b/tests/Feature/CloneVolumeDataSourceSafetyTest.php@@ -0,0 +1,612 @@+<?php++use App\Actions\Application\StopApplication;+use App\Actions\Database\StartDatabase;+use App\Actions\Database\StopDatabase;+use App\Actions\Service\StartService;+use App\Actions\Service\StopService;+use App\Jobs\VolumeCloneJob;+use App\Livewire\Project\CloneMe;+use App\Livewire\Project\Shared\ResourceOperations;+use App\Models\Application;+use App\Models\ApplicationDeploymentQueue;+use App\Models\Environment;+use App\Models\InstanceSettings;+use App\Models\LocalPersistentVolume;+use App\Models\PrivateKey;+use App\Models\Project;+use App\Models\Server;+use App\Models\Service;+use App\Models\StandaloneDocker;+use App\Models\StandalonePostgresql;+use App\Models\Team;+use App\Models\User;+use App\Notifications\Internal\GeneralNotification;+use Illuminate\Database\Schema\Blueprint;+use Illuminate\Foundation\Testing\RefreshDatabase;+use Illuminate\Process\PendingProcess;+use Illuminate\Queue\CallQueuedClosure;+use Illuminate\Support\Facades\Artisan;+use Illuminate\Support\Facades\Bus;+use Illuminate\Support\Facades\DB;+use Illuminate\Support\Facades\Gate;+use Illuminate\Support\Facades\Notification;+use Illuminate\Support\Facades\Process;+use Illuminate\Support\Facades\Schema;+use Illuminate\Support\Str;+use Laravel\Horizon\Tags;+use Livewire\Livewire;++uses(RefreshDatabase::class);++const CLONE_SAFETY_SERVICE_COMPOSE = <<<'YAML'+services:+ web:+ image: nginx:alpine+ volumes:+ - 'web-data:/usr/share/nginx/html'+YAML;++beforeEach(function () {+ $this->withoutVite();+ Server::flushIdentityMap();+ InstanceSettings::unguarded(fn () => InstanceSettings::firstOrCreate(['id' => 0]));+ config(['constants.ssh.mux_enabled' => false]);+ Notification::fake();+ Process::fake(['*' => Process::result(output: '')]);++ $this->user = User::factory()->create();+ $this->team = Team::factory()->create();+ $this->user->teams()->attach($this->team, ['role' => 'owner']);++ $privateKey = PrivateKey::factory()->create(['team_id' => $this->team->id]);+ $server = Server::factory()->create(['team_id' => $this->team->id, 'private_key_id' => $privateKey->id]);+ $server->settings()->update(['is_reachable' => true, 'is_usable' => true]);+ $this->destination = StandaloneDocker::where('server_id', $server->id)->firstOrFail();+ $this->project = Project::factory()->create(['team_id' => $this->team->id]);+ $this->environment = $this->project->environments()->first()+ ?? Environment::factory()->create(['project_id' => $this->project->id]);++ $this->service = Service::factory()->create([+ 'docker_compose_raw' => CLONE_SAFETY_SERVICE_COMPOSE,+ 'environment_id' => $this->environment->id,+ 'server_id' => $server->id,+ 'destination_id' => $this->destination->id,+ 'destination_type' => $this->destination->getMorphClass(),+ ]);+ $this->service->parse();++ $this->actingAs($this->user);+ session(['currentTeam' => $this->team]);+});++/**+ * Runs the queued jobs with the database driver, so every chain step and the catch callback+ * are serialized and restored like on a Horizon worker.+ */+function cloneSafetyUseDatabaseQueue(): void+{+ Schema::create('jobs', function (Blueprint $table) {+ $table->id();+ $table->string('queue')->index();+ $table->longText('payload');+ $table->unsignedTinyInteger('attempts');+ $table->unsignedInteger('reserved_at')->nullable();+ $table->unsignedInteger('available_at');+ $table->unsignedInteger('created_at');+ });+ config(['queue.default' => 'database']);+}++/**+ * The worker runs in the test process, so a high `--memory` keeps the default 128 MB limit from+ * stopping it after the first chain step when earlier tests have grown the process.+ */+function cloneSafetyWorkQueue(): void+{+ Artisan::call('queue:work', ['connection' => 'database', '--stop-when-empty' => true, '--queue' => 'high', '--sleep' => 0, '--memory' => 4096]);+}++/** The volume copy (`cp -a` in a helper container) fails like on a full disk; every other remote command works. */+function cloneSafetyFailVolumeCopy(): void+{+ Process::fake(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), 'cp -a')+ ? Process::result(errorOutput: 'No space left on device', exitCode: 1)+ : Process::result(output: ''));+}++/** `docker ps` (running containers only) lists the given container names; every other remote command works. */+function cloneSafetyListRunningContainers(string ...$names): void+{+ Process::fake(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), 'docker ps --filter')+ ? Process::result(output: implode("\n", $names))+ : Process::result(output: ''));+}++function cloneSafetyMarkServiceRunning(Service $service): void+{+ $service->applications()->update(['status' => 'running:healthy']);+ $service->databases()->update(['status' => 'running:healthy']);+}++test('a failed volume copy starts the running source service again and notifies the team', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ cloneSafetyFailVolumeCopy();+ cloneSafetyMarkServiceRunning($this->service);+ StopService::shouldRun()->once()->withArgs(fn (Service $service) => $service->is($this->service));+ StartService::shouldRun()->once()->withArgs(fn (Service $service) => $service->is($this->service));++ Livewire::test(ResourceOperations::class, ['resource' => $this->service])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ cloneSafetyWorkQueue();++ expect(DB::table('failed_jobs')->count())->toBe(1);+ Notification::assertSentTo($this->team, GeneralNotification::class, function (GeneralNotification $notification) {+ return $notification->success === false+ && str_contains($notification->message, $this->service->name)+ && str_contains($notification->message, 'No space left on device');+ });+});++test('a volume copy of a stopped source service neither stops nor starts it', function () {+ cloneSafetyUseDatabaseQueue();+ StopService::shouldRun()->never();+ StartService::shouldRun()->never();++ Livewire::test(ResourceOperations::class, ['resource' => $this->service])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ cloneSafetyWorkQueue();++ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ expect(DB::table('failed_jobs')->count())->toBe(0);+ Notification::assertNothingSent();+});++test('a successful volume copy stops the running source service and starts it again', function () {+ cloneSafetyUseDatabaseQueue();+ cloneSafetyMarkServiceRunning($this->service);+ StopService::shouldRun()->once();+ StartService::shouldRun()->once();++ Livewire::test(ResourceOperations::class, ['resource' => $this->service])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ cloneSafetyWorkQueue();++ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ expect(DB::table('failed_jobs')->count())->toBe(0);+ Notification::assertNothingSent();+});++test('a failed stop that leaves some containers of the source service running starts only the stopped containers again', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ cloneSafetyMarkServiceRunning($this->service);+ cloneSafetyListRunningContainers('web-'.$this->service->uuid);+ StopService::shouldRun()->once()->andReturn('docker stop timed out');+ StartService::shouldRun()->never();++ Livewire::test(ResourceOperations::class, ['resource' => $this->service])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ cloneSafetyWorkQueue();++ $workdir = $this->service->workdir();+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker compose --project-directory {$workdir} -f {$workdir}/docker-compose.yml --project-name {$this->service->uuid} up -d --no-recreate < /dev/null"));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), '--force-recreate'));+ Notification::assertSentTo($this->team, GeneralNotification::class, fn (GeneralNotification $notification) => $notification->success === false+ && str_contains($notification->message, 'did not copy')+ && str_contains($notification->message, 'started its stopped containers again')+ && str_contains($notification->message, 'docker stop timed out'));+});++test('a failed start of the stopped containers of a partly stopped source service is in the notification', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ cloneSafetyMarkServiceRunning($this->service);+ Process::fake(fn (PendingProcess $process) => match (true) {+ str_contains(remoteProcessScript($process), 'docker ps --filter') => Process::result(output: 'web-'.$this->service->uuid),+ str_contains(remoteProcessScript($process), 'up -d --no-recreate') => Process::result(errorOutput: 'Cannot connect to the Docker daemon', exitCode: 1),+ default => Process::result(output: ''),+ });+ StopService::shouldRun()->once()->andReturn(null);+ StartService::shouldRun()->never();++ Livewire::test(ResourceOperations::class, ['resource' => $this->service])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ cloneSafetyWorkQueue();++ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ Notification::assertSentTo($this->team, GeneralNotification::class, fn (GeneralNotification $notification) => str_contains($notification->message, 'did not copy')+ && str_contains($notification->message, 'could not start its stopped containers again'));+});++test('a stop that leaves containers of the source service running copies no data', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ cloneSafetyMarkServiceRunning($this->service);+ cloneSafetyListRunningContainers('web-'.$this->service->uuid);+ StopService::shouldRun()->once()->andReturn(null);+ StartService::shouldRun()->never();++ Livewire::test(ResourceOperations::class, ['resource' => $this->service])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ cloneSafetyWorkQueue();++ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ Notification::assertSentTo($this->team, GeneralNotification::class, fn (GeneralNotification $notification) => str_contains($notification->message, 'did not copy')+ && str_contains($notification->message, 'web-'.$this->service->uuid));+});++test('a failed stop that left the source service down starts it again', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ cloneSafetyMarkServiceRunning($this->service);+ StopService::shouldRun()->once()->andReturn('docker stop timed out');+ StartService::shouldRun()->once()->withArgs(fn (Service $service) => $service->is($this->service));++ Livewire::test(ResourceOperations::class, ['resource' => $this->service])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ cloneSafetyWorkQueue();++ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ Notification::assertSentTo($this->team, GeneralNotification::class, fn (GeneralNotification $notification) => str_contains($notification->message, 'did not copy')+ && str_contains($notification->message, 'started')+ && str_contains($notification->message, 'docker stop timed out'));+});++test('cloning volume data needs the permission to stop the source resource', function () {+ Bus::fake();+ Gate::before(fn ($user, string $ability) => $ability === 'stop' ? false : null);++ Livewire::test(ResourceOperations::class, ['resource' => $this->service])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid);++ expect(Service::query()->count())->toBe(1);+ Bus::assertNotDispatched(CallQueuedClosure::class);++ Livewire::test(CloneMe::class, [+ 'project_uuid' => $this->project->uuid,+ 'environment_uuid' => $this->environment->uuid,+ ])+ ->set('selectedDestination', $this->destination->uuid)+ ->set('newName', 'cloned-env')+ ->set('cloneVolumeData', true)+ ->call('clone', 'environment');++ expect(Service::query()->count())->toBe(1)+ ->and($this->project->environments()->where('name', 'cloned-env')->exists())->toBeFalse();+ Bus::assertNotDispatched(CallQueuedClosure::class);+});++test('cloning without volume data needs no stop permission and stops nothing', function () {+ Bus::fake();+ Gate::before(fn ($user, string $ability) => $ability === 'stop' ? false : null);++ Livewire::test(ResourceOperations::class, ['resource' => $this->service])+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ expect(Service::query()->count())->toBe(2);+ Bus::assertNotDispatched(CallQueuedClosure::class);+ Bus::assertNotDispatched(VolumeCloneJob::class);+});++test('cloning a running database with volume data copies its volumes in one chain between stop and start', function (string $component) {+ Bus::fake();+ $database = StandalonePostgresql::create([+ 'name' => 'orders-db', 'postgres_user' => 'postgres', 'postgres_password' => 'password', 'postgres_db' => 'db',+ 'image' => 'postgres:17', 'environment_id' => $this->environment->id,+ 'destination_id' => $this->destination->id, 'destination_type' => $this->destination->getMorphClass(),+ ]);+ $database->forceFill(['status' => 'running:healthy'])->saveQuietly();+ LocalPersistentVolume::query()->where('resource_type', $database->getMorphClass())->where('resource_id', $database->id)->delete();+ foreach (['postgres-data-'.$database->uuid, $database->uuid.'-extra'] as $name) {+ LocalPersistentVolume::create([+ 'name' => $name,+ 'mount_path' => '/data/'.$name,+ 'resource_id' => $database->id,+ 'resource_type' => $database->getMorphClass(),+ ]);+ }+ $this->service->delete();++ if ($component === ResourceOperations::class) {+ Livewire::test(ResourceOperations::class, ['resource' => $database])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();+ } else {+ Livewire::test(CloneMe::class, [+ 'project_uuid' => $this->project->uuid,+ 'environment_uuid' => $this->environment->uuid,+ ])+ ->set('selectedDestination', $this->destination->uuid)+ ->set('newName', 'cloned-env')+ ->set('cloneVolumeData', true)+ ->call('clone', 'environment')+ ->assertHasNoErrors();+ }++ Bus::assertNotDispatched(VolumeCloneJob::class);+ Bus::assertChained([+ CallQueuedClosure::class,+ VolumeCloneJob::class,+ VolumeCloneJob::class,+ CallQueuedClosure::class,+ ]);+})->with([+ 'resource operations' => [ResourceOperations::class],+ 'environment clone' => [CloneMe::class],+]);++test('a restored same-server volume clone job has explicit Horizon tags', function () {+ $server = $this->destination->server;+ $volume = LocalPersistentVolume::create(['name' => 'clone-target', 'mount_path' => '/data', 'resource_id' => 1, 'resource_type' => Service::class]);++ // Without explicit tags, Horizon compares the job's models with `==`. Two restored copies of one+ // server fail that comparison on PHP 8.4 ("Nesting level too deep"), so the chain stops before the copy.+ $restoredJob = unserialize(serialize(new VolumeCloneJob('clone-source', 'clone-target', $server, $server, $volume)));++ expect(Tags::extractExplicitTags($restoredJob))->toBe([Server::class.':'.$server->id, LocalPersistentVolume::class.':'.$volume->id]);+});++function cloneSafetyCreateDatabase(object $test, string $status): StandalonePostgresql+{+ $database = StandalonePostgresql::create([+ 'name' => 'api-db', 'postgres_user' => 'postgres', 'postgres_password' => 'password', 'postgres_db' => 'db',+ 'image' => 'postgres:17', 'environment_id' => $test->environment->id,+ 'destination_id' => $test->destination->id, 'destination_type' => $test->destination->getMorphClass(),+ ]);+ $database->forceFill(['status' => $status])->saveQuietly();++ return $database;+}++function cloneSafetyApiHeaders(User $user, int $teamId): array+{+ $plainTextToken = Str::random(40);+ $token = $user->tokens()->create([+ 'name' => 'clone-safety',+ 'token' => hash('sha256', $plainTextToken),+ 'abilities' => ['*'],+ 'team_id' => $teamId,+ ]);+ auth()->forgetGuards();++ return ['Authorization' => 'Bearer '.$token->getKey().'|'.$plainTextToken];+}++function cloneSafetyCreateApplication(object $test, string $status): Application+{+ $application = Application::factory()->create([+ 'environment_id' => $test->environment->id,+ 'destination_id' => $test->destination->id,+ 'destination_type' => $test->destination->getMorphClass(),+ ]);+ $application->forceFill(['status' => $status])->saveQuietly();+ LocalPersistentVolume::create([+ 'name' => $application->uuid.'-data',+ 'mount_path' => '/data',+ 'resource_id' => $application->id,+ 'resource_type' => $application->getMorphClass(),+ ]);++ return $application;+}++test('a failed volume copy of a running database cloned through the API starts it again and notifies the team', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ cloneSafetyFailVolumeCopy();+ $database = cloneSafetyCreateDatabase($this, 'running:healthy');+ StopDatabase::shouldRun()->once()->withArgs(fn ($source) => $source->is($database));+ StartDatabase::shouldRun()->once()->withArgs(fn ($source) => $source->is($database));++ $this->withHeaders(cloneSafetyApiHeaders($this->user, $this->team->id))+ ->postJson("/api/v1/databases/{$database->uuid}/clone", [+ 'destination_uuid' => $this->destination->uuid,+ 'clone_volumes' => true,+ ])+ ->assertCreated();++ cloneSafetyWorkQueue();++ Notification::assertSentTo($this->team, GeneralNotification::class, fn (GeneralNotification $notification) => $notification->success === false+ && str_contains($notification->message, 'api-db')+ && str_contains($notification->message, 'No space left on device'));+});++test('a failed stop of a running database cloned through the API copies no data and leaves it running', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ $database = cloneSafetyCreateDatabase($this, 'running:healthy');+ cloneSafetyListRunningContainers($database->uuid);+ StopDatabase::shouldRun()->once()->andReturn('Database stop failed: Connection refused');+ StartDatabase::shouldRun()->never();++ $this->withHeaders(cloneSafetyApiHeaders($this->user, $this->team->id))+ ->postJson("/api/v1/databases/{$database->uuid}/clone", [+ 'destination_uuid' => $this->destination->uuid,+ 'clone_volumes' => true,+ ])+ ->assertCreated();++ cloneSafetyWorkQueue();++ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ Notification::assertSentTo($this->team, GeneralNotification::class, fn (GeneralNotification $notification) => str_contains($notification->message, 'did not copy')+ && str_contains($notification->message, 'api-db')+ && str_contains($notification->message, 'Connection refused'));+});++test('cloning a running database through the API copies its volumes after a successful stop', function () {+ cloneSafetyUseDatabaseQueue();+ $database = cloneSafetyCreateDatabase($this, 'running:healthy');+ StopDatabase::shouldRun()->once()->andReturn('Database stopped successfully');+ StartDatabase::shouldRun()->once();++ $this->withHeaders(cloneSafetyApiHeaders($this->user, $this->team->id))+ ->postJson("/api/v1/databases/{$database->uuid}/clone", [+ 'destination_uuid' => $this->destination->uuid,+ 'clone_volumes' => true,+ ])+ ->assertCreated();++ cloneSafetyWorkQueue();++ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ expect(DB::table('failed_jobs')->count())->toBe(0);+ Notification::assertNothingSent();+});++test('cloning a stopped database through the API copies its volumes without stopping or starting it', function () {+ Bus::fake();+ $database = cloneSafetyCreateDatabase($this, 'exited');++ $this->withHeaders(cloneSafetyApiHeaders($this->user, $this->team->id))+ ->postJson("/api/v1/databases/{$database->uuid}/clone", [+ 'destination_uuid' => $this->destination->uuid,+ 'clone_volumes' => true,+ ])+ ->assertCreated();++ Bus::assertNotDispatched(CallQueuedClosure::class);+ Bus::assertDispatched(VolumeCloneJob::class);+});++/**+ * `docker ps` lists the running containers: before the stop the given ones, after it the remaining ones.+ *+ * @param list<string> $before+ * @param list<string> $after+ */+function cloneSafetyListRunningContainersBeforeAndAfterStop(array $before, array $after): void+{+ $listings = 0;+ Process::fake(function (PendingProcess $process) use (&$listings, $before, $after) {+ if (! str_contains(remoteProcessScript($process), 'docker ps --filter')) {+ return Process::result(output: '');+ }++ return Process::result(output: implode("\n", $listings++ === 0 ? $before : $after));+ });+}++function cloneSafetyCloneApplicationWithVolumes(object $test, Application $application): void+{+ $test->service->delete();+ Livewire::test(ResourceOperations::class, ['resource' => $application])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $test->destination->uuid)+ ->assertHasNoErrors();+ cloneSafetyWorkQueue();+}++test('a failed stop that took down some containers of the source application redeploys the commit that ran before', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ $application = cloneSafetyCreateApplication($this, 'running:healthy');+ $runningCommit = str_repeat('a', 40);+ foreach ([$runningCommit => now()->subHour(), str_repeat('b', 40) => now()->subDay()] as $commit => $updatedAt) {+ ApplicationDeploymentQueue::create([+ 'application_id' => $application->id, 'deployment_uuid' => new_public_id(), 'pull_request_id' => 0,+ 'commit' => $commit, 'status' => 'finished', 'server_id' => $this->destination->server_id, 'destination_id' => $this->destination->id,+ ])->forceFill(['updated_at' => $updatedAt])->saveQuietly();+ }+ cloneSafetyListRunningContainersBeforeAndAfterStop(['web-'.$application->uuid, 'worker-'.$application->uuid], ['worker-'.$application->uuid]);+ StopApplication::shouldRun()->once()->andReturn('docker stop failed for worker');++ cloneSafetyCloneApplicationWithVolumes($this, $application);++ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ $deployments = ApplicationDeploymentQueue::query()->where('application_id', $application->id)->orderBy('id')->get();+ expect($deployments)->toHaveCount(3);+ $redeploy = $deployments->last();+ expect($redeploy->commit)->toBe($runningCommit)+ ->and($redeploy->pull_request_id)->toBe(0);+ Notification::assertSentTo($this->team, GeneralNotification::class, fn (GeneralNotification $notification) => $notification->success === false+ && str_contains($notification->message, 'did not copy')+ && str_contains($notification->message, "started a redeploy of '{$application->name}'")+ && str_contains($notification->message, 'docker stop failed for worker'));+});++test('a failed stop that took down containers of a source application without a known commit redeploys its configured commit', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ $application = cloneSafetyCreateApplication($this, 'running:healthy');+ cloneSafetyListRunningContainersBeforeAndAfterStop(['web-'.$application->uuid, 'worker-'.$application->uuid], ['worker-'.$application->uuid]);+ StopApplication::shouldRun()->once()->andReturn(null);++ cloneSafetyCloneApplicationWithVolumes($this, $application);++ expect(ApplicationDeploymentQueue::query()->where('application_id', $application->id)->sole()->commit)->toBe('HEAD');+ Notification::assertSentTo($this->team, GeneralNotification::class, fn (GeneralNotification $notification) => str_contains($notification->message, "started a redeploy of '{$application->name}'")+ && str_contains($notification->message, 'could not find the commit'));+});++test('a failed stop that took down no container of the source application does not redeploy it', function () {+ $this->team->webhookNotificationSettings()->update(['webhook_enabled' => true, 'webhook_url' => 'https://example.com/webhook']);+ cloneSafetyUseDatabaseQueue();+ $application = cloneSafetyCreateApplication($this, 'running:healthy');+ cloneSafetyListRunningContainersBeforeAndAfterStop(['web-'.$application->uuid], ['web-'.$application->uuid]);+ StopApplication::shouldRun()->once()->andReturn('Server is not functional');++ cloneSafetyCloneApplicationWithVolumes($this, $application);++ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'cp -a'));+ expect(ApplicationDeploymentQueue::query()->where('application_id', $application->id)->count())->toBe(0);+ Notification::assertSentTo($this->team, GeneralNotification::class, fn (GeneralNotification $notification) => str_contains($notification->message, 'did not copy')+ && str_contains($notification->message, 'is still running'));+});++test('cloning a running application with volume data copies its volumes in one chain between stop and start', function () {+ Bus::fake();+ $application = cloneSafetyCreateApplication($this, 'running:healthy');+ $this->service->delete();++ Livewire::test(ResourceOperations::class, ['resource' => $application])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ Bus::assertChained([CallQueuedClosure::class, VolumeCloneJob::class, CallQueuedClosure::class]);+ expect(ApplicationDeploymentQueue::query()->count())->toBe(0);+});++test('cloning a stopped application with volume data neither stops nor redeploys it', function () {+ Bus::fake();+ $application = cloneSafetyCreateApplication($this, 'exited');+ $this->service->delete();++ Livewire::test(ResourceOperations::class, ['resource' => $application])+ ->set('cloneVolumeData', true)+ ->call('cloneTo', $this->destination->uuid)+ ->assertHasNoErrors();++ Bus::assertNotDispatched(CallQueuedClosure::class);+ Bus::assertDispatched(VolumeCloneJob::class);+ expect(ApplicationDeploymentQueue::query()->count())->toBe(0);+});diff --git a/tests/Feature/CloudflareTunnelApiSetupTest.php b/tests/Feature/CloudflareTunnelApiSetupTest.phpindex 3597e7f0dc..acc1e22256 100644--- a/tests/Feature/CloudflareTunnelApiSetupTest.php+++ b/tests/Feature/CloudflareTunnelApiSetupTest.php@@ -272,7 +272,7 @@ describe('disable', function () {->assertNotDispatched('warning');// Detached, so stopping cloudflared does not cut the SSH session that runs through the tunnel.- Process::assertRan(fn ($process) => str_contains($process->command, "nohup setsid sh -c 'sleep 2; docker rm -f coolify-cloudflared' > /dev/null 2>&1 < /dev/null &"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "nohup setsid sh -c 'sleep 2; docker rm -f coolify-cloudflared' > /dev/null 2>&1 < /dev/null &"));Sleep::assertNeverSlept();Http::assertSent(fn (Request $request) => $request->method() === 'DELETE' && str_ends_with($request->url(), CF_TUNNEL_API.'/tun-1'));@@ -335,7 +335,7 @@ describe('disable', function () {Livewire::test(CloudflareTunnel::class, ['server_uuid' => $this->server->uuid])->call('toggleCloudflareTunnels');- Process::assertRan(fn ($process) => str_contains($process->command,+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process),"sudo bash -c 'nohup setsid sh -c '\\''sleep 2; docker rm -f coolify-cloudflared'\\'' > /dev/null 2>&1 < /dev/null &'"));});diff --git a/tests/Feature/ComposeContentVolumeConfinementTest.php b/tests/Feature/ComposeContentVolumeConfinementTest.phpindex 9df9ac2466..7218de04b6 100644--- a/tests/Feature/ComposeContentVolumeConfinementTest.php+++ b/tests/Feature/ComposeContentVolumeConfinementTest.php@@ -132,8 +132,8 @@ function contentHostPathUseServerUser(string $user): voidfunction contentHostPathFakeServer(): void{Process::fake(fn ($process) => Process::result(output: match (true) {- str_contains($process->command, 'readlink -f') => 'OK',- str_contains($process->command, 'test -') => 'NOK',+ str_contains(remoteProcessScript($process), 'readlink -f') => 'OK',+ str_contains(remoteProcessScript($process), 'test -') => 'NOK',default => '',}));}@@ -165,8 +165,8 @@ function contentHostPathAssertWritten(string $path, string $content, string $use$escapedParent = escapeshellarg(dirname($path));$base64 = base64_encode($content);- Process::assertRan(fn ($process) => str_contains($process->command, "{$sudo}mkdir -p {$escapedParent}"));- Process::assertRan(fn ($process) => str_contains($process->command, "echo '{$base64}' | {$sudo}base64 -d | {$sudo}tee {$escapedPath} > /dev/null"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "{$sudo}mkdir -p {$escapedParent}"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "echo '{$base64}' | {$sudo}base64 -d | {$sudo}tee {$escapedPath} > /dev/null"));}/**@@ -230,8 +230,8 @@ test('loading files for a service writes content inside the resource directory',$sudo = $user === 'root' ? '' : 'sudo ';$escapedPath = escapeshellarg($writePath);$base64 = base64_encode($content);- Process::assertRan(fn ($process) => str_contains($process->command, "{$sudo}mkdir -p -- \"$({$sudo}dirname -- {$escapedPath})\""));- Process::assertRan(fn ($process) => str_contains($process->command, "echo '{$base64}' | {$sudo}base64 -d | {$sudo}tee -- {$escapedPath}"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "{$sudo}mkdir -p -- \"$({$sudo}dirname -- {$escapedPath})\""));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "echo '{$base64}' | {$sudo}base64 -d | {$sudo}tee -- {$escapedPath}"));})->with(['home directory' => ['/etc/b.conf', 'b'],'bare relative path' => ['/etc/c.conf', 'c'],@@ -247,7 +247,7 @@ test('loading files for a service does not write content outside the resource dicontentHostPathFakeServer();getFilesystemVolumesFromServer($fileVolume->resource, true);- Process::assertNotRan(fn ($process) => str_contains($process->command, base64_encode($content)));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), base64_encode($content)));})->with(['absolute host file' => ['/etc/a.conf', 'a'],'parent directory' => ['/etc/d.conf', 'd'],@@ -266,8 +266,8 @@ test('a quote in a content path is shell-escaped in every write command', functi$fileVolume->saveStorageOnServer();$sudo = $user === 'root' ? '' : 'sudo ';- Process::assertRan(fn ($process) => str_contains($process->command, "{$sudo}tee '/etc/my'\\''app/app.conf' > /dev/null"));- Process::assertRan(fn ($process) => str_contains($process->command, "{$sudo}mkdir -p '/etc/my'\\''app'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "{$sudo}tee '/etc/my'\\''app/app.conf' > /dev/null"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "{$sudo}mkdir -p '/etc/my'\\''app'"));})->with(['root', 'coolify']);test('the parsers reject shell injection in a content volume source', function () {@@ -294,8 +294,8 @@ test('a bind mount without content keeps its administrator-selected host path',contentHostPathFakeServer();$fileVolume->saveStorageOnServer();- Process::assertRan(fn ($process) => str_contains($process->command, "mkdir -p '/srv/shared'"));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'readlink -f'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "mkdir -p '/srv/shared'"));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'readlink -f'));});test('the service stack form saves content volumes with host paths', function () {@@ -357,7 +357,7 @@ test('the application General form rejects shell injection in a content volume sfunction contentHostPathFakeGitCompose(string $compose): void{Process::fake(function ($process) use ($compose) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: match (true) {str_contains($command, 'git cat-file blob') => $compose,@@ -436,5 +436,5 @@ test('loading files for a service does not create directories outside the resourcontentHostPathFakeServer();getFilesystemVolumesFromServer($fileVolume->resource, true);- Process::assertNotRan(fn ($process) => str_contains($process->command, '/etc/cron.d/coolify-test') && str_contains($process->command, 'mkdir'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), '/etc/cron.d/coolify-test') && str_contains(remoteProcessScript($process), 'mkdir'));});diff --git a/tests/Feature/ComposeExternalVolumeTest.php b/tests/Feature/ComposeExternalVolumeTest.phpindex 08d8f16c09..af9dcee98f 100644--- a/tests/Feature/ComposeExternalVolumeTest.php+++ b/tests/Feature/ComposeExternalVolumeTest.php@@ -261,7 +261,7 @@ function loginExternalVolumeOwner(): voidfunction fakeExternalVolumeServer(array &$commands): void{Process::fake(function ($process) use (&$commands) {- $commands[] = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $commands[] = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: '');});diff --git a/tests/Feature/ComposeFileLoadCommandsTest.php b/tests/Feature/ComposeFileLoadCommandsTest.phpindex b9ed445fe9..92dd32f45e 100644--- a/tests/Feature/ComposeFileLoadCommandsTest.php+++ b/tests/Feature/ComposeFileLoadCommandsTest.php@@ -1,9 +1,14 @@<?phpuse App\Models\Application;+use App\Models\GiteaApp;+use App\Models\GithubApp;+use App\Models\GitlabApp;use App\Models\InstanceSettings;+use App\Models\PrivateKey;use App\Models\Server;use Illuminate\Foundation\Testing\RefreshDatabase;+use Illuminate\Support\Facades\Storage;use Symfony\Component\Process\Process;uses(RefreshDatabase::class);@@ -240,3 +245,165 @@ test('a symlinked Compose file inside a Git submodule is rejected', function ()->and($process->getErrorOutput())->toContain('symbolic link')->and($process->getErrorOutput())->not->toContain('linked-target-value');});++test('the Compose file loads from a Git submodule with a deploy key', function (string $user) {+ Storage::fake('ssh-keys');+ $submoduleSource = $this->root.'/infra-src';+ mkdir($submoduleSource);+ file_put_contents("{$submoduleSource}/docker-compose.yml", $this->compose);+ $gitEnv = ['GIT_CONFIG_COUNT' => '1', 'GIT_CONFIG_KEY_0' => 'protocol.file.allow', 'GIT_CONFIG_VALUE_0' => 'always'];+ foreach ([+ [$submoduleSource, ['git', 'init', '-q', '-b', 'main']],+ [$submoduleSource, ['git', 'add', '-A']],+ [$submoduleSource, ['git', '-c', '[email protected]', '-c', 'user.name=Test', 'commit', '-qm', 'infra']],+ [$submoduleSource, ['git', 'clone', '-q', '--bare', '.', $this->root.'/infra.git']],+ [$this->root.'/src', ['git', 'submodule', '-q', 'add', $this->root.'/infra.git', 'infra']],+ [$this->root.'/src', ['git', '-c', '[email protected]', '-c', 'user.name=Test', 'commit', '-qm', 'add infra']],+ [$this->root.'/src', ['git', 'push', '-q', $this->root.'/repo.git', 'main']],+ ] as [$cwd, $command]) {+ (new Process($command, $cwd, $gitEnv))->mustRun();+ }++ // A git stub that clones the SSH remote from the local bare repository, and fails a submodule+ // update like an SSH authentication error when the deploy key of GIT_SSH_COMMAND is missing.+ $realGit = trim((new Process(['sh', '-c', 'command -v git']))->mustRun()->getOutput());+ file_put_contents($this->root.'/bin/git', <<<SH+ #!/bin/bash+ args=()+ for arg in "\$@"; do+ [[ \$arg == git@* ]] && arg='{$this->root}/repo.git'+ args+=("\$arg")+ done+ if [[ " \$* " == *" submodule update "* ]]; then+ key=\$(printf '%s' "\$GIT_SSH_COMMAND" | sed -n 's/.* -i \\([^ ]*\\).*/\\1/p')+ if [ ! -f "\$key" ]; then echo 'Permission denied (publickey).' >&2; exit 128; fi+ fi+ exec {$realGit} "\${args[@]}"+ SH);+ chmod($this->root.'/bin/git', 0755);++ $application = Application::factory()->create([+ 'build_pack' => 'dockercompose',+ 'git_repository' => '[email protected]:org/repo.git',+ 'git_branch' => 'main',+ 'git_commit_sha' => 'HEAD',+ 'base_directory' => '/',+ 'docker_compose_location' => '/infra/docker-compose.yml',+ 'private_key_id' => PrivateKey::factory()->create()->id,+ ]);+ $application->settings->update(['is_git_submodules_enabled' => true]);+ $uuid = 'compose-load-'.bin2hex(random_bytes(4));++ $commands = (new ReflectionMethod(Application::class, 'composeFileReadCommands'))->invoke($application->fresh(), $uuid);+ // Keep the deploy key out of the real /root/.ssh of the test host.+ $commands = $commands->map(fn (string $command) => str_replace('/root/.ssh', $this->root.'/ssh', $command));+ if ($user !== 'root') {+ $server = Mockery::mock(Server::class)->makePartial();+ $server->shouldReceive('getAttribute')->with('user')->andReturn($user);+ $server->shouldReceive('setAttribute')->andReturnSelf();+ $commands = collect(parseCommandsByLineForSudo($commands, $server));+ }++ $process = new Process(['bash', '-se'], env: ['PATH' => $this->root.'/bin:'.getenv('PATH')] + $gitEnv);+ $process->setInput($commands->implode("\n")."\n");+ $process->run();+ (new Process(['rm', '-rf', "/tmp/{$uuid}"]))->run();++ expect($process->getErrorOutput())->not->toContain('Permission denied')+ ->and($process->getExitCode())->toBe(0)+ ->and($process->getOutput())->toBe($this->compose)+ ->and(glob($this->root.'/ssh/*'))->toBe([]);+})->with([+ 'root' => ['root'],+ 'non-root' => ['ubuntu'],+]);++test('a pinned commit loads the Compose file of that commit for every source type', function (string $sourceType, bool $shallow, string $user) {+ Storage::fake('ssh-keys');+ $pinnedCommit = $this->commit;+ file_put_contents($this->root.'/src/apps/web/docker-compose.yml', "services:\n api:\n image: branch-head\n");+ foreach ([+ ['git', '-c', '[email protected]', '-c', 'user.name=Test', 'commit', '-qam', 'branch head'],+ ['git', 'push', '-q', $this->root.'/repo.git', 'main'],+ ] as $command) {+ (new Process($command, $this->root.'/src'))->mustRun();+ }++ // A git stub that sends every remote URL (HTTPS with a token, or SSH) to the local bare repository.+ $realGit = trim((new Process(['sh', '-c', 'command -v git']))->mustRun()->getOutput());+ file_put_contents($this->root.'/bin/git', <<<SH+ #!/bin/bash+ args=()+ for arg in "\$@"; do+ [[ \$arg == git@* || \$arg == https://* ]] && arg='{$this->root}/repo.git'+ args+=("\$arg")+ done+ exec {$realGit} "\${args[@]}"+ SH);+ chmod($this->root.'/bin/git', 0755);++ $privateKey = PrivateKey::factory()->create();+ $application = Application::factory()->create([+ 'build_pack' => 'dockercompose',+ 'git_repository' => $sourceType === 'other' ? $this->root.'/repo.git' : '[email protected]:org/repo.git',+ 'git_branch' => 'main',+ 'git_commit_sha' => $pinnedCommit,+ 'base_directory' => '/apps/web',+ 'docker_compose_location' => '/docker-compose.yml',+ 'private_key_id' => $sourceType === 'deploy key' ? $privateKey->id : null,+ ]);+ $application->settings->update(['is_git_shallow_clone_enabled' => $shallow]);+ $application = $application->fresh();++ if ($sourceType === 'GitHub App') {+ $application->git_repository = 'org/repo';+ $source = Mockery::mock(GithubApp::class)->makePartial();+ $source->forceFill(['html_url' => 'https://github.example.test', 'is_public' => false]);+ $source->shouldReceive('getMorphClass')->andReturn(GithubApp::class);+ $source->shouldReceive('installationToken')->andReturn('ghs_token');+ $application->setRelation('source', $source);+ } elseif ($sourceType === 'Gitea token') {+ $application->git_repository = 'org/repo';+ $source = (new GiteaApp)->forceFill(['html_url' => 'https://gitea.example.test', 'api_url' => 'https://gitea.example.test/api/v1', 'is_public' => false]);+ $source->access_token = 'gitea-token';+ $source->refresh_token = 'gitea-refresh-token';+ $source->expires_at = time() + 3600;+ $application->setRelation('source', $source);+ } elseif ($sourceType === 'GitLab SSH key') {+ $source = (new GitlabApp)->forceFill(['html_url' => 'https://gitlab.example.test', 'team_id' => $privateKey->team_id, 'custom_port' => 22]);+ $source->setRelation('privateKey', $privateKey);+ $application->setRelation('source', $source);+ }++ $uuid = 'compose-load-'.bin2hex(random_bytes(4));+ $commands = (new ReflectionMethod(Application::class, 'composeFileReadCommands'))->invoke($application, $uuid);+ // Keep the deploy key out of the real /root/.ssh of the test host.+ $commands = $commands->map(fn (string $command) => str_replace('/root/.ssh', $this->root.'/ssh', $command));+ if ($user !== 'root') {+ $server = Mockery::mock(Server::class)->makePartial();+ $server->shouldReceive('getAttribute')->with('user')->andReturn($user);+ $server->shouldReceive('setAttribute')->andReturnSelf();+ $commands = collect(parseCommandsByLineForSudo($commands, $server));+ }++ $process = new Process(['bash', '-se'], env: ['PATH' => $this->root.'/bin:'.getenv('PATH')]);+ $process->setInput($commands->implode("\n")."\n");+ $process->run();+ (new Process(['rm', '-rf', "/tmp/{$uuid}"]))->run();++ expect($process->getErrorOutput())->not->toContain('syntax error')+ ->and($process->getExitCode())->toBe(0)+ ->and($process->getOutput())->toBe($this->compose);+})->with([+ 'public URL' => ['other'],+ 'deploy key' => ['deploy key'],+ 'GitHub App' => ['GitHub App'],+ 'Gitea token' => ['Gitea token'],+ 'GitLab SSH key' => ['GitLab SSH key'],+])->with([+ 'full clone' => [false],+ 'shallow clone' => [true],+])->with([+ 'root' => ['root'],+ 'non-root' => ['ubuntu'],+]);diff --git a/tests/Feature/ComposeFileLoadErrorTest.php b/tests/Feature/ComposeFileLoadErrorTest.phpindex 5513cfe8d4..c266f5c795 100644--- a/tests/Feature/ComposeFileLoadErrorTest.php+++ b/tests/Feature/ComposeFileLoadErrorTest.php@@ -35,7 +35,7 @@ beforeEach(function () {function fakeComposeLoadServer(?string $failingStep, string $errorOutput): void{Process::fake(function ($process) use ($failingStep, $errorOutput) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);if ($failingStep !== null && str_contains($command, $failingStep)) {return Process::result(errorOutput: $errorOutput, exitCode: 128);}diff --git a/tests/Feature/ComposePreviewEnvironmentValuesTest.php b/tests/Feature/ComposePreviewEnvironmentValuesTest.phpnew file mode 100644index 0000000000..eb6657c480--- /dev/null+++ b/tests/Feature/ComposePreviewEnvironmentValuesTest.php@@ -0,0 +1,109 @@+<?php++use App\Models\Application;+use App\Models\ApplicationPreview;+use App\Models\Environment;+use App\Models\EnvironmentVariable;+use App\Models\Project;+use App\Models\Server;+use App\Models\SharedEnvironmentVariable;+use App\Models\StandaloneDocker;+use App\Models\Team;+use Illuminate\Foundation\Testing\RefreshDatabase;++uses(RefreshDatabase::class);++beforeEach(function () {+ Server::flushIdentityMap();++ $team = Team::factory()->create();+ $server = Server::factory()->create(['team_id' => $team->id]);+ $destination = StandaloneDocker::query()->where('server_id', $server->id)->first()+ ?? StandaloneDocker::factory()->create(['server_id' => $server->id]);+ $project = Project::factory()->create(['team_id' => $team->id]);+ $environment = Environment::factory()->create(['project_id' => $project->id]);++ SharedEnvironmentVariable::create([+ 'key' => 'TEAM_SECRET', 'value' => 'team-secret-value', 'type' => 'team', 'team_id' => $team->id,+ ]);++ $this->application = Application::factory()->create([+ 'build_pack' => 'dockercompose',+ 'environment_id' => $environment->id,+ 'destination_id' => $destination->id,+ 'destination_type' => $destination->getMorphClass(),+ 'docker_compose_raw' => <<<'YAML'+services:+ web:+ image: nginx:alpine+ environment:+ CHANGED_FOR_PREVIEW: ''+ SAME_AS_PRODUCTION: ''+ FROM_TEAM: '{{team.TEAM_SECRET}}'+YAML,+ ]);++ // Creating a production variable also creates a preview copy with the same value.+ foreach (['CHANGED_FOR_PREVIEW', 'SAME_AS_PRODUCTION'] as $key) {+ EnvironmentVariable::create([+ 'key' => $key,+ 'value' => 'production-value',+ 'resourceable_type' => Application::class,+ 'resourceable_id' => $this->application->id,+ 'is_preview' => false,+ ]);+ }+ $this->application->environment_variables_preview()->where('key', 'CHANGED_FOR_PREVIEW')->firstOrFail()+ ->update(['value' => 'preview-value']);+});++function composePreviewEnvironmentValues(Application $application, int $pullRequestId = 0): array+{+ $previewId = null;+ if ($pullRequestId !== 0) {+ $previewId = ApplicationPreview::create([+ 'application_id' => $application->id,+ 'pull_request_id' => $pullRequestId,+ 'pull_request_html_url' => "https://github.com/example/repository/pull/{$pullRequestId}",+ ])->id;+ }++ $serviceName = $pullRequestId === 0 ? 'web' : "web-pr-{$pullRequestId}";+ $environment = collect(data_get($application->fresh()->parse(pull_request_id: $pullRequestId, preview_id: $previewId)->toArray(), "services.{$serviceName}.environment"));++ return $environment->only(['CHANGED_FOR_PREVIEW', 'SAME_AS_PRODUCTION', 'FROM_TEAM'])->all();+}++it('fills empty compose values of a production deployment with the production variables', function () {+ expect(composePreviewEnvironmentValues($this->application))->toBe([+ 'CHANGED_FOR_PREVIEW' => 'production-value',+ 'SAME_AS_PRODUCTION' => 'production-value',+ 'FROM_TEAM' => 'team-secret-value',+ ]);+});++it('fills empty compose values of a pull request deployment with the preview variables', function () {+ expect(composePreviewEnvironmentValues($this->application, 7))->toBe([+ 'CHANGED_FOR_PREVIEW' => 'preview-value',+ 'SAME_AS_PRODUCTION' => 'production-value',+ 'FROM_TEAM' => 'team-secret-value',+ ]);+});++it('gives a public pull request deployment no production values or shared variables from the compose file', function () {+ $this->application->settings->update(['is_pr_deployments_public_enabled' => true]);++ expect(composePreviewEnvironmentValues($this->application, 7))->toBe([+ 'CHANGED_FOR_PREVIEW' => 'preview-value',+ 'SAME_AS_PRODUCTION' => '',+ 'FROM_TEAM' => '{{team.TEAM_SECRET}}',+ ]);+});++it('resolves shared variables in a preview value of a public pull request deployment', function () {+ $this->application->settings->update(['is_pr_deployments_public_enabled' => true]);+ $this->application->environment_variables_preview()->where('key', 'CHANGED_FOR_PREVIEW')->firstOrFail()+ ->update(['value' => '{{team.TEAM_SECRET}}']);++ expect(composePreviewEnvironmentValues($this->application, 7)['CHANGED_FOR_PREVIEW'])->toBe('team-secret-value');+});diff --git a/tests/Feature/ComposePreviewMagicVariablesTest.php b/tests/Feature/ComposePreviewMagicVariablesTest.phpnew file mode 100644index 0000000000..fcdf777695--- /dev/null+++ b/tests/Feature/ComposePreviewMagicVariablesTest.php@@ -0,0 +1,217 @@+<?php++use App\Models\Application;+use App\Models\ApplicationPreview;+use App\Models\Environment;+use App\Models\EnvironmentVariable;+use App\Models\Project;+use App\Models\Server;+use App\Models\StandaloneDocker;+use App\Models\Team;+use Illuminate\Foundation\Testing\RefreshDatabase;++uses(RefreshDatabase::class);++beforeEach(function () {+ Server::flushIdentityMap();++ $team = Team::factory()->create();+ $server = Server::factory()->create(['team_id' => $team->id]);+ $destination = StandaloneDocker::query()->where('server_id', $server->id)->first()+ ?? StandaloneDocker::factory()->create(['server_id' => $server->id]);+ $project = Project::factory()->create(['team_id' => $team->id]);+ $environment = Environment::factory()->create(['project_id' => $project->id]);++ $this->application = Application::factory()->create([+ 'build_pack' => 'dockercompose',+ 'environment_id' => $environment->id,+ 'destination_id' => $destination->id,+ 'destination_type' => $destination->getMorphClass(),+ 'docker_compose_raw' => <<<'YAML'+services:+ db:+ image: postgres:16-alpine+ environment:+ POSTGRES_USER: ${SERVICE_USER_DB}+ POSTGRES_PASSWORD: ${SERVICE_PASSWORD_DB}+YAML,+ ]);+});++function composePreviewMagicParse(Application $application, int $pullRequestId = 0): void+{+ $previewId = null;+ if ($pullRequestId !== 0) {+ $previewId = ApplicationPreview::query()->firstOrCreate([+ 'application_id' => $application->id,+ 'pull_request_id' => $pullRequestId,+ ], [+ 'pull_request_html_url' => "https://github.com/example/repository/pull/{$pullRequestId}",+ ])->id;+ }++ $application->fresh()->parse(pull_request_id: $pullRequestId, preview_id: $previewId);+}++/**+ * @return array{production: array<string, ?string>, preview: array<string, ?string>}+ */+function composePreviewMagicValues(Application $application): array+{+ $keys = ['SERVICE_PASSWORD_DB', 'SERVICE_USER_DB'];++ return [+ 'production' => $application->environment_variables()->whereIn('key', $keys)->pluck('value', 'key')->sortKeys()->all(),+ 'preview' => $application->environment_variables_preview()->whereIn('key', $keys)->pluck('value', 'key')->sortKeys()->all(),+ ];+}++it('generates the magic variables of a first pull request parse only in the preview variables', function () {+ composePreviewMagicParse($this->application, 7);++ $values = composePreviewMagicValues($this->application);+ expect($values['production'])->toBe([])+ ->and(array_keys($values['preview']))->toBe(['SERVICE_PASSWORD_DB', 'SERVICE_USER_DB'])+ ->and($values['preview']['SERVICE_PASSWORD_DB'])->not->toBeEmpty();++ // The production deployment generates its own values and keeps the preview values.+ composePreviewMagicParse($this->application);++ $afterProduction = composePreviewMagicValues($this->application);+ expect($afterProduction['preview'])->toBe($values['preview'])+ ->and($afterProduction['production']['SERVICE_PASSWORD_DB'])->not->toBeEmpty()+ ->and($afterProduction['production']['SERVICE_PASSWORD_DB'])->not->toBe($values['preview']['SERVICE_PASSWORD_DB']);+});++it('does not copy a generated production magic secret to the preview variables', function (string $key) {+ EnvironmentVariable::create([+ 'key' => $key,+ 'value' => 'production-secret',+ 'is_preview' => false,+ 'resourceable_type' => Application::class,+ 'resourceable_id' => $this->application->id,+ ]);++ expect($this->application->environment_variables_preview()->where('key', $key)->exists())->toBeFalse();+})->with([+ 'SERVICE_PASSWORD_DB',+ 'SERVICE_PASSWORD_64_DB',+ 'SERVICE_PASSWORDWITHSYMBOLS_DB',+ 'SERVICE_USER_DB',+ 'SERVICE_LOWERCASEUSER_DB',+ 'SERVICE_BASE64_DB',+ 'SERVICE_BASE64_64_DB',+ 'SERVICE_BASE64_128_DB',+ 'SERVICE_REALBASE64_DB',+ 'SERVICE_HEX_32_DB',+]);++it('still copies a normal or service url production variable to the preview variables', function (string $key) {+ EnvironmentVariable::create([+ 'key' => $key,+ 'value' => 'production-value',+ 'is_preview' => false,+ 'resourceable_type' => Application::class,+ 'resourceable_id' => $this->application->id,+ ]);++ expect($this->application->environment_variables_preview()->where('key', $key)->value('value'))->toBe('production-value');+})->with(['DATABASE_HOST', 'SERVICE_URL_DB', 'SERVICE_FQDN_DB', 'SERVICE_PASSWORD_DB_MAIN']);++it('gives a private pull request its own magic values instead of the production values', function () {+ composePreviewMagicParse($this->application);+ $production = composePreviewMagicValues($this->application)['production'];++ composePreviewMagicParse($this->application, 7);++ $values = composePreviewMagicValues($this->application);+ expect($values['production'])->toBe($production)+ ->and(array_keys($values['preview']))->toBe(['SERVICE_PASSWORD_DB', 'SERVICE_USER_DB'])+ ->and($values['preview']['SERVICE_PASSWORD_DB'])->not->toBeEmpty()+ ->and($values['preview']['SERVICE_PASSWORD_DB'])->not->toBe($production['SERVICE_PASSWORD_DB'])+ ->and($values['preview']['SERVICE_USER_DB'])->not->toBe($production['SERVICE_USER_DB']);++ // The next pull request parse keeps the preview values, so the preview database keeps its password.+ composePreviewMagicParse($this->application, 7);++ expect(composePreviewMagicValues($this->application))->toBe($values);+});++it('keeps an existing preview copy of a production magic value for a private pull request', function () {+ composePreviewMagicParse($this->application);+ $production = composePreviewMagicValues($this->application)['production'];+ // A preview copy that an earlier version created; existing preview databases use this password.+ $this->application->environment_variables_preview()->create([+ 'key' => 'SERVICE_PASSWORD_DB',+ 'value' => $production['SERVICE_PASSWORD_DB'],+ 'is_preview' => true,+ ]);++ composePreviewMagicParse($this->application, 7);++ expect(composePreviewMagicValues($this->application)['preview']['SERVICE_PASSWORD_DB'])->toBe($production['SERVICE_PASSWORD_DB']);+});++it('keeps a preview magic variable that the user created', function () {+ EnvironmentVariable::create([+ 'key' => 'SERVICE_PASSWORD_DB',+ 'value' => 'user-preview-password',+ 'is_preview' => true,+ 'resourceable_type' => Application::class,+ 'resourceable_id' => $this->application->id,+ ]);++ composePreviewMagicParse($this->application, 7);++ expect(composePreviewMagicValues($this->application)['preview']['SERVICE_PASSWORD_DB'])->toBe('user-preview-password');+});++it('gives a public pull request its own magic values instead of the production values', function () {+ composePreviewMagicParse($this->application);+ $production = composePreviewMagicValues($this->application)['production'];+ $this->application->settings->update(['is_pr_deployments_public_enabled' => true]);++ composePreviewMagicParse($this->application, 7);++ $values = composePreviewMagicValues($this->application);+ expect($values['production'])->toBe($production)+ ->and($values['preview']['SERVICE_PASSWORD_DB'])->not->toBeEmpty()+ ->and($values['preview']['SERVICE_PASSWORD_DB'])->not->toBe($production['SERVICE_PASSWORD_DB'])+ ->and($values['preview']['SERVICE_USER_DB'])->not->toBe($production['SERVICE_USER_DB']);++ // The next pull request parse keeps the preview values, so the preview database keeps its password.+ composePreviewMagicParse($this->application, 7);++ expect(composePreviewMagicValues($this->application))->toBe($values);+});++it('keeps a changed preview magic value of a public pull request', function () {+ composePreviewMagicParse($this->application);+ $this->application->settings->update(['is_pr_deployments_public_enabled' => true]);+ $this->application->environment_variables_preview()->create([+ 'key' => 'SERVICE_PASSWORD_DB',+ 'value' => 'preview-password',+ 'is_preview' => true,+ ]);++ composePreviewMagicParse($this->application, 7);++ expect(composePreviewMagicValues($this->application)['preview']['SERVICE_PASSWORD_DB'])->toBe('preview-password');+});++it('replaces an existing preview copy of a production magic value for a public pull request', function () {+ composePreviewMagicParse($this->application);+ $production = composePreviewMagicValues($this->application)['production'];+ $this->application->environment_variables_preview()->create([+ 'key' => 'SERVICE_PASSWORD_DB',+ 'value' => $production['SERVICE_PASSWORD_DB'],+ 'is_preview' => true,+ ]);+ $this->application->settings->update(['is_pr_deployments_public_enabled' => true]);++ composePreviewMagicParse($this->application, 7);++ expect(composePreviewMagicValues($this->application)['preview']['SERVICE_PASSWORD_DB'])+ ->not->toBeEmpty()+ ->not->toBe($production['SERVICE_PASSWORD_DB']);+});diff --git a/tests/Feature/ComposeStaleVolumeTest.php b/tests/Feature/ComposeStaleVolumeTest.phpindex 17b2b251d4..ad85fc3729 100644--- a/tests/Feature/ComposeStaleVolumeTest.php+++ b/tests/Feature/ComposeStaleVolumeTest.php@@ -95,7 +95,7 @@ it('deletes the entry of a volume that the Compose file no longer declares', fun$name = staleVolume('third')->name;$commands = [];Process::fake(function ($process) use (&$commands) {- $commands[] = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $commands[] = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: '');});diff --git a/tests/Feature/ComposeVolumeDriverOptionsTest.php b/tests/Feature/ComposeVolumeDriverOptionsTest.phpindex ebd79a1dc3..46668b1f1f 100644--- a/tests/Feature/ComposeVolumeDriverOptionsTest.php+++ b/tests/Feature/ComposeVolumeDriverOptionsTest.php@@ -527,7 +527,7 @@ describe('storage page', function () {$volume = LocalPersistentVolume::where('name', $name)->sole();$commands = [];Process::fake(function ($process) use (&$commands) {- $commands[] = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $commands[] = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: '');});diff --git a/tests/Feature/ContainerFilesystemServiceTest.php b/tests/Feature/ContainerFilesystemServiceTest.phpindex aa24f3fafe..24d221c064 100644--- a/tests/Feature/ContainerFilesystemServiceTest.php+++ b/tests/Feature/ContainerFilesystemServiceTest.php@@ -157,9 +157,9 @@ it('lists through a locked-down helper container that runs as the container proc$server->update(['user' => 'root']);$commands = [];Process::fake(function ($process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);- return str_contains($process->command, '/status')+ return str_contains(remoteProcessScript($process), '/status')? Process::result(output: fsProcessStatus('65532', '0', '0 1000', '0000000000000000')): Process::result(output: "regular file\t9\t1\t644\troot\troot\t./users.yml");});@@ -189,7 +189,7 @@ it('wraps the helper command in one sudo shell for non-root servers', function ($server->update(['user' => 'deploy']);$commands = [];Process::fake(function ($process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return count($commands) === 1? Process::result(output: fsProcessStatus('0', '0', '0', '00000000a80425fb'))@@ -264,7 +264,7 @@ it('writes content near or above the shell argument limit via docker cp', functi(new ContainerFilesystemService($server, 'app-123'))->write('/app/big.txt', str_repeat('a', $size));- Process::assertRan(fn ($process) => str_contains($process->command, 'docker cp'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker cp'));})->with([96 * 1024, 200_000]);it('base64-encodes content in the write command', function () {@@ -363,8 +363,8 @@ it('uploads by scp-ing to the server then docker cp into the container', functio(new ContainerFilesystemService($server, 'app-123'))->upload($local, '/app/plugins/x.jar');- Process::assertRan(fn ($process) => str_contains($process->command, 'docker cp')- && str_contains($process->command, 'app-123'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker cp')+ && str_contains(remoteProcessScript($process), 'app-123'));@unlink($local);});@@ -379,7 +379,7 @@ it('executes read guards for real file bytes through the non-root sudo parser',$path = tempnam(sys_get_temp_dir(), 'fs-read-');file_put_contents($path, $content);Process::fake(function ($process) {- $lines = explode("\n", $process->command);+ $lines = explode("\n", remoteProcessScript($process));$body = implode("\n", array_slice($lines, 1, -1));$local = new Symfony\Component\Process\Process(['bash', '-c',@@ -415,7 +415,7 @@ it('rejects a symlink to an oversized file before reading its content', functionfclose($handle);symlink($path, $link);Process::fake(function ($process) {- $body = implode("\n", array_slice(explode("\n", $process->command), 1, -1));+ $body = implode("\n", array_slice(explode("\n", remoteProcessScript($process)), 1, -1));$local = new Symfony\Component\Process\Process(['bash', '-c', 'sudo() { "$@"; }; docker() { shift 2; "$@"; }; export -f sudo docker;'."\n".$body]);$local->run();@@ -461,13 +461,13 @@ it('streams the remote command output to the writer in chunks', function () {$server = fsServer();$svc = new class($server, 'app-123') extends ContainerFilesystemService{- public ?string $remoteCommand = null;+ public string $stdinFile;- protected function sshCommand(string $command): string+ protected function sshCommand(): string{- $this->remoteCommand = $command;+ $this->stdinFile = tempnam(sys_get_temp_dir(), 'fs-stream-');- return 'head -c 200000 /dev/zero';+ return 'cat > '.escapeshellarg($this->stdinFile).'; head -c 200000 /dev/zero';}};$received = '';@@ -478,7 +478,9 @@ it('streams the remote command output to the writer in chunks', function () {$chunks++;});- expect($svc->remoteCommand)->toEndWith("docker exec 'app-123' cat -- '/app/a.bin'");+ $remoteCommand = file_get_contents($svc->stdinFile);+ unlink($svc->stdinFile);+ expect($remoteCommand)->toEndWith("docker exec 'app-123' cat -- '/app/a.bin'\n");expect(strlen($received))->toBe(200000);expect($chunks)->toBeGreaterThan(1);});diff --git a/tests/Feature/ContainerNameShellQuotingTest.php b/tests/Feature/ContainerNameShellQuotingTest.phpindex 0fa8a6c6eb..db32576d83 100644--- a/tests/Feature/ContainerNameShellQuotingTest.php+++ b/tests/Feature/ContainerNameShellQuotingTest.php@@ -85,12 +85,12 @@ function quotingUseNonRootUser(): voidfunction quotingAssertCommandRan(string $command): void{- Process::assertRan(fn ($process) => str_contains($process->command, $command));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), $command));}function quotingAssertNoUnquotedName(): void{- Process::assertDidntRun(fn ($process) => str_contains($process->command, ' app;touch'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), ' app;touch'));}test('stopping a service application quotes its container name', function (bool $nonRoot, string $sudo) {@@ -155,7 +155,7 @@ test('a scheduled task in a service quotes the container name', function (bool $function quotingFakeApplicationContainer(string $name): void{Process::fake(function ($process) use ($name) {- if (str_contains($process->command, 'docker ps -a --filter')) {+ if (str_contains(remoteProcessScript($process), 'docker ps -a --filter')) {return Process::result(output: json_encode(['Names' => $name, 'Labels' => 'coolify.applicationId=1', 'State' => 'running']));}@@ -183,7 +183,7 @@ test('stopping an application quotes the container names from docker ps', functiStopApplication::run($application, dockerCleanup: false);quotingAssertCommandRan("{$sudo}docker stop --timeout=");- Process::assertRan(fn ($process) => preg_match('/docker stop --timeout=\d+ '.preg_quote($quoted, '/').'\n/', $process->command) === 1);+ Process::assertRan(fn ($process) => preg_match('/docker stop --timeout=\d+ '.preg_quote($quoted, '/').'\n/', remoteProcessScript($process)) === 1);quotingAssertCommandRan("\n{$sudo}docker rm -f {$quoted}\n");quotingAssertNoUnquotedName();})->with(['root' => [false, ''], 'non-root' => [true, 'sudo ']]);@@ -196,7 +196,7 @@ test('stopping an application preview quotes the container names from docker ps''pull_request_html_url' => 'https://example.com/pull/1',]);Process::fake(function ($process) {- if (str_contains($process->command, 'docker ps -a --filter')) {+ if (str_contains(remoteProcessScript($process), 'docker ps -a --filter')) {return Process::result(output: json_encode(['Names' => QUOTING_HOSTILE_NAME, 'Labels' => 'coolify.applicationId=1,coolify.pullRequestId=1']));}@@ -206,7 +206,7 @@ test('stopping an application preview quotes the container names from docker ps'StopApplicationPreview::run($preview);- Process::assertRan(fn ($process) => preg_match('/docker stop --timeout=\d+ '.preg_quote($quoted, '/').'\n/', $process->command) === 1);+ Process::assertRan(fn ($process) => preg_match('/docker stop --timeout=\d+ '.preg_quote($quoted, '/').'\n/', remoteProcessScript($process)) === 1);quotingAssertCommandRan("\ndocker rm -f {$quoted}\n");quotingAssertNoUnquotedName();});@@ -218,7 +218,7 @@ test('stopping an application on one server quotes the container names from dockStopApplicationOneServer::run($application, $this->server);- Process::assertRan(fn ($process) => preg_match('/docker stop --timeout=\d+ '.preg_quote($quoted, '/').'\n/', $process->command) === 1);+ Process::assertRan(fn ($process) => preg_match('/docker stop --timeout=\d+ '.preg_quote($quoted, '/').'\n/', remoteProcessScript($process)) === 1);quotingAssertNoUnquotedName();});diff --git a/tests/Feature/ContainerOwnerUuidLabelsTest.php b/tests/Feature/ContainerOwnerUuidLabelsTest.phpindex 52da7c52e5..8472c90b57 100644--- a/tests/Feature/ContainerOwnerUuidLabelsTest.php+++ b/tests/Feature/ContainerOwnerUuidLabelsTest.php@@ -87,7 +87,7 @@ describe('container lookups for services and databases', function () {]);$commands = collect();Process::fake(function ($process) use ($commands) {- $commands->push($process->command);+ $commands->push(remoteProcessScript($process));return Process::result(output: '');});diff --git a/tests/Feature/ContentFilesBeforeComposeUpTest.php b/tests/Feature/ContentFilesBeforeComposeUpTest.phpindex f4b5ec8f6c..054b812b17 100644--- a/tests/Feature/ContentFilesBeforeComposeUpTest.php+++ b/tests/Feature/ContentFilesBeforeComposeUpTest.php@@ -97,7 +97,7 @@ function fakeContentFilesServer(array $states): voidContentFilesUpRecorder::$commands = [];Process::fake(function ($process) use ($states) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);ContentFilesUpRecorder::$commands[] = ['ssh', $command];if (str_contains($command, 'empty-directory')) {@@ -132,9 +132,9 @@ function contentFilesStateChecks(): arrayfunction contentFilesAssertNoWrite(): void{- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'base64 -d'));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'rmdir'));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'rm -fr'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'base64 -d'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'rmdir'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'rm -fr'));}beforeEach(function () {@@ -256,9 +256,9 @@ test('a Compose deployment checks all content files with one server command', fu->toContain(escapeshellarg($second->fs_path))->toContain(escapeshellarg($third->fs_path))->and($this->logEntries)->toContain(['Writing 1 missing configuration file.', 'stdout']);- Process::assertRan(fn ($process) => str_contains($process->command, 'tee '.escapeshellarg($second->fs_path)));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'tee '.escapeshellarg($first->fs_path)));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'tee '.escapeshellarg($third->fs_path)));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'tee '.escapeshellarg($second->fs_path)));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'tee '.escapeshellarg($first->fs_path)));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'tee '.escapeshellarg($third->fs_path)));});test('a Compose deployment does not change a content file that exists on the server', function () {@@ -286,7 +286,7 @@ test('a Compose deployment replaces an empty directory with the content file', f->and($write)->toBeLessThan(ContentFilesUpRecorder::indexOf('deployment', ' up -d'))->and(strpos($writeCommand, "rmdir {$escapedPath}"))->toBeLessThan(strpos($writeCommand, "tee {$escapedPath}"))->and($this->logEntries)->toContain(['Writing 1 missing configuration file.', 'stdout']);- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'rm -fr'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'rm -fr'));});test('a Compose deployment keeps a directory that is not empty and shows a warning', function () {@@ -420,8 +420,8 @@ test('the content file check and write are safe for non-root servers', function$escapedPath = escapeshellarg($volume->fs_path);expect(contentFilesStateChecks())->not->toBeEmpty()->and(contentFilesStateChecks()[0])->toContain("sudo bash -c 'sh -c ");- Process::assertRan(fn ($process) => str_contains($process->command, "sudo rmdir {$escapedPath}")- && str_contains($process->command, "| sudo tee {$escapedPath}"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "sudo rmdir {$escapedPath}")+ && str_contains(remoteProcessScript($process), "| sudo tee {$escapedPath}"));});test('starting a service writes missing content files before docker compose up', function () {@@ -446,10 +446,10 @@ test('starting a service writes missing content files before docker compose up',->and($command)->toContain("echo 'Warning: A directory that is not empty is at {$workerConf->fs_path} on the server.")->and(strpos($command, 'Writing 1 missing configuration file.'))->toBeLessThan(strpos($command, ' up -d'))->and($command)->not->toContain(base64_encode('listen 8080;'));- Process::assertRan(fn ($process) => str_contains($process->command, 'base64 -d | tee '.escapeshellarg($appConf->fs_path))- && str_contains($process->command, base64_encode('listen 8080;')));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'tee '.escapeshellarg($workerConf->fs_path)));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'rm -fr'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'base64 -d | tee '.escapeshellarg($appConf->fs_path))+ && str_contains(remoteProcessScript($process), base64_encode('listen 8080;')));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'tee '.escapeshellarg($workerConf->fs_path)));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'rm -fr'));});test('deploying one service application writes only its missing content files', function () {@@ -472,8 +472,8 @@ test('deploying one service application writes only its missing content files',expect(contentFilesStateChecks())->toHaveCount(1)->and(contentFilesStateChecks()[0])->not->toContain(escapeshellarg($workerConf->fs_path))->and(strpos($command, 'Writing 1 missing configuration file.'))->toBeLessThan(strpos($command, ' up -d'));- Process::assertRan(fn ($process) => str_contains($process->command, 'tee '.escapeshellarg($appConf->fs_path)));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'tee '.escapeshellarg($workerConf->fs_path)));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'tee '.escapeshellarg($appConf->fs_path)));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'tee '.escapeshellarg($workerConf->fs_path)));});test('starting a service without content files does not run the file check', function () {@@ -523,8 +523,8 @@ test('the storage save job writes the content again when the same file exists',$writes = array_filter(ContentFilesUpRecorder::ssh(), fn (string $command) => str_contains($command, 'base64 -d | tee '.escapeshellarg($volume->fs_path)));expect($writes)->toHaveCount(2);- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'rm -fr'));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'rmdir'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'rm -fr'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'rmdir'));});test('the storage save job replaces an empty directory with the content file', function () {@@ -534,11 +534,11 @@ test('the storage save job replaces an empty directory with the content file', f(new ServerStorageSaveJob($volume))->handle();$escapedPath = escapeshellarg($volume->fs_path);- Process::assertRan(fn ($process) => str_contains($process->command, "rmdir {$escapedPath}")- && str_contains($process->command, base64_encode(CONTENT_FILES_UP_CONTENT))- && strpos($process->command, "rmdir {$escapedPath}") < strpos($process->command, "tee {$escapedPath}"));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'rm -fr'));- Process::assertDidntRun(fn ($process) => str_contains($process->command, "touch {$escapedPath}"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "rmdir {$escapedPath}")+ && str_contains(remoteProcessScript($process), base64_encode(CONTENT_FILES_UP_CONTENT))+ && strpos(remoteProcessScript($process), "rmdir {$escapedPath}") < strpos(remoteProcessScript($process), "tee {$escapedPath}"));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'rm -fr'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), "touch {$escapedPath}"));});test('the storage save job does not delete a directory that is not empty', function () {@@ -573,8 +573,8 @@ test('the server file sync replaces an empty directory with the content file', fexpect($volume->fresh())->is_directory->toBeFalsy()->content->toBe(CONTENT_FILES_UP_CONTENT);- Process::assertRan(fn ($process) => str_contains($process->command, 'rmdir'));- Process::assertRan(fn ($process) => str_contains($process->command, 'base64 -d'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'rmdir'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'base64 -d'));});test('the server file sync still marks a storage without content as a directory', function () {diff --git a/tests/Feature/CreateScheduledBackupValidationTest.php b/tests/Feature/CreateScheduledBackupValidationTest.phpindex 071fdd2035..a2fd3367f0 100644--- a/tests/Feature/CreateScheduledBackupValidationTest.php+++ b/tests/Feature/CreateScheduledBackupValidationTest.php@@ -226,7 +226,7 @@ it('backs up sqlite files added to the database after the backup was scheduled',$commands = collect();Process::fake(function ($process) use ($commands) {- $commands->push(is_array($process->command) ? implode(' ', $process->command) : $process->command);+ $commands->push(is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process));return Process::result(output: '');});diff --git a/tests/Feature/DashboardServerMetricsChartTest.php b/tests/Feature/DashboardServerMetricsChartTest.phpindex f5e1522757..fce49f1aab 100644--- a/tests/Feature/DashboardServerMetricsChartTest.php+++ b/tests/Feature/DashboardServerMetricsChartTest.php@@ -102,5 +102,5 @@ it('loads dashboard metrics over SSH for a reachable server', function () {],);- Process::assertRan(fn ($process) => str_contains($process->command, 'coolify-sentinel'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'coolify-sentinel'));});diff --git a/tests/Feature/DatabaseBackupInstanceDatabaseTest.php b/tests/Feature/DatabaseBackupInstanceDatabaseTest.phpindex acad0bbf28..839cd8ef0e 100644--- a/tests/Feature/DatabaseBackupInstanceDatabaseTest.php+++ b/tests/Feature/DatabaseBackupInstanceDatabaseTest.php@@ -53,7 +53,7 @@ it('backs up a user database named coolify-db as an ordinary database', function])->save();$commands = collect();Process::fake(function ($process) use ($commands) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands->push($command);return Process::result(output: str_contains($command, 'du -b') ? '128' : '');diff --git a/tests/Feature/DatabaseBackupMultipleS3DestinationsTest.php b/tests/Feature/DatabaseBackupMultipleS3DestinationsTest.phpindex 7b6a26bd22..4acf47a048 100644--- a/tests/Feature/DatabaseBackupMultipleS3DestinationsTest.php+++ b/tests/Feature/DatabaseBackupMultipleS3DestinationsTest.php@@ -87,7 +87,7 @@ function runMultiS3BackupJob(array $failingStorages): array$backup = createMultiS3Backup();$commands = collect();Process::fake(function ($process) use ($commands) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands->push($command);return Process::result(output: str_contains($command, 'du -b') ? '128' : '');@@ -321,15 +321,27 @@ describe('backup job', function () {$sshDisk = Storage::disk('ssh-keys');Storage::shouldReceive('disk')->with('ssh-keys')->andReturn($sshDisk);Storage::shouldReceive('build')->andReturn($disk);- Process::fake(fn ($process) => Process::result(output: str_contains($process->command, 'du -b') ? '128' : ''));+ $commands = collect();+ Process::fake(function ($process) use ($commands) {+ $commands->push(remoteProcessScript($process));++ return Process::result(output: str_contains(remoteProcessScript($process), 'du -b') ? '128' : '');+ });(new DatabaseBackupJob($backup))->handle();expect(ScheduledDatabaseBackupExecution::query()->sole()->s3_uploaded)->toBeTrue();- Process::assertRan(fn ($process) => str_contains($process->command, "-e 'MC_REGION=ru-central-1'")- && str_contains($process->command, 'temporary/first-bucket/'));- Process::assertRan(fn ($process) => str_contains($process->command, "-e 'MC_REGION=us-east-1'")- && str_contains($process->command, 'temporary/second-bucket/'));+ $regionOfBucket = [];+ $region = null;+ foreach ($commands as $command) {+ if (preg_match("/-e 'MC_REGION=([^']+)'/", $command, $match)) {+ $region = $match[1];+ }+ if (preg_match("#mc cp .*'temporary/([^/']+)/#", $command, $match)) {+ $regionOfBucket[$match[1]] = $region;+ }+ }+ expect($regionOfBucket)->toBe(['first-bucket' => 'ru-central-1', 'second-bucket' => 'us-east-1']);});it('keeps the local file and names the failed destination when one upload fails', function () {@@ -442,7 +454,7 @@ it('requires a live S3 copy for local retention only when local backups are disa}$commands = collect();Process::fake(function ($process) use ($commands) {- $commands->push(is_array($process->command) ? implode(' ', $process->command) : $process->command);+ $commands->push(is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process));return Process::result();});diff --git a/tests/Feature/DatabaseBackupPerDatabaseStateTest.php b/tests/Feature/DatabaseBackupPerDatabaseStateTest.phpindex 65821b03b5..ab80731f60 100644--- a/tests/Feature/DatabaseBackupPerDatabaseStateTest.php+++ b/tests/Feature/DatabaseBackupPerDatabaseStateTest.php@@ -54,7 +54,7 @@ it('keeps the successful execution of an earlier database when creating the next])->save();$commands = collect();Process::fake(function ($process) use ($commands) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands->push($command);return Process::result(output: str_contains($command, 'du -b') ? '128' : '');@@ -95,7 +95,7 @@ it('finishes the execution of every database in a run with several databases', f'team_id' => $team->id,])->save();Process::fake(function ($process) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);if (str_contains($command, 'pg_dump') && str_contains($command, 'second')) {return Process::result(errorOutput: 'pg_dump: database "second" does not exist', exitCode: 1);}diff --git a/tests/Feature/DatabaseBackupUploadValidationTest.php b/tests/Feature/DatabaseBackupUploadValidationTest.phpindex 31589dc7f8..756795fbeb 100644--- a/tests/Feature/DatabaseBackupUploadValidationTest.php+++ b/tests/Feature/DatabaseBackupUploadValidationTest.php@@ -257,16 +257,42 @@ test('postgresql backup safety scanner allows copy from stdin data', function ()expect(DatabaseBackupFileValidator::containsPostgresqlProgramExecution($dump))->toBeFalse();});-// Decided trade-off: an unanchored "TO/FROM PROGRAM" match cannot tell a real COPY-// directive from the same two words in a data row, and we keep it unanchored so it-// also catches dynamic SQL. A dump whose data literally contains "from program" or-// "to program" is therefore blocked, with an error that points to the container-// terminal for trusted dumps. This is rare in practice.-test('postgresql backup safety scanner blocks the program token pair even inside table data', function () {- $dump = "COPY notes FROM stdin;\n1\tcopy files from program storage\n\\.\n";+// pg_dump writes table data as COPY ... FROM stdin rows that end with a "\." line. The rows are+// data, not SQL, so ordinary text in them must not block the restore. Statements outside such a+// block are still scanned, also when they follow one.+dataset('postgresql copy data rows', [+ 'program words' => ["COPY public.notes (id, body) FROM stdin;\n1\thow to program\n2\timported from program files\n3\tcopy files to program storage\n\\.\n"],+ 'large object call text' => ["COPY public.notes (id, body) FROM stdin;\n1\tcall lo_import(path) on the server\n\\.\n"],+ 'alter system text' => ["COPY public.notes (id, body) FROM stdin;\n1\trun ALTER SYSTEM SET work_mem then reload\n\\.\n"],+ 'escaped tab before backquotes' => ["COPY public.notes (id, body) FROM stdin;\n1\ttab\\t `backticks` here\n\\.\n"],+ 'null row before text' => ["COPY a (x) FROM stdin;\n\\N\nfrom program\n\\.\n"],+ 'several blocks' => ["COPY a (x) FROM stdin;\nfrom program\n\\.\n\nCOPY b (y) FROM stdin;\n\\.\nCOPY c (z) FROM stdin;\nto program\n\\.\nSELECT 1;\n"],+]);++dataset('postgresql statements around copy data', [+ 'program statement after a copy block' => ["COPY notes FROM stdin;\n1\tok\n\\.\nCOPY x FROM PROGRAM 'id';\n"],+ 'shell command after a copy block' => ["COPY notes FROM stdin;\n1\tok\n\\.\n\\! id\n"],+ 'copy header that is not alone on its line' => ["SELECT 1; COPY notes FROM stdin;\nCOPY x FROM PROGRAM 'id';\n\\.\n"],+ 'program statement before a copy block' => ["COPY x TO PROGRAM 'id';\nCOPY notes FROM stdin;\n1\tok\n\\.\n"],+ 'commented-out program statement' => ["-- COPY x FROM PROGRAM 'id'\nSELECT 1;\n"],+ 'program statement in a block comment' => ["/* COPY x TO PROGRAM 'id' */\nSELECT 1;\n"],+]);++test('postgresql backup safety scanner allows text in copy data rows', function (string $dump) {+ expect(DatabaseBackupFileValidator::containsPostgresqlProgramExecution($dump))->toBeFalse();+})->with('postgresql copy data rows');+test('remote postgresql scanner allows text in copy data rows', function (string $dump) {+ expect(scannerBlocks(postgresScanScript(writeScanPayload($dump))))->toBeFalse();+})->with('postgresql copy data rows');++test('postgresql backup safety scanner blocks statements around copy data', function (string $dump) {expect(DatabaseBackupFileValidator::containsPostgresqlProgramExecution($dump))->toBeTrue();-});+})->with('postgresql statements around copy data');++test('remote postgresql scanner blocks statements around copy data', function (string $dump) {+ expect(scannerBlocks(postgresScanScript(writeScanPayload($dump))))->toBeTrue();+})->with('postgresql statements around copy data');test('postgresql backup safety scanner blocks program execution hidden in dynamic sql', function (string $payload) {expect(DatabaseBackupFileValidator::containsPostgresqlProgramExecution($payload))->toBeTrue();@@ -281,10 +307,13 @@ test('postgresql backup safety scanner blocks program execution hidden in dynami'mid-statement psql shell' => ["SELECT 1; \\! id\n"],]);-test('backup validator rejects plaintext .dump containing program execution', function () {- $file = makeTemporaryUpload('evil.dump', "COPY x FROM PROGRAM 'id';\n");+// The upload does not know the target database type, so it does not run the PostgreSQL scan:+// a MySQL dump can hold the same words. A PostgreSQL import scans the whole file when it starts.+test('backup validator accepts SQL uploads that the PostgreSQL scan would block', function () {+ $file = makeTemporaryUpload('mysql.sql', "INSERT INTO `notes` VALUES (1,'imported from program files'),(2,'\\! not a command');\n");- expect(DatabaseBackupFileValidator::isUploadAllowed($file, 10 * 1024 * 1024))->toBeFalse();+ expect(DatabaseBackupFileValidator::isUploadAllowed($file, 10 * 1024 * 1024))->toBeTrue()+ ->and(DatabaseBackupFileValidator::fileContainsPostgresqlProgramExecution($file->getPathname()))->toBeTrue();});test('remote postgresql scanner blocks bypass payloads', function (string $content, bool $gzip) {@@ -320,7 +349,6 @@ test('remote postgresql scanner blocks bypass payloads', function (string $conte'large object import' => ["SELECT lo_import('/etc/passwd');\n", false],'alter system' => ["ALTER SYSTEM SET archive_command = 'id';\n", false],'commented out payload is now blocked' => ["-- COPY x FROM PROGRAM 'id'\nSELECT 1;\n", false],- 'program words in table data are blocked' => ["COPY notes FROM stdin;\n1\tcopy files from program storage\n\\.\n", false],]);test('remote postgresql scanner allows legitimate restores', function (string $content, bool $gzip) {diff --git a/tests/Feature/DatabaseDumpExitStatusTest.php b/tests/Feature/DatabaseDumpExitStatusTest.phpindex 7a9382a74b..b9a867acd8 100644--- a/tests/Feature/DatabaseDumpExitStatusTest.php+++ b/tests/Feature/DatabaseDumpExitStatusTest.php@@ -90,7 +90,7 @@ it('records a failed SQLite dump as failed and removes the partial backup file',]);$commands = collect();Process::fake(function ($process) use ($commands) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands->push($command);return str_contains($command, 'VACUUM INTO')diff --git a/tests/Feature/DatabaseFlushTest.php b/tests/Feature/DatabaseFlushTest.phpindex 476aed154d..072a8e4e64 100644--- a/tests/Feature/DatabaseFlushTest.php+++ b/tests/Feature/DatabaseFlushTest.php@@ -63,7 +63,7 @@ it('flushes every key with the engine CLI', function (string $model, array $attr->assertReturned(true)->assertDispatched('success');- Process::assertRan(fn ($process) => str_contains($process->command, "sudo bash -c 'docker exec {$database->uuid} sh -c '\\''REDISCLI_AUTH=\"\$REDIS_PASSWORD\" {$cli} FLUSHALL ASYNC'\\'''"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "sudo bash -c 'docker exec {$database->uuid} sh -c '\\''REDISCLI_AUTH=\"\$REDIS_PASSWORD\" {$cli} FLUSHALL ASYNC'\\'''"));expect(AuditEvent::query()->where('event', 'ui.database.flushed')->where('resource_uuid', $database->uuid)->exists())->toBeTrue();})->with(['redis' => [StandaloneRedis::class, [], 'redis-cli'],diff --git a/tests/Feature/DatabaseImportInterruptedCleanupTest.php b/tests/Feature/DatabaseImportInterruptedCleanupTest.phpindex f82b8cc438..baabb45a09 100644--- a/tests/Feature/DatabaseImportInterruptedCleanupTest.php+++ b/tests/Feature/DatabaseImportInterruptedCleanupTest.php@@ -205,9 +205,9 @@ test('the cleanup runs at most once per import', function () {$listener->handle(new DatabaseImportFinished($cleanup));$listener->handle(new DatabaseImportFinished([...$cleanup, 'stopRestore' => true]));- Process::assertRanTimes(fn ($process) => str_contains($process->command, "s3-restore-{$operation}"), 1);- Process::assertRan(fn ($process) => str_contains($process->command, "/tmp/restore_{$operation}")- && str_contains($process->command, 'Stopping the database import processes'));+ Process::assertRanTimes(fn ($process) => str_contains(remoteProcessScript($process), "s3-restore-{$operation}"), 1);+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "/tmp/restore_{$operation}")+ && str_contains(remoteProcessScript($process), 'Stopping the database import processes'));});test('the cleanup of an import without an operation id still runs every time', function () {@@ -218,7 +218,7 @@ test('the cleanup of an import without an operation id still runs every time', f$listener->handle(new DatabaseImportFinished($cleanup));$listener->handle(new DatabaseImportFinished($cleanup));- Process::assertRanTimes(fn ($process) => str_contains($process->command, $cleanup['containerName']), 2);+ Process::assertRanTimes(fn ($process) => str_contains(remoteProcessScript($process), $cleanup['containerName']), 2);});it('keeps the stop message when the stopped import process ends afterwards', function () {@@ -277,7 +277,7 @@ it('keeps the stop message when the stopped import job fails', function () {test('an import task may run longer than the SSH command timeout while other remote tasks keep the default timeout', function (int $commandTimeout) {config()->set('constants.ssh.command_timeout', $commandTimeout);Process::swap(new ProcessFactory);- Process::fake([+ fakeRemoteProcesses(['*stat -c %s*' => Process::result('2048'),'*od -An -tx1*' => Process::result(' 2d 2d 20 50 6f 73'),'*' => Process::result(''),@@ -314,8 +314,8 @@ test('a failed import task stops the remote restore and blocks the database unti(new CleanupDatabaseImport)->handle(new DatabaseImportFinished([...$cleanup, 'stopRestore' => true]));- Process::assertRan(fn ($process) => str_contains($process->command, "/tmp/restore_{$operation}")- && str_contains($process->command, 'Stopping the database import processes'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "/tmp/restore_{$operation}")+ && str_contains(remoteProcessScript($process), 'Stopping the database import processes'));$activity->refresh();expect(data_get($activity, 'properties.status'))->toBe(ProcessStatus::ERROR->value)->and(data_get($activity, 'properties.error'))->toBe('App\\Jobs\\CoolifyTask has timed out. '.ResourceStartActivity::IMPORT_STOPPED_MESSAGE)@@ -379,7 +379,7 @@ test('an import whose SSH command ends without the restore result stops the restProcess::fake();(new CleanupDatabaseImport)->handle(new DatabaseImportFinished([...$cleanup, 'stopRestore' => true]));- Process::assertRan(fn ($process) => str_contains($process->command, 'Stopping the database import processes'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'Stopping the database import processes'));$activity->refresh();expect(data_get($activity, 'properties.status'))->toBe(ProcessStatus::ERROR->value)->and(data_get($activity, 'properties.error'))->toBe("{$reason} ".ResourceStartActivity::IMPORT_STOPPED_MESSAGE)@@ -426,7 +426,7 @@ test('a restore that exits like a lost SSH connection or timeout reports a norma$this->server->update(['user' => $user]);Server::flushIdentityMap();Process::swap(new ProcessFactory);- Process::fake([+ fakeRemoteProcesses(['*stat -c %s*' => Process::result('2048'),'*od -An -tx1*' => Process::result(' 2d 2d 20 50 6f 73'),'*' => Process::result(''),diff --git a/tests/Feature/DatabaseImportRestoreOptionsTest.php b/tests/Feature/DatabaseImportRestoreOptionsTest.phpindex bb254d96e0..64e47c021d 100644--- a/tests/Feature/DatabaseImportRestoreOptionsTest.php+++ b/tests/Feature/DatabaseImportRestoreOptionsTest.php@@ -70,7 +70,7 @@ beforeEach(function () {*/function importRestoreScript(Model $database, DatabaseImportSource $source): string{- Process::fake([+ fakeRemoteProcesses(['*stat -c %s*' => Process::result('2048'),'*od -An -tx1*' => Process::result(' 1f 8b 08 00 00 00'),'*' => Process::result(''),diff --git a/tests/Feature/DatabaseStartContentFilesTest.php b/tests/Feature/DatabaseStartContentFilesTest.phpindex 88de7023b8..c3be8071d8 100644--- a/tests/Feature/DatabaseStartContentFilesTest.php+++ b/tests/Feature/DatabaseStartContentFilesTest.php@@ -30,7 +30,7 @@ beforeEach(function () {// Every path on the server is missing, and every path is inside the resource directory.Process::fake(function ($process) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);if (str_contains($command, 'empty-directory')) {return Process::result(output: collect(range(1, substr_count($command, '/ssl/server.')))->map(fn (int $index) => "{$index}:missing")->implode("\n"));}@@ -80,7 +80,7 @@ it('writes a new SSL certificate file on the server before the container starts'foreach ($files as $file) {$path = escapeshellarg($database->workdir()."/ssl/{$file}");- Process::assertRan(fn ($process) => str_contains($process->command, "base64 -d | tee {$path}"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "base64 -d | tee {$path}"));}$commands = implode("\n", $this->executor->commands);expect($commands)->toContain('missing configuration file')diff --git a/tests/Feature/DeleteResourceJobAtomicityTest.php b/tests/Feature/DeleteResourceJobAtomicityTest.phpindex a23ed15c42..04b90e3950 100644--- a/tests/Feature/DeleteResourceJobAtomicityTest.php+++ b/tests/Feature/DeleteResourceJobAtomicityTest.php@@ -189,7 +189,7 @@ it('removes service containers before its volumes and local metadata', function$service->server->settings()->update(['is_reachable' => true, 'is_usable' => true]);$commands = collect();Process::fake(function ($process) use ($commands) {- $commands->push($process->command);+ $commands->push(remoteProcessScript($process));return Process::result(output: '');});@@ -227,7 +227,7 @@ it('targets a service subresource container by its Docker labels', function () {$service->server->settings()->update(['is_reachable' => true, 'is_usable' => true]);$commands = collect();Process::fake(function ($process) use ($commands) {- $commands->push($process->command);+ $commands->push(remoteProcessScript($process));return Process::result(output: '');});@@ -260,7 +260,7 @@ it('removes only the database container when an application of the same service$service->server->settings()->update(['is_reachable' => true, 'is_usable' => true]);$commands = collect();Process::fake(function ($process) use ($commands) {- $commands->push($process->command);+ $commands->push(remoteProcessScript($process));return Process::result(output: '');});@@ -347,7 +347,7 @@ it('removes service containers with a command that works for non-root SSH users'$service->server->settings()->update(['is_reachable' => true, 'is_usable' => true]);$commands = collect();Process::fake(function ($process) use ($commands) {- $commands->push($process->command);+ $commands->push(remoteProcessScript($process));return Process::result(output: '');});diff --git a/tests/Feature/DeleteServiceRemoteCleanupTest.php b/tests/Feature/DeleteServiceRemoteCleanupTest.phpindex 52affebdfe..bb98cdd4a9 100644--- a/tests/Feature/DeleteServiceRemoteCleanupTest.php+++ b/tests/Feature/DeleteServiceRemoteCleanupTest.php@@ -60,8 +60,8 @@ beforeEach(function () {it('removes the other volumes, networks and the configuration directory when one volume is in use', function () {$commands = collect();Process::fake(function ($process) use ($commands) {- $commands->push($process->command);- if (str_contains($process->command, "docker volume rm -f '{$this->service->uuid}_web-data'")) {+ $commands->push(remoteProcessScript($process));+ if (str_contains(remoteProcessScript($process), "docker volume rm -f '{$this->service->uuid}_web-data'")) {return Process::result(errorOutput: "Error response from daemon: remove {$this->service->uuid}_web-data: volume is in use", exitCode: 1);}diff --git a/tests/Feature/DeploymentCancellationCleanupTest.php b/tests/Feature/DeploymentCancellationCleanupTest.phpindex 0721fbc4d2..4b68e76dd5 100644--- a/tests/Feature/DeploymentCancellationCleanupTest.php+++ b/tests/Feature/DeploymentCancellationCleanupTest.php@@ -68,7 +68,7 @@ beforeEach(function () {$this->remoteCommands = [];Process::fake(function (PendingProcess $process) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$this->remoteCommands[] = $command;return Process::result(str_contains($command, 'docker ps -a') ? 'cancel-cleanup-uuid' : '');diff --git a/tests/Feature/DeploymentConfigurationSymlinkWriteTest.php b/tests/Feature/DeploymentConfigurationSymlinkWriteTest.phpindex ff654dd00a..a08f90fe54 100644--- a/tests/Feature/DeploymentConfigurationSymlinkWriteTest.php+++ b/tests/Feature/DeploymentConfigurationSymlinkWriteTest.php@@ -58,8 +58,10 @@ beforeEach(function () {SH;Process::fake(function (PendingProcess $process) use ($shellPrelude) {$command = $process->command;- if (preg_match('~<< \\\\(\S+)\n(.*)\n\1$~s', $command, $matches)) {- $script = $matches[2];+ $input = $process->input;+ if (str_ends_with($command, 'exec sh -se; fi"\'')) {+ $script = $input;+ $input = null;} elseif (preg_match("~ '((?:[^']|'\\\\'')*)'$~s", $command, $matches)) {$script = str_replace("'\\''", "'", $matches[1]);} else {@@ -67,7 +69,7 @@ beforeEach(function () {}$shell = new SymfonyProcess(['bash', '-c', $shellPrelude."\nset -e\n".$script]);- $shell->setInput($process->input);+ $shell->setInput($input);$shell->run();return Process::result(output: $shell->getOutput(), errorOutput: $shell->getErrorOutput(), exitCode: $shell->getExitCode());diff --git a/tests/Feature/DeploymentSensitiveCommandFailureTest.php b/tests/Feature/DeploymentSensitiveCommandFailureTest.phpindex e8924c758e..9e2b99e61a 100644--- a/tests/Feature/DeploymentSensitiveCommandFailureTest.php+++ b/tests/Feature/DeploymentSensitiveCommandFailureTest.php@@ -166,7 +166,7 @@ it('hides the .env write command and its base64 payload when the write fails', f$payloads = [];Process::fake(function (PendingProcess $process) use (&$payloads) {- $payloads = array_merge($payloads, base64PayloadsInCommand($process->command));+ $payloads = array_merge($payloads, base64PayloadsInCommand(remoteProcessScript($process)));return Process::result(errorOutput: 'tee: /artifacts/sensitive-failure/.env: No space left on device', exitCode: 1);});@@ -199,7 +199,7 @@ it('redacts the sensitive payload when the error output echoes the command', fun$payloads = [];Process::fake(function (PendingProcess $process) use (&$payloads) {- $commandPayloads = base64PayloadsInCommand($process->command);+ $commandPayloads = base64PayloadsInCommand(remoteProcessScript($process));$payloads = array_merge($payloads, $commandPayloads);return Process::result(errorOutput: "bash: line 1: echo '".($commandPayloads[0] ?? '')."': unexpected failure", exitCode: 2);@@ -223,7 +223,7 @@ it('hides the build-time env write command when the write fails', function () {$payloads = [];Process::fake(function (PendingProcess $process) use (&$payloads) {- $payloads = array_merge($payloads, base64PayloadsInCommand($process->command));+ $payloads = array_merge($payloads, base64PayloadsInCommand(remoteProcessScript($process)));return Process::result(errorOutput: 'tee: /artifacts/build-time.env: No such file or directory', exitCode: 1);});@@ -308,9 +308,9 @@ it('logs only the variable names of the .env files in development mode', functio$ranCommands = [];Process::fake(function (PendingProcess $process) use (&$ranCommands) {- $ranCommands[] = $process->command;+ $ranCommands[] = remoteProcessScript($process);- return Process::result(output: str_contains($process->command, 'cat ') ? 'APP_SECRET='.SENSITIVE_FAILURE_SECRET : '');+ return Process::result(output: str_contains(remoteProcessScript($process), 'cat ') ? 'APP_SECRET='.SENSITIVE_FAILURE_SECRET : '');});invokeSensitiveFailureJobMethod($job, $reflection, 'save_runtime_environment_variables');@@ -353,11 +353,11 @@ it('hides the helper container command when build secrets are passed as environm]);Process::fake(function (PendingProcess $process) {- if (str_contains($process->command, 'docker run -d')) {+ if (str_contains(remoteProcessScript($process), 'docker run -d')) {return Process::result(errorOutput: 'docker: Error response from daemon: network not found.', exitCode: 125);}- return Process::result(output: str_contains($process->command, 'echo $HOME') ? '/root' : 'NOK');+ return Process::result(output: str_contains(remoteProcessScript($process), 'echo $HOME') ? '/root' : 'NOK');});$exception = captureDeploymentException(fn () => invokeSensitiveFailureJobMethod($job, $reflection, 'prepare_builder_image'));@@ -381,11 +381,11 @@ it('hides the Railpack prepare command that passes build-time values', function]);Process::fake(function (PendingProcess $process) {- if (str_contains($process->command, 'railpack prepare')) {+ if (str_contains(remoteProcessScript($process), 'railpack prepare')) {return Process::result(errorOutput: 'railpack: failed to detect a provider', exitCode: 1);}- return Process::result(output: str_contains($process->command, 'buildx version') ? 'available' : 'missing');+ return Process::result(output: str_contains(remoteProcessScript($process), 'buildx version') ? 'available' : 'missing');});$exception = captureDeploymentException(fn () => invokeSensitiveFailureJobMethod($job, $reflection, 'build_railpack_image'));@@ -412,7 +412,7 @@ it('hides the Nixpacks plan write command that contains build-time values', func$payloads = [];Process::fake(function (PendingProcess $process) use (&$payloads) {- $commandPayloads = base64PayloadsInCommand($process->command);+ $commandPayloads = base64PayloadsInCommand(remoteProcessScript($process));if ($commandPayloads !== []) {$payloads = array_merge($payloads, $commandPayloads);@@ -567,7 +567,7 @@ it('tells how to log in when the build server has no Docker config for the SSH u$server = readSensitiveFailureJobProperty($job, $reflection, 'server');$server->forceFill(['user' => 'coolify', 'name' => 'build-1'])->save();setSensitiveFailureJobProperties($job, $reflection, ['server' => $server->fresh(), 'use_build_server' => true]);- Process::fake(fn (PendingProcess $process) => Process::result(output: str_contains($process->command, 'echo $HOME') ? '/home/coolify' : 'NOK'));+ Process::fake(fn (PendingProcess $process) => Process::result(output: str_contains(remoteProcessScript($process), 'echo $HOME') ? '/home/coolify' : 'NOK'));$exception = captureDeploymentException(fn () => invokeSensitiveFailureJobMethod($job, $reflection, 'prepare_builder_image'));@@ -575,7 +575,7 @@ it('tells how to log in when the build server has no Docker config for the SSH u->toContain('of the SSH user (coolify) not found on the build server (build-1)')->toContain('Server → Docker Registries')->toContain('"sudo docker login" is saved for root');- Process::assertNotRan(fn (PendingProcess $process) => str_contains($process->command, 'docker run -d'));+ Process::assertNotRan(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), 'docker run -d'));});it('mounts the seccomp profile from custom docker options into the helper container', function () {@@ -593,13 +593,13 @@ it('mounts the seccomp profile from custom docker options into the helper contai$helperCommands = [];Process::fake(function (PendingProcess $process) use (&$helperCommands) {- if (str_contains($process->command, 'docker run -d')) {- $helperCommands[] = $process->command;+ if (str_contains(remoteProcessScript($process), 'docker run -d')) {+ $helperCommands[] = remoteProcessScript($process);return Process::result(errorOutput: 'docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /etc/docker/seccomp-profile.json', exitCode: 125);}- return Process::result(output: str_contains($process->command, 'echo $HOME') ? '/root' : 'NOK');+ return Process::result(output: str_contains(remoteProcessScript($process), 'echo $HOME') ? '/root' : 'NOK');});$exception = captureDeploymentException(fn () => invokeSensitiveFailureJobMethod($job, $reflection, 'prepare_builder_image'));diff --git a/tests/Feature/DevHostDockerPathTest.php b/tests/Feature/DevHostDockerPathTest.phpindex 484264c461..fec0273e2d 100644--- a/tests/Feature/DevHostDockerPathTest.php+++ b/tests/Feature/DevHostDockerPathTest.php@@ -60,7 +60,7 @@ beforeEach(function () {$this->commands = [];Process::fake(function ($process) {- $this->commands[] = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $this->commands[] = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: '');});diff --git a/tests/Feature/DevelopmentQemuVmTest.php b/tests/Feature/DevelopmentQemuVmTest.phpindex 180c861b12..68c43ba143 100644--- a/tests/Feature/DevelopmentQemuVmTest.php+++ b/tests/Feature/DevelopmentQemuVmTest.php@@ -83,19 +83,19 @@ it('stores vm disks in a libvirt-accessible directory', function () {it('automatically configures the qemu host', function () {Process::fake(function ($process) {- if (str_contains($process->command, 'command -v')) {+ if (str_contains(remoteProcessScript($process), 'command -v')) {return Process::result();}- if (str_contains($process->command, 'net-info')) {+ if (str_contains(remoteProcessScript($process), 'net-info')) {return Process::result(exitCode: 1);}- if (str_contains($process->command, 'network inspect')) {+ if (str_contains(remoteProcessScript($process), 'network inspect')) {return Process::result(output: "172.18.0.0/16\n");}- if (str_contains($process->command, 'iptables -C')) {+ if (str_contains(remoteProcessScript($process), 'iptables -C')) {return Process::result(exitCode: 1);}@@ -104,22 +104,22 @@ it('automatically configures the qemu host', function () {ConfigureDevelopmentQemuHost::run();- Process::assertRan(fn ($process) => str_contains($process->command, 'command -v') && str_contains($process->command, 'xorriso'));- Process::assertRan(fn ($process) => str_contains($process->command, 'systemctl enable --now libvirtd'));- Process::assertRan(fn ($process) => str_contains($process->command, 'virsh net-define'));- Process::assertRan(fn ($process) => str_contains($process->command, 'virsh net-start'));- Process::assertRan(fn ($process) => str_contains($process->command, 'virsh net-autostart'));- Process::assertRan(fn ($process) => str_contains($process->command, 'sysctl -w net.ipv4.ip_forward=1'));- Process::assertRan(fn ($process) => str_contains($process->command, 'iptables -I LIBVIRT_FWI'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'command -v') && str_contains(remoteProcessScript($process), 'xorriso'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'systemctl enable --now libvirtd'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh net-define'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh net-start'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh net-autostart'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'sysctl -w net.ipv4.ip_forward=1'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'iptables -I LIBVIRT_FWI'));});it('does not restart an active libvirt network', function () {Process::fake(function ($process) {- if (str_contains($process->command, 'net-info')) {+ if (str_contains(remoteProcessScript($process), 'net-info')) {return Process::result(output: "Name: default\nActive: yes\n");}- if (str_contains($process->command, 'network inspect')) {+ if (str_contains(remoteProcessScript($process), 'network inspect')) {return Process::result(output: "172.18.0.0/16\n");}@@ -128,9 +128,9 @@ it('does not restart an active libvirt network', function () {ConfigureDevelopmentQemuHost::run();- Process::assertNotRan(fn ($process) => str_contains($process->command, 'virsh net-start'));- Process::assertRan(fn ($process) => str_contains($process->command, 'iptables -D LIBVIRT_FWI'));- Process::assertRan(fn ($process) => str_contains($process->command, 'iptables -I LIBVIRT_FWI 1'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh net-start'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'iptables -D LIBVIRT_FWI'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'iptables -I LIBVIRT_FWI 1'));});it('seeds one predefined root qemu server', function () {@@ -196,8 +196,8 @@ it('seeds localhost through the container command', function () {]))->toBe(Command::SUCCESS)->and(Server::query()->findOrFail(0)->ip)->toBe('192.168.122.10');- Process::assertRan(fn ($process) => str_contains($process->command, 'docker run -d --name coolify-db')- && str_contains($process->command, 'POSTGRES_DB='));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker run -d --name coolify-db')+ && str_contains(remoteProcessScript($process), 'POSTGRES_DB='));});it('starts the instance database container with sudo on a non-root localhost vm', function () {@@ -206,9 +206,9 @@ it('starts the instance database container with sudo on a non-root localhost vm'Artisan::call('dev:qemu:seed', ['profile' => 'ubuntu-non-root', '--as-localhost' => true]);- Process::assertRan(fn ($process) => str_contains($process->command, 'if ! sudo docker inspect coolify-db')- && str_contains($process->command, 'sudo docker run -d --name coolify-db')- && str_contains($process->command, 'sudo docker start coolify-db'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'if ! sudo docker inspect coolify-db')+ && str_contains(remoteProcessScript($process), 'sudo docker run -d --name coolify-db')+ && str_contains(remoteProcessScript($process), 'sudo docker start coolify-db'));});it('still seeds localhost when the instance database container cannot start', function () {@@ -257,13 +257,13 @@ it('reuses an existing vm without touching other managed vms', function () {$ran = [];Process::fake(function ($process) use (&$ran) {- $ran[] = $process->command;+ $ran[] = remoteProcessScript($process);return match (true) {- str_contains($process->command, ' net-dumpxml ') => Process::result(output: '<network></network>'),- str_contains($process->command, ' network inspect ') => Process::result(output: "172.18.0.0/16\n"),- str_contains($process->command, ' dominfo ') => Process::result(output: 'exists'),- str_contains($process->command, ' domstate ') => Process::result(output: "shut off\n"),+ str_contains(remoteProcessScript($process), ' net-dumpxml ') => Process::result(output: '<network></network>'),+ str_contains(remoteProcessScript($process), ' network inspect ') => Process::result(output: "172.18.0.0/16\n"),+ str_contains(remoteProcessScript($process), ' dominfo ') => Process::result(output: 'exists'),+ str_contains(remoteProcessScript($process), ' domstate ') => Process::result(output: "shut off\n"),default => Process::result(),};});@@ -275,12 +275,12 @@ it('reuses an existing vm without touching other managed vms', function () {expect($resize)->not->toBeFalse()->and($commands->search("virsh setmem 'coolify-dev-ubuntu-non-root' 1024M --config"))->toBeGreaterThan($resize)->and($commands->search("virsh start 'coolify-dev-ubuntu-non-root'"))->toBeGreaterThan($resize);- Process::assertNotRan(fn ($process) => str_starts_with($process->command, 'virt-install '));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'virsh destroy'));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'virsh undefine'));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'coolify-dev-ubuntu-root'));- Process::assertRan(fn ($process) => str_contains($process->command, 'iptables -I LIBVIRT_FWI'));- Process::assertRan(fn ($process) => str_contains($process->command, 'docker exec') && str_contains($process->command, 'coolify'));+ Process::assertNotRan(fn ($process) => str_starts_with(remoteProcessScript($process), 'virt-install '));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh destroy'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh undefine'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'coolify-dev-ubuntu-root'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'iptables -I LIBVIRT_FWI'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker exec') && str_contains(remoteProcessScript($process), 'coolify'));expect(File::get("{$storagePath}/coolify-dev-ubuntu-root.qcow2"))->toBe('other vm data')->and(File::get("{$storagePath}/coolify-dev-ubuntu-non-root.qcow2"))->toBe('vm data');});@@ -297,10 +297,10 @@ it('does not restart an already running vm', function () {StartDevelopmentQemuVm::run('ubuntu-root');- Process::assertNotRan(fn ($process) => str_contains($process->command, 'virsh start'));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'virsh setmem'));- Process::assertNotRan(fn ($process) => str_starts_with($process->command, 'virt-install '));- Process::assertRan(fn ($process) => str_contains($process->command, 'docker exec') && str_contains($process->command, 'fsockopen'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh start'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh setmem'));+ Process::assertNotRan(fn ($process) => str_starts_with(remoteProcessScript($process), 'virt-install '));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker exec') && str_contains(remoteProcessScript($process), 'fsockopen'));});it('freshly recreates only the selected vm', function () {@@ -320,12 +320,12 @@ it('freshly recreates only the selected vm', function () {StartDevelopmentQemuVm::run('ubuntu-non-root', true);- Process::assertRan(fn ($process) => $process->command === "virsh destroy 'coolify-dev-ubuntu-non-root'");- Process::assertRan(fn ($process) => $process->command === "virsh undefine 'coolify-dev-ubuntu-non-root'");- Process::assertNotRan(fn ($process) => str_contains($process->command, 'coolify-dev-ubuntu-root'));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'virsh start'));- Process::assertRan(fn ($process) => str_contains($process->command, 'virt-install') && str_contains($process->command, 'coolify-dev-ubuntu-non-root'));- Process::assertRan(fn ($process) => str_contains($process->command, 'net-update') && str_contains($process->command, 'ip-dhcp-host') && str_contains($process->command, '192.168.122.11'));+ Process::assertRan(fn ($process) => remoteProcessScript($process) === "virsh destroy 'coolify-dev-ubuntu-non-root'");+ Process::assertRan(fn ($process) => remoteProcessScript($process) === "virsh undefine 'coolify-dev-ubuntu-non-root'");+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'coolify-dev-ubuntu-root'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh start'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virt-install') && str_contains(remoteProcessScript($process), 'coolify-dev-ubuntu-non-root'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'net-update') && str_contains(remoteProcessScript($process), 'ip-dhcp-host') && str_contains(remoteProcessScript($process), '192.168.122.11'));expect(File::get("{$storagePath}/coolify-dev-ubuntu-root.qcow2"))->toBe('other vm data')->and(File::exists("{$storagePath}/coolify-dev-ubuntu-non-root.qcow2"))->toBeFalse()->and(File::exists("{$storagePath}/coolify-dev-ubuntu-non-root-prepared.qcow2"))->toBeTrue();@@ -342,9 +342,9 @@ it('passes the fresh option from the command to the selected vms only', functionexpect(Artisan::call('dev:qemu', ['profiles' => ['debian-root'], '--fresh' => true]))->toBe(Command::SUCCESS);- Process::assertRan(fn ($process) => $process->command === "virsh undefine 'coolify-dev-debian-root'");- Process::assertRan(fn ($process) => str_contains($process->command, 'virt-install') && str_contains($process->command, 'coolify-dev-debian-root'));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'virsh destroy') && ! str_contains($process->command, 'coolify-dev-debian-root'));+ Process::assertRan(fn ($process) => remoteProcessScript($process) === "virsh undefine 'coolify-dev-debian-root'");+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virt-install') && str_contains(remoteProcessScript($process), 'coolify-dev-debian-root'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh destroy') && ! str_contains(remoteProcessScript($process), 'coolify-dev-debian-root'));});it('rejects qemu vm management outside development', function () {@@ -360,19 +360,19 @@ it('can create a vm without a host database connection', function () {]);DB::enableQueryLog();Process::fake(function ($process) {- if (str_contains($process->command, 'net-dumpxml')) {+ if (str_contains(remoteProcessScript($process), 'net-dumpxml')) {return Process::result(output: '<network></network>');}- if (str_contains($process->command, 'network inspect')) {+ if (str_contains(remoteProcessScript($process), 'network inspect')) {return Process::result(output: "172.18.0.0/16\n");}- if (str_contains($process->command, 'virsh dominfo')) {+ if (str_contains(remoteProcessScript($process), 'virsh dominfo')) {return Process::result(exitCode: 1);}- if (str_contains($process->command, 'iptables -C')) {+ if (str_contains(remoteProcessScript($process), 'iptables -C')) {return Process::result(exitCode: 1);}@@ -382,14 +382,14 @@ it('can create a vm without a host database connection', function () {StartDevelopmentQemuVm::run('ubuntu-root');expect(DB::getQueryLog())->toBeEmpty();- Process::assertRan(fn ($process) => str_contains($process->command, 'virt-install'));- Process::assertRan(fn ($process) => str_contains($process->command, 'iptables -I LIBVIRT_FWI'));- Process::assertRan(fn ($process) => str_contains($process->command, 'virsh domstate') && str_contains($process->command, 'shut off'));- Process::assertRan(fn ($process) => str_contains($process->command, 'mv ') && str_contains($process->command, 'prepared.qcow2'));- Process::assertRan(fn ($process) => str_contains($process->command, 'qemu-img create') && str_contains($process->command, 'prepared.qcow2'));- Process::assertRan(fn ($process) => str_contains($process->command, 'xorriso -as mkisofs -V cidata -graft-points'));- Process::assertRan(fn ($process) => str_contains($process->command, 'virt-install') && str_contains($process->command, 'bus=virtio,readonly=on'));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'mac_in_use'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virt-install'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'iptables -I LIBVIRT_FWI'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh domstate') && str_contains(remoteProcessScript($process), 'shut off'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'mv ') && str_contains(remoteProcessScript($process), 'prepared.qcow2'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'qemu-img create') && str_contains(remoteProcessScript($process), 'prepared.qcow2'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'xorriso -as mkisofs -V cidata -graft-points'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virt-install') && str_contains(remoteProcessScript($process), 'bus=virtio,readonly=on'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'mac_in_use'));$userData = File::get(config('development-qemu.storage_path').'/coolify-dev-ubuntu-root-user-data.yaml');expect($userData)->toContain('docker compose version', 'docker buildx version', 'docker info', 'cloud-init.disabled');$cloudInit = Yaml::parse($userData);@@ -410,10 +410,10 @@ it('reuses a prepared vm image without reinstalling docker', function () {StartDevelopmentQemuVm::run('ubuntu-root');- Process::assertNotRan(fn ($process) => str_contains($process->command, 'curl --fail --location'));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'virsh domstate'));- Process::assertRan(fn ($process) => str_contains($process->command, 'qemu-img create') && str_contains($process->command, 'coolify-dev-ubuntu-root-prepared.qcow2'));- Process::assertRan(fn ($process) => str_contains($process->command, 'virt-install') && ! str_contains($process->command, '--cloud-init'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'curl --fail --location'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'virsh domstate'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'qemu-img create') && str_contains(remoteProcessScript($process), 'coolify-dev-ubuntu-root-prepared.qcow2'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virt-install') && ! str_contains(remoteProcessScript($process), '--cloud-init'));expect(File::exists("{$storagePath}/coolify-dev-ubuntu-root-prepared.qcow2"))->toBeTrue();});@@ -439,26 +439,26 @@ it('prepares alpine with the compose and buildx packages for a non-root user', fit('does not cache a vm when preparation does not finish', function () {config(['development-qemu.storage_path' => sys_get_temp_dir().'/coolify-qemu-failed-preparation-'.uniqid()]);Process::fake(function ($process) {- if (str_contains($process->command, 'net-dumpxml')) {+ if (str_contains(remoteProcessScript($process), 'net-dumpxml')) {return Process::result(output: '<network></network>');}- if (str_contains($process->command, 'network inspect')) {+ if (str_contains(remoteProcessScript($process), 'network inspect')) {return Process::result(output: "172.18.0.0/16\n");}- if (str_contains($process->command, 'virsh dominfo')) {+ if (str_contains(remoteProcessScript($process), 'virsh dominfo')) {return Process::result(exitCode: 1);}- return str_contains($process->command, 'timeout 900 bash')+ return str_contains(remoteProcessScript($process), 'timeout 900 bash')? Process::result(exitCode: 124): Process::result();});expect(fn () => StartDevelopmentQemuVm::run('ubuntu-root'))->toThrow(RuntimeException::class);- Process::assertNotRan(fn ($process) => str_contains($process->command, 'mv ') && str_contains($process->command, 'prepared.qcow2'));- Process::assertNotRan(fn ($process) => str_starts_with($process->command, 'virt-install ') && ! str_contains($process->command, 'readonly=on'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'mv ') && str_contains(remoteProcessScript($process), 'prepared.qcow2'));+ Process::assertNotRan(fn ($process) => str_starts_with(remoteProcessScript($process), 'virt-install ') && ! str_contains(remoteProcessScript($process), 'readonly=on'));});it('seeds through the coolify container when the host database is unavailable', function () {@@ -476,7 +476,7 @@ it('seeds through the coolify container when the host database is unavailable',ManageDevelopmentQemuVm::run('ubuntu-root');- Process::assertRan(fn ($process) => str_contains($process->command, "docker exec 'coolify' php artisan dev:qemu:seed") && str_contains($process->command, 'ubuntu-root'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker exec 'coolify' php artisan dev:qemu:seed") && str_contains(remoteProcessScript($process), 'ubuntu-root'));});it('passes localhost mode to the container when the host database is unavailable', function () {@@ -493,7 +493,7 @@ it('passes localhost mode to the container when the host database is unavailableManageDevelopmentQemuVm::run('ubuntu-root', true);- Process::assertRan(fn ($process) => str_contains($process->command, 'dev:qemu:seed') && str_contains($process->command, '--as-localhost'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'dev:qemu:seed') && str_contains(remoteProcessScript($process), '--as-localhost'));});it('starts and seeds root and non-root profiles together', function () {@@ -511,8 +511,8 @@ it('starts and seeds root and non-root profiles together', function () {'development-qemu-ubuntu-root','development-qemu-ubuntu-non-root',])->count())->toBe(2);- Process::assertRan(fn ($process) => str_contains($process->command, 'virt-install') && str_contains($process->command, 'coolify-dev-ubuntu-root'));- Process::assertRan(fn ($process) => str_contains($process->command, 'virt-install') && str_contains($process->command, 'coolify-dev-ubuntu-non-root'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virt-install') && str_contains(remoteProcessScript($process), 'coolify-dev-ubuntu-root'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'virt-install') && str_contains(remoteProcessScript($process), 'coolify-dev-ubuntu-non-root'));});it('seeds through the instance coolify container when the host database is unavailable', function () {@@ -532,9 +532,9 @@ it('seeds through the instance coolify container when the host database is unavaManageDevelopmentQemuVm::run('ubuntu-root');- Process::assertRan(fn ($process) => str_contains($process->command, "docker exec 'coolify-fix-deploy-env' php artisan dev:qemu:seed"));- Process::assertRan(fn ($process) => str_contains($process->command, "docker exec 'coolify-fix-deploy-env' php -r"));- Process::assertNotRan(fn ($process) => str_contains($process->command, "docker exec 'coolify' "));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker exec 'coolify-fix-deploy-env' php artisan dev:qemu:seed"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker exec 'coolify-fix-deploy-env' php -r"));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), "docker exec 'coolify' "));});it('resolves legacy values when no dev instance is configured', function () {@@ -600,12 +600,12 @@ it('configures the instance libvirt network and forwarding rule', function () {'<ip address="10.221.3.1" netmask="255.255.255.0">','<range start="10.221.3.2" end="10.221.3.254"/>',)->not->toContain('virbr0', '192.168.122');- Process::assertRan(fn ($process) => $process->command === "virsh net-info 'coolify-dev-3'");- Process::assertRan(fn ($process) => $process->command === "virsh net-start 'coolify-dev-3'");- Process::assertRan(fn ($process) => $process->command === "virsh net-autostart 'coolify-dev-3'");- Process::assertRan(fn ($process) => str_contains($process->command, "docker network inspect 'coolify-main'"));- Process::assertRan(fn ($process) => $process->command === "iptables -I LIBVIRT_FWI 1 -s '172.30.0.0/16' -d '10.221.3.0/24' -o 'cdevbr3' -j ACCEPT");- Process::assertNotRan(fn ($process) => str_contains($process->command, 'virbr0'));+ Process::assertRan(fn ($process) => remoteProcessScript($process) === "virsh net-info 'coolify-dev-3'");+ Process::assertRan(fn ($process) => remoteProcessScript($process) === "virsh net-start 'coolify-dev-3'");+ Process::assertRan(fn ($process) => remoteProcessScript($process) === "virsh net-autostart 'coolify-dev-3'");+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker network inspect 'coolify-main'"));+ Process::assertRan(fn ($process) => remoteProcessScript($process) === "iptables -I LIBVIRT_FWI 1 -s '172.30.0.0/16' -d '10.221.3.0/24' -o 'cdevbr3' -j ACCEPT");+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'virbr0'));});it('keeps the legacy libvirt network definition unchanged', function () {@@ -625,7 +625,7 @@ it('keeps the legacy libvirt network definition unchanged', function () {'<ip address="192.168.122.1" netmask="255.255.255.0">','<range start="192.168.122.2" end="192.168.122.254"/>',);- Process::assertRan(fn ($process) => $process->command === "iptables -I LIBVIRT_FWI 1 -s '172.18.0.0/16' -d '192.168.122.0/24' -o 'virbr0' -j ACCEPT");+ Process::assertRan(fn ($process) => remoteProcessScript($process) === "iptables -I LIBVIRT_FWI 1 -s '172.18.0.0/16' -d '192.168.122.0/24' -o 'virbr0' -j ACCEPT");});it('creates instance vms with instance domains, ips, and shared prepared images', function () {@@ -642,17 +642,17 @@ it('creates instance vms with instance domains, ips, and shared prepared images'StartDevelopmentQemuVm::run('ubuntu-root');- Process::assertRan(fn ($process) => $process->command === "virsh dominfo 'coolify-dev-main--ubuntu-root'");- Process::assertRan(fn ($process) => str_starts_with($process->command, "virsh net-update 'coolify-dev-3' add-last ip-dhcp-host") && str_contains($process->command, '52:54:00:ca:00:01') && str_contains($process->command, '10.221.3.10'));- Process::assertRan(fn ($process) => str_contains($process->command, 'qemu-img create')- && str_contains($process->command, "'{$storagePath}/coolify-dev-ubuntu-root-prepared.qcow2'")- && str_contains($process->command, "'{$storagePath}/coolify-dev-main--ubuntu-root.qcow2'"));- Process::assertRan(fn ($process) => str_starts_with($process->command, 'virt-install ')- && str_contains($process->command, "--name 'coolify-dev-main--ubuntu-root'")- && str_contains($process->command, "--network network='coolify-dev-3',model=virtio,mac='52:54:00:ca:00:01'")- && str_ends_with($process->command, '--check mac_in_use=off'));- Process::assertRan(fn ($process) => str_contains($process->command, "docker exec 'coolify-main' php -r") && str_contains($process->command, "'10.221.3.10'"));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'curl --fail --location'));+ Process::assertRan(fn ($process) => remoteProcessScript($process) === "virsh dominfo 'coolify-dev-main--ubuntu-root'");+ Process::assertRan(fn ($process) => str_starts_with(remoteProcessScript($process), "virsh net-update 'coolify-dev-3' add-last ip-dhcp-host") && str_contains(remoteProcessScript($process), '52:54:00:ca:00:01') && str_contains(remoteProcessScript($process), '10.221.3.10'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'qemu-img create')+ && str_contains(remoteProcessScript($process), "'{$storagePath}/coolify-dev-ubuntu-root-prepared.qcow2'")+ && str_contains(remoteProcessScript($process), "'{$storagePath}/coolify-dev-main--ubuntu-root.qcow2'"));+ Process::assertRan(fn ($process) => str_starts_with(remoteProcessScript($process), 'virt-install ')+ && str_contains(remoteProcessScript($process), "--name 'coolify-dev-main--ubuntu-root'")+ && str_contains(remoteProcessScript($process), "--network network='coolify-dev-3',model=virtio,mac='52:54:00:ca:00:01'")+ && str_ends_with(remoteProcessScript($process), '--check mac_in_use=off'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker exec 'coolify-main' php -r") && str_contains(remoteProcessScript($process), "'10.221.3.10'"));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'curl --fail --location'));});it('rejects invalid instance names and slots', function (string $instance, int $slot) {@@ -731,10 +731,10 @@ it('creates a vm with every function of the host nvidia gpu and more memory', fuStartDevelopmentQemuVm::run('ubuntu-root', true, true);- Process::assertRan(fn ($process) => $process->command === 'systemctl stop nvidia-persistenced');- Process::assertRan(fn ($process) => str_starts_with($process->command, 'virt-install ')- && str_contains($process->command, '--memory 4096 ')- && str_contains($process->command, "--hostdev 'pci_0000_01_00_0' --hostdev 'pci_0000_01_00_1'"));+ Process::assertRan(fn ($process) => remoteProcessScript($process) === 'systemctl stop nvidia-persistenced');+ Process::assertRan(fn ($process) => str_starts_with(remoteProcessScript($process), 'virt-install ')+ && str_contains(remoteProcessScript($process), '--memory 4096 ')+ && str_contains(remoteProcessScript($process), "--hostdev 'pci_0000_01_00_0' --hostdev 'pci_0000_01_00_1'"));});it('stops an existing vm to attach the host gpu and starts it with the gpu memory', function () {@@ -742,16 +742,16 @@ it('stops an existing vm to attach the host gpu and starts it with the gpu memor$hostdevAttached = false;$ran = [];Process::fake(function ($process) use (&$hostdevAttached, &$ran) {- $ran[] = $process->command;- $hostdevAttached = $hostdevAttached || str_starts_with($process->command, 'virt-xml ');+ $ran[] = remoteProcessScript($process);+ $hostdevAttached = $hostdevAttached || str_starts_with(remoteProcessScript($process), 'virt-xml ');return match (true) {- str_contains($process->command, ' net-dumpxml ') => Process::result(output: '<network></network>'),- str_contains($process->command, ' network inspect ') => Process::result(output: "172.18.0.0/16\n"),- str_contains($process->command, ' dominfo ') => Process::result(output: 'exists'),- str_contains($process->command, ' domstate ') => Process::result(output: $hostdevAttached ? "shut off\n" : "running\n"),- str_contains($process->command, ' dumpxml ') => Process::result(output: $hostdevAttached ? "<domain><hostdev mode='subsystem'/></domain>" : '<domain></domain>'),- str_starts_with($process->command, 'lspci ') => Process::result(output: "0000:01:00.0 \"0300\" \"10de\" \"2584\"\n0000:01:00.1 \"0403\" \"10de\" \"2291\"\n"),+ str_contains(remoteProcessScript($process), ' net-dumpxml ') => Process::result(output: '<network></network>'),+ str_contains(remoteProcessScript($process), ' network inspect ') => Process::result(output: "172.18.0.0/16\n"),+ str_contains(remoteProcessScript($process), ' dominfo ') => Process::result(output: 'exists'),+ str_contains(remoteProcessScript($process), ' domstate ') => Process::result(output: $hostdevAttached ? "shut off\n" : "running\n"),+ str_contains(remoteProcessScript($process), ' dumpxml ') => Process::result(output: $hostdevAttached ? "<domain><hostdev mode='subsystem'/></domain>" : '<domain></domain>'),+ str_starts_with(remoteProcessScript($process), 'lspci ') => Process::result(output: "0000:01:00.0 \"0300\" \"10de\" \"2584\"\n0000:01:00.1 \"0403\" \"10de\" \"2291\"\n"),default => Process::result(),};});@@ -766,7 +766,7 @@ it('stops an existing vm to attach the host gpu and starts it with the gpu memor->and($commands)->toContain("virt-xml 'coolify-dev-ubuntu-root' --add-device --hostdev 'pci_0000_01_00_1'")->and($commands->search("virsh setmem 'coolify-dev-ubuntu-root' 4096M --config"))->toBeGreaterThan($attach)->and($commands->search("virsh start 'coolify-dev-ubuntu-root'"))->toBeGreaterThan($commands->search('systemctl stop nvidia-persistenced'));- Process::assertNotRan(fn ($process) => str_starts_with($process->command, 'virt-install '));+ Process::assertNotRan(fn ($process) => str_starts_with(remoteProcessScript($process), 'virt-install '));});it('keeps the gpu memory of a gpu vm that starts without the gpu option', function () {@@ -777,8 +777,8 @@ it('keeps the gpu memory of a gpu vm that starts without the gpu option', functiProcess::assertRan("virsh setmem 'coolify-dev-ubuntu-root' 4096M --config");Process::assertRan('systemctl stop nvidia-persistenced');- Process::assertNotRan(fn ($process) => str_starts_with($process->command, 'virt-xml '));- Process::assertNotRan(fn ($process) => str_contains($process->command, '1024M'));+ Process::assertNotRan(fn ($process) => str_starts_with(remoteProcessScript($process), 'virt-xml '));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), '1024M'));});it('fails before it changes the vm when the host has no nvidia gpu', function () {@@ -793,7 +793,7 @@ it('fails before it changes the vm when the host has no nvidia gpu', function ()expect(fn () => StartDevelopmentQemuVm::run('ubuntu-root', false, true))->toThrow(RuntimeException::class, 'No NVIDIA GPU found');- Process::assertNotRan(fn ($process) => str_starts_with($process->command, 'virt-xml ') || str_contains($process->command, 'virsh start'));+ Process::assertNotRan(fn ($process) => str_starts_with(remoteProcessScript($process), 'virt-xml ') || str_contains(remoteProcessScript($process), 'virsh start'));});it('allows gpu passthrough only for one ubuntu root profile', function (array $profiles) {diff --git a/tests/Feature/DockerCleanupJobTest.php b/tests/Feature/DockerCleanupJobTest.phpindex d3c1b01b02..a12808c15b 100644--- a/tests/Feature/DockerCleanupJobTest.php+++ b/tests/Feature/DockerCleanupJobTest.php@@ -70,7 +70,7 @@ function dockerCleanupFakeProcesses(?Closure $onRun = null): Collection{$commands = collect();Process::fake(function ($process) use ($commands, $onRun) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands->push($command);if ($onRun) {$onRun($command);@@ -455,7 +455,7 @@ it('cleans after a stop only when the disk usage reaches the server threshold',$server->settings()->update(['docker_cleanup_threshold' => 80]);$commands = collect();Process::fake(function ($process) use ($commands, $diskUsage) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands->push($command);return Process::result(output: str_contains($command, 'df / --output=pcent') ? $diskUsage : '');@@ -500,7 +500,7 @@ it('scans application images with a fixed number of remote commands and keeps th$commands = collect();Process::fake(function ($process) use ($commands, $containers, $images) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands->push($command);return match (true) {@@ -662,7 +662,7 @@ it('keeps an application image name with a single quote inside the grep pattern'->and(explode("\n", $output))->toBe(['rmi postgres:16']);})->with(['root user' => false, 'non-root user' => true]);-function dockerCleanupDeployment(Server $server, string $status, string $column = 'server_id', ?Carbon $updatedAt = null): void+function dockerCleanupDeployment(Server $server, string $status, string $column = 'server_id', ?Carbon $updatedAt = null): ApplicationDeploymentQueue{$deployment = ApplicationDeploymentQueue::create(['application_id' => '1',@@ -673,22 +673,45 @@ function dockerCleanupDeployment(Server $server, string $status, string $columnif ($updatedAt) {$deployment->forceFill(['updated_at' => $updatedAt])->saveQuietly();}++ return $deployment;}-it('skips a forced scheduled cleanup while a deployment is queued or running on the server, so the build cache is not pruned', function (string $status, string $column) {+/**+ * @param Collection<int, string> $commands+ */+function dockerCleanupPrunedBuildCache(Collection $commands): bool+{+ return $commands->contains(fn (string $command) => str_contains($command, 'docker builder prune') || str_contains($command, 'docker buildx prune'));+}++/**+ * @param Collection<int, string> $commands+ */+function dockerCleanupPrunedImagesAndContainers(Collection $commands): bool+{+ return $commands->contains(fn (string $command) => str_contains($command, 'docker image prune -f'))+ && $commands->contains(fn (string $command) => str_contains($command, 'xargs -r docker rm'));+}++it('cleans a forced scheduled cleanup while a deployment is queued or running on the server, but keeps the build cache', function (string $status, string $column) {Notification::fake();Event::fake([DockerCleanupDone::class]);$server = dockerCleanupReachableServer();$server->settings()->update(['force_docker_cleanup' => true]);- dockerCleanupDeployment($server, $status, $column);+ $deployment = dockerCleanupDeployment($server, $status, $column);$commands = dockerCleanupFakeProcesses();(new DockerCleanupJob($server->fresh()))->handle();$execution = DockerCleanupExecution::query()->sole();- expect($commands->contains(fn (string $command) => str_contains($command, 'prune')))->toBeFalse()+ expect(dockerCleanupPrunedImagesAndContainers($commands))->toBeTrue()+ ->and(dockerCleanupPrunedBuildCache($commands))->toBeFalse()->and($execution->status)->toBe('success')- ->and($execution->message)->toContain('skipped')+ ->and($execution->message)->toContain('Forced Docker cleanup')+ ->and($execution->message)->toContain('build cache was not pruned')+ ->and($execution->message)->toContain($deployment->deployment_uuid)+ ->and(collect(json_decode($execution->cleanup_log, true))->pluck('output')->implode("\n"))->toContain($deployment->deployment_uuid)->and($execution->finished_at)->not->toBeNull();})->with(['running on the server' => ['in_progress', 'server_id'],@@ -696,6 +719,27 @@ it('skips a forced scheduled cleanup while a deployment is queued or running on'building on the server' => ['in_progress', 'build_server_id'],]);+it('cleans a scheduled cleanup above the disk threshold while a deployment is running, but keeps the build cache', function () {+ Notification::fake();+ Event::fake([DockerCleanupDone::class]);+ $server = dockerCleanupReachableServer();+ $server->settings()->update(['force_docker_cleanup' => false, 'docker_cleanup_threshold' => 80]);+ dockerCleanupDeployment($server, 'in_progress');+ $commands = collect();+ Process::fake(function ($process) use ($commands) {+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);+ $commands->push($command);++ return Process::result(output: str_contains($command, 'df / --output=pcent') ? '90' : '');+ });++ (new DockerCleanupJob($server->fresh()))->handle();++ expect(dockerCleanupPrunedImagesAndContainers($commands))->toBeTrue()+ ->and(dockerCleanupPrunedBuildCache($commands))->toBeFalse()+ ->and(DockerCleanupExecution::query()->sole()->message)->toContain('build cache was not pruned');+});+it('runs a forced scheduled cleanup when the only deployments are finished, on another server, or stuck', function () {Notification::fake();Event::fake([DockerCleanupDone::class]);@@ -724,15 +768,17 @@ it('runs a manual cleanup while a deployment is running, because the user starteexpect($commands->contains(fn (string $command) => str_contains($command, 'docker builder prune')))->toBeTrue();});-it('does not clean after a stop while a deployment is running on the server', function () {+it('cleans after a stop while a deployment is running on the server, but keeps the build cache', function () {$server = dockerCleanupReachableServer();$server->settings()->update(['docker_cleanup_threshold' => 10]);dockerCleanupDeployment($server, 'in_progress');$commands = dockerCleanupFakeProcesses();- expect(CleanupDocker::run($server->fresh(), false, false, true))->toBe([])- ->and($commands)->toBeEmpty()- ->and(Cache::has(CleanupDocker::lastRunCacheKey($server)))->toBeFalse();+ CleanupDocker::run($server->fresh(), false, false, true);++ expect(dockerCleanupPrunedImagesAndContainers($commands))->toBeTrue()+ ->and(dockerCleanupPrunedBuildCache($commands))->toBeFalse()+ ->and(Cache::has(CleanupDocker::lastRunCacheKey($server)))->toBeTrue();});/**@@ -747,7 +793,7 @@ function dockerCleanupImageRemovals(Server $server, array $containers, array $im{$commands = collect();Process::fake(function ($process) use ($commands, $containers, $images) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands->push($command);return match (true) {diff --git a/tests/Feature/DockerVersionStorageTest.php b/tests/Feature/DockerVersionStorageTest.phpindex 4f99e382fc..74a5a29b46 100644--- a/tests/Feature/DockerVersionStorageTest.php+++ b/tests/Feature/DockerVersionStorageTest.php@@ -117,7 +117,7 @@ it('stores a parsed compose version on the server settings', function () {it('records docker and compose versions during connection checks', function () {$server = createServerForDockerVersionStorageTest();- Process::fake([+ fakeRemoteProcesses(['*compose*' => Process::result(output: 'v2.32.4', exitCode: 0),'*' => Process::result(output: '{"Server":{"Version":"29.4.3"}}',diff --git a/tests/Feature/EnvironmentVariable/RefreshEnvsTargetingTest.php b/tests/Feature/EnvironmentVariable/RefreshEnvsTargetingTest.phpindex b58c017626..d14cbb52e5 100644--- a/tests/Feature/EnvironmentVariable/RefreshEnvsTargetingTest.php+++ b/tests/Feature/EnvironmentVariable/RefreshEnvsTargetingTest.php@@ -74,7 +74,7 @@ test('reloading the compose file refreshes only the environment variable list','docker_compose_location' => '/docker-compose.yml',]);Process::fake(function ($process) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);if (str_contains($command, 'git cat-file blob')) {return Process::result(output: "services:\n web:\n image: nginx:alpine\n");}diff --git a/tests/Feature/FileDownloadStorageTest.php b/tests/Feature/FileDownloadStorageTest.phpindex 2cf526c1c0..75436da765 100644--- a/tests/Feature/FileDownloadStorageTest.php+++ b/tests/Feature/FileDownloadStorageTest.php@@ -167,7 +167,7 @@ it('uploads to S3, stores the link, and queues the delete', function () {(new PrepareFileDownloadJob('dl-1', 7, $server->id, 'app-123', '/data/a.txt', false, 'a.txt', $this->storage->id))->handle($downloads);expect(Cache::get(PrepareFileDownloadJob::cacheKey('dl-1')))->toBe(['user_id' => 7, 'status' => 'ready', 'url' => 'https://s3.example.com/get']);- Process::assertRan(fn ($process) => str_contains($process->command, 'https://s3.example.com/put'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'https://s3.example.com/put'));Bus::assertDispatched(DeleteFileDownloadJob::class, fn (DeleteFileDownloadJob $job) => $job->s3StorageId === $this->storage->id&& str_starts_with($job->key, FileDownloadStorageService::PREFIX)&& $job->delay->equalTo(now()->addHours(6)));diff --git a/tests/Feature/GeneratedComposeOutsideBuildContextTest.php b/tests/Feature/GeneratedComposeOutsideBuildContextTest.phpindex 4b3f1413db..4c47a1169e 100644--- a/tests/Feature/GeneratedComposeOutsideBuildContextTest.php+++ b/tests/Feature/GeneratedComposeOutsideBuildContextTest.php@@ -56,7 +56,7 @@ beforeEach(function () {$this->commands = collect();Process::fake(function (PendingProcess $process) {- $command = (string) $process->command;+ $command = remoteProcessScript($process);$this->commands->push($command);return Process::result(output: match (true) {diff --git a/tests/Feature/GetLogsCommandInjectionTest.php b/tests/Feature/GetLogsCommandInjectionTest.phpindex c2ff1eb50b..ca04c917a7 100644--- a/tests/Feature/GetLogsCommandInjectionTest.php+++ b/tests/Feature/GetLogsCommandInjectionTest.php@@ -81,7 +81,7 @@ describe('GetLogs Livewire action validation', function () {])->save();$server = Server::with('settings')->findOrFail($this->server->id);- Process::fake(['*' => Process::result(output: 'all logs')]);+ fakeRemoteProcesses(['*' => Process::result(output: 'all logs')]);Livewire::test(GetLogs::class, ['server' => $server,@@ -94,7 +94,7 @@ describe('GetLogs Livewire action validation', function () {->assertSet('numberOfLines', -1)->assertReturned('all logs');- Process::assertRan(fn ($process) => str_contains($process->command, 'docker logs -n all -t test-container'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker logs -n all -t test-container'));});test('getLogs marks ANSI-colored output truncated based on raw bytes', function () {@@ -108,7 +108,7 @@ describe('GetLogs Livewire action validation', function () {expect(strlen($output))->toBe(GetLogs::MAX_DISPLAY_SIZE_BYTES + 1);- Process::fake(['*' => Process::result(output: $output)]);+ fakeRemoteProcesses(['*' => Process::result(output: $output)]);$component = new GetLogs;$component->server = $server;@@ -195,7 +195,7 @@ describe('GetLogs browser-rendered output', function () {});test('getLogs returns timestamped output without storing it in component state', function () {- Process::fake(['*' => Process::result(output: "2026-09-28T10:00:02Z second\n2026-09-28T10:00:01Z first")]);+ fakeRemoteProcesses(['*' => Process::result(output: "2026-09-28T10:00:02Z second\n2026-09-28T10:00:01Z first")]);$component = Livewire::test(GetLogs::class, ['server' => $this->functionalServer,@@ -207,11 +207,11 @@ describe('GetLogs browser-rendered output', function () {->assertReturned("2026-09-28T10:00:01Z first\n2026-09-28T10:00:02Z second");expect($component->instance())->not->toHaveProperty('outputs');- Process::assertRan(fn ($process) => str_contains($process->command, 'docker logs -n 100 -t test-container'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker logs -n 100 -t test-container'));});test('getLogs only requests new lines when streaming from a timestamp', function () {- Process::fake(['*' => Process::result(output: '2026-09-28T10:00:03Z new')]);+ fakeRemoteProcesses(['*' => Process::result(output: '2026-09-28T10:00:03Z new')]);Livewire::test(GetLogs::class, ['server' => $this->functionalServer,@@ -221,12 +221,12 @@ describe('GetLogs browser-rendered output', function () {->call('getLogs', '2026-09-28T10:00:02.123456789Z')->assertReturned('2026-09-28T10:00:03Z new');- Process::assertRan(fn ($process) => str_contains($process->command, 'docker logs --since 2026-09-28T10:00:02.123456789Z -t test-container'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker logs --since 2026-09-28T10:00:02.123456789Z -t test-container'));});test('getLogs prefixes the streaming command with sudo for non-root servers', function () {$this->functionalServer->update(['user' => 'ubuntu']);- Process::fake(['*' => Process::result(output: '')]);+ fakeRemoteProcesses(['*' => Process::result(output: '')]);Livewire::test(GetLogs::class, ['server' => $this->functionalServer->fresh(['settings']),@@ -234,11 +234,11 @@ describe('GetLogs browser-rendered output', function () {'container' => 'test-container',])->call('getLogs', '2026-09-28T10:00:02Z');- Process::assertRan(fn ($process) => str_contains($process->command, '(sudo docker logs --since 2026-09-28T10:00:02Z -t test-container) 2>&1 | head -c'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), '(sudo docker logs --since 2026-09-28T10:00:02Z -t test-container) 2>&1 | head -c'));});test('getLogs ignores a since value that is not a docker timestamp', function (string $since) {- Process::fake(['*' => Process::result(output: 'logs')]);+ fakeRemoteProcesses(['*' => Process::result(output: 'logs')]);Livewire::test(GetLogs::class, ['server' => $this->functionalServer,@@ -246,8 +246,8 @@ describe('GetLogs browser-rendered output', function () {'container' => 'test-container',])->call('getLogs', $since);- Process::assertRan(fn ($process) => str_contains($process->command, 'docker logs -n 100 -t test-container')- && ! str_contains($process->command, '--since'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker logs -n 100 -t test-container')+ && ! str_contains(remoteProcessScript($process), '--since'));})->with(['command substitution' => ['2026-09-28T10:00:02Z$(id)'],'shell separator' => ['2026-09-28T10:00:02Z; id'],@@ -255,7 +255,7 @@ describe('GetLogs browser-rendered output', function () {]);test('copyLogs keeps line order and omits timestamps when timestamps are hidden', function () {- Process::fake(['*' => Process::result(output: "zeta\nalpha")]);+ fakeRemoteProcesses(['*' => Process::result(output: "zeta\nalpha")]);Livewire::test(GetLogs::class, ['server' => $this->functionalServer,@@ -266,7 +266,7 @@ describe('GetLogs browser-rendered output', function () {->call('copyLogs')->assertReturned("zeta\nalpha");- Process::assertRan(fn ($process) => str_contains($process->command, 'docker logs -n 100 test-container'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker logs -n 100 test-container'));});test('copyLogs returns empty for unauthorized server', function () {@@ -282,7 +282,7 @@ describe('GetLogs browser-rendered output', function () {});test('streaming does not render log lines or poll through Livewire', function () {- Process::fake(['*' => Process::result(output: '2026-09-28T10:00:01Z secret-looking line')]);+ fakeRemoteProcesses(['*' => Process::result(output: '2026-09-28T10:00:01Z secret-looking line')]);Livewire::test(GetLogs::class, ['server' => $this->functionalServer,diff --git a/tests/Feature/GetLogsExportRedactionTest.php b/tests/Feature/GetLogsExportRedactionTest.phpindex dfb016efab..1e8710abba 100644--- a/tests/Feature/GetLogsExportRedactionTest.php+++ b/tests/Feature/GetLogsExportRedactionTest.php@@ -105,9 +105,9 @@ it('redacts stored and effective database passwords in exported logs', function$database->secretManagerLink()->create(['integration_token_id' => $token->id]);}Process::fake(fn ($process) => match (true) {- str_contains($process->command, 'docker ps') => Process::result(output: json_encode(['ID' => 'abc123', 'Names' => $database->uuid, 'Labels' => '', 'State' => 'running'])),- str_contains($process->command, 'docker inspect') => Process::result(output: json_encode(['State' => ['Status' => 'running']])),- str_contains($process->command, 'docker logs') => Process::result(output: 'booted xy effective-database-canary'),+ str_contains(remoteProcessScript($process), 'docker ps') => Process::result(output: json_encode(['ID' => 'abc123', 'Names' => $database->uuid, 'Labels' => '', 'State' => 'running'])),+ str_contains(remoteProcessScript($process), 'docker inspect') => Process::result(output: json_encode(['State' => ['Status' => 'running']])),+ str_contains(remoteProcessScript($process), 'docker logs') => Process::result(output: 'booted xy effective-database-canary'),default => Process::result(output: ''),});@@ -151,9 +151,9 @@ it('still exports database logs when its remote secret source fails', function (]);$database->secretManagerLink()->create(['integration_token_id' => $token->id]);Process::fake(fn ($process) => match (true) {- str_contains($process->command, 'docker ps') => Process::result(output: json_encode(['ID' => 'abc123', 'Names' => $database->uuid, 'Labels' => '', 'State' => 'running'])),- str_contains($process->command, 'docker inspect') => Process::result(output: json_encode(['State' => ['Status' => 'running']])),- str_contains($process->command, 'docker logs') => Process::result(output: 'booted xy {{vault.DB_PASSWORD}}'),+ str_contains(remoteProcessScript($process), 'docker ps') => Process::result(output: json_encode(['ID' => 'abc123', 'Names' => $database->uuid, 'Labels' => '', 'State' => 'running'])),+ str_contains(remoteProcessScript($process), 'docker inspect') => Process::result(output: json_encode(['State' => ['Status' => 'running']])),+ str_contains(remoteProcessScript($process), 'docker logs') => Process::result(output: 'booted xy {{vault.DB_PASSWORD}}'),default => Process::result(output: ''),});diff --git a/tests/Feature/GetLogsMergedSourcesTest.php b/tests/Feature/GetLogsMergedSourcesTest.phpindex 9485b2bccc..55c16a7fe4 100644--- a/tests/Feature/GetLogsMergedSourcesTest.php+++ b/tests/Feature/GetLogsMergedSourcesTest.php@@ -118,7 +118,7 @@ it('rejects a foreign server or an invalid container name before reading logs',it('merges container logs by timestamp and tags each line with its source', function () {Process::fake(function (PendingProcess $process) {- $command = is_array($process->command) ? implode(' ', $process->command) : (string) $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);if (str_contains($command, 'web-helsinki')) {return Process::result(output: "2026-09-28T10:00:02Z second\n2026-09-28T10:00:01Z first\n");}@@ -161,7 +161,7 @@ it('merges container logs by timestamp and tags each line with its source', funcit('asks each source only for lines after its own timestamp', function () {$commands = [];Process::fake(function (PendingProcess $process) use (&$commands) {- $command = is_array($process->command) ? implode(' ', $process->command) : (string) $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands[] = $command;return Process::result(output: "2026-09-28T10:00:03Z line\n");diff --git a/tests/Feature/GetRollbackImagesTest.php b/tests/Feature/GetRollbackImagesTest.phpindex 89481424da..0a9bf1a6bc 100644--- a/tests/Feature/GetRollbackImagesTest.php+++ b/tests/Feature/GetRollbackImagesTest.php@@ -107,7 +107,7 @@ describe('on a reachable server', function () {]))->implode(PHP_EOL);$images = implode(PHP_EOL, ["{$uuid}#aaaaaaa#2026-10-01 10:00:00", "{$uuid}#bbbbbbb#2026-10-02 10:00:00"]);- Process::fake(fn ($process) => Process::result(output: str_contains($process->command, 'docker images') ? $images : $ps));+ Process::fake(fn ($process) => Process::result(output: str_contains(remoteProcessScript($process), 'docker images') ? $images : $ps));}it('finds the current image of containers that are not named after the application uuid', function (string $name) {diff --git a/tests/Feature/GithubRunnerFlowTest.php b/tests/Feature/GithubRunnerFlowTest.phpindex 2be3ec6d50..360c2196ee 100644--- a/tests/Feature/GithubRunnerFlowTest.php+++ b/tests/Feature/GithubRunnerFlowTest.php@@ -506,7 +506,7 @@ describe('provisioning', function () {$commands = '';Process::assertRan(function (PendingProcess $process) use (&$commands) {- $commands .= $process->command."\n";+ $commands .= remoteProcessScript($process)."\n";return true;});@@ -596,7 +596,7 @@ describe('provisioning', function () {(new ProvisionGithubRunnerJob($execution->id))->handle();expect(GithubRunnerExecution::find($execution->id))->toBeNull();- Process::assertDidntRun(fn (PendingProcess $process) => str_contains($process->command, '/home/runner/run.sh'));+ Process::assertDidntRun(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), '/home/runner/run.sh'));Queue::assertPushed(DeregisterGithubRunnerJob::class, fn ($job) => $job->githubAppId === $this->githubApp->id && $job->runnerId === 42);Queue::assertNotPushed(ProvisionGithubRunnerJob::class);});@@ -648,7 +648,7 @@ describe('provisioning', function () {->and($execution->server_id)->toBeNull()->and($execution->runner_name)->toBeNull()->and($execution->error_message)->toContain('Attempt 1 of 3 failed')->toContain('Resource not accessible by integration');- Process::assertRan(fn (PendingProcess $process) => str_contains($process->command, "docker rm -f -v coolify-runner-{$uuid}"));+ Process::assertRan(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), "docker rm -f -v coolify-runner-{$uuid}"));Queue::assertPushed(ProvisionGithubRunnerJob::class, fn ($job) => $job->executionId === $execution->id && $job->delay !== null);Queue::assertNotPushed(CleanupGithubRunnerJob::class);});@@ -701,7 +701,7 @@ describe('provisioning', function () {expect($execution->fresh()->status)->toBe(GithubRunnerStatus::Cancelled)->and($execution->fresh()->ready_at)->toBeNull();- Process::assertDidntRun(fn (PendingProcess $process) => str_contains($process->command, '/home/runner/run.sh'));+ Process::assertDidntRun(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), '/home/runner/run.sh'));Queue::assertPushed(CleanupGithubRunnerJob::class, fn ($job) => $job->executionId === $execution->id);});@@ -732,12 +732,12 @@ describe('provisioning', function () {expect($execution->fresh()->status)->toBe(GithubRunnerStatus::Cancelled)->and($execution->fresh()->ready_at)->toBeNull();- Process::assertDidntRun(fn (PendingProcess $process) => str_contains($process->command, '/home/runner/run.sh'));+ Process::assertDidntRun(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), '/home/runner/run.sh'));Queue::assertPushed(CleanupGithubRunnerJob::class, fn ($job) => $job->executionId === $execution->id);(new CleanupGithubRunnerJob($execution->id))->handle();- Process::assertDidntRun(fn (PendingProcess $process) => str_contains($process->command, 'docker rm'));+ Process::assertDidntRun(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), 'docker rm'));Http::assertNotSent(fn ($request) => $request->method() === 'DELETE');})->with(['the transfer cancelled the execution' => true, 'the transfer commits after the last check' => false]);@@ -747,7 +747,7 @@ describe('provisioning', function () {$execution = runnerTestExecution($this->githubApp);fakeRunnerGithubApi($this->githubApp);Process::fake(['*' => function (PendingProcess $process) use ($execution, $duringStart) {- if (! $duringStart || str_contains($process->command, '/home/runner/run.sh')) {+ if (! $duringStart || str_contains(remoteProcessScript($process), '/home/runner/run.sh')) {$execution->finish(GithubRunnerStatus::Cancelled);}@@ -770,7 +770,7 @@ describe('provisioning', function () {$execution = runnerTestExecution($this->githubApp);fakeRejectedJitConfig();Process::fake(['*' => function (PendingProcess $process) use ($execution) {- if (str_contains($process->command, 'docker rm -f -v')) {+ if (str_contains(remoteProcessScript($process), 'docker rm -f -v')) {$execution->finish(GithubRunnerStatus::Cancelled, 'Stopped externally');}@@ -817,7 +817,7 @@ describe('cleanup and reconciliation', function () {(new CleanupGithubRunnerJob($execution->id))->handle();- Process::assertRan(fn (PendingProcess $process) => str_contains($process->command, "docker rm -f -v coolify-runner-{$execution->uuid}"));+ Process::assertRan(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), "docker rm -f -v coolify-runner-{$execution->uuid}"));Http::assertSent(fn ($request) => $request->method() === 'DELETE' && str_ends_with($request->url(), '/orgs/acme/actions/runners/42'));Queue::assertPushed(ProvisionGithubRunnerJob::class, fn ($job) => $job->executionId === $next->id);});@@ -846,8 +846,8 @@ describe('cleanup and reconciliation', function () {->and($long->fresh()->status)->toBe(GithubRunnerStatus::TimedOut)->and($healthy->fresh()->status)->toBe(GithubRunnerStatus::Running);Queue::assertPushed(CleanupGithubRunnerJob::class, 3);- Process::assertRan(fn (PendingProcess $process) => str_contains($process->command, 'docker rm -f -v coolify-runner-orphanuuid'));- Process::assertDidntRun(fn (PendingProcess $process) => str_contains($process->command, "docker rm -f -v coolify-runner-{$healthy->uuid}"));+ Process::assertRan(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), 'docker rm -f -v coolify-runner-orphanuuid'));+ Process::assertDidntRun(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), "docker rm -f -v coolify-runner-{$healthy->uuid}"));});it('lets a late completed webhook replace a failure guessed from a stopped container', function () {diff --git a/tests/Feature/GithubRunnerSettingsTest.php b/tests/Feature/GithubRunnerSettingsTest.phpindex e06a072e8e..16fe8067df 100644--- a/tests/Feature/GithubRunnerSettingsTest.php+++ b/tests/Feature/GithubRunnerSettingsTest.php@@ -473,7 +473,7 @@ describe('isolated Docker with Sysbox', function () {->test(GithubRunners::class, ['server_uuid' => $this->server->uuid])->call('checkSysbox');- Process::assertRan(fn ($process) => str_contains($process->command, 'test -x /usr/bin/fusermount3'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'test -x /usr/bin/fusermount3'));});it('shows the install button when Sysbox is missing', function () {@@ -495,7 +495,7 @@ describe('isolated Docker with Sysbox', function () {->call('installSysbox')->assertNotDispatched('activityMonitor');- Process::assertRanTimes(fn ($process) => str_contains($process->command, 'apt-get'), 1);+ Process::assertRanTimes(fn ($process) => str_contains(remoteProcessScript($process), 'apt-get'), 1);});it('does not let members install Sysbox', function () {diff --git a/tests/Feature/InstanceBackupMultipleS3DestinationsTest.php b/tests/Feature/InstanceBackupMultipleS3DestinationsTest.phpindex 551e0cef0f..017ddb95ed 100644--- a/tests/Feature/InstanceBackupMultipleS3DestinationsTest.php+++ b/tests/Feature/InstanceBackupMultipleS3DestinationsTest.php@@ -111,7 +111,7 @@ it('saves several S3 destinations for the instance backup and drops storages ofit('uploads the instance backup to every selected destination', function () {$this->backup->forceFill(['save_s3' => true, 'databases_to_backup' => 'coolify'])->save();$this->backup->syncS3Storages([$this->firstStorage->id, $this->secondStorage->id]);- Process::fake(fn ($process) => Process::result(output: str_contains(implode(' ', (array) $process->command), 'du -b') ? '128' : ''));+ Process::fake(fn ($process) => Process::result(output: str_contains(implode(' ', (array) remoteProcessScript($process)), 'du -b') ? '128' : ''));$uploadedTo = collect();$job = Mockery::mock(DatabaseBackupJob::class.'[upload_to_s3]', [$this->backup->fresh()])->shouldAllowMockingProtectedMethods();diff --git a/tests/Feature/Livewire/FileBrowserTest.php b/tests/Feature/Livewire/FileBrowserTest.phpindex f9afb3bef6..2a2ce5d26a 100644--- a/tests/Feature/Livewire/FileBrowserTest.php+++ b/tests/Feature/Livewire/FileBrowserTest.php@@ -384,7 +384,7 @@ it('rejects unsafe child names without deleting the current folder', function (s->call('loadFiles')->call('deleteEntry', $name)->assertDispatched('error');- Process::assertNotRan(fn ($process) => str_contains($process->command, 'rm -rf'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'rm -rf'));})->with(['', '.', '..', '../data', 'a/b', 'a\\b']);it('saves the opened file and clears editor state', function () {@@ -560,8 +560,8 @@ it('lists files for a running application and excludes stopped and preview conta])->map(fn ($container) => json_encode($container))->implode("\n");Process::fake(function ($process) use ($containers) {$output = match (true) {- str_contains($process->command, 'docker ps') => $containers,- str_contains($process->command, 'docker inspect') => '/app',+ str_contains(remoteProcessScript($process), 'docker ps') => $containers,+ str_contains(remoteProcessScript($process), 'docker inspect') => '/app',default => "file\t12\t1\t644\tconfig.yml",};@@ -622,7 +622,7 @@ it('allows retrying a failed initial container lookup', function () {]);$fail = true;Process::fake(function ($process) use (&$fail) {- if (str_contains($process->command, 'docker ps')) {+ if (str_contains(remoteProcessScript($process), 'docker ps')) {if ($fail) {$fail = false;@@ -634,7 +634,7 @@ it('allows retrying a failed initial container lookup', function () {]));}- return Process::result(output: str_contains($process->command, 'docker inspect') ? '/app' : '');+ return Process::result(output: str_contains(remoteProcessScript($process), 'docker inspect') ? '/app' : '');});$this->actingAs($this->admin);session(['currentTeam' => $this->team]);diff --git a/tests/Feature/Livewire/MetricsLiveRefreshTest.php b/tests/Feature/Livewire/MetricsLiveRefreshTest.phpnew file mode 100644index 0000000000..dfc766075d--- /dev/null+++ b/tests/Feature/Livewire/MetricsLiveRefreshTest.php@@ -0,0 +1,121 @@+<?php++use App\Livewire\Fleet\Metrics as FleetMetrics;+use App\Livewire\Project\Shared\Metrics as ResourceMetrics;+use App\Livewire\Server\Charts;+use App\Models\Application;+use App\Models\Environment;+use App\Models\InstanceSettings;+use App\Models\Project;+use App\Models\Server;+use App\Models\StandaloneDocker;+use App\Models\Team;+use App\Models\User;+use Illuminate\Foundation\Testing\RefreshDatabase;+use Illuminate\Support\Facades\Queue;+use Livewire\Features\SupportTesting\Testable;+use Livewire\Livewire;++uses(RefreshDatabase::class);++/*+ * Every metrics poll runs SSH commands that fetch the full range. These servers have metrics+ * off, so loadData() runs without SSH; its refreshChartData event shows that a fetch ran.+ */+beforeEach(function () {+ Queue::fake();+ Server::flushIdentityMap();+ InstanceSettings::unguarded(fn () => InstanceSettings::query()->firstOrCreate(['id' => 0]));++ $this->team = Team::factory()->create();+ $this->user = User::factory()->create();+ $this->user->teams()->attach($this->team, ['role' => 'owner']);+ $this->actingAs($this->user);+ session(['currentTeam' => $this->team]);++ $this->server = Server::factory()->create(['team_id' => $this->team->id]);+});++dataset('range metrics components', [+ 'server metrics' => [fn () => [+ Livewire::test(Charts::class, ['server_uuid' => $this->server->uuid]),+ 'refreshChartData-server-metrics',+ ]],+ 'resource metrics' => [fn () => [+ Livewire::test(ResourceMetrics::class, ['resource' => Application::factory()->create([+ 'environment_id' => Environment::factory()->create([+ 'project_id' => Project::factory()->create(['team_id' => $this->team->id])->id,+ ])->id,+ 'destination_id' => StandaloneDocker::query()->where('server_id', $this->server->id)->firstOrFail()->id,+ 'destination_type' => StandaloneDocker::class,+ ])]),+ 'refreshChartData-metrics-cpu',+ ]],+]);++it('refreshes short ranges live every five seconds by default', function (Closure $build) {+ /** @var Testable $component */+ [$component, $event] = $build->call($this);++ $component->assertSet('live', true);+ expect($component->instance()->pollSeconds())->toBe(5);++ $component->call('pollData')->assertDispatched($event);+})->with('range metrics components');++it('loads long ranges once and does not poll them by default', function (Closure $build) {+ /** @var Testable $component */+ [$component, $event] = $build->call($this);++ $component->set('interval', 43200)->call('setInterval')+ ->assertSet('live', false)+ ->assertDispatched($event);+ expect($component->instance()->pollSeconds())->toBeNull();++ $component->call('pollData')->assertNotDispatched($event);+})->with('range metrics components');++it('refreshes a long range every minute after the user turns live refresh on', function (Closure $build) {+ /** @var Testable $component */+ [$component, $event] = $build->call($this);++ $component->set('interval', 10080)->call('setInterval')+ ->set('live', true)+ ->assertDispatched($event);+ expect($component->instance()->pollSeconds())->toBe(60);++ // A client that polls faster than the range allows gets no new data.+ $component->call('pollData')->assertNotDispatched($event);++ $this->travel(61)->seconds();+ $component->call('pollData')->assertDispatched($event);+})->with('range metrics components');++it('turns live refresh back on when the user returns to a short range', function (Closure $build) {+ /** @var Testable $component */+ [$component] = $build->call($this);++ $component->set('interval', 720)->assertSet('live', false)+ ->set('interval', 30)->assertSet('live', true);+ expect($component->instance()->pollSeconds())->toBe(5);+})->with('range metrics components');++it('does not poll fleet metrics until the user turns live refresh on', function () {+ $component = loadLazy(Livewire::test(FleetMetrics::class))->assertSet('live', false);+ expect($component->instance()->pollSeconds())->toBeNull();+ $component->call('pollData')->assertNotDispatched('refreshChartData-fleet-metrics');++ $component->set('live', true)->assertDispatched('refreshChartData-fleet-metrics');+ expect($component->instance()->pollSeconds())->toBe(60);+ $component->call('pollData')->assertNotDispatched('refreshChartData-fleet-metrics');++ $this->travel(61)->seconds();+ $component->call('pollData')->assertDispatched('refreshChartData-fleet-metrics');+});++it('pauses fleet live refresh when the user picks another range', function () {+ loadLazy(Livewire::test(FleetMetrics::class))+ ->set('live', true)+ ->call('setRange', '30d')+ ->assertSet('live', false);+});diff --git a/tests/Feature/LocalFileVolumeRelativePathTest.php b/tests/Feature/LocalFileVolumeRelativePathTest.phpindex a60bcddbf5..8fc83ca274 100644--- a/tests/Feature/LocalFileVolumeRelativePathTest.php+++ b/tests/Feature/LocalFileVolumeRelativePathTest.php@@ -43,8 +43,8 @@ beforeEach(function () {$this->serviceApplication = ServiceApplication::create(['name' => 'app', 'service_id' => $this->service->id]);Process::fake(fn ($process) => Process::result(output: match (true) {- str_contains($process->command, 'readlink -f') => 'OK',- str_contains($process->command, 'test -') => 'NOK',+ str_contains(remoteProcessScript($process), 'readlink -f') => 'OK',+ str_contains(remoteProcessScript($process), 'test -') => 'NOK',default => '',}));});@@ -67,15 +67,15 @@ function relativePathFileVolume(string $fsPath, bool $isDirectory, ?string $cont*/function assertNoRelativeRemotePath(): void{- Process::assertDidntRun(fn ($process) => str_contains($process->command, "'./"));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), "'./"));}test('a relative directory path is created inside the resource directory, not in the home directory', function () {$workdir = $this->service->workdir();relativePathFileVolume('./data/sub', isDirectory: true)->saveStorageOnServer();- Process::assertRan(fn ($process) => str_contains($process->command, "mkdir -p '{$workdir}/data/sub'"));- Process::assertRan(fn ($process) => str_contains($process->command, "mkdir -p '{$workdir}/data'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "mkdir -p '{$workdir}/data/sub'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "mkdir -p '{$workdir}/data'"));assertNoRelativeRemotePath();});@@ -83,8 +83,8 @@ test('a relative file path without content gets its parent directory inside the$workdir = $this->service->workdir();relativePathFileVolume('./config/app.conf', isDirectory: false)->saveStorageOnServer();- Process::assertRan(fn ($process) => str_contains($process->command, "mkdir -p '{$workdir}/config'"));- Process::assertRan(fn ($process) => str_contains($process->command, "touch '{$workdir}/config/app.conf'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "mkdir -p '{$workdir}/config'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "touch '{$workdir}/config/app.conf'"));assertNoRelativeRemotePath();});@@ -92,7 +92,7 @@ test('a relative file path with content is written inside the resource directory$workdir = $this->service->workdir();relativePathFileVolume('./config/app.conf', isDirectory: false, content: 'listen 8080;')->saveStorageOnServer();- Process::assertRan(fn ($process) => str_contains($process->command, "| base64 -d | tee '{$workdir}/config/app.conf' > /dev/null"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "| base64 -d | tee '{$workdir}/config/app.conf' > /dev/null"));assertNoRelativeRemotePath();});@@ -100,20 +100,20 @@ test('an absolute directory path keeps working unchanged', function () {$workdir = $this->service->workdir();relativePathFileVolume($workdir.'/data', isDirectory: true)->saveStorageOnServer();- Process::assertRan(fn ($process) => str_contains($process->command, "mkdir -p '{$workdir}/data'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "mkdir -p '{$workdir}/data'"));assertNoRelativeRemotePath();});test('deleting a relative path removes it from the resource directory', function () {Process::fake(fn ($process) => Process::result(output: match (true) {- str_contains($process->command, 'readlink -f') => 'OK',- str_contains($process->command, 'test -f') => 'OK',+ str_contains(remoteProcessScript($process), 'readlink -f') => 'OK',+ str_contains(remoteProcessScript($process), 'test -f') => 'OK',default => 'NOK',}));$workdir = $this->service->workdir();relativePathFileVolume('./config/app.conf', isDirectory: false)->deleteStorageOnServer();- Process::assertRan(fn ($process) => str_contains($process->command, "rm -rf '{$workdir}/config/app.conf'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "rm -rf '{$workdir}/config/app.conf'"));assertNoRelativeRemotePath();});diff --git a/tests/Feature/LocalhostServerDetailsTest.php b/tests/Feature/LocalhostServerDetailsTest.phpindex 68e2e4d9ec..ff9ae31443 100644--- a/tests/Feature/LocalhostServerDetailsTest.php+++ b/tests/Feature/LocalhostServerDetailsTest.php@@ -182,7 +182,7 @@ it('collects localhost server details after a successful connection check', func// The connection check also runs SSH multiplexing commands, so answer by command, not by order.Process::fake(function ($process) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return str_contains($command, '---PRETTY_NAME---')? Process::result(output: localhostMetadataProcessOutput(), exitCode: 0)diff --git a/tests/Feature/LockedDomainRowsAndDockerImagesTest.php b/tests/Feature/LockedDomainRowsAndDockerImagesTest.phpindex cb4a072cea..1e0a8fd042 100644--- a/tests/Feature/LockedDomainRowsAndDockerImagesTest.php+++ b/tests/Feature/LockedDomainRowsAndDockerImagesTest.php@@ -71,7 +71,7 @@ it('rejects a client update of the service domain state', function (string $propit('rejects a client update of the docker image list', function () {$imageId = 'sha256:'.str_repeat('a', 64);- Process::fake([+ fakeRemoteProcesses(['*docker image ls*' => Process::result(output: json_encode(['Repository' => 'nginx', 'Tag' => '1.27', 'ID' => $imageId, 'Size' => '52.5MB', 'CreatedSince' => '3 weeks ago'])),'*docker ps -a*' => Process::result(output: "{$imageId}#/web\n"),'*docker system df*' => Process::result(output: ''),diff --git a/tests/Feature/LogDrain/CloudwatchLogDrainTest.php b/tests/Feature/LogDrain/CloudwatchLogDrainTest.phpindex 26034973b9..bd219a5279 100644--- a/tests/Feature/LogDrain/CloudwatchLogDrainTest.php+++ b/tests/Feature/LogDrain/CloudwatchLogDrainTest.php@@ -240,5 +240,5 @@ it('keeps the Fluent Bit drain running if CloudWatch is also enabled, matching tStartLogDrain::run($server);- Process::assertRan(fn ($process) => str_contains($process->command, 'log-drains/docker-compose.yml up -d'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'log-drains/docker-compose.yml up -d'));});diff --git a/tests/Feature/LogDrain/LogDrainServerNameTest.php b/tests/Feature/LogDrain/LogDrainServerNameTest.phpindex 0993fee879..3b8fcf6c51 100644--- a/tests/Feature/LogDrain/LogDrainServerNameTest.php+++ b/tests/Feature/LogDrain/LogDrainServerNameTest.php@@ -31,7 +31,7 @@ test('a server name with a line break cannot add a section to the fluent-bit con$commands = new Collection;Process::fake(function ($process) use ($commands) {- $commands->push($process->command);+ $commands->push(remoteProcessScript($process));return Process::result();});@@ -60,7 +60,7 @@ test('a server name with spaces is written as a single fluent-bit value', functi$commands = new Collection;Process::fake(function ($process) use ($commands) {- $commands->push($process->command);+ $commands->push(remoteProcessScript($process));return Process::result();});diff --git a/tests/Feature/Mcp/McpLogRedactionTest.php b/tests/Feature/Mcp/McpLogRedactionTest.phpindex 2bf6d01c59..2d6e80b62c 100644--- a/tests/Feature/Mcp/McpLogRedactionTest.php+++ b/tests/Feature/Mcp/McpLogRedactionTest.php@@ -84,9 +84,9 @@ function mcpLogRedactionCall(array $abilities, string $tool, array $arguments):function fakeMcpApplicationContainer(Closure $logs): void{Process::fake(fn ($process) => match (true) {- str_contains($process->command, 'docker ps') => Process::result(output: json_encode(['ID' => 'abc123', 'Names' => 'app', 'Labels' => '', 'State' => 'running'])),- str_contains($process->command, 'docker inspect') => Process::result(output: json_encode(['State' => ['Status' => 'running']])),- str_contains($process->command, 'docker logs') => $logs(),+ str_contains(remoteProcessScript($process), 'docker ps') => Process::result(output: json_encode(['ID' => 'abc123', 'Names' => 'app', 'Labels' => '', 'State' => 'running'])),+ str_contains(remoteProcessScript($process), 'docker inspect') => Process::result(output: json_encode(['State' => ['Status' => 'running']])),+ str_contains(remoteProcessScript($process), 'docker logs') => $logs(),default => Process::result(output: ''),});}diff --git a/tests/Feature/Mcp/McpServerToolsTest.php b/tests/Feature/Mcp/McpServerToolsTest.phpindex 36dafa8f07..7adbc2e0f0 100644--- a/tests/Feature/Mcp/McpServerToolsTest.php+++ b/tests/Feature/Mcp/McpServerToolsTest.php@@ -434,7 +434,7 @@ test('manage_registry logs in on its own server without echoing the password', fexpect(mcpSrvIsError($response))->toBeFalse(mcpSrvText($response))->and(mcpSrvText($response))->not->toContain('REGISTRY_PASSWORD_VALUE');- Process::assertRan(fn ($process) => str_contains(is_array($process->command) ? implode(' ', $process->command) : $process->command, 'docker'));+ Process::assertRan(fn ($process) => str_contains(is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process), 'docker'));});test('manage_registry cannot touch registries on a server of another team', function (string $action, string $ability, array $extra) {diff --git a/tests/Feature/NonRootDataDirectoryCommandsTest.php b/tests/Feature/NonRootDataDirectoryCommandsTest.phpindex fb99fac25d..cde366c7fa 100644--- a/tests/Feature/NonRootDataDirectoryCommandsTest.php+++ b/tests/Feature/NonRootDataDirectoryCommandsTest.php@@ -47,7 +47,7 @@ function nonRootDataDirectoryRemoteLines(): array{$lines = [];Process::assertRan(function ($process) use (&$lines) {- array_push($lines, ...explode("\n", $process->command));+ array_push($lines, ...explode("\n", remoteProcessScript($process)));return true;});@@ -124,7 +124,7 @@ it('stages clone archives in a random mktemp directory only for a non-root SSH u};$rootServer = Server::factory()->create(['team_id' => $this->server->team_id, 'user' => 'root', 'private_key_id' => $this->server->private_key_id]);Process::fake(fn ($process) => Process::result(- output: str_contains($process->command, 'mktemp -d') ? '/var/tmp/coolify-clone.Ab3dE6gH9k' : '',+ output: str_contains(remoteProcessScript($process), 'mktemp -d') ? '/var/tmp/coolify-clone.Ab3dE6gH9k' : '',));$nonRootDirectory = $stager->createCloneArchiveDirectory($this->server, 'volume-a');@@ -132,7 +132,7 @@ it('stages clone archives in a random mktemp directory only for a non-root SSH u$commands = [];Process::assertRan(function ($process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return true;});@@ -156,11 +156,11 @@ it('rejects a clone archive directory that mktemp did not create', function (str}};Process::fake(fn ($process) => Process::result(- output: str_contains($process->command, 'mktemp -d') ? $output : '',+ output: str_contains(remoteProcessScript($process), 'mktemp -d') ? $output : '',));expect(fn () => $stager->createCloneArchiveDirectory($this->server, 'volume-a'))->toThrow(RuntimeException::class);- Process::assertNotRan(fn ($process) => str_contains($process->command, 'chown') || str_contains($process->command, 'chmod'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'chown') || str_contains(remoteProcessScript($process), 'chmod'));})->with(['empty' => [''],'predictable path' => ['/var/tmp/coolify-clone/volume-a'],diff --git a/tests/Feature/PersistentStoragePerformanceTest.php b/tests/Feature/PersistentStoragePerformanceTest.phpindex 33a01e7792..db0853a8bd 100644--- a/tests/Feature/PersistentStoragePerformanceTest.php+++ b/tests/Feature/PersistentStoragePerformanceTest.php@@ -25,7 +25,7 @@ beforeEach(function () {$this->withoutVite();config(['app.maintenance.store' => 'array', 'cache.default' => 'array']);Process::fake(fn ($process) => Process::result(- output: str_contains($process->command, 'readlink -f') ? 'OK' : ''+ output: str_contains(remoteProcessScript($process), 'readlink -f') ? 'OK' : ''));InstanceSettings::unguarded(fn () => InstanceSettings::updateOrCreate(['id' => 0],diff --git a/tests/Feature/Proxy/MaintenanceModeTest.php b/tests/Feature/Proxy/MaintenanceModeTest.phpindex 5a16cec8db..e0791cbc63 100644--- a/tests/Feature/Proxy/MaintenanceModeTest.php+++ b/tests/Feature/Proxy/MaintenanceModeTest.php@@ -90,7 +90,7 @@ function maintenanceScriptLines(): array{$lines = [];Process::assertRan(function ($process) use (&$lines) {- array_push($lines, ...explode("\n", $process->command));+ array_push($lines, ...explode("\n", remoteProcessScript($process)));return true;});diff --git a/tests/Feature/Proxy/NonRootProxyCommandsTest.php b/tests/Feature/Proxy/NonRootProxyCommandsTest.phpindex 71c6d906e4..67a82c78a2 100644--- a/tests/Feature/Proxy/NonRootProxyCommandsTest.php+++ b/tests/Feature/Proxy/NonRootProxyCommandsTest.php@@ -43,7 +43,7 @@ function nonRootProxyScriptLines(): array{$lines = [];Process::assertRan(function ($process) use (&$lines) {- array_push($lines, ...explode("\n", $process->command));+ array_push($lines, ...explode("\n", remoteProcessScript($process)));return true;});diff --git a/tests/Feature/Proxy/ProxyErrorPageTest.php b/tests/Feature/Proxy/ProxyErrorPageTest.phpindex 4d1f939459..a9c22a8974 100644--- a/tests/Feature/Proxy/ProxyErrorPageTest.php+++ b/tests/Feature/Proxy/ProxyErrorPageTest.php@@ -54,7 +54,7 @@ function errorPageScriptLines(): array{$lines = [];Process::assertRan(function ($process) use (&$lines) {- array_push($lines, ...explode("\n", $process->command));+ array_push($lines, ...explode("\n", remoteProcessScript($process)));return true;});diff --git a/tests/Feature/Proxy/ProxyLogrotateSidecarCleanupTest.php b/tests/Feature/Proxy/ProxyLogrotateSidecarCleanupTest.phpindex d1639e1c30..0ca2cc616b 100644--- a/tests/Feature/Proxy/ProxyLogrotateSidecarCleanupTest.php+++ b/tests/Feature/Proxy/ProxyLogrotateSidecarCleanupTest.php@@ -45,7 +45,7 @@ function logrotateCleanupScriptLines(): array{$lines = [];Process::assertRan(function ($process) use (&$lines) {- array_push($lines, ...explode("\n", $process->command));+ array_push($lines, ...explode("\n", remoteProcessScript($process)));return true;});diff --git a/tests/Feature/Proxy/ProxyPortVariablesTest.php b/tests/Feature/Proxy/ProxyPortVariablesTest.phpindex b76c423be2..233beb0fa0 100644--- a/tests/Feature/Proxy/ProxyPortVariablesTest.php+++ b/tests/Feature/Proxy/ProxyPortVariablesTest.php@@ -71,7 +71,7 @@ it('starts a proxy whose ports use variables', function (string $port) {StartProxy::run($this->server, async: false, force: true);- Process::assertRan(fn ($process) => str_contains($process->command, 'docker compose -f /data/coolify/proxy/docker-compose.yml up -d --wait --remove-orphans'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker compose -f /data/coolify/proxy/docker-compose.yml up -d --wait --remove-orphans'));})->with('proxy port variables');it('restarts a proxy whose ports use variables', function (string $port) {@@ -92,8 +92,8 @@ it('asks to start a stopped proxy whose ports use variables and checks only concexpect(CheckProxy::run($this->server))->toBeTrue();- Process::assertRan(fn ($process) => str_contains($process->command, "sport = ':8081'"));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'sport = ') && str_contains($process->command, '${'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "sport = ':8081'"));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'sport = ') && str_contains(remoteProcessScript($process), '${'));});it('asks to start a stopped proxy whose ports all use variables without defaults', function () {@@ -102,7 +102,7 @@ it('asks to start a stopped proxy whose ports all use variables without defaultsexpect(CheckProxy::run($this->server))->toBeTrue();- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'sport = '));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'sport = '));});it('does not ask to start a stopped proxy without published ports', function () {@@ -119,7 +119,7 @@ it('saves and starts a proxy whose ports key is empty', function () {StartProxy::run($this->server->fresh(), async: false, force: true);expect($this->server->fresh()->proxy->last_saved_proxy_configuration)->toBe($configuration);- Process::assertRan(fn ($process) => str_contains($process->command, 'docker compose -f /data/coolify/proxy/docker-compose.yml up -d --wait --remove-orphans'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker compose -f /data/coolify/proxy/docker-compose.yml up -d --wait --remove-orphans'));});it('configures traffic analytics for a proxy whose ports use variables', function () {diff --git a/tests/Feature/Proxy/TraefikCertificateResolverTest.php b/tests/Feature/Proxy/TraefikCertificateResolverTest.phpindex 9b698f4d78..51bbe605ca 100644--- a/tests/Feature/Proxy/TraefikCertificateResolverTest.php+++ b/tests/Feature/Proxy/TraefikCertificateResolverTest.php@@ -88,7 +88,7 @@ test('coolify dashboard routes follow the server certificate resolver', function$command = '';Process::assertRan(function ($process) use (&$command) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return str_contains($command, 'coolify.yaml');});@@ -112,7 +112,7 @@ test('the catch-all router follows the server certificate resolver', function (?$command = '';Process::assertRan(function ($process) use (&$command) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return str_contains($command, 'default_redirect_503.yaml') && str_contains($command, 'base64 -d');});diff --git a/tests/Feature/RemoteProcessRedactionTest.php b/tests/Feature/RemoteProcessRedactionTest.phpindex 525dda6614..c3f30e848a 100644--- a/tests/Feature/RemoteProcessRedactionTest.php+++ b/tests/Feature/RemoteProcessRedactionTest.php@@ -141,7 +141,7 @@ it('redacts a database start secret that is split across output chunks', functio$activity = remote_process(['echo ok'], $this->server);// The process fake ends every output chunk with a newline, so the chunks are sent by hand.$invoked = Mockery::mock(InvokedProcess::class);- Process::shouldReceive('timeout->idleTimeout->start')->andReturnUsing(function (string $command, Closure $output) use ($invoked) {+ Process::shouldReceive('timeout->idleTimeout->input->start')->andReturnUsing(function (string $command, Closure $output) use ($invoked) {$invoked->shouldReceive('wait')->andReturnUsing(function () use ($output) {foreach (['value is split-sec', "ret-value\nnext ", 'line split-secret-value'] as $chunk) {$output('out', $chunk);@@ -169,7 +169,7 @@ function runRemoteProcessWithChunks(Activity $activity, array $chunks, ?Throwabl// The process fake ends every output chunk with a newline, so the chunks are sent by hand.$invoked = Mockery::mock(InvokedProcess::class);$invoked->shouldReceive('id')->andReturn(123);- Process::shouldReceive('timeout->start')->andReturnUsing(function (string $command, Closure $output) use ($invoked, $chunks, $failure) {+ Process::shouldReceive('timeout->input->start')->andReturnUsing(function (string $command, Closure $output) use ($invoked, $chunks, $failure) {$invoked->shouldReceive('wait')->andReturnUsing(function () use ($output, $chunks, $failure) {foreach ($chunks as [$type, $chunk]) {$output($type, $chunk);diff --git a/tests/Feature/ReverbAndTerminalPackagingTest.php b/tests/Feature/ReverbAndTerminalPackagingTest.phpindex 440e069875..df1932ad49 100644--- a/tests/Feature/ReverbAndTerminalPackagingTest.php+++ b/tests/Feature/ReverbAndTerminalPackagingTest.php@@ -122,11 +122,13 @@ it('runs Reverb and terminal websocket services inside the Coolify containers',it('removes the dedicated realtime service from bundled compose files', function (string $composeFile, bool $hasRuntimeEnvironment) {$composeContents = file_get_contents(base_path($composeFile));- $soketi = Yaml::parse($composeContents)['services']['soketi'] ?? null;+ $services = Yaml::parse($composeContents)['services'];+ $soketi = $services['soketi'] ?? null;// Production keeps "coolify-realtime" only as a network alias of the coolify container, and "soketi"- // only as a placeholder with no replicas for old overrides in docker-compose.custom.yml.- expect($soketi === null || $soketi === ['image' => 'busybox:1.37', 'deploy' => ['replicas' => 0]])->toBeTrue();+ // only as a placeholder with no replicas for old overrides in docker-compose.custom.yml. The placeholder+ // uses the coolify image, so install and upgrade do not pull an extra image.+ expect($soketi === null || $soketi === ['image' => $services['coolify']['image'], 'deploy' => ['replicas' => 0]])->toBeTrue();expect($composeContents)->not->toContain('container_name: coolify-realtime')->not->toContain('coolify-realtime:')@@ -336,3 +338,30 @@ it('accepts a soketi override from 4.3.23 in docker-compose.custom.yml with zero'production compose, full service' => ['docker-compose.prod.yml', "services:\n soketi:\n image: quay.io/soketi/soketi:1.6-16-alpine\n container_name: coolify-realtime\n ports:\n - '6001:6001'\n"],'nightly production compose, partial override' => ['other/nightly/docker-compose.prod.yml', "services:\n soketi:\n environment:\n SOKETI_DEBUG: 'true'\n"],]);++it('does not pull an extra image for the soketi placeholder', function (string $composeFile) {+ if (! (new ExecutableFinder)->find('docker')) {+ $this->markTestSkipped('Docker is not installed.');+ }++ $directory = sys_get_temp_dir().'/coolify-compose-images-'.Str::random(8);+ mkdir($directory);+ copy(base_path('docker-compose.yml'), "{$directory}/docker-compose.yml");+ copy(base_path($composeFile), "{$directory}/docker-compose.prod.yml");++ try {+ // upgrade.sh pulls every image that "config --images" lists and stops when a pull fails.+ $result = Process::path($directory)+ ->env(['REGISTRY_URL' => 'registry.example.com', 'LATEST_IMAGE' => '4.5.1'])+ ->run('docker compose -f docker-compose.yml -f docker-compose.prod.yml config --no-env-resolution --images');+ } finally {+ File::deleteDirectory($directory);+ }++ expect($result->successful())->toBeTrue($result->errorOutput())+ ->and(collect(explode("\n", trim($result->output())))->unique()->sort()->values()->all())+ ->toBe(['postgres:15-alpine', 'redis:7-alpine', 'registry.example.com/coollabsio/coolify:4.5.1']);+})->with([+ 'production compose' => ['docker-compose.prod.yml'],+ 'nightly production compose' => ['other/nightly/docker-compose.prod.yml'],+]);diff --git a/tests/Feature/RollingUpdateProxyDrainTest.php b/tests/Feature/RollingUpdateProxyDrainTest.phpindex 2c95ddf943..4ad85d0d63 100644--- a/tests/Feature/RollingUpdateProxyDrainTest.php+++ b/tests/Feature/RollingUpdateProxyDrainTest.php@@ -65,7 +65,7 @@ function createApplicationForRollingUpdateDrainTest(string $proxyType = 'TRAEFIKfunction runRollingUpdateDrainStep(Application $application, bool $newVersionIsHealthy, ?array $oldProxyLabels = ['traefik.enable' => 'true'], array $newProxyLabels = ['traefik.enable' => 'true']): array{Process::fake(function ($process) use ($application, $oldProxyLabels, $newProxyLabels) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);if (str_contains($command, 'docker inspect')) {return Process::result(output: json_encode(['coolify.name' => 'new', ...$newProxyLabels])."\n".json_encode(['coolify.name' => 'old', ...$oldProxyLabels]));}diff --git a/tests/Feature/SafeDeploymentLogTest.php b/tests/Feature/SafeDeploymentLogTest.phpindex 1d4f5dcc69..289760fd6b 100644--- a/tests/Feature/SafeDeploymentLogTest.php+++ b/tests/Feature/SafeDeploymentLogTest.php@@ -391,14 +391,14 @@ function makeRestLogRedactionEndpoint(string $resource, string $envSecret): arraProcess::fake(function ($process) use ($containerName, $envSecret) {return match (true) {- str_contains($process->command, 'docker ps') => Process::result(output: json_encode([+ str_contains(remoteProcessScript($process), 'docker ps') => Process::result(output: json_encode(['ID' => 'abc123','Names' => $containerName,'Labels' => "coolify.name={$containerName}",'State' => 'running',])),- str_contains($process->command, 'docker inspect') => Process::result(output: json_encode(['State' => ['Status' => 'running']])),- str_contains($process->command, 'docker logs') => Process::result(output: "booted with {$envSecret}\npassword=coolify-canary-pattern-secret"),+ str_contains(remoteProcessScript($process), 'docker inspect') => Process::result(output: json_encode(['State' => ['Status' => 'running']])),+ str_contains(remoteProcessScript($process), 'docker logs') => Process::result(output: "booted with {$envSecret}\npassword=coolify-canary-pattern-secret"),default => Process::result(output: ''),};});diff --git a/tests/Feature/ScheduledJobManagerDispatchTest.php b/tests/Feature/ScheduledJobManagerDispatchTest.phpindex b02e1bb0a2..cc648b3216 100644--- a/tests/Feature/ScheduledJobManagerDispatchTest.php+++ b/tests/Feature/ScheduledJobManagerDispatchTest.php@@ -1102,11 +1102,11 @@ it('fails only its own execution and notifies the team when the worker timeout s$commands = collect();$atTimeout = null;Process::fake(function ($process) use ($application, $task, $queuedCopy, $team, $commands, &$atTimeout) {- $commands->push($process->command);- if (str_contains($process->command, 'docker ps')) {+ $commands->push(remoteProcessScript($process));+ if (str_contains(remoteProcessScript($process), 'docker ps')) {return Process::result(output: json_encode(['Names' => 'app-'.$application->uuid, 'Labels' => "coolify.applicationUuid={$application->uuid}", 'State' => 'running']));}- if (str_contains($process->command, 'docker exec')) {+ if (str_contains(remoteProcessScript($process), 'docker exec')) {Carbon::setTestNow(now()->addMinute());$newerRun = ScheduledTaskExecution::create(['scheduled_task_id' => $task->id, 'status' => 'running', 'started_at' => now()]);$ownRun = ScheduledTaskExecution::query()->whereKeyNot([$newerRun->id])->where('status', 'running')->sole();diff --git a/tests/Feature/ScheduledTaskApplicationContainerTest.php b/tests/Feature/ScheduledTaskApplicationContainerTest.phpindex fd1162f644..c5d4226f28 100644--- a/tests/Feature/ScheduledTaskApplicationContainerTest.php+++ b/tests/Feature/ScheduledTaskApplicationContainerTest.php@@ -56,7 +56,7 @@ function fakeApplicationContainers(array $containers): void->map(fn (string $state, string $name) => json_encode(['ID' => $name, 'Names' => $name, 'Labels' => 'coolify.applicationId=1', 'State' => $state]))->implode("\n");- Process::fake(fn ($process) => str_contains($process->command, 'docker ps -a --filter')+ Process::fake(fn ($process) => str_contains(remoteProcessScript($process), 'docker ps -a --filter')? Process::result(output: $output): Process::result(output: 'done'));}@@ -67,8 +67,8 @@ test('a scheduled task ignores stopped application containers', function () {(new ScheduledTaskJob($this->task))->handle();- Process::assertRan(fn ($process) => str_contains($process->command, "docker exec '{$uuid}-222222222222' sh -c"));- Process::assertDidntRun(fn ($process) => str_contains($process->command, "docker exec '{$uuid}-111111111111'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker exec '{$uuid}-222222222222' sh -c"));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), "docker exec '{$uuid}-111111111111'"));expect($this->task->executions()->sole()->status)->toBe('success');});@@ -79,6 +79,6 @@ test('a scheduled task runs in the application container with the exact name', f(new ScheduledTaskJob($this->task->refresh()))->handle();- Process::assertRan(fn ($process) => str_contains($process->command, "docker exec '{$uuid}-222222222222' sh -c"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker exec '{$uuid}-222222222222' sh -c"));expect($this->task->executions()->sole()->status)->toBe('success');});diff --git a/tests/Feature/Server/CheckAndStartSentinelJobTest.php b/tests/Feature/Server/CheckAndStartSentinelJobTest.phpindex ac2213e736..e79fe1981c 100644--- a/tests/Feature/Server/CheckAndStartSentinelJobTest.php+++ b/tests/Feature/Server/CheckAndStartSentinelJobTest.php@@ -49,14 +49,14 @@ it('does not connect to an unreachable or unusable server', function (string $fl})->with(['is_reachable', 'is_usable']);it('starts Sentinel when the container is missing', function () {- Process::fake(['*' => Process::result(output: '')]);+ fakeRemoteProcesses(['*' => Process::result(output: '')]);StartSentinel::shouldRun()->once();(new CheckAndStartSentinelJob($this->server))->handle();});it('does not treat an SSH failure as a missing container', function () {- Process::fake(['*' => Process::result(errorOutput: 'ssh: connect to host 1.2.3.4 port 22: Connection timed out', exitCode: 255)]);+ fakeRemoteProcesses(['*' => Process::result(errorOutput: 'ssh: connect to host 1.2.3.4 port 22: Connection timed out', exitCode: 255)]);StartSentinel::shouldNotRun();expect(fn () => (new CheckAndStartSentinelJob($this->server))->handle())@@ -65,7 +65,7 @@ it('does not treat an SSH failure as a missing container', function () {it('does not restart a running Sentinel that is up to date', function () {$this->server->forceFill(['sentinel_updated_at' => now()])->save();- Process::fake([+ fakeRemoteProcesses(['*docker ps*' => Process::result(output: 'running'),'*api/version*' => Process::result(output: '1.0.0'),]);@@ -76,7 +76,7 @@ it('does not restart a running Sentinel that is up to date', function () {it('uses sudo for the status check on a non-root server', function () {$this->server->forceFill(['user' => 'ubuntu', 'sentinel_updated_at' => now()])->save();- Process::fake([+ fakeRemoteProcesses(['*docker ps*' => Process::result(output: 'running'),'*api/version*' => Process::result(output: '1.0.0'),]);@@ -84,12 +84,12 @@ it('uses sudo for the status check on a non-root server', function () {(new CheckAndStartSentinelJob($this->server->refresh()))->handle();- Process::assertRan(fn ($process) => str_contains($process->command, 'sudo docker ps -a'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'sudo docker ps -a'));});it('restarts an unhealthy Sentinel and remembers why', function () {$this->server->forceFill(['sentinel_updated_at' => now()])->save();- Process::fake(['*docker ps*' => Process::result(output: 'running Up 2 hours (unhealthy)')]);+ fakeRemoteProcesses(['*docker ps*' => Process::result(output: 'running Up 2 hours (unhealthy)')]);StartSentinel::shouldRun()->once();(new CheckAndStartSentinelJob($this->server))->handle();@@ -99,7 +99,7 @@ it('restarts an unhealthy Sentinel and remembers why', function () {it('does not restart a Sentinel whose health check is still starting', function () {$this->server->forceFill(['sentinel_updated_at' => now()])->save();- Process::fake([+ fakeRemoteProcesses(['*docker ps*' => Process::result(output: 'running Up 30 seconds (health: starting)'),'*api/version*' => Process::result(output: '1.0.0'),]);@@ -112,7 +112,7 @@ it('restarts a running Sentinel that stopped pushing, at most once per hour', fuCache::flush();Cache::put('coolify:versions:all', ['coolify' => ['sentinel' => ['version' => '1.0.0']]], 3600);$this->server->forceFill(['sentinel_updated_at' => now()->subMinutes(15), 'sentinel_waiting_since' => null])->save();- Process::fake([+ fakeRemoteProcesses(['*docker ps*' => Process::result(output: 'running Up 2 hours (healthy)'),'*api/version*' => Process::result(output: '1.0.0'),]);@@ -126,7 +126,7 @@ it('does not restart a Sentinel that started recently and has not pushed yet', fCache::flush();Cache::put('coolify:versions:all', ['coolify' => ['sentinel' => ['version' => '1.0.0']]], 3600);$this->server->forceFill(['sentinel_updated_at' => now()->subDays(5), 'sentinel_waiting_since' => now()->subMinutes(3)])->save();- Process::fake([+ fakeRemoteProcesses(['*docker ps*' => Process::result(output: 'running Up 3 minutes (healthy)'),'*api/version*' => Process::result(output: '1.0.0'),]);@@ -144,7 +144,7 @@ it('remembers the push error that Sentinel reports', function () {'last_status' => null,'consecutive_failures' => 4,]);- Process::fake([+ fakeRemoteProcesses(['*docker ps*' => Process::result(output: 'running Up 2 hours (healthy)'),'*api/version*' => Process::result(output: "1.0.0\n{$pushStatus}"),]);@@ -157,20 +157,20 @@ it('remembers the push error that Sentinel reports', function () {it('reads the push status with the token from inside the Sentinel container on a non-root server', function () {$this->server->forceFill(['user' => 'ubuntu', 'sentinel_updated_at' => now()])->save();- Process::fake([+ fakeRemoteProcesses(['*docker ps*' => Process::result(output: 'running Up 2 hours (healthy)'),'*api/version*' => Process::result(output: '1.0.0'),]);(new CheckAndStartSentinelJob($this->server->refresh()))->handle();- Process::assertRan(fn ($process) => str_contains($process->command, "sudo bash -c 'docker exec coolify-sentinel sh -c '\\''curl")- && str_contains($process->command, 'Bearer $TOKEN'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "sudo bash -c 'docker exec coolify-sentinel sh -c '\\''curl")+ && str_contains(remoteProcessScript($process), 'Bearer $TOKEN'));});it('does not restart a starting Sentinel whose API does not answer yet', function () {$this->server->forceFill(['sentinel_updated_at' => now()->subDays(5), 'sentinel_waiting_since' => now()])->save();- Process::fake([+ fakeRemoteProcesses(['*docker ps*' => Process::result(output: 'running Up 3 seconds (health: starting)'),'*api/version*' => Process::result(output: ''),]);diff --git a/tests/Feature/Server/CloudflareTunnelChangedNotificationTest.php b/tests/Feature/Server/CloudflareTunnelChangedNotificationTest.phpindex d57f68a4b5..a6481e4489 100644--- a/tests/Feature/Server/CloudflareTunnelChangedNotificationTest.php+++ b/tests/Feature/Server/CloudflareTunnelChangedNotificationTest.php@@ -34,7 +34,7 @@ beforeEach(function () {it('uses sudo for the health check on a non-root server', function () {Event::fake();Sleep::fake();- Process::fake([+ fakeRemoteProcesses(['*sudo docker inspect coolify-cloudflared*' => Process::result(output: 'true'),'*' => Process::result(errorOutput: 'permission denied while trying to connect to the Docker API', exitCode: 1),]);diff --git a/tests/Feature/Server/ServerMetricsSettingsTest.php b/tests/Feature/Server/ServerMetricsSettingsTest.phpindex 6d4a069702..86022a44e8 100644--- a/tests/Feature/Server/ServerMetricsSettingsTest.php+++ b/tests/Feature/Server/ServerMetricsSettingsTest.php@@ -53,19 +53,19 @@ it('validates metrics collection settings on the server metrics page', function]);});-it('refreshes server metrics live for every time range until paused', function () {+it('refreshes short server metric ranges live and starts long ranges paused', function () {$server = Server::factory()->create(['team_id' => $this->team->id]);$server->settings->is_sentinel_enabled = true;$server->settings->is_metrics_enabled = true;$server->settings->saveQuietly();- Livewire::test(Charts::class, ['server_uuid' => $server->uuid])- ->assertSet('live', true)- ->assertSeeHtml('wire:poll.5s="loadData"')- ->set('interval', 43200)- ->assertSet('live', true)- ->assertSeeHtml('wire:poll.60s="loadData"')- ->set('live', false)- ->assertDontSeeHtml('wire:poll')- ->assertSee('Paused');+ $component = Livewire::test(Charts::class, ['server_uuid' => $server->uuid])+ ->assertSet('live', true);+ expect($component->instance()->pollSeconds())->toBe(5);++ $component->set('interval', 43200)->assertSet('live', false);+ expect($component->instance()->pollSeconds())->toBeNull();++ $component->set('live', true);+ expect($component->instance()->pollSeconds())->toBe(60);});diff --git a/tests/Feature/Server/StartSentinelLockTest.php b/tests/Feature/Server/StartSentinelLockTest.phpindex cec5438a45..310d89da3f 100644--- a/tests/Feature/Server/StartSentinelLockTest.php+++ b/tests/Feature/Server/StartSentinelLockTest.php@@ -48,7 +48,7 @@ function queuedSentinelStarts(): intit('holds the per-server lock while it replaces the Sentinel container', function () {$lockHeldDuringDockerRun = null;Process::fake(function ($process) use (&$lockHeldDuringDockerRun) {- $script = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $script = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);if (str_contains($script, 'docker run -d')) {$lockHeldDuringDockerRun = ! Cache::lock(StartSentinel::lockKey($this->server), 1)->get();}@@ -69,7 +69,7 @@ it('starts Sentinel with the settings saved after the server was loaded', functi$scripts = [];Process::fake(function ($process) use (&$scripts) {- $scripts[] = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $scripts[] = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: '');});diff --git a/tests/Feature/ServerConnectionCheckIsolationTest.php b/tests/Feature/ServerConnectionCheckIsolationTest.phpindex ee308cb52f..506b80937f 100644--- a/tests/Feature/ServerConnectionCheckIsolationTest.php+++ b/tests/Feature/ServerConnectionCheckIsolationTest.php@@ -71,7 +71,7 @@ it('does not call cloud provider APIs during an SSH connection check', function'instance' => ['id' => 'instance-1', 'status' => 'active'],]),]);- Process::fake([+ fakeRemoteProcesses(['*' => Process::result(output: '{"Server":{"Version":"27.0.0"}}',exitCode: 0,@@ -89,7 +89,7 @@ it('resets unreachable_count after a successful connection check', function () {$server->unreachable_count = 5;$server->save();- Process::fake([+ fakeRemoteProcesses(['*' => Process::result(output: '{"Server":{"Version":"27.0.0"}}',exitCode: 0,@@ -107,7 +107,7 @@ it('logs the checking node and ssh error only when the connection state changes'$logPath = tempnam(sys_get_temp_dir(), 'coolify-scheduled-log-');config(['logging.channels.scheduled' => ['driver' => 'single', 'path' => $logPath, 'level' => 'debug']]);Log::forgetChannel('scheduled');- Process::fake([+ fakeRemoteProcesses(['*' => Process::result(errorOutput: 'ssh: connect to host 203.0.113.10 port 22: Connection refused', exitCode: 255),]);@@ -127,7 +127,7 @@ it('logs the checking node and ssh error only when the connection state changes'it('keeps the server online after one failed SSH check', function () {$server = createServerForConnectionIsolationTest(['ip' => '203.0.113.10']);$server->settings->update(['is_reachable' => true, 'is_usable' => true]);- Process::fake([+ fakeRemoteProcesses(['*' => Process::result(errorOutput: 'Connection timed out', exitCode: 255),]);@@ -141,7 +141,7 @@ it('keeps the server online after one failed SSH check', function () {it('marks the server offline after consecutive failed SSH checks', function () {$server = createServerForConnectionIsolationTest(['ip' => '203.0.113.10']);$server->settings->update(['is_reachable' => true, 'is_usable' => true]);- Process::fake([+ fakeRemoteProcesses(['*' => Process::result(errorOutput: 'Connection timed out', exitCode: 255),]);@@ -159,7 +159,7 @@ it('does not mark the server offline when the check fails for a reason other tha$server->settings->update(['is_reachable' => true, 'is_usable' => true]);$server->unreachable_count = 1;$server->save();- Process::fake([+ fakeRemoteProcesses(['*' => Process::result(output: '{"Server":{"Version":"27.0.0"}}', exitCode: 0),]);// Fails the unreachable_count reset, after SSH and Docker checks passed.@@ -178,7 +178,7 @@ it('retries a failed SSH attempt once within the same check', function () {$server->settings->update(['is_reachable' => false, 'is_usable' => false]);$server->unreachable_count = 3;$server->save();- Process::fake([+ fakeRemoteProcesses(['*ls -la /*' => Process::sequence()->push(Process::result(errorOutput: 'Connection timed out', exitCode: 255))->push(Process::result(exitCode: 0)),@@ -189,7 +189,7 @@ it('retries a failed SSH attempt once within the same check', function () {(new ServerConnectionCheckJob($server->fresh(), disableMux: false))->handle();Sleep::assertSleptTimes(1);- Process::assertRanTimes(fn ($process) => str_contains($process->command, 'ls -la /'), 2);+ Process::assertRanTimes(fn ($process) => str_contains(remoteProcessScript($process), 'ls -la /'), 2);expect($server->fresh()->unreachable_count)->toBe(0)->and((bool) $server->settings->fresh()->is_reachable)->toBeTrue()->and((bool) $server->settings->fresh()->is_usable)->toBeTrue();@@ -198,14 +198,14 @@ it('retries a failed SSH attempt once within the same check', function () {it('counts one failed check when both SSH attempts fail', function () {$server = createServerForConnectionIsolationTest(['ip' => '203.0.113.10']);$server->settings->update(['is_reachable' => true, 'is_usable' => true]);- Process::fake([+ fakeRemoteProcesses(['*' => Process::result(errorOutput: 'Connection timed out', exitCode: 255),]);(new ServerConnectionCheckJob($server->fresh(), disableMux: false))->handle();Sleep::assertSleptTimes(1);- Process::assertRanTimes(fn ($process) => str_contains($process->command, 'ls -la /'), 2);+ Process::assertRanTimes(fn ($process) => str_contains(remoteProcessScript($process), 'ls -la /'), 2);expect($server->fresh()->unreachable_count)->toBe(1)->and((bool) $server->settings->fresh()->is_reachable)->toBeTrue();});@@ -213,21 +213,21 @@ it('counts one failed check when both SSH attempts fail', function () {it('does not retry the Docker check within the same check', function () {$server = createServerForConnectionIsolationTest(['ip' => '203.0.113.10']);$server->settings->update(['is_reachable' => true, 'is_usable' => true]);- Process::fake([+ fakeRemoteProcesses(['*docker version*' => Process::result(errorOutput: 'Cannot connect to the Docker daemon', exitCode: 1),'*' => Process::result(exitCode: 0),]);(new ServerConnectionCheckJob($server->fresh(), disableMux: false))->handle();- Process::assertRanTimes(fn ($process) => str_contains($process->command, 'docker version'), 1);+ Process::assertRanTimes(fn ($process) => str_contains(remoteProcessScript($process), 'docker version'), 1);expect((bool) $server->settings->fresh()->is_reachable)->toBeTrue()->and((bool) $server->settings->fresh()->is_usable)->toBeFalse();});it('keeps the server usable when only the compose version check fails', function () {$server = createServerForConnectionIsolationTest(['ip' => '203.0.113.10']);- Process::fake([+ fakeRemoteProcesses(['*compose*' => Process::result(errorOutput: 'docker: compose is not a docker command', exitCode: 1),'*' => Process::result(output: '{"Server":{"Version":"29.4.3"}}', exitCode: 0),]);@@ -251,7 +251,7 @@ it('stops managing a server when its owner marker names a newer claim of another'server_metadata' => ['transfer' => ['status' => 'claimed', 'claim' => ['claim_id' => 'claim-of-this-instance', 'claimed_at' => '2026-01-01T00:00:00+00:00']]],]);$server->settings->update(['is_reachable' => true, 'is_usable' => true]);- Process::fake([+ fakeRemoteProcesses(['*ls -la /*' => Process::result(output: '{"claim_id":"claim-of-instance-b","instance_url":"https://8.8.8.8","claimed_at":"2026-02-01T00:00:00+00:00"}', exitCode: 0),'*' => Process::result(output: '{"Server":{"Version":"29.4.3"}}', exitCode: 0),]);@@ -262,7 +262,7 @@ it('stops managing a server when its owner marker names a newer claim of anotherexpect($server->isTransferredAway())->toBeTrue()->and(data_get($server->server_metadata, 'transfer.target_instance_url'))->toBe('https://8.8.8.8')->and((bool) $server->settings->force_disabled)->toBeTrue();- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'docker version'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'docker version'));});it('keeps managing a server when its owner marker is its own claim', function () {@@ -270,7 +270,7 @@ it('keeps managing a server when its owner marker is its own claim', function ()'ip' => '203.0.113.10','server_metadata' => ['transfer' => ['status' => 'claimed', 'claim' => ['claim_id' => 'claim-of-this-instance']]],]);- Process::fake([+ fakeRemoteProcesses(['*ls -la /*' => Process::result(output: '{"claim_id":"claim-of-this-instance"}', exitCode: 0),'*compose*' => Process::result(output: 'v2.32.4', exitCode: 0),'*' => Process::result(output: '{"Server":{"Version":"29.4.3"}}', exitCode: 0),diff --git a/tests/Feature/ServerDockerImagesAuthorizationTest.php b/tests/Feature/ServerDockerImagesAuthorizationTest.phpindex 8d50c24b8e..71836c7255 100644--- a/tests/Feature/ServerDockerImagesAuthorizationTest.php+++ b/tests/Feature/ServerDockerImagesAuthorizationTest.php@@ -23,7 +23,7 @@ beforeEach(function () {$this->server->settings->update(['is_reachable' => true, 'is_usable' => true]);$this->actingAs($this->user);session(['currentTeam' => $this->team]);- Process::fake([+ fakeRemoteProcesses(['*docker image ls*' => Process::result(output: json_encode(['Repository' => 'nginx', 'Tag' => '1.27', 'ID' => 'sha256:'.str_repeat('a', 64), 'Size' => '1MB', 'CreatedSince' => 'now'])),'*' => Process::result(output: ''),]);diff --git a/tests/Feature/ServerDockerImagesTest.php b/tests/Feature/ServerDockerImagesTest.phpindex d056d8c9e4..da36681d1a 100644--- a/tests/Feature/ServerDockerImagesTest.php+++ b/tests/Feature/ServerDockerImagesTest.php@@ -29,7 +29,7 @@ $nginxId = 'sha256:'.str_repeat('a', 64);$danglingId = 'sha256:'.str_repeat('b', 64);$fakeDocker = function (string $nginxId, string $danglingId) {- Process::fake([+ fakeRemoteProcesses(['*docker image ls*' => Process::result(output: implode("\n", [json_encode(['Repository' => 'nginx', 'Tag' => '1.27', 'ID' => $nginxId, 'Size' => '52.5MB', 'CreatedSince' => '3 weeks ago']),json_encode(['Repository' => '<none>', 'Tag' => '<none>', 'ID' => $danglingId, 'Size' => '11MB', 'CreatedSince' => '2 days ago']),@@ -78,7 +78,7 @@ it('deletes an unused image', function () use ($fakeDocker, $nginxId, $danglingI->call('delete', $danglingId)->assertDispatched('success');- Process::assertRan(fn ($process) => str_contains($process->command, "docker image rm '{$danglingId}'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker image rm '{$danglingId}'"));});it('refuses to delete an image that a container is using', function () use ($fakeDocker, $nginxId, $danglingId) {@@ -89,7 +89,7 @@ it('refuses to delete an image that a container is using', function () use ($fak->call('delete', 'nginx:1.27')->assertDispatched('error');- Process::assertNotRan(fn ($process) => str_contains($process->command, 'docker image rm'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker image rm'));});it('filters images by the search term', function () use ($fakeDocker, $nginxId, $danglingId) {@@ -103,7 +103,7 @@ it('filters images by the search term', function () use ($fakeDocker, $nginxId,});it('paginates images and returns to the first page when the search changes', function () {- Process::fake([+ fakeRemoteProcesses(['*docker image ls*' => Process::result(output: collect(range(1, 12))->map(fn (int $i) => json_encode(['Repository' => sprintf('app-%02d', $i), 'Tag' => 'latest', 'ID' => 'sha256:'.str_repeat(dechex($i), 64),diff --git a/tests/Feature/ServerDockerRegistriesTest.php b/tests/Feature/ServerDockerRegistriesTest.phpindex 9c7cd9ee45..3cecdc14dc 100644--- a/tests/Feature/ServerDockerRegistriesTest.php+++ b/tests/Feature/ServerDockerRegistriesTest.php@@ -76,7 +76,7 @@ it('shows which registries a server is logged in to and which ones its applicatimakeDockerRegistriesServerReachable($this->server);Process::fake(function ($process) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: str_contains($command, '.docker/config.json')? json_encode(['auths' => ['ghcr.io' => ['auth' => base64_encode('user:SECRET_TOKEN')]]])@@ -171,7 +171,7 @@ function fakeDockerRegistryServer(string $loginError = ''): array{$commands = [];Process::fake(function ($process) use (&$commands, $loginError) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$commands[] = $command;if ($loginError !== '' && str_contains($command, ' login ')) {@@ -191,7 +191,7 @@ it('logs in to a registry with the token on stdin and never as plain text in themakeDockerRegistriesServerReachable($this->server);$commands = [];Process::fake(function ($process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return Process::result(output: '');});@@ -226,7 +226,7 @@ it('lets docker choose the Docker Hub key and runs the login through sudo for nomakeDockerRegistriesServerReachable($this->server);$commands = [];Process::fake(function ($process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return Process::result(output: '');});@@ -271,7 +271,7 @@ it('validates the registry host and username before running a command', function->call('login')->assertHasErrors($errorField);- Process::assertDidntRun(fn ($process) => str_contains($process->command, ' login '));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), ' login '));})->with(['shell characters in registry' => ["ghcr.io';reboot;'", 'user', 'registry'],'space in username' => ['ghcr.io', 'my user', 'username'],@@ -289,7 +289,7 @@ it('logs out from a registry', function () {->call('logout', 'ghcr.io', '')->assertDispatched('success');- Process::assertRan(fn ($process) => str_contains($process->command, "docker --config \"\$HOME/.docker\" logout 'ghcr.io'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker --config \"\$HOME/.docker\" logout 'ghcr.io'"));expect(AuditEvent::query()->where('event', 'ui.server.registry_logout')->sole()->metadata['registry'])->toBe('ghcr.io');});@@ -309,7 +309,7 @@ it('does not let a user who lost the admin role log in or log out', function (st? $component->call('login')->assertForbidden(): $component->call('logout', 'ghcr.io')->assertForbidden();- Process::assertDidntRun(fn ($process) => str_contains($process->command, ' login ') || str_contains($process->command, ' logout '));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), ' login ') || str_contains(remoteProcessScript($process), ' logout '));})->with(['login', 'logout']);it('does not render the registries page again for a user who lost the admin role', function () {@@ -364,7 +364,7 @@ it('edits an existing login with the registry kept fixed', function () {makeDockerRegistriesServerReachable($this->server);$commands = [];Process::fake(function ($process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return Process::result(output: '');});@@ -440,8 +440,8 @@ it('always uses the fixed host of a provider, even if the browser sends another->assertHasNoErrors()->assertDispatched('success', 'Logged in to ghcr.io.');- Process::assertRan(fn ($process) => str_contains($process->command, "--password-stdin 'ghcr.io'"));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'evil.example.com'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "--password-stdin 'ghcr.io'"));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'evil.example.com'));});it('checks that the host belongs to the chosen cloud provider', function (string $provider, string $registry, bool $valid) {@@ -650,7 +650,7 @@ it('tests a saved login and shows the result on the row', function () {makeDockerRegistriesServerReachable($this->server);$checkFails = false;Process::fake(function ($process) use (&$checkFails) {- if (str_contains($process->command, '</dev/null')) {+ if (str_contains(remoteProcessScript($process), '</dev/null')) {return $checkFails? Process::result(errorOutput: "Authenticating with existing credentials... [Username: octocat]\nLogin did not succeed, error: Error response from daemon: unauthorized\nerror: cannot perform an interactive login from a non-TTY device", exitCode: 1): Process::result(output: 'Login Succeeded');@@ -663,7 +663,7 @@ it('tests a saved login and shows the result on the row', function () {->call('testLogin', 'ghcr.io')->assertSet('loginChecks', ['ghcr.io' => ['ok' => true, 'message' => 'The login works.']])->assertDispatched('success', 'The login for ghcr.io works.');- Process::assertRan(fn ($process) => str_contains($process->command, "docker --config \"\$HOME/.docker\" login 'ghcr.io' </dev/null"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker --config \"\$HOME/.docker\" login 'ghcr.io' </dev/null"));$checkFails = true;$component->call('testLogin', 'ghcr.io')@@ -682,7 +682,7 @@ it('does not test registries that are not logged in', function () {->call('testLogin', 'registry.example.com')->assertSet('loginChecks', []);- Process::assertDidntRun(fn ($process) => str_contains($process->command, '</dev/null'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), '</dev/null'));});it('logs in to several servers at once from the overview page', function () {@@ -743,7 +743,7 @@ it('refuses a multi-server login to a server of another team', function () {->call('login')->assertForbidden();- Process::assertDidntRun(fn ($process) => str_contains($process->command, ' login '));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), ' login '));});it('does not open the overview login form for members', function () {diff --git a/tests/Feature/ServerMetricsPrefetchTest.php b/tests/Feature/ServerMetricsPrefetchTest.phpindex 85a51e1ac4..ea82720640 100644--- a/tests/Feature/ServerMetricsPrefetchTest.php+++ b/tests/Feature/ServerMetricsPrefetchTest.php@@ -24,7 +24,7 @@ beforeEach(function () {it('fetches several metric types in one SSH command', function () {$commands = [];Process::fake(function (PendingProcess $process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return Process::result(implode("\x1e", ['[{"time":1000,"percent":12.5}]',@@ -55,7 +55,7 @@ it('keeps the Sentinel error of one prefetched metric', function () {it('falls back to one fetch per metric when the batch fails', function () {$commands = [];Process::fake(function (PendingProcess $process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return count($commands) === 1? Process::result('', 'ssh: connect to host failed', 1)@@ -71,7 +71,7 @@ it('falls back to one fetch per metric when the batch fails', function () {it('runs the batched metrics script with sudo for a non-root user', function () {$commands = [];Process::fake(function (PendingProcess $process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return Process::result("[]\x1e[]");});diff --git a/tests/Feature/ServerProxyDynamicConfigurationsAuthorizationTest.php b/tests/Feature/ServerProxyDynamicConfigurationsAuthorizationTest.phpindex 674c829b1b..bd122fae24 100644--- a/tests/Feature/ServerProxyDynamicConfigurationsAuthorizationTest.php+++ b/tests/Feature/ServerProxyDynamicConfigurationsAuthorizationTest.php@@ -31,10 +31,10 @@ beforeEach(function () {$this->server->settings->forceFill(['is_reachable' => true, 'is_usable' => true, 'force_disabled' => false])->save();Process::fake(function ($process) {- if (str_contains($process->command, 'ls -1')) {+ if (str_contains(remoteProcessScript($process), 'ls -1')) {return Process::result(output: "zzfix-auth.yaml\n");}- if (str_contains($process->command, 'zzfix-auth.yaml')) {+ if (str_contains(remoteProcessScript($process), 'zzfix-auth.yaml')) {return Process::result(output: "http:\n middlewares:\n auth:\n basicAuth:\n users:\n - 'zzfix:\$apr1\$zzfixsecrethash'\n");}@@ -65,7 +65,7 @@ it('does not load dynamic configuration contents for a team member', function ()->and(json_encode($component->snapshot))->not->toContain('zzfixsecrethash')->and($component->html())->not->toContain('zzfixsecrethash')->and($component->html())->toContain('zzfix-auth.yaml');- Process::assertNotRan(fn ($process) => str_contains($process->command, 'head -c'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'head -c'));});it('loads dynamic configuration contents for a team admin', function () {diff --git a/tests/Feature/ServerStorageSaveDispatchTest.php b/tests/Feature/ServerStorageSaveDispatchTest.phpindex 917f98422a..b179890d1b 100644--- a/tests/Feature/ServerStorageSaveDispatchTest.php+++ b/tests/Feature/ServerStorageSaveDispatchTest.php@@ -38,7 +38,7 @@ beforeEach(function () {]);Process::fake(fn ($process) => Process::result(- output: str_contains($process->command, 'readlink -f') ? 'OK' : 'NOK'+ output: str_contains(remoteProcessScript($process), 'readlink -f') ? 'OK' : 'NOK'));$this->processedSaves = 0;@@ -65,13 +65,13 @@ it('runs the storage save after its mount is committed', function () {$volume = createStorageSaveTestMount($this->application);expect($this->processedSaves)->toBe(0);- Process::assertNotRan(fn ($process) => str_contains($process->command, 'mkdir -p'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'mkdir -p'));DB::commit();expect($this->processedSaves)->toBe(1)->and($volume->exists)->toBeTrue();- Process::assertRan(fn ($process) => str_contains($process->command, 'mkdir -p'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'mkdir -p'));});it('does not run the storage save after its mount is rolled back', function () {@@ -81,7 +81,7 @@ it('does not run the storage save after its mount is rolled back', function () {expect($this->processedSaves)->toBe(0)->and(LocalFileVolume::query()->count())->toBe(0);- Process::assertNotRan(fn ($process) => str_contains($process->command, 'mkdir -p'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'mkdir -p'));});it('writes a new file mount immediately when there is no transaction', function () {@@ -95,7 +95,7 @@ it('writes a new file mount immediately when there is no transaction', function]);expect($this->processedSaves)->toBe(1);- Process::assertRan(fn ($process) => str_contains($process->command, base64_encode('KEY=value')));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), base64_encode('KEY=value')));});it('does not write compose content outside the resource directory before a deployment', function () {@@ -114,7 +114,7 @@ it('does not write compose content outside the resource directory before a deplo]);expect($this->processedSaves)->toBe(0);- Process::assertNotRan(fn ($process) => str_contains($process->command, base64_encode('outside-content')));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), base64_encode('outside-content')));});it('writes compose content inside the resource directory immediately', function () {@@ -133,7 +133,7 @@ it('writes compose content inside the resource directory immediately', function]);expect($this->processedSaves)->toBe(1);- Process::assertRan(fn ($process) => str_contains($process->command, base64_encode('inside-content')));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), base64_encode('inside-content')));});it('does not write a mount outside the resource directory before a deployment', function () {@@ -147,5 +147,5 @@ it('does not write a mount outside the resource directory before a deployment',]);expect($this->processedSaves)->toBe(0);- Process::assertNotRan(fn ($process) => str_contains($process->command, base64_encode('outside-without-compose')));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), base64_encode('outside-without-compose')));});diff --git a/tests/Feature/Service/DeleteServiceSubresourceTest.php b/tests/Feature/Service/DeleteServiceSubresourceTest.phpindex 97bdd0d0f6..2955984e19 100644--- a/tests/Feature/Service/DeleteServiceSubresourceTest.php+++ b/tests/Feature/Service/DeleteServiceSubresourceTest.php@@ -82,7 +82,7 @@ it('deletes a service part from Coolify when the server stops responding while s->assertNotDispatched('error');expect($model::find($this->{$part}->id))->toBeNull();- Process::assertRan(fn ($process) => str_contains($process->command, 'docker rm -f $container_ids'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker rm -f $container_ids'));Log::shouldHaveReceived('warning')->withArgs(fn (string $message) => str_contains($message, 'service part'))->atLeast()->once();})->with('service parts');@@ -107,5 +107,5 @@ it('removes the container of a service part and reports success when the server->assertNotDispatched('warning');expect($model::find($this->{$part}->id))->toBeNull();- Process::assertRan(fn ($process) => str_contains($process->command, "coolify.service.subUuid={$this->{$part}->uuid}"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "coolify.service.subUuid={$this->{$part}->uuid}"));})->with('service parts');diff --git a/tests/Feature/Service/StartDatabaseProxyTest.php b/tests/Feature/Service/StartDatabaseProxyTest.phpindex a0e98a9241..f67976f485 100644--- a/tests/Feature/Service/StartDatabaseProxyTest.php+++ b/tests/Feature/Service/StartDatabaseProxyTest.php@@ -93,7 +93,7 @@ test('database proxy resolves the database container on every connection', funct$commands = [];Process::fake(function ($process) use (&$commands) {- $commands[] = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $commands[] = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: '');});@@ -126,7 +126,7 @@ test('database proxy runs Docker Compose only with the compose file that Coolify$commands = [];Process::fake(function ($process) use (&$commands) {- $commands[] = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $commands[] = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: '');});diff --git a/tests/Feature/ServiceComposeStdinWriteTest.php b/tests/Feature/ServiceComposeStdinWriteTest.phpindex 2f7844c05f..d24b8a6580 100644--- a/tests/Feature/ServiceComposeStdinWriteTest.php+++ b/tests/Feature/ServiceComposeStdinWriteTest.php@@ -73,7 +73,7 @@ it('streams a large compose file over stdin instead of the command line', functi->and($envWrite['command'])->toMatch('/'.preg_quote($tee, '/').' \S+\.env\.tmp/');$this->processes->each(fn (array $process) => expect(strlen($process['command']))->toBeLessThan(8 * 1024));- expect($this->processes->last()['command'])->toMatch('/mv \S+\.env\.tmp '.preg_quote($workdir, '/').'\/\.env/');+ expect($this->processes->last()['command']."\n".$this->processes->last()['input'])->toMatch('/mv \S+\.env\.tmp '.preg_quote($workdir, '/').'\/\.env/');})->with(['root' => ['root', 'tee'],'non-root' => ['cooluser', 'sudo tee'],diff --git a/tests/Feature/ServiceParserReferenceTest.php b/tests/Feature/ServiceParserReferenceTest.phpindex fb7a0aed60..95f5d108d2 100644--- a/tests/Feature/ServiceParserReferenceTest.php+++ b/tests/Feature/ServiceParserReferenceTest.php@@ -31,6 +31,7 @@use App\Models\Application;use App\Models\ApplicationPreview;use App\Models\Environment;+use App\Models\EnvironmentVariable;use App\Models\InstanceSettings;use App\Models\Project;use App\Models\Server;@@ -185,6 +186,12 @@ it('parses the application compose exactly as recorded', function (array $option]);$application->forceFill(['uuid' => 'referenceapplication0001'])->save();$application->settings->update($options['settings'] ?? []);+ // Creating the production variable also creates its preview copy; the preview gets its own value.+ EnvironmentVariable::create([+ 'key' => 'MAP_EMPTY_FROM_DATABASE', 'value' => 'production-value', 'is_preview' => false,+ 'resourceable_type' => Application::class, 'resourceable_id' => $application->id,+ ]);+ $application->environment_variables_preview()->where('key', 'MAP_EMPTY_FROM_DATABASE')->firstOrFail()->update(['value' => 'preview-value']);$commit = 'abcdef0123456789abcdef0123456789abcdef01';$snapshot = [];diff --git a/tests/Feature/SettingsBackupEmptyStateTest.php b/tests/Feature/SettingsBackupEmptyStateTest.phpindex 30392acc49..f9c4ef9faf 100644--- a/tests/Feature/SettingsBackupEmptyStateTest.php+++ b/tests/Feature/SettingsBackupEmptyStateTest.php@@ -208,7 +208,7 @@ test('configure backup creates the instance database instead of using a team datStorage::fake('ssh-mux');Server::query()->whereKey(0)->update(['private_key_id' => PrivateKey::factory()->create(['team_id' => 0])->id]);Server::flushIdentityMap();- Process::fake([+ fakeRemoteProcesses(['*docker inspect coolify-db*' => Process::result(output: json_encode([['Config' => ['Env' => ['POSTGRES_USER=coolify', 'POSTGRES_PASSWORD=instance-secret', 'POSTGRES_DB=coolify']],]])),diff --git a/tests/Feature/SqliteConnectApplicationTest.php b/tests/Feature/SqliteConnectApplicationTest.phpindex 9757498c80..3cdfd40de0 100644--- a/tests/Feature/SqliteConnectApplicationTest.php+++ b/tests/Feature/SqliteConnectApplicationTest.php@@ -264,8 +264,8 @@ it('keeps a docker volume shared with another resource when the application volu$this->application->deleteVolumes();- Process::assertRan(fn ($process) => str_contains($process->command, 'docker volume rm -f '.escapeshellarg($this->application->uuid.'-data')));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'sqlite-data-'.$this->sqlite->uuid));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker volume rm -f '.escapeshellarg($this->application->uuid.'-data')));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'sqlite-data-'.$this->sqlite->uuid));});it('keeps the sqlite data volume when the database volumes are deleted while an application mounts it', function () {@@ -275,7 +275,7 @@ it('keeps the sqlite data volume when the database volumes are deleted while an$this->sqlite->deleteVolumes();- Process::assertNotRan(fn ($process) => str_contains($process->command, $databaseVolume->name));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), $databaseVolume->name));});it('removes the sqlite data volume when no application mounts it', function () {@@ -284,7 +284,7 @@ it('removes the sqlite data volume when no application mounts it', function () {$this->sqlite->deleteVolumes();- Process::assertRan(fn ($process) => str_contains($process->command, 'docker volume rm -f '.escapeshellarg($databaseVolume->name)));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker volume rm -f '.escapeshellarg($databaseVolume->name)));});function connectSqliteVolume(Application $application, StandaloneSqlite $sqlite): LocalPersistentVolumediff --git a/tests/Feature/SshBatchStdinTransportTest.php b/tests/Feature/SshBatchStdinTransportTest.phpnew file mode 100644index 0000000000..d190760289--- /dev/null+++ b/tests/Feature/SshBatchStdinTransportTest.php@@ -0,0 +1,290 @@+<?php++use App\Actions\CoolifyTask\RunRemoteProcess;+use App\Enums\ProcessStatus;+use App\Exceptions\DeploymentException;+use App\Models\ApplicationDeploymentQueue;+use App\Models\PrivateKey;+use App\Models\Server;+use App\Models\User;+use App\Services\DatabaseStartCommandExecutor;+use App\Services\DockerRegistryLogins;+use App\Support\RemoteProcessCommand;+use App\Traits\ExecuteRemoteCommand;+use Illuminate\Database\Eloquent\Model;+use Illuminate\Foundation\Testing\RefreshDatabase;+use Illuminate\Support\Collection;+use Illuminate\Support\Facades\File;+use Illuminate\Support\Facades\Storage;+use Spatie\Activitylog\Models\Activity;++/**+ * Remote command batches must reach ssh on stdin. A batch inside the argv of a local process+ * (for example a heredoc in `sh -c`) shows its secrets to every user of `ps` on the Coolify host.+ *+ * These tests run the real generated command. A stub `ssh` on PATH records its argv, the argv of+ * every ancestor process up to PHP, and its stdin, then runs the remote shell command locally.+ */+uses(RefreshDatabase::class);++const SSH_STDIN_SECRET = 'rr-n5-secret-Zq81xLp';++beforeEach(function () {+ config()->set('constants.ssh.mux_enabled', false);+ config()->set('constants.ssh.max_retries', 3);+ config()->set('constants.ssh.retry_base_delay', 0);+ config()->set('constants.ssh.retry_max_delay', 0);+ Server::flushIdentityMap();++ $this->stubDir = sys_get_temp_dir().'/ssh-stdin-stub-'.uniqid();+ File::ensureDirectoryExists($this->stubDir.'/bin');+ File::ensureDirectoryExists($this->stubDir.'/log');++ $log = $this->stubDir.'/log';+ $phpPid = getmypid();+ File::put($this->stubDir.'/bin/ssh', <<<BASH+#!/bin/bash+id="\$(date +%s%N).\$\$"+printf '%s\\0' "\$@" > "{$log}/argv.\$id"+pid=\$\$+while [ "\$pid" -gt 1 ] && [ "\$pid" != "{$phpPid}" ]; do+ tr '\\0' ' ' < /proc/\$pid/cmdline >> "{$log}/ancestors.\$id"+ echo >> "{$log}/ancestors.\$id"+ pid=\$(sed 's/.*) //' /proc/\$pid/stat | cut -d' ' -f2)+done+tee "{$log}/stdin.\$id" | sh -c "\${@: -1}"+BASH);+ File::put($this->stubDir.'/bin/sudo', "#!/bin/sh\nexec \"\$@\"\n");+ chmod($this->stubDir.'/bin/ssh', 0755);+ chmod($this->stubDir.'/bin/sudo', 0755);+ File::put($this->stubDir.'/bin/docker', "#!/bin/sh\n[ \"\$3\" = login ] && { cat > \"{$log}/docker-login.stdin\"; echo 'Login Succeeded'; }\nexit 0\n");+ chmod($this->stubDir.'/bin/docker', 0755);++ $this->originalPath = getenv('PATH');+ $path = $this->stubDir.'/bin:'.$this->originalPath;+ putenv("PATH={$path}");+ $_SERVER['PATH'] = $_ENV['PATH'] = $path;+});++afterEach(function () {+ putenv("PATH={$this->originalPath}");+ $_SERVER['PATH'] = $_ENV['PATH'] = $this->originalPath;+ File::deleteDirectory($this->stubDir);+ Server::flushIdentityMap();+});++function sshStdinStubServer(string $user = 'root'): Server+{+ $owner = User::factory()->create();+ $team = $owner->teams()->first();++ Storage::fake('ssh-keys');+ $privateKey = PrivateKey::factory()->create(['team_id' => $team->id]);+ Storage::disk('ssh-keys')->put("ssh_key@{$privateKey->uuid}", $privateKey->private_key);++ return Server::factory()->create([+ 'team_id' => $team->id,+ 'private_key_id' => $privateKey->id,+ 'user' => $user,+ ]);+}++/**+ * @return array{invocations: int, argv: string, stdin: string}+ */+function sshStdinStubRecords(string $stubDir): array+{+ $files = fn (string $prefix) => glob("{$stubDir}/log/{$prefix}.*") ?: [];+ $read = fn (string $prefix) => collect($files($prefix))->map(fn (string $file) => (string) file_get_contents($file))->implode("\n");++ return [+ 'invocations' => count($files('argv')),+ 'argv' => $read('argv')."\n".$read('ancestors'),+ 'stdin' => $read('stdin'),+ ];+}++function sshStdinRemoteProcessActivity(Server $server, string $command): Activity+{+ return activity()+ ->withProperties([+ 'server_uuid' => $server->uuid,+ ...RemoteProcessCommand::properties($command),+ 'type' => 'inline',+ 'status' => ProcessStatus::QUEUED->value,+ 'team_id' => $server->team_id,+ ])+ ->event('inline')+ ->log('[]');+}++function sshStdinDeploymentRunner(Server $server): object+{+ $queue = ApplicationDeploymentQueue::create([+ 'deployment_uuid' => 'ssh-stdin-deployment',+ 'application_id' => 0,+ 'server_id' => $server->id,+ 'status' => 'in_progress',+ ]);++ return new class($server, $queue)+ {+ use ExecuteRemoteCommand;++ public Collection $saved_outputs;++ public function __construct(public Server $server, public ApplicationDeploymentQueue $application_deployment_queue)+ {+ $this->saved_outputs = collect();+ }+ };+}++it('runs an instant remote batch with its secret only on ssh stdin', function (string $user) {+ $server = sshStdinStubServer($user);++ $output = instant_remote_process([+ 'export DB_PASSWORD='.SSH_STDIN_SECRET,+ 'test -n "$DB_PASSWORD"',+ 'echo "len=${#DB_PASSWORD} literal=\$HOME tick=`echo t`"',+ ], $server);++ $records = sshStdinStubRecords($this->stubDir);+ expect($output)->toBe('len='.strlen(SSH_STDIN_SECRET).' literal=$HOME tick=t')+ ->and($records['invocations'])->toBe(1)+ ->and($records['stdin'])->toContain(SSH_STDIN_SECRET)+ ->and($records['argv'])->toContain('bash -se')+ ->not->toContain(SSH_STDIN_SECRET);+ if ($user !== 'root') {+ expect($records['stdin'])->toContain('sudo test -n');+ }+})->with(['root' => 'root', 'non-root' => 'rr-deployer']);++it('does not retry a remote command that fails with exit code 1', function () {+ $server = sshStdinStubServer();++ expect(fn () => instant_remote_process(['echo '.SSH_STDIN_SECRET.' > /dev/null', 'echo failed >&2', 'exit 1', 'echo never'], $server))+ ->toThrow(RuntimeException::class, 'failed');++ $records = sshStdinStubRecords($this->stubDir);+ expect($records['invocations'])->toBe(1)+ ->and($records['argv'])->not->toContain(SSH_STDIN_SECRET);+});++it('retries the whole batch when ssh exits with the transport code 255', function () {+ $server = sshStdinStubServer();++ expect(fn () => instant_remote_process(['echo '.SSH_STDIN_SECRET.' > /dev/null', 'echo "Connection refused" >&2', 'exit 255'], $server))+ ->toThrow(RuntimeException::class);++ $records = sshStdinStubRecords($this->stubDir);+ expect($records['invocations'])->toBe(3)+ ->and($records['argv'])->not->toContain(SSH_STDIN_SECRET);+});++it('returns nothing for a failed batch with errors ignored', function () {+ $server = sshStdinStubServer();++ expect(instant_remote_process(['exit 3'], $server, throwError: false))->toBeNull();+});++it('runs batches larger than the 128 KiB argument limit and returns large output', function () {+ $server = sshStdinStubServer();+ $payload = str_repeat('a', 200 * 1024);++ $output = instant_remote_process([+ "PAYLOAD='{$payload}'",+ 'echo "${#PAYLOAD}"',+ 'head -c 3000000 /dev/zero | tr "\0" "b"',+ ], $server);++ expect(strlen($output))->toBe(strlen((string) (200 * 1024)) + 1 + 3000000)+ ->and($output)->toStartWith((200 * 1024)."\nbbb");+});++it('streams remote process output into the activity log with the secret only on stdin', function (string $user) {+ $server = sshStdinStubServer($user);+ $activity = sshStdinRemoteProcessActivity($server, implode("\n", [+ 'export TOKEN='.SSH_STDIN_SECRET,+ 'echo first',+ 'sleep 0.5',+ 'echo second',+ ]));++ $result = (new RunRemoteProcess($activity))();++ $entries = collect(json_decode($activity->fresh()->description, true));+ $records = sshStdinStubRecords($this->stubDir);+ expect($result->exitCode())->toBe(0)+ ->and($entries->pluck('output')->map(fn ($output) => trim($output))->all())->toBe(['first', 'second'])+ ->and($activity->fresh()->getExtraProperty('status'))->toBe(ProcessStatus::FINISHED->value)+ ->and($records['stdin'])->toContain(SSH_STDIN_SECRET)+ ->and($records['argv'])->not->toContain(SSH_STDIN_SECRET);+})->with(['root' => 'root', 'non-root' => 'rr-deployer']);++it('keeps the exit code of a failed remote process and honours ignore_errors', function () {+ $server = sshStdinStubServer();++ $failing = sshStdinRemoteProcessActivity($server, "echo broken >&2\nexit 7");+ expect(fn () => (new RunRemoteProcess($failing))())->toThrow(RuntimeException::class, 'broken');+ expect($failing->fresh()->getExtraProperty('exitCode'))->toBe(7)+ ->and($failing->fresh()->getExtraProperty('status'))->toBe(ProcessStatus::ERROR->value);++ $ignored = sshStdinRemoteProcessActivity($server, 'exit 7');+ expect((new RunRemoteProcess($ignored, ignore_errors: true))()->exitCode())->toBe(7);+});++it('runs deployment commands with the secret only on stdin and streams their output', function (string $user) {+ $server = sshStdinStubServer($user);+ $runner = sshStdinDeploymentRunner($server);++ $runner->execute_remote_command(+ ['command' => 'test -n '.SSH_STDIN_SECRET.' && echo deploy-42', 'hidden' => true],+ ['command' => 'cat', 'input' => "from-input\n"],+ ['command' => 'false', 'ignore_errors' => true],+ );++ $logs = collect(json_decode($runner->application_deployment_queue->fresh()->logs, true))->pluck('output');+ $records = sshStdinStubRecords($this->stubDir);+ expect($logs->all())->toContain('deploy-42', 'from-input')+ ->and($records['argv'])->not->toContain(SSH_STDIN_SECRET)+ ->and($records['stdin'])->toContain(SSH_STDIN_SECRET);+})->with(['root' => 'root', 'non-root' => 'rr-deployer']);++it('fails a deployment command with its remote exit code', function () {+ $server = sshStdinStubServer();+ $runner = sshStdinDeploymentRunner($server);++ expect(fn () => $runner->execute_remote_command(['command' => 'echo nope >&2; exit 4']))+ ->toThrow(DeploymentException::class, 'exit code 4');+ expect(sshStdinStubRecords($this->stubDir)['invocations'])->toBe(1);+});++it('starts a database with its password only on ssh stdin', function (string $user) {+ $server = sshStdinStubServer($user);+ $database = new class extends Model {};+ $database->setRelation('destination', (object) ['server' => $server]);+ $activity = activity()->withProperties(['status' => ProcessStatus::QUEUED->value])->event('inline')->log('[]');++ app(DatabaseStartCommandExecutor::class)->execute([+ 'export POSTGRES_PASSWORD='.SSH_STDIN_SECRET,+ 'echo "started ${#POSTGRES_PASSWORD}"',+ ], $database, $activity);++ $records = sshStdinStubRecords($this->stubDir);+ expect($activity->fresh()->getExtraProperty('status'))->toBe(ProcessStatus::FINISHED->value)+ ->and(collect(json_decode($activity->fresh()->description, true))->pluck('output')->implode(''))->toContain('started '.strlen(SSH_STDIN_SECRET))+ ->and($records['stdin'])->toContain(SSH_STDIN_SECRET)+ ->and($records['argv'])->not->toContain(SSH_STDIN_SECRET);+})->with(['root' => 'root', 'non-root' => 'rr-deployer']);++it('logs in to a registry with the password only on ssh stdin', function (string $user) {+ $server = sshStdinStubServer($user);++ DockerRegistryLogins::login($server, 'registry.example.com', 'deployer', SSH_STDIN_SECRET);++ $records = sshStdinStubRecords($this->stubDir);+ expect(file_get_contents($this->stubDir.'/log/docker-login.stdin'))->toBe(SSH_STDIN_SECRET)+ ->and($records['argv'])->not->toContain(SSH_STDIN_SECRET)+ ->not->toContain(base64_encode(SSH_STDIN_SECRET));+})->with(['root' => 'root', 'non-root' => 'rr-deployer']);diff --git a/tests/Feature/StartClickhouseVolumesTest.php b/tests/Feature/StartClickhouseVolumesTest.phpindex 8193cdb95e..dc4c96b80b 100644--- a/tests/Feature/StartClickhouseVolumesTest.php+++ b/tests/Feature/StartClickhouseVolumesTest.php@@ -56,7 +56,7 @@ function startedClickhouseCompose(): array{$compose = null;Process::assertRan(function ($process) use (&$compose) {- if (preg_match("/echo '([A-Za-z0-9+\/=]+)' \| base64 -d \| (?:sudo )?tee \S+\/docker-compose\.yml/", $process->command, $matches)) {+ if (preg_match("/echo '([A-Za-z0-9+\/=]+)' \| base64 -d \| (?:sudo )?tee \S+\/docker-compose\.yml/", remoteProcessScript($process), $matches)) {$compose = Yaml::parse(base64_decode($matches[1]));}@@ -91,7 +91,7 @@ it('keeps the data volume when a file mount is added', function () {function fakeClickhouseDataMount(string $volumeName): void{Process::fake(fn ($process) => Process::result(- output: str_contains($process->command, 'docker inspect') ? "volume {$volumeName}" : 'OK',+ output: str_contains(remoteProcessScript($process), 'docker inspect') ? "volume {$volumeName}" : 'OK',));}@@ -99,7 +99,7 @@ function ranRemoteCommandContaining(string $needle): bool{$ran = false;Process::assertRan(function ($process) use ($needle, &$ran) {- $ran = $ran || str_contains($process->command, $needle);+ $ran = $ran || str_contains(remoteProcessScript($process), $needle);return true;});diff --git a/tests/Feature/StartDatabaseImportBackupPreparationTest.php b/tests/Feature/StartDatabaseImportBackupPreparationTest.phpindex b49d607a84..1741d157e7 100644--- a/tests/Feature/StartDatabaseImportBackupPreparationTest.php+++ b/tests/Feature/StartDatabaseImportBackupPreparationTest.php@@ -62,7 +62,7 @@ beforeEach(function () {function importServerBackup(object $test, string $path, string $magicHex): string{- Process::fake([+ fakeRemoteProcesses(['*stat -c %s*' => Process::result('2048'),'*od -An -tx1*' => Process::result(' '.implode(' ', str_split($magicHex, 2))),'*' => Process::result(''),@@ -149,7 +149,7 @@ test('bz2, xz, and zip server backups are prepared in the helper image', functio->toContain("docker rm -f 'backup-decompress-")->not->toContain("docker cp '{$path}' '{$this->database->uuid}:");- Process::assertRan(fn ($process) => str_contains($process->command, "docker run -d --network none --name 'backup-decompress-"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "docker run -d --network none --name 'backup-decompress-"));Queue::assertPushed(CoolifyTask::class, fn (CoolifyTask $job) => str_starts_with((string) ($job->call_event_data['containerName'] ?? ''), 'backup-decompress-'));})->with(['bz2' => ['/srv/backups/app.sql.bz2', '425a68393141'],diff --git a/tests/Feature/StartDatabaseImportS3CredentialsTest.php b/tests/Feature/StartDatabaseImportS3CredentialsTest.phpindex b66750e6f7..c53b343711 100644--- a/tests/Feature/StartDatabaseImportS3CredentialsTest.php+++ b/tests/Feature/StartDatabaseImportS3CredentialsTest.php@@ -12,15 +12,19 @@ use App\Models\StandaloneDocker;use App\Models\StandalonePostgresql;use App\Models\Team;use App\Support\DatabaseImport\DatabaseImportCleanup;+use App\Support\DatabaseImport\DatabaseImportException;use App\Support\DatabaseImport\DatabaseImportSource;use App\Support\RemoteProcessCommand;use Illuminate\Filesystem\FilesystemAdapter;use Illuminate\Filesystem\FilesystemManager;use Illuminate\Foundation\Testing\RefreshDatabase;+use Illuminate\Support\Collection;use Illuminate\Support\Facades\Process;use Illuminate\Support\Facades\Queue;use Illuminate\Support\Facades\Storage;use Illuminate\Support\Str;+use Spatie\Activitylog\Models\Activity;+use Symfony\Component\Process\Process as SymfonyProcess;uses(RefreshDatabase::class);@@ -94,7 +98,7 @@ test('s3 import activity command does not contain storage key or secret', functi->not->toContain('.env')->not->toContain('S3_ACCESS_KEY=')->not->toContain('S3_SECRET_KEY=')- ->toContain('mc alias set s3temp "$S3_ENDPOINT" "$S3_ACCESS_KEY" "$S3_SECRET_KEY"');+ ->not->toContain('mc alias set');Queue::assertPushed(CoolifyTask::class, function (CoolifyTask $job) {$cleanup = $job->call_event_data;@@ -186,6 +190,134 @@ test('s3 import helper signs requests with the storage region', function () {$this->team->id,);- Process::assertRan(fn ($process) => str_contains($process->command, "-e 'MC_REGION=ru-central-1'")- && str_contains($process->command, '-e S3_ENDPOINT='));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "-e 'MC_REGION=ru-central-1'")+ && str_contains(remoteProcessScript($process), 'docker run -d'));+});++const IMPORT_S3_ARGV_KEY = 'AK/+$x"y';+const IMPORT_S3_ARGV_SECRET = "s3c/r+e\$t 'q\"uo\\te`d` \\n";++/**+ * Starts an S3 import against a faked SSH layer and records every SSH process with its stdin.+ *+ * @return array{0: Collection<int, array{command: string, input: ?string}>, 1: ?Activity, 2: ?Throwable}+ */+function importS3ArgvTestRun(object $test, string $sshUser, bool $aliasFails = false): array+{+ $test->server->update(['user' => $sshUser]);+ $storage = S3Storage::create([+ 'name' => 'Import S3',+ 'region' => 'ru-central-1',+ 'key' => IMPORT_S3_ARGV_KEY,+ 'secret' => IMPORT_S3_ARGV_SECRET,+ 'bucket' => 'test-bucket',+ 'endpoint' => 'https://8.8.8.8',+ 'is_usable' => true,+ 'team_id' => $test->team->id,+ ]);++ $disk = Mockery::mock(FilesystemAdapter::class);+ $disk->shouldReceive('exists')->andReturn(true);+ $disk->shouldReceive('size')->andReturn(1024);+ $filesystem = Mockery::mock(FilesystemManager::class, [app()])->makePartial();+ $filesystem->shouldReceive('build')->andReturn($disk);+ Storage::swap($filesystem);++ $processes = collect();+ Process::fake(function ($process) use ($processes, $aliasFails) {+ $processes->push(['command' => $process->command, 'input' => $process->input]);++ return $aliasFails && str_contains($process->command, 'mc alias set')+ ? Process::result(errorOutput: 'mc: invalid credentials', exitCode: 1)+ : Process::result();+ });+ Queue::fake();++ $activity = null;+ $error = null;+ try {+ $activity = app(StartDatabaseImport::class)->handle(+ $test->database->fresh(),+ new DatabaseImportSource('s3', path: 'backups/restore.sql', s3StorageUuid: $storage->uuid),+ $test->team->id,+ );+ } catch (Throwable $e) {+ $error = $e;+ }++ return [$processes, $activity, $error];+}++/**+ * Runs the recorded `mc alias set` SSH process in a real shell with its stdin. Stubs for `timeout`, `ssh`+ * (runs the remote command in `sh -c`, like a login shell), `sudo`, `docker` and `mc` log their arguments.+ *+ * @param array{command: string, input: ?string} $aliasProcess+ * @return array{argv: string, mcStdin: string}+ */+function importS3ArgvTestRunAliasProcess(array $aliasProcess): array+{+ $bin = sys_get_temp_dir().'/import-s3-argv-'.bin2hex(random_bytes(6));+ mkdir($bin);+ $log = "{$bin}/argv.log";+ $stubs = [+ 'timeout' => "shift\nexec \"\$@\"",+ 'ssh' => "for last; do :; done\nexec sh -c \"\$last\"",+ 'sudo' => 'exec "$@"',+ 'docker' => "[ \"\$1\" = exec ] || exit 0\nshift\nwhile [ \"\${1#-}\" != \"\$1\" ]; do shift; done\nshift\nexec \"\$@\"",+ 'mc' => "cat > {$bin}/mc.stdin",+ ];+ foreach ($stubs as $name => $body) {+ file_put_contents("{$bin}/{$name}", "#!/bin/sh\nprintf '%s\\n' \"{$name} \$*\" >> {$log}\n{$body}\n");+ chmod("{$bin}/{$name}", 0755);+ }++ try {+ (new SymfonyProcess(['bash', '-c', $aliasProcess['command']], env: ['PATH' => "{$bin}:/usr/bin:/bin"], input: $aliasProcess['input']))->mustRun();++ return ['argv' => (string) file_get_contents($log), 'mcStdin' => (string) file_get_contents("{$bin}/mc.stdin")];+ } finally {+ array_map('unlink', glob("{$bin}/*"));+ rmdir($bin);+ }+}++it('passes the S3 keys of an import to mc on stdin, not in a command line', function (string $sshUser) {+ [$processes, $activity, $error] = importS3ArgvTestRun($this, $sshUser);++ expect($error)->toBeNull();+ $commands = $processes->pluck('command')->push((string) RemoteProcessCommand::read($activity));+ foreach ($commands as $command) {+ expect($command)+ ->not->toContain(IMPORT_S3_ARGV_SECRET)+ ->not->toContain(escapeshellarg(IMPORT_S3_ARGV_SECRET))+ ->not->toContain(escapeshellarg(IMPORT_S3_ARGV_KEY))+ ->not->toContain(base64_encode(IMPORT_S3_ARGV_SECRET))+ ->not->toContain(base64_encode(IMPORT_S3_ARGV_KEY));+ }++ $aliasProcess = $processes->sole(fn (array $process) => str_contains($process['command'], 'mc alias set'));+ $run = importS3ArgvTestRunAliasProcess($aliasProcess);++ expect($run['mcStdin'])->toBe(IMPORT_S3_ARGV_KEY."\n".IMPORT_S3_ARGV_SECRET."\n")+ ->and($run['argv'])->toContain('mc alias set s3temp https://8.8.8.8')+ ->and($run['argv'])->not->toContain(IMPORT_S3_ARGV_SECRET)+ ->and($run['argv'])->not->toContain(IMPORT_S3_ARGV_KEY)+ ->and($processes->contains(fn (array $process) => str_contains($process['command']."\n".$process['input'], "-e 'MC_REGION=ru-central-1'")))->toBeTrue()+ ->and((string) RemoteProcessCommand::read($activity))->toContain('mc cp '.escapeshellarg('s3temp/test-bucket/backups/restore.sql'));+ if ($sshUser !== 'root') {+ expect($run['argv'])->toContain('sudo docker exec -i');+ }+})->with(['root', 'deploy']);++it('removes the S3 restore helper when the mc alias cannot be set', function () {+ [$processes, $activity, $error] = importS3ArgvTestRun($this, 'root', aliasFails: true);++ expect($activity)->toBeNull()+ ->and($error)->toBeInstanceOf(DatabaseImportException::class);+ $aliasIndex = $processes->search(fn (array $process) => str_contains($process['command'], 'mc alias set'));+ $removedAfterAlias = $processes->slice($aliasIndex + 1)+ ->contains(fn (array $process) => preg_match("/docker rm -f '?s3-restore-/", $process['command']."\n".$process['input']) === 1);+ expect($removedAfterAlias)->toBeTrue();+ Queue::assertNothingPushed();});diff --git a/tests/Feature/StopApplicationFailureTest.php b/tests/Feature/StopApplicationFailureTest.phpindex b7ad6b4b56..bf0f2cb90a 100644--- a/tests/Feature/StopApplicationFailureTest.php+++ b/tests/Feature/StopApplicationFailureTest.php@@ -56,11 +56,11 @@ beforeEach(function () {function fakeRunningContainerWithStopResult(ProcessResult $stopResult): void{Process::fake(function ($process) use ($stopResult) {- if (str_contains($process->command, 'docker ps -a --filter')) {+ if (str_contains(remoteProcessScript($process), 'docker ps -a --filter')) {return Process::result(output: json_encode(['Names' => 'app-container', 'Labels' => '', 'State' => 'running']));}- return str_contains($process->command, 'docker stop') ? $stopResult : Process::result(output: '');+ return str_contains(remoteProcessScript($process), 'docker stop') ? $stopResult : Process::result(output: '');});}diff --git a/tests/Feature/StorageAndServiceAuditTest.php b/tests/Feature/StorageAndServiceAuditTest.phpindex e0520fa867..6b9a3974d3 100644--- a/tests/Feature/StorageAndServiceAuditTest.php+++ b/tests/Feature/StorageAndServiceAuditTest.php@@ -47,7 +47,7 @@ beforeEach(function () {Bus::fake();// Remote `test -f` / `test -d` probes report a missing path; path confinement checks pass.Process::fake(fn ($process) => Process::result(- output: str_contains(implode(' ', (array) $process->command), '&& echo OK || echo NOK') ? 'NOK' : 'OK'+ output: str_contains(implode(' ', (array) remoteProcessScript($process)), '&& echo OK || echo NOK') ? 'NOK' : 'OK'));Server::flushIdentityMap();diff --git a/tests/Feature/StorageHostPathTest.php b/tests/Feature/StorageHostPathTest.phpindex 37239aa74d..856037e55e 100644--- a/tests/Feature/StorageHostPathTest.php+++ b/tests/Feature/StorageHostPathTest.php@@ -64,9 +64,9 @@ beforeEach(function () {function storageHostPathFakeServer(string $symlink = 'OK', string $exists = 'NOK'): void{Process::fake(fn ($process) => Process::result(output: match (true) {- str_contains($process->command, 'test -L') => $symlink,- str_contains($process->command, 'readlink -f') => 'OK',- str_contains($process->command, 'test -') => $exists,+ str_contains(remoteProcessScript($process), 'test -L') => $symlink,+ str_contains(remoteProcessScript($process), 'readlink -f') => 'OK',+ str_contains(remoteProcessScript($process), 'test -') => $exists,default => '',}));}@@ -138,7 +138,7 @@ function storageHostPathFileVolume(string $fsPath, ?string $content, bool $isDirfunction storageHostPathAssertNoDelete(): void{- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'rm -') || str_contains($process->command, 'rmdir'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'rm -') || str_contains(remoteProcessScript($process), 'rmdir'));}test('an admin creates a directory mount anywhere on the host in the UI', function () {@@ -153,7 +153,7 @@ test('an admin creates a directory mount anywhere on the host in the UI', functi->and($volume->is_directory)->toBeTrue();$volume->saveStorageOnServer();- Process::assertRan(fn ($process) => str_contains($process->command, "mkdir -p '/srv/zz'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "mkdir -p '/srv/zz'"));});test('an admin creates a file mount with content anywhere on the host in the UI', function () {@@ -169,8 +169,8 @@ test('an admin creates a file mount with content anywhere on the host in the UI'->and($volume->mount_path)->toBe('/etc/app.conf');$volume->saveStorageOnServer();- Process::assertRan(fn ($process) => str_contains($process->command, "mkdir -p '/etc/zz'"));- Process::assertRan(fn ($process) => preg_match("#base64 -d \\| (sudo )?tee '/etc/zz/app.conf' > /dev/null#", $process->command) === 1);+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "mkdir -p '/etc/zz'"));+ Process::assertRan(fn ($process) => preg_match("#base64 -d \\| (sudo )?tee '/etc/zz/app.conf' > /dev/null#", remoteProcessScript($process)) === 1);expect($volume->contentPathOnServer())->toBe('/etc/zz/app.conf');});@@ -252,8 +252,8 @@ test('a symbolic link at a file path outside the resource directory is refused',expect(fn () => $volume->saveStorageOnServer())->toThrow(RuntimeException::class, '/etc/zz/app.conf is a symbolic link on the server.');- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'base64 -d'));- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'touch '));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'base64 -d'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'touch '));$this->withHeaders(['Authorization' => 'Bearer '.storageHostPathToken($this->admin)])->postJson("/api/v1/applications/{$this->application->uuid}/storages", [@@ -290,7 +290,7 @@ test('deleting a mount inside the resource directory still deletes it on the hosstorageHostPathFileVolume($path, null, isDirectory: true)->deleteStorageOnServer();- Process::assertRan(fn ($process) => str_contains($process->command, "rm -rf '{$path}'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "rm -rf '{$path}'"));});test('the UI deletes a mount inside the resource directory on the host only when the user keeps the option checked', function (bool $isDirectory, array $selectedActions, bool $deletesOnHost, string $message) {@@ -304,7 +304,7 @@ test('the UI deletes a mount inside the resource directory on the host only whenexpect($volume->fresh())->toBeNull();if ($deletesOnHost) {- Process::assertRan(fn ($process) => str_contains($process->command, "rm -rf '{$path}'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "rm -rf '{$path}'"));} else {storageHostPathAssertNoDelete();}@@ -333,10 +333,10 @@ test('the delete dialog offers deletion on the host only for mounts inside the rtest('a file outside the resource directory never replaces a directory on the host', function () {Process::fake(fn ($process) => Process::result(output: match (true) {- str_contains($process->command, 'test -L') => 'OK',- str_contains($process->command, 'test -d') => 'OK',- str_contains($process->command, 'test -f') => 'NOK',- str_contains($process->command, 'sh -c') => '1:empty-directory',+ str_contains(remoteProcessScript($process), 'test -L') => 'OK',+ str_contains(remoteProcessScript($process), 'test -d') => 'OK',+ str_contains(remoteProcessScript($process), 'test -f') => 'NOK',+ str_contains(remoteProcessScript($process), 'sh -c') => '1:empty-directory',default => '',}));@@ -364,8 +364,8 @@ test('relative host paths stay inside the resource directory', function () {expect(LocalFileVolume::query()->orderBy('id')->pluck('fs_path')->all())->toBe([$workdir.'/data/zz', $workdir.'/config/app.conf']);- Process::assertRan(fn ($process) => str_contains($process->command, 'readlink -f')- && str_contains($process->command, "'{$workdir}' '{$workdir}/data/zz'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'readlink -f')+ && str_contains(remoteProcessScript($process), "'{$workdir}' '{$workdir}/data/zz'"));});test('a token without deploy cannot create a mount outside the resource directory', function (string $resourceType, array $mount) {diff --git a/tests/Feature/TeamTwoFactorRequirementTest.php b/tests/Feature/TeamTwoFactorRequirementTest.phpindex 45334f57da..ebee17ca9c 100644--- a/tests/Feature/TeamTwoFactorRequirementTest.php+++ b/tests/Feature/TeamTwoFactorRequirementTest.php@@ -1,7 +1,9 @@<?php+use App\Livewire\SwitchTeam;use App\Livewire\Team\Index;use App\Models\InstanceSettings;+use App\Models\OauthIdentity;use App\Models\Team;use App\Models\User;use Illuminate\Foundation\Testing\RefreshDatabase;@@ -113,3 +115,96 @@ test('the setup page shows the error for a wrong 2fa code', function () {->assertOk()->assertSee('The provided two factor authentication code was invalid.');});++test('a member without 2fa can confirm with a passkey on the setup page', function () {+ $user = twoFactorTeamUser($this->team);+ $this->team->update(['is_two_factor_required' => true]);+ $this->actingAs($user)->withSession(['currentTeam' => $this->team]);++ $this->getJson('/passkeys/confirm/options')->assertOk();+ $this->postJson('/passkeys/confirm', [])->assertUnprocessable();+});++test('a member with a password must confirm it before enabling 2fa', function () {+ $user = twoFactorTeamUser($this->team);+ $this->team->update(['is_two_factor_required' => true]);+ $this->actingAs($user)->withSession(['currentTeam' => $this->team]);++ $this->post('/user/two-factor-authentication')->assertRedirect(route('password.confirm'));++ expect($user->fresh()->two_factor_secret)->toBeNull();+});++test('an oauth member without a known password can enable 2fa without a password confirmation', function () {+ $user = twoFactorTeamUser($this->team);+ OauthIdentity::create([+ 'user_id' => $user->id,+ 'provider' => 'oidc',+ 'issuer' => 'https://idp.example.com',+ 'provider_user_id' => 'oidc-user-1',+ 'email' => $user->email,+ ]);+ $this->team->update(['is_two_factor_required' => true]);+ $this->actingAs($user)->withSession(['currentTeam' => $this->team]);++ $this->from('/two-factor-required')+ ->post('/user/two-factor-authentication')+ ->assertRedirect('/two-factor-required');++ expect($user->fresh()->two_factor_secret)->not->toBeNull();+});++test('an invited member who must set a password can open the password page in a team that requires 2fa', function () {+ $user = twoFactorTeamUser($this->team);+ $user->forceFill(['force_password_reset' => true])->save();+ $this->team->update(['is_two_factor_required' => true]);+ $this->actingAs($user)->withSession(['currentTeam' => $this->team]);++ $this->get('/force-password-reset')->assertOk();+ $this->get('/two-factor-required')->assertRedirect(route('auth.force-password-reset'));+});++test('a member without 2fa can switch from the setup page to a team that does not require 2fa', function () {+ $user = twoFactorTeamUser($this->team);+ $personalTeam = $user->teams()->where('personal_team', true)->firstOrFail();+ $personalTeam->update(['show_boarding' => false]);+ $this->team->update(['is_two_factor_required' => true]);+ $this->actingAs($user)->withSession(['currentTeam' => $this->team]);+ $this->get('/two-factor-required')->assertSee($personalTeam->name);++ $this->post(route('two-factor.required.switch-team'), ['team_id' => $personalTeam->id])+ ->assertRedirect(route('dashboard'));++ expect(data_get(session('currentTeam'), 'id'))->toBe($personalTeam->id);+ $this->get('/profile')->assertOk();+});++test('a member without 2fa cannot switch from the setup page to a team that requires 2fa or is not theirs', function () {+ $user = twoFactorTeamUser($this->team);+ $otherRequiredTeam = Team::factory()->create(['show_boarding' => false, 'is_two_factor_required' => true]);+ $otherRequiredTeam->members()->attach($user->id, ['role' => 'member']);+ $foreignTeam = Team::factory()->create(['show_boarding' => false]);+ $this->team->update(['is_two_factor_required' => true]);+ $this->actingAs($user)->withSession(['currentTeam' => $this->team]);++ $this->post(route('two-factor.required.switch-team'), ['team_id' => $otherRequiredTeam->id])+ ->assertRedirect(route('two-factor.required'));+ $this->post(route('two-factor.required.switch-team'), ['team_id' => $foreignTeam->id])+ ->assertRedirect(route('two-factor.required'));++ expect(data_get(session('currentTeam'), 'id'))->toBe($this->team->id);+});++test('a member without 2fa who switches back to a team that requires 2fa is sent to the setup page again', function () {+ $user = twoFactorTeamUser($this->team);+ $personalTeam = $user->teams()->where('personal_team', true)->firstOrFail();+ $personalTeam->update(['show_boarding' => false]);+ $this->team->update(['is_two_factor_required' => true]);+ $this->actingAs($user);+ session(['currentTeam' => $personalTeam]);++ Livewire::test(SwitchTeam::class)->call('switch_to', $this->team->id);++ expect(data_get(session('currentTeam'), 'id'))->toBe($this->team->id);+ $this->get('/profile')->assertRedirect(route('two-factor.required'));+});diff --git a/tests/Feature/TerminalCapabilitiesTest.php b/tests/Feature/TerminalCapabilitiesTest.phpindex 7bb648d9b1..a8dd5d36ca 100644--- a/tests/Feature/TerminalCapabilitiesTest.php+++ b/tests/Feature/TerminalCapabilitiesTest.php@@ -33,20 +33,20 @@ it('probes container capabilities with exactly one elevation layer', function (sexpect((new ReflectionMethod(Terminal::class, 'checkShellAvailability'))->invoke($terminal, $server, 'game-1'))->toBeTrue();Process::assertRan(function ($process) use ($sudo) {- if (! str_contains($process->command, 'docker inspect')) {+ if (! str_contains(remoteProcessScript($process), 'docker inspect')) {return false;}- expect($process->command)+ expect(remoteProcessScript($process))->toContain("\n{$sudo}docker inspect --format '{{.Config.OpenStdin}}' 'game-1' 2>/dev/null\n")->not->toContain('sudo sudo');- return str_contains($process->command, 'docker inspect');+ return str_contains(remoteProcessScript($process), 'docker inspect');});Process::assertRan(function ($process) use ($sudo) {- if (! str_contains($process->command, 'docker exec')) {+ if (! str_contains(remoteProcessScript($process), 'docker exec')) {return false;}- expect($process->command)+ expect(remoteProcessScript($process))->toContain("{$sudo}docker exec 'game-1' bash -c 'exit 0' 2>/dev/null || {$sudo}docker exec 'game-1' sh -c 'exit 0' 2>/dev/null")->not->toContain('sudo sudo');diff --git a/tests/Feature/TerminalSessionSecurityTest.php b/tests/Feature/TerminalSessionSecurityTest.phpindex 55650df4ca..93bd8aa8bf 100644--- a/tests/Feature/TerminalSessionSecurityTest.php+++ b/tests/Feature/TerminalSessionSecurityTest.php@@ -110,7 +110,7 @@ it('carries the console mode in the token and attaches instead of exec on redeem'private_key_id' => $privateKey->id,]);$service = app(TerminalSessionService::class);- Process::fake([+ fakeRemoteProcesses(['*docker inspect*' => Process::result(output: json_encode(['State' => ['Status' => 'running']])),'*' => Process::result(),]);diff --git a/tests/Feature/TerminalSessionStartFailureTest.php b/tests/Feature/TerminalSessionStartFailureTest.phpindex 6871500448..8a17b1fed7 100644--- a/tests/Feature/TerminalSessionStartFailureTest.php+++ b/tests/Feature/TerminalSessionStartFailureTest.php@@ -55,7 +55,7 @@ beforeEach(function () {function fakeTerminalContainer(string $state, bool $hasShell = true, array $runningContainers = []): void{Process::fake(function ($process) use ($state, $hasShell, $runningContainers) {- $command = (string) $process->command;+ $command = remoteProcessScript($process);if (str_contains($command, 'docker ps -a')) {return Process::result(output: collect($runningContainers)diff --git a/tests/Feature/TraefikAcmeBackupTest.php b/tests/Feature/TraefikAcmeBackupTest.phpindex 841cd351fe..fa46a8d46d 100644--- a/tests/Feature/TraefikAcmeBackupTest.php+++ b/tests/Feature/TraefikAcmeBackupTest.php@@ -53,7 +53,7 @@ beforeEach(function () {$this->remoteLines = [];Process::fake(function ($process) {- $command = $process->command;+ $command = remoteProcessScript($process);if (str_contains($command, ' scp ') && preg_match("~'([^']+)' \\S+:'([^']+)'$~", $command, $matches)) {copy($matches[1], $matches[2]);diff --git a/tests/Feature/TraefikCertificateRemoteFileTest.php b/tests/Feature/TraefikCertificateRemoteFileTest.phpindex 972d0f4986..f584461e99 100644--- a/tests/Feature/TraefikCertificateRemoteFileTest.php+++ b/tests/Feature/TraefikCertificateRemoteFileTest.php@@ -40,7 +40,7 @@ beforeEach(function () {$this->uploadedContents = null;Process::fake(function ($process) {- $command = $process->command;+ $command = remoteProcessScript($process);if (str_starts_with($command, 'timeout') && str_contains($command, ' scp ')&& preg_match("~'(".preg_quote(sys_get_temp_dir(), '~')."/coolify-acme-[^']+)'~", $command, $matches)) {$this->uploadedContents = file_get_contents($matches[1]);@@ -56,8 +56,8 @@ it('reads the ACME file with sudo on non-root servers', function () {$certificates = GetTraefikCertificates::run($this->server);expect($certificates)->toHaveCount(40);- Process::assertRan(fn ($process) => str_contains($process->command, "\nsudo sh -c 'if [ ! -f")- && str_contains($process->command, 'head -c'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "\nsudo sh -c 'if [ ! -f")+ && str_contains(remoteProcessScript($process), 'head -c'));});it('uploads a large ACME file instead of passing it as a shell argument', function () {@@ -70,10 +70,10 @@ it('uploads a large ACME file instead of passing it as a shell argument', functi->and($remaining)->toHaveCount(39)->and(collect($remaining)->pluck('domain.main'))->not->toContain('app1.example.com');- Process::assertRan(fn ($process) => str_contains($process->command, "\nsudo sh -c 'set -e;")- && str_contains($process->command, 'umask 077')- && str_contains($process->command, 'chmod 600')- && str_contains($process->command, 'mv --'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "\nsudo sh -c 'set -e;")+ && str_contains(remoteProcessScript($process), 'umask 077')+ && str_contains(remoteProcessScript($process), 'chmod 600')+ && str_contains(remoteProcessScript($process), 'mv --'));Process::assertDidntRun(fn ($process) => strlen($process->command) > 128 * 1024);expect(glob(sys_get_temp_dir().'/coolify-acme-*'))->toBeEmpty();});@@ -164,7 +164,7 @@ it('does not show or delete certificates of another team', function () {->call('deleteTraefikCertificate', $certificateId)->assertDispatched('error');- Process::assertDidntRun(fn ($process) => str_contains($process->command, 'acme.json'));+ Process::assertDidntRun(fn ($process) => str_contains(remoteProcessScript($process), 'acme.json'));expect($this->uploadedContents)->toBeNull()->and($this->server->fresh()->hasPendingProxyConfiguration())->toBeFalse();});diff --git a/tests/Feature/TrafficAnalytics/SentinelTrafficAvailabilityTest.php b/tests/Feature/TrafficAnalytics/SentinelTrafficAvailabilityTest.phpindex d1f824f1f6..ddaf9c8a58 100644--- a/tests/Feature/TrafficAnalytics/SentinelTrafficAvailabilityTest.php+++ b/tests/Feature/TrafficAnalytics/SentinelTrafficAvailabilityTest.php@@ -91,7 +91,7 @@ it('fetches a healthy server with a bounded ssh and curl timeout', function () {expect($overview->requests)->toBe(7);Process::assertRan(fn (PendingProcess $process) => $process->timeout <= 30- && str_contains($process->command, '--max-time'));+ && str_contains(remoteProcessScript($process), '--max-time'));});it('shows no data without an error when Sentinel has no traffic routes (0.0.x answers 404)', function () {@@ -116,5 +116,5 @@ it('shows no data without an error when Sentinel has no traffic routes (0.0.x an->assertSet('series', [])->assertNotDispatched('error');- Process::assertRan(fn (PendingProcess $process) => str_contains($process->command, '/api/traffic/dashboard'));+ Process::assertRan(fn (PendingProcess $process) => str_contains(remoteProcessScript($process), '/api/traffic/dashboard'));});diff --git a/tests/Feature/TrafficAnalytics/SentinelTrafficTokenStdinTest.php b/tests/Feature/TrafficAnalytics/SentinelTrafficTokenStdinTest.phpindex 7919c5e212..dad221fa3a 100644--- a/tests/Feature/TrafficAnalytics/SentinelTrafficTokenStdinTest.php+++ b/tests/Feature/TrafficAnalytics/SentinelTrafficTokenStdinTest.php@@ -71,9 +71,8 @@ function runSentSentinelScriptWithFakes(string $bin): array{$script = null;Process::assertRan(function ($process) use (&$script) {- // generateSshCommand sends the script as the body of a heredoc: first line opens it, last line closes it.- $lines = explode("\n", $process->command);- $script = implode("\n", array_slice($lines, 1, -1));+ // The script goes to ssh on stdin.+ $script = rtrim((string) $process->input, "\n");return true;});diff --git a/tests/Feature/TrafficAnalytics/StartSentinelTrafficTest.php b/tests/Feature/TrafficAnalytics/StartSentinelTrafficTest.phpindex 3bb5a818f8..7d16345f5c 100644--- a/tests/Feature/TrafficAnalytics/StartSentinelTrafficTest.php+++ b/tests/Feature/TrafficAnalytics/StartSentinelTrafficTest.php@@ -143,7 +143,7 @@ function runStartSentinelAndCaptureScript(Server $server): stringEvent::fake([SentinelRestarted::class]);$scripts = [];Process::fake(function ($process) use (&$scripts) {- $scripts[] = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $scripts[] = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);return Process::result(output: '');});diff --git a/tests/Feature/VolumeBackupMultipleS3DestinationsTest.php b/tests/Feature/VolumeBackupMultipleS3DestinationsTest.phpindex 4f95a3fe66..45e5a81200 100644--- a/tests/Feature/VolumeBackupMultipleS3DestinationsTest.php+++ b/tests/Feature/VolumeBackupMultipleS3DestinationsTest.php@@ -93,7 +93,7 @@ function fakeMultiDestinationS3Disks(): voidit('archives locally and uploads to every destination instead of streaming', function () {[$backup, $first, $second] = createMultiDestinationVolumeBackup(['disable_local_backup' => true]);fakeMultiDestinationS3Disks();- Process::fake([+ fakeRemoteProcesses(['*du -b*' => '128','*' => '',]);@@ -101,11 +101,11 @@ it('archives locally and uploads to every destination instead of streaming', fun(new VolumeBackupJob($backup))->handle();$execution = ScheduledVolumeBackupExecution::query()->sole();- Process::assertNotRan(fn ($process) => str_contains($process->command, 'mc pipe'));- Process::assertRan(fn ($process) => str_contains($process->command, 'tar -I') && str_contains($process->command, ' > '));- Process::assertRan(fn ($process) => str_contains($process->command, 'mc cp') && str_contains($process->command, 'temporary/first-bucket/'));- Process::assertRan(fn ($process) => str_contains($process->command, 'mc cp') && str_contains($process->command, 'temporary/second-bucket/'));- Process::assertRan(fn ($process) => preg_match("/rm -f '[^']+\\.tar\\.gz'/", $process->command) === 1);+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'mc pipe'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'tar -I') && str_contains(remoteProcessScript($process), ' > '));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'mc cp') && str_contains(remoteProcessScript($process), 'temporary/first-bucket/'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'mc cp') && str_contains(remoteProcessScript($process), 'temporary/second-bucket/'));+ Process::assertRan(fn ($process) => preg_match("/rm -f '[^']+\\.tar\\.gz'/", remoteProcessScript($process)) === 1);expect($execution->status)->toBe('success')->and($execution->s3_storage_id)->toBe($first->id)->and($execution->s3_uploaded)->toBeTrue()@@ -120,23 +120,32 @@ it('signs each upload with the region of its destination', function () {[$backup, $first] = createMultiDestinationVolumeBackup();$first->update(['region' => 'ru-central-1']);fakeMultiDestinationS3Disks();- Process::fake([- '*du -b*' => '128',- '*' => '',- ]);+ $commands = collect();+ Process::fake(function ($process) use ($commands) {+ $commands->push(remoteProcessScript($process));++ return Process::result(output: str_contains(remoteProcessScript($process), 'du -b') ? '128' : '');+ });(new VolumeBackupJob($backup))->handle();- Process::assertRan(fn ($process) => str_contains($process->command, "-e 'MC_REGION=ru-central-1'")- && str_contains($process->command, 'temporary/first-bucket/'));- Process::assertRan(fn ($process) => str_contains($process->command, "-e 'MC_REGION=us-east-1'")- && str_contains($process->command, 'temporary/second-bucket/'));+ $regionOfBucket = [];+ $region = null;+ foreach ($commands as $command) {+ if (preg_match("/-e 'MC_REGION=([^']+)'/", $command, $match)) {+ $region = $match[1];+ }+ if (preg_match("#mc cp .*'temporary/([^/']+)/#", $command, $match)) {+ $regionOfBucket[$match[1]] = $region;+ }+ }+ expect($regionOfBucket)->toBe(['first-bucket' => 'ru-central-1', 'second-bucket' => 'us-east-1']);});it('keeps the local archive and cleans the partial upload when one destination fails', function () {[$backup, $first, $second] = createMultiDestinationVolumeBackup(['disable_local_backup' => true]);fakeMultiDestinationS3Disks();- Process::fake([+ fakeRemoteProcesses(['*du -b*' => '128','*temporary/second-bucket/*' => Process::result(errorOutput: 'access denied', exitCode: 1),'*' => '',@@ -147,7 +156,7 @@ it('keeps the local archive and cleans the partial upload when one destination f$execution = ScheduledVolumeBackupExecution::query()->sole();$firstReplica = $execution->s3Replicas()->where('s3_storage_id', $first->id)->sole();$secondReplica = $execution->s3Replicas()->where('s3_storage_id', $second->id)->sole();- Process::assertNotRan(fn ($process) => preg_match("/rm -f '[^']+\\.tar\\.gz'/", $process->command) === 1);+ Process::assertNotRan(fn ($process) => preg_match("/rm -f '[^']+\\.tar\\.gz'/", remoteProcessScript($process)) === 1);expect($execution->status)->toBe('success')->and($execution->message)->toContain('S3 upload failed: second storage:')->and($execution->s3_uploaded)->toBeFalse()diff --git a/tests/Feature/VolumeBackupRecoveryBoundsTest.php b/tests/Feature/VolumeBackupRecoveryBoundsTest.phpindex 08d90ab774..bc7ffc0e89 100644--- a/tests/Feature/VolumeBackupRecoveryBoundsTest.php+++ b/tests/Feature/VolumeBackupRecoveryBoundsTest.php@@ -250,7 +250,7 @@ it('cleans the S3 upload even when container recovery fails', function () {Storage::fake('ssh-mux');[, , , $backup] = createRecoveryBoundsBackup(Team::factory()->create(), ['stop_during_backup' => true]);$execution = createRecoveryBoundsExecution($backup, ['stop_recovery_pending' => true]);- Process::fake([+ fakeRemoteProcesses(['*' => Process::result(errorOutput: 'ssh: connect to host 203.0.113.20 port 22: Connection refused', exitCode: 255),]);$disk = Mockery::mock();@@ -314,7 +314,7 @@ it('only blocks new scheduled backups while stopped containers await recovery',]);it('keeps executions with pending recovery during retention cleanup', function () {- Process::fake([+ fakeRemoteProcesses(['*du -b*' => '128','*' => '',]);diff --git a/tests/Feature/VolumeBackupRecoveryMissingContainerTest.php b/tests/Feature/VolumeBackupRecoveryMissingContainerTest.phpindex fe5190a7b1..2210fd4014 100644--- a/tests/Feature/VolumeBackupRecoveryMissingContainerTest.php+++ b/tests/Feature/VolumeBackupRecoveryMissingContainerTest.php@@ -109,7 +109,7 @@ function missingContainerRecoveryBackup(string $user): arrayfunction fakeMissingContainerRemote(string $fakeDockerDir): void{Process::fake(function ($process) use ($fakeDockerDir) {- $command = is_array($process->command) ? implode(' ', $process->command) : $process->command;+ $command = is_array(remoteProcessScript($process)) ? implode(' ', remoteProcessScript($process)) : remoteProcessScript($process);$lines = explode(PHP_EOL, $command);$remote = implode(PHP_EOL, array_slice($lines, 1, -1));if (str_starts_with($remote, 'sudo ')) {diff --git a/tests/Feature/VolumeBackupTest.php b/tests/Feature/VolumeBackupTest.phpindex 9b82d22785..cb4086ae5e 100644--- a/tests/Feature/VolumeBackupTest.php+++ b/tests/Feature/VolumeBackupTest.php@@ -817,8 +817,8 @@ it('deletes an individual volume backup archive and execution', function () {->assertDispatched('success');expect($execution->fresh())->toBeNull();- Process::assertRan(fn ($process) => str_contains($process->command, 'rm -f')- && str_contains($process->command, 'archive.tar.gz'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'rm -f')+ && str_contains(remoteProcessScript($process), 'archive.tar.gz'));});it('deletes an individual S3 archive from the storage recorded on its execution', function () {@@ -1608,8 +1608,8 @@ it('deletes local archives before deleting a volume backup schedule', function (]);expect(ScheduledVolumeBackup::query()->count())->toBe(0);- Process::assertRan(fn ($process) => str_contains($process->command, 'rm -f')- && str_contains($process->command, 'archive.tar.gz'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'rm -f')+ && str_contains(remoteProcessScript($process), 'archive.tar.gz'));});it('deletes a volume backup schedule without deleting unselected archives', function () {@@ -1812,8 +1812,8 @@ it('deletes disabled volume backup archives before deleting their application',expect($backup->fresh())->toBeNull()->and($execution->fresh())->toBeNull();- Process::assertRan(fn ($process) => str_contains($process->command, 'rm -f')- && str_contains($process->command, 'application-delete.tar.gz'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'rm -f')+ && str_contains(remoteProcessScript($process), 'application-delete.tar.gz'));});it('marks a running execution failed even when the job instance lost its execution state', function () {@@ -1836,8 +1836,8 @@ it('marks a running execution failed even when the job instance lost its executi->and($execution->fresh()->message)->toBe('Worker timed out')->and($execution->fresh()->finished_at)->not->toBeNull()->and($execution->fresh()->filename)->toBeNull();- Process::assertRan(fn ($process) => str_contains($process->command, 'rm -f')- && str_contains($process->command, 'timed-out.tar.gz'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'rm -f')+ && str_contains(remoteProcessScript($process), 'timed-out.tar.gz'));});it('archives a named volume using the server compression CPU percentage', function (int $compressionCpuPercentage) {@@ -1855,7 +1855,7 @@ it('archives a named volume using the server compression CPU percentage', functi'timeout' => 3600,]);- Process::fake([+ fakeRemoteProcesses(['*docker ps -q*' => 'abc123','*du -b*' => '128','*' => '',@@ -1874,17 +1874,17 @@ it('archives a named volume using the server compression CPU percentage', functi->and($execution->size)->toBe(128)->and($execution->filename)->toEndWith('.tar.gz');- Process::assertRan(fn ($process) => str_contains($process->command, 'docker volume inspect')- && str_contains($process->command, 'docker run --rm --name ')- && str_contains($process->command, 'app-data:/volume:ro')- && str_contains($process->command, 'command -v pigz')- && str_contains($process->command, 'pigz -3 -p')- && str_contains($process->command, "\$(nproc) * {$compressionCpuPercentage} + 99")- && str_contains($process->command, 'gzip -3')- && str_contains($process->command, 'tar -I "$compressor" -cf -')- && str_contains($process->command, '> ')- && str_contains($process->command, '.tar.gz')- && ! str_contains($process->command, ':/backup'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker volume inspect')+ && str_contains(remoteProcessScript($process), 'docker run --rm --name ')+ && str_contains(remoteProcessScript($process), 'app-data:/volume:ro')+ && str_contains(remoteProcessScript($process), 'command -v pigz')+ && str_contains(remoteProcessScript($process), 'pigz -3 -p')+ && str_contains(remoteProcessScript($process), "\$(nproc) * {$compressionCpuPercentage} + 99")+ && str_contains(remoteProcessScript($process), 'gzip -3')+ && str_contains(remoteProcessScript($process), 'tar -I "$compressor" -cf -')+ && str_contains(remoteProcessScript($process), '> ')+ && str_contains(remoteProcessScript($process), '.tar.gz')+ && ! str_contains(remoteProcessScript($process), ':/backup'));})->with(['low' => 25,'high' => 75,@@ -1906,7 +1906,7 @@ it('logs the selected volume backup compressor in development', function (string'retention_max_storage_s3' => 0,]);- Process::fake([+ fakeRemoteProcesses(['*command -v pigz*' => $detectedCompressor,'*du -b*' => '128','*' => '',@@ -1953,7 +1953,7 @@ it('keeps the upload destination on the volume backup execution', function () {$disk->shouldReceive('delete')->zeroOrMoreTimes()->andReturnTrue();Storage::shouldReceive('disk')->with('ssh-keys')->andReturn($sshDisk);Storage::shouldReceive('build')->zeroOrMoreTimes()->andReturn($disk);- Process::fake([+ fakeRemoteProcesses(['*du -b*' => '128','*' => '',]);@@ -1997,7 +1997,7 @@ it('streams S3-only volume backups with Content-MD5 on Object Lock buckets and w$disk->shouldReceive('delete')->zeroOrMoreTimes()->andReturnTrue();Storage::shouldReceive('disk')->with('ssh-keys')->andReturn($sshDisk);Storage::shouldReceive('build')->once()->andReturn($disk);- Process::fake([+ fakeRemoteProcesses(['*mc pipe*' => "Added `temporary` successfully.\n128 bytes -> `temporary/bucket/archive.tar.gz`\n128",'*' => '',]);@@ -2006,21 +2006,23 @@ it('streams S3-only volume backups with Content-MD5 on Object Lock buckets and w$execution = ScheduledVolumeBackupExecution::query()->sole();$expectedFilename = 'volume-app-data-1786795200.tar.gz';- Process::assertRan(fn ($process) => str_contains($process->command, 'tar -I')- && str_contains($process->command, 'mc pipe')- && str_contains($process->command, "-e 'MC_REGION=ru-central-1'")- && ! str_contains($process->command, '--quiet')- && str_contains($process->command, 'lock=$(mc retention info --default --json --resolve')- && str_contains($process->command, '*) checksum=--checksum=MD5 ;; *) checksum= ;; esac && (compressor=')- && str_contains($process->command, ' $checksum ')- && str_contains($process->command, '>/dev/null 2>/tmp/mc-pipe.log || {')- && str_contains($process->command, '/tmp/mc-pipe.log >&2; exit $status; }) && mc stat')- && str_contains($process->command, 'temporary/bucket/')- && str_contains($process->command, $expectedFilename)- && substr_count($process->command, '--resolve') === 4- && substr_count($process->command, '>/dev/null') >= 2- && ! str_contains($process->command, ' > ')- && ! str_contains($process->command, 'mc cp'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'tar -I')+ && str_contains(remoteProcessScript($process), 'mc pipe')+ && ! str_contains(remoteProcessScript($process), '--quiet')+ && str_contains(remoteProcessScript($process), 'lock=$(mc retention info --default --json --resolve')+ && str_contains(remoteProcessScript($process), '*) checksum=--checksum=MD5 ;; *) checksum= ;; esac && (compressor=')+ && str_contains(remoteProcessScript($process), ' $checksum ')+ && str_contains(remoteProcessScript($process), '>/dev/null 2>/tmp/mc-pipe.log || {')+ && str_contains(remoteProcessScript($process), '/tmp/mc-pipe.log >&2; exit $status; }) && mc stat')+ && str_contains(remoteProcessScript($process), 'temporary/bucket/')+ && str_contains(remoteProcessScript($process), $expectedFilename)+ && substr_count(remoteProcessScript($process), '--resolve') === 3+ && substr_count(remoteProcessScript($process), '>/dev/null') >= 1+ && ! str_contains(remoteProcessScript($process), ' > ')+ && ! str_contains(remoteProcessScript($process), 'mc cp'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker run -d --rm --name')+ && str_contains(remoteProcessScript($process), "-e 'MC_REGION=ru-central-1'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'mc alias set --resolve'));expect($execution->status)->toBe('success')->and($execution->size)->toBe(128)->and($execution->s3_uploaded)->toBeTrue()@@ -2056,7 +2058,7 @@ it('fails an unsupported S3 stream without falling back to a local archive', fun$disk->shouldReceive('delete')->once()->andReturnTrue();Storage::shouldReceive('disk')->with('ssh-keys')->andReturn($sshDisk);Storage::shouldReceive('build')->zeroOrMoreTimes()->andReturn($disk);- Process::fake([+ fakeRemoteProcesses(['*mc pipe*' => Process::result(errorOutput: 'streaming upload is unsupported', exitCode: 1),'*' => '',]);@@ -2070,11 +2072,11 @@ it('fails an unsupported S3 stream without falling back to a local archive', fun->and($execution->message)->toContain('Enable local backups to use the local archive upload method.')->and($execution->filename)->toBeNull()->and($execution->local_storage_deleted)->toBeTrue();- Process::assertRan(fn ($process) => str_contains($process->command, 'mc pipe')- && ! str_contains($process->command, 'mc cp')- && ! str_contains($process->command, ' > '));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'mkdir -p')- || (str_contains($process->command, 'rm -f') && str_contains($process->command, '.tar.gz')));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'mc pipe')+ && ! str_contains(remoteProcessScript($process), 'mc cp')+ && ! str_contains(remoteProcessScript($process), ' > '));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'mkdir -p')+ || preg_match("/(^|\n)rm -f '[^\n]*\.tar\.gz'/", remoteProcessScript($process)) === 1);});it('keeps local-first archive creation and mc copy when retaining a local volume backup', function () {@@ -2105,18 +2107,18 @@ it('keeps local-first archive creation and mc copy when retaining a local volume$disk->shouldReceive('delete')->zeroOrMoreTimes()->andReturnTrue();Storage::shouldReceive('disk')->with('ssh-keys')->andReturn($sshDisk);Storage::shouldReceive('build')->once()->andReturn($disk);- Process::fake([+ fakeRemoteProcesses(['*du -b*' => '128','*' => '',]);(new VolumeBackupJob($backup))->handle();- Process::assertRan(fn ($process) => str_contains($process->command, 'tar -I')- && str_contains($process->command, '>')- && ! str_contains($process->command, 'mc pipe'));- Process::assertRan(fn ($process) => str_contains($process->command, 'mc cp --resolve ')- && ! str_contains($process->command, 'mc pipe'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'tar -I')+ && str_contains(remoteProcessScript($process), '>')+ && ! str_contains(remoteProcessScript($process), 'mc pipe'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'mc cp --resolve ')+ && ! str_contains(remoteProcessScript($process), 'mc pipe'));});it('removes local volume backups older than the configured retention days', function () {@@ -2146,7 +2148,7 @@ it('removes local volume backups older than the configured retention days', func]);ScheduledVolumeBackupExecution::query()->whereKey($recentExecution)->update(['created_at' => now()->subDay()]);- Process::fake([+ fakeRemoteProcesses(['*du -b*' => '128','*' => '',]);@@ -2155,8 +2157,8 @@ it('removes local volume backups older than the configured retention days', funcexpect($expiredExecution->fresh())->toBeNull()->and($recentExecution->fresh())->not->toBeNull();- Process::assertRan(fn ($process) => str_contains($process->command, 'rm -f')- && str_contains($process->command, 'expired.tar.gz'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'rm -f')+ && str_contains(remoteProcessScript($process), 'expired.tar.gz'));});it('removes oldest local volume backups over the configured storage limit', function () {@@ -2179,7 +2181,7 @@ it('removes oldest local volume backups over the configured storage limit', func]);ScheduledVolumeBackupExecution::query()->whereKey($oldExecution)->update(['created_at' => now()->subDay()]);- Process::fake([+ fakeRemoteProcesses(['*du -b*' => '128','*' => '',]);@@ -2188,8 +2190,8 @@ it('removes oldest local volume backups over the configured storage limit', funcexpect($oldExecution->fresh())->toBeNull()->and($backup->executions()->where('status', 'success')->count())->toBe(1);- Process::assertRan(fn ($process) => str_contains($process->command, 'rm -f')- && str_contains($process->command, 'over-limit.tar.gz'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'rm -f')+ && str_contains(remoteProcessScript($process), 'over-limit.tar.gz'));});it('removes retained S3 archives from the storage recorded on each execution', function () {@@ -2319,7 +2321,7 @@ it('keeps a successful backup successful when retention cleanup fails', function'created_at' => now()->subDay(),]);- Process::fake([+ fakeRemoteProcesses(['*rm -f*' => Process::result(errorOutput: 'permission denied', exitCode: 1),'*du -b*' => '128','*' => '',@@ -2345,7 +2347,7 @@ it('stops and restarts containers that use the volume when requested', function'stop_during_backup' => true,]);- Process::fake([+ fakeRemoteProcesses(['*docker ps -q*' => "abc123\ndef456",'*du -b*' => '128','*' => '',@@ -2353,16 +2355,16 @@ it('stops and restarts containers that use the volume when requested', function(new VolumeBackupJob($backup))->handle();- Process::assertRan(fn ($process) => str_contains($process->command, 'trap cleanup EXIT')- && str_contains($process->command, 'exit 143')- && str_contains($process->command, 'TERM')- && str_contains($process->command, 'docker stop')- && str_contains($process->command, 'docker start')- && str_contains($process->command, 'state_file='));- Process::assertRan(fn ($process) => str_contains($process->command, '{{println .Source}}{{println .Name}}')- && str_contains($process->command, "grep -Fqx -- 'app-data'")- && str_contains($process->command, '{{if not .State.Paused}}')- && str_contains($process->command, 'continue'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'trap cleanup EXIT')+ && str_contains(remoteProcessScript($process), 'exit 143')+ && str_contains(remoteProcessScript($process), 'TERM')+ && str_contains(remoteProcessScript($process), 'docker stop')+ && str_contains(remoteProcessScript($process), 'docker start')+ && str_contains(remoteProcessScript($process), 'state_file='));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), '{{println .Source}}{{println .Name}}')+ && str_contains(remoteProcessScript($process), "grep -Fqx -- 'app-data'")+ && str_contains(remoteProcessScript($process), '{{if not .State.Paused}}')+ && str_contains(remoteProcessScript($process), 'continue'));});/**@@ -2374,10 +2376,10 @@ function runVolumeContainerScan(string $fakeDocker): array{$scan = '';Process::assertRan(function ($process) use (&$scan): bool {- if (! str_contains($process->command, 'docker ps -q')) {+ if (! str_contains(remoteProcessScript($process), 'docker ps -q')) {return false;}- $scan = $process->command;+ $scan = remoteProcessScript($process);return true;});@@ -2406,7 +2408,7 @@ it('skips containers that are removed while scanning for volume users', function'stop_during_backup' => true,]);- Process::fake(['*du -b*' => '128', '*' => '']);+ fakeRemoteProcesses(['*du -b*' => '128', '*' => '']);(new VolumeBackupJob($backup))->handle();$docker = <<<'SH'@@ -2436,7 +2438,7 @@ it('finds containers using a bind mounted host path before stopping', function ('stop_during_backup' => true,]);- Process::fake([+ fakeRemoteProcesses(['*docker ps -q*' => 'abc123','*du -b*' => '128','*' => '',@@ -2444,8 +2446,8 @@ it('finds containers using a bind mounted host path before stopping', function ((new VolumeBackupJob($backup))->handle();- Process::assertRan(fn ($process) => str_contains($process->command, '{{println .Source}}{{println .Name}}')- && str_contains($process->command, "grep -Fqx -- '/srv/app-data'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), '{{println .Source}}{{println .Name}}')+ && str_contains(remoteProcessScript($process), "grep -Fqx -- '/srv/app-data'"));});it('archives an application directory mount from its resolved host path', function () {@@ -2468,7 +2470,7 @@ it('archives an application directory mount from its resolved host path', functi]);$source = $application->workdir().'/uploads';- Process::fake([+ fakeRemoteProcesses(['*docker ps -q*' => 'abc123','*du -b*' => '128','*' => '',@@ -2477,13 +2479,13 @@ it('archives an application directory mount from its resolved host path', functi(new VolumeBackupJob($backup))->handle();expect(ScheduledVolumeBackupExecution::query()->sole()->status)->toBe('success');- Process::assertRan(fn ($process) => str_contains($process->command, 'test -d '.escapeshellarg($source)));- Process::assertRan(fn ($process) => str_contains($process->command, $source)- && str_contains($process->command, ':/volume:ro')- && str_contains($process->command, 'directory-uploads-'));- Process::assertRan(fn ($process) => str_contains($process->command, "grep -Fqx -- '".$source."'"));- Process::assertRan(fn ($process) => str_contains($process->command, 'docker stop')- && str_contains($process->command, 'docker start'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'test -d '.escapeshellarg($source)));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), $source)+ && str_contains(remoteProcessScript($process), ':/volume:ro')+ && str_contains(remoteProcessScript($process), 'directory-uploads-'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "grep -Fqx -- '".$source."'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'docker stop')+ && str_contains(remoteProcessScript($process), 'docker start'));});it('retries container recovery when a timed out backup left a container stopped', function () {@@ -2499,7 +2501,7 @@ it('retries container recovery when a timed out backup left a container stopped''status' => 'running','stop_recovery_pending' => true,]);- Process::fake([+ fakeRemoteProcesses(['*cat *coolify-volume-backup-*' => 'abc123','*docker start*' => Process::result(errorOutput: 'daemon unavailable', exitCode: 1),]);@@ -2523,7 +2525,7 @@ it('restarts containers and removes a partial archive when archiving fails', fun'frequency' => 'daily','stop_during_backup' => true,]);- Process::fake([+ fakeRemoteProcesses(['*docker ps -q*' => 'abc123','*trap cleanup EXIT*' => Process::result(errorOutput: 'archive failed', exitCode: 1),'*' => '',@@ -2534,8 +2536,8 @@ it('restarts containers and removes a partial archive when archiving fails', fun$execution = ScheduledVolumeBackupExecution::query()->sole();expect($execution->status)->toBe('failed')->and($execution->stop_container_ids)->toBeNull();- Process::assertRan(fn ($process) => str_contains($process->command, 'rm -f')- && str_contains($process->command, '.tar.gz'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'rm -f')+ && str_contains(remoteProcessScript($process), '.tar.gz'));});it('throws when the S3 filesystem reports that deletion failed', function () {@@ -2887,7 +2889,7 @@ it('notifies the team when a volume backup succeeds', function () {'team_id' => $team->id,'frequency' => 'daily',]);- Process::fake([+ fakeRemoteProcesses(['*du -b*' => '128','*' => '',]);diff --git a/tests/Feature/VolumeCloneJobTest.php b/tests/Feature/VolumeCloneJobTest.phpindex 8a3c585f69..b0f6e627d4 100644--- a/tests/Feature/VolumeCloneJobTest.php+++ b/tests/Feature/VolumeCloneJobTest.php@@ -44,7 +44,7 @@ beforeEach(function () {function fakeVolumeCloneServer(?array $sourceVolume): void{Process::fake(fn ($process) => Process::result(- output: str_contains($process->command, 'docker volume inspect')+ output: str_contains(remoteProcessScript($process), 'docker volume inspect')? json_encode($sourceVolume === null ? [] : [$sourceVolume]): '',));@@ -54,7 +54,7 @@ function volumeCloneCommands(): string{$commands = [];Process::assertRan(function ($process) use (&$commands) {- $commands[] = $process->command;+ $commands[] = remoteProcessScript($process);return true;});diff --git a/tests/Fixtures/parser/application-reference.yaml b/tests/Fixtures/parser/application-reference.yamlindex 9f40212547..d78609688a 100644--- a/tests/Fixtures/parser/application-reference.yaml+++ b/tests/Fixtures/parser/application-reference.yaml@@ -54,6 +54,8 @@ services:SERVICE_FQDN_API: /apiMAP_LITERAL: map-valueMAP_NULL:+ # Empty value: filled from the stored variable (the preview variable for a pull request).+ MAP_EMPTY_FROM_DATABASE: ''ports:- "3001:3001"- target: 3002diff --git a/tests/Fixtures/parser/expected/application-with-exact-caddy-labels.yaml b/tests/Fixtures/parser/expected/application-with-exact-caddy-labels.yamlindex 885372bac3..63897e9c8f 100644--- a/tests/Fixtures/parser/expected/application-with-exact-caddy-labels.yaml+++ b/tests/Fixtures/parser/expected/application-with-exact-caddy-labels.yaml@@ -34,6 +34,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -172,6 +173,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: production-valueCOOLIFY_BRANCH: '"main"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-20260102T030405@@ -677,6 +679,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -819,6 +822,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: production-valueCOOLIFY_BRANCH: '"main"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-20260102T030405@@ -1373,6 +1377,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -1386,9 +1391,9 @@required: trueSELF_REF: nullSERVICE_BASE64_64_WEB: '<generated:SERVICE_BASE64_64_WEB:64>'- SERVICE_FQDN_ADMIN: referenceapplication0001.10.0.0.1.sslip.io+ SERVICE_FQDN_ADMIN: admin.example.comSERVICE_FQDN_ADMIN_8080: null- SERVICE_FQDN_API: referenceapplication0001.10.0.0.1.sslip.io/api+ SERVICE_FQDN_API: www.api.example.comSERVICE_FQDN_API_V2: api-v2.example.comSERVICE_FQDN_BLOG: blog.example.comSERVICE_FQDN_MY_APP: my-app.example.com@@ -1516,6 +1521,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: preview-valueCOOLIFY_BRANCH: '"pull/7/head"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-pr-7@@ -2072,6 +2078,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -2085,9 +2092,9 @@required: trueSELF_REF: nullSERVICE_BASE64_64_WEB: '<generated:SERVICE_BASE64_64_WEB:64>'- SERVICE_FQDN_ADMIN: referenceapplication0001.10.0.0.1.sslip.io+ SERVICE_FQDN_ADMIN: admin.example.comSERVICE_FQDN_ADMIN_8080: null- SERVICE_FQDN_API: referenceapplication0001.10.0.0.1.sslip.io/api+ SERVICE_FQDN_API: www.api.example.comSERVICE_FQDN_API_V2: api-v2.example.comSERVICE_FQDN_BLOG: blog.example.comSERVICE_FQDN_MY_APP: my-app.example.com@@ -2193,6 +2200,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: preview-valueCOOLIFY_BRANCH: '"pull/8/head"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-pr-8diff --git a/tests/Fixtures/parser/expected/application-with-exact-traefik-labels.yaml b/tests/Fixtures/parser/expected/application-with-exact-traefik-labels.yamlindex bf5d8ae42f..83792ffb74 100644--- a/tests/Fixtures/parser/expected/application-with-exact-traefik-labels.yaml+++ b/tests/Fixtures/parser/expected/application-with-exact-traefik-labels.yaml@@ -34,6 +34,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -174,6 +175,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: production-valueCOOLIFY_BRANCH: '"main"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-20260102T030405@@ -690,6 +692,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -841,6 +844,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: production-valueCOOLIFY_BRANCH: '"main"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-20260102T030405@@ -1439,6 +1443,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -1452,9 +1457,9 @@required: trueSELF_REF: nullSERVICE_BASE64_64_WEB: '<generated:SERVICE_BASE64_64_WEB:64>'- SERVICE_FQDN_ADMIN: referenceapplication0001.10.0.0.1.sslip.io+ SERVICE_FQDN_ADMIN: admin.example.comSERVICE_FQDN_ADMIN_8080: null- SERVICE_FQDN_API: referenceapplication0001.10.0.0.1.sslip.io/api+ SERVICE_FQDN_API: www.api.example.comSERVICE_FQDN_API_V2: api-v2.example.comSERVICE_FQDN_BLOG: blog.example.comSERVICE_FQDN_MY_APP: my-app.example.com@@ -1591,6 +1596,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: preview-valueCOOLIFY_BRANCH: '"pull/7/head"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-pr-7@@ -2189,6 +2195,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -2202,9 +2209,9 @@required: trueSELF_REF: nullSERVICE_BASE64_64_WEB: '<generated:SERVICE_BASE64_64_WEB:64>'- SERVICE_FQDN_ADMIN: referenceapplication0001.10.0.0.1.sslip.io+ SERVICE_FQDN_ADMIN: admin.example.comSERVICE_FQDN_ADMIN_8080: null- SERVICE_FQDN_API: referenceapplication0001.10.0.0.1.sslip.io/api+ SERVICE_FQDN_API: www.api.example.comSERVICE_FQDN_API_V2: api-v2.example.comSERVICE_FQDN_BLOG: blog.example.comSERVICE_FQDN_MY_APP: my-app.example.com@@ -2310,6 +2317,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: preview-valueCOOLIFY_BRANCH: '"pull/8/head"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-pr-8diff --git a/tests/Fixtures/parser/expected/application-with-settings.yaml b/tests/Fixtures/parser/expected/application-with-settings.yamlindex 32c3ad3a7b..3363af1f60 100644--- a/tests/Fixtures/parser/expected/application-with-settings.yaml+++ b/tests/Fixtures/parser/expected/application-with-settings.yaml@@ -34,6 +34,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -186,6 +187,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: production-valueCOOLIFY_BRANCH: '"main"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-20260102T030405@@ -785,6 +787,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -948,6 +951,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: production-valueCOOLIFY_BRANCH: '"main"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-20260102T030405@@ -1635,6 +1639,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -1648,9 +1653,9 @@required: trueSELF_REF: nullSERVICE_BASE64_64_WEB: '<generated:SERVICE_BASE64_64_WEB:64>'- SERVICE_FQDN_ADMIN: referenceapplication0001.10.0.0.1.sslip.io+ SERVICE_FQDN_ADMIN: admin.example.comSERVICE_FQDN_ADMIN_8080: null- SERVICE_FQDN_API: referenceapplication0001.10.0.0.1.sslip.io/api+ SERVICE_FQDN_API: www.api.example.comSERVICE_FQDN_API_V2: api-v2.example.comSERVICE_FQDN_BLOG: blog.example.comSERVICE_FQDN_MY_APP: my-app.example.com@@ -1801,6 +1806,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: preview-valueCOOLIFY_BRANCH: '"pull/7/head"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-pr-7@@ -2495,6 +2501,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -2508,9 +2515,9 @@required: trueSELF_REF: nullSERVICE_BASE64_64_WEB: '<generated:SERVICE_BASE64_64_WEB:64>'- SERVICE_FQDN_ADMIN: referenceapplication0001.10.0.0.1.sslip.io+ SERVICE_FQDN_ADMIN: admin.example.comSERVICE_FQDN_ADMIN_8080: null- SERVICE_FQDN_API: referenceapplication0001.10.0.0.1.sslip.io/api+ SERVICE_FQDN_API: www.api.example.comSERVICE_FQDN_API_V2: api-v2.example.comSERVICE_FQDN_BLOG: blog.example.comSERVICE_FQDN_MY_APP: my-app.example.com@@ -2624,6 +2631,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: preview-valueCOOLIFY_BRANCH: '"pull/8/head"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-pr-8diff --git a/tests/Fixtures/parser/expected/application.yaml b/tests/Fixtures/parser/expected/application.yamlindex 6e52c2b0ee..fc3380cdbf 100644--- a/tests/Fixtures/parser/expected/application.yaml+++ b/tests/Fixtures/parser/expected/application.yaml@@ -34,6 +34,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -181,6 +182,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: production-valueCOOLIFY_BRANCH: '"main"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-20260102T030405@@ -725,6 +727,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -883,6 +886,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: production-valueCOOLIFY_BRANCH: '"main"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-20260102T030405@@ -1517,6 +1521,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -1530,9 +1535,9 @@required: trueSELF_REF: nullSERVICE_BASE64_64_WEB: '<generated:SERVICE_BASE64_64_WEB:64>'- SERVICE_FQDN_ADMIN: referenceapplication0001.10.0.0.1.sslip.io+ SERVICE_FQDN_ADMIN: admin.example.comSERVICE_FQDN_ADMIN_8080: null- SERVICE_FQDN_API: referenceapplication0001.10.0.0.1.sslip.io/api+ SERVICE_FQDN_API: www.api.example.comSERVICE_FQDN_API_V2: api-v2.example.comSERVICE_FQDN_BLOG: blog.example.comSERVICE_FQDN_MY_APP: my-app.example.com@@ -1677,6 +1682,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: preview-valueCOOLIFY_BRANCH: '"pull/7/head"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-pr-7@@ -2315,6 +2321,7 @@BARE_PASSTHROUGH: nullBUILD_VERSION: nullDASH_DEFAULT_VAR: null+ MAP_EMPTY_FROM_DATABASE: production-valueMAP_NULL: nullNESTED_INNER: nullNESTED_NO_DEFAULT_INNER: null@@ -2328,9 +2335,9 @@required: trueSELF_REF: nullSERVICE_BASE64_64_WEB: '<generated:SERVICE_BASE64_64_WEB:64>'- SERVICE_FQDN_ADMIN: referenceapplication0001.10.0.0.1.sslip.io+ SERVICE_FQDN_ADMIN: admin.example.comSERVICE_FQDN_ADMIN_8080: null- SERVICE_FQDN_API: referenceapplication0001.10.0.0.1.sslip.io/api+ SERVICE_FQDN_API: www.api.example.comSERVICE_FQDN_API_V2: api-v2.example.comSERVICE_FQDN_BLOG: blog.example.comSERVICE_FQDN_MY_APP: my-app.example.com@@ -2436,6 +2443,7 @@environment:MAP_LITERAL: map-valueMAP_NULL: null+ MAP_EMPTY_FROM_DATABASE: preview-valueCOOLIFY_BRANCH: '"pull/8/head"'COOLIFY_RESOURCE_UUID: referenceapplication0001COOLIFY_CONTAINER_NAME: api-referenceapplication0001-pr-8diff --git a/tests/Pest.php b/tests/Pest.phpindex b0a054c607..93ec29d1b0 100644--- a/tests/Pest.php+++ b/tests/Pest.php@@ -31,6 +31,7 @@ uses(TestCase::class)->in('Feature', 'v4/Feature', 'v4/Browser');require_once __DIR__.'/Support/BrowserTestHelpers.php';require_once __DIR__.'/Support/ManagedDnsTestHelpers.php';require_once __DIR__.'/Support/StripeTestHelpers.php';+require_once __DIR__.'/Support/RemoteProcessTestHelpers.php';function remoteOutputSource(string $path): string{diff --git a/tests/Support/RemoteProcessTestHelpers.php b/tests/Support/RemoteProcessTestHelpers.phpnew file mode 100644index 0000000000..fd28084079--- /dev/null+++ b/tests/Support/RemoteProcessTestHelpers.php@@ -0,0 +1,36 @@+<?php++use Illuminate\Process\PendingProcess;+use Illuminate\Support\Facades\Process;+use Illuminate\Support\Str;++/**+ * The text of a faked process: its command line and its stdin. SSH command batches go to the+ * server on stdin, so assertions on remote commands must read both.+ */+function remoteProcessScript(PendingProcess $process): string+{+ $command = is_array($process->command) ? implode(' ', $process->command) : (string) $process->command;++ return is_string($process->input) && $process->input !== '' ? $command."\n".$process->input : $command;+}++/**+ * Process::fake() with command patterns that also match the SSH command batch on stdin.+ * Unmatched processes get an empty successful result.+ *+ * @param array<string, mixed> $handlers pattern => result or closure, as for Process::fake()+ */+function fakeRemoteProcesses(array $handlers): void+{+ Process::fake(function (PendingProcess $process) use ($handlers) {+ $script = remoteProcessScript($process);+ foreach ($handlers as $pattern => $handler) {+ if (is_numeric($pattern) || $pattern === '*' || Str::is($pattern, $script)) {+ return $handler instanceof Closure ? $handler($process) : $handler;+ }+ }++ return Process::result();+ });+}diff --git a/tests/Unit/DatabaseImport/DatabaseImportCommandBuilderTest.php b/tests/Unit/DatabaseImport/DatabaseImportCommandBuilderTest.phpindex c78c56d066..c44a19ada9 100644--- a/tests/Unit/DatabaseImport/DatabaseImportCommandBuilderTest.php+++ b/tests/Unit/DatabaseImport/DatabaseImportCommandBuilderTest.php@@ -83,8 +83,8 @@ test('restores dump-all PostgreSQL custom archives with pg_restore from the deco// pg_restore cannot read gzip files, so it must receive the decompressed archive on stdin.expect($command)- ->toContain('stream | pg_restore --exit-on-error -U "$pg_user" -d "$db"')- ->toContain('stream | psql -v ON_ERROR_STOP=1 -U "$pg_user" -d "$db"')+ ->toContain('stream | pg_restore -U "$pg_user" -d "$db"')+ ->toContain('stream | psql -U "$pg_user" -d "$db"')->not->toContain("-d \"\$db\" '/tmp/backup.dump.gz'");});diff --git a/tests/Unit/DatabaseImport/DatabaseImportRestoreScriptTest.php b/tests/Unit/DatabaseImport/DatabaseImportRestoreScriptTest.phpindex f7007689d6..47ebd39e69 100644--- a/tests/Unit/DatabaseImport/DatabaseImportRestoreScriptTest.php+++ b/tests/Unit/DatabaseImport/DatabaseImportRestoreScriptTest.php@@ -562,6 +562,7 @@ test('restores PostgreSQL backups containing all databases after checking the fo['psql', '', ["-U {$user} -d template1 -t -c", 'SELECT datname FROM pg_database']],['dropdb', '', ["-U {$user} --maintenance-db=template1 --if-exists app"]],['createdb', '', ["-U {$user} app"]],+ ['createdb', '', ["-U {$user} postgres"]],];$inspect = $restoreCall[0] === 'pg_restore' ? [['pg_restore', 'PGDMP', ['-l']]] : [];diff --git a/tests/Unit/DeployKeyDedicatedPathTest.php b/tests/Unit/DeployKeyDedicatedPathTest.phpindex 3586994bca..cf84aa2d39 100644--- a/tests/Unit/DeployKeyDedicatedPathTest.php+++ b/tests/Unit/DeployKeyDedicatedPathTest.php@@ -115,7 +115,7 @@ it('supports skipping command log entries without adding a hidden command entry'->and($logs[0]['command'])->toBeNull()->and($logs[0]['output'])->toBe('materialized '.REDACTED);expect($storedLogs)->not->toContain($payload)->not->toContain($command);- Process::assertRan(fn ($process) => str_contains($process->command, $command));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), $command));});it('writes a deploy key to a per-deployment path and cleans it up for ls-remote on the host', function () use ($keyPath) {diff --git a/tests/Unit/InstallScriptAuthorizedKeysTest.php b/tests/Unit/InstallScriptAuthorizedKeysTest.phpindex b2fd5136e4..a105cea8b0 100644--- a/tests/Unit/InstallScriptAuthorizedKeysTest.php+++ b/tests/Unit/InstallScriptAuthorizedKeysTest.php@@ -2,7 +2,7 @@use Symfony\Component\Process\Process;-it('removes only the previous Coolify key from authorized_keys', function (string $path) {+it('removes only previous Coolify keys from authorized_keys', function (string $path) {$script = file_get_contents(dirname(__DIR__, 2).'/'.$path);preg_match('/^\s*(sed -i .*~\/\.ssh\/authorized_keys)$/m', $script, $matches);expect($matches)->toHaveKey(1);@@ -12,9 +12,13 @@ it('removes only the previous Coolify key from authorized_keys', function (strinfile_put_contents($home.'/.ssh/authorized_keys', implode("\n", ['ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey coolify-disposable','ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey operator-lab',- 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHostKey root@coolify',+ 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey me@coolify-laptop',+ 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey coolify-ops key',+ 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey admin@[email protected]','no-pty ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOptions [email protected]','ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOldCoolifyKey coolify',+ // Installers before September 2024 generated their key with the comment "root@coolify".+ 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOlderCoolifyKey root@coolify','',]));@@ -24,7 +28,9 @@ it('removes only the previous Coolify key from authorized_keys', function (strinexpect(file_get_contents($home.'/.ssh/authorized_keys'))->toBe(implode("\n", ['ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey coolify-disposable','ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey operator-lab',- 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHostKey root@coolify',+ 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey me@coolify-laptop',+ 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey coolify-ops key',+ 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOperatorKey admin@[email protected]','no-pty ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOptions [email protected]','',]));diff --git a/tests/Unit/Livewire/Database/S3RestoreTest.php b/tests/Unit/Livewire/Database/S3RestoreTest.phpindex d620c737e8..51bea457a9 100644--- a/tests/Unit/Livewire/Database/S3RestoreTest.php+++ b/tests/Unit/Livewire/Database/S3RestoreTest.php@@ -124,6 +124,19 @@ test('buildRestoreCommand dump-all PostgreSQL restore is pg_restore for PGDMP ot->and(strpos($command, 'kind=archive'))->toBeLessThan(strpos($command, 'dropdb'));});+test('dump-all PostgreSQL restore does not stop on the errors every pg_dumpall dump produces', function () {+ $component = importFormWithResource('App\Models\StandalonePostgresql');+ $component->dumpAll = true;++ // A pg_dumpall dump always runs CREATE ROLE for the existing superuser (and CREATE DATABASE+ // for the recreated target database), and an archive with owners can reference missing+ // roles. Stopping on that first error would leave the server without the dropped databases.+ expect($component->buildRestoreCommand('/tmp/test.dump'))+ ->not->toContain('ON_ERROR_STOP')+ ->not->toContain('--exit-on-error')+ ->not->toContain('--single-transaction');+});+test('dump-all PostgreSQL import text shows the client chosen by the dump format', function () {$component = importFormWithResource('App\Models\StandalonePostgresql');$component->updatedDumpAll(true);@@ -132,8 +145,9 @@ test('dump-all PostgreSQL import text shows the client chosen by the dump format});test('dump-all PostgreSQL restore selects the client for the dump format', function (string $contents, bool $gzip, string $restore, array $preflight) {- // Check the archive, terminate sessions, list databases, recreate the target database, then restore.- expect(invokedPostgresRestoreClients($contents, $gzip))->toBe([...$preflight, 'psql:', 'psql:', 'psql:', 'createdb:', $restore]);+ // Check the archive, terminate sessions, list databases, recreate the target database and the+ // postgres database (POSTGRES_DB is "app" here), then restore.+ expect(invokedPostgresRestoreClients($contents, $gzip))->toBe([...$preflight, 'psql:', 'psql:', 'psql:', 'createdb:', 'createdb:', $restore]);})->with(['custom archive' => ['PGDMP'.str_repeat("\0", 16), false, 'pg_restore:PGDMP', ['pg_restore:PGDMP']],'gzip custom archive' => ['PGDMP'.str_repeat("\0", 16), true, 'pg_restore:PGDMP', ['pg_restore:PGDMP']],diff --git a/tests/Unit/LocalFileVolumeContentSizeTest.php b/tests/Unit/LocalFileVolumeContentSizeTest.phpindex f7b220522f..5ee1e3775d 100644--- a/tests/Unit/LocalFileVolumeContentSizeTest.php+++ b/tests/Unit/LocalFileVolumeContentSizeTest.php@@ -90,11 +90,11 @@ it('does not read regular bind-mounted file contents while loading service setti$application->setRelation('destination', (object) ['server' => $server]);Process::fake(function ($process) {- if (str_contains($process->command, "test -f '/data/large.bin'")) {+ if (str_contains(remoteProcessScript($process), "test -f '/data/large.bin'")) {return Process::result(output: 'OK');}- if (str_contains($process->command, "test -d '/data/large.bin'")) {+ if (str_contains(remoteProcessScript($process), "test -d '/data/large.bin'")) {return Process::result(output: 'NOK');}@@ -104,8 +104,8 @@ it('does not read regular bind-mounted file contents while loading service settigetFilesystemVolumesFromServer($application);expect($volume->is_directory)->toBeFalse();- Process::assertRan(fn ($process) => str_contains($process->command, "test -f '/data/large.bin'"));- Process::assertNotRan(fn ($process) => str_contains($process->command, 'cat /data/large.bin') || str_contains($process->command, 'head -c'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "test -f '/data/large.bin'"));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'cat /data/large.bin') || str_contains(remoteProcessScript($process), 'head -c'));});it('does not interpolate unsafe persisted file-storage paths into remote commands', function (string $path) {@@ -132,7 +132,7 @@ it('does not interpolate unsafe persisted file-storage paths into remote commandProcess::fake();expect(fn () => getFilesystemVolumesFromServer($application, true))->toThrow(Exception::class);- Process::assertNotRan(fn ($process) => str_contains($process->command, $path));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), $path));})->with(['/tmp/evil`id`', '/tmp/evil$(id)', '/tmp/evil;id', '/tmp/evil|id', '${DATA:-/tmp/evil$(id)}', '/srv/$HOME;id', '${DATA:-/srv/app;id}/config.yml', '${DATA:-${HOME:-$(id)}}', '${DATA:+/srv/app;id}/config.yml']);it('preserves safe remote-shell path expansion', function (string $path, array $environment, string $expected) {@@ -194,17 +194,17 @@ it('quotes literal file-storage paths and safely expands persisted expressions',$expression->setRelation('resource', $application);Process::fake(fn ($process) => Process::result(output: match (true) {- str_contains($process->command, 'readlink -f') => 'OK',- str_contains($process->command, 'test -') => 'NOK',+ str_contains(remoteProcessScript($process), 'readlink -f') => 'OK',+ str_contains(remoteProcessScript($process), 'test -') => 'NOK',default => '',}));getFilesystemVolumesFromServer($application, true);- Process::assertRan(fn ($process) => str_contains($process->command, "test -f '/data/application/my files/config.yaml'"));- Process::assertRan(fn ($process) => str_contains($process->command, "mkdir -p -- '/data/application/my files/config.yaml'"));- Process::assertRan(fn ($process) => str_contains($process->command, "dirname -- '/data/application/my files/settings.json'"));- Process::assertRan(fn ($process) => str_contains($process->command, "tee -- '/data/application/my files/settings.json'"));- Process::assertRan(fn ($process) => str_contains($process->command, '${DATA_PATH:-/srv/app/config.yaml}'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "test -f '/data/application/my files/config.yaml'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "mkdir -p -- '/data/application/my files/config.yaml'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "dirname -- '/data/application/my files/settings.json'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), "tee -- '/data/application/my files/settings.json'"));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), '${DATA_PATH:-/srv/app/config.yaml}'));});it('bounds the remote file read itself to prevent a size-check race', function () {diff --git a/tests/Unit/RemoteOutputSizeLimitsTest.php b/tests/Unit/RemoteOutputSizeLimitsTest.phpindex 2dacc7ff28..22a9407691 100644--- a/tests/Unit/RemoteOutputSizeLimitsTest.php+++ b/tests/Unit/RemoteOutputSizeLimitsTest.php@@ -36,15 +36,15 @@ it('bounds dynamic proxy configuration files and their combined Livewire payload$files = collect(range(1, 101))->map(fn (int $number) => sprintf('file%03d.yml', $number))->implode("\n");Process::fake(function ($process) use ($files) {- if (str_contains($process->command, 'ls -1')) {+ if (str_contains(remoteProcessScript($process), 'ls -1')) {return Process::result(output: $files);}- if (str_contains($process->command, 'file001.yml')) {+ if (str_contains(remoteProcessScript($process), 'file001.yml')) {return Process::result(output: str_repeat('x', DynamicConfigurations::MAX_CONFIGURATION_FILE_SIZE_BYTES + 1));}- if (preg_match('/file00[2-6]\.yml/', $process->command)) {+ if (preg_match('/file00[2-6]\.yml/', remoteProcessScript($process))) {return Process::result(output: str_repeat('x', DynamicConfigurations::MAX_CONFIGURATION_FILE_SIZE_BYTES));}@@ -60,7 +60,7 @@ it('bounds dynamic proxy configuration files and their combined Livewire payloadexpect($component->contents)->toHaveCount(5)->toHaveKeys(['file002|yml', 'file003|yml', 'file004|yml', 'file005|yml', 'file006|yml'])->not->toHaveKeys(['file001|yml', 'file007|yml', 'file101|yml']);- Process::assertNotRan(fn ($process) => str_contains($process->command, 'file101.yml'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'file101.yml'));});it('fails explicitly when a source fixture cannot be read', function () {remoteOutputSource('missing-source-fixture.php');@@ -93,7 +93,7 @@ it('rejects oversized docker compose files loaded from git before parsing', func$application->shouldReceive('save')->once();Process::fake(function ($process) {- if (str_contains($process->command, 'git cat-file -s')) {+ if (str_contains(remoteProcessScript($process), 'git cat-file -s')) {return Process::result(output: '__COOLIFY_COMPOSE_TOO_LARGE__');}@@ -103,7 +103,7 @@ it('rejects oversized docker compose files loaded from git before parsing', funcexpect(fn () => $application->loadComposeFile())->toThrow(RuntimeException::class, 'Docker Compose file exceeds the 5 MiB size limit.');expect($application->docker_compose_raw)->toBeNull();- Process::assertRan(fn ($process) => str_contains($process->command, '-gt '.Application::MAX_DOCKER_COMPOSE_SIZE_BYTES));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), '-gt '.Application::MAX_DOCKER_COMPOSE_SIZE_BYTES));});it('bounds scheduled task output before storing or notifying', function () {$source = remoteOutputSource('app/Jobs/ScheduledTaskJob.php');@@ -176,5 +176,5 @@ it('rejects oversized proxy configuration backfill before persistence', functionexpect(fn () => $method->invoke(new GetProxyConfiguration, $server))->toThrow(RuntimeException::class, 'Proxy configuration exceeds the 5 MiB size limit.');- Process::assertRan(fn ($process) => str_contains($process->command, 'head -c '.(GetProxyConfiguration::MAX_CONFIGURATION_SIZE_BYTES + 1)));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'head -c '.(GetProxyConfiguration::MAX_CONFIGURATION_SIZE_BYTES + 1)));});diff --git a/tests/Unit/ServerTransfer/ServerTransferBundleTest.php b/tests/Unit/ServerTransfer/ServerTransferBundleTest.phpindex aa27c86120..5364f99fb4 100644--- a/tests/Unit/ServerTransfer/ServerTransferBundleTest.php+++ b/tests/Unit/ServerTransfer/ServerTransferBundleTest.php@@ -92,3 +92,38 @@ test('validate rejects an unsafe id in an id list', function () {expect(ServerTransferBundle::validate($bundle)['errors'])->toContain('volume_backups.0.s3_storage_uuids.1 is not a valid id. An id may only contain letters, digits, hyphens, and underscores (at most 64 characters).');});++function bundleTestWithDatabasePorts(array $attributes): array+{+ $bundle = bundleTestMinimalBundle();+ $bundle['projects'] = [['environments' => [['databases' => [['attributes' => $attributes]]]]]];++ return $bundle;+}++test('validate accepts the port values that older databases have', function (array $attributes) {+ expect(ServerTransferBundle::validate(bundleTestWithDatabasePorts($attributes))['errors'])->toBe([]);+})->with([+ 'no ports' => [['ports_mappings' => null, 'public_port' => null]],+ 'empty mappings' => [['ports_mappings' => '']],+ 'mapping list' => [['ports_mappings' => '5432:5432,8080:80']],+ 'spaces and a trailing comma' => [['ports_mappings' => '5432:5432, 8080:80,']],+ 'container port only' => [['ports_mappings' => '5432']],+ 'ip, range and protocol' => [['ports_mappings' => '127.0.0.1:8000-8010:8000-8010/udp,[::1]::80']],+ 'public port below 1024' => [['public_port' => 543]],+ 'public port as digit string' => [['public_port' => '5432']],+]);++test('validate rejects garbage database port values', function (array $attributes, string $field) {+ $errors = ServerTransferBundle::validate(bundleTestWithDatabasePorts($attributes))['errors'];++ expect(implode(' ', $errors))->toContain("projects.0.environments.0.databases.0.{$field}");+})->with([+ 'shell in mappings' => [['ports_mappings' => '5432:5432; rm -rf /'], 'ports_mappings'],+ 'yaml in mappings' => [['ports_mappings' => "5432:5432\nprivileged: true"], 'ports_mappings'],+ 'letters in mappings' => [['ports_mappings' => 'abc:def'], 'ports_mappings'],+ 'mappings as array' => [['ports_mappings' => ['5432:5432']], 'ports_mappings'],+ 'public port zero' => [['public_port' => 0], 'public_port'],+ 'public port too high' => [['public_port' => 70000], 'public_port'],+ 'public port text' => [['public_port' => '5432; id'], 'public_port'],+]);diff --git a/tests/Unit/UpdateCoolifyTest.php b/tests/Unit/UpdateCoolifyTest.phpindex 73f0ae6619..c842dffec8 100644--- a/tests/Unit/UpdateCoolifyTest.php+++ b/tests/Unit/UpdateCoolifyTest.php@@ -424,11 +424,11 @@ it('starts an upgrade when the previous upgrade status is older than the lock exit('backs up the instance database before the update', function () {updateCoolifyTestPrepareUpdate();- Process::fake(['*du -b*' => '128', '*' => '']);+ fakeRemoteProcesses(['*du -b*' => '128', '*' => '']);(new UpdateCoolify)->handle(manual_update: true);- Process::assertRan(fn ($process) => str_contains($process->command, 'pg_dump'));+ Process::assertRan(fn ($process) => str_contains(remoteProcessScript($process), 'pg_dump'));expect(RemoteProcessCommand::read(Activity::query()->latest('id')->first()))->toContain("bash /data/coolify/source/upgrade.sh '4.0.10'");});@@ -439,13 +439,13 @@ it('skips the backup before the update when it is disabled', function () {(new UpdateCoolify)->handle(manual_update: true);- Process::assertNotRan(fn ($process) => str_contains($process->command, 'pg_dump'));+ Process::assertNotRan(fn ($process) => str_contains(remoteProcessScript($process), 'pg_dump'));expect(Activity::query()->count())->toBe(1);});it('does not start the update when the instance backup fails', function () {updateCoolifyTestPrepareUpdate();- Process::fake(['*pg_dump*' => Process::result(exitCode: 1), '*' => '']);+ fakeRemoteProcesses(['*pg_dump*' => Process::result(exitCode: 1), '*' => '']);expect(fn () => (new UpdateCoolify)->handle(manual_update: true))->toThrow(Exception::class, 'The database backup failed, so the update was not started.')diff --git a/tests/v4/Browser/CloneVolumeDataConfirmationTest.php b/tests/v4/Browser/CloneVolumeDataConfirmationTest.phpnew file mode 100644index 0000000000..055ef26f74--- /dev/null+++ b/tests/v4/Browser/CloneVolumeDataConfirmationTest.php@@ -0,0 +1,53 @@+<?php++use App\Models\Project;+use App\Models\StandalonePostgresql;+use Illuminate\Foundation\Testing\RefreshDatabase;++uses(RefreshDatabase::class);++beforeEach(function () {+ config()->set('app.maintenance.store', 'array');+ $this->stack = seedBrowserResourceStack();+ $this->database = createBrowserPostgresql($this->stack, ['uuid' => 'db-clone-volume-data', 'name' => 'Clone Source DB']);+});++it('asks for a confirmation before a resource clone with volume data', function () {+ loginAndSkipBoarding();++ $page = visit(databaseConfigurationUrl($this->stack['project'], $this->stack['environment'], $this->database).'/resource-operations')+ ->assertSee('Clone volume data')+ ->assertSee('The running resource will not be changed.')+ ->click('input[id^="cloneVolumeData-"]')+ ->wait(1)+ ->assertSee('A running resource is stopped while its volume data is copied.')+ ->click('#clone-resource-section button.button:has-text("Clone resource")')+ ->wait(1)+ ->assertSee('Clone this resource with its volume data?')+ ->assertSee('If the resource is running, it is stopped while its volume data is copied.')+ ->screenshot(filename: 'clone-resource-volume-data-confirmation');++ expect(StandalonePostgresql::query()->count())->toBe(1);+});++it('asks for a confirmation before an environment clone with volume data', function () {+ loginAndSkipBoarding();++ $project = $this->stack['project'];+ $environment = $this->stack['environment'];++ visit("/project/{$project->uuid}/environment/{$environment->uuid}/clone")+ ->assertSee('Clone volume data')+ ->click('input[id^="cloneVolumeData-"]')+ ->wait(1)+ ->click('coolify')+ ->wait(1)+ ->click('Clone to environment')+ ->wait(1)+ ->assertSee('Clone with volume data?')+ ->assertSee('Running source resources are stopped while their volume data is copied.')+ ->screenshot(filename: 'clone-environment-volume-data-confirmation');++ expect(Project::query()->count())->toBe(1)+ ->and($environment->project->environments()->count())->toBe(1);+});