Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Make MSP isolation checks credential-safe and TLS-verified
pulse-triage[bot]5e6d5af3b53e
diff --git a/docs/MSP.md b/docs/MSP.mdindex cb746ec8a7..7561f520c6 100644--- a/docs/MSP.md+++ b/docs/MSP.md@@ -110,11 +110,37 @@ the tunnels and keep the management port out of them.### Validation checklist (run after setup, repeat after network changes)+Use your actual HTTPS addresses and keep certificate verification enabled.+For a private CA, add `--cacert /path/to/trusted-ca.pem` using a CA you have+verified independently; do not bypass a certificate error with `--insecure`.+The examples make bounded **GET** requests, discard response bodies and print+only the HTTP status. They do not send notifications or change settings.++For the authenticated checks, prepare two separate private header files with+[the API token file procedure](API.md#api-token-recommended), substituting+`agent-header` and `org-a-header` for `api-header`. Keep the directory mode+`700` and files mode `600`, owned by the account running curl. Enter the token+only in your local editor, never in a shell command, URL or report. Each file+contains one `X-API-Token` header: the existing `agent:report` token in+`agent-header`, and an existing org-A token with `monitoring:read` in+`org-a-header`. Do not broaden scopes or create a privileged token just to run+these checks. Keep the files outside repositories, backups shared for support+and diagnostic exports; remove them locally when no longer needed.++Keep `--disable` first so a local curl configuration cannot enable tracing,+extra credentials or redirects. Do not add verbose/trace output or follow+redirects with these tokens. For the HTTP-status checks below, a TLS, DNS,+timeout or connection error is a **failed check**, not proof of isolation:+require curl exit **0** as well as the expected HTTP status. `000` is not an HTTP result. Do not use `--fail` for these+checks because the expected rejections are HTTP errors.+1. **Agent port excludes management surfaces.** Both must return `404`:```bash- curl -sk -o /dev/null -w '%{http_code}\n' https://agents.example.com:7656/ # 404- curl -sk -o /dev/null -w '%{http_code}\n' https://agents.example.com:7656/api/login # 404+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' https://agents.example.com:7656/ # 404+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' https://agents.example.com:7656/api/login # 404```When commands are enabled, also verify that `/api/agent/ws` reaches Pulse@@ -123,33 +149,45 @@ the tunnels and keep the management port out of them.2. **Management port is not reachable from a client site.** From a clientnetwork (or through a client tunnel), a connection to `FRONTEND_PORT` must- time out or be refused by your firewall — not answer.+ time out or be refused by your firewall — not answer. That observation alone+ does not distinguish a firewall from a DNS, TLS or server failure; confirm+ the same management endpoint works from the authorised staff network.-3. **Agent tokens cannot manage.** A request to a management endpoint with an- agent token must be rejected:+3. **Agent tokens cannot manage.** On an authentication-enabled instance, a+ request to a management endpoint with the agent token must be rejected:```bash- curl -sk -o /dev/null -w '%{http_code}\n' \- -H "X-API-Token: <agent:report token>" https://pulse.internal:7655/api/notifications/webhooks # 401/403+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' \+ --header "@$HOME/.config/pulse/agent-header" \+ https://pulse.internal:7655/api/notifications/webhooks # 401/403```-4. **Cross-tenant isolation (shared-process mode only).** A token bound to one- organization must get `403` when targeting another organization AND when- targeting the default org (a leaked client-site token must not read the- provider's own estate):+4. **Cross-tenant isolation (shared-process mode only).** A valid token bound+ to org-A with `monitoring:read` must get `403` when targeting another+ organization AND when targeting the default org (a leaked client-site+ token must not read the provider's own estate):```bash- curl -sk -o /dev/null -w '%{http_code}\n' \- -H "X-API-Token: <org-A token>" -H "X-Pulse-Org-ID: org-b" \- https://pulse.internal:7655/api/alerts/active # 403- curl -sk -o /dev/null -w '%{http_code}\n' \- -H "X-API-Token: <org-A token>" -H "X-Pulse-Org-ID: default" \- https://pulse.internal:7655/api/alerts/active # 403+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' \+ --header "@$HOME/.config/pulse/org-a-header" --header 'X-Pulse-Org-ID: org-b' \+ https://pulse.internal:7655/api/alerts/active # 403+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' \+ --header "@$HOME/.config/pulse/org-a-header" --header 'X-Pulse-Org-ID: default' \+ https://pulse.internal:7655/api/alerts/active # 403```+ An expired or otherwise invalid token can give `401` before the tenant+ boundary is checked. First confirm that this same token can read+ `/api/alerts/active` in org-A through the private header-file method; do not+ count an authentication failure as cross-tenant acceptance.+Keep your own monitoring estate in its own organization too, rather thanin the default org, so every boundary in the instance is an explicit org- boundary.+ boundary. For help, share only the check, redacted endpoint role, status and+ curl exit; never the header file, token, response body or trace.## Connecting a client's Proxmox or PBS over your VPN@@ -169,8 +207,11 @@ Per client, the steps are:1. Make the client's PVE/PBS API address reachable from the Docker host thatruns the client workspaces (route or interface into that client's- tunnel). From the host, `curl -sk https://<client-pve>:8006` should- answer before you involve Pulse.+ tunnel). From that host, make a bounded unauthenticated HTTPS request to+ the PVE/PBS API with certificate verification enabled. For a private or+ self-signed certificate, use `--cacert` with an independently verified+ certificate/CA. A TLS error is not proof of a blocked tunnel, and an HTTP+ response is not proof that Pulse's configured account can collect data.2. Open the client's workspace (portal → workspace → **Open**) and add thenode under **Settings → Infrastructure**, using the tunnel-reachableaddress. The guided flow generates a setup command to run once on thediff --git a/frontend-modern/public/docs/MSP.md b/frontend-modern/public/docs/MSP.mdindex cb746ec8a7..7561f520c6 100644--- a/frontend-modern/public/docs/MSP.md+++ b/frontend-modern/public/docs/MSP.md@@ -110,11 +110,37 @@ the tunnels and keep the management port out of them.### Validation checklist (run after setup, repeat after network changes)+Use your actual HTTPS addresses and keep certificate verification enabled.+For a private CA, add `--cacert /path/to/trusted-ca.pem` using a CA you have+verified independently; do not bypass a certificate error with `--insecure`.+The examples make bounded **GET** requests, discard response bodies and print+only the HTTP status. They do not send notifications or change settings.++For the authenticated checks, prepare two separate private header files with+[the API token file procedure](API.md#api-token-recommended), substituting+`agent-header` and `org-a-header` for `api-header`. Keep the directory mode+`700` and files mode `600`, owned by the account running curl. Enter the token+only in your local editor, never in a shell command, URL or report. Each file+contains one `X-API-Token` header: the existing `agent:report` token in+`agent-header`, and an existing org-A token with `monitoring:read` in+`org-a-header`. Do not broaden scopes or create a privileged token just to run+these checks. Keep the files outside repositories, backups shared for support+and diagnostic exports; remove them locally when no longer needed.++Keep `--disable` first so a local curl configuration cannot enable tracing,+extra credentials or redirects. Do not add verbose/trace output or follow+redirects with these tokens. For the HTTP-status checks below, a TLS, DNS,+timeout or connection error is a **failed check**, not proof of isolation:+require curl exit **0** as well as the expected HTTP status. `000` is not an HTTP result. Do not use `--fail` for these+checks because the expected rejections are HTTP errors.+1. **Agent port excludes management surfaces.** Both must return `404`:```bash- curl -sk -o /dev/null -w '%{http_code}\n' https://agents.example.com:7656/ # 404- curl -sk -o /dev/null -w '%{http_code}\n' https://agents.example.com:7656/api/login # 404+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' https://agents.example.com:7656/ # 404+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' https://agents.example.com:7656/api/login # 404```When commands are enabled, also verify that `/api/agent/ws` reaches Pulse@@ -123,33 +149,45 @@ the tunnels and keep the management port out of them.2. **Management port is not reachable from a client site.** From a clientnetwork (or through a client tunnel), a connection to `FRONTEND_PORT` must- time out or be refused by your firewall — not answer.+ time out or be refused by your firewall — not answer. That observation alone+ does not distinguish a firewall from a DNS, TLS or server failure; confirm+ the same management endpoint works from the authorised staff network.-3. **Agent tokens cannot manage.** A request to a management endpoint with an- agent token must be rejected:+3. **Agent tokens cannot manage.** On an authentication-enabled instance, a+ request to a management endpoint with the agent token must be rejected:```bash- curl -sk -o /dev/null -w '%{http_code}\n' \- -H "X-API-Token: <agent:report token>" https://pulse.internal:7655/api/notifications/webhooks # 401/403+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' \+ --header "@$HOME/.config/pulse/agent-header" \+ https://pulse.internal:7655/api/notifications/webhooks # 401/403```-4. **Cross-tenant isolation (shared-process mode only).** A token bound to one- organization must get `403` when targeting another organization AND when- targeting the default org (a leaked client-site token must not read the- provider's own estate):+4. **Cross-tenant isolation (shared-process mode only).** A valid token bound+ to org-A with `monitoring:read` must get `403` when targeting another+ organization AND when targeting the default org (a leaked client-site+ token must not read the provider's own estate):```bash- curl -sk -o /dev/null -w '%{http_code}\n' \- -H "X-API-Token: <org-A token>" -H "X-Pulse-Org-ID: org-b" \- https://pulse.internal:7655/api/alerts/active # 403- curl -sk -o /dev/null -w '%{http_code}\n' \- -H "X-API-Token: <org-A token>" -H "X-Pulse-Org-ID: default" \- https://pulse.internal:7655/api/alerts/active # 403+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' \+ --header "@$HOME/.config/pulse/org-a-header" --header 'X-Pulse-Org-ID: org-b' \+ https://pulse.internal:7655/api/alerts/active # 403+ curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \+ --output /dev/null --write-out '%{http_code}\n' \+ --header "@$HOME/.config/pulse/org-a-header" --header 'X-Pulse-Org-ID: default' \+ https://pulse.internal:7655/api/alerts/active # 403```+ An expired or otherwise invalid token can give `401` before the tenant+ boundary is checked. First confirm that this same token can read+ `/api/alerts/active` in org-A through the private header-file method; do not+ count an authentication failure as cross-tenant acceptance.+Keep your own monitoring estate in its own organization too, rather thanin the default org, so every boundary in the instance is an explicit org- boundary.+ boundary. For help, share only the check, redacted endpoint role, status and+ curl exit; never the header file, token, response body or trace.## Connecting a client's Proxmox or PBS over your VPN@@ -169,8 +207,11 @@ Per client, the steps are:1. Make the client's PVE/PBS API address reachable from the Docker host thatruns the client workspaces (route or interface into that client's- tunnel). From the host, `curl -sk https://<client-pve>:8006` should- answer before you involve Pulse.+ tunnel). From that host, make a bounded unauthenticated HTTPS request to+ the PVE/PBS API with certificate verification enabled. For a private or+ self-signed certificate, use `--cacert` with an independently verified+ certificate/CA. A TLS error is not proof of a blocked tunnel, and an HTTP+ response is not proof that Pulse's configured account can collect data.2. Open the client's workspace (portal → workspace → **Open**) and add thenode under **Settings → Infrastructure**, using the tunnel-reachableaddress. The guided flow generates a setup command to run once on thediff --git a/scripts/installtests/msp_diagnostic_guidance_test.go b/scripts/installtests/msp_diagnostic_guidance_test.gonew file mode 100644index 0000000000..59cd38de1d--- /dev/null+++ b/scripts/installtests/msp_diagnostic_guidance_test.go@@ -0,0 +1,266 @@+package installtests++import (+ "context"+ "encoding/pem"+ "fmt"+ "io"+ "log"+ "net/http"+ "net/http/httptest"+ "os"+ "os/exec"+ "path/filepath"+ "regexp"+ "strings"+ "testing"+ "time"+)++// Exercise the copied recipes against a synthetic TLS peer, not a customer's+// management API. This proves curl transport/credential handling, not Pulse's+// tenant authorisation or an installed provider deployment.+func mspValidationRecipes(t *testing.T) []string {+ t.Helper()+ doc, err := os.ReadFile(repoFile("docs", "MSP.md"))+ if err != nil {+ t.Fatal(err)+ }+ const heading = "### Validation checklist (run after setup, repeat after network changes)\n"+ _, section, found := strings.Cut(string(doc), heading)+ if !found {+ t.Fatal("MSP validation checklist is missing")+ }+ section, _, _ = strings.Cut(section, "\n## ")+ blocks := regexp.MustCompile("(?s)```bash\n(.*?)```").FindAllStringSubmatch(section, -1)+ var recipes []string+ for _, block := range blocks {+ // A continuation is one command; test each command separately so the+ // second public probe cannot mask the first probe's failed exit.+ joined := regexp.MustCompile(`\\\n[ \t]*`).ReplaceAllString(block[1], " ")+ for _, line := range strings.Split(joined, "\n") {+ line = strings.TrimSpace(line)+ if line == "" || strings.HasPrefix(line, "#") {+ continue+ }+ if err := mspRecipeSafety(line); err != nil {+ t.Fatalf("unsafe copied MSP recipe: %v", err)+ }+ recipes = append(recipes, line)+ }+ }+ if len(recipes) != 5 {+ t.Fatalf("want five HTTP checks, got %d", len(recipes))+ }+ return recipes+}++func mspRecipeSafety(recipe string) error {+ for _, required := range []string{+ "curl --disable ", "--silent --show-error", "--proto '=https'",+ "--connect-timeout 5", "--max-time 10", "--output /dev/null",+ "--write-out '%{http_code}\\n'", "https://",+ } {+ if !strings.Contains(recipe, required) {+ return fmt.Errorf("missing %s", required)+ }+ }+ for _, forbidden := range []string{+ "--insecure", "--location", "--config", "--trace", "--verbose",+ "--fail", "X-API-Token:", "Authorization:", "--data", "--request",+ } {+ if strings.Contains(recipe, forbidden) {+ return fmt.Errorf("forbidden %s", forbidden)+ }+ }+ if regexp.MustCompile(`(?:^|\s)-[A-Za-z]`).MatchString(recipe) ||+ strings.ContainsAny(recipe, "|;`") || strings.Contains(recipe, "$(") {+ return fmt.Errorf("unexpected short flag, pipeline or shell expansion")+ }+ if strings.Contains(recipe, "pulse.internal") &&+ !strings.Contains(recipe, `--header "@$HOME/.config/pulse/`) {+ return fmt.Errorf("authenticated request lacks private header file")+ }+ return nil+}++func TestMSPDiagnosticGuidanceRejectsUnsafeRecipes(t *testing.T) {+ recipe := mspValidationRecipes(t)[2]+ for name, unsafe := range map[string]string{+ "curl config": strings.Replace(recipe, "--disable ", "", 1),+ "TLS bypass": strings.Replace(recipe, "--disable ", "--disable --insecure ", 1),+ "short bypass": strings.Replace(recipe, "--disable ", "--disable -k ", 1),+ "redirect": strings.Replace(recipe, "--disable ", "--disable --location ", 1),+ "trace": strings.Replace(recipe, "--disable ", "--disable --trace /tmp/trace ", 1),+ "inline token": strings.Replace(recipe, `"@$HOME/.config/pulse/agent-header"`,+ `"X-API-Token: synthetic-inline-secret"`, 1),+ "missing deadline": strings.Replace(recipe, "--max-time 10 ", "", 1),+ "missing connect bound": strings.Replace(recipe, "--connect-timeout 5 ", "", 1),+ "body disclosure": strings.Replace(recipe, "--output /dev/null ", "", 1),+ "HTTP": strings.Replace(recipe, "https://", "http://", 1),+ "pipe hides exit": recipe + " | cat",+ } {+ t.Run(name, func(t *testing.T) {+ if mspRecipeSafety(unsafe) == nil {+ t.Fatal("unsafe mutation was accepted")+ }+ })+ }+}++type mspProbeRequest struct {+ method, path, token, org string+}++type mspProbeLab struct {+ home, ca, capture, trace, curl string+ server *httptest.Server+ requests chan mspProbeRequest+}++func newMSPProbeLab(t *testing.T, redirect bool) *mspProbeLab {+ t.Helper()+ realCurl, err := exec.LookPath("curl")+ if err != nil {+ t.Fatal("curl is required to verify copied MSP diagnostics")+ }+ lab := &mspProbeLab{home: t.TempDir(), curl: realCurl, requests: make(chan mspProbeRequest, 8)}+ lab.ca = filepath.Join(lab.home, "trusted-ca.pem")+ lab.capture = filepath.Join(lab.home, "argv")+ lab.trace = filepath.Join(lab.home, "forbidden-trace")+ lab.server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ lab.requests <- mspProbeRequest{r.Method, r.URL.Path, r.Header.Get("X-API-Token"), r.Header.Get("X-Pulse-Org-ID")}+ if redirect {+ http.Redirect(w, r, lab.server.URL+"/must-not-follow", http.StatusFound)+ return+ }+ code := http.StatusForbidden+ if r.URL.Path == "/" || r.URL.Path == "/api/login" {+ code = http.StatusNotFound+ }+ w.WriteHeader(code)+ // Even an upstream body echoing credentials must be discarded.+ fmt.Fprint(w, "synthetic-agent-secret synthetic-org-secret private-body")+ }))+ lab.server.Config.ErrorLog = log.New(io.Discard, "", 0)+ lab.server.StartTLS()+ t.Cleanup(lab.server.Close)+ cert := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: lab.server.Certificate().Raw})+ if err := os.WriteFile(lab.ca, cert, 0600); err != nil {+ t.Fatal(err)+ }+ dir := filepath.Join(lab.home, ".config", "pulse")+ if err := os.MkdirAll(dir, 0700); err != nil {+ t.Fatal(err)+ }+ for name, token := range map[string]string{"agent-header": "synthetic-agent-secret", "org-a-header": "synthetic-org-secret"} {+ if err := os.WriteFile(filepath.Join(dir, name), []byte("X-API-Token: "+token+"\n"), 0600); err != nil {+ t.Fatal(err)+ }+ }+ // --disable must ignore settings that would disclose headers, bypass TLS+ // or follow redirects, even on a user's otherwise configured curl.+ if err := os.WriteFile(filepath.Join(lab.home, ".curlrc"), []byte("insecure\nlocation\ntrace = "+lab.trace+"\n"), 0600); err != nil {+ t.Fatal(err)+ }+ bin := filepath.Join(lab.home, "bin")+ if err := os.Mkdir(bin, 0700); err != nil {+ t.Fatal(err)+ }+ wrapper := "#!/bin/sh\nprintf '%s\\0' \"$@\" > \"$MSP_ARGV\"\nexec \"$MSP_CURL\" \"$@\"\n"+ if err := os.WriteFile(filepath.Join(bin, "curl"), []byte(wrapper), 0700); err != nil {+ t.Fatal(err)+ }+ return lab+}++func (lab *mspProbeLab) run(t *testing.T, recipe string, trustCA bool) (string, int) {+ t.Helper()+ // Replace only documented endpoint placeholders; the CA option is the+ // guide's explicit private-CA alternative. Credentials never enter argv.+ recipe = strings.ReplaceAll(recipe, "https://agents.example.com:7656", lab.server.URL)+ recipe = strings.ReplaceAll(recipe, "https://pulse.internal:7655", lab.server.URL)+ if trustCA {+ recipe = strings.Replace(recipe, "curl --disable ", `curl --disable --cacert "$MSP_CA" `, 1)+ }+ ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)+ defer cancel()+ command := exec.CommandContext(ctx, "bash", "-c", recipe)+ command.Env = append(os.Environ(), "HOME="+lab.home, "PATH="+filepath.Join(lab.home, "bin")+":"+os.Getenv("PATH"),+ "MSP_ARGV="+lab.capture, "MSP_CURL="+lab.curl, "MSP_CA="+lab.ca,+ "https_proxy=", "HTTPS_PROXY=", "http_proxy=", "HTTP_PROXY=", "ALL_PROXY=", "all_proxy=")+ output, err := command.CombinedOutput()+ if ctx.Err() != nil {+ t.Fatalf("copied recipe exceeded test deadline: %v", ctx.Err())+ }+ exit := 0+ if err != nil {+ failure, ok := err.(*exec.ExitError)+ if !ok {+ t.Fatal(err)+ }+ exit = failure.ExitCode()+ }+ argv, err := os.ReadFile(lab.capture)+ if err != nil {+ t.Fatal(err)+ }+ for _, secret := range []string{"synthetic-agent-secret", "synthetic-org-secret", "private-body"} {+ if strings.Contains(string(argv), secret) || strings.Contains(string(output), secret) {+ t.Fatal("copied recipe disclosed synthetic private data")+ }+ }+ if _, err := os.Stat(lab.trace); !os.IsNotExist(err) {+ t.Fatal("curl local configuration enabled a trace")+ }+ return string(output), exit+}++func TestMSPDiagnosticGuidanceCopiedChecksUsePrivateHeadersAndTrustedTLS(t *testing.T) {+ lab := newMSPProbeLab(t, false)+ want := []mspProbeRequest{+ {"GET", "/", "", ""}, {"GET", "/api/login", "", ""},+ {"GET", "/api/notifications/webhooks", "synthetic-agent-secret", ""},+ {"GET", "/api/alerts/active", "synthetic-org-secret", "org-b"},+ {"GET", "/api/alerts/active", "synthetic-org-secret", "default"},+ }+ for i, recipe := range mspValidationRecipes(t) {+ t.Run(fmt.Sprint(i+1), func(t *testing.T) {+ output, exit := lab.run(t, recipe, true)+ expected := "403\n"+ if i < 2 {+ expected = "404\n"+ }+ if exit != 0 || output != expected {+ t.Fatalf("curl result = %q, exit %d; want %q, exit 0", output, exit, expected)+ }+ select {+ case got := <-lab.requests:+ if got != want[i] {+ t.Fatal("request method, path or private header did not match the documented check")+ }+ default:+ t.Fatal("copied check sent no request")+ }+ })+ }+}++func TestMSPDiagnosticGuidanceUntrustedTLSRemainsFailure(t *testing.T) {+ lab := newMSPProbeLab(t, false)+ output, exit := lab.run(t, mspValidationRecipes(t)[2], false)+ if exit != 60 || !strings.HasSuffix(output, "000\n") {+ t.Fatalf("want certificate failure/000, got exit %d, %q", exit, output)+ }+ if len(lab.requests) != 0 {+ t.Fatal("untrusted peer received an authenticated HTTP request")+ }+}++func TestMSPDiagnosticGuidanceDoesNotFollowRedirects(t *testing.T) {+ lab := newMSPProbeLab(t, true)+ output, exit := lab.run(t, mspValidationRecipes(t)[2], true)+ if exit != 0 || output != "302\n" || len(lab.requests) != 1 {+ t.Fatalf("redirect must remain an unexpected status, not a followed check: %q, exit %d", output, exit)+ }+}