Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Bound copied log diagnostics by bytes as well as records

pulse-triage[bot]fcc633337ed3
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index cd310d3d04..731908474a 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -350,14 +350,31 @@ incident's time window; no new action or request ID is needed.
```bash
# systemd / Proxmox LXC — warning and higher priority
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm cat; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse -p warning --since '15 minutes ago' --lines 200 --no-pager 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs"
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+journalctl -u pulse -p warning --since '15 minutes ago' --lines 200 --no-pager 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
+ fi
+ cat -- "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -366,10 +383,11 @@ fi
)
```
-This requires **GNU `timeout`**, reads at most 200 records from the last 15
-minutes and allows eight seconds plus a one-second termination grace. A failed
-or timed-out read is unavailable, not an empty warning search; no partial
-excerpt is shown. Stop if the utility or journal access is unavailable; do not
+This requires **GNU `timeout`** and the listed local utilities, reads at most
+200 records from the last 15 minutes, and limits the excerpt to **256 KiB**.
+The eight-second hard deadline covers the reader and byte limiter together.
+A failed, oversized or timed-out read is unavailable, not an empty warning
+search; no partial excerpt is shown. Stop if a utility or journal access is unavailable; do not
use an unbounded substitute. These local excerpts are **not sanitised**; follow
[log-sharing precautions](#inspect-notification-logs) before sharing any line.
@@ -1283,22 +1301,42 @@ inside the Pulse container, not on the Proxmox host. Adjust the time window to
the original incident and substitute your actual service or container name
(`pulse-backend` on some older systemd installs). These examples read at most
200 records from the last 15 minutes; they do not follow the log or send a test.
-Each requires **GNU `timeout`** and gives the reader eight seconds plus a
-one-second termination grace. If it is unavailable, stop rather than running
-an unbounded substitute. These are Bash examples; the enclosing subshell keeps
-an unavailable read from exiting your interactive shell.
+Each requires **GNU `timeout`** and the listed local utilities. A **256 KiB**
+byte limit complements the record limit: a single log record can be large.
+The eight-second hard deadline kills only the read-only collection group,
+including the reader and byte limiter, not Pulse or Docker. If a required
+utility is unavailable, stop rather than running an unbounded substitute.
+These are Bash examples; the enclosing subshell keeps an unavailable read from
+exiting your interactive shell.
```bash
# systemd / Proxmox LXC
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm cat; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs"
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
+ fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
fi
+ cat -- "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -1310,14 +1348,31 @@ fi
```bash
# Docker
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm cat; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s docker logs --since 15m --tail 200 pulse 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs"
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+docker logs --since 15m --tail 200 pulse 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
+ fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
fi
+ cat -- "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -1327,10 +1382,14 @@ fi
```
Application logs can be written to stdout or stderr; these commands collect
-both into one local excerpt, without preserving stdout/stderr attribution,
-and display it **only after the reader completes successfully**.
-A failed or timed-out read displays only an unavailable message and its exit
-code, not a partial excerpt that could be mistaken for complete evidence.
+both into one owner-only temporary file, without preserving stdout/stderr
+attribution, and display it **only after the reader completes successfully**
+and the byte limit is checked. The file holds at most 256 KiB plus one byte
+used to detect overflow; it is removed when the subshell exits. A failed,
+oversized or timed-out read shows no partial excerpt that could be mistaken
+for complete evidence. If the shell is forcibly interrupted, an owner-only
+`pulse-log.*` file may remain in its temporary directory; remove that exact
+file privately, not unrelated logs or deployment data.
Do not pipe the reader into `grep email`: it can miss SMTP or webhook errors
and hide a failed read behind a matching partial line. A nonzero reader exit,
access error or missing service/container is a failed read, not “no delivery
@@ -1358,25 +1417,44 @@ Replace `abc123` below with the response's ID. Run only the command for your
deployment, on the Pulse host using an account authorised to read its logs.
These examples limit collection to the last 15 minutes and 1,000 lines; adjust
the time window to the original incident rather than repeating the failed action.
-Like the notification-log examples above, these require **GNU `timeout`**,
-allow eight seconds plus a one-second termination grace, and search **only a
-successfully completed read**. A deadline or failure shows no partial match;
-missing `timeout` does not fall back to an unbounded reader.
+Like the notification-log examples above, these require **GNU `timeout`**
+and the listed local utilities, enforce an eight-second hard deadline and a
+**256 KiB** byte limit, and search **only a successfully completed read**.
+They use an owner-only temporary file with the same cleanup precautions. A
+deadline, overflow or failure shows no partial match; a missing utility does
+not fall back to an unbounded reader. Keep the original response status, time
+and ID if the chosen window is too large; do not raise the limit or repeat the
+failed action to fill that gap.
```bash
# systemd / Proxmox LXC
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm grep; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
REQUEST_ID='abc123'
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse --since '15 minutes ago' --lines 1000 --no-pager 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs" | grep -F -- "$REQUEST_ID"
- else
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+journalctl -u pulse --since '15 minutes ago' --lines 1000 --no-pager 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
exit 1
fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
+ fi
+ grep -aF -- "$REQUEST_ID" "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -1388,17 +1466,32 @@ fi
```bash
# Docker
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm grep; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
REQUEST_ID='abc123'
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s docker logs --since 15m --tail 1000 pulse 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs" | grep -F -- "$REQUEST_ID"
- else
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+docker logs --since 15m --tail 1000 pulse 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
+ fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
exit 1
fi
+ grep -aF -- "$REQUEST_ID" "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -1408,9 +1501,11 @@ fi
```
A log-reader failure is not an empty search result: resolve any access or
-container/service error locally first. Exit **124** means the reader exceeded
-the deadline; **137** can mean the termination grace also elapsed. Neither
-establishes that Pulse itself is hung. Do not restart Pulse or Docker, enable
+container/service error locally first. Exit **137** can mean the hard deadline
+killed the collection group; an oversized excerpt is also unavailable. The
+byte limiter can close the pipe before the reader finishes, producing a
+nonzero reader exit such as **141**. These are collection results, not proof
+that Pulse itself is hung. Do not restart Pulse or Docker, enable
Debug, widen collection or repeat the failed action merely to obtain logs.
Even a successful read with no match does not prove the request succeeded. At the default log level, this middleware
logs HTTP 5xx failures but not successful requests; HTTP 4xx failures are logged
diff --git a/docs/TRUENAS.md b/docs/TRUENAS.md
index eceeab1a81..42b718a448 100644
--- a/docs/TRUENAS.md
+++ b/docs/TRUENAS.md
@@ -302,10 +302,12 @@ actual service or container name (`pulse-backend` on some older systemd
installs), and adjust the time window to the original incident.
Those Bash recipes read at most 200 records from the last 15 minutes and
-require **GNU `timeout`**, with an eight-second deadline and one-second
-termination grace. A record limit alone does not bound a hung reader. If
-`timeout` is unavailable, stop; do not use an unbounded substitute. They do
-not follow logs, make API requests or change logging.
+require **GNU `timeout`** and the listed local utilities, with an eight-second
+hard deadline and a **256 KiB** byte limit. A record limit alone does not bound
+a hung reader or a large record. If a required utility is unavailable, stop;
+do not use an unbounded substitute. They do not follow logs, make API requests
+or change logging. The linked reader explains its owner-only temporary file
+and cleanup precautions.
Both output streams are captured because Docker can write application logs
to stderr. An excerpt is shown only after a successful read; failure or
diff --git a/docs/UNIFIED_AGENT.md b/docs/UNIFIED_AGENT.md
index 59eab8aec2..469b68c8b7 100644
--- a/docs/UNIFIED_AGENT.md
+++ b/docs/UNIFIED_AGENT.md
@@ -1007,14 +1007,31 @@ contact Pulse, a container runtime or a guest agent.
```bash
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Agent log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm cat; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Agent log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! agent_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Agent log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
-if agent_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse-agent.service --since '15 minutes ago' --lines 200 --no-pager --output=cat 2>&1); then
- if [ -n "$agent_logs" ]; then
- printf '%s\n' "$agent_logs"
+fi
+trap 'rm -- "$agent_log_file"' EXIT
+if timeout --signal=KILL 8s bash -o pipefail > "$agent_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+journalctl -u pulse-agent.service --since '15 minutes ago' --lines 200 --no-pager --output=cat 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! agent_log_bytes=$(wc -c < "$agent_log_file" 2>/dev/null); then
+ printf 'Agent log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
fi
+ if [ "$agent_log_bytes" -gt 262144 ]; then
+ printf 'Agent log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
+ fi
+ cat -- "$agent_log_file"
else
agent_log_status=$?
printf 'Agent log read unavailable (exit %s); no partial excerpt shown.\n' "$agent_log_status" >&2
@@ -1023,15 +1040,23 @@ fi
)
```
-This requires **GNU `timeout`**, reads at most 200 records from the default
-15-minute window, and allows eight seconds plus a one-second termination grace.
-A failed or timed-out read is unavailable, not an empty search; partial output
-is withheld. Stop if the utility, service journal or existing access is
+This requires **GNU `timeout`** and the listed local utilities, reads at most
+200 records from the default 15-minute window, and limits the excerpt to
+**256 KiB**: a record limit alone does not bound bytes. The eight-second hard
+deadline kills only the read-only collection group, including the reader and
+byte limiter, not the agent. A failed, oversized or timed-out read is
+unavailable, not an empty search; partial output is withheld. Stop if a
+utility, service journal or existing access is
unavailable; do not follow the journal indefinitely, remove the deadline or
broaden permissions to collect it. A successful empty read is inconclusive:
the event may be outside the window or absent at the configured log level.
Do not restart, re-enrol or enable debug logging just to obtain an entry.
+Both streams are buffered in an owner-only temporary file, capped at 256 KiB
+plus one overflow-detection byte and removed when the subshell exits. If the
+shell is forcibly interrupted, a private `pulse-log.*` file may remain in its
+temporary directory; remove that exact file locally, not saved agent state.
+
The excerpt is **not sanitised**. Review it locally before searching for the
relevant error or sharing a manually redacted timestamp and failure stage.
Tokens, cookies, URLs, private names or addresses can appear in errors; do not
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index cd310d3d04..731908474a 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -350,14 +350,31 @@ incident's time window; no new action or request ID is needed.
```bash
# systemd / Proxmox LXC — warning and higher priority
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm cat; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse -p warning --since '15 minutes ago' --lines 200 --no-pager 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs"
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+journalctl -u pulse -p warning --since '15 minutes ago' --lines 200 --no-pager 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
+ fi
+ cat -- "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -366,10 +383,11 @@ fi
)
```
-This requires **GNU `timeout`**, reads at most 200 records from the last 15
-minutes and allows eight seconds plus a one-second termination grace. A failed
-or timed-out read is unavailable, not an empty warning search; no partial
-excerpt is shown. Stop if the utility or journal access is unavailable; do not
+This requires **GNU `timeout`** and the listed local utilities, reads at most
+200 records from the last 15 minutes, and limits the excerpt to **256 KiB**.
+The eight-second hard deadline covers the reader and byte limiter together.
+A failed, oversized or timed-out read is unavailable, not an empty warning
+search; no partial excerpt is shown. Stop if a utility or journal access is unavailable; do not
use an unbounded substitute. These local excerpts are **not sanitised**; follow
[log-sharing precautions](#inspect-notification-logs) before sharing any line.
@@ -1283,22 +1301,42 @@ inside the Pulse container, not on the Proxmox host. Adjust the time window to
the original incident and substitute your actual service or container name
(`pulse-backend` on some older systemd installs). These examples read at most
200 records from the last 15 minutes; they do not follow the log or send a test.
-Each requires **GNU `timeout`** and gives the reader eight seconds plus a
-one-second termination grace. If it is unavailable, stop rather than running
-an unbounded substitute. These are Bash examples; the enclosing subshell keeps
-an unavailable read from exiting your interactive shell.
+Each requires **GNU `timeout`** and the listed local utilities. A **256 KiB**
+byte limit complements the record limit: a single log record can be large.
+The eight-second hard deadline kills only the read-only collection group,
+including the reader and byte limiter, not Pulse or Docker. If a required
+utility is unavailable, stop rather than running an unbounded substitute.
+These are Bash examples; the enclosing subshell keeps an unavailable read from
+exiting your interactive shell.
```bash
# systemd / Proxmox LXC
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm cat; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs"
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
+ fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
fi
+ cat -- "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -1310,14 +1348,31 @@ fi
```bash
# Docker
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm cat; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s docker logs --since 15m --tail 200 pulse 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs"
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+docker logs --since 15m --tail 200 pulse 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
+ fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
fi
+ cat -- "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -1327,10 +1382,14 @@ fi
```
Application logs can be written to stdout or stderr; these commands collect
-both into one local excerpt, without preserving stdout/stderr attribution,
-and display it **only after the reader completes successfully**.
-A failed or timed-out read displays only an unavailable message and its exit
-code, not a partial excerpt that could be mistaken for complete evidence.
+both into one owner-only temporary file, without preserving stdout/stderr
+attribution, and display it **only after the reader completes successfully**
+and the byte limit is checked. The file holds at most 256 KiB plus one byte
+used to detect overflow; it is removed when the subshell exits. A failed,
+oversized or timed-out read shows no partial excerpt that could be mistaken
+for complete evidence. If the shell is forcibly interrupted, an owner-only
+`pulse-log.*` file may remain in its temporary directory; remove that exact
+file privately, not unrelated logs or deployment data.
Do not pipe the reader into `grep email`: it can miss SMTP or webhook errors
and hide a failed read behind a matching partial line. A nonzero reader exit,
access error or missing service/container is a failed read, not “no delivery
@@ -1358,25 +1417,44 @@ Replace `abc123` below with the response's ID. Run only the command for your
deployment, on the Pulse host using an account authorised to read its logs.
These examples limit collection to the last 15 minutes and 1,000 lines; adjust
the time window to the original incident rather than repeating the failed action.
-Like the notification-log examples above, these require **GNU `timeout`**,
-allow eight seconds plus a one-second termination grace, and search **only a
-successfully completed read**. A deadline or failure shows no partial match;
-missing `timeout` does not fall back to an unbounded reader.
+Like the notification-log examples above, these require **GNU `timeout`**
+and the listed local utilities, enforce an eight-second hard deadline and a
+**256 KiB** byte limit, and search **only a successfully completed read**.
+They use an owner-only temporary file with the same cleanup precautions. A
+deadline, overflow or failure shows no partial match; a missing utility does
+not fall back to an unbounded reader. Keep the original response status, time
+and ID if the chosen window is too large; do not raise the limit or repeat the
+failed action to fill that gap.
```bash
# systemd / Proxmox LXC
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm grep; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
REQUEST_ID='abc123'
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse --since '15 minutes ago' --lines 1000 --no-pager 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs" | grep -F -- "$REQUEST_ID"
- else
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+journalctl -u pulse --since '15 minutes ago' --lines 1000 --no-pager 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
exit 1
fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
+ fi
+ grep -aF -- "$REQUEST_ID" "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -1388,17 +1466,32 @@ fi
```bash
# Docker
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm grep; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
+fi
+trap 'rm -- "$pulse_log_file"' EXIT
REQUEST_ID='abc123'
-if pulse_logs=$(timeout --signal=TERM --kill-after=1s 8s docker logs --since 15m --tail 1000 pulse 2>&1); then
- if [ -n "$pulse_logs" ]; then
- printf '%s\n' "$pulse_logs" | grep -F -- "$REQUEST_ID"
- else
+if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+docker logs --since 15m --tail 1000 pulse 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
+ printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
+ fi
+ if [ "$pulse_log_bytes" -gt 262144 ]; then
+ printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
exit 1
fi
+ grep -aF -- "$REQUEST_ID" "$pulse_log_file"
else
pulse_log_status=$?
printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
@@ -1408,9 +1501,11 @@ fi
```
A log-reader failure is not an empty search result: resolve any access or
-container/service error locally first. Exit **124** means the reader exceeded
-the deadline; **137** can mean the termination grace also elapsed. Neither
-establishes that Pulse itself is hung. Do not restart Pulse or Docker, enable
+container/service error locally first. Exit **137** can mean the hard deadline
+killed the collection group; an oversized excerpt is also unavailable. The
+byte limiter can close the pipe before the reader finishes, producing a
+nonzero reader exit such as **141**. These are collection results, not proof
+that Pulse itself is hung. Do not restart Pulse or Docker, enable
Debug, widen collection or repeat the failed action merely to obtain logs.
Even a successful read with no match does not prove the request succeeded. At the default log level, this middleware
logs HTTP 5xx failures but not successful requests; HTTP 4xx failures are logged
diff --git a/frontend-modern/public/docs/TRUENAS.md b/frontend-modern/public/docs/TRUENAS.md
index eceeab1a81..42b718a448 100644
--- a/frontend-modern/public/docs/TRUENAS.md
+++ b/frontend-modern/public/docs/TRUENAS.md
@@ -302,10 +302,12 @@ actual service or container name (`pulse-backend` on some older systemd
installs), and adjust the time window to the original incident.
Those Bash recipes read at most 200 records from the last 15 minutes and
-require **GNU `timeout`**, with an eight-second deadline and one-second
-termination grace. A record limit alone does not bound a hung reader. If
-`timeout` is unavailable, stop; do not use an unbounded substitute. They do
-not follow logs, make API requests or change logging.
+require **GNU `timeout`** and the listed local utilities, with an eight-second
+hard deadline and a **256 KiB** byte limit. A record limit alone does not bound
+a hung reader or a large record. If a required utility is unavailable, stop;
+do not use an unbounded substitute. They do not follow logs, make API requests
+or change logging. The linked reader explains its owner-only temporary file
+and cleanup precautions.
Both output streams are captured because Docker can write application logs
to stderr. An excerpt is shown only after a successful read; failure or
diff --git a/frontend-modern/public/docs/UNIFIED_AGENT.md b/frontend-modern/public/docs/UNIFIED_AGENT.md
index 59eab8aec2..469b68c8b7 100644
--- a/frontend-modern/public/docs/UNIFIED_AGENT.md
+++ b/frontend-modern/public/docs/UNIFIED_AGENT.md
@@ -1007,14 +1007,31 @@ contact Pulse, a container runtime or a guest agent.
```bash
(
-command -v timeout >/dev/null 2>&1 || {
- printf 'Agent log read unavailable: GNU timeout is required; no unbounded fallback.\n' >&2
+umask 077
+for tool in timeout bash head mktemp wc rm cat; do
+ command -v "$tool" >/dev/null 2>&1 || {
+ printf 'Agent log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
+ exit 1
+ }
+done
+if ! agent_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
+ printf 'Agent log read unavailable: private temporary file could not be created.\n' >&2
exit 1
-}
-if agent_logs=$(timeout --signal=TERM --kill-after=1s 8s journalctl -u pulse-agent.service --since '15 minutes ago' --lines 200 --no-pager --output=cat 2>&1); then
- if [ -n "$agent_logs" ]; then
- printf '%s\n' "$agent_logs"
+fi
+trap 'rm -- "$agent_log_file"' EXIT
+if timeout --signal=KILL 8s bash -o pipefail > "$agent_log_file" 2>/dev/null <<'PULSE_LOG_READ'
+journalctl -u pulse-agent.service --since '15 minutes ago' --lines 200 --no-pager --output=cat 2>&1 | head -c 262145
+PULSE_LOG_READ
+then
+ if ! agent_log_bytes=$(wc -c < "$agent_log_file" 2>/dev/null); then
+ printf 'Agent log read unavailable: excerpt size could not be checked.\n' >&2
+ exit 1
fi
+ if [ "$agent_log_bytes" -gt 262144 ]; then
+ printf 'Agent log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
+ exit 1
+ fi
+ cat -- "$agent_log_file"
else
agent_log_status=$?
printf 'Agent log read unavailable (exit %s); no partial excerpt shown.\n' "$agent_log_status" >&2
@@ -1023,15 +1040,23 @@ fi
)
```
-This requires **GNU `timeout`**, reads at most 200 records from the default
-15-minute window, and allows eight seconds plus a one-second termination grace.
-A failed or timed-out read is unavailable, not an empty search; partial output
-is withheld. Stop if the utility, service journal or existing access is
+This requires **GNU `timeout`** and the listed local utilities, reads at most
+200 records from the default 15-minute window, and limits the excerpt to
+**256 KiB**: a record limit alone does not bound bytes. The eight-second hard
+deadline kills only the read-only collection group, including the reader and
+byte limiter, not the agent. A failed, oversized or timed-out read is
+unavailable, not an empty search; partial output is withheld. Stop if a
+utility, service journal or existing access is
unavailable; do not follow the journal indefinitely, remove the deadline or
broaden permissions to collect it. A successful empty read is inconclusive:
the event may be outside the window or absent at the configured log level.
Do not restart, re-enrol or enable debug logging just to obtain an entry.
+Both streams are buffered in an owner-only temporary file, capped at 256 KiB
+plus one overflow-detection byte and removed when the subshell exits. If the
+shell is forcibly interrupted, a private `pulse-log.*` file may remain in its
+temporary directory; remove that exact file locally, not saved agent state.
+
The excerpt is **not sanitised**. Review it locally before searching for the
relevant error or sharing a manually redacted timestamp and failure stage.
Tokens, cookies, URLs, private names or addresses can appear in errors; do not
diff --git a/scripts/tests/test_agent_log_docs.py b/scripts/tests/test_agent_log_docs.py
index f477524fd8..4c4ba33301 100644
--- a/scripts/tests/test_agent_log_docs.py
+++ b/scripts/tests/test_agent_log_docs.py
@@ -51,7 +51,7 @@ class AgentLogDocsTest(unittest.TestCase):
self.assertEqual(result.stderr, "")
self.assertEqual(argv, ["-u", "pulse-agent.service", "--since", "15 minutes ago",
"--lines", "200", "--no-pager", "--output=cat"])
- self.assertIn("timeout --signal=TERM --kill-after=1s 8s", copied)
+ self.assertIn("timeout --signal=KILL 8s", copied)
self.assertIn("2>&1", copied)
self.assertNotRegex(copied, r"\b(?:grep|curl|docker|systemctl|printenv|sudo)\b|--follow|--token")
@@ -73,12 +73,12 @@ class AgentLogDocsTest(unittest.TestCase):
def test_hung_reader_is_stopped_by_the_documented_deadline(self):
result, _ = self.exercise(stdout=LOG, stderr=PRIVATE_ERROR, hang="term")
- self.assertEqual(result.returncode, 124, result.stderr)
+ self.assertEqual(result.returncode, 137, result.stderr)
self.assertEqual(result.stdout, "")
- self.assertIn("Agent log read unavailable (exit 124)", result.stderr)
+ self.assertIn("Agent log read unavailable (exit 137)", result.stderr)
self.assertNotIn(PRIVATE_ERROR.strip(), result.stderr)
- def test_reader_ignoring_term_is_killed_after_the_documented_grace(self):
+ def test_reader_ignoring_term_is_killed_at_the_hard_deadline(self):
result, _ = self.exercise(stdout=LOG, hang="ignore-term")
self.assertEqual(result.returncode, 137, result.stderr)
self.assertEqual(result.stdout, "")
diff --git a/scripts/tests/test_notification_troubleshooting_docs.py b/scripts/tests/test_notification_troubleshooting_docs.py
index edddb95d73..180e17de46 100644
--- a/scripts/tests/test_notification_troubleshooting_docs.py
+++ b/scripts/tests/test_notification_troubleshooting_docs.py
@@ -65,7 +65,7 @@ class NotificationTroubleshootingDocsTest(unittest.TestCase):
self.assertEqual(argv, expected[reader])
self.assertEqual(result.stdout, SMTP_ERROR + WEBHOOK_ERROR)
self.assertNotRegex(copied, r"\b(?:grep|curl|inspect|restart|printenv)\b|--follow|--token")
- self.assertIn("--signal=TERM --kill-after=1s 8s", copied)
+ self.assertIn("--signal=KILL 8s", copied)
def test_docker_application_logs_on_both_streams_remain_visible(self):
result, _ = self.exercise("docker", stdout=WEBHOOK_ERROR, stderr=SMTP_ERROR)
@@ -91,11 +91,11 @@ class NotificationTroubleshootingDocsTest(unittest.TestCase):
for reader in recipes():
with self.subTest(reader=reader):
result, _ = self.exercise(reader, stdout=PARTIAL_EMAIL_LOG, hang="term")
- self.assertEqual(result.returncode, 124, result.stderr)
+ self.assertEqual(result.returncode, 137, result.stderr)
self.assertEqual(result.stdout, "")
self.assertIn("Log read unavailable", result.stderr)
- def test_reader_ignoring_term_is_killed_after_one_second_grace(self):
+ def test_reader_ignoring_term_is_killed_at_the_hard_deadline(self):
result, _ = self.exercise("docker", stdout=PARTIAL_EMAIL_LOG, hang="ignore-term")
self.assertEqual(result.returncode, 137, result.stderr)
self.assertEqual(result.stdout, "")
diff --git a/scripts/tests/test_troubleshooting_logs.py b/scripts/tests/test_troubleshooting_logs.py
index 47b96f4ea1..fbe0c43349 100644
--- a/scripts/tests/test_troubleshooting_logs.py
+++ b/scripts/tests/test_troubleshooting_logs.py
@@ -15,6 +15,7 @@ import signal
import subprocess
import sys
import tempfile
+import time
import unittest
@@ -44,20 +45,28 @@ def recipes(heading: str = "### Correlate Logs with Requests") -> dict[str, str]
READER = f'''#!{sys.executable}
import json, os, pathlib, signal, sys, time
pathlib.Path(os.environ['READER_ARGV']).write_text(json.dumps(sys.argv[1:]))
-sys.stdout.write(os.environ.get('LOG_STDOUT', ''))
-sys.stderr.write(os.environ.get('LOG_STDERR', ''))
+pathlib.Path(os.environ['READER_PID']).write_text(json.dumps({{
+ 'pid': os.getpid(),
+ 'start': pathlib.Path('/proc/self/stat').read_text().split(')', 1)[1].split()[19],
+}}))
+sys.stdout.write(pathlib.Path(os.environ['LOG_STDOUT_FILE']).read_text())
+sys.stderr.write(pathlib.Path(os.environ['LOG_STDERR_FILE']).read_text())
sys.stdout.flush()
sys.stderr.flush()
if os.environ.get('READER_HANG'):
- if os.environ['READER_HANG'] == 'ignore-term':
+ if os.environ['READER_HANG'] in ('ignore-term', 'ignore-term-closed-pipes'):
signal.signal(signal.SIGTERM, signal.SIG_IGN)
+ if os.environ['READER_HANG'] == 'ignore-term-closed-pipes':
+ os.close(1)
+ os.close(2)
time.sleep(30)
sys.exit(int(os.environ.get('READER_EXIT', '0')))
'''
def exercise_log_recipe(reader: str, copied: str, *, stdout: str = "", stderr: str = "",
- exit_code: int = 0, hang: str = "", missing_timeout: bool = False):
+ exit_code: int = 0, hang: str = "", missing_timeout: bool = False,
+ missing_tool: str = "", unavailable_temp: bool = False):
"""Own and clean up only this synthetic shell's process group on failure."""
with tempfile.TemporaryDirectory() as temporary:
directory = Path(temporary)
@@ -65,9 +74,41 @@ def exercise_log_recipe(reader: str, copied: str, *, stdout: str = "", stderr: s
command.write_text(READER, encoding="utf-8")
command.chmod(0o700)
argv = directory / "argv.json"
- path = str(directory) if missing_timeout else f"{directory}:{os.environ['PATH']}"
- env = dict(os.environ, PATH=path, READER_ARGV=str(argv), LOG_STDOUT=stdout,
- LOG_STDERR=stderr, READER_EXIT=str(exit_code), READER_HANG=hang)
+ pid_file = directory / "reader.pid"
+ stdout_file, stderr_file = directory / "stdout.txt", directory / "stderr.txt"
+ # Files keep the large-record controls out of exec's argument/environment
+ # size limit. These are synthetic bytes, never a real journal or token.
+ stdout_file.write_text(stdout, encoding="utf-8")
+ stderr_file.write_text(stderr, encoding="utf-8")
+ log_directory = directory / "log-temporary"
+ log_directory.mkdir(mode=0o700)
+ cleanup_file = directory / "cleanup.jsonl"
+ real_rm = shutil.which("rm")
+ (directory / "rm").write_text(f'''#!{sys.executable}
+import json, os, pathlib, stat, sys
+path = pathlib.Path(sys.argv[2])
+assert sys.argv[1] == '--' and len(sys.argv) == 3
+assert path.parent == pathlib.Path(os.environ['LOG_TEMPORARY'])
+assert path.name.startswith('pulse-log.')
+info = path.stat()
+with pathlib.Path(os.environ['LOG_CLEANUP']).open('a') as receipt:
+ receipt.write(json.dumps({{'mode': stat.S_IMODE(info.st_mode), 'bytes': info.st_size}}) + '\\n')
+os.execv({real_rm!r}, [{real_rm!r}] + sys.argv[1:])
+''', encoding="utf-8")
+ (directory / "rm").chmod(0o700)
+ if missing_tool:
+ for tool in ("bash", "timeout", "head", "mktemp", "wc", "cat", "grep"):
+ if tool != missing_tool:
+ (directory / tool).symlink_to(shutil.which(tool))
+ if missing_tool == "rm":
+ (directory / "rm").unlink()
+ path = (str(directory) if missing_timeout or missing_tool
+ else f"{directory}:{os.environ['PATH']}")
+ env = dict(os.environ, PATH=path, READER_ARGV=str(argv), READER_PID=str(pid_file),
+ LOG_STDOUT_FILE=str(stdout_file), LOG_STDERR_FILE=str(stderr_file),
+ READER_EXIT=str(exit_code), READER_HANG=hang,
+ LOG_TEMPORARY=str(log_directory), LOG_CLEANUP=str(cleanup_file),
+ TMPDIR=str(log_directory / "absent" if unavailable_temp else log_directory))
with subprocess.Popen([shutil.which("bash"), "-c", copied], env=env,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True, start_new_session=True) as process:
@@ -81,6 +122,37 @@ def exercise_log_recipe(reader: str, copied: str, *, stdout: str = "", stderr: s
process.communicate(timeout=5)
raise
result = subprocess.CompletedProcess(process.args, process.returncode, output, errors)
+ result.log_cleanup = ([json.loads(line) for line in cleanup_file.read_text().splitlines()]
+ if cleanup_file.exists() else [])
+ if list(log_directory.iterdir()):
+ raise AssertionError("copied recipe left a private log file behind")
+ # The reader is a child of timeout/the collection shell, not ours to
+ # waitpid. A zombie is terminated, not still collecting. Observe only
+ # the recorded fixture PID; never search for or stop a real service.
+ result.reader_running = False
+ if pid_file.exists():
+ identity = json.loads(pid_file.read_text())
+ try:
+ pid_fd = os.pidfd_open(identity["pid"])
+ except ProcessLookupError:
+ pid_fd = None
+ if pid_fd is not None:
+ try:
+ for _ in range(50):
+ try:
+ fields = Path(f"/proc/{identity['pid']}/stat").read_text().split(")", 1)[1].split()
+ except FileNotFoundError:
+ break
+ if fields[19] != identity["start"] or fields[0] == "Z":
+ break
+ time.sleep(0.01)
+ else:
+ # The pidfd cannot signal a recycled PID. Clean up
+ # only the captured synthetic child, then fail proof.
+ signal.pidfd_send_signal(pid_fd, signal.SIGKILL)
+ result.reader_running = True
+ finally:
+ os.close(pid_fd)
return result, json.loads(argv.read_text()) if argv.exists() else None
@@ -140,7 +212,7 @@ class TroubleshootingLogRecipesTest(unittest.TestCase):
for reader in recipes():
with self.subTest(reader=reader):
result = self.exercise(reader, stdout=JSON_LOG, hang="term")
- self.assertEqual(result.returncode, 124, result.stderr)
+ self.assertEqual(result.returncode, 137, result.stderr)
self.assertEqual(result.stdout, "")
self.assertIn("Log read unavailable", result.stderr)
@@ -154,7 +226,7 @@ class TroubleshootingLogRecipesTest(unittest.TestCase):
def test_console_only_selector_is_a_negative_control_for_json(self):
copied = recipes()["journalctl"].replace(
- 'grep -F -- "$REQUEST_ID"', 'grep -F -- "request_id=$REQUEST_ID"'
+ 'grep -aF -- "$REQUEST_ID"', 'grep -aF -- "request_id=$REQUEST_ID"'
)
result = self.exercise("journalctl", stdout=JSON_LOG, recipe=copied)
self.assertEqual(result.returncode, 1)
@@ -222,9 +294,9 @@ class GeneralLogEvidenceTest(unittest.TestCase):
for reader in recipes("### Inspect Notification Logs"):
with self.subTest(reader=reader):
result = self.exercise(reader, stdout="partial error\n", hang="term")
- self.assertEqual(result.returncode, 124, result.stderr)
+ self.assertEqual(result.returncode, 137, result.stderr)
self.assertEqual(result.stdout, "")
- self.assertIn("Log read unavailable (exit 124)", result.stderr)
+ self.assertIn("Log read unavailable (exit 137)", result.stderr)
def test_general_readers_do_not_fall_back_when_timeout_is_missing(self):
for reader in recipes("### Inspect Notification Logs"):
@@ -256,7 +328,7 @@ class SeverityLogEvidenceTest(unittest.TestCase):
def test_priority_reader_keeps_incident_bounds_without_privilege_or_mutation(self):
copied = self.copied()
- for bound in ("--signal=TERM --kill-after=1s 8s", "--since '15 minutes ago'",
+ for bound in ("--signal=KILL 8s", "--since '15 minutes ago'",
"--lines 200", "--no-pager"):
self.assertIn(bound, copied)
self.assertNotRegex(copied, r"\b(?:sudo|systemctl|curl|grep)\b|--follow")
@@ -279,9 +351,9 @@ class SeverityLogEvidenceTest(unittest.TestCase):
def test_real_deadline_withholds_partial_warning(self):
result = self.exercise(stdout=JSON_LOG, hang="term")
- self.assertEqual(result.returncode, 124, result.stderr)
+ self.assertEqual(result.returncode, 137, result.stderr)
self.assertEqual(result.stdout, "")
- self.assertIn("Log read unavailable (exit 124)", result.stderr)
+ self.assertIn("Log read unavailable (exit 137)", result.stderr)
def test_missing_timeout_stops_before_reading(self):
result = self.exercise(missing_timeout=True)
@@ -306,5 +378,103 @@ class SeverityLogEvidenceTest(unittest.TestCase):
self.assertNotIn("systemctl cat", guide)
+class LogByteBoundsTest(unittest.TestCase):
+ """Read the actual server and agent entry points, not a duplicate helper."""
+
+ def copied_recipes(self):
+ copied = []
+ for heading in ("### Inspect Notification Logs", "### Correlate Logs with Requests"):
+ copied.extend((heading, reader, recipe)
+ for reader, recipe in recipes(heading).items())
+ copied.append(("severity", "journalctl", SeverityLogEvidenceTest().copied()))
+ agent_doc = (ROOT / "docs/UNIFIED_AGENT.md").read_text()
+ agent_section = agent_doc.split("### Collect agent logs safely\n", 1)[1].split("\n### ", 1)[0]
+ agent_blocks = re.findall(r"```bash\n(.*?)```", agent_section, re.DOTALL)
+ self.assertEqual(len(agent_blocks), 1)
+ copied.append(("agent", "journalctl", agent_blocks[0]))
+ self.assertEqual(len(copied), 6)
+ return copied
+
+ def exercise(self, reader, recipe, **settings):
+ result, argv = exercise_log_recipe(reader, recipe, **settings)
+ self.assertFalse(result.reader_running, "reader survived the copied collection shell")
+ for cleanup in result.log_cleanup:
+ self.assertEqual(cleanup["mode"], 0o600)
+ self.assertLessEqual(cleanup["bytes"], 262145)
+ return result, argv
+
+ def test_exact_byte_limit_preserves_a_complete_record_and_private_file_cleanup(self):
+ log = "abc123 " + "x" * (262144 - 8) + "\n"
+ self.assertEqual(len(log.encode()), 262144)
+ for heading, reader, recipe in self.copied_recipes():
+ with self.subTest(section=heading, reader=reader):
+ result, _ = self.exercise(reader, recipe, stdout=log)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(result.stdout, log)
+ self.assertEqual(result.stderr, "")
+ self.assertEqual(result.log_cleanup, [{"mode": 0o600, "bytes": 262144}])
+
+ def test_single_large_record_and_large_stderr_are_unavailable_not_a_partial_match(self):
+ for heading, reader, recipe in self.copied_recipes():
+ for stream in ("stdout", "stderr"):
+ with self.subTest(section=heading, reader=reader, stream=stream):
+ result, _ = self.exercise(reader, recipe, **{stream: "abc123 " + "x" * 1048576})
+ self.assertNotEqual(result.returncode, 0)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("unavailable", result.stderr)
+ self.assertNotIn("abc123", result.stderr)
+ self.assertEqual(result.log_cleanup, [{"mode": 0o600, "bytes": 262145}])
+
+ def test_limit_counts_actual_bytes_including_unicode_nuls_and_trailing_newlines(self):
+ for heading, reader, recipe in self.copied_recipes():
+ for log in ("abc123" + "\n" * 262139, "é" * 131073, "\0" * 262145):
+ with self.subTest(section=heading, reader=reader, first_byte=repr(log[0])):
+ result, _ = self.exercise(reader, recipe, stdout=log)
+ self.assertNotEqual(result.returncode, 0)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("unavailable", result.stderr)
+ self.assertEqual(result.log_cleanup, [{"mode": 0o600, "bytes": 262145}])
+
+ def test_failed_reader_still_has_no_excerpt_and_removes_its_private_buffer(self):
+ for heading, reader, recipe in self.copied_recipes():
+ with self.subTest(section=heading, reader=reader):
+ result, _ = self.exercise(reader, recipe, stdout="abc123 partial\n", exit_code=2)
+ self.assertEqual(result.returncode, 2)
+ self.assertEqual(result.stdout, "")
+ self.assertEqual(len(result.log_cleanup), 1)
+
+ def test_missing_limiter_or_private_temp_path_stops_before_the_reader(self):
+ for heading, reader, recipe in self.copied_recipes():
+ for settings in ({"missing_tool": "head"}, {"unavailable_temp": True}):
+ with self.subTest(section=heading, reader=reader, settings=settings):
+ result, argv = self.exercise(reader, recipe, **settings)
+ self.assertIsNone(argv)
+ self.assertEqual(result.returncode, 1)
+ self.assertEqual(result.stdout, "")
+ self.assertIn("unavailable", result.stderr)
+ self.assertEqual(result.log_cleanup, [])
+
+ def test_hard_deadline_terminates_closed_pipe_term_resistant_readers_and_cleans_buffer(self):
+ selected = [("docker", recipes("### Inspect Notification Logs")["docker"]),
+ ("journalctl", self.copied_recipes()[-1][2])]
+ for reader, recipe in selected:
+ with self.subTest(reader=reader):
+ result, _ = self.exercise(reader, recipe, stdout="partial private log\n",
+ hang="ignore-term-closed-pipes")
+ self.assertEqual(result.returncode, 137, result.stderr)
+ self.assertEqual(result.stdout, "")
+ self.assertNotIn("partial private log", result.stderr)
+ self.assertEqual(len(result.log_cleanup), 1)
+
+ def test_each_entry_keeps_collection_deadline_pipe_failures_and_byte_limiter_together(self):
+ for heading, reader, recipe in self.copied_recipes():
+ with self.subTest(section=heading, reader=reader):
+ subprocess.run(["bash", "-n", "-c", recipe], check=True, capture_output=True)
+ for bound in ("umask 077", "timeout --signal=KILL 8s bash -o pipefail",
+ "2>&1 | head -c 262145", '" -gt 262144 ]', "trap 'rm --"):
+ self.assertIn(bound, recipe)
+ self.assertNotIn("--foreground", recipe)
+
+
if __name__ == "__main__":
unittest.main()
diff --git a/scripts/tests/test_truenas_docs.py b/scripts/tests/test_truenas_docs.py
index 34ea5ade28..2b36f69b77 100644
--- a/scripts/tests/test_truenas_docs.py
+++ b/scripts/tests/test_truenas_docs.py
@@ -159,7 +159,7 @@ class TrueNASDocsTest(unittest.TestCase):
self.assertCountEqual(result.stdout.splitlines(), ["startup failed", "TrueNAS collection failed"])
self.assertEqual(result.stderr, "")
self.assertNotRegex(recipe, r"\b(?:grep|curl|inspect|restart|printenv)\b|--follow")
- self.assertIn("--signal=TERM --kill-after=1s 8s", recipe)
+ self.assertIn("--signal=KILL 8s", recipe)
def test_log_reader_failure_is_preserved_even_with_a_matching_partial_line(self):
for reader, recipe in self.log_recipes().items():
@@ -176,11 +176,11 @@ class TrueNASDocsTest(unittest.TestCase):
for reader, recipe in self.log_recipes().items():
with self.subTest(reader=reader):
result, _ = self.exercise_log(reader, recipe, stdout="TrueNAS polling\n", hang="term")
- self.assertEqual(result.returncode, 124, result.stderr)
+ self.assertEqual(result.returncode, 137, result.stderr)
self.assertEqual(result.stdout, "")
- self.assertIn("Log read unavailable (exit 124)", result.stderr)
+ self.assertIn("Log read unavailable (exit 137)", result.stderr)
- def test_reader_ignoring_term_is_killed_after_the_grace_period(self):
+ def test_reader_ignoring_term_is_killed_at_the_hard_deadline(self):
result, _ = self.exercise_log("docker", self.log_recipes()["docker"],
stdout="TrueNAS polling\n", hang="ignore-term")
self.assertEqual(result.returncode, 137, result.stderr)
@@ -224,7 +224,7 @@ class TrueNASDocsTest(unittest.TestCase):
"not sanitised", "not the whole excerpt", "manually redacted",
"anything echoed in an error", "enable Debug",
"record limit alone does not bound a hung reader", "GNU `timeout`",
- "eight-second deadline and one-second termination grace",
+ "eight-second hard deadline",
"no request ID is required", "only after a successful read",
"withholds partial output", "do not use an unbounded substitute"):
self.assertIn(boundary, prose)