Skip to content

Free, Open Source, Self-Hosted WhatsApp API Gateway

fix(scripts): only reject the env line that sets the key (#1823)

Yudhi Armyndharisc05f6adf6bb9
diff --git a/docs/11-operational-runbooks.md b/docs/11-operational-runbooks.md
index cb61e8bb4..5960a4b2a 100644
--- a/docs/11-operational-runbooks.md
+++ b/docs/11-operational-runbooks.md
@@ -786,31 +786,53 @@ OPENWA_DATA_DIR=/srv/openwa/data \
> the container, where it names a path in the volume, while the scripts read it against the current
> directory.
>
-> On every install, inside a container or not, a quoted value followed by a `#` comment or not
-> closed on its line, a quoted value containing its own quote character or ending in a backslash, a
-> double-quoted value with a backslash, or a `KEY: value` line in `./.env` or `.env.generated` stops
-> the script with an error naming the key, before anything is archived or restored. So does a bare
-> CR, U+2028 or U+2029 on any line naming the key, a comment included, since the app starts a new
-> line there; a NUL or a byte that is not UTF-8 on a line setting the key; a byte-order mark or a
-> Unicode blank (such as a no-break space) before the key, around its `=` or at either end of its
-> value; a bare `NAME:` line right before the key's line, which the app reads as `NAME`'s value;
-> and a key whose value the app takes from a later line, either a bare `KEY` line followed, past
-> any blank lines, by one starting with `=` or an empty `KEY=` followed by a quoted value. The app
-> reads such a line, so neither the script default nor a guess is safe to use. `DATABASE_NAME` is
-> not read by a PostgreSQL backup through `DATABASE_URL` or by the restore of a PostgreSQL archive,
-> and an unreadable `ENGINE_TYPE` only skips the warning about missing Baileys state.
+> On every install, inside a container or not, the scripts stop with an error naming the key, before
+> anything is archived or restored, when the last line setting it in `./.env` or `.env.generated`,
+> which is the one the app keeps, has one of these forms:
+>
+> - a quoted value followed by a `#` comment or not closed on its line
+> - a quoted value containing its own quote character, or ending in a backslash the app can read
+> past to a later quote of that kind followed only by blanks or a comment
+> - a double-quoted value with a backslash
+> - a `KEY: value` or `KEY:value` line (Docker Compose reads both from `./.env` into the container)
+> - a NUL or a byte that is not UTF-8
+> - a byte-order mark or a Unicode blank (such as a no-break space) before the key, around its `=`,
+> at either end of an unquoted value or after a closing quote
+> - a bare `NAME:` line right before the key's line, which the app reads as `NAME`'s value
+> - a bare `KEY` line followed, past any blank lines, by one starting with `=`, or an empty `KEY=`
+> followed by a quoted value, which the app reads as the key's value
+> - a quoted value opened on an earlier line, the key's own or another key's, that the app can read
+> on to the key's line, which then belongs to that value
+>
+> A bare CR, U+2028 or U+2029 on any line naming the key, a comment included, stops the run too,
+> since the app can start a setting right after one. Like the app, the scripts break lines at a bare
+> CR, and at a U+2028 or U+2029 outside an unquoted value. The app reads each of these lines
+> (`KEY:value` only through Docker Compose, from `./.env`), so neither the script default nor a
+> guess is safe to use. `DATABASE_NAME` is not read by a PostgreSQL backup through `DATABASE_URL` or
+> by the restore of a PostgreSQL archive, and an unreadable `ENGINE_TYPE` only skips the warning
+> about missing Baileys state.
+>
> Move the comment to its own line and keep the quotes (an unquoted value ends at a `#` and loses
> its outer blanks), wrap the value in a quote character it does not contain, single-quote a value
> whose backslashes are literal and not at its end (inside double quotes the app turns `\n` and `\r`
> into line breaks; if that is intended, pass the key in the environment), write `KEY=value` on one
-> line for `KEY: value` and for a value on a later line, give a bare `NAME:` line a value or remove
-> it, save the file as UTF-8 with ASCII blanks and LF or CRLF line endings, or pass the key in the
-> environment. A restore reads the archive's `.env.generated`, which cannot be edited in place, so a
-> key it holds in such a form has to be passed in the environment. Blanks around `=`, CRLF line
-> endings, a value in one pair of quotes and a `#` comment after an unquoted value are read as the
-> app reads them. The scripts read the file line by line, so a line for the key inside a quoted
-> value that spans several lines, another key's or an earlier one for the same key, is taken as a
-> setting, although the app reads it as part of that value.
+> line for `KEY: value`, `KEY:value` and a value on a later line, close a quote left open on an
+> earlier line, give a bare `NAME:` line a value or remove it, save the file as UTF-8 with ASCII
+> blanks and LF or CRLF line endings, or pass the key in the environment. A restore reads the
+> archive's `.env.generated`, which cannot be edited in place, so a key it holds in such a form has
+> to be passed in the environment, and so does a `DATABASE_*` value saved from
+> Dashboard > Infrastructure in one of these forms, since the next save writes it the same way. The
+> scripts need `tr`, `tail`, `sed` and `grep` to read either file and stop when one is missing.
+> Blanks around `=`, CRLF line endings, a value in one pair of quotes and a `#` comment after an
+> unquoted value are read as the app reads them, and so is a quoted value that spans several lines
+> (a PEM key, say) and closes before the key's line, even when it holds quotes of another kind.
+>
+> Under Docker Compose the container gets `./.env` through Compose, which reads some values
+> differently from the app's own reader: it keeps backticks and a `#` with no blank before it as part
+> of the value, and expands `$NAME` outside single quotes. Run inside the container, the scripts see
+> the value Compose passed in; run on the host, they read `./.env` as the app reads a `.env` file, so
+> a value of that kind can resolve differently there. Single-quote such a value in `./.env`, or pass
+> the key in the environment on the host.
**Verification:**
diff --git a/scripts/backup.sh b/scripts/backup.sh
index 7c7662b5a..6ea401df5 100755
--- a/scripts/backup.sh
+++ b/scripts/backup.sh
@@ -297,8 +297,11 @@ if [ -d "$BAILEYS_DIR" ]; then
if [ -n "$(find -H "$BAILEYS_DIR" -mindepth 1 -maxdepth 1 -type d -print -quit)" ]; then
record_engine_state_note "baileys/ (recorded whenever Baileys state exists; it cannot show whether it was live)"
fi
-# The warning is advisory, so an ENGINE_TYPE line the scripts cannot parse skips it instead of the backup.
-elif ! ENGINE_TYPE_RESOLVED="$(openwa_resolve ENGINE_TYPE '')"; then
+# The warning is advisory, so an ENGINE_TYPE line the scripts cannot parse skips it instead of the backup,
+# and its error is reported as a warning. A failed lookup prints nothing on stdout and a successful one
+# nothing on stderr, so both share the substitution.
+elif ! ENGINE_TYPE_RESOLVED="$(openwa_resolve ENGINE_TYPE '' 2>&1)"; then
+ sed -e 's/^\[config\] ERROR: /[config] WARN: /' -e 's/^\[config\] /[config] /' <<<"$ENGINE_TYPE_RESOLVED" >&2
log "WARN: ENGINE_TYPE could not be read (see above); skipping only the check for missing Baileys state"
elif [ "$ENGINE_TYPE_RESOLVED" = "baileys" ]; then
log "WARN: ENGINE_TYPE=baileys but $BAILEYS_DIR was not found — restored sessions will require pairing"
diff --git a/scripts/lib-env.sh b/scripts/lib-env.sh
index 829f3e491..4b23d3fd0 100644
--- a/scripts/lib-env.sh
+++ b/scripts/lib-env.sh
@@ -17,77 +17,278 @@
#
# Deliberately conservative: only the `KEY=value` forms dotenv reads plainly are honoured. Blanks
# around the `=` and the value, CRLF line endings, a value wrapped in one pair of quotes and a comment
-# after an unquoted value are read as dotenv reads them. Anything else (a quoted value followed by a
-# comment, containing its own quote character or ending in a backslash, a double-quoted value with
-# escapes, a `KEY: value` line, a bare CR, U+2028 or U+2029 on a line naming the key, a NUL or byte
-# that is not UTF-8 on its line, a byte-order mark or Unicode blank next to the key, its `=` or the
-# ends of its value, a bare `NAME:` line just before it, a bare `KEY` or an empty `KEY=` the app
-# completes from a later line) stops the run rather than being guessed at: the app still reads such a
-# line, so neither a guess nor the default is safe to back up or restore to, and a silently
-# mis-resolved path is the exact failure this exists to prevent. The file is still read line by line:
-# a line for the key inside a quoted value that spans lines, whether another key's or an earlier one
-# of the same key, is taken as a setting, although the app reads it as part of that value. Nothing
-# here exports anything: each key is looked up by name, so a stray entry in an operator's .env can
-# never reach the script's own environment.
+# after an unquoted value are read as dotenv reads them. As in dotenv, the last line setting the key
+# wins, so only that line is checked. Anything else (a quoted value followed by a comment, containing
+# its own quote character or ending in a backslash the app can read on past to a later quote, a
+# double-quoted value with escapes, a `KEY: value` or `KEY:value` line, a bare CR, U+2028 or U+2029 on
+# any line naming the key, a NUL or byte that is not UTF-8 on its line, a byte-order mark or Unicode
+# blank next to the key, its `=`, the ends of an unquoted value or a closing quote, a bare `NAME:` line
+# just before it, a bare `KEY` or an empty `KEY=` the app completes from a later line) stops the run
+# rather than being guessed at: the app still reads such a line, so neither a guess nor the default is
+# safe to back up or restore to, and a silently mis-resolved path is the exact failure this exists to
+# prevent. So does a quoted value opened on an earlier line, the key's own or another key's, that the
+# app can read on to the key's line, where lines break as the app breaks them: at a bare CR, and at
+# U+2028 or U+2029 outside an unquoted value. Nothing here exports anything: each key is looked up by
+# name, so a stray entry in an operator's .env can never reach the script's own environment.
+
+# openwa_env_lines - copy stdin with a line break wherever dotenv starts a line: at a bare CR, and at
+# U+2028 or U+2029 unless an unquoted value without a `#` runs on past it. Each added line starts with
+# the CR or a U+2028, which the patterns read as a blank, and CRLF line ends lose their CR. It also
+# splits inside a quoted value, and after a stray quote in an unquoted one; openwa_env_opener skips
+# the lines that adds inside a value.
+openwa_env_lines() (
+ export LC_ALL=C
+ local line seg a b cr=$'\r' ls=$'\xe2\x80\xa8'
+ local setting='^(export[[:space:]]+)?[A-Za-z0-9_.-]+([[:space:]]*=|:$|:[[:space:]])(.*)$'
+ sed -e $'s/\r$//' -e $'s/\xe2\x80\xa9/\xe2\x80\xa8/g' | while IFS= read -r line || [ -n "$line" ]; do
+ seg=""
+ while :; do
+ a="${line%%"$cr"*}"
+ b="${line%%"$ls"*}"
+ [ "${#a}" -lt "${#line}" ] || [ "${#b}" -lt "${#line}" ] || break
+ if [ "${#a}" -lt "${#b}" ]; then
+ printf '%s\n' "$seg$a"
+ seg="$cr"
+ line="${line:${#a}+1}"
+ continue
+ fi
+ seg="$seg$b"
+ line="${line:${#b}+3}"
+ a="${seg#"$cr"}"
+ a="${a#"$ls"}"
+ a="${a#"${a%%[![:space:]]*}"}"
+ if [[ $a =~ $setting ]] && [[ ${BASH_REMATCH[3]} != *[#\'\"\`]* ]]; then
+ seg="$seg$ls"
+ continue
+ fi
+ printf '%s\n' "$seg"
+ seg="$ls"
+ done
+ printf '%s\n' "$seg$line"
+ done
+)
+
+# openwa_env_opener <s> <closes> - read the lines before the key's line, split by openwa_env_lines, and
+# print the number of the one that opens a quoted value dotenv reads on to the key's line. <s> matches
+# one blank; <closes> lists the quote characters that a quote on or after the key's line, up to the
+# first such quote without a backslash before it, can close a value with: one only blanks or a comment
+# follow. The lines are read in order, as dotenv reads them, so a line inside a value opens nothing.
+# A quote opens a value after `NAME=` or `NAME: `, after a bare `NAME` and an `=` on the next line that
+# is not blank, or at the start of the line after a bare `NAME:` or of a line after an empty `NAME=` or
+# `NAME:` or a bare `NAME` and a lone `=`, past blank lines. The value ends at the last quote that only
+# blanks or a comment follow, up to the first quote without a backslash; with none, dotenv reads the
+# line after `NAME=`, `NAME: ` or a bare `NAME:` as an unquoted value, which runs on past a U+2028 or
+# U+2029 until a `#`, and any other line as no setting at all.
+openwa_env_opener() (
+ local LC_ALL=C
+ local s="$1" closes="$2" l v r q c i=0 j n e u kind last prior open closer ls=$'\xe2\x80\xa8'
+ local set="^$s*(export$s+)?[A-Za-z0-9_.-]+($s*=|:$s)" colon="^$s*(export$s+)?[A-Za-z0-9_.-]+:\$"
+ local bare="^$s*(export$s+)?[A-Za-z0-9_.-]+$s*\$" empty="^$s*(export$s+)?[A-Za-z0-9_.-]+($s*=|:)$s*\$"
+ local blankline="^$s*\$" eq="^$s*=" lone="^$s*=$s*\$" quote="^$s*[=\"'\`]"
+ # after[i]: line i is the value of a bare `NAME:` line before it. free[i]: dotenv starts reading at
+ # line i, which is not inside a value.
+ local -a lines after free
+ while IFS= read -r l; do
+ i=$((i + 1))
+ lines[i]="$l"
+ done
+ n=$i
+ i=1
+ while [ "$i" -le "$n" ]; do
+ l="${lines[i]}"
+ free[i]=1
+ # A line without a quote opens nothing; only a bare `NAME:` line, the value of one, or a value that
+ # runs on past a U+2028 needs the checks below.
+ case "$l" in
+ *[\"\'\`]* | *:) ;;
+ *)
+ if [ -z "${after[i]:-}" ] && [[ ${lines[i + 1]:-} != "$ls"* ]]; then
+ i=$((i + 1))
+ continue
+ fi
+ ;;
+ esac
+ kind=""
+ v="$l"
+ if [ -n "${after[i]:-}" ]; then
+ kind=value
+ elif [[ $l =~ $set ]]; then
+ kind=value
+ v="${l#*[=:]}"
+ elif [[ $l =~ $colon ]]; then
+ after[i + 1]=1
+ elif [[ $l =~ $quote ]]; then
+ # The two lines before it that are not blank, which have to start its value.
+ j=$i last="" prior=""
+ while [ "$j" -gt 1 ]; do
+ j=$((j - 1))
+ [[ ! ${lines[j]} =~ $blankline ]] || continue
+ [ -z "$last" ] || { prior=$j && break; }
+ last=$j
+ done
+ if [ -z "$last" ] || [ -z "${free[last]:-}" ] || [ -n "${after[last]:-}" ]; then
+ :
+ elif [[ $l =~ $eq ]]; then
+ if [[ ${lines[last]} =~ $bare ]]; then
+ kind=value
+ v="${l#*=}"
+ fi
+ elif [[ ${lines[last]} =~ $lone ]]; then
+ if [ -n "$prior" ] && [ -n "${free[prior]:-}" ] && [ -z "${after[prior]:-}" ] && [[ ${lines[prior]} =~ $bare ]]; then
+ kind=quoted
+ fi
+ elif [[ ${lines[last]} =~ $empty ]]; then
+ kind=quoted
+ fi
+ fi
+ if [ -z "$kind" ]; then
+ i=$((i + 1))
+ continue
+ fi
+ q=""
+ for c in "'" '"' '`'; do
+ open="^$s*$c"
+ if [[ $v =~ $open ]]; then q="$c" && break; fi
+ done
+ if [ -n "$q" ]; then
+ open="(^|[^\\\\])$q"
+ closer="^([^$q]|\\\\$q)*$q$s*(#.*)?\$"
+ r="${v#*"$q"}"
+ e=""
+ if [[ $r =~ $open ]]; then
+ [[ ! $r =~ $closer ]] || e=$i
+ else
+ j=$i u=""
+ while [ "$j" -lt "$n" ]; do
+ j=$((j + 1))
+ [[ ! ${lines[j]} =~ $closer ]] || e=$j
+ if [[ ${lines[j]} =~ $open ]]; then u=$j && break; fi
+ done
+ if [ -z "$u" ] && [[ $closes == *"$q"* ]]; then
+ printf '%s' "$i"
+ return 0
+ fi
+ [ -n "$e" ] || [[ ! $r =~ $closer ]] || e=$i
+ fi
+ if [ -n "$e" ]; then
+ i=$((e + 1))
+ continue
+ fi
+ if [ "$kind" = quoted ]; then
+ i=$((i + 1))
+ continue
+ fi
+ fi
+ # An unquoted value, which runs on past a U+2028 or U+2029 up to a `#`.
+ i=$((i + 1))
+ while [ "$i" -le "$n" ] && [[ $v != *'#'* ]] && [[ ${lines[i]} == "$ls"* ]]; do
+ v="${lines[i]}"
+ i=$((i + 1))
+ done
+ done
+)
# openwa_env_file_value <file> <key> - print the value from one env-file layer. Returns 1 when the layer
# does not set the key and 2 when it may set it in a form rejected below; a blank value succeeds and
# prints nothing.
openwa_env_file_value() {
- local file="$1" key="$2" line match value n next
- local keyline="^[[:space:]]*(export[[:space:]]+)?${key}[[:space:]]*([=:]|\$)"
+ local file="$1" key="$2" line match value n next prev rest tool q o closes bad=""
+ # The trims below take only ASCII blanks whatever the operator's locale, so a Unicode blank after a
+ # closing quote stops the run on every host.
+ local LC_ALL=C
# The code points beyond ASCII that dotenv's \s matches: the Unicode blanks and the byte-order mark.
local blank=$'\xc2\xa0|\xe1\x9a\x80|\xe2\x80[\x80-\x8a\xa8\xa9\xaf]|\xe2\x81\x9f|\xe3\x80\x80|\xef\xbb\xbf'
+ local s="([[:space:]]|$blank)"
+ # A line dotenv or Docker Compose may read as setting the key: `KEY=`, `KEY:` or a bare `KEY`.
+ # dotenv skips `KEY:value`, but Compose interpolates ./.env into the container and reads it as a
+ # setting, so that line stops the run too.
+ local keyline="^$s*(export$s+)?${key}$s*([=:]|\$)"
# A line of well-formed UTF-8, without NUL.
local utf8=$'([\x01-\x7f]|[\xc2-\xdf][\x80-\xbf]|\xe0[\xa0-\xbf][\x80-\xbf]|[\xe1-\xec\xee\xef][\x80-\xbf]{2}|\xed[\x80-\x9f][\x80-\xbf]|\xf0[\x90-\xbf][\x80-\xbf]{2}|[\xf1-\xf3][\x80-\xbf]{3}|\xf4[\x80-\x8f][\x80-\xbf]{2})*'
[ -f "$file" ] || return 1
- # dotenv also takes a byte-order mark or a Unicode blank (a no-break space, say) for whitespace and a
- # bare CR, U+2028 or U+2029 for a line break, and decodes the file as UTF-8, so a byte that is not UTF-8
- # reaches the app as U+FFFD. grep and the trims below do none of that, so the key could be missed or
- # read with other bytes. Such a line break on any line naming the key, comments included, is rejected,
- # and so is a line setting the key with a blank before the key, around its `=` or at either end of an
- # unquoted value, or with a NUL or a byte that is not UTF-8 anywhere. Every grep here runs with -a in
- # the C locale: otherwise a NUL anywhere in the file, or a byte that is not UTF-8 in a UTF-8 locale, can
- # hide the key's line. A NUL, which bash would drop from the line, becomes \377.
- line="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C grep -aE "(^|[^A-Za-z0-9_.-])${key}([^A-Za-z0-9_.-]|\$)")" || true
- if ! LC_ALL=C grep -qE $'\r.|\xe2\x80[\xa8\xa9]' <<<"$line"; then
- line="$(LC_ALL=C grep -aE "^([[:space:]]|$blank)*(export([[:space:]]|$blank)+)?${key}([[:space:]]|$blank)*([=:]|\$)" <<<"$line")" || true
- [ -z "$line" ] || LC_ALL=C grep -qvE "$keyline" <<<"$line" ||
- LC_ALL=C grep -qE "^[^=:]*[=:][[:space:]]*($blank)|^[^#]*($blank)[[:space:]]*(#|\$)" <<<"$line" ||
- LC_ALL=C grep -qvxE "$utf8" <<<"$line" || line=""
- fi
- if [ -n "$line" ]; then
- echo "[config] ERROR: $file has a bare CR, U+2028 or U+2029 on a line naming $key, a NUL or a byte that is" >&2
- echo "[config] not UTF-8 on a line setting it, or a byte-order mark or Unicode blank next to $key, its =" >&2
- echo "[config] or the ends of its value. Save it as UTF-8 with plain ASCII blanks and LF or CRLF line" >&2
- echo "[config] ends, or pass $key in the environment." >&2
- return 2
- fi
- # dotenv reads the line after a bare `NAME:` line as NAME's value, so a line for the key right there
- # does not set the key for the app, although the scripts would read it. dotenv also starts a line
- # after a bare CR, U+2028 or U+2029, so `# note<U+2028>NAME:` is such a line too.
- line="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C grep -aB1 -E "$keyline")" || true
- if LC_ALL=C grep -qE "(^|"$'\r|\xe2\x80[\xa8\xa9]'")([[:space:]]|$blank)*(export([[:space:]]|$blank)+)?[A-Za-z0-9_.-]+:"$'\r'"?\$" <<<"$line"; then
- echo "[config] ERROR: $file has a bare NAME: line just before a line setting $key, which the app reads as" >&2
- echo "[config] NAME's value instead. Give NAME: a value or remove it, or pass $key in the environment." >&2
+ # Without these the key's line would read as absent and the lookup would fall through to a default.
+ for tool in tr tail sed grep; do
+ command -v "$tool" >/dev/null 2>&1 && continue
+ echo "[config] ERROR: $tool is required to read $file; install it or pass $key in the environment." >&2
return 2
- fi
- # The last line naming the key wins, as in dotenv. `KEY: value` is matched only so that it stops the
- # run. dotenv skips a bare `KEY` line (`export KEY`, say), unless the next line that is not blank
- # starts with `=`: its `KEY\s*=` crosses line breaks, so that pair sets the key and stops the run too.
+ done
+ # dotenv takes a bare CR for a line break and can start a setting right after U+2028 or U+2029 (after
+ # a comment or a quoted value, say), so a line grep sees can hold the key's setting in its middle.
+ # Such a character on any line naming the key, comments included, is rejected. Every grep here runs
+ # with -a in the C locale: otherwise a NUL anywhere in the file, or a byte that is not UTF-8 in a
+ # UTF-8 locale, can hide the key's line. A NUL, which bash would drop from the line, becomes \377.
+ line="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C grep -aE "(^|[^A-Za-z0-9_.-])${key}([^A-Za-z0-9_.-]|\$)")" || true
+ if LC_ALL=C grep -qE $'\r.|\xe2\x80[\xa8\xa9]' <<<"$line"; then bad=1; fi
+ # The last line setting the key wins, as in dotenv, so only that one is checked below. `KEY: value`
+ # and `KEY:value` are matched only so that they stop the run. dotenv skips a bare `KEY` line (`export
+ # KEY`, say), unless the next line that is not blank starts with `=`: its `KEY\s*=` crosses line
+ # breaks, so that pair sets the key and stops the run too.
line=""
while IFS= read -r match; do
value="${match#*"$key"}"
- if [ -z "${value//[[:space:]]/}" ]; then
+ if LC_ALL=C grep -qxE "$s*" <<<"$value"; then
next="$(LC_ALL=C tail -n +"$((${match%%:*} + 1))" "$file" | LC_ALL=C tr '\000' '\377' |
- LC_ALL=C grep -avE -m 1 "^([[:space:]]|$blank)*\$")" || true
- LC_ALL=C grep -qE "^([[:space:]]|$blank)*=" <<<"$next" || continue
+ LC_ALL=C grep -avE -m 1 "^$s*\$")" || true
+ LC_ALL=C grep -qE "^$s*=" <<<"$next" || continue
fi
line="$match"
- done < <(LC_ALL=C grep -anE "$keyline" "$file" 2>/dev/null)
- [ -n "$line" ] || return 1
+ done < <(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C grep -anE "$keyline")
+ [ -n "$line$bad" ] || return 1
n="${line%%:*}"
line="${line#*:}"
+ # dotenv also takes a byte-order mark or a Unicode blank (a no-break space, say) for whitespace and
+ # decodes the file as UTF-8, so a byte that is not UTF-8 reaches the app as U+FFFD. grep and the trims
+ # below do neither, so the line is rejected with such a blank before the key, around its `=`, or at
+ # either end of an unquoted value (a quoted value keeps its blanks), or with a NUL or a byte that is
+ # not UTF-8 anywhere.
+ if [ -n "$bad" ] || ! LC_ALL=C grep -qE "^[[:space:]]*(export[[:space:]]+)?${key}[[:space:]]*([=:]|\$)" <<<"$line" ||
+ LC_ALL=C grep -qE "^[^=:]*[=:][[:space:]]*($blank)|^[^=:]*[=:][[:space:]]*([^\"'\`[:space:]#][^#]*)?($blank)[[:space:]]*(#|\$)" <<<"$line" ||
+ LC_ALL=C grep -qvxE "$utf8" <<<"$line"; then
+ echo "[config] ERROR: $file has a bare CR, U+2028 or U+2029 on a line naming $key, a NUL or a byte that is" >&2
+ echo "[config] not UTF-8 on the line setting it, or a byte-order mark or Unicode blank next to $key, its =" >&2
+ echo "[config] or the ends of its unquoted value. Save it as UTF-8 with plain ASCII blanks and LF or CRLF" >&2
+ echo "[config] line ends, or pass $key in the environment." >&2
+ return 2
+ fi
+ # dotenv reads the line after a bare `NAME:` line as NAME's value, so the key's line right there does
+ # not set the key for the app, although the scripts would read it. dotenv can also start a setting
+ # after a bare CR, U+2028 or U+2029, so `# note<U+2028>NAME:` is such a line too.
+ prev=""
+ [ "$n" -le 1 ] || prev="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C sed -n "$((n - 1))p")"
+ if LC_ALL=C grep -qE "(^|"$'\r|\xe2\x80[\xa8\xa9]'")$s*(export$s+)?[A-Za-z0-9_.-]+:"$'\r'"?\$" <<<"$prev"; then
+ echo "[config] ERROR: $file has a bare NAME: line just before a line setting $key, which the app reads as" >&2
+ echo "[config] NAME's value instead. Give NAME: a value or remove it, or pass $key in the environment." >&2
+ return 2
+ fi
+ # dotenv reads a quoted value on across lines, so a line inside one does not set the key. The value
+ # ends at the first quote without a backslash before it or, when more than blanks or a comment follow
+ # that quote on its line, at the last quote before it that only blanks or a comment follow. So the
+ # line is rejected when, for some quote character, such a quote sits on it or after it, up to the
+ # first quote without a backslash, and an earlier line opens a value with that quote which no such
+ # quote closes before the line (openwa_env_opener). Lines are split where dotenv splits them
+ # (openwa_env_lines).
+ rest="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C tail -n +"$n" | openwa_env_lines)"
+ closes=""
+ for q in "'" '"' '`'; do
+ next="$rest"
+ o="$(LC_ALL=C grep -anE -m 1 "(^|[^\\\\])$q" <<<"$next")" || true
+ [ -z "$o" ] || next="$(LC_ALL=C sed -n "1,${o%%:*}p" <<<"$next")"
+ ! LC_ALL=C grep -aqE "^([^$q]|\\\\$q)*$q$s*(#.*)?\$" <<<"$next" || closes="$closes$q"
+ done
+ o=""
+ if [ "$n" -gt 1 ] && [ -n "$closes" ]; then
+ prev="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C sed -n "1,$((n - 1))p" | openwa_env_lines)"
+ o="$(openwa_env_opener "$s" "$closes" <<<"$prev")"
+ fi
+ if [ -n "$o" ]; then
+ # Count only the lines the file has, not the ones openwa_env_lines added. A file line that starts with
+ # a CR or U+2028 still counts: openwa_env_lines prints the empty text before that character first.
+ o="$(LC_ALL=C sed -n "1,${o}p" <<<"$prev" | LC_ALL=C grep -acvE "^("$'\r|\xe2\x80\xa8'")")" || true
+ echo "[config] ERROR: $file opens a quoted value on line $o that the app can read on to line $n, which sets" >&2
+ echo "[config] $key, as part of that value. Close the quote on the line that opens it, or pass $key in" >&2
+ echo "[config] the environment." >&2
+ return 2
+ fi
value="${line#*"$key"}"
value="${value#"${value%%[![:space:]]*}"}"
case "$value" in
@@ -103,11 +304,23 @@ openwa_env_file_value() {
value="${value%"${value##*[![:space:]]}"}"
case "$value" in
\"*\\*) ;; # dotenv expands \n and \r inside double quotes
- \'*\\\' | \`*\\\`) ;; # dotenv may take the backslash and quote for an escaped quote and read on
\"*\" | \'*\' | \`*\`)
# A quote inside means the pair does not wrap the whole value: a comment ending in one follows it.
case "${value:1:${#value}-2}" in
*"${value:0:1}"*) ;;
+ *\\)
+ # dotenv may take the backslash and quote for an escaped quote and read on, up to the first
+ # quote without a backslash, to a quote that only blanks or a comment follow on its line, so
+ # the value ends here only when no later line has such a quote.
+ q="${value:0:1}"
+ next="$(LC_ALL=C sed 1d <<<"$rest")"
+ o="$(LC_ALL=C grep -anE -m 1 "(^|[^\\\\])$q" <<<"$next")" || true
+ [ -z "$o" ] || next="$(LC_ALL=C sed -n "1,${o%%:*}p" <<<"$next")"
+ if ! LC_ALL=C grep -aqE "^([^$q]|\\\\$q)*$q$s*(#.*)?\$" <<<"$next"; then
+ printf '%s' "${value:1:${#value}-2}"
+ return 0
+ fi
+ ;;
*)
printf '%s' "${value:1:${#value}-2}"
return 0
@@ -138,7 +351,8 @@ openwa_env_file_value() {
esac
echo "[config] ERROR: $file sets $key in a form these scripts do not parse: a quoted value followed by a" >&2
echo "[config] comment or not closed on its line, a quoted value containing its own quote character or" >&2
- echo "[config] ending in a backslash, a double-quoted value with a backslash, or KEY: value." >&2
+ echo "[config] ending in a backslash the app can read on past to a later quote, a double-quoted value" >&2
+ echo "[config] with a backslash, or KEY: value." >&2
echo "[config] Move a comment to its own line and keep the quotes, close the quote on the same line, wrap a" >&2
echo "[config] value in a quote character it does not contain, single-quote a value whose backslashes are" >&2
printf '%s\n' "[config] literal and not at its end (inside double quotes the app turns \\n and \\r into line breaks)," >&2
diff --git a/scripts/smoke-test-backup-restore.sh b/scripts/smoke-test-backup-restore.sh
index 684fdcc44..a5e245f39 100755
--- a/scripts/smoke-test-backup-restore.sh
+++ b/scripts/smoke-test-backup-restore.sh
@@ -107,7 +107,7 @@ db_fingerprint() {
# to shadow tar (simulating an incomplete archive) without touching the real scripts.
populate_shim() {
shim_dir="$1"
- tools="env bash sh cp tar gzip mktemp date rm sed mkdir ls cat chmod grep printf uname dirname"
+ tools="env bash sh cp tar gzip mktemp date rm sed mkdir ls cat chmod grep printf uname dirname tr tail"
if [ "${2:-}" = "with-sqlite3" ]; then
tools="$tools sqlite3"
fi
@@ -1562,6 +1562,10 @@ done
if ! grep -q 'sets ENGINE_TYPE in a form' <<<"$OUT_AG" || ! grep -q 'skipping only the check for missing Baileys state' <<<"$OUT_AG"; then
fail "(ag) backup did not report the ENGINE_TYPE line it cannot parse and that it carried on: $OUT_AG"
fi
+# The run carries on, so the line is reported as a warning rather than an error.
+if grep -q 'ERROR' <<<"$OUT_AG" || ! grep -q '^\[config\] WARN: .* sets ENGINE_TYPE in a form' <<<"$OUT_AG"; then
+ fail "(ag) backup reported the ENGINE_TYPE line it carries on past as an error: $OUT_AG"
+fi
rm -f "$AGB/data/act"$'\343\200\200'"ive.sqlite"
# dotenv reads the line after a bare `NAME:` line as NAME's value, so the app never sets the key from it.
printf 'LOG_LEVEL:\nDATABASE_NAME=./data/active.sqlite\n' >"$AGB/.env"
@@ -1591,20 +1595,157 @@ for check in 'DATABASE_NAME\n=./data/active.sqlite|names DATABASE_NAME on a line
fi
done
# dotenv can take the backslash and quote that end a quoted value for an escaped quote, and read on to a
-# quote on a later line.
-printf '%s\n' "DATABASE_NAME='./data/active.sqlite\\'" "'" >"$AGB/.env"
-set +e
-OUT_AG="$(cd "$AGB" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"
-RC_AG=$?
-set -e
-if [ "$RC_AG" -eq 0 ] || ! grep -q 'ending in a backslash' <<<"$OUT_AG"; then
- fail "(ag) a quoted DATABASE_NAME ending in a backslash did not fail the lookup (rc $RC_AG): $OUT_AG"
-fi
+# later quote that only blanks or a comment follow, escaped or not.
+for ag_next in "'" "x\\' # note"; do
+ printf '%s\n' "DATABASE_NAME='./data/active.sqlite\\'" "$ag_next" >"$AGB/.env"
+ set +e
+ OUT_AG="$(cd "$AGB" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"
+ RC_AG=$?
+ set -e
+ if [ "$RC_AG" -eq 0 ] || ! grep -q 'ending in a backslash' <<<"$OUT_AG"; then
+ fail "(ag) a quoted DATABASE_NAME ending in a backslash before '$ag_next' did not fail the lookup (rc $RC_AG): $OUT_AG"
+ fi
+done
# An empty `KEY=` followed by blank lines and an unquoted line stays empty for dotenv too.
printf 'DATABASE_NAME=\n\n# data store\nLOG_LEVEL=info\n' >"$AGB/.env"
if [ "$(cd "$AGB" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT)" != DEFAULT ]; then
fail "(ag) an empty DATABASE_NAME= followed by a comment and another key did not resolve to the default"
fi
+# dotenv reads each of these unambiguously, and the scripts used to stop on them: a Unicode blank inside
+# a quoted value; a Unicode blank, a byte that is not UTF-8 or a bare NAME: line before it on an earlier
+# line for the key, which the last one overrides; and a value single-quoted for its `#` and ending in a
+# backslash, as the dashboard writes it, with no single quote on a later line or only one followed by
+# more than a comment. An empty value followed by a comment holding a Unicode blank is still empty, so
+# the default applies. The key's line also sets it after a quoted value that closes on its own lines,
+# even on a line that looks like it opens one, or that nothing closes, which dotenv then reads without
+# its quote, and after one closed by an escaped quote whose later quote is followed by more than a
+# comment. A line that would open a quoted value but is the value of a bare NAME: line before it
+# opens nothing, and neither does a U+2028 inside an unquoted value start a line. Nor does a quote
+# inside a value opened with another quote character open a value, even after a U+2029 there, or a
+# quote in an unquoted value, which runs on past a U+2028. Fields: ./.env, .env.generated, the key and
+# the value the app reads, all with printf escapes.
+AGC="$WORK/ag-last-line"
+mkdir -p "$AGC/data"
+for check in 'DATABASE_NAME=\047a\302\240#b\047||DATABASE_NAME|a\302\240#b' \
+ 'LOG_LEVEL:\nDATABASE_NAME=a\nDATABASE_NAME=b||DATABASE_NAME|b' \
+ 'DATABASE_NAME=a\302\240\nDATABASE_NAME=b||DATABASE_NAME|b' \
+ 'DATABASE_NAME=a # donn\351es\nDATABASE_NAME=b||DATABASE_NAME|b' \
+ 'DATABASE_NAME= # note\302\240||DATABASE_NAME|DEFAULT' \
+ 'DATABASE_NAME=#a\302\240#b\nLOG_LEVEL=info||DATABASE_NAME|DEFAULT' \
+ '|DATABASE_PASSWORD=\047p#w\134\047\nDATABASE_HOST="db"|DATABASE_PASSWORD|p#w\134' \
+ '|DATABASE_PASSWORD=\047p#w\134\047\nREDIS_PASSWORD=\047x#y\047|DATABASE_PASSWORD|p#w\134' \
+ 'N:\nB=\047y\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b' \
+ 'M:\nN=\n\047x\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b' \
+ 'A=1\342\200\250B=\047y\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b' \
+ '|DATABASE_PASSWORD=\047pw\302\240#x\047|DATABASE_PASSWORD|pw\302\240#x' \
+ 'CERT="BEGIN\nabc\nEND"\nDATABASE_NAME=b||DATABASE_NAME|b' \
+ 'LOG_LEVEL=\047x\nDATABASE_NAME=b||DATABASE_NAME|b' \
+ 'LOG_LEVEL=\047a\134\047\nDATABASE_NAME=b\nY=\047c\047||DATABASE_NAME|b' \
+ 'CERT=\047a\nNOTE=\047 # c\nDATABASE_NAME=b\nY=x\047||DATABASE_NAME|b' \
+ 'A=\047x\nB="y\n\047\nDATABASE_NAME=b\nC=z"||DATABASE_NAME|b' \
+ 'A=\047x\342\200\251B="y\047 # c\nDATABASE_NAME=b\nC=z"||DATABASE_NAME|b' \
+ 'A=x\047y\342\200\250B=\047z\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b' \
+ 'N:\nx\047\342\200\250B=\047z\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b'; do
+ IFS='|' read -r ag_env ag_gen ag_key ag_want <<<"$check"
+ # shellcheck disable=SC2059 # the fields' escapes are the point
+ printf "$ag_env\n" >"$AGC/.env"
+ # shellcheck disable=SC2059
+ printf "$ag_gen\n" >"$AGC/data/.env.generated"
+ # shellcheck disable=SC2059
+ ag_want="$(printf "$ag_want")"
+ set +e
+ got="$(cd "$AGC" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve "$ag_key" DEFAULT 2>&1)"
+ rc=$?
+ set -e
+ if [ "$rc" -ne 0 ] || [ "$got" != "$ag_want" ]; then
+ fail "(ag) $ag_key in '$ag_env' / '$ag_gen' resolved to '$got' (rc $rc) instead of the app's '$ag_want'"
+ fi
+done
+# dotenv reads the line after a bare `DATABASE_NAME:` as its value, and Docker Compose, which
+# interpolates ./.env into the container, reads `KEY:value` and `KEY :value` as settings, so each stops
+# the run even when .env.generated sets the key. The error names the `KEY: value` form, not a bare
+# NAME: line before the key's line.
+printf 'DATABASE_NAME=./data/gen\n' >"$AGC/data/.env.generated"
+for ag_env in 'DATABASE_NAME:\n./data/active.sqlite' 'DATABASE_NAME:./data/a.sqlite' \
+ 'DATABASE_NAME=./data/b.sqlite\nDATABASE_NAME :./data/a.sqlite'; do
+ # shellcheck disable=SC2059 # the escapes are the point
+ printf "$ag_env\n" >"$AGC/.env"
+ set +e
+ OUT_AG="$(cd "$AGC" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"
+ RC_AG=$?
+ set -e
+ if [ "$RC_AG" -ne 2 ] || ! grep -q 'sets DATABASE_NAME in a form' <<<"$OUT_AG" || grep -q 'bare NAME:' <<<"$OUT_AG"; then
+ fail "(ag) '$ag_env' was not reported as KEY: value (rc $RC_AG): $OUT_AG"
+ fi
+done
+# dotenv reads a quoted value on to its closing quote, so a line for the key inside it does not set the
+# key. Whether the quote opens on an earlier line for the key or another key's, at the start of a line
+# after an empty NAME=, after a bare NAME: line that is itself the value of the one before it, or after
+# a bare CR, U+2028 or U+2029, where dotenv starts a line, and whether it closes before one of those,
+# the run stops instead of reading the key's last line or a line inside the value. A quote followed by
+# a colon opens a value too.
+for ag_env in 'DATABASE_NAME=\047a\302\240\nDATABASE_NAME=./data/a.sqlite\047' \
+ 'DATABASE_NAME=\047x\nDATABASE_NAME=\377a\nDATABASE_NAME=./data/a.sqlite\047' \
+ 'LOG_LEVEL=\047x\nDATABASE_NAME=a\302\240\nDATABASE_NAME=./data/a.sqlite\nY\047' \
+ 'LOG_LEVEL=\047x\nLOG:\nDATABASE_NAME=a\nDATABASE_NAME=./data/a.sqlite\nY\047' \
+ 'DATABASE_NAME=\047a\nDATABASE_NAME=./data/a.sqlite\047' \
+ 'LOG_LEVEL=\n\n"x\nDATABASE_NAME=./data/a.sqlite" # note' \
+ 'W:\nN:\nB=\047y\nDATABASE_NAME=./data/a.sqlite\nz\047' \
+ 'x\rA=\047y\nDATABASE_NAME=./data/a.sqlite\nz\047' \
+ '# n\342\200\250A=\047y\nDATABASE_NAME=./data/a.sqlite\nz\047' \
+ 'A=\047x\nDATABASE_NAME=./data/a.sqlite\nz\047\342\200\251more' \
+ 'A=\047x\nDATABASE_NAME=./data/a.sqlite\nz\047\rmore' \
+ 'A=\047x\nB=\047:\nDATABASE_NAME=./data/a.sqlite\n\047'; do
+ # shellcheck disable=SC2059 # the escapes are the point
+ printf "$ag_env\n" >"$AGC/.env"
+ set +e
+ OUT_AG="$(cd "$AGC" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"
+ RC_AG=$?
+ set -e
+ if [ "$RC_AG" -ne 2 ] || ! grep -q 'opens a quoted value on line' <<<"$OUT_AG"; then
+ fail "(ag) DATABASE_NAME inside the quoted value in '$ag_env' resolved to '$OUT_AG' (rc $RC_AG) instead of stopping"
+ fi
+done
+# The error names the file's line that opens the value, not one counted after a line break dotenv adds,
+# and still counts a line of the file that starts with U+2029.
+printf 'Q=1\n\342\200\251# n\nx\ry\342\200\250A=\047v\nDATABASE_NAME=./data/a.sqlite\nz\047\n' >"$AGC/.env"
+set +e
+OUT_AG="$(cd "$AGC" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"
+set -e
+if ! grep -q 'opens a quoted value on line 3 that the app can read on to line 4,' <<<"$OUT_AG"; then
+ fail "(ag) the quoted value opened after a bare CR and U+2028 was not reported on line 3: $OUT_AG"
+fi
+# The scripts trim only ASCII blanks, so a no-break space after a closing quote stops the run in a
+# UTF-8 locale too, as it does in the C locale.
+ag_loc="$(locale -a 2>/dev/null | grep -m 1 -ixE 'C\.UTF-?8|en_US\.UTF-?8')" || true
+if [ -n "$ag_loc" ]; then
+ printf 'DATABASE_NAME=\047\047 \302\240\n' >"$AGC/.env"
+ set +e
+ # shellcheck disable=SC2016 # expanded by the inner shell
+ OUT_AG="$(cd "$AGC" && LC_ALL="$ag_loc" bash -c 'DATA_DIR=./data && . "$1" && openwa_resolve DATABASE_NAME DEFAULT' _ "$REPO_ROOT/scripts/lib-env.sh" 2>&1)"
+ RC_AG=$?
+ set -e
+ if [ "$RC_AG" -ne 2 ] || ! grep -q 'sets DATABASE_NAME in a form' <<<"$OUT_AG"; then
+ fail "(ag) a no-break space after a closing quote resolved to '$OUT_AG' (rc $RC_AG) under LC_ALL=$ag_loc"
+ fi
+fi
+# Without tr, tail, sed or grep the key's line cannot be read, so the lookup stops instead of taking
+# the default.
+printf 'DATABASE_SSL=true\n' >"$AGC/.env"
+for ag_tool in tr tail sed grep; do
+ rm -rf "$AGC/shim"
+ mkdir -p "$AGC/shim"
+ populate_shim "$AGC/shim"
+ rm -f "$AGC/shim/$ag_tool"
+ set +e
+ # shellcheck disable=SC2016 # expanded by the inner shell
+ OUT_AG="$(cd "$AGC" && PATH="$AGC/shim" bash -c 'DATA_DIR=./data && . "$1" && openwa_resolve DATABASE_SSL false' _ "$REPO_ROOT/scripts/lib-env.sh" 2>&1)"
+ RC_AG=$?
+ set -e
+ if [ "$RC_AG" -ne 2 ] || ! grep -q "$ag_tool is required" <<<"$OUT_AG"; then
+ fail "(ag) without $ag_tool DATABASE_SSL resolved to '$OUT_AG' (rc $RC_AG) instead of stopping"
+ fi
+done
# A quoted value holding its own quote character is named as such, and the remedy warns that double
# quotes expand \n and \r. The other quote style, which the error suggests, archives the configured store.
make_fixture "$AGB/data/o'brien.sqlite" "ag-active"