Free, Open Source, Self-Hosted WhatsApp API Gateway
TypeScript apibotgatewayself-hostedwhatsappwhatsapp-apiwhatsapp-automationwhatsapp-botwhatsapp-gatewaywhatsapp-gateway-apiwhatsapp-webwhatsapp-web-api
fix(scripts): only reject the env line that sets the key (#1823)
Yudhi Armyndharisc05f6adf6bb9
diff --git a/docs/11-operational-runbooks.md b/docs/11-operational-runbooks.mdindex cb61e8bb4..5960a4b2a 100644--- a/docs/11-operational-runbooks.md+++ b/docs/11-operational-runbooks.md@@ -786,31 +786,53 @@ OPENWA_DATA_DIR=/srv/openwa/data \> the container, where it names a path in the volume, while the scripts read it against the current> directory.>-> On every install, inside a container or not, a quoted value followed by a `#` comment or not-> closed on its line, a quoted value containing its own quote character or ending in a backslash, a-> double-quoted value with a backslash, or a `KEY: value` line in `./.env` or `.env.generated` stops-> the script with an error naming the key, before anything is archived or restored. So does a bare-> CR, U+2028 or U+2029 on any line naming the key, a comment included, since the app starts a new-> line there; a NUL or a byte that is not UTF-8 on a line setting the key; a byte-order mark or a-> Unicode blank (such as a no-break space) before the key, around its `=` or at either end of its-> value; a bare `NAME:` line right before the key's line, which the app reads as `NAME`'s value;-> and a key whose value the app takes from a later line, either a bare `KEY` line followed, past-> any blank lines, by one starting with `=` or an empty `KEY=` followed by a quoted value. The app-> reads such a line, so neither the script default nor a guess is safe to use. `DATABASE_NAME` is-> not read by a PostgreSQL backup through `DATABASE_URL` or by the restore of a PostgreSQL archive,-> and an unreadable `ENGINE_TYPE` only skips the warning about missing Baileys state.+> On every install, inside a container or not, the scripts stop with an error naming the key, before+> anything is archived or restored, when the last line setting it in `./.env` or `.env.generated`,+> which is the one the app keeps, has one of these forms:+>+> - a quoted value followed by a `#` comment or not closed on its line+> - a quoted value containing its own quote character, or ending in a backslash the app can read+> past to a later quote of that kind followed only by blanks or a comment+> - a double-quoted value with a backslash+> - a `KEY: value` or `KEY:value` line (Docker Compose reads both from `./.env` into the container)+> - a NUL or a byte that is not UTF-8+> - a byte-order mark or a Unicode blank (such as a no-break space) before the key, around its `=`,+> at either end of an unquoted value or after a closing quote+> - a bare `NAME:` line right before the key's line, which the app reads as `NAME`'s value+> - a bare `KEY` line followed, past any blank lines, by one starting with `=`, or an empty `KEY=`+> followed by a quoted value, which the app reads as the key's value+> - a quoted value opened on an earlier line, the key's own or another key's, that the app can read+> on to the key's line, which then belongs to that value+>+> A bare CR, U+2028 or U+2029 on any line naming the key, a comment included, stops the run too,+> since the app can start a setting right after one. Like the app, the scripts break lines at a bare+> CR, and at a U+2028 or U+2029 outside an unquoted value. The app reads each of these lines+> (`KEY:value` only through Docker Compose, from `./.env`), so neither the script default nor a+> guess is safe to use. `DATABASE_NAME` is not read by a PostgreSQL backup through `DATABASE_URL` or+> by the restore of a PostgreSQL archive, and an unreadable `ENGINE_TYPE` only skips the warning+> about missing Baileys state.+>> Move the comment to its own line and keep the quotes (an unquoted value ends at a `#` and loses> its outer blanks), wrap the value in a quote character it does not contain, single-quote a value> whose backslashes are literal and not at its end (inside double quotes the app turns `\n` and `\r`> into line breaks; if that is intended, pass the key in the environment), write `KEY=value` on one-> line for `KEY: value` and for a value on a later line, give a bare `NAME:` line a value or remove-> it, save the file as UTF-8 with ASCII blanks and LF or CRLF line endings, or pass the key in the-> environment. A restore reads the archive's `.env.generated`, which cannot be edited in place, so a-> key it holds in such a form has to be passed in the environment. Blanks around `=`, CRLF line-> endings, a value in one pair of quotes and a `#` comment after an unquoted value are read as the-> app reads them. The scripts read the file line by line, so a line for the key inside a quoted-> value that spans several lines, another key's or an earlier one for the same key, is taken as a-> setting, although the app reads it as part of that value.+> line for `KEY: value`, `KEY:value` and a value on a later line, close a quote left open on an+> earlier line, give a bare `NAME:` line a value or remove it, save the file as UTF-8 with ASCII+> blanks and LF or CRLF line endings, or pass the key in the environment. A restore reads the+> archive's `.env.generated`, which cannot be edited in place, so a key it holds in such a form has+> to be passed in the environment, and so does a `DATABASE_*` value saved from+> Dashboard > Infrastructure in one of these forms, since the next save writes it the same way. The+> scripts need `tr`, `tail`, `sed` and `grep` to read either file and stop when one is missing.+> Blanks around `=`, CRLF line endings, a value in one pair of quotes and a `#` comment after an+> unquoted value are read as the app reads them, and so is a quoted value that spans several lines+> (a PEM key, say) and closes before the key's line, even when it holds quotes of another kind.+>+> Under Docker Compose the container gets `./.env` through Compose, which reads some values+> differently from the app's own reader: it keeps backticks and a `#` with no blank before it as part+> of the value, and expands `$NAME` outside single quotes. Run inside the container, the scripts see+> the value Compose passed in; run on the host, they read `./.env` as the app reads a `.env` file, so+> a value of that kind can resolve differently there. Single-quote such a value in `./.env`, or pass+> the key in the environment on the host.**Verification:**diff --git a/scripts/backup.sh b/scripts/backup.shindex 7c7662b5a..6ea401df5 100755--- a/scripts/backup.sh+++ b/scripts/backup.sh@@ -297,8 +297,11 @@ if [ -d "$BAILEYS_DIR" ]; thenif [ -n "$(find -H "$BAILEYS_DIR" -mindepth 1 -maxdepth 1 -type d -print -quit)" ]; thenrecord_engine_state_note "baileys/ (recorded whenever Baileys state exists; it cannot show whether it was live)"fi-# The warning is advisory, so an ENGINE_TYPE line the scripts cannot parse skips it instead of the backup.-elif ! ENGINE_TYPE_RESOLVED="$(openwa_resolve ENGINE_TYPE '')"; then+# The warning is advisory, so an ENGINE_TYPE line the scripts cannot parse skips it instead of the backup,+# and its error is reported as a warning. A failed lookup prints nothing on stdout and a successful one+# nothing on stderr, so both share the substitution.+elif ! ENGINE_TYPE_RESOLVED="$(openwa_resolve ENGINE_TYPE '' 2>&1)"; then+ sed -e 's/^\[config\] ERROR: /[config] WARN: /' -e 's/^\[config\] /[config] /' <<<"$ENGINE_TYPE_RESOLVED" >&2log "WARN: ENGINE_TYPE could not be read (see above); skipping only the check for missing Baileys state"elif [ "$ENGINE_TYPE_RESOLVED" = "baileys" ]; thenlog "WARN: ENGINE_TYPE=baileys but $BAILEYS_DIR was not found — restored sessions will require pairing"diff --git a/scripts/lib-env.sh b/scripts/lib-env.shindex 829f3e491..4b23d3fd0 100644--- a/scripts/lib-env.sh+++ b/scripts/lib-env.sh@@ -17,77 +17,278 @@## Deliberately conservative: only the `KEY=value` forms dotenv reads plainly are honoured. Blanks# around the `=` and the value, CRLF line endings, a value wrapped in one pair of quotes and a comment-# after an unquoted value are read as dotenv reads them. Anything else (a quoted value followed by a-# comment, containing its own quote character or ending in a backslash, a double-quoted value with-# escapes, a `KEY: value` line, a bare CR, U+2028 or U+2029 on a line naming the key, a NUL or byte-# that is not UTF-8 on its line, a byte-order mark or Unicode blank next to the key, its `=` or the-# ends of its value, a bare `NAME:` line just before it, a bare `KEY` or an empty `KEY=` the app-# completes from a later line) stops the run rather than being guessed at: the app still reads such a-# line, so neither a guess nor the default is safe to back up or restore to, and a silently-# mis-resolved path is the exact failure this exists to prevent. The file is still read line by line:-# a line for the key inside a quoted value that spans lines, whether another key's or an earlier one-# of the same key, is taken as a setting, although the app reads it as part of that value. Nothing-# here exports anything: each key is looked up by name, so a stray entry in an operator's .env can-# never reach the script's own environment.+# after an unquoted value are read as dotenv reads them. As in dotenv, the last line setting the key+# wins, so only that line is checked. Anything else (a quoted value followed by a comment, containing+# its own quote character or ending in a backslash the app can read on past to a later quote, a+# double-quoted value with escapes, a `KEY: value` or `KEY:value` line, a bare CR, U+2028 or U+2029 on+# any line naming the key, a NUL or byte that is not UTF-8 on its line, a byte-order mark or Unicode+# blank next to the key, its `=`, the ends of an unquoted value or a closing quote, a bare `NAME:` line+# just before it, a bare `KEY` or an empty `KEY=` the app completes from a later line) stops the run+# rather than being guessed at: the app still reads such a line, so neither a guess nor the default is+# safe to back up or restore to, and a silently mis-resolved path is the exact failure this exists to+# prevent. So does a quoted value opened on an earlier line, the key's own or another key's, that the+# app can read on to the key's line, where lines break as the app breaks them: at a bare CR, and at+# U+2028 or U+2029 outside an unquoted value. Nothing here exports anything: each key is looked up by+# name, so a stray entry in an operator's .env can never reach the script's own environment.++# openwa_env_lines - copy stdin with a line break wherever dotenv starts a line: at a bare CR, and at+# U+2028 or U+2029 unless an unquoted value without a `#` runs on past it. Each added line starts with+# the CR or a U+2028, which the patterns read as a blank, and CRLF line ends lose their CR. It also+# splits inside a quoted value, and after a stray quote in an unquoted one; openwa_env_opener skips+# the lines that adds inside a value.+openwa_env_lines() (+ export LC_ALL=C+ local line seg a b cr=$'\r' ls=$'\xe2\x80\xa8'+ local setting='^(export[[:space:]]+)?[A-Za-z0-9_.-]+([[:space:]]*=|:$|:[[:space:]])(.*)$'+ sed -e $'s/\r$//' -e $'s/\xe2\x80\xa9/\xe2\x80\xa8/g' | while IFS= read -r line || [ -n "$line" ]; do+ seg=""+ while :; do+ a="${line%%"$cr"*}"+ b="${line%%"$ls"*}"+ [ "${#a}" -lt "${#line}" ] || [ "${#b}" -lt "${#line}" ] || break+ if [ "${#a}" -lt "${#b}" ]; then+ printf '%s\n' "$seg$a"+ seg="$cr"+ line="${line:${#a}+1}"+ continue+ fi+ seg="$seg$b"+ line="${line:${#b}+3}"+ a="${seg#"$cr"}"+ a="${a#"$ls"}"+ a="${a#"${a%%[![:space:]]*}"}"+ if [[ $a =~ $setting ]] && [[ ${BASH_REMATCH[3]} != *[#\'\"\`]* ]]; then+ seg="$seg$ls"+ continue+ fi+ printf '%s\n' "$seg"+ seg="$ls"+ done+ printf '%s\n' "$seg$line"+ done+)++# openwa_env_opener <s> <closes> - read the lines before the key's line, split by openwa_env_lines, and+# print the number of the one that opens a quoted value dotenv reads on to the key's line. <s> matches+# one blank; <closes> lists the quote characters that a quote on or after the key's line, up to the+# first such quote without a backslash before it, can close a value with: one only blanks or a comment+# follow. The lines are read in order, as dotenv reads them, so a line inside a value opens nothing.+# A quote opens a value after `NAME=` or `NAME: `, after a bare `NAME` and an `=` on the next line that+# is not blank, or at the start of the line after a bare `NAME:` or of a line after an empty `NAME=` or+# `NAME:` or a bare `NAME` and a lone `=`, past blank lines. The value ends at the last quote that only+# blanks or a comment follow, up to the first quote without a backslash; with none, dotenv reads the+# line after `NAME=`, `NAME: ` or a bare `NAME:` as an unquoted value, which runs on past a U+2028 or+# U+2029 until a `#`, and any other line as no setting at all.+openwa_env_opener() (+ local LC_ALL=C+ local s="$1" closes="$2" l v r q c i=0 j n e u kind last prior open closer ls=$'\xe2\x80\xa8'+ local set="^$s*(export$s+)?[A-Za-z0-9_.-]+($s*=|:$s)" colon="^$s*(export$s+)?[A-Za-z0-9_.-]+:\$"+ local bare="^$s*(export$s+)?[A-Za-z0-9_.-]+$s*\$" empty="^$s*(export$s+)?[A-Za-z0-9_.-]+($s*=|:)$s*\$"+ local blankline="^$s*\$" eq="^$s*=" lone="^$s*=$s*\$" quote="^$s*[=\"'\`]"+ # after[i]: line i is the value of a bare `NAME:` line before it. free[i]: dotenv starts reading at+ # line i, which is not inside a value.+ local -a lines after free+ while IFS= read -r l; do+ i=$((i + 1))+ lines[i]="$l"+ done+ n=$i+ i=1+ while [ "$i" -le "$n" ]; do+ l="${lines[i]}"+ free[i]=1+ # A line without a quote opens nothing; only a bare `NAME:` line, the value of one, or a value that+ # runs on past a U+2028 needs the checks below.+ case "$l" in+ *[\"\'\`]* | *:) ;;+ *)+ if [ -z "${after[i]:-}" ] && [[ ${lines[i + 1]:-} != "$ls"* ]]; then+ i=$((i + 1))+ continue+ fi+ ;;+ esac+ kind=""+ v="$l"+ if [ -n "${after[i]:-}" ]; then+ kind=value+ elif [[ $l =~ $set ]]; then+ kind=value+ v="${l#*[=:]}"+ elif [[ $l =~ $colon ]]; then+ after[i + 1]=1+ elif [[ $l =~ $quote ]]; then+ # The two lines before it that are not blank, which have to start its value.+ j=$i last="" prior=""+ while [ "$j" -gt 1 ]; do+ j=$((j - 1))+ [[ ! ${lines[j]} =~ $blankline ]] || continue+ [ -z "$last" ] || { prior=$j && break; }+ last=$j+ done+ if [ -z "$last" ] || [ -z "${free[last]:-}" ] || [ -n "${after[last]:-}" ]; then+ :+ elif [[ $l =~ $eq ]]; then+ if [[ ${lines[last]} =~ $bare ]]; then+ kind=value+ v="${l#*=}"+ fi+ elif [[ ${lines[last]} =~ $lone ]]; then+ if [ -n "$prior" ] && [ -n "${free[prior]:-}" ] && [ -z "${after[prior]:-}" ] && [[ ${lines[prior]} =~ $bare ]]; then+ kind=quoted+ fi+ elif [[ ${lines[last]} =~ $empty ]]; then+ kind=quoted+ fi+ fi+ if [ -z "$kind" ]; then+ i=$((i + 1))+ continue+ fi+ q=""+ for c in "'" '"' '`'; do+ open="^$s*$c"+ if [[ $v =~ $open ]]; then q="$c" && break; fi+ done+ if [ -n "$q" ]; then+ open="(^|[^\\\\])$q"+ closer="^([^$q]|\\\\$q)*$q$s*(#.*)?\$"+ r="${v#*"$q"}"+ e=""+ if [[ $r =~ $open ]]; then+ [[ ! $r =~ $closer ]] || e=$i+ else+ j=$i u=""+ while [ "$j" -lt "$n" ]; do+ j=$((j + 1))+ [[ ! ${lines[j]} =~ $closer ]] || e=$j+ if [[ ${lines[j]} =~ $open ]]; then u=$j && break; fi+ done+ if [ -z "$u" ] && [[ $closes == *"$q"* ]]; then+ printf '%s' "$i"+ return 0+ fi+ [ -n "$e" ] || [[ ! $r =~ $closer ]] || e=$i+ fi+ if [ -n "$e" ]; then+ i=$((e + 1))+ continue+ fi+ if [ "$kind" = quoted ]; then+ i=$((i + 1))+ continue+ fi+ fi+ # An unquoted value, which runs on past a U+2028 or U+2029 up to a `#`.+ i=$((i + 1))+ while [ "$i" -le "$n" ] && [[ $v != *'#'* ]] && [[ ${lines[i]} == "$ls"* ]]; do+ v="${lines[i]}"+ i=$((i + 1))+ done+ done+)# openwa_env_file_value <file> <key> - print the value from one env-file layer. Returns 1 when the layer# does not set the key and 2 when it may set it in a form rejected below; a blank value succeeds and# prints nothing.openwa_env_file_value() {- local file="$1" key="$2" line match value n next- local keyline="^[[:space:]]*(export[[:space:]]+)?${key}[[:space:]]*([=:]|\$)"+ local file="$1" key="$2" line match value n next prev rest tool q o closes bad=""+ # The trims below take only ASCII blanks whatever the operator's locale, so a Unicode blank after a+ # closing quote stops the run on every host.+ local LC_ALL=C# The code points beyond ASCII that dotenv's \s matches: the Unicode blanks and the byte-order mark.local blank=$'\xc2\xa0|\xe1\x9a\x80|\xe2\x80[\x80-\x8a\xa8\xa9\xaf]|\xe2\x81\x9f|\xe3\x80\x80|\xef\xbb\xbf'+ local s="([[:space:]]|$blank)"+ # A line dotenv or Docker Compose may read as setting the key: `KEY=`, `KEY:` or a bare `KEY`.+ # dotenv skips `KEY:value`, but Compose interpolates ./.env into the container and reads it as a+ # setting, so that line stops the run too.+ local keyline="^$s*(export$s+)?${key}$s*([=:]|\$)"# A line of well-formed UTF-8, without NUL.local utf8=$'([\x01-\x7f]|[\xc2-\xdf][\x80-\xbf]|\xe0[\xa0-\xbf][\x80-\xbf]|[\xe1-\xec\xee\xef][\x80-\xbf]{2}|\xed[\x80-\x9f][\x80-\xbf]|\xf0[\x90-\xbf][\x80-\xbf]{2}|[\xf1-\xf3][\x80-\xbf]{3}|\xf4[\x80-\x8f][\x80-\xbf]{2})*'[ -f "$file" ] || return 1- # dotenv also takes a byte-order mark or a Unicode blank (a no-break space, say) for whitespace and a- # bare CR, U+2028 or U+2029 for a line break, and decodes the file as UTF-8, so a byte that is not UTF-8- # reaches the app as U+FFFD. grep and the trims below do none of that, so the key could be missed or- # read with other bytes. Such a line break on any line naming the key, comments included, is rejected,- # and so is a line setting the key with a blank before the key, around its `=` or at either end of an- # unquoted value, or with a NUL or a byte that is not UTF-8 anywhere. Every grep here runs with -a in- # the C locale: otherwise a NUL anywhere in the file, or a byte that is not UTF-8 in a UTF-8 locale, can- # hide the key's line. A NUL, which bash would drop from the line, becomes \377.- line="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C grep -aE "(^|[^A-Za-z0-9_.-])${key}([^A-Za-z0-9_.-]|\$)")" || true- if ! LC_ALL=C grep -qE $'\r.|\xe2\x80[\xa8\xa9]' <<<"$line"; then- line="$(LC_ALL=C grep -aE "^([[:space:]]|$blank)*(export([[:space:]]|$blank)+)?${key}([[:space:]]|$blank)*([=:]|\$)" <<<"$line")" || true- [ -z "$line" ] || LC_ALL=C grep -qvE "$keyline" <<<"$line" ||- LC_ALL=C grep -qE "^[^=:]*[=:][[:space:]]*($blank)|^[^#]*($blank)[[:space:]]*(#|\$)" <<<"$line" ||- LC_ALL=C grep -qvxE "$utf8" <<<"$line" || line=""- fi- if [ -n "$line" ]; then- echo "[config] ERROR: $file has a bare CR, U+2028 or U+2029 on a line naming $key, a NUL or a byte that is" >&2- echo "[config] not UTF-8 on a line setting it, or a byte-order mark or Unicode blank next to $key, its =" >&2- echo "[config] or the ends of its value. Save it as UTF-8 with plain ASCII blanks and LF or CRLF line" >&2- echo "[config] ends, or pass $key in the environment." >&2- return 2- fi- # dotenv reads the line after a bare `NAME:` line as NAME's value, so a line for the key right there- # does not set the key for the app, although the scripts would read it. dotenv also starts a line- # after a bare CR, U+2028 or U+2029, so `# note<U+2028>NAME:` is such a line too.- line="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C grep -aB1 -E "$keyline")" || true- if LC_ALL=C grep -qE "(^|"$'\r|\xe2\x80[\xa8\xa9]'")([[:space:]]|$blank)*(export([[:space:]]|$blank)+)?[A-Za-z0-9_.-]+:"$'\r'"?\$" <<<"$line"; then- echo "[config] ERROR: $file has a bare NAME: line just before a line setting $key, which the app reads as" >&2- echo "[config] NAME's value instead. Give NAME: a value or remove it, or pass $key in the environment." >&2+ # Without these the key's line would read as absent and the lookup would fall through to a default.+ for tool in tr tail sed grep; do+ command -v "$tool" >/dev/null 2>&1 && continue+ echo "[config] ERROR: $tool is required to read $file; install it or pass $key in the environment." >&2return 2- fi- # The last line naming the key wins, as in dotenv. `KEY: value` is matched only so that it stops the- # run. dotenv skips a bare `KEY` line (`export KEY`, say), unless the next line that is not blank- # starts with `=`: its `KEY\s*=` crosses line breaks, so that pair sets the key and stops the run too.+ done+ # dotenv takes a bare CR for a line break and can start a setting right after U+2028 or U+2029 (after+ # a comment or a quoted value, say), so a line grep sees can hold the key's setting in its middle.+ # Such a character on any line naming the key, comments included, is rejected. Every grep here runs+ # with -a in the C locale: otherwise a NUL anywhere in the file, or a byte that is not UTF-8 in a+ # UTF-8 locale, can hide the key's line. A NUL, which bash would drop from the line, becomes \377.+ line="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C grep -aE "(^|[^A-Za-z0-9_.-])${key}([^A-Za-z0-9_.-]|\$)")" || true+ if LC_ALL=C grep -qE $'\r.|\xe2\x80[\xa8\xa9]' <<<"$line"; then bad=1; fi+ # The last line setting the key wins, as in dotenv, so only that one is checked below. `KEY: value`+ # and `KEY:value` are matched only so that they stop the run. dotenv skips a bare `KEY` line (`export+ # KEY`, say), unless the next line that is not blank starts with `=`: its `KEY\s*=` crosses line+ # breaks, so that pair sets the key and stops the run too.line=""while IFS= read -r match; dovalue="${match#*"$key"}"- if [ -z "${value//[[:space:]]/}" ]; then+ if LC_ALL=C grep -qxE "$s*" <<<"$value"; thennext="$(LC_ALL=C tail -n +"$((${match%%:*} + 1))" "$file" | LC_ALL=C tr '\000' '\377' |- LC_ALL=C grep -avE -m 1 "^([[:space:]]|$blank)*\$")" || true- LC_ALL=C grep -qE "^([[:space:]]|$blank)*=" <<<"$next" || continue+ LC_ALL=C grep -avE -m 1 "^$s*\$")" || true+ LC_ALL=C grep -qE "^$s*=" <<<"$next" || continuefiline="$match"- done < <(LC_ALL=C grep -anE "$keyline" "$file" 2>/dev/null)- [ -n "$line" ] || return 1+ done < <(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C grep -anE "$keyline")+ [ -n "$line$bad" ] || return 1n="${line%%:*}"line="${line#*:}"+ # dotenv also takes a byte-order mark or a Unicode blank (a no-break space, say) for whitespace and+ # decodes the file as UTF-8, so a byte that is not UTF-8 reaches the app as U+FFFD. grep and the trims+ # below do neither, so the line is rejected with such a blank before the key, around its `=`, or at+ # either end of an unquoted value (a quoted value keeps its blanks), or with a NUL or a byte that is+ # not UTF-8 anywhere.+ if [ -n "$bad" ] || ! LC_ALL=C grep -qE "^[[:space:]]*(export[[:space:]]+)?${key}[[:space:]]*([=:]|\$)" <<<"$line" ||+ LC_ALL=C grep -qE "^[^=:]*[=:][[:space:]]*($blank)|^[^=:]*[=:][[:space:]]*([^\"'\`[:space:]#][^#]*)?($blank)[[:space:]]*(#|\$)" <<<"$line" ||+ LC_ALL=C grep -qvxE "$utf8" <<<"$line"; then+ echo "[config] ERROR: $file has a bare CR, U+2028 or U+2029 on a line naming $key, a NUL or a byte that is" >&2+ echo "[config] not UTF-8 on the line setting it, or a byte-order mark or Unicode blank next to $key, its =" >&2+ echo "[config] or the ends of its unquoted value. Save it as UTF-8 with plain ASCII blanks and LF or CRLF" >&2+ echo "[config] line ends, or pass $key in the environment." >&2+ return 2+ fi+ # dotenv reads the line after a bare `NAME:` line as NAME's value, so the key's line right there does+ # not set the key for the app, although the scripts would read it. dotenv can also start a setting+ # after a bare CR, U+2028 or U+2029, so `# note<U+2028>NAME:` is such a line too.+ prev=""+ [ "$n" -le 1 ] || prev="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C sed -n "$((n - 1))p")"+ if LC_ALL=C grep -qE "(^|"$'\r|\xe2\x80[\xa8\xa9]'")$s*(export$s+)?[A-Za-z0-9_.-]+:"$'\r'"?\$" <<<"$prev"; then+ echo "[config] ERROR: $file has a bare NAME: line just before a line setting $key, which the app reads as" >&2+ echo "[config] NAME's value instead. Give NAME: a value or remove it, or pass $key in the environment." >&2+ return 2+ fi+ # dotenv reads a quoted value on across lines, so a line inside one does not set the key. The value+ # ends at the first quote without a backslash before it or, when more than blanks or a comment follow+ # that quote on its line, at the last quote before it that only blanks or a comment follow. So the+ # line is rejected when, for some quote character, such a quote sits on it or after it, up to the+ # first quote without a backslash, and an earlier line opens a value with that quote which no such+ # quote closes before the line (openwa_env_opener). Lines are split where dotenv splits them+ # (openwa_env_lines).+ rest="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C tail -n +"$n" | openwa_env_lines)"+ closes=""+ for q in "'" '"' '`'; do+ next="$rest"+ o="$(LC_ALL=C grep -anE -m 1 "(^|[^\\\\])$q" <<<"$next")" || true+ [ -z "$o" ] || next="$(LC_ALL=C sed -n "1,${o%%:*}p" <<<"$next")"+ ! LC_ALL=C grep -aqE "^([^$q]|\\\\$q)*$q$s*(#.*)?\$" <<<"$next" || closes="$closes$q"+ done+ o=""+ if [ "$n" -gt 1 ] && [ -n "$closes" ]; then+ prev="$(LC_ALL=C tr '\000' '\377' 2>/dev/null <"$file" | LC_ALL=C sed -n "1,$((n - 1))p" | openwa_env_lines)"+ o="$(openwa_env_opener "$s" "$closes" <<<"$prev")"+ fi+ if [ -n "$o" ]; then+ # Count only the lines the file has, not the ones openwa_env_lines added. A file line that starts with+ # a CR or U+2028 still counts: openwa_env_lines prints the empty text before that character first.+ o="$(LC_ALL=C sed -n "1,${o}p" <<<"$prev" | LC_ALL=C grep -acvE "^("$'\r|\xe2\x80\xa8'")")" || true+ echo "[config] ERROR: $file opens a quoted value on line $o that the app can read on to line $n, which sets" >&2+ echo "[config] $key, as part of that value. Close the quote on the line that opens it, or pass $key in" >&2+ echo "[config] the environment." >&2+ return 2+ fivalue="${line#*"$key"}"value="${value#"${value%%[![:space:]]*}"}"case "$value" in@@ -103,11 +304,23 @@ openwa_env_file_value() {value="${value%"${value##*[![:space:]]}"}"case "$value" in\"*\\*) ;; # dotenv expands \n and \r inside double quotes- \'*\\\' | \`*\\\`) ;; # dotenv may take the backslash and quote for an escaped quote and read on\"*\" | \'*\' | \`*\`)# A quote inside means the pair does not wrap the whole value: a comment ending in one follows it.case "${value:1:${#value}-2}" in*"${value:0:1}"*) ;;+ *\\)+ # dotenv may take the backslash and quote for an escaped quote and read on, up to the first+ # quote without a backslash, to a quote that only blanks or a comment follow on its line, so+ # the value ends here only when no later line has such a quote.+ q="${value:0:1}"+ next="$(LC_ALL=C sed 1d <<<"$rest")"+ o="$(LC_ALL=C grep -anE -m 1 "(^|[^\\\\])$q" <<<"$next")" || true+ [ -z "$o" ] || next="$(LC_ALL=C sed -n "1,${o%%:*}p" <<<"$next")"+ if ! LC_ALL=C grep -aqE "^([^$q]|\\\\$q)*$q$s*(#.*)?\$" <<<"$next"; then+ printf '%s' "${value:1:${#value}-2}"+ return 0+ fi+ ;;*)printf '%s' "${value:1:${#value}-2}"return 0@@ -138,7 +351,8 @@ openwa_env_file_value() {esacecho "[config] ERROR: $file sets $key in a form these scripts do not parse: a quoted value followed by a" >&2echo "[config] comment or not closed on its line, a quoted value containing its own quote character or" >&2- echo "[config] ending in a backslash, a double-quoted value with a backslash, or KEY: value." >&2+ echo "[config] ending in a backslash the app can read on past to a later quote, a double-quoted value" >&2+ echo "[config] with a backslash, or KEY: value." >&2echo "[config] Move a comment to its own line and keep the quotes, close the quote on the same line, wrap a" >&2echo "[config] value in a quote character it does not contain, single-quote a value whose backslashes are" >&2printf '%s\n' "[config] literal and not at its end (inside double quotes the app turns \\n and \\r into line breaks)," >&2diff --git a/scripts/smoke-test-backup-restore.sh b/scripts/smoke-test-backup-restore.shindex 684fdcc44..a5e245f39 100755--- a/scripts/smoke-test-backup-restore.sh+++ b/scripts/smoke-test-backup-restore.sh@@ -107,7 +107,7 @@ db_fingerprint() {# to shadow tar (simulating an incomplete archive) without touching the real scripts.populate_shim() {shim_dir="$1"- tools="env bash sh cp tar gzip mktemp date rm sed mkdir ls cat chmod grep printf uname dirname"+ tools="env bash sh cp tar gzip mktemp date rm sed mkdir ls cat chmod grep printf uname dirname tr tail"if [ "${2:-}" = "with-sqlite3" ]; thentools="$tools sqlite3"fi@@ -1562,6 +1562,10 @@ doneif ! grep -q 'sets ENGINE_TYPE in a form' <<<"$OUT_AG" || ! grep -q 'skipping only the check for missing Baileys state' <<<"$OUT_AG"; thenfail "(ag) backup did not report the ENGINE_TYPE line it cannot parse and that it carried on: $OUT_AG"fi+# The run carries on, so the line is reported as a warning rather than an error.+if grep -q 'ERROR' <<<"$OUT_AG" || ! grep -q '^\[config\] WARN: .* sets ENGINE_TYPE in a form' <<<"$OUT_AG"; then+ fail "(ag) backup reported the ENGINE_TYPE line it carries on past as an error: $OUT_AG"+firm -f "$AGB/data/act"$'\343\200\200'"ive.sqlite"# dotenv reads the line after a bare `NAME:` line as NAME's value, so the app never sets the key from it.printf 'LOG_LEVEL:\nDATABASE_NAME=./data/active.sqlite\n' >"$AGB/.env"@@ -1591,20 +1595,157 @@ for check in 'DATABASE_NAME\n=./data/active.sqlite|names DATABASE_NAME on a linefidone# dotenv can take the backslash and quote that end a quoted value for an escaped quote, and read on to a-# quote on a later line.-printf '%s\n' "DATABASE_NAME='./data/active.sqlite\\'" "'" >"$AGB/.env"-set +e-OUT_AG="$(cd "$AGB" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"-RC_AG=$?-set -e-if [ "$RC_AG" -eq 0 ] || ! grep -q 'ending in a backslash' <<<"$OUT_AG"; then- fail "(ag) a quoted DATABASE_NAME ending in a backslash did not fail the lookup (rc $RC_AG): $OUT_AG"-fi+# later quote that only blanks or a comment follow, escaped or not.+for ag_next in "'" "x\\' # note"; do+ printf '%s\n' "DATABASE_NAME='./data/active.sqlite\\'" "$ag_next" >"$AGB/.env"+ set +e+ OUT_AG="$(cd "$AGB" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"+ RC_AG=$?+ set -e+ if [ "$RC_AG" -eq 0 ] || ! grep -q 'ending in a backslash' <<<"$OUT_AG"; then+ fail "(ag) a quoted DATABASE_NAME ending in a backslash before '$ag_next' did not fail the lookup (rc $RC_AG): $OUT_AG"+ fi+done# An empty `KEY=` followed by blank lines and an unquoted line stays empty for dotenv too.printf 'DATABASE_NAME=\n\n# data store\nLOG_LEVEL=info\n' >"$AGB/.env"if [ "$(cd "$AGB" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT)" != DEFAULT ]; thenfail "(ag) an empty DATABASE_NAME= followed by a comment and another key did not resolve to the default"fi+# dotenv reads each of these unambiguously, and the scripts used to stop on them: a Unicode blank inside+# a quoted value; a Unicode blank, a byte that is not UTF-8 or a bare NAME: line before it on an earlier+# line for the key, which the last one overrides; and a value single-quoted for its `#` and ending in a+# backslash, as the dashboard writes it, with no single quote on a later line or only one followed by+# more than a comment. An empty value followed by a comment holding a Unicode blank is still empty, so+# the default applies. The key's line also sets it after a quoted value that closes on its own lines,+# even on a line that looks like it opens one, or that nothing closes, which dotenv then reads without+# its quote, and after one closed by an escaped quote whose later quote is followed by more than a+# comment. A line that would open a quoted value but is the value of a bare NAME: line before it+# opens nothing, and neither does a U+2028 inside an unquoted value start a line. Nor does a quote+# inside a value opened with another quote character open a value, even after a U+2029 there, or a+# quote in an unquoted value, which runs on past a U+2028. Fields: ./.env, .env.generated, the key and+# the value the app reads, all with printf escapes.+AGC="$WORK/ag-last-line"+mkdir -p "$AGC/data"+for check in 'DATABASE_NAME=\047a\302\240#b\047||DATABASE_NAME|a\302\240#b' \+ 'LOG_LEVEL:\nDATABASE_NAME=a\nDATABASE_NAME=b||DATABASE_NAME|b' \+ 'DATABASE_NAME=a\302\240\nDATABASE_NAME=b||DATABASE_NAME|b' \+ 'DATABASE_NAME=a # donn\351es\nDATABASE_NAME=b||DATABASE_NAME|b' \+ 'DATABASE_NAME= # note\302\240||DATABASE_NAME|DEFAULT' \+ 'DATABASE_NAME=#a\302\240#b\nLOG_LEVEL=info||DATABASE_NAME|DEFAULT' \+ '|DATABASE_PASSWORD=\047p#w\134\047\nDATABASE_HOST="db"|DATABASE_PASSWORD|p#w\134' \+ '|DATABASE_PASSWORD=\047p#w\134\047\nREDIS_PASSWORD=\047x#y\047|DATABASE_PASSWORD|p#w\134' \+ 'N:\nB=\047y\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b' \+ 'M:\nN=\n\047x\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b' \+ 'A=1\342\200\250B=\047y\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b' \+ '|DATABASE_PASSWORD=\047pw\302\240#x\047|DATABASE_PASSWORD|pw\302\240#x' \+ 'CERT="BEGIN\nabc\nEND"\nDATABASE_NAME=b||DATABASE_NAME|b' \+ 'LOG_LEVEL=\047x\nDATABASE_NAME=b||DATABASE_NAME|b' \+ 'LOG_LEVEL=\047a\134\047\nDATABASE_NAME=b\nY=\047c\047||DATABASE_NAME|b' \+ 'CERT=\047a\nNOTE=\047 # c\nDATABASE_NAME=b\nY=x\047||DATABASE_NAME|b' \+ 'A=\047x\nB="y\n\047\nDATABASE_NAME=b\nC=z"||DATABASE_NAME|b' \+ 'A=\047x\342\200\251B="y\047 # c\nDATABASE_NAME=b\nC=z"||DATABASE_NAME|b' \+ 'A=x\047y\342\200\250B=\047z\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b' \+ 'N:\nx\047\342\200\250B=\047z\nDATABASE_NAME=b\nz\047||DATABASE_NAME|b'; do+ IFS='|' read -r ag_env ag_gen ag_key ag_want <<<"$check"+ # shellcheck disable=SC2059 # the fields' escapes are the point+ printf "$ag_env\n" >"$AGC/.env"+ # shellcheck disable=SC2059+ printf "$ag_gen\n" >"$AGC/data/.env.generated"+ # shellcheck disable=SC2059+ ag_want="$(printf "$ag_want")"+ set +e+ got="$(cd "$AGC" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve "$ag_key" DEFAULT 2>&1)"+ rc=$?+ set -e+ if [ "$rc" -ne 0 ] || [ "$got" != "$ag_want" ]; then+ fail "(ag) $ag_key in '$ag_env' / '$ag_gen' resolved to '$got' (rc $rc) instead of the app's '$ag_want'"+ fi+done+# dotenv reads the line after a bare `DATABASE_NAME:` as its value, and Docker Compose, which+# interpolates ./.env into the container, reads `KEY:value` and `KEY :value` as settings, so each stops+# the run even when .env.generated sets the key. The error names the `KEY: value` form, not a bare+# NAME: line before the key's line.+printf 'DATABASE_NAME=./data/gen\n' >"$AGC/data/.env.generated"+for ag_env in 'DATABASE_NAME:\n./data/active.sqlite' 'DATABASE_NAME:./data/a.sqlite' \+ 'DATABASE_NAME=./data/b.sqlite\nDATABASE_NAME :./data/a.sqlite'; do+ # shellcheck disable=SC2059 # the escapes are the point+ printf "$ag_env\n" >"$AGC/.env"+ set +e+ OUT_AG="$(cd "$AGC" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"+ RC_AG=$?+ set -e+ if [ "$RC_AG" -ne 2 ] || ! grep -q 'sets DATABASE_NAME in a form' <<<"$OUT_AG" || grep -q 'bare NAME:' <<<"$OUT_AG"; then+ fail "(ag) '$ag_env' was not reported as KEY: value (rc $RC_AG): $OUT_AG"+ fi+done+# dotenv reads a quoted value on to its closing quote, so a line for the key inside it does not set the+# key. Whether the quote opens on an earlier line for the key or another key's, at the start of a line+# after an empty NAME=, after a bare NAME: line that is itself the value of the one before it, or after+# a bare CR, U+2028 or U+2029, where dotenv starts a line, and whether it closes before one of those,+# the run stops instead of reading the key's last line or a line inside the value. A quote followed by+# a colon opens a value too.+for ag_env in 'DATABASE_NAME=\047a\302\240\nDATABASE_NAME=./data/a.sqlite\047' \+ 'DATABASE_NAME=\047x\nDATABASE_NAME=\377a\nDATABASE_NAME=./data/a.sqlite\047' \+ 'LOG_LEVEL=\047x\nDATABASE_NAME=a\302\240\nDATABASE_NAME=./data/a.sqlite\nY\047' \+ 'LOG_LEVEL=\047x\nLOG:\nDATABASE_NAME=a\nDATABASE_NAME=./data/a.sqlite\nY\047' \+ 'DATABASE_NAME=\047a\nDATABASE_NAME=./data/a.sqlite\047' \+ 'LOG_LEVEL=\n\n"x\nDATABASE_NAME=./data/a.sqlite" # note' \+ 'W:\nN:\nB=\047y\nDATABASE_NAME=./data/a.sqlite\nz\047' \+ 'x\rA=\047y\nDATABASE_NAME=./data/a.sqlite\nz\047' \+ '# n\342\200\250A=\047y\nDATABASE_NAME=./data/a.sqlite\nz\047' \+ 'A=\047x\nDATABASE_NAME=./data/a.sqlite\nz\047\342\200\251more' \+ 'A=\047x\nDATABASE_NAME=./data/a.sqlite\nz\047\rmore' \+ 'A=\047x\nB=\047:\nDATABASE_NAME=./data/a.sqlite\n\047'; do+ # shellcheck disable=SC2059 # the escapes are the point+ printf "$ag_env\n" >"$AGC/.env"+ set +e+ OUT_AG="$(cd "$AGC" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"+ RC_AG=$?+ set -e+ if [ "$RC_AG" -ne 2 ] || ! grep -q 'opens a quoted value on line' <<<"$OUT_AG"; then+ fail "(ag) DATABASE_NAME inside the quoted value in '$ag_env' resolved to '$OUT_AG' (rc $RC_AG) instead of stopping"+ fi+done+# The error names the file's line that opens the value, not one counted after a line break dotenv adds,+# and still counts a line of the file that starts with U+2029.+printf 'Q=1\n\342\200\251# n\nx\ry\342\200\250A=\047v\nDATABASE_NAME=./data/a.sqlite\nz\047\n' >"$AGC/.env"+set +e+OUT_AG="$(cd "$AGC" && DATA_DIR=./data && . "$REPO_ROOT/scripts/lib-env.sh" && openwa_resolve DATABASE_NAME DEFAULT 2>&1)"+set -e+if ! grep -q 'opens a quoted value on line 3 that the app can read on to line 4,' <<<"$OUT_AG"; then+ fail "(ag) the quoted value opened after a bare CR and U+2028 was not reported on line 3: $OUT_AG"+fi+# The scripts trim only ASCII blanks, so a no-break space after a closing quote stops the run in a+# UTF-8 locale too, as it does in the C locale.+ag_loc="$(locale -a 2>/dev/null | grep -m 1 -ixE 'C\.UTF-?8|en_US\.UTF-?8')" || true+if [ -n "$ag_loc" ]; then+ printf 'DATABASE_NAME=\047\047 \302\240\n' >"$AGC/.env"+ set +e+ # shellcheck disable=SC2016 # expanded by the inner shell+ OUT_AG="$(cd "$AGC" && LC_ALL="$ag_loc" bash -c 'DATA_DIR=./data && . "$1" && openwa_resolve DATABASE_NAME DEFAULT' _ "$REPO_ROOT/scripts/lib-env.sh" 2>&1)"+ RC_AG=$?+ set -e+ if [ "$RC_AG" -ne 2 ] || ! grep -q 'sets DATABASE_NAME in a form' <<<"$OUT_AG"; then+ fail "(ag) a no-break space after a closing quote resolved to '$OUT_AG' (rc $RC_AG) under LC_ALL=$ag_loc"+ fi+fi+# Without tr, tail, sed or grep the key's line cannot be read, so the lookup stops instead of taking+# the default.+printf 'DATABASE_SSL=true\n' >"$AGC/.env"+for ag_tool in tr tail sed grep; do+ rm -rf "$AGC/shim"+ mkdir -p "$AGC/shim"+ populate_shim "$AGC/shim"+ rm -f "$AGC/shim/$ag_tool"+ set +e+ # shellcheck disable=SC2016 # expanded by the inner shell+ OUT_AG="$(cd "$AGC" && PATH="$AGC/shim" bash -c 'DATA_DIR=./data && . "$1" && openwa_resolve DATABASE_SSL false' _ "$REPO_ROOT/scripts/lib-env.sh" 2>&1)"+ RC_AG=$?+ set -e+ if [ "$RC_AG" -ne 2 ] || ! grep -q "$ag_tool is required" <<<"$OUT_AG"; then+ fail "(ag) without $ag_tool DATABASE_SSL resolved to '$OUT_AG' (rc $RC_AG) instead of stopping"+ fi+done# A quoted value holding its own quote character is named as such, and the remedy warns that double# quotes expand \n and \r. The other quote style, which the error suggests, archives the configured store.make_fixture "$AGB/data/o'brien.sqlite" "ag-active"