Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

fix(stepfunctions): bound $range allocation at the AWS memory limit (#2790)

Facundo Paredesf7f16cd2a3e7
diff --git a/docs/services/step-functions.md b/docs/services/step-functions.md
index 5de3e6b94f..df7a299ddc 100644
--- a/docs/services/step-functions.md
+++ b/docs/services/step-functions.md
@@ -189,7 +189,7 @@ top of the JSONata language, alongside every function JSONata itself provides.
| Function | Returns |
| --- | --- |
-| `$parse(jsonString)` | the deserialized value; the replacement for `$eval`, which AWS disables |
+| `$parse(jsonString)` | the deserialized value; the replacement for `$eval`, which AWS disables and so does Floci, answering `T1006` to a call |
| `$partition(array, chunkSize)` | `array` split into chunks of `chunkSize`, the last one holding the remainder |
| `$range(start, end, step)` | the values from `start` to `end`, inclusive when `step` lands on `end` |
| `$hash(str, algorithm)` | the hex digest of `str`; `algorithm` is `MD5`, `SHA-1`, `SHA-256`, `SHA-384` or `SHA-512`, case-sensitive |
@@ -210,16 +210,47 @@ JSONata's own `$string` follows AWS's number notation: a whole number is written
`1e21` and in exponent notation from there, on both signs, so `$string(1e20)` is
`100000000000000000000` and `$string(1e21)` is `1e+21`.
-Evaluation is bounded, as it is on AWS: one expression may run for five seconds and nest 500
-levels deep, and past either the state fails with `States.QueryEvaluationError`. Both bounds sit
-well above what AWS itself accepts, so an expression that evaluates there evaluates here. AWS
-refuses a non-tail-recursive function past a nesting depth near 100, and the largest sequence it
-accepts evaluates here in about a tenth of a second.
-
-One deviation. AWS also bounds the *memory* an expression may use, refusing
-`[1..900000] ~> $count()` with `Expression evaluation memory limit exceeded` while accepting the
-same expression at 800,000 elements. Floci bounds time and depth but not memory, and its ranges
-are lazy, so that expression answers immediately at any size.
+The execution input reaching `$states.input` follows the same number model as `$parse`: an integer
+stays exact while it fits in a `long` and switches to a `double` past that boundary, matching AWS.
+
+JSONata's own `$formatNumber` checks its picture string against the fourteen rules of XPath F&O
+4.7.3, as AWS does, so `$formatNumber(1, "x")` fails with `D3086` rather than answering `x1`: a
+picture with no digit in it describes no number.
+
+Evaluation is bounded on three axes, as it is on AWS, and past any of them the state fails with
+`States.QueryEvaluationError`.
+
+- **Depth**: one expression may nest 100 levels, which is AWS's own ceiling: AWS accepts `1+1+…+1`
+ at 100 terms, 99 parentheses, 99 brackets, 99 `~>` stages, and a non-tail-recursive `$f(31)`,
+ which nests `3n+5`, and refuses one level more of each. The refusal names the depth reached, as
+ in `Stack overflow error: … Depth=101 max=100`.
+- **Memory**: one value an expression builds may hold 6,990,256 bytes, counting a number as eight
+ bytes and a character as one. That is AWS's own bound: AWS accepts `[1..873782]` and refuses one
+ element more, and it accepts a string doubled 22 times, 2^22 characters, refusing the 23rd. The
+ refusal is AWS's own
+ `Expression evaluation memory limit exceeded`, which is what `[1..900000] ~> $count()` and
+ `$sum([1..900000])` now answer. `$range` holds a tighter bound of its own, checked before it
+ allocates rather than on the array it would have built: AWS accepts `$range(1, 360145, 1)`, all
+ 360,145 elements, and refuses one element more with the same refusal. A lazy literal range such
+ as `[1..873782]` is exempt from both bounds on AWS, and stays exempt here.
+- **Time**: five seconds, which is the library's own default and roughly fifty times the slowest
+ evaluation of a payload AWS itself accepts. A recursive expression with no base case is a tail
+ call, so it loops rather than nesting and only the clock ends it.
+
+JSON has no literal for a non-finite number, so AWS writes each one as the string JavaScript names
+it by, wherever it lands: `1/0` is `"Infinity"`, `1e308 * 10` is `"Infinity"`, `0/0` is `"NaN"` and
+`$parseInteger("abc", "0")` is `"NaN"`, and nested, `[1/0]` is `["Infinity"]` and `{"k": 1/0}` is
+`{"k": "Infinity"}`.
+
+One deviation, on the arithmetic half of that. A non-finite number a **function** answers is a value
+like any other here, so `$parseInteger("abc", "0")` is `"NaN"` on its own, inside an array and
+inside an object, as on AWS. One the **arithmetic** produces is not: the JSONata library refuses to
+carry it and raises instead, so Floci sees the value only in that refusal, after it has unwound
+whatever was being built around it. `1/0`, `-1/0` and `1e308 * 10` are answered because the
+expression's own result is what was refused; `[1/0]`, `{"k": 1/0}` and `$string(1/0)` fail the state
+with `States.QueryEvaluationError` where AWS answers a string, and `0/0` fails the field that holds
+it, because NaN is dropped as not-a-number without even a refusal to read it from. A state that
+fails is one a `Catch` fires on, which is the half of the divergence worth keeping.
## Nested workflows
diff --git a/src/main/java/io/github/hectorvent/floci/services/stepfunctions/FormatNumberPicture.java b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/FormatNumberPicture.java
new file mode 100644
index 0000000000..e8b4f6943a
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/FormatNumberPicture.java
@@ -0,0 +1,238 @@
+package io.github.hectorvent.floci.services.stepfunctions;
+
+import com.dashjoin.jsonata.JException;
+
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/**
+ * The picture string of {@code $formatNumber}, checked against the rules XPath F&O 4.7.3 gives
+ * it, which is what AWS applies and the bundled JSONata library does not: it formats through
+ * {@link java.text.DecimalFormat} and answers {@code "x1"} for {@code $formatNumber(1, "x")} where
+ * AWS fails the state with D3086.
+ *
+ * <p>A picture may break more than one of the fourteen rules, and the code reported is the last
+ * one, which is the one AWS reports: a picture with no digit at all breaks two at once, the
+ * mantissa holding no digit (D3085) and the active part holding a passive character (D3086), and
+ * D3086 is the answer.
+ *
+ * <p>The symbols the rules are written in terms of are the built-in ones, each replaceable through
+ * the third argument of {@code $formatNumber}.
+ */
+final class FormatNumberPicture {
+
+ private final char decimalSeparator;
+ private final char exponentSeparator;
+ private final char groupingSeparator;
+ private final char digit;
+ private final char patternSeparator;
+ private final char percent;
+ private final char perMille;
+ private final char zeroDigit;
+
+ private FormatNumberPicture(Map<?, ?> options) {
+ decimalSeparator = symbol(options, "decimal-separator", '.');
+ exponentSeparator = symbol(options, "exponent-separator", 'e');
+ groupingSeparator = symbol(options, "grouping-separator", ',');
+ digit = symbol(options, "digit", '#');
+ patternSeparator = symbol(options, "pattern-separator", ';');
+ percent = symbol(options, "percent", '%');
+ perMille = symbol(options, "per-mille", '‰');
+ zeroDigit = symbol(options, "zero-digit", '0');
+ }
+
+ private static char symbol(Map<?, ?> options, String name, char builtIn) {
+ Object override = options == null ? null : options.get(name);
+ return override instanceof String text && text.length() == 1 ? text.charAt(0) : builtIn;
+ }
+
+ /**
+ * Fails with the code of the rule the picture breaks, or returns when it breaks none. The
+ * picture is split into sub-pictures first, one for positive numbers and an optional second
+ * one for negative numbers.
+ */
+ static void validate(String picture, Map<?, ?> options) {
+ FormatNumberPicture symbols = new FormatNumberPicture(options);
+ String[] subPictures =
+ picture.split(Pattern.quote(String.valueOf(symbols.patternSeparator)), -1);
+ if (subPictures.length > 2) {
+ throw new JException("D3080", -1);
+ }
+ for (String subPicture : subPictures) {
+ String error = symbols.errorIn(symbols.split(subPicture));
+ if (error != null) {
+ throw new JException(error, -1);
+ }
+ }
+ }
+
+ /**
+ * The parts a sub-picture is read in. The prefix and the suffix are the passive characters at
+ * either edge, so the active part is what sits between them; the mantissa is the active part
+ * up to the exponent separator, and the integer and fractional parts are the mantissa either
+ * side of the decimal separator.
+ */
+ private record Parts(String subPicture, String activePart, String mantissaPart,
+ String integerPart, String fractionalPart, String exponentPart) { }
+
+ private Parts split(String subPicture) {
+ int activeStart = 0;
+ while (activeStart < subPicture.length() && !isActiveOutsideExponent(subPicture.charAt(activeStart))) {
+ activeStart++;
+ }
+ int activeEnd = subPicture.length();
+ while (activeEnd > 0 && !isActiveOutsideExponent(subPicture.charAt(activeEnd - 1))) {
+ activeEnd--;
+ }
+ // A sub-picture with no active character at all is all active part and has no edges.
+ if (activeEnd == 0) {
+ activeStart = 0;
+ activeEnd = subPicture.length();
+ }
+ String activePart = subPicture.substring(activeStart, activeEnd);
+ int exponentAt = subPicture.indexOf(exponentSeparator, activeStart) - activeStart;
+ boolean hasExponent = exponentAt >= 0 && exponentAt <= activePart.length();
+ String mantissaPart = hasExponent ? upTo(activePart, exponentAt) : activePart;
+ String exponentPart = hasExponent ? from(activePart, exponentAt + 1) : null;
+ int decimalAt = mantissaPart.indexOf(decimalSeparator);
+ String integerPart = decimalAt < 0 ? mantissaPart : mantissaPart.substring(0, decimalAt);
+ String fractionalPart =
+ decimalAt < 0 ? subPicture.substring(activeEnd) : mantissaPart.substring(decimalAt + 1);
+ return new Parts(subPicture, activePart, mantissaPart, integerPart, fractionalPart, exponentPart);
+ }
+
+ /** The exponent separator is located in the sub-picture, so it can point past the active part. */
+ private static String upTo(String text, int end) {
+ return text.substring(0, Math.min(end, text.length()));
+ }
+
+ private static String from(String text, int start) {
+ return text.substring(Math.min(start, text.length()));
+ }
+
+ /**
+ * The code of the last rule the sub-picture breaks, or null when it breaks none. The rules are
+ * read in the order the codes number them, and a later one replaces what an earlier one found.
+ */
+ private String errorIn(Parts parts) {
+ String error = separatorCountError(parts.subPicture());
+ error = later(error, digitError(parts));
+ error = later(error, groupingSeparatorError(parts));
+ error = later(error, optionalDigitError(parts));
+ error = later(error, exponentError(parts));
+ return error;
+ }
+
+ private static String later(String error, String found) {
+ return found != null ? found : error;
+ }
+
+ /** D3081 to D3084: a symbol that may appear once appears twice, or two that exclude each other. */
+ private String separatorCountError(String subPicture) {
+ String error = null;
+ if (subPicture.indexOf(decimalSeparator) != subPicture.lastIndexOf(decimalSeparator)) {
+ error = "D3081";
+ }
+ if (subPicture.indexOf(percent) != subPicture.lastIndexOf(percent)) {
+ error = "D3082";
+ }
+ if (subPicture.indexOf(perMille) != subPicture.lastIndexOf(perMille)) {
+ error = "D3083";
+ }
+ if (subPicture.indexOf(percent) >= 0 && subPicture.indexOf(perMille) >= 0) {
+ error = "D3084";
+ }
+ return error;
+ }
+
+ /** D3085 and D3086: the mantissa holds no digit, and the active part holds a passive character. */
+ private String digitError(Parts parts) {
+ String error = null;
+ if (!holdsDigit(parts.mantissaPart())) {
+ error = "D3085";
+ }
+ if (!parts.activePart().chars().allMatch(character -> isActive((char) character))) {
+ error = "D3086";
+ }
+ return error;
+ }
+
+ /** D3087 to D3089: a grouping separator next to the decimal separator, at the end, or doubled. */
+ private String groupingSeparatorError(Parts parts) {
+ String subPicture = parts.subPicture();
+ int decimalAt = subPicture.indexOf(decimalSeparator);
+ String error = null;
+ if (decimalAt >= 0 && (characterAt(subPicture, decimalAt - 1) == groupingSeparator
+ || characterAt(subPicture, decimalAt + 1) == groupingSeparator)) {
+ error = "D3087";
+ } else if (decimalAt < 0 && !parts.integerPart().isEmpty()
+ && parts.integerPart().charAt(parts.integerPart().length() - 1) == groupingSeparator) {
+ error = "D3088";
+ }
+ if (subPicture.indexOf("" + groupingSeparator + groupingSeparator) >= 0) {
+ error = "D3089";
+ }
+ return error;
+ }
+
+ /**
+ * D3090 and D3091: an optional digit with a mandatory one on the side the number grows from,
+ * which is to its left in the integer part and to its right in the fractional part.
+ */
+ private String optionalDigitError(Parts parts) {
+ String error = null;
+ int firstOptional = parts.integerPart().indexOf(digit);
+ if (firstOptional >= 0 && holdsDecimalDigit(parts.integerPart().substring(0, firstOptional))) {
+ error = "D3090";
+ }
+ int lastOptional = parts.fractionalPart().lastIndexOf(digit);
+ if (lastOptional >= 0 && holdsDecimalDigit(parts.fractionalPart().substring(lastOptional))) {
+ error = "D3091";
+ }
+ return error;
+ }
+
+ /** D3092 and D3093: an exponent alongside a percent, and an exponent that is not all digits. */
+ private String exponentError(Parts parts) {
+ if (parts.exponentPart() == null) {
+ return null;
+ }
+ boolean scaled = parts.subPicture().indexOf(percent) >= 0
+ || parts.subPicture().indexOf(perMille) >= 0;
+ String error = null;
+ if (!parts.exponentPart().isEmpty() && scaled) {
+ error = "D3092";
+ }
+ if (parts.exponentPart().isEmpty()
+ || !parts.exponentPart().chars().allMatch(character -> isDecimalDigit((char) character))) {
+ error = "D3093";
+ }
+ return error;
+ }
+
+ private static char characterAt(String text, int index) {
+ return index >= 0 && index < text.length() ? text.charAt(index) : '\0';
+ }
+
+ private boolean holdsDigit(String text) {
+ return text.chars().anyMatch(character -> isDecimalDigit((char) character) || character == digit);
+ }
+
+ private boolean holdsDecimalDigit(String text) {
+ return text.chars().anyMatch(character -> isDecimalDigit((char) character));
+ }
+
+ private boolean isDecimalDigit(char character) {
+ return character >= zeroDigit && character < zeroDigit + 10;
+ }
+
+ private boolean isActive(char character) {
+ return isDecimalDigit(character) || character == decimalSeparator
+ || character == exponentSeparator || character == groupingSeparator
+ || character == digit || character == patternSeparator;
+ }
+
+ private boolean isActiveOutsideExponent(char character) {
+ return isActive(character) && character != exponentSeparator;
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluator.java b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluator.java
index 61209dbeff..63f84d5544 100644
--- a/src/main/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluator.java
+++ b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluator.java
@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.stepfunctions;
import com.dashjoin.jsonata.Functions;
import com.dashjoin.jsonata.JException;
import com.dashjoin.jsonata.Jsonata;
+import com.dashjoin.jsonata.Utils;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.DeserializationFeature;
@@ -13,6 +14,7 @@ import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.MissingNode;
import com.fasterxml.jackson.databind.node.NullNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fasterxml.jackson.databind.node.TextNode;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
@@ -21,6 +23,7 @@ import java.math.BigInteger;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
+import java.text.ParseException;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.Iterator;
@@ -91,14 +94,54 @@ public class JsonataEvaluator {
static final long EVALUATION_TIMEOUT_MILLIS = 5_000;
/**
- * How deep evaluation may nest before the state fails. A non-tail-recursive function nests at
- * roughly {@code 3n+5} for {@code $f(n)}; measured against AWS with {@code test-state}, AWS
- * accepts {@code n=30} (depth 95) and refuses {@code n=40} (depth 125), so its own ceiling
- * sits near 100. Five times that leaves no expression AWS accepts able to reach this bound,
- * and it still trips before the JVM stack does: on a 1 MB thread stack the bound raises a
- * JSONata error while 1000 or more overflows the stack instead.
+ * How deep evaluation may nest before the state fails, which is the ceiling AWS itself holds:
+ * AWS accepts an evaluation at depth 100 and refuses one at 101, whatever the nesting is made
+ * of. An addition chain nests one level per term, parentheses, brackets and the chain operator
+ * nest one level more than their count, and a non-tail-recursive {@code $f(n)} nests
+ * {@code 3n+5}.
+ *
+ * <p>The bound trips before the JVM stack does: on a 1 MB thread stack it raises a JSONata
+ * error, which the state fails on, where 1000 or more overflows the stack instead.
+ */
+ static final int MAX_EVALUATION_DEPTH = 100;
+
+ /** What one number, boolean or null of a value counts as against {@link #MAX_EXPRESSION_BYTES}. */
+ private static final int BYTES_PER_SCALAR = 8;
+
+ /**
+ * How much memory one value an expression builds may hold before the state fails, 6,990,256
+ * bytes, which is the bound AWS itself holds: AWS accepts {@code [1..873782]} and refuses one
+ * element more, and it accepts a string of 2^22 characters and refuses one of 2^23. Both of
+ * those refusals land just past this bound when a number counts as {@link #BYTES_PER_SCALAR}
+ * bytes and a character as one.
+ *
+ * <p>What is bounded is the largest single value an evaluation step produces, not the sum of
+ * everything alive at once. Checking it walks that value and stops as soon as the walk passes
+ * the bound, so one check costs at most 873,783 steps, and a lazy range is counted from its
+ * length without being materialised.
+ */
+ static final long MAX_EXPRESSION_BYTES = 873_782L * BYTES_PER_SCALAR;
+
+ /**
+ * The largest number of elements a single {@code $range} call may produce before floci
+ * refuses it, the bound AWS itself holds specifically for {@code $range}: AWS accepts
+ * {@code $range(1, 360145, 1)}, returning all 360,145 elements, and refuses one element more
+ * with the same memory refusal as {@link #MAX_EXPRESSION_BYTES}. That general bound is looser
+ * (873,782 scalars) and is checked only on the value an evaluation step has already built; a
+ * lazy literal range such as {@code [1..873782]} is exempt from it on AWS, and stays exempt
+ * here. {@code $range} gets a bound of its own because it is the one construct that actually
+ * allocates a list of that size, so the general check would see the memory only after the
+ * allocation already happened.
+ */
+ static final int MAX_RANGE_ELEMENTS = 360_145;
+
+ /**
+ * AWS's own sentence for an expression that exceeded its evaluation memory limit, shared by
+ * the bound on a built value ({@link #boundHeldMemory}) and the pre-allocation cap on
+ * {@code $range} ({@link #range}).
*/
- static final int MAX_EVALUATION_DEPTH = 500;
+ private static final String MEMORY_LIMIT_EXCEEDED =
+ "Expression evaluation memory limit exceeded: Check for excessive memory usage";
private final ObjectMapper objectMapper;
private final ObjectReader strictJsonReader;
@@ -161,6 +204,7 @@ public class JsonataEvaluator {
JsonNode evaluate(String expression, JsonNode statesVar, JsonNode variables) {
String expr = isExpression(expression) ? unwrap(expression) : expression;
+ ExpressionNesting nesting = new ExpressionNesting();
try {
Jsonata jsonataExpr = jsonata(expr);
// Off, the library keeps JSONata's null marker in the result instead of flattening it
@@ -173,7 +217,12 @@ public class JsonataEvaluator {
// case loops for ever and the execution never leaves RUNNING. The bounds raise a
// JSONata error, which the catch below turns into States.QueryEvaluationError.
frame.setRuntimeBounds(evaluationTimeoutMillis, maxEvaluationDepth);
+ nesting.trackOn(frame);
+ boundHeldMemory(frame);
stepFunctionsExtensions.forEach(frame::bind);
+ // AWS disables $eval, and the library ships it. An unbound name called as a function is
+ // the library's own T1006, which is the code AWS answers. $parse is the replacement.
+ frame.bind("eval", (Object) null);
// Workflow variables (the Assign field) are referenced as top-level $name in AWS's
// JSONata dialect, e.g. $CheckpointCount. They are bound alongside $states, never
// inside it: AWS reserves $states for input/result/errorOutput/context only. $states is
@@ -190,36 +239,165 @@ public class JsonataEvaluator {
Object result = jsonataExpr.evaluate(null, frame);
return toJsonNode(result);
} catch (Exception e) {
- throw new AslExecutor.FailStateException("States.QueryEvaluationError", queryEvaluationCause(e));
+ JsonNode nonFinite = nesting.isTheWholeExpression() ? nonFiniteRefusal(e) : null;
+ if (nonFinite != null) {
+ return nonFinite;
+ }
+ throw queryEvaluationFailure(e);
+ } catch (StackOverflowError | OutOfMemoryError e) {
+ // These two are the resource an expression itself spends: nesting deep enough to
+ // exhaust the parser's stack, and asking for a value larger than the JVM can hold.
+ // AWS bounds both and fails the state with States.QueryEvaluationError, so the state's
+ // own Retry and Catch (and a Catch on States.ALL) still see it. Every other Error says
+ // the runtime is broken rather than this expression, and it keeps going to
+ // AslExecutor's terminal catch (Error e), which fails the execution as States.Runtime
+ // and rethrows. The message is often null here, so toString() carries the diagnosis,
+ // the same way that terminal catch already renders it. QueryEvaluationFailure (rather
+ // than a bare FailStateException) is what lets evaluateField prepend AWS's sentence,
+ // colon-separated since this cause carries no JSONata code.
+ throw new QueryEvaluationFailure(e.toString(), ": ");
+ }
+ }
+
+ /**
+ * How deeply nested the evaluation step running is, the expression's own root node being 1.
+ * A step that fails never leaves, so once an exception has unwound the count is the depth of
+ * the step that raised it.
+ */
+ private static final class ExpressionNesting {
+
+ private int depth;
+
+ /**
+ * Counts on the same pair of callbacks the library installs its bounds through, keeping
+ * the library's own and running after it.
+ */
+ void trackOn(Jsonata.Frame frame) {
+ Jsonata.EntryCallback boundsOnEntry = (Jsonata.EntryCallback) frame.lookup("__evaluate_entry");
+ frame.setEvaluateEntryCallback((expression, input, environment) -> {
+ boundsOnEntry.callback(expression, input, environment);
+ depth++;
+ });
+ afterEachEvaluationStep(frame, (expression, input, environment, result) -> depth--);
+ }
+
+ /** Whether the step that failed is the expression itself rather than a step inside it. */
+ boolean isTheWholeExpression() {
+ return depth == 1;
}
}
/**
- * The cause AWS reports for a failed JSONata expression: the error code, then the message
- * jsonata-js renders for it, as in {@code T0412: Argument 1 of function "sum" must be an array
- * of "numbers"}. AWS puts a sentence naming the expression and the field in front of that,
- * which needs the field path {@link #resolveTemplate} does not thread today (#2665).
+ * The value behind a D1001, which the library raises rather than carrying a non-finite number:
+ * {@code 1/0} and {@code 1e308*10} both fail there where AWS answers {@code "Infinity"}. The
+ * exception is the only place the library leaves the value, and it carries no way back to where
+ * the arithmetic ran, so this recovers the value only when the step that raised it is the whole
+ * expression. {@code [1/0]} fails the state instead of answering AWS's {@code ["Infinity"]},
+ * which is the conservative half of the divergence: a state that fails is one a Catch fires on.
+ */
+ private JsonNode nonFiniteRefusal(Exception e) {
+ if (e instanceof JException jsonataError && "D1001".equals(jsonataError.getError())
+ && jsonataError.getCurrent() instanceof Number value) {
+ return fromJsonataValue(value.doubleValue());
+ }
+ return null;
+ }
+
+ /**
+ * Adds the memory bound to the time and depth bounds the library installs. The exit callback
+ * carries the value every evaluation step produces, which is what the bound is on.
+ */
+ private static void boundHeldMemory(Jsonata.Frame frame) {
+ afterEachEvaluationStep(frame, (expression, input, environment, result) -> {
+ if (heldBytes(result, 0) > MAX_EXPRESSION_BYTES) {
+ throw new JException(MEMORY_LIMIT_EXCEEDED, -1);
+ }
+ });
+ }
+
+ /**
+ * Runs {@code addition} after every evaluation step, keeping whatever the frame already runs
+ * there. The library installs its time and depth bounds through that same callback, so an
+ * addition extends them rather than replacing them.
+ */
+ private static void afterEachEvaluationStep(Jsonata.Frame frame, Jsonata.ExitCallback addition) {
+ Jsonata.ExitCallback installed = (Jsonata.ExitCallback) frame.lookup("__evaluate_exit");
+ frame.setEvaluateExitCallback((expression, input, environment, result) -> {
+ installed.callback(expression, input, environment, result);
+ addition.callback(expression, input, environment, result);
+ });
+ }
+
+ /**
+ * What {@code value} adds to the {@code held} bytes already counted, stopping as soon as the
+ * total passes {@link #MAX_EXPRESSION_BYTES} so a value far over the bound costs no more to
+ * refuse than one just over it. A range is counted from its length: the library materialises it
+ * lazily, and AWS refuses it on the elements it stands for rather than on what it has built.
+ */
+ private static long heldBytes(Object value, long held) {
+ if (held > MAX_EXPRESSION_BYTES) {
+ return held;
+ }
+ if (value instanceof String text) {
+ return held + text.length();
+ }
+ if (value instanceof Utils.RangeList range) {
+ return held + (long) range.size() * BYTES_PER_SCALAR;
+ }
+ if (value instanceof List<?> list) {
+ return elementsHeldBytes(list, held);
+ }
+ if (value instanceof Map<?, ?> map) {
+ return elementsHeldBytes(map.entrySet(), held);
+ }
+ if (value instanceof Map.Entry<?, ?> entry) {
+ return heldBytes(entry.getValue(), heldBytes(entry.getKey(), held));
+ }
+ return held + BYTES_PER_SCALAR;
+ }
+
+ private static long elementsHeldBytes(Iterable<?> elements, long held) {
+ long total = held;
+ for (Object element : elements) {
+ total = heldBytes(element, total);
+ if (total > MAX_EXPRESSION_BYTES) {
+ return total;
+ }
+ }
+ return total;
+ }
+
+ /**
+ * The state failure AWS reports for an expression that threw. Its cause is the error code and
+ * the message jsonata-js renders for it, as in {@code T0412: Argument 1 of function "sum" must
+ * be an array of "numbers"}; for the handful of failures with no JSONata code of their own (a
+ * timeout, a stack-depth bound) it is the bare message the exception came with.
*
- * <p>The dashjoin port renders the message itself, and three things go wrong on the way. Its
- * copy of the catalog has the word "function" replaced by "Object" throughout; its substituter
- * fills the first two placeholders of a template and leaves a third one standing; and it quotes
- * every value it inserts, where jsonata-js JSON-renders it. So the message is composed here
- * from the code and the values {@link JException} carries instead.
+ * <p>The dashjoin port renders the coded message itself, and three things go wrong on the way.
+ * Its copy of the catalog has the word "function" replaced by "Object" throughout; its
+ * substituter fills the first two placeholders of a template and leaves a third one standing;
+ * and it quotes every value it inserts, where jsonata-js JSON-renders it. So the message is
+ * composed here from the code and the values {@link JException} carries instead.
*/
- private String queryEvaluationCause(Exception e) {
+ private QueryEvaluationFailure queryEvaluationFailure(Exception e) {
if (!(e instanceof JException jsonataError) || jsonataError.getError() == null) {
- return e.getMessage();
+ return new QueryEvaluationFailure(e.getMessage(), ": ");
}
String code = jsonataError.getError();
Object current = jsonataError.getCurrent();
Object expected = jsonataError.getExpected();
// Both this class's own functions and a few library call sites throw with the whole
// message in the code slot. The catalog lookup then misses and dashjoin prefixes its own
- // class name, which AWS never emits; the message is the code slot itself.
+ // class name, which AWS never emits; the message is the code slot itself. Most of these
+ // already open with a real code ("D3137: Invalid JSON"), a sentence of its own that joins
+ // AWS's with ". "; the two bound messages (a timeout, a stack-depth or memory ceiling) open
+ // with English instead and are bare, joining with ": " like any other uncoded message.
if ((UNKNOWN_CODE_PREFIX + code).equals(jsonataError.getMessage())) {
- return code;
+ String bound = withoutDepthBoundSuffix(code);
+ String separator = bound.matches("[A-Z]\\d{4}:.*") ? ". " : ": ";
+ return new QueryEvaluationFailure(bound, separator);
}
- return switch (code) {
+ String tail = switch (code) {
// The only two templates in the catalog with a third placeholder. Neither third value
// reaches the exception: T0412 carries the offending argument and the element type it
// wanted but not the argument index nor the function name, and T2009 carries the two
@@ -230,6 +408,32 @@ public class JsonataEvaluator {
+ " either side of the operator must be of the same data type";
default -> code + ": " + renderTemplate(code, current, expected);
};
+ return new QueryEvaluationFailure(tail, ". ");
+ }
+
+ /**
+ * The library's stack-depth bound names the depth and the ceiling it hit,
+ * {@code ". Depth=101 max=100"}, which AWS's own sentence for the same failure never carries.
+ * Every other bare message this method renders has no such suffix, so stripping it here is
+ * specific to that one failure without needing to name it.
+ */
+ private static String withoutDepthBoundSuffix(String message) {
+ return message.replaceFirst("\\. Depth=\\d+ max=\\d+$", "");
+ }
+
+ /**
+ * A States.QueryEvaluationError the expression itself raised, carrying the punctuation that
+ * joins its cause to the sentence {@link #evaluateField} puts in front: {@code ". "} after a
+ * coded JSONata message, which is a sentence of its own, and {@code ": "} before a bare one.
+ */
+ private static final class QueryEvaluationFailure extends FailStateException {
+
+ final String sentenceSeparator;
+
+ QueryEvaluationFailure(String cause, String sentenceSeparator) {
+ super("States.QueryEvaluationError", cause);
+ this.sentenceSeparator = sentenceSeparator;
+ }
}
/**
@@ -256,19 +460,29 @@ public class JsonataEvaluator {
}
/**
- * Evaluate the single expression an ASL field holds, failing the state when it returns nothing.
- * AWS names the field in the cause and a {@code Catch} on States.QueryEvaluationError fires:
- * a Wait whose Seconds returned nothing does not wait zero seconds, it fails.
+ * Evaluate the single expression an ASL field holds, failing the state when it returns nothing
+ * or when the expression itself throws. AWS names the expression and the field in the cause
+ * either way and a {@code Catch} on States.QueryEvaluationError fires: a Wait whose Seconds
+ * returned nothing does not wait zero seconds, it fails, and {@code $abs("x")} in a field fails
+ * it with AWS's "threw an error during evaluation" sentence in front of the JSONata message.
*
* <p>{@code field} is the field's path relative to the state, as AWS writes it: {@code Seconds},
* {@code Choices[1]/Condition}, {@code Output/a/b}.
*/
JsonNode evaluateField(String expression, String field, JsonNode statesVar, JsonNode variables) {
- JsonNode value = evaluate(expression, statesVar, variables);
+ String expr = isExpression(expression) ? unwrap(expression) : expression;
+ JsonNode value;
+ try {
+ value = evaluate(expression, statesVar, variables);
+ } catch (QueryEvaluationFailure failure) {
+ throw new FailStateException(failure.error,
+ "The JSONata expression '" + expr + "' specified for the field '" + field
+ + "' threw an error during evaluation" + failure.sentenceSeparator + failure.cause);
+ }
if (value.isMissingNode()) {
throw new FailStateException("States.QueryEvaluationError",
- "The JSONata expression '" + (isExpression(expression) ? unwrap(expression) : expression)
- + "' specified for the field '" + field + "' returned nothing (undefined).");
+ "The JSONata expression '" + expr + "' specified for the field '" + field
+ + "' returned nothing (undefined).");
}
return value;
}
@@ -356,16 +570,31 @@ public class JsonataEvaluator {
list.forEach(element -> array.add(fromJsonataValue(element)));
return array;
}
+ // JSON has no literal for a non-finite number, so AWS writes each one as the string
+ // JavaScript names it by: 1/0 comes back as "Infinity" and $parseInteger("abc","0") as
+ // "NaN".
+ if (value instanceof Double number && !Double.isFinite(number)) {
+ return TextNode.valueOf(nonFiniteName(number));
+ }
return objectMapper.valueToTree(value);
}
+ private static String nonFiniteName(double value) {
+ if (Double.isNaN(value)) {
+ return "NaN";
+ }
+ return value > 0 ? "Infinity" : "-Infinity";
+ }
+
/**
* The JSONata functions Floci binds on the evaluation frame. Six of them are what the Step
* Functions dialect adds on top of the JSONata language, per
* https://docs.aws.amazon.com/step-functions/latest/dg/transforming-data.html: five the
* dashjoin library does not have at all, and $random it has with the wrong arity, so the
- * binding shadows it to accept AWS's optional seed. The seventh, $string, is a JSONata
- * function the library has but writes with a different number notation than AWS.
+ * binding shadows it to accept AWS's optional seed. The other three are JSONata functions the
+ * library has and answers differently from AWS: $string writes a number in another notation,
+ * $parseInteger answers nothing where AWS answers NaN, and $formatNumber accepts picture
+ * strings AWS refuses.
*
* <p>Each of the six declares one optional slot more than the arity AWS documents. The library
* pads a short call with Java nulls up to the declared arity and refuses a call longer than
@@ -385,7 +614,42 @@ public class JsonataEvaluator {
"hash", new Jsonata.JFunction((input, arguments) -> hash(arguments), "<x?x?x?:x>"),
"random", new Jsonata.JFunction((input, arguments) -> random(arguments), "<x?x?:x>"),
"uuid", new Jsonata.JFunction((input, arguments) -> uuid(arguments), "<x?:x>"),
- "string", new Jsonata.JFunction((input, arguments) -> string(arguments), "<x-b?:s>"));
+ "string", new Jsonata.JFunction((input, arguments) -> string(arguments), "<x-b?:s>"),
+ "parseInteger", new Jsonata.JFunction((input, arguments) -> parseInteger(arguments), "<s-s:n>"),
+ "formatNumber", new Jsonata.JFunction((input, arguments) -> formatNumber(arguments), "<n-so?:s>"));
+ }
+
+ /**
+ * $parseInteger(value, picture): the JSONata function, answering NaN where the picture cannot
+ * read the value, which is what AWS answers. The library answers nothing there, and an ASL
+ * field that evaluates to nothing fails the state.
+ */
+ private static Object parseInteger(List<Object> arguments) {
+ Object value = argument(arguments, 0);
+ if (value == null) {
+ return null;
+ }
+ try {
+ Number parsed = Functions.parseInteger((String) value, (String) argument(arguments, 1));
+ return parsed == null ? Double.NaN : parsed;
+ } catch (ParseException e) {
+ throw new JException(e.getMessage(), -1);
+ }
+ }
+
+ /**
+ * $formatNumber(value, picture, options): the JSONata function, with the picture string checked
+ * against the rules AWS applies to it first. See {@link FormatNumberPicture}.
+ */
+ private static Object formatNumber(List<Object> arguments) {
+ Object value = argument(arguments, 0);
+ if (value == null) {
+ return null;
+ }
+ Map<?, ?> options = (Map<?, ?>) argument(arguments, 2);
+ String picture = (String) argument(arguments, 1);
+ FormatNumberPicture.validate(picture, options);
+ return Functions.formatNumber((Number) value, picture, options);
}
/**
@@ -607,6 +871,9 @@ public class JsonataEvaluator {
.subtract(BigInteger.valueOf(start))
.divide(BigInteger.valueOf(step))
.add(BigInteger.ONE);
+ if (elements.compareTo(BigInteger.valueOf(MAX_RANGE_ELEMENTS)) > 0) {
+ throw new JException(MEMORY_LIMIT_EXCEEDED, -1);
+ }
List<Object> values = new ArrayList<>();
long value = start;
for (BigInteger emitted = BigInteger.ZERO;
diff --git a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluatorTest.java b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluatorTest.java
index 8715ea7ea2..b29e50be70 100644
--- a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluatorTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluatorTest.java
@@ -10,6 +10,7 @@ import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;
+import java.util.Map;
import java.util.stream.Stream;
import static org.junit.jupiter.api.Assertions.*;
@@ -349,6 +350,50 @@ class JsonataEvaluatorTest {
assertTrue(ex.cause.contains("T0410"));
}
+ /**
+ * The AWS bound on how many elements a single {@code $range} call may produce, measured
+ * directly (us-east-1, test-state, 2026-08-30): AWS accepts {@code $range(1, 360145, 1)},
+ * returning all 360,145 elements.
+ */
+ @Test
+ void rangeAtTheAwsElementBoundBuildsTheFullArray() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+ JsonNode result = evaluator.evaluate("{% $count($range(1, 360145, 1)) %}", statesVar);
+ assertEquals(360145, result.asInt());
+ }
+
+ /**
+ * One element past the bound, AWS fails the state with {@code States.QueryEvaluationError}
+ * and this cause byte for byte (us-east-1, test-state, 2026-08-30).
+ */
+ @Test
+ void rangeOneElementPastTheAwsElementBoundFailsWithTheMemoryRefusal() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluateField("{% $range(1, 360146, 1) %}", "Output/v", statesVar, null));
+ assertEquals("States.QueryEvaluationError", failure.error);
+ assertEquals("The JSONata expression '$range(1, 360146, 1)' specified for the field 'Output/v' threw "
+ + "an error during evaluation: Expression evaluation memory limit exceeded: Check for excessive "
+ + "memory usage", failure.cause);
+ }
+
+ /**
+ * Far past the bound, the refusal has to come from the pre-allocation cap in {@code range()}
+ * rather than from the memory bound on the built value: an unbounded call would spend
+ * gigabytes building the array before that check ever runs.
+ */
+ @Test
+ @Timeout(60)
+ void rangeFarPastTheBoundFailsWithoutAllocating() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluateField("{% $range(1, 1000000000, 1) %}", "Output/v", statesVar, null));
+ assertEquals("States.QueryEvaluationError", failure.error);
+ assertEquals("The JSONata expression '$range(1, 1000000000, 1)' specified for the field 'Output/v' threw "
+ + "an error during evaluation: Expression evaluation memory limit exceeded: Check for excessive "
+ + "memory usage", failure.cause);
+ }
+
@Test
void hash_computesKnownDigestsForEachAlgorithm() {
JsonNode statesVar = objectMapper.createObjectNode();
@@ -519,6 +564,33 @@ class JsonataEvaluatorTest {
assertEquals("1.5", evaluator.evaluate("{% $parse('1.5') %}", statesVar).toString());
}
+ /**
+ * The execution input reaching {@code $states.input} follows the same number model as
+ * {@code $parse} (see {@link #parse_dropsATrailingZeroOnAnIntegerValuedNumber}): both route
+ * through {@code toJsonataValue}/{@code toJsonataNumber}. Measured against real AWS
+ * (us-east-1, test-state, 2026-08-30): a Pass state with
+ * {@code Output: {"v": "{% $states.input.amount %}"}} and input {@code {"amount": 1.0}}
+ * answers {@code {"v": 1}}.
+ */
+ @Test
+ void statesInputDropsATrailingZeroOnAnIntegerValuedNumber() throws Exception {
+ JsonNode statesVar = objectMapper.readTree("{\"input\": {\"amount\": 1.0}}");
+ assertEquals("1", evaluator.evaluate("{% $states.input.amount %}", statesVar).toString());
+ }
+
+ /**
+ * Same model, the other side of the long boundary (see
+ * {@link #parse_keepsAnIntegerExactWhileItFitsInALong}). Measured against real AWS
+ * (us-east-1, test-state, 2026-08-30): input {@code {"amount": 9223372036854775808}} answers
+ * {@code {"v": 9.223372036854776E18}}.
+ */
+ @Test
+ void statesInputSwitchesToADoublePastTheLongBoundary() throws Exception {
+ JsonNode statesVar = objectMapper.readTree("{\"input\": {\"amount\": 9223372036854775808}}");
+ assertEquals("9.223372036854776E18",
+ evaluator.evaluate("{% $states.input.amount %}", statesVar).toString());
+ }
+
@Test
void parse_rejectsJsonAwsRejects() {
JsonNode statesVar = objectMapper.createObjectNode();
@@ -582,9 +654,9 @@ class JsonataEvaluatorTest {
* without a depth bound it ends in a StackOverflowError, which is an Error and so escapes the
* evaluator's catch. The shipped bound is what turns it into a state failure.
*
- * <p>The cause is AWS's own string for this failure, followed by the {@code Depth=N max=M}
- * suffix that names the bound that was hit. AWS writes one space after the full stop where
- * the library writes two.
+ * <p>The cause is the library's own string for this failure, with its trailing {@code Depth=N
+ * max=M} suffix stripped: AWS never emits it. The library writes two spaces after the full
+ * stop where AWS writes one; that divergence is left as is.
*/
@Test
@Timeout(30)
@@ -597,18 +669,13 @@ class JsonataEvaluatorTest {
assertEquals("States.QueryEvaluationError", ex.error);
assertEquals("Stack overflow error: Check for non-terminating recursive"
- + " function. Consider rewriting as tail-recursive. Depth=501 max=500", ex.cause);
+ + " function. Consider rewriting as tail-recursive", ex.cause);
}
/**
* The boundary from below: the risk of a bound is that it fails a working state machine, so
- * these must all still evaluate. Measured with {@code aws stepfunctions test-state}, AWS
- * returns 30 for the first and 800000 for the second, which makes them the ceiling of what
- * AWS itself accepts on each axis: it refuses the recursion at n=40 (nesting depth 125) and
- * the sequence at 900,000 elements.
- *
- * <p>The last two AWS refuses on its memory limit and Floci evaluates, which is where the
- * bounds are deliberately looser than AWS rather than tighter.
+ * these must all still evaluate. AWS accepts every one of them and refuses the next value on
+ * each axis: the recursion at n=32, the sequence at 873,783 elements.
*/
@Test
@Timeout(60)
@@ -617,13 +684,251 @@ class JsonataEvaluatorTest {
String recursionHead = "{% ($f := function($x){ $x <= 0 ? 0 : 1 + $f($x-1) }; $f(";
assertEquals(30, evaluator.evaluate(recursionHead + "30)) %}", statesVar).asInt());
+ assertEquals(31, evaluator.evaluate(recursionHead + "31)) %}", statesVar).asInt());
assertEquals(800000, evaluator.evaluate("{% [1..800000] ~> $count() %}", statesVar).asInt());
-
- assertEquals(160, evaluator.evaluate(recursionHead + "160)) %}", statesVar).asInt());
assertEquals(40000200000L,
evaluator.evaluate("{% $sum($map([1..200000], function($v){$v * 2})) %}", statesVar).asLong());
}
+ /**
+ * The four nesting shapes of issue #2737, at the last {@code n} AWS accepts and at the first
+ * it refuses. An addition chain nests one level per term, so AWS accepts 100 terms;
+ * parentheses, brackets and the chain operator nest one level more than their count, so AWS
+ * accepts 99 of each. Both put AWS's ceiling at
+ * nesting depth 100, which is what {@link JsonataEvaluator#MAX_EVALUATION_DEPTH} holds.
+ */
+ @Test
+ @Timeout(30)
+ void everyNestingShapeFailsAtTheDepthAwsFailsAt() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ assertEquals(100, evaluator.evaluate("{% 1" + "+1".repeat(99) + " %}", statesVar).asInt());
+ assertEquals(1, evaluator.evaluate("{% " + "(".repeat(99) + "1" + ")".repeat(99) + " %}", statesVar).asInt());
+ assertTrue(evaluator.evaluate("{% " + "[".repeat(99) + "1" + "]".repeat(99) + " %}",
+ statesVar).isArray());
+ assertEquals("1", evaluator.evaluate("{% 1" + "~>$string".repeat(99) + " %}", statesVar).asText());
+
+ for (String tooDeep : new String[]{
+ "1" + "+1".repeat(100),
+ "(".repeat(100) + "1" + ")".repeat(100),
+ "[".repeat(100) + "1" + "]".repeat(100),
+ "1" + "~>$string".repeat(100)}) {
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate("{% " + tooDeep + " %}", statesVar),
+ "expected the depth bound to refuse " + tooDeep.substring(0, 20) + "...");
+ assertEquals("States.QueryEvaluationError", ex.error);
+ assertTrue(ex.cause.endsWith("Consider rewriting as tail-recursive"), ex.cause);
+ }
+ }
+
+ /**
+ * The non-tail-recursive shape of issue #2737, which nests three levels per call: AWS accepts
+ * {@code $f(31)} and refuses {@code $f(32)}, and the depth bound reproduces both.
+ */
+ @Test
+ @Timeout(30)
+ void nonTailRecursionFailsAtTheCallDepthAwsFailsAt() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+ String recursionHead = "{% ($f := function($x){ $x <= 0 ? 0 : 1 + $f($x-1) }; $f(";
+
+ assertEquals(31, evaluator.evaluate(recursionHead + "31)) %}", statesVar).asInt());
+
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate(recursionHead + "32)) %}", statesVar));
+ assertEquals("States.QueryEvaluationError", ex.error);
+ assertTrue(ex.cause.endsWith("Consider rewriting as tail-recursive"), ex.cause);
+ }
+
+ /**
+ * The memory bound of issue #2737 on the sequence AWS draws it on: AWS accepts
+ * {@code [1..873782]} and refuses one element more, and counting a number as eight bytes puts
+ * those two on either side of {@link JsonataEvaluator#MAX_EXPRESSION_BYTES} here as well.
+ */
+ @Test
+ @Timeout(60)
+ void theLargestSequenceAwsAcceptsEvaluatesAndTheNextOneFailsOnTheMemoryBound() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ assertEquals(873782, evaluator.evaluate("{% [1..873782] ~> $count() %}", statesVar).asInt());
+
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate("{% [1..873783] ~> $count() %}", statesVar));
+ assertEquals("States.QueryEvaluationError", ex.error);
+ assertEquals("Expression evaluation memory limit exceeded: Check for excessive memory usage", ex.cause);
+ }
+
+ /**
+ * The same memory bound on a string: a string that doubles 22 times evaluates on AWS and one
+ * that doubles 23 times does not. A character counts as one byte, which leaves 2^22 characters
+ * inside the bound and 2^23 outside it, as on AWS.
+ */
+ @Test
+ @Timeout(60)
+ void aStringDoublingStopsAtTheSameBoundAwsStopsAt() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+ String doublingHead = "{% ($f := function($s, $n){ $n = 0 ? $s : $f($s & $s, $n - 1) };"
+ + " $length($f('x', ";
+
+ assertEquals(4194304, evaluator.evaluate(doublingHead + "22))) %}", statesVar).asInt());
+
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate(doublingHead + "23))) %}", statesVar));
+ assertEquals("States.QueryEvaluationError", ex.error);
+ assertEquals("Expression evaluation memory limit exceeded: Check for excessive memory usage", ex.cause);
+ }
+
+ /**
+ * The two expressions of issue #2737 that AWS refuses on its memory limit. They fail the state
+ * here too, so a Catch fires on them rather than the state succeeding with a value.
+ */
+ @Test
+ @Timeout(60)
+ void theSequencesAwsRefusesOnItsMemoryLimitFailTheStateHere() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ for (String expression : new String[]{"[1..900000] ~> $count()", "$sum([1..900000])"}) {
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate("{% " + expression + " %}", statesVar),
+ "expected the memory bound to refuse " + expression);
+ assertEquals("States.QueryEvaluationError", ex.error);
+ assertEquals("Expression evaluation memory limit exceeded: Check for excessive memory usage", ex.cause);
+ }
+ }
+
+ /**
+ * AWS disables $eval and answers T1006 for a call to it. The library ships $eval, so the name
+ * is left unbound here and a call to it answers the same code; $parse is the replacement AWS
+ * documents.
+ */
+ @Test
+ void evalIsNotBoundBecauseAwsDisablesIt() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate("{% $eval('1+1') %}", statesVar));
+
+ assertEquals("States.QueryEvaluationError", ex.error);
+ assertEquals("T1006: Attempted to invoke a non-function", ex.cause);
+ }
+
+ /**
+ * JSON has no literal for a non-finite number, so AWS writes each one as the string JavaScript
+ * names it by. The library refuses to carry the value at all and raises D1001 instead, and the
+ * value that refusal carries is what these answer.
+ */
+ @Test
+ void aNonFiniteNumberIsTheStringAwsWritesForIt() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ assertEquals("Infinity", evaluator.evaluate("{% 1/0 %}", statesVar).asText());
+ assertEquals("Infinity", evaluator.evaluate("{% 1e308 * 10 %}", statesVar).asText());
+ assertEquals("-Infinity", evaluator.evaluate("{% -1/0 %}", statesVar).asText());
+ }
+
+ /**
+ * The shapes AWS answers a string for and Floci does not. A non-finite number the library's own
+ * arithmetic produces reaches Floci only as the exception the library raises instead of carrying
+ * the value, and by then the exception has unwound whatever value was being built around it, so
+ * only an expression whose own result is non-finite can be answered. AWS answers
+ * {@code ["Infinity"]} for the first, {@code {"k":"Infinity"}} for the second and
+ * {@code "NaN"} for {@code 0/0}.
+ *
+ * <p>Failing the state is the half of the divergence a Catch fires on, which is why the value is
+ * not answered at the top of an expression that was building something else with it.
+ */
+ @Test
+ void aNonFiniteNumberTheArithmeticBuildsInsideAValueFailsTheStateInstead() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ for (String expression : new String[]{"[1/0]", "{'k': 1/0}", "[1/0, 2]", "$string(1/0)"}) {
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate("{% " + expression + " %}", statesVar),
+ "expected " + expression + " to fail the state");
+ assertEquals("States.QueryEvaluationError", ex.error);
+ assertEquals("D1001: Number out of range: \"Infinity\"", ex.cause);
+ }
+
+ // NaN raises nothing at all: the library reads it as not a number and drops it, so the
+ // expression returns nothing and the field it was written in is what fails.
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluateField("{% 0/0 %}", "Output/r", statesVar, null));
+ assertEquals("The JSONata expression '0/0' specified for the field 'Output/r' "
+ + "returned nothing (undefined).", ex.cause);
+ }
+
+ /**
+ * $parseInteger answers NaN for a value its picture cannot read, where the library answers
+ * nothing and an ASL field that evaluates to nothing fails the state.
+ */
+ @Test
+ void parseIntegerAnswersNaNForAValueThePictureCannotRead() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ assertEquals("NaN", evaluator.evaluateField("{% $parseInteger('abc', '0') %}",
+ "Output/r", statesVar, null).asText());
+ // A non-finite number a function answers is carried like any other value, so it reaches the
+ // output as AWS writes it wherever it lands, which is what arithmetic cannot do.
+ assertEquals("[\"NaN\"]",
+ evaluator.evaluate("{% [$parseInteger('abc', '0')] %}", statesVar).toString());
+ assertEquals("{\"k\":\"NaN\"}",
+ evaluator.evaluate("{% {'k': $parseInteger('abc', '0')} %}", statesVar).toString());
+ // A value the picture does read stays the integer it was, not a double.
+ assertEquals("123", evaluator.evaluate("{% $parseInteger('123', '0') %}", statesVar).toString());
+ }
+
+ /**
+ * A picture string with no digit in it describes no number, and AWS fails the state on it
+ * where the library formats through DecimalFormat and answers "x1".
+ */
+ @Test
+ void formatNumberRefusesAPictureThatDescribesNoNumber() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate("{% $formatNumber(1, 'x') %}", statesVar));
+
+ assertEquals("States.QueryEvaluationError", ex.error);
+ assertEquals("D3086: The sub-picture must not contain a passive character that is preceded"
+ + " by an active character and that is followed by another active character", ex.cause);
+ }
+
+ @Test
+ void formatNumberNamesTheRuleThePictureBreaks() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+ Map<String, String> expectedCodes = Map.ofEntries(
+ Map.entry("0.0;0.0;0.0", "D3080"), Map.entry("0.0.0", "D3081"),
+ Map.entry("0%%", "D3082"), Map.entry("0‰‰", "D3083"),
+ Map.entry("0%‰", "D3084"), Map.entry("0,.0", "D3087"),
+ Map.entry("0,", "D3088"), Map.entry("0,,0", "D3089"),
+ Map.entry("0#", "D3090"), Map.entry("##0.0#0", "D3091"),
+ Map.entry("0e0%", "D3092"), Map.entry("#0e", "D3093"));
+
+ expectedCodes.forEach((picture, code) -> {
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate("{% $formatNumber(1, '" + picture + "') %}", statesVar),
+ "expected " + picture + " to fail");
+ assertTrue(ex.cause.startsWith(code + ":"), picture + " gave " + ex.cause);
+ });
+ }
+
+ @Test
+ void formatNumberStillFormatsThePicturesTheRulesAllow() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ assertEquals("1,234.57", evaluator.evaluate("{% $formatNumber(1234.5678, '#,##0.00') %}",
+ statesVar).asText());
+ assertEquals("12.3%", evaluator.evaluate("{% $formatNumber(0.123, '#0.0%') %}", statesVar).asText());
+ assertEquals("$1,234.57", evaluator.evaluate("{% $formatNumber(1234.5678, '$#,##0.00') %}",
+ statesVar).asText());
+ // A passive character before the first active one (a currency prefix, or the parenthesis
+ // AWS's own negative sub-picture opens with) shifted the exponent separator's index without
+ // shifting it back onto the active part, so any of these with an exponent was refused with a
+ // spurious D3093.
+ assertEquals("-$12e2", evaluator.evaluate("{% $formatNumber(-1200, '$#0e0') %}", statesVar).asText());
+ assertEquals("-12e2", evaluator.evaluate("{% $formatNumber(-1200, '#0e0;-#0e0') %}", statesVar).asText());
+ assertEquals("(12e2)", evaluator.evaluate("{% $formatNumber(-1200, '#0e0;(#0e0)') %}", statesVar).asText());
+ }
+
@Test
void string_writesAWholeNumberInFullBelowTheExponentBoundary() {
// dashjoin stops writing digits at Long.MAX_VALUE, 9.223372036854776E18, and prints
@@ -908,6 +1213,51 @@ class JsonataEvaluatorTest {
+ expectedFieldPath + "' returned nothing (undefined).", failure.cause);
}
+ /**
+ * #2738: {@code evaluate} caught {@code Exception}, not {@code Error}, so a
+ * {@code java.lang.Error} raised while parsing or evaluating an expression escaped straight
+ * past this method to {@code AslExecutor}'s last-resort {@code catch (Error e)}, failing the
+ * whole execution as {@code States.Runtime} before the state's own {@code Retry}/{@code Catch}
+ * ever ran. This pins the {@code StackOverflowError} arm of the catch, with an expression
+ * nested 200,000 parentheses deep that overflows the JVM call stack during parsing in a few
+ * milliseconds and never grows a string past a few hundred KB. The other arm,
+ * {@code OutOfMemoryError}, remains a guard with no cheap trigger through this library: the
+ * memory bound now trips first on every expression measured against it, including the one the
+ * issue reproduced with (see
+ * {@code StepFunctionsJsonataIntegrationTest.parallelBranchErrorIsCatchableByStatesAll}).
+ */
+ @Test
+ void aStackOverflowErrorDuringParsingBecomesAQueryEvaluationError() {
+ JsonNode statesVar = objectMapper.createObjectNode();
+ String deeplyNested = "(".repeat(200_000) + "1" + ")".repeat(200_000);
+
+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluate("{% " + deeplyNested + " %}", statesVar));
+
+ assertEquals("States.QueryEvaluationError", ex.error);
+ assertTrue(ex.cause.contains("StackOverflowError"), ex.cause);
+ }
+
+ /**
+ * The {@code StackOverflowError}/{@code OutOfMemoryError} arm used to throw a bare
+ * {@code FailStateException}, which {@link JsonataEvaluator#evaluateField} does not catch as a
+ * {@code QueryEvaluationFailure}, so the field's expression and AWS's sentence never reached the
+ * cause: a Catch on this failure saw the bare {@code java.lang.StackOverflowError} instead of
+ * the sentence every other evaluation failure carries.
+ */
+ @Test
+ void evaluateFieldWrapsAStackOverflowErrorWithTheSentenceAndAColonSeparator() throws Exception {
+ JsonNode statesVar = objectMapper.createObjectNode();
+ String deeplyNested = "(".repeat(200_000) + "1" + ")".repeat(200_000);
+
+ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluateField("{% " + deeplyNested + " %}", "Output/v", statesVar, null));
+
+ assertEquals("States.QueryEvaluationError", failure.error);
+ assertTrue(failure.cause.contains("threw an error during evaluation: "), failure.cause);
+ assertTrue(failure.cause.contains("StackOverflowError"), failure.cause);
+ }
+
@Test
void evaluateFieldNamesTheFieldOfAPositionThatHoldsASingleExpression() throws Exception {
JsonNode statesVar = objectMapper.readTree("{\"input\": {}}");
@@ -920,6 +1270,79 @@ class JsonataEvaluatorTest {
+ "returned nothing (undefined).", failure.cause);
}
+ @Test
+ void evaluateFieldWrapsAThrownJsonataErrorWithTheSentenceAndACodedSeparator() throws Exception {
+ // A coded JSONata failure carries AWS's sentence in front of the code, joined by ". ".
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluateField("{% $sum(['a']) %}", "Output/v", statesVar, null));
+
+ assertEquals("States.QueryEvaluationError", failure.error);
+ assertEquals("The JSONata expression '$sum(['a'])' specified for the field 'Output/v' threw an error "
+ + "during evaluation. T0412: Argument [\"a\"] must be an array of \"numbers\"", failure.cause);
+ }
+
+ @Test
+ void evaluateFieldWrapsABoundFailureWithTheSentenceAndAColonSeparator() throws Exception {
+ // A failure with no JSONata code and no template to render (see
+ // aNonJsonataFailureKeepsTheMessageItCameWith) carries AWS's sentence in front of the bare
+ // message it came with, joined by ": " where a coded one joins with ". ".
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluateField("{% $toMillis('nope') %}", "Output/v", statesVar, null));
+
+ assertEquals("States.QueryEvaluationError", failure.error);
+ assertTrue(failure.cause.startsWith("The JSONata expression '$toMillis('nope')' specified for the field "
+ + "'Output/v' threw an error during evaluation: "), failure.cause);
+ assertTrue(failure.cause.contains("nope"), failure.cause);
+ }
+
+ /**
+ * The memory bound's message carries no JSONata code slot, so it is a bare bound message like
+ * {@code $toMillis('nope')}'s and joins AWS's sentence with {@code ": "}, not the {@code ". "}
+ * a coded message joins with. Measured against real AWS (us-east-1, test-state): this is its
+ * cause byte for byte.
+ */
+ @Test
+ @Timeout(60)
+ void evaluateFieldWrapsTheMemoryBoundFailureWithTheSentenceAndAColonSeparator() throws Exception {
+ JsonNode statesVar = objectMapper.createObjectNode();
+
+ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluateField("{% [1..900000] ~> $count() %}", "Output/r", statesVar, null));
+
+ assertEquals("States.QueryEvaluationError", failure.error);
+ assertEquals("The JSONata expression '[1..900000] ~> $count()' specified for the field 'Output/r' threw "
+ + "an error during evaluation: Expression evaluation memory limit exceeded: Check for excessive "
+ + "memory usage", failure.cause);
+ }
+
+ /**
+ * The depth bound's message is the same shape: no JSONata code slot, so {@code ": "} joins it
+ * to AWS's sentence too. AWS's own cause for this same 101-term chain, measured the same way,
+ * ends {@code "...evaluation: Stack overflow error: Check for non-terminating recursive
+ * function. Consider rewriting as tail-recursive"}; the library's own double space after
+ * "function." is the one divergence {@link JsonataEvaluator#withoutDepthBoundSuffix} leaves in
+ * place, so this checks the sentence and the join rather than the byte-exact tail.
+ */
+ @Test
+ @Timeout(30)
+ void evaluateFieldWrapsTheDepthBoundFailureWithTheSentenceAndAColonSeparator() throws Exception {
+ JsonNode statesVar = objectMapper.createObjectNode();
+ String chain = "1" + "+1".repeat(100);
+
+ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,
+ () -> evaluator.evaluateField("{% " + chain + " %}", "Output/r", statesVar, null));
+
+ assertEquals("States.QueryEvaluationError", failure.error);
+ assertTrue(failure.cause.startsWith("The JSONata expression '" + chain + "' specified for the field "
+ + "'Output/r' threw an error during evaluation: Stack overflow error: Check for non-terminating"),
+ failure.cause);
+ assertTrue(failure.cause.endsWith("Consider rewriting as tail-recursive"), failure.cause);
+ }
+
@Test
void anExplicitNullIsAValueAndKeepsEveryPositionEvaluating() throws Exception {
JsonNode statesVar = objectMapper.readTree("{\"input\": {\"bar\": null}}");
diff --git a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataFunctionsIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataFunctionsIntegrationTest.java
index 7e692b46e6..bb6aebb812 100644
--- a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataFunctionsIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataFunctionsIntegrationTest.java
@@ -247,7 +247,38 @@ class StepFunctionsJsonataFunctionsIntegrationTest {
Response resp = waitForExecutionFailure(startExecution(smArn, "{}"));
assertEquals("States.QueryEvaluationError", resp.jsonPath().getString("error"));
- assertEquals("T0412: Argument [\"a\"] must be an array of \"numbers\"",
+ assertEquals("The JSONata expression '$sum(['a'])' specified for the field 'Output/v' threw an error "
+ + "during evaluation. T0412: Argument [\"a\"] must be an array of \"numbers\"",
+ resp.jsonPath().getString("cause"));
+ }
+
+ @Test
+ void aFailedJsonataExpressionOpensWithTheSentenceNamingTheExpressionAndTheField() throws Exception {
+ // AWS on the same definition: States.QueryEvaluationError,
+ // "The JSONata expression '$abs(\"x\")' specified for the field 'Output/v' threw an error
+ // during evaluation. T0410: Argument 1 of function \"abs\" does not match function signature"
+ String definition = """
+ {
+ "QueryLanguage": "JSONata",
+ "StartAt": "Abs",
+ "States": {
+ "Abs": {
+ "Type": "Pass",
+ "Output": {
+ "v": "{% $abs(\\"x\\") %}"
+ },
+ "End": true
+ }
+ }
+ }
+ """;
+
+ String smArn = createStateMachine("jsonata-abs-error-cause", definition);
+ Response resp = waitForExecutionFailure(startExecution(smArn, "{}"));
+
+ assertEquals("States.QueryEvaluationError", resp.jsonPath().getString("error"));
+ assertEquals("The JSONata expression '$abs(\"x\")' specified for the field 'Output/v' threw an error "
+ + "during evaluation. T0410: Argument 1 of function \"abs\" does not match function signature",
resp.jsonPath().getString("cause"));
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataIntegrationTest.java
index 164d46419d..bb61192e00 100644
--- a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataIntegrationTest.java
@@ -1311,6 +1311,62 @@ class StepFunctionsJsonataIntegrationTest {
assertTrue(caught.path("caught").asBoolean(), caught.toString());
}
+ /**
+ * #2738: a {@code java.lang.Error} raised while evaluating a branch's expression used to
+ * escape {@code JsonataEvaluator}'s catch and reach {@code AslExecutor}'s own last-resort
+ * {@code catch (Error e)}, failing the whole execution as {@code States.Runtime} before the
+ * {@code Parallel}'s own {@code Catch} ever ran. On real AWS (measured in us-east-1) the same
+ * definition ends {@code SUCCEEDED} with output {@code {"caught":true}}.
+ *
+ * <p>The branch expression is the issue's own reproduction verbatim (quotes swapped to
+ * JSONata's single-quote string literal, which embeds in this JSON body without escaping):
+ * a tail-recursive function doubles a string 31 times, which dashjoin loops rather than
+ * recursing on (JSONata optimises the tail call). Since the memory bound
+ * {@code JsonataEvaluator} now holds, the doubling trips it at the 23rd iteration &mdash;
+ * {@code 'x'} doubled 23 times is 8,388,608 characters, past
+ * {@link JsonataEvaluator#MAX_EXPRESSION_BYTES}'s 6,990,256 &mdash; and fails the state with
+ * {@code Expression evaluation memory limit exceeded} through the ordinary {@code JException}
+ * path, eight doublings short of ever reaching the 31st and nowhere near the JVM's maximum
+ * array length. Whichever of the two carries the failure, the {@code Parallel}'s own
+ * {@code Catch} still sees it, which is what this test pins: the execution still ends
+ * {@code SUCCEEDED} with {@code {"caught":true}}, matching AWS. The {@code OutOfMemoryError}
+ * arm of {@code JsonataEvaluator.evaluate}'s catch remains a guard with no cheap trigger
+ * through this library; its {@code StackOverflowError} sibling is pinned by
+ * {@code JsonataEvaluatorTest.aStackOverflowErrorDuringParsingBecomesAQueryEvaluationError}.
+ */
+ @Test
+ void parallelBranchErrorIsCatchableByStatesAll() throws Exception {
+ String definition = """
+ {
+ "QueryLanguage": "JSONata",
+ "StartAt": "P",
+ "States": {
+ "P": {
+ "Type": "Parallel",
+ "Branches": [{
+ "StartAt": "E",
+ "States": {
+ "E": {
+ "Type": "Pass",
+ "Output": {"r": "{% ($p := function($s, $n) { $n = 0 ? $s : $p($s & $s, $n - 1) }; $length($p('x', 31))) %}"},
+ "End": true
+ }
+ }
+ }],
+ "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "Caught"}],
+ "End": true
+ },
+ "Caught": {"Type": "Pass", "Output": {"caught": true}, "End": true}
+ }
+ }
+ """;
+
+ String smArn = createStateMachine("sfn-error-catchable-test", definition);
+ JsonNode output = objectMapper.readTree(waitForExecution(startExecution(smArn, "{}")));
+
+ assertTrue(output.path("caught").asBoolean(), output.toString());
+ }
+
@Test
void assignExpressionReturningNothingFailsTheState() throws Exception {
// Real AWS names 'Assign/x' for this definition; before this guard the variable was never