Light, fluffy, and always free - The AWS Local Emulator alternative
fix(stepfunctions): bound $range allocation at the AWS memory limit (#2790)
Facundo Paredesf7f16cd2a3e7
diff --git a/docs/services/step-functions.md b/docs/services/step-functions.mdindex 5de3e6b94f..df7a299ddc 100644--- a/docs/services/step-functions.md+++ b/docs/services/step-functions.md@@ -189,7 +189,7 @@ top of the JSONata language, alongside every function JSONata itself provides.| Function | Returns || --- | --- |-| `$parse(jsonString)` | the deserialized value; the replacement for `$eval`, which AWS disables |+| `$parse(jsonString)` | the deserialized value; the replacement for `$eval`, which AWS disables and so does Floci, answering `T1006` to a call || `$partition(array, chunkSize)` | `array` split into chunks of `chunkSize`, the last one holding the remainder || `$range(start, end, step)` | the values from `start` to `end`, inclusive when `step` lands on `end` || `$hash(str, algorithm)` | the hex digest of `str`; `algorithm` is `MD5`, `SHA-1`, `SHA-256`, `SHA-384` or `SHA-512`, case-sensitive |@@ -210,16 +210,47 @@ JSONata's own `$string` follows AWS's number notation: a whole number is written`1e21` and in exponent notation from there, on both signs, so `$string(1e20)` is`100000000000000000000` and `$string(1e21)` is `1e+21`.-Evaluation is bounded, as it is on AWS: one expression may run for five seconds and nest 500-levels deep, and past either the state fails with `States.QueryEvaluationError`. Both bounds sit-well above what AWS itself accepts, so an expression that evaluates there evaluates here. AWS-refuses a non-tail-recursive function past a nesting depth near 100, and the largest sequence it-accepts evaluates here in about a tenth of a second.--One deviation. AWS also bounds the *memory* an expression may use, refusing-`[1..900000] ~> $count()` with `Expression evaluation memory limit exceeded` while accepting the-same expression at 800,000 elements. Floci bounds time and depth but not memory, and its ranges-are lazy, so that expression answers immediately at any size.+The execution input reaching `$states.input` follows the same number model as `$parse`: an integer+stays exact while it fits in a `long` and switches to a `double` past that boundary, matching AWS.++JSONata's own `$formatNumber` checks its picture string against the fourteen rules of XPath F&O+4.7.3, as AWS does, so `$formatNumber(1, "x")` fails with `D3086` rather than answering `x1`: a+picture with no digit in it describes no number.++Evaluation is bounded on three axes, as it is on AWS, and past any of them the state fails with+`States.QueryEvaluationError`.++- **Depth**: one expression may nest 100 levels, which is AWS's own ceiling: AWS accepts `1+1+…+1`+ at 100 terms, 99 parentheses, 99 brackets, 99 `~>` stages, and a non-tail-recursive `$f(31)`,+ which nests `3n+5`, and refuses one level more of each. The refusal names the depth reached, as+ in `Stack overflow error: … Depth=101 max=100`.+- **Memory**: one value an expression builds may hold 6,990,256 bytes, counting a number as eight+ bytes and a character as one. That is AWS's own bound: AWS accepts `[1..873782]` and refuses one+ element more, and it accepts a string doubled 22 times, 2^22 characters, refusing the 23rd. The+ refusal is AWS's own+ `Expression evaluation memory limit exceeded`, which is what `[1..900000] ~> $count()` and+ `$sum([1..900000])` now answer. `$range` holds a tighter bound of its own, checked before it+ allocates rather than on the array it would have built: AWS accepts `$range(1, 360145, 1)`, all+ 360,145 elements, and refuses one element more with the same refusal. A lazy literal range such+ as `[1..873782]` is exempt from both bounds on AWS, and stays exempt here.+- **Time**: five seconds, which is the library's own default and roughly fifty times the slowest+ evaluation of a payload AWS itself accepts. A recursive expression with no base case is a tail+ call, so it loops rather than nesting and only the clock ends it.++JSON has no literal for a non-finite number, so AWS writes each one as the string JavaScript names+it by, wherever it lands: `1/0` is `"Infinity"`, `1e308 * 10` is `"Infinity"`, `0/0` is `"NaN"` and+`$parseInteger("abc", "0")` is `"NaN"`, and nested, `[1/0]` is `["Infinity"]` and `{"k": 1/0}` is+`{"k": "Infinity"}`.++One deviation, on the arithmetic half of that. A non-finite number a **function** answers is a value+like any other here, so `$parseInteger("abc", "0")` is `"NaN"` on its own, inside an array and+inside an object, as on AWS. One the **arithmetic** produces is not: the JSONata library refuses to+carry it and raises instead, so Floci sees the value only in that refusal, after it has unwound+whatever was being built around it. `1/0`, `-1/0` and `1e308 * 10` are answered because the+expression's own result is what was refused; `[1/0]`, `{"k": 1/0}` and `$string(1/0)` fail the state+with `States.QueryEvaluationError` where AWS answers a string, and `0/0` fails the field that holds+it, because NaN is dropped as not-a-number without even a refusal to read it from. A state that+fails is one a `Catch` fires on, which is the half of the divergence worth keeping.## Nested workflowsdiff --git a/src/main/java/io/github/hectorvent/floci/services/stepfunctions/FormatNumberPicture.java b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/FormatNumberPicture.javanew file mode 100644index 0000000000..e8b4f6943a--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/FormatNumberPicture.java@@ -0,0 +1,238 @@+package io.github.hectorvent.floci.services.stepfunctions;++import com.dashjoin.jsonata.JException;++import java.util.Map;+import java.util.regex.Pattern;++/**+ * The picture string of {@code $formatNumber}, checked against the rules XPath F&O 4.7.3 gives+ * it, which is what AWS applies and the bundled JSONata library does not: it formats through+ * {@link java.text.DecimalFormat} and answers {@code "x1"} for {@code $formatNumber(1, "x")} where+ * AWS fails the state with D3086.+ *+ * <p>A picture may break more than one of the fourteen rules, and the code reported is the last+ * one, which is the one AWS reports: a picture with no digit at all breaks two at once, the+ * mantissa holding no digit (D3085) and the active part holding a passive character (D3086), and+ * D3086 is the answer.+ *+ * <p>The symbols the rules are written in terms of are the built-in ones, each replaceable through+ * the third argument of {@code $formatNumber}.+ */+final class FormatNumberPicture {++ private final char decimalSeparator;+ private final char exponentSeparator;+ private final char groupingSeparator;+ private final char digit;+ private final char patternSeparator;+ private final char percent;+ private final char perMille;+ private final char zeroDigit;++ private FormatNumberPicture(Map<?, ?> options) {+ decimalSeparator = symbol(options, "decimal-separator", '.');+ exponentSeparator = symbol(options, "exponent-separator", 'e');+ groupingSeparator = symbol(options, "grouping-separator", ',');+ digit = symbol(options, "digit", '#');+ patternSeparator = symbol(options, "pattern-separator", ';');+ percent = symbol(options, "percent", '%');+ perMille = symbol(options, "per-mille", '‰');+ zeroDigit = symbol(options, "zero-digit", '0');+ }++ private static char symbol(Map<?, ?> options, String name, char builtIn) {+ Object override = options == null ? null : options.get(name);+ return override instanceof String text && text.length() == 1 ? text.charAt(0) : builtIn;+ }++ /**+ * Fails with the code of the rule the picture breaks, or returns when it breaks none. The+ * picture is split into sub-pictures first, one for positive numbers and an optional second+ * one for negative numbers.+ */+ static void validate(String picture, Map<?, ?> options) {+ FormatNumberPicture symbols = new FormatNumberPicture(options);+ String[] subPictures =+ picture.split(Pattern.quote(String.valueOf(symbols.patternSeparator)), -1);+ if (subPictures.length > 2) {+ throw new JException("D3080", -1);+ }+ for (String subPicture : subPictures) {+ String error = symbols.errorIn(symbols.split(subPicture));+ if (error != null) {+ throw new JException(error, -1);+ }+ }+ }++ /**+ * The parts a sub-picture is read in. The prefix and the suffix are the passive characters at+ * either edge, so the active part is what sits between them; the mantissa is the active part+ * up to the exponent separator, and the integer and fractional parts are the mantissa either+ * side of the decimal separator.+ */+ private record Parts(String subPicture, String activePart, String mantissaPart,+ String integerPart, String fractionalPart, String exponentPart) { }++ private Parts split(String subPicture) {+ int activeStart = 0;+ while (activeStart < subPicture.length() && !isActiveOutsideExponent(subPicture.charAt(activeStart))) {+ activeStart++;+ }+ int activeEnd = subPicture.length();+ while (activeEnd > 0 && !isActiveOutsideExponent(subPicture.charAt(activeEnd - 1))) {+ activeEnd--;+ }+ // A sub-picture with no active character at all is all active part and has no edges.+ if (activeEnd == 0) {+ activeStart = 0;+ activeEnd = subPicture.length();+ }+ String activePart = subPicture.substring(activeStart, activeEnd);+ int exponentAt = subPicture.indexOf(exponentSeparator, activeStart) - activeStart;+ boolean hasExponent = exponentAt >= 0 && exponentAt <= activePart.length();+ String mantissaPart = hasExponent ? upTo(activePart, exponentAt) : activePart;+ String exponentPart = hasExponent ? from(activePart, exponentAt + 1) : null;+ int decimalAt = mantissaPart.indexOf(decimalSeparator);+ String integerPart = decimalAt < 0 ? mantissaPart : mantissaPart.substring(0, decimalAt);+ String fractionalPart =+ decimalAt < 0 ? subPicture.substring(activeEnd) : mantissaPart.substring(decimalAt + 1);+ return new Parts(subPicture, activePart, mantissaPart, integerPart, fractionalPart, exponentPart);+ }++ /** The exponent separator is located in the sub-picture, so it can point past the active part. */+ private static String upTo(String text, int end) {+ return text.substring(0, Math.min(end, text.length()));+ }++ private static String from(String text, int start) {+ return text.substring(Math.min(start, text.length()));+ }++ /**+ * The code of the last rule the sub-picture breaks, or null when it breaks none. The rules are+ * read in the order the codes number them, and a later one replaces what an earlier one found.+ */+ private String errorIn(Parts parts) {+ String error = separatorCountError(parts.subPicture());+ error = later(error, digitError(parts));+ error = later(error, groupingSeparatorError(parts));+ error = later(error, optionalDigitError(parts));+ error = later(error, exponentError(parts));+ return error;+ }++ private static String later(String error, String found) {+ return found != null ? found : error;+ }++ /** D3081 to D3084: a symbol that may appear once appears twice, or two that exclude each other. */+ private String separatorCountError(String subPicture) {+ String error = null;+ if (subPicture.indexOf(decimalSeparator) != subPicture.lastIndexOf(decimalSeparator)) {+ error = "D3081";+ }+ if (subPicture.indexOf(percent) != subPicture.lastIndexOf(percent)) {+ error = "D3082";+ }+ if (subPicture.indexOf(perMille) != subPicture.lastIndexOf(perMille)) {+ error = "D3083";+ }+ if (subPicture.indexOf(percent) >= 0 && subPicture.indexOf(perMille) >= 0) {+ error = "D3084";+ }+ return error;+ }++ /** D3085 and D3086: the mantissa holds no digit, and the active part holds a passive character. */+ private String digitError(Parts parts) {+ String error = null;+ if (!holdsDigit(parts.mantissaPart())) {+ error = "D3085";+ }+ if (!parts.activePart().chars().allMatch(character -> isActive((char) character))) {+ error = "D3086";+ }+ return error;+ }++ /** D3087 to D3089: a grouping separator next to the decimal separator, at the end, or doubled. */+ private String groupingSeparatorError(Parts parts) {+ String subPicture = parts.subPicture();+ int decimalAt = subPicture.indexOf(decimalSeparator);+ String error = null;+ if (decimalAt >= 0 && (characterAt(subPicture, decimalAt - 1) == groupingSeparator+ || characterAt(subPicture, decimalAt + 1) == groupingSeparator)) {+ error = "D3087";+ } else if (decimalAt < 0 && !parts.integerPart().isEmpty()+ && parts.integerPart().charAt(parts.integerPart().length() - 1) == groupingSeparator) {+ error = "D3088";+ }+ if (subPicture.indexOf("" + groupingSeparator + groupingSeparator) >= 0) {+ error = "D3089";+ }+ return error;+ }++ /**+ * D3090 and D3091: an optional digit with a mandatory one on the side the number grows from,+ * which is to its left in the integer part and to its right in the fractional part.+ */+ private String optionalDigitError(Parts parts) {+ String error = null;+ int firstOptional = parts.integerPart().indexOf(digit);+ if (firstOptional >= 0 && holdsDecimalDigit(parts.integerPart().substring(0, firstOptional))) {+ error = "D3090";+ }+ int lastOptional = parts.fractionalPart().lastIndexOf(digit);+ if (lastOptional >= 0 && holdsDecimalDigit(parts.fractionalPart().substring(lastOptional))) {+ error = "D3091";+ }+ return error;+ }++ /** D3092 and D3093: an exponent alongside a percent, and an exponent that is not all digits. */+ private String exponentError(Parts parts) {+ if (parts.exponentPart() == null) {+ return null;+ }+ boolean scaled = parts.subPicture().indexOf(percent) >= 0+ || parts.subPicture().indexOf(perMille) >= 0;+ String error = null;+ if (!parts.exponentPart().isEmpty() && scaled) {+ error = "D3092";+ }+ if (parts.exponentPart().isEmpty()+ || !parts.exponentPart().chars().allMatch(character -> isDecimalDigit((char) character))) {+ error = "D3093";+ }+ return error;+ }++ private static char characterAt(String text, int index) {+ return index >= 0 && index < text.length() ? text.charAt(index) : '\0';+ }++ private boolean holdsDigit(String text) {+ return text.chars().anyMatch(character -> isDecimalDigit((char) character) || character == digit);+ }++ private boolean holdsDecimalDigit(String text) {+ return text.chars().anyMatch(character -> isDecimalDigit((char) character));+ }++ private boolean isDecimalDigit(char character) {+ return character >= zeroDigit && character < zeroDigit + 10;+ }++ private boolean isActive(char character) {+ return isDecimalDigit(character) || character == decimalSeparator+ || character == exponentSeparator || character == groupingSeparator+ || character == digit || character == patternSeparator;+ }++ private boolean isActiveOutsideExponent(char character) {+ return isActive(character) && character != exponentSeparator;+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluator.java b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluator.javaindex 61209dbeff..63f84d5544 100644--- a/src/main/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluator.java+++ b/src/main/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluator.java@@ -3,6 +3,7 @@ package io.github.hectorvent.floci.services.stepfunctions;import com.dashjoin.jsonata.Functions;import com.dashjoin.jsonata.JException;import com.dashjoin.jsonata.Jsonata;+import com.dashjoin.jsonata.Utils;import com.fasterxml.jackson.core.JsonParser;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.DeserializationFeature;@@ -13,6 +14,7 @@ import com.fasterxml.jackson.databind.node.ArrayNode;import com.fasterxml.jackson.databind.node.MissingNode;import com.fasterxml.jackson.databind.node.NullNode;import com.fasterxml.jackson.databind.node.ObjectNode;+import com.fasterxml.jackson.databind.node.TextNode;import jakarta.enterprise.context.ApplicationScoped;import jakarta.inject.Inject;@@ -21,6 +23,7 @@ import java.math.BigInteger;import java.nio.charset.StandardCharsets;import java.security.MessageDigest;import java.security.NoSuchAlgorithmException;+import java.text.ParseException;import java.util.ArrayList;import java.util.HexFormat;import java.util.Iterator;@@ -91,14 +94,54 @@ public class JsonataEvaluator {static final long EVALUATION_TIMEOUT_MILLIS = 5_000;/**- * How deep evaluation may nest before the state fails. A non-tail-recursive function nests at- * roughly {@code 3n+5} for {@code $f(n)}; measured against AWS with {@code test-state}, AWS- * accepts {@code n=30} (depth 95) and refuses {@code n=40} (depth 125), so its own ceiling- * sits near 100. Five times that leaves no expression AWS accepts able to reach this bound,- * and it still trips before the JVM stack does: on a 1 MB thread stack the bound raises a- * JSONata error while 1000 or more overflows the stack instead.+ * How deep evaluation may nest before the state fails, which is the ceiling AWS itself holds:+ * AWS accepts an evaluation at depth 100 and refuses one at 101, whatever the nesting is made+ * of. An addition chain nests one level per term, parentheses, brackets and the chain operator+ * nest one level more than their count, and a non-tail-recursive {@code $f(n)} nests+ * {@code 3n+5}.+ *+ * <p>The bound trips before the JVM stack does: on a 1 MB thread stack it raises a JSONata+ * error, which the state fails on, where 1000 or more overflows the stack instead.+ */+ static final int MAX_EVALUATION_DEPTH = 100;++ /** What one number, boolean or null of a value counts as against {@link #MAX_EXPRESSION_BYTES}. */+ private static final int BYTES_PER_SCALAR = 8;++ /**+ * How much memory one value an expression builds may hold before the state fails, 6,990,256+ * bytes, which is the bound AWS itself holds: AWS accepts {@code [1..873782]} and refuses one+ * element more, and it accepts a string of 2^22 characters and refuses one of 2^23. Both of+ * those refusals land just past this bound when a number counts as {@link #BYTES_PER_SCALAR}+ * bytes and a character as one.+ *+ * <p>What is bounded is the largest single value an evaluation step produces, not the sum of+ * everything alive at once. Checking it walks that value and stops as soon as the walk passes+ * the bound, so one check costs at most 873,783 steps, and a lazy range is counted from its+ * length without being materialised.+ */+ static final long MAX_EXPRESSION_BYTES = 873_782L * BYTES_PER_SCALAR;++ /**+ * The largest number of elements a single {@code $range} call may produce before floci+ * refuses it, the bound AWS itself holds specifically for {@code $range}: AWS accepts+ * {@code $range(1, 360145, 1)}, returning all 360,145 elements, and refuses one element more+ * with the same memory refusal as {@link #MAX_EXPRESSION_BYTES}. That general bound is looser+ * (873,782 scalars) and is checked only on the value an evaluation step has already built; a+ * lazy literal range such as {@code [1..873782]} is exempt from it on AWS, and stays exempt+ * here. {@code $range} gets a bound of its own because it is the one construct that actually+ * allocates a list of that size, so the general check would see the memory only after the+ * allocation already happened.+ */+ static final int MAX_RANGE_ELEMENTS = 360_145;++ /**+ * AWS's own sentence for an expression that exceeded its evaluation memory limit, shared by+ * the bound on a built value ({@link #boundHeldMemory}) and the pre-allocation cap on+ * {@code $range} ({@link #range}).*/- static final int MAX_EVALUATION_DEPTH = 500;+ private static final String MEMORY_LIMIT_EXCEEDED =+ "Expression evaluation memory limit exceeded: Check for excessive memory usage";private final ObjectMapper objectMapper;private final ObjectReader strictJsonReader;@@ -161,6 +204,7 @@ public class JsonataEvaluator {JsonNode evaluate(String expression, JsonNode statesVar, JsonNode variables) {String expr = isExpression(expression) ? unwrap(expression) : expression;+ ExpressionNesting nesting = new ExpressionNesting();try {Jsonata jsonataExpr = jsonata(expr);// Off, the library keeps JSONata's null marker in the result instead of flattening it@@ -173,7 +217,12 @@ public class JsonataEvaluator {// case loops for ever and the execution never leaves RUNNING. The bounds raise a// JSONata error, which the catch below turns into States.QueryEvaluationError.frame.setRuntimeBounds(evaluationTimeoutMillis, maxEvaluationDepth);+ nesting.trackOn(frame);+ boundHeldMemory(frame);stepFunctionsExtensions.forEach(frame::bind);+ // AWS disables $eval, and the library ships it. An unbound name called as a function is+ // the library's own T1006, which is the code AWS answers. $parse is the replacement.+ frame.bind("eval", (Object) null);// Workflow variables (the Assign field) are referenced as top-level $name in AWS's// JSONata dialect, e.g. $CheckpointCount. They are bound alongside $states, never// inside it: AWS reserves $states for input/result/errorOutput/context only. $states is@@ -190,36 +239,165 @@ public class JsonataEvaluator {Object result = jsonataExpr.evaluate(null, frame);return toJsonNode(result);} catch (Exception e) {- throw new AslExecutor.FailStateException("States.QueryEvaluationError", queryEvaluationCause(e));+ JsonNode nonFinite = nesting.isTheWholeExpression() ? nonFiniteRefusal(e) : null;+ if (nonFinite != null) {+ return nonFinite;+ }+ throw queryEvaluationFailure(e);+ } catch (StackOverflowError | OutOfMemoryError e) {+ // These two are the resource an expression itself spends: nesting deep enough to+ // exhaust the parser's stack, and asking for a value larger than the JVM can hold.+ // AWS bounds both and fails the state with States.QueryEvaluationError, so the state's+ // own Retry and Catch (and a Catch on States.ALL) still see it. Every other Error says+ // the runtime is broken rather than this expression, and it keeps going to+ // AslExecutor's terminal catch (Error e), which fails the execution as States.Runtime+ // and rethrows. The message is often null here, so toString() carries the diagnosis,+ // the same way that terminal catch already renders it. QueryEvaluationFailure (rather+ // than a bare FailStateException) is what lets evaluateField prepend AWS's sentence,+ // colon-separated since this cause carries no JSONata code.+ throw new QueryEvaluationFailure(e.toString(), ": ");+ }+ }++ /**+ * How deeply nested the evaluation step running is, the expression's own root node being 1.+ * A step that fails never leaves, so once an exception has unwound the count is the depth of+ * the step that raised it.+ */+ private static final class ExpressionNesting {++ private int depth;++ /**+ * Counts on the same pair of callbacks the library installs its bounds through, keeping+ * the library's own and running after it.+ */+ void trackOn(Jsonata.Frame frame) {+ Jsonata.EntryCallback boundsOnEntry = (Jsonata.EntryCallback) frame.lookup("__evaluate_entry");+ frame.setEvaluateEntryCallback((expression, input, environment) -> {+ boundsOnEntry.callback(expression, input, environment);+ depth++;+ });+ afterEachEvaluationStep(frame, (expression, input, environment, result) -> depth--);+ }++ /** Whether the step that failed is the expression itself rather than a step inside it. */+ boolean isTheWholeExpression() {+ return depth == 1;}}/**- * The cause AWS reports for a failed JSONata expression: the error code, then the message- * jsonata-js renders for it, as in {@code T0412: Argument 1 of function "sum" must be an array- * of "numbers"}. AWS puts a sentence naming the expression and the field in front of that,- * which needs the field path {@link #resolveTemplate} does not thread today (#2665).+ * The value behind a D1001, which the library raises rather than carrying a non-finite number:+ * {@code 1/0} and {@code 1e308*10} both fail there where AWS answers {@code "Infinity"}. The+ * exception is the only place the library leaves the value, and it carries no way back to where+ * the arithmetic ran, so this recovers the value only when the step that raised it is the whole+ * expression. {@code [1/0]} fails the state instead of answering AWS's {@code ["Infinity"]},+ * which is the conservative half of the divergence: a state that fails is one a Catch fires on.+ */+ private JsonNode nonFiniteRefusal(Exception e) {+ if (e instanceof JException jsonataError && "D1001".equals(jsonataError.getError())+ && jsonataError.getCurrent() instanceof Number value) {+ return fromJsonataValue(value.doubleValue());+ }+ return null;+ }++ /**+ * Adds the memory bound to the time and depth bounds the library installs. The exit callback+ * carries the value every evaluation step produces, which is what the bound is on.+ */+ private static void boundHeldMemory(Jsonata.Frame frame) {+ afterEachEvaluationStep(frame, (expression, input, environment, result) -> {+ if (heldBytes(result, 0) > MAX_EXPRESSION_BYTES) {+ throw new JException(MEMORY_LIMIT_EXCEEDED, -1);+ }+ });+ }++ /**+ * Runs {@code addition} after every evaluation step, keeping whatever the frame already runs+ * there. The library installs its time and depth bounds through that same callback, so an+ * addition extends them rather than replacing them.+ */+ private static void afterEachEvaluationStep(Jsonata.Frame frame, Jsonata.ExitCallback addition) {+ Jsonata.ExitCallback installed = (Jsonata.ExitCallback) frame.lookup("__evaluate_exit");+ frame.setEvaluateExitCallback((expression, input, environment, result) -> {+ installed.callback(expression, input, environment, result);+ addition.callback(expression, input, environment, result);+ });+ }++ /**+ * What {@code value} adds to the {@code held} bytes already counted, stopping as soon as the+ * total passes {@link #MAX_EXPRESSION_BYTES} so a value far over the bound costs no more to+ * refuse than one just over it. A range is counted from its length: the library materialises it+ * lazily, and AWS refuses it on the elements it stands for rather than on what it has built.+ */+ private static long heldBytes(Object value, long held) {+ if (held > MAX_EXPRESSION_BYTES) {+ return held;+ }+ if (value instanceof String text) {+ return held + text.length();+ }+ if (value instanceof Utils.RangeList range) {+ return held + (long) range.size() * BYTES_PER_SCALAR;+ }+ if (value instanceof List<?> list) {+ return elementsHeldBytes(list, held);+ }+ if (value instanceof Map<?, ?> map) {+ return elementsHeldBytes(map.entrySet(), held);+ }+ if (value instanceof Map.Entry<?, ?> entry) {+ return heldBytes(entry.getValue(), heldBytes(entry.getKey(), held));+ }+ return held + BYTES_PER_SCALAR;+ }++ private static long elementsHeldBytes(Iterable<?> elements, long held) {+ long total = held;+ for (Object element : elements) {+ total = heldBytes(element, total);+ if (total > MAX_EXPRESSION_BYTES) {+ return total;+ }+ }+ return total;+ }++ /**+ * The state failure AWS reports for an expression that threw. Its cause is the error code and+ * the message jsonata-js renders for it, as in {@code T0412: Argument 1 of function "sum" must+ * be an array of "numbers"}; for the handful of failures with no JSONata code of their own (a+ * timeout, a stack-depth bound) it is the bare message the exception came with.*- * <p>The dashjoin port renders the message itself, and three things go wrong on the way. Its- * copy of the catalog has the word "function" replaced by "Object" throughout; its substituter- * fills the first two placeholders of a template and leaves a third one standing; and it quotes- * every value it inserts, where jsonata-js JSON-renders it. So the message is composed here- * from the code and the values {@link JException} carries instead.+ * <p>The dashjoin port renders the coded message itself, and three things go wrong on the way.+ * Its copy of the catalog has the word "function" replaced by "Object" throughout; its+ * substituter fills the first two placeholders of a template and leaves a third one standing;+ * and it quotes every value it inserts, where jsonata-js JSON-renders it. So the message is+ * composed here from the code and the values {@link JException} carries instead.*/- private String queryEvaluationCause(Exception e) {+ private QueryEvaluationFailure queryEvaluationFailure(Exception e) {if (!(e instanceof JException jsonataError) || jsonataError.getError() == null) {- return e.getMessage();+ return new QueryEvaluationFailure(e.getMessage(), ": ");}String code = jsonataError.getError();Object current = jsonataError.getCurrent();Object expected = jsonataError.getExpected();// Both this class's own functions and a few library call sites throw with the whole// message in the code slot. The catalog lookup then misses and dashjoin prefixes its own- // class name, which AWS never emits; the message is the code slot itself.+ // class name, which AWS never emits; the message is the code slot itself. Most of these+ // already open with a real code ("D3137: Invalid JSON"), a sentence of its own that joins+ // AWS's with ". "; the two bound messages (a timeout, a stack-depth or memory ceiling) open+ // with English instead and are bare, joining with ": " like any other uncoded message.if ((UNKNOWN_CODE_PREFIX + code).equals(jsonataError.getMessage())) {- return code;+ String bound = withoutDepthBoundSuffix(code);+ String separator = bound.matches("[A-Z]\\d{4}:.*") ? ". " : ": ";+ return new QueryEvaluationFailure(bound, separator);}- return switch (code) {+ String tail = switch (code) {// The only two templates in the catalog with a third placeholder. Neither third value// reaches the exception: T0412 carries the offending argument and the element type it// wanted but not the argument index nor the function name, and T2009 carries the two@@ -230,6 +408,32 @@ public class JsonataEvaluator {+ " either side of the operator must be of the same data type";default -> code + ": " + renderTemplate(code, current, expected);};+ return new QueryEvaluationFailure(tail, ". ");+ }++ /**+ * The library's stack-depth bound names the depth and the ceiling it hit,+ * {@code ". Depth=101 max=100"}, which AWS's own sentence for the same failure never carries.+ * Every other bare message this method renders has no such suffix, so stripping it here is+ * specific to that one failure without needing to name it.+ */+ private static String withoutDepthBoundSuffix(String message) {+ return message.replaceFirst("\\. Depth=\\d+ max=\\d+$", "");+ }++ /**+ * A States.QueryEvaluationError the expression itself raised, carrying the punctuation that+ * joins its cause to the sentence {@link #evaluateField} puts in front: {@code ". "} after a+ * coded JSONata message, which is a sentence of its own, and {@code ": "} before a bare one.+ */+ private static final class QueryEvaluationFailure extends FailStateException {++ final String sentenceSeparator;++ QueryEvaluationFailure(String cause, String sentenceSeparator) {+ super("States.QueryEvaluationError", cause);+ this.sentenceSeparator = sentenceSeparator;+ }}/**@@ -256,19 +460,29 @@ public class JsonataEvaluator {}/**- * Evaluate the single expression an ASL field holds, failing the state when it returns nothing.- * AWS names the field in the cause and a {@code Catch} on States.QueryEvaluationError fires:- * a Wait whose Seconds returned nothing does not wait zero seconds, it fails.+ * Evaluate the single expression an ASL field holds, failing the state when it returns nothing+ * or when the expression itself throws. AWS names the expression and the field in the cause+ * either way and a {@code Catch} on States.QueryEvaluationError fires: a Wait whose Seconds+ * returned nothing does not wait zero seconds, it fails, and {@code $abs("x")} in a field fails+ * it with AWS's "threw an error during evaluation" sentence in front of the JSONata message.** <p>{@code field} is the field's path relative to the state, as AWS writes it: {@code Seconds},* {@code Choices[1]/Condition}, {@code Output/a/b}.*/JsonNode evaluateField(String expression, String field, JsonNode statesVar, JsonNode variables) {- JsonNode value = evaluate(expression, statesVar, variables);+ String expr = isExpression(expression) ? unwrap(expression) : expression;+ JsonNode value;+ try {+ value = evaluate(expression, statesVar, variables);+ } catch (QueryEvaluationFailure failure) {+ throw new FailStateException(failure.error,+ "The JSONata expression '" + expr + "' specified for the field '" + field+ + "' threw an error during evaluation" + failure.sentenceSeparator + failure.cause);+ }if (value.isMissingNode()) {throw new FailStateException("States.QueryEvaluationError",- "The JSONata expression '" + (isExpression(expression) ? unwrap(expression) : expression)- + "' specified for the field '" + field + "' returned nothing (undefined).");+ "The JSONata expression '" + expr + "' specified for the field '" + field+ + "' returned nothing (undefined).");}return value;}@@ -356,16 +570,31 @@ public class JsonataEvaluator {list.forEach(element -> array.add(fromJsonataValue(element)));return array;}+ // JSON has no literal for a non-finite number, so AWS writes each one as the string+ // JavaScript names it by: 1/0 comes back as "Infinity" and $parseInteger("abc","0") as+ // "NaN".+ if (value instanceof Double number && !Double.isFinite(number)) {+ return TextNode.valueOf(nonFiniteName(number));+ }return objectMapper.valueToTree(value);}+ private static String nonFiniteName(double value) {+ if (Double.isNaN(value)) {+ return "NaN";+ }+ return value > 0 ? "Infinity" : "-Infinity";+ }+/*** The JSONata functions Floci binds on the evaluation frame. Six of them are what the Step* Functions dialect adds on top of the JSONata language, per* https://docs.aws.amazon.com/step-functions/latest/dg/transforming-data.html: five the* dashjoin library does not have at all, and $random it has with the wrong arity, so the- * binding shadows it to accept AWS's optional seed. The seventh, $string, is a JSONata- * function the library has but writes with a different number notation than AWS.+ * binding shadows it to accept AWS's optional seed. The other three are JSONata functions the+ * library has and answers differently from AWS: $string writes a number in another notation,+ * $parseInteger answers nothing where AWS answers NaN, and $formatNumber accepts picture+ * strings AWS refuses.** <p>Each of the six declares one optional slot more than the arity AWS documents. The library* pads a short call with Java nulls up to the declared arity and refuses a call longer than@@ -385,7 +614,42 @@ public class JsonataEvaluator {"hash", new Jsonata.JFunction((input, arguments) -> hash(arguments), "<x?x?x?:x>"),"random", new Jsonata.JFunction((input, arguments) -> random(arguments), "<x?x?:x>"),"uuid", new Jsonata.JFunction((input, arguments) -> uuid(arguments), "<x?:x>"),- "string", new Jsonata.JFunction((input, arguments) -> string(arguments), "<x-b?:s>"));+ "string", new Jsonata.JFunction((input, arguments) -> string(arguments), "<x-b?:s>"),+ "parseInteger", new Jsonata.JFunction((input, arguments) -> parseInteger(arguments), "<s-s:n>"),+ "formatNumber", new Jsonata.JFunction((input, arguments) -> formatNumber(arguments), "<n-so?:s>"));+ }++ /**+ * $parseInteger(value, picture): the JSONata function, answering NaN where the picture cannot+ * read the value, which is what AWS answers. The library answers nothing there, and an ASL+ * field that evaluates to nothing fails the state.+ */+ private static Object parseInteger(List<Object> arguments) {+ Object value = argument(arguments, 0);+ if (value == null) {+ return null;+ }+ try {+ Number parsed = Functions.parseInteger((String) value, (String) argument(arguments, 1));+ return parsed == null ? Double.NaN : parsed;+ } catch (ParseException e) {+ throw new JException(e.getMessage(), -1);+ }+ }++ /**+ * $formatNumber(value, picture, options): the JSONata function, with the picture string checked+ * against the rules AWS applies to it first. See {@link FormatNumberPicture}.+ */+ private static Object formatNumber(List<Object> arguments) {+ Object value = argument(arguments, 0);+ if (value == null) {+ return null;+ }+ Map<?, ?> options = (Map<?, ?>) argument(arguments, 2);+ String picture = (String) argument(arguments, 1);+ FormatNumberPicture.validate(picture, options);+ return Functions.formatNumber((Number) value, picture, options);}/**@@ -607,6 +871,9 @@ public class JsonataEvaluator {.subtract(BigInteger.valueOf(start)).divide(BigInteger.valueOf(step)).add(BigInteger.ONE);+ if (elements.compareTo(BigInteger.valueOf(MAX_RANGE_ELEMENTS)) > 0) {+ throw new JException(MEMORY_LIMIT_EXCEEDED, -1);+ }List<Object> values = new ArrayList<>();long value = start;for (BigInteger emitted = BigInteger.ZERO;diff --git a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluatorTest.java b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluatorTest.javaindex 8715ea7ea2..b29e50be70 100644--- a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluatorTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/JsonataEvaluatorTest.java@@ -10,6 +10,7 @@ import org.junit.jupiter.params.ParameterizedTest;import org.junit.jupiter.params.provider.Arguments;import org.junit.jupiter.params.provider.MethodSource;+import java.util.Map;import java.util.stream.Stream;import static org.junit.jupiter.api.Assertions.*;@@ -349,6 +350,50 @@ class JsonataEvaluatorTest {assertTrue(ex.cause.contains("T0410"));}+ /**+ * The AWS bound on how many elements a single {@code $range} call may produce, measured+ * directly (us-east-1, test-state, 2026-08-30): AWS accepts {@code $range(1, 360145, 1)},+ * returning all 360,145 elements.+ */+ @Test+ void rangeAtTheAwsElementBoundBuildsTheFullArray() {+ JsonNode statesVar = objectMapper.createObjectNode();+ JsonNode result = evaluator.evaluate("{% $count($range(1, 360145, 1)) %}", statesVar);+ assertEquals(360145, result.asInt());+ }++ /**+ * One element past the bound, AWS fails the state with {@code States.QueryEvaluationError}+ * and this cause byte for byte (us-east-1, test-state, 2026-08-30).+ */+ @Test+ void rangeOneElementPastTheAwsElementBoundFailsWithTheMemoryRefusal() {+ JsonNode statesVar = objectMapper.createObjectNode();+ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluateField("{% $range(1, 360146, 1) %}", "Output/v", statesVar, null));+ assertEquals("States.QueryEvaluationError", failure.error);+ assertEquals("The JSONata expression '$range(1, 360146, 1)' specified for the field 'Output/v' threw "+ + "an error during evaluation: Expression evaluation memory limit exceeded: Check for excessive "+ + "memory usage", failure.cause);+ }++ /**+ * Far past the bound, the refusal has to come from the pre-allocation cap in {@code range()}+ * rather than from the memory bound on the built value: an unbounded call would spend+ * gigabytes building the array before that check ever runs.+ */+ @Test+ @Timeout(60)+ void rangeFarPastTheBoundFailsWithoutAllocating() {+ JsonNode statesVar = objectMapper.createObjectNode();+ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluateField("{% $range(1, 1000000000, 1) %}", "Output/v", statesVar, null));+ assertEquals("States.QueryEvaluationError", failure.error);+ assertEquals("The JSONata expression '$range(1, 1000000000, 1)' specified for the field 'Output/v' threw "+ + "an error during evaluation: Expression evaluation memory limit exceeded: Check for excessive "+ + "memory usage", failure.cause);+ }+@Testvoid hash_computesKnownDigestsForEachAlgorithm() {JsonNode statesVar = objectMapper.createObjectNode();@@ -519,6 +564,33 @@ class JsonataEvaluatorTest {assertEquals("1.5", evaluator.evaluate("{% $parse('1.5') %}", statesVar).toString());}+ /**+ * The execution input reaching {@code $states.input} follows the same number model as+ * {@code $parse} (see {@link #parse_dropsATrailingZeroOnAnIntegerValuedNumber}): both route+ * through {@code toJsonataValue}/{@code toJsonataNumber}. Measured against real AWS+ * (us-east-1, test-state, 2026-08-30): a Pass state with+ * {@code Output: {"v": "{% $states.input.amount %}"}} and input {@code {"amount": 1.0}}+ * answers {@code {"v": 1}}.+ */+ @Test+ void statesInputDropsATrailingZeroOnAnIntegerValuedNumber() throws Exception {+ JsonNode statesVar = objectMapper.readTree("{\"input\": {\"amount\": 1.0}}");+ assertEquals("1", evaluator.evaluate("{% $states.input.amount %}", statesVar).toString());+ }++ /**+ * Same model, the other side of the long boundary (see+ * {@link #parse_keepsAnIntegerExactWhileItFitsInALong}). Measured against real AWS+ * (us-east-1, test-state, 2026-08-30): input {@code {"amount": 9223372036854775808}} answers+ * {@code {"v": 9.223372036854776E18}}.+ */+ @Test+ void statesInputSwitchesToADoublePastTheLongBoundary() throws Exception {+ JsonNode statesVar = objectMapper.readTree("{\"input\": {\"amount\": 9223372036854775808}}");+ assertEquals("9.223372036854776E18",+ evaluator.evaluate("{% $states.input.amount %}", statesVar).toString());+ }+@Testvoid parse_rejectsJsonAwsRejects() {JsonNode statesVar = objectMapper.createObjectNode();@@ -582,9 +654,9 @@ class JsonataEvaluatorTest {* without a depth bound it ends in a StackOverflowError, which is an Error and so escapes the* evaluator's catch. The shipped bound is what turns it into a state failure.*- * <p>The cause is AWS's own string for this failure, followed by the {@code Depth=N max=M}- * suffix that names the bound that was hit. AWS writes one space after the full stop where- * the library writes two.+ * <p>The cause is the library's own string for this failure, with its trailing {@code Depth=N+ * max=M} suffix stripped: AWS never emits it. The library writes two spaces after the full+ * stop where AWS writes one; that divergence is left as is.*/@Test@Timeout(30)@@ -597,18 +669,13 @@ class JsonataEvaluatorTest {assertEquals("States.QueryEvaluationError", ex.error);assertEquals("Stack overflow error: Check for non-terminating recursive"- + " function. Consider rewriting as tail-recursive. Depth=501 max=500", ex.cause);+ + " function. Consider rewriting as tail-recursive", ex.cause);}/*** The boundary from below: the risk of a bound is that it fails a working state machine, so- * these must all still evaluate. Measured with {@code aws stepfunctions test-state}, AWS- * returns 30 for the first and 800000 for the second, which makes them the ceiling of what- * AWS itself accepts on each axis: it refuses the recursion at n=40 (nesting depth 125) and- * the sequence at 900,000 elements.- *- * <p>The last two AWS refuses on its memory limit and Floci evaluates, which is where the- * bounds are deliberately looser than AWS rather than tighter.+ * these must all still evaluate. AWS accepts every one of them and refuses the next value on+ * each axis: the recursion at n=32, the sequence at 873,783 elements.*/@Test@Timeout(60)@@ -617,13 +684,251 @@ class JsonataEvaluatorTest {String recursionHead = "{% ($f := function($x){ $x <= 0 ? 0 : 1 + $f($x-1) }; $f(";assertEquals(30, evaluator.evaluate(recursionHead + "30)) %}", statesVar).asInt());+ assertEquals(31, evaluator.evaluate(recursionHead + "31)) %}", statesVar).asInt());assertEquals(800000, evaluator.evaluate("{% [1..800000] ~> $count() %}", statesVar).asInt());-- assertEquals(160, evaluator.evaluate(recursionHead + "160)) %}", statesVar).asInt());assertEquals(40000200000L,evaluator.evaluate("{% $sum($map([1..200000], function($v){$v * 2})) %}", statesVar).asLong());}+ /**+ * The four nesting shapes of issue #2737, at the last {@code n} AWS accepts and at the first+ * it refuses. An addition chain nests one level per term, so AWS accepts 100 terms;+ * parentheses, brackets and the chain operator nest one level more than their count, so AWS+ * accepts 99 of each. Both put AWS's ceiling at+ * nesting depth 100, which is what {@link JsonataEvaluator#MAX_EVALUATION_DEPTH} holds.+ */+ @Test+ @Timeout(30)+ void everyNestingShapeFailsAtTheDepthAwsFailsAt() {+ JsonNode statesVar = objectMapper.createObjectNode();++ assertEquals(100, evaluator.evaluate("{% 1" + "+1".repeat(99) + " %}", statesVar).asInt());+ assertEquals(1, evaluator.evaluate("{% " + "(".repeat(99) + "1" + ")".repeat(99) + " %}", statesVar).asInt());+ assertTrue(evaluator.evaluate("{% " + "[".repeat(99) + "1" + "]".repeat(99) + " %}",+ statesVar).isArray());+ assertEquals("1", evaluator.evaluate("{% 1" + "~>$string".repeat(99) + " %}", statesVar).asText());++ for (String tooDeep : new String[]{+ "1" + "+1".repeat(100),+ "(".repeat(100) + "1" + ")".repeat(100),+ "[".repeat(100) + "1" + "]".repeat(100),+ "1" + "~>$string".repeat(100)}) {+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate("{% " + tooDeep + " %}", statesVar),+ "expected the depth bound to refuse " + tooDeep.substring(0, 20) + "...");+ assertEquals("States.QueryEvaluationError", ex.error);+ assertTrue(ex.cause.endsWith("Consider rewriting as tail-recursive"), ex.cause);+ }+ }++ /**+ * The non-tail-recursive shape of issue #2737, which nests three levels per call: AWS accepts+ * {@code $f(31)} and refuses {@code $f(32)}, and the depth bound reproduces both.+ */+ @Test+ @Timeout(30)+ void nonTailRecursionFailsAtTheCallDepthAwsFailsAt() {+ JsonNode statesVar = objectMapper.createObjectNode();+ String recursionHead = "{% ($f := function($x){ $x <= 0 ? 0 : 1 + $f($x-1) }; $f(";++ assertEquals(31, evaluator.evaluate(recursionHead + "31)) %}", statesVar).asInt());++ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate(recursionHead + "32)) %}", statesVar));+ assertEquals("States.QueryEvaluationError", ex.error);+ assertTrue(ex.cause.endsWith("Consider rewriting as tail-recursive"), ex.cause);+ }++ /**+ * The memory bound of issue #2737 on the sequence AWS draws it on: AWS accepts+ * {@code [1..873782]} and refuses one element more, and counting a number as eight bytes puts+ * those two on either side of {@link JsonataEvaluator#MAX_EXPRESSION_BYTES} here as well.+ */+ @Test+ @Timeout(60)+ void theLargestSequenceAwsAcceptsEvaluatesAndTheNextOneFailsOnTheMemoryBound() {+ JsonNode statesVar = objectMapper.createObjectNode();++ assertEquals(873782, evaluator.evaluate("{% [1..873782] ~> $count() %}", statesVar).asInt());++ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate("{% [1..873783] ~> $count() %}", statesVar));+ assertEquals("States.QueryEvaluationError", ex.error);+ assertEquals("Expression evaluation memory limit exceeded: Check for excessive memory usage", ex.cause);+ }++ /**+ * The same memory bound on a string: a string that doubles 22 times evaluates on AWS and one+ * that doubles 23 times does not. A character counts as one byte, which leaves 2^22 characters+ * inside the bound and 2^23 outside it, as on AWS.+ */+ @Test+ @Timeout(60)+ void aStringDoublingStopsAtTheSameBoundAwsStopsAt() {+ JsonNode statesVar = objectMapper.createObjectNode();+ String doublingHead = "{% ($f := function($s, $n){ $n = 0 ? $s : $f($s & $s, $n - 1) };"+ + " $length($f('x', ";++ assertEquals(4194304, evaluator.evaluate(doublingHead + "22))) %}", statesVar).asInt());++ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate(doublingHead + "23))) %}", statesVar));+ assertEquals("States.QueryEvaluationError", ex.error);+ assertEquals("Expression evaluation memory limit exceeded: Check for excessive memory usage", ex.cause);+ }++ /**+ * The two expressions of issue #2737 that AWS refuses on its memory limit. They fail the state+ * here too, so a Catch fires on them rather than the state succeeding with a value.+ */+ @Test+ @Timeout(60)+ void theSequencesAwsRefusesOnItsMemoryLimitFailTheStateHere() {+ JsonNode statesVar = objectMapper.createObjectNode();++ for (String expression : new String[]{"[1..900000] ~> $count()", "$sum([1..900000])"}) {+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate("{% " + expression + " %}", statesVar),+ "expected the memory bound to refuse " + expression);+ assertEquals("States.QueryEvaluationError", ex.error);+ assertEquals("Expression evaluation memory limit exceeded: Check for excessive memory usage", ex.cause);+ }+ }++ /**+ * AWS disables $eval and answers T1006 for a call to it. The library ships $eval, so the name+ * is left unbound here and a call to it answers the same code; $parse is the replacement AWS+ * documents.+ */+ @Test+ void evalIsNotBoundBecauseAwsDisablesIt() {+ JsonNode statesVar = objectMapper.createObjectNode();++ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate("{% $eval('1+1') %}", statesVar));++ assertEquals("States.QueryEvaluationError", ex.error);+ assertEquals("T1006: Attempted to invoke a non-function", ex.cause);+ }++ /**+ * JSON has no literal for a non-finite number, so AWS writes each one as the string JavaScript+ * names it by. The library refuses to carry the value at all and raises D1001 instead, and the+ * value that refusal carries is what these answer.+ */+ @Test+ void aNonFiniteNumberIsTheStringAwsWritesForIt() {+ JsonNode statesVar = objectMapper.createObjectNode();++ assertEquals("Infinity", evaluator.evaluate("{% 1/0 %}", statesVar).asText());+ assertEquals("Infinity", evaluator.evaluate("{% 1e308 * 10 %}", statesVar).asText());+ assertEquals("-Infinity", evaluator.evaluate("{% -1/0 %}", statesVar).asText());+ }++ /**+ * The shapes AWS answers a string for and Floci does not. A non-finite number the library's own+ * arithmetic produces reaches Floci only as the exception the library raises instead of carrying+ * the value, and by then the exception has unwound whatever value was being built around it, so+ * only an expression whose own result is non-finite can be answered. AWS answers+ * {@code ["Infinity"]} for the first, {@code {"k":"Infinity"}} for the second and+ * {@code "NaN"} for {@code 0/0}.+ *+ * <p>Failing the state is the half of the divergence a Catch fires on, which is why the value is+ * not answered at the top of an expression that was building something else with it.+ */+ @Test+ void aNonFiniteNumberTheArithmeticBuildsInsideAValueFailsTheStateInstead() {+ JsonNode statesVar = objectMapper.createObjectNode();++ for (String expression : new String[]{"[1/0]", "{'k': 1/0}", "[1/0, 2]", "$string(1/0)"}) {+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate("{% " + expression + " %}", statesVar),+ "expected " + expression + " to fail the state");+ assertEquals("States.QueryEvaluationError", ex.error);+ assertEquals("D1001: Number out of range: \"Infinity\"", ex.cause);+ }++ // NaN raises nothing at all: the library reads it as not a number and drops it, so the+ // expression returns nothing and the field it was written in is what fails.+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluateField("{% 0/0 %}", "Output/r", statesVar, null));+ assertEquals("The JSONata expression '0/0' specified for the field 'Output/r' "+ + "returned nothing (undefined).", ex.cause);+ }++ /**+ * $parseInteger answers NaN for a value its picture cannot read, where the library answers+ * nothing and an ASL field that evaluates to nothing fails the state.+ */+ @Test+ void parseIntegerAnswersNaNForAValueThePictureCannotRead() {+ JsonNode statesVar = objectMapper.createObjectNode();++ assertEquals("NaN", evaluator.evaluateField("{% $parseInteger('abc', '0') %}",+ "Output/r", statesVar, null).asText());+ // A non-finite number a function answers is carried like any other value, so it reaches the+ // output as AWS writes it wherever it lands, which is what arithmetic cannot do.+ assertEquals("[\"NaN\"]",+ evaluator.evaluate("{% [$parseInteger('abc', '0')] %}", statesVar).toString());+ assertEquals("{\"k\":\"NaN\"}",+ evaluator.evaluate("{% {'k': $parseInteger('abc', '0')} %}", statesVar).toString());+ // A value the picture does read stays the integer it was, not a double.+ assertEquals("123", evaluator.evaluate("{% $parseInteger('123', '0') %}", statesVar).toString());+ }++ /**+ * A picture string with no digit in it describes no number, and AWS fails the state on it+ * where the library formats through DecimalFormat and answers "x1".+ */+ @Test+ void formatNumberRefusesAPictureThatDescribesNoNumber() {+ JsonNode statesVar = objectMapper.createObjectNode();++ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate("{% $formatNumber(1, 'x') %}", statesVar));++ assertEquals("States.QueryEvaluationError", ex.error);+ assertEquals("D3086: The sub-picture must not contain a passive character that is preceded"+ + " by an active character and that is followed by another active character", ex.cause);+ }++ @Test+ void formatNumberNamesTheRuleThePictureBreaks() {+ JsonNode statesVar = objectMapper.createObjectNode();+ Map<String, String> expectedCodes = Map.ofEntries(+ Map.entry("0.0;0.0;0.0", "D3080"), Map.entry("0.0.0", "D3081"),+ Map.entry("0%%", "D3082"), Map.entry("0‰‰", "D3083"),+ Map.entry("0%‰", "D3084"), Map.entry("0,.0", "D3087"),+ Map.entry("0,", "D3088"), Map.entry("0,,0", "D3089"),+ Map.entry("0#", "D3090"), Map.entry("##0.0#0", "D3091"),+ Map.entry("0e0%", "D3092"), Map.entry("#0e", "D3093"));++ expectedCodes.forEach((picture, code) -> {+ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate("{% $formatNumber(1, '" + picture + "') %}", statesVar),+ "expected " + picture + " to fail");+ assertTrue(ex.cause.startsWith(code + ":"), picture + " gave " + ex.cause);+ });+ }++ @Test+ void formatNumberStillFormatsThePicturesTheRulesAllow() {+ JsonNode statesVar = objectMapper.createObjectNode();++ assertEquals("1,234.57", evaluator.evaluate("{% $formatNumber(1234.5678, '#,##0.00') %}",+ statesVar).asText());+ assertEquals("12.3%", evaluator.evaluate("{% $formatNumber(0.123, '#0.0%') %}", statesVar).asText());+ assertEquals("$1,234.57", evaluator.evaluate("{% $formatNumber(1234.5678, '$#,##0.00') %}",+ statesVar).asText());+ // A passive character before the first active one (a currency prefix, or the parenthesis+ // AWS's own negative sub-picture opens with) shifted the exponent separator's index without+ // shifting it back onto the active part, so any of these with an exponent was refused with a+ // spurious D3093.+ assertEquals("-$12e2", evaluator.evaluate("{% $formatNumber(-1200, '$#0e0') %}", statesVar).asText());+ assertEquals("-12e2", evaluator.evaluate("{% $formatNumber(-1200, '#0e0;-#0e0') %}", statesVar).asText());+ assertEquals("(12e2)", evaluator.evaluate("{% $formatNumber(-1200, '#0e0;(#0e0)') %}", statesVar).asText());+ }+@Testvoid string_writesAWholeNumberInFullBelowTheExponentBoundary() {// dashjoin stops writing digits at Long.MAX_VALUE, 9.223372036854776E18, and prints@@ -908,6 +1213,51 @@ class JsonataEvaluatorTest {+ expectedFieldPath + "' returned nothing (undefined).", failure.cause);}+ /**+ * #2738: {@code evaluate} caught {@code Exception}, not {@code Error}, so a+ * {@code java.lang.Error} raised while parsing or evaluating an expression escaped straight+ * past this method to {@code AslExecutor}'s last-resort {@code catch (Error e)}, failing the+ * whole execution as {@code States.Runtime} before the state's own {@code Retry}/{@code Catch}+ * ever ran. This pins the {@code StackOverflowError} arm of the catch, with an expression+ * nested 200,000 parentheses deep that overflows the JVM call stack during parsing in a few+ * milliseconds and never grows a string past a few hundred KB. The other arm,+ * {@code OutOfMemoryError}, remains a guard with no cheap trigger through this library: the+ * memory bound now trips first on every expression measured against it, including the one the+ * issue reproduced with (see+ * {@code StepFunctionsJsonataIntegrationTest.parallelBranchErrorIsCatchableByStatesAll}).+ */+ @Test+ void aStackOverflowErrorDuringParsingBecomesAQueryEvaluationError() {+ JsonNode statesVar = objectMapper.createObjectNode();+ String deeplyNested = "(".repeat(200_000) + "1" + ")".repeat(200_000);++ AslExecutor.FailStateException ex = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluate("{% " + deeplyNested + " %}", statesVar));++ assertEquals("States.QueryEvaluationError", ex.error);+ assertTrue(ex.cause.contains("StackOverflowError"), ex.cause);+ }++ /**+ * The {@code StackOverflowError}/{@code OutOfMemoryError} arm used to throw a bare+ * {@code FailStateException}, which {@link JsonataEvaluator#evaluateField} does not catch as a+ * {@code QueryEvaluationFailure}, so the field's expression and AWS's sentence never reached the+ * cause: a Catch on this failure saw the bare {@code java.lang.StackOverflowError} instead of+ * the sentence every other evaluation failure carries.+ */+ @Test+ void evaluateFieldWrapsAStackOverflowErrorWithTheSentenceAndAColonSeparator() throws Exception {+ JsonNode statesVar = objectMapper.createObjectNode();+ String deeplyNested = "(".repeat(200_000) + "1" + ")".repeat(200_000);++ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluateField("{% " + deeplyNested + " %}", "Output/v", statesVar, null));++ assertEquals("States.QueryEvaluationError", failure.error);+ assertTrue(failure.cause.contains("threw an error during evaluation: "), failure.cause);+ assertTrue(failure.cause.contains("StackOverflowError"), failure.cause);+ }+@Testvoid evaluateFieldNamesTheFieldOfAPositionThatHoldsASingleExpression() throws Exception {JsonNode statesVar = objectMapper.readTree("{\"input\": {}}");@@ -920,6 +1270,79 @@ class JsonataEvaluatorTest {+ "returned nothing (undefined).", failure.cause);}+ @Test+ void evaluateFieldWrapsAThrownJsonataErrorWithTheSentenceAndACodedSeparator() throws Exception {+ // A coded JSONata failure carries AWS's sentence in front of the code, joined by ". ".+ JsonNode statesVar = objectMapper.createObjectNode();++ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluateField("{% $sum(['a']) %}", "Output/v", statesVar, null));++ assertEquals("States.QueryEvaluationError", failure.error);+ assertEquals("The JSONata expression '$sum(['a'])' specified for the field 'Output/v' threw an error "+ + "during evaluation. T0412: Argument [\"a\"] must be an array of \"numbers\"", failure.cause);+ }++ @Test+ void evaluateFieldWrapsABoundFailureWithTheSentenceAndAColonSeparator() throws Exception {+ // A failure with no JSONata code and no template to render (see+ // aNonJsonataFailureKeepsTheMessageItCameWith) carries AWS's sentence in front of the bare+ // message it came with, joined by ": " where a coded one joins with ". ".+ JsonNode statesVar = objectMapper.createObjectNode();++ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluateField("{% $toMillis('nope') %}", "Output/v", statesVar, null));++ assertEquals("States.QueryEvaluationError", failure.error);+ assertTrue(failure.cause.startsWith("The JSONata expression '$toMillis('nope')' specified for the field "+ + "'Output/v' threw an error during evaluation: "), failure.cause);+ assertTrue(failure.cause.contains("nope"), failure.cause);+ }++ /**+ * The memory bound's message carries no JSONata code slot, so it is a bare bound message like+ * {@code $toMillis('nope')}'s and joins AWS's sentence with {@code ": "}, not the {@code ". "}+ * a coded message joins with. Measured against real AWS (us-east-1, test-state): this is its+ * cause byte for byte.+ */+ @Test+ @Timeout(60)+ void evaluateFieldWrapsTheMemoryBoundFailureWithTheSentenceAndAColonSeparator() throws Exception {+ JsonNode statesVar = objectMapper.createObjectNode();++ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluateField("{% [1..900000] ~> $count() %}", "Output/r", statesVar, null));++ assertEquals("States.QueryEvaluationError", failure.error);+ assertEquals("The JSONata expression '[1..900000] ~> $count()' specified for the field 'Output/r' threw "+ + "an error during evaluation: Expression evaluation memory limit exceeded: Check for excessive "+ + "memory usage", failure.cause);+ }++ /**+ * The depth bound's message is the same shape: no JSONata code slot, so {@code ": "} joins it+ * to AWS's sentence too. AWS's own cause for this same 101-term chain, measured the same way,+ * ends {@code "...evaluation: Stack overflow error: Check for non-terminating recursive+ * function. Consider rewriting as tail-recursive"}; the library's own double space after+ * "function." is the one divergence {@link JsonataEvaluator#withoutDepthBoundSuffix} leaves in+ * place, so this checks the sentence and the join rather than the byte-exact tail.+ */+ @Test+ @Timeout(30)+ void evaluateFieldWrapsTheDepthBoundFailureWithTheSentenceAndAColonSeparator() throws Exception {+ JsonNode statesVar = objectMapper.createObjectNode();+ String chain = "1" + "+1".repeat(100);++ AslExecutor.FailStateException failure = assertThrows(AslExecutor.FailStateException.class,+ () -> evaluator.evaluateField("{% " + chain + " %}", "Output/r", statesVar, null));++ assertEquals("States.QueryEvaluationError", failure.error);+ assertTrue(failure.cause.startsWith("The JSONata expression '" + chain + "' specified for the field "+ + "'Output/r' threw an error during evaluation: Stack overflow error: Check for non-terminating"),+ failure.cause);+ assertTrue(failure.cause.endsWith("Consider rewriting as tail-recursive"), failure.cause);+ }+@Testvoid anExplicitNullIsAValueAndKeepsEveryPositionEvaluating() throws Exception {JsonNode statesVar = objectMapper.readTree("{\"input\": {\"bar\": null}}");diff --git a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataFunctionsIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataFunctionsIntegrationTest.javaindex 7e692b46e6..bb6aebb812 100644--- a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataFunctionsIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataFunctionsIntegrationTest.java@@ -247,7 +247,38 @@ class StepFunctionsJsonataFunctionsIntegrationTest {Response resp = waitForExecutionFailure(startExecution(smArn, "{}"));assertEquals("States.QueryEvaluationError", resp.jsonPath().getString("error"));- assertEquals("T0412: Argument [\"a\"] must be an array of \"numbers\"",+ assertEquals("The JSONata expression '$sum(['a'])' specified for the field 'Output/v' threw an error "+ + "during evaluation. T0412: Argument [\"a\"] must be an array of \"numbers\"",+ resp.jsonPath().getString("cause"));+ }++ @Test+ void aFailedJsonataExpressionOpensWithTheSentenceNamingTheExpressionAndTheField() throws Exception {+ // AWS on the same definition: States.QueryEvaluationError,+ // "The JSONata expression '$abs(\"x\")' specified for the field 'Output/v' threw an error+ // during evaluation. T0410: Argument 1 of function \"abs\" does not match function signature"+ String definition = """+ {+ "QueryLanguage": "JSONata",+ "StartAt": "Abs",+ "States": {+ "Abs": {+ "Type": "Pass",+ "Output": {+ "v": "{% $abs(\\"x\\") %}"+ },+ "End": true+ }+ }+ }+ """;++ String smArn = createStateMachine("jsonata-abs-error-cause", definition);+ Response resp = waitForExecutionFailure(startExecution(smArn, "{}"));++ assertEquals("States.QueryEvaluationError", resp.jsonPath().getString("error"));+ assertEquals("The JSONata expression '$abs(\"x\")' specified for the field 'Output/v' threw an error "+ + "during evaluation. T0410: Argument 1 of function \"abs\" does not match function signature",resp.jsonPath().getString("cause"));}diff --git a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataIntegrationTest.javaindex 164d46419d..bb61192e00 100644--- a/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/stepfunctions/StepFunctionsJsonataIntegrationTest.java@@ -1311,6 +1311,62 @@ class StepFunctionsJsonataIntegrationTest {assertTrue(caught.path("caught").asBoolean(), caught.toString());}+ /**+ * #2738: a {@code java.lang.Error} raised while evaluating a branch's expression used to+ * escape {@code JsonataEvaluator}'s catch and reach {@code AslExecutor}'s own last-resort+ * {@code catch (Error e)}, failing the whole execution as {@code States.Runtime} before the+ * {@code Parallel}'s own {@code Catch} ever ran. On real AWS (measured in us-east-1) the same+ * definition ends {@code SUCCEEDED} with output {@code {"caught":true}}.+ *+ * <p>The branch expression is the issue's own reproduction verbatim (quotes swapped to+ * JSONata's single-quote string literal, which embeds in this JSON body without escaping):+ * a tail-recursive function doubles a string 31 times, which dashjoin loops rather than+ * recursing on (JSONata optimises the tail call). Since the memory bound+ * {@code JsonataEvaluator} now holds, the doubling trips it at the 23rd iteration —+ * {@code 'x'} doubled 23 times is 8,388,608 characters, past+ * {@link JsonataEvaluator#MAX_EXPRESSION_BYTES}'s 6,990,256 — and fails the state with+ * {@code Expression evaluation memory limit exceeded} through the ordinary {@code JException}+ * path, eight doublings short of ever reaching the 31st and nowhere near the JVM's maximum+ * array length. Whichever of the two carries the failure, the {@code Parallel}'s own+ * {@code Catch} still sees it, which is what this test pins: the execution still ends+ * {@code SUCCEEDED} with {@code {"caught":true}}, matching AWS. The {@code OutOfMemoryError}+ * arm of {@code JsonataEvaluator.evaluate}'s catch remains a guard with no cheap trigger+ * through this library; its {@code StackOverflowError} sibling is pinned by+ * {@code JsonataEvaluatorTest.aStackOverflowErrorDuringParsingBecomesAQueryEvaluationError}.+ */+ @Test+ void parallelBranchErrorIsCatchableByStatesAll() throws Exception {+ String definition = """+ {+ "QueryLanguage": "JSONata",+ "StartAt": "P",+ "States": {+ "P": {+ "Type": "Parallel",+ "Branches": [{+ "StartAt": "E",+ "States": {+ "E": {+ "Type": "Pass",+ "Output": {"r": "{% ($p := function($s, $n) { $n = 0 ? $s : $p($s & $s, $n - 1) }; $length($p('x', 31))) %}"},+ "End": true+ }+ }+ }],+ "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "Caught"}],+ "End": true+ },+ "Caught": {"Type": "Pass", "Output": {"caught": true}, "End": true}+ }+ }+ """;++ String smArn = createStateMachine("sfn-error-catchable-test", definition);+ JsonNode output = objectMapper.readTree(waitForExecution(startExecution(smArn, "{}")));++ assertTrue(output.path("caught").asBoolean(), output.toString());+ }+@Testvoid assignExpressionReturningNothingFailsTheState() throws Exception {// Real AWS names 'Assign/x' for this definition; before this guard the variable was never