Light, fluffy, and always free - The AWS Local Emulator alternative
fix(cloudfront): serve viewer Range GETs (#5377)
Priamo Germosenff20750d9123
diff --git a/docs/services/cloudfront.md b/docs/services/cloudfront.mdindex f5ee9caf74..f58b563669 100644--- a/docs/services/cloudfront.md+++ b/docs/services/cloudfront.md@@ -196,7 +196,8 @@ PUT, PATCH and DELETE).union of both). `HeaderBehavior` `none`, `whitelist`, `allViewer`,`allViewerAndWhitelistCloudFront` and `allExcept`, `CookieBehavior` and `QueryStringBehavior``none`, `whitelist`, `allExcept` and `all` are evaluated. A cache policy that selects nothing and no- origin request policy forwards no viewer headers, cookies or query strings, as on AWS.+ origin request policy forwards no viewer headers, cookies or query strings, as on AWS, apart from+ the headers below that CloudFront forwards whatever the policies.- A cache behavior without a cache policy uses its legacy `ForwardedValues`: `QueryString=true`forwards every query string (`QueryStringCacheKeys` only narrows the cache key), cookies follow`Forward` `none`, `whitelist` (with `*` and `?` wildcards) or `all`, and viewer headers are@@ -209,11 +210,13 @@ PUT, PATCH and DELETE).forwarded, appends the viewer address to `X-Forwarded-For`, adds a`Via: <viewer HTTP version> <distribution domain> (CloudFront)` hop and an `X-Amz-Cf-Id`.`Authorization` is always forwarded on POST, PUT, PATCH and DELETE, and on OPTIONS unless OPTIONS- is a cached method; `Content-Type` travels with a request body. Cache policy compression settings- normalize `Accept-Encoding` to `br,gzip`, `gzip`, `br` or `identity`. Forwarded cookies are sorted- by name, and cookies that are not `name=value` or whose name starts with `$` are dropped.- Hop-by-hop headers, `Expect`, `X-Edge-*`, `X-Real-IP`, `X-Forwarded-Proto` and viewer-supplied- `CloudFront-*` headers are never forwarded.+ is a cached method; `Content-Type` travels with a request body; a GET or HEAD with a `Range`+ carries the viewer's `Range` and `If-Range`, since CloudFront forwards a range GET to the origin,+ and a HEAD can then report the range's length. Cache policy compression settings normalize+ `Accept-Encoding` to `br,gzip`, `gzip`, `br` or `identity`.+ Forwarded cookies are sorted by name, and cookies that are not `name=value` or whose name starts+ with `$` are dropped. Hop-by-hop headers, `Expect`, `X-Edge-*`, `X-Real-IP`, `X-Forwarded-Proto`+ and viewer-supplied `CloudFront-*` headers are never forwarded.- Of the CloudFront request headers, `CloudFront-Forwarded-Proto`, `CloudFront-Viewer-Address` and`CloudFront-Viewer-Http-Version` are generated when a policy names them, or when an `allExcept`header behavior does not exclude them (AWS documents Managed-AllViewerExceptHostHeader as adding@@ -225,11 +228,22 @@ PUT, PATCH and DELETE).`CachingOptimized` and `CachingOptimizedForUncompressedObjects` but not `Amplify`,`Elemental-MediaPackage` or the `UseOriginCacheControlHeaders` policies.- In-process S3 origins do not apply forwarding settings: viewer query strings, cookies and headers- do not change the S3 read.+ do not change the S3 read, apart from the viewer's `Range` and `If-Range` on a GET or HEAD.- Origin responses are streamed to the viewer, from in-process S3 origins and custom origins alike,so a large object is never held in memory. A custom origin's `Content-Length` is passed on, and aresponse it sends chunked stays chunked. Floci does not cache, so it serves objects larger than50 GB the way CloudFront does with caching disabled; with caching enabled, CloudFront refuses them.+- Range GETs: an S3 origin serves the one byte range a viewer's `Range` asks for as 206 with its+ `Content-Range`, and answers 416 when the range starts past the end of the object. Any other+ `Range` gets the whole object: one CloudFront does not accept (ranges out of order or overlapping,+ or a negative value), several ranges, which S3 does not serve in one request, or a last byte+ before the first, as does a `Range` whose `If-Range` no longer describes the object, which S3+ answers with the whole object. A HEAD is answered as S3's HeadObject answers a `Range`: a+ satisfiable range changes only its `Content-Length`, and one past the end gets 416. A custom+ origin gets the viewer's `Range` and `If-Range` on every GET or HEAD with a `Range`, whatever the+ forwarding settings, and its answer (206, the whole object, or 416) is passed on. Floci does not+ cache, so unlike CloudFront it cannot serve a later range request from an object the origin sent+ whole. A custom error page is served whole, whatever the `Range` of the request it replaces.- Origin `Set-Cookie` headers always reach the viewer. With legacy `Forward=none`, AWS strips themfrom the response; Floci does not.- Custom origins that resolve to loopback, private, link-local, carrier-grade NAT, or other non-routable addresses are rejected by default. Development-only private origins must be explicitly allowlisted by exact hostname.diff --git a/docs/services/s3.md b/docs/services/s3.mdindex 490d59d4cb..886611b29b 100644--- a/docs/services/s3.md+++ b/docs/services/s3.md@@ -106,6 +106,10 @@ Redirect-all and advanced routing-rule configurations are not implemented.Suffix ranges against empty objects return an empty `200` response, matching AWS behavior used by clients that issue tail reads.+`HeadObject` honors `Range` as the API Reference describes: a satisfiable range changes `Content-Length` to the length of the range, and one that cannot be satisfied returns `416`, while a `Range` that is not one valid range, such as several ranges or a last byte before the first, is ignored. Like a ranged `GetObject`, it leaves out the whole-object checksum headers.++An `If-Range` header makes the range conditional, so a resumed download never mixes two versions of an object. The range is served while the validator still describes the object: a quoted ETag equal to the object's (a multipart ETag compared whole, `-N` suffix included), or an HTTP-date equal to its `Last-Modified`, an exact match as RFC 9110 section 13.1.5 requires, so a later date does not qualify. Otherwise, including a weak (`W/`) or malformed validator, `GetObject` returns the whole object with `200`. A presigned URL that signs `Range` must also sign an `If-Range` the request carries, or the request fails with `403 AccessDenied` naming it in `HeadersNotSigned`.+## S3 Select`SelectObjectContent` runs SQL queries directly against S3 objects without downloading the entire file. Floci supports CSV, JSON Lines, JSON arrays, and Parquet inputs. The SQL dialect follows [AWS S3 Select SQL reference](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-select-sql-reference.html).diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilder.java b/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilder.javaindex 819a7221e6..8d55c03b0f 100644--- a/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilder.java+++ b/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilder.java@@ -28,6 +28,10 @@ import java.util.concurrent.ThreadLocalRandom;* {@code User-Agent} is forwarded, an {@code X-Forwarded-For} ending in the viewer address,* CloudFront's {@code Via} hop and an {@code X-Amz-Cf-Id}. {@code Host} is the origin's own* name unless the viewer's {@code Host} is forwarded.</li>+ * <li>Whatever the policies, {@code Authorization} goes with the methods CloudFront does not cache,+ * {@code Content-Type} with a request body, and the viewer's {@code Range} and+ * {@code If-Range} with a GET or HEAD that has a {@code Range}, since CloudFront forwards a+ * range GET to the origin.</li>* </ul>** <p>This class holds no state and does no I/O. Origin custom headers are applied afterwards by the@@ -113,6 +117,8 @@ final class CloudFrontOriginRequestBuilder {Set<String> connectionTokens = connectionTokens(viewer.headers());boolean forwardsAuthorization = BODY_METHODS.contains(viewer.method())|| ("OPTIONS".equals(viewer.method()) && !forwarding.optionsCached());+ boolean rangeRead = ("GET".equals(viewer.method()) || "HEAD".equals(viewer.method()))+ && viewer.headers().stream().anyMatch(header -> "range".equals(normalize(header.name())));List<Header> headers = new ArrayList<>();List<String> cookieHeaders = new ArrayList<>();@@ -151,6 +157,14 @@ final class CloudFrontOriginRequestBuilder {headers.add(header);}}+ case "range", "if-range" -> {+ // CloudFront forwards a range GET to the origin whatever the cache settings, so the+ // origin can answer with the range alone. A HEAD gets them too, so it can report the+ // range's length as S3's HeadObject does. If-Range only qualifies a Range.+ if (rangeRead || selections.forwardsHeader(name)) {+ headers.add(header);+ }+ }default -> {if (selections.forwardsHeader(name)) {headers.add(header);diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontServingController.java b/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontServingController.javaindex 37551b01b8..b67111045e 100644--- a/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontServingController.java+++ b/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontServingController.java@@ -8,6 +8,7 @@ import io.github.hectorvent.floci.services.cloudfront.model.Distribution;import io.github.hectorvent.floci.services.cloudfront.model.DistributionConfig;import io.github.hectorvent.floci.services.cloudfront.model.Origin;import io.github.hectorvent.floci.services.cloudfront.model.OriginAccessControl;+import io.github.hectorvent.floci.services.s3.ByteRange;import io.github.hectorvent.floci.services.s3.S3Service;import io.github.hectorvent.floci.services.s3.model.S3Object;import io.quarkus.vertx.http.runtime.CurrentVertxRequest;@@ -473,8 +474,9 @@ public class CloudFrontServingController {}String key = CloudFrontRequestRouter.resolveOriginKey(origin.getOriginPath(), decodedViewerPath, config.getDefaultRootObject());- return fetchFromS3(- distribution, origin, key, viewerAuthorization, !"HEAD".equals(method));+ HttpServerRequest request = currentVertxRequest.getCurrent().request();+ return fetchFromS3(distribution, origin, key, viewerAuthorization, !"HEAD".equals(method),+ new ViewerRange(request.getHeader("Range"), request.getHeader("If-Range")));}String forwardUri = CloudFrontRequestRouter.resolveForwardUri(origin.getOriginPath(), rawViewerPath, config.getDefaultRootObject());@@ -565,7 +567,8 @@ public class CloudFrontServingController {Origin origin,String key,String viewerAuthorization,- boolean includeBody) {+ boolean includeBody,+ ViewerRange viewerRange) {String bucket = CloudFrontRequestRouter.bucketFromS3Domain(origin.getDomainName());if (bucket == null) {return OriginResponse.error(502, "Could not determine S3 bucket for origin.");@@ -577,14 +580,9 @@ public class CloudFrontServingController {if (includeBody) {// The body is streamed to the viewer, so an object of any size is served without// being read into the heap; the metadata and the stream come from one snapshot.- S3Service.ObjectRead read = s3Service.openObject(bucket, key, null);- S3Object obj = read.object();- response = new OriginResponse(- 200, contentType(obj), null, read.body(), obj.getSize(), s3ObjectHeaders(obj));+ response = s3ObjectResponse(s3Service.openObject(bucket, key, null), viewerRange);} else {- S3Object meta = s3Service.headObject(bucket, key);- response = new OriginResponse(- 200, contentType(meta), null, meta.getSize(), s3ObjectHeaders(meta));+ response = s3HeadResponse(s3Service.headObject(bucket, key), viewerRange);}} catch (AwsException e) {response = OriginResponse.error(e.getHttpStatus(), e.getMessage());@@ -598,6 +596,69 @@ public class CloudFrontServingController {}}+ /**+ * The GET of an S3 object, answered as S3 answers the viewer's {@code Range}: the one range it+ * asks for, or 416 when that range starts past the end of the object. CloudFront serves the+ * whole object, as for no {@code Range} at all, when the header is not one valid range, which+ * covers several ranges, since S3 serves one range per request, and when an {@code If-Range}+ * no longer describes the object.+ */+ private static OriginResponse s3ObjectResponse(S3Service.ObjectRead read, ViewerRange viewerRange) {+ S3Object obj = read.object();+ ByteRange.Resolution resolution = resolveRange(viewerRange, obj);+ return switch (resolution.outcome()) {+ case PART -> {+ ByteRange range = resolution.range();+ Map<String, List<String>> headers = s3ObjectHeaders(obj);+ putHeader(headers, "Content-Range", range.contentRange(obj.getSize()));+ yield new OriginResponse(206, contentType(obj), null, range.slice(read.body()),+ range.length(), headers);+ }+ case NOT_SATISFIABLE -> {+ closeUnsent(read);+ yield rangeNotSatisfiable(obj.getSize());+ }+ case WHOLE, INVALID -> new OriginResponse(+ 200, contentType(obj), null, read.body(), obj.getSize(), s3ObjectHeaders(obj));+ };+ }++ /**+ * The HEAD of an S3 object, answered as HeadObject answers a {@code Range}: "If the Range is+ * satisfiable, only the ContentLength is affected in the response. If the Range is not+ * satisfiable, S3 returns a 416". A header that is not one valid range is ignored, as for a GET.+ */+ private static OriginResponse s3HeadResponse(S3Object meta, ViewerRange viewerRange) {+ ByteRange.Resolution resolution = resolveRange(viewerRange, meta);+ return switch (resolution.outcome()) {+ case PART -> new OriginResponse(+ 200, contentType(meta), null, resolution.range().length(), s3ObjectHeaders(meta));+ case NOT_SATISFIABLE -> rangeNotSatisfiable(meta.getSize());+ case WHOLE, INVALID -> new OriginResponse(+ 200, contentType(meta), null, meta.getSize(), s3ObjectHeaders(meta));+ };+ }++ /** The viewer's {@code Range} resolved against an object, unless its {@code If-Range} no longer matches it. */+ private static ByteRange.Resolution resolveRange(ViewerRange viewerRange, S3Object obj) {+ String rangeHeader = ByteRange.rangeApplies(viewerRange.ifRange(), obj.getETag(), obj.getLastModified())+ ? viewerRange.range() : null;+ return ByteRange.resolve(rangeHeader, obj.getSize());+ }++ private static OriginResponse rangeNotSatisfiable(long size) {+ return OriginResponse.error(416, ByteRange.NOT_SATISFIABLE_MESSAGE)+ .withHeaders(Map.of("Content-Range", List.of(ByteRange.unsatisfiedContentRange(size))));+ }++ private static void closeUnsent(S3Service.ObjectRead read) {+ try {+ read.close();+ } catch (IOException e) {+ LOG.debugv("Could not close an unsent S3 object: {0}", e.getMessage());+ }+ }+private void authorizeS3OriginRead(Distribution distribution,Origin origin,@@ -759,6 +820,11 @@ public class CloudFrontServingController {record ViewerBody(InputStream stream, long length) {}+ /** The viewer's {@code Range} and the {@code If-Range} that qualifies it, either one possibly absent. */+ private record ViewerRange(String range, String ifRange) {+ static final ViewerRange NONE = new ViewerRange(null, null);+ }+private static String originAccessIdentityId(Origin origin) {Map<String, String> config = origin.getS3OriginConfig();String value = config != null ? config.get("OriginAccessIdentity") : null;@@ -911,8 +977,9 @@ public class CloudFrontServingController {OriginResponse page;if (CloudFrontRequestRouter.isS3Origin(errOrigin)) {String key = CloudFrontRequestRouter.resolveOriginKey(errOrigin.getOriginPath(), errNormalized, null);+ // The viewer's Range is for the object it asked for, not for the page served instead.page = fetchFromS3(- distribution, errOrigin, key, viewerAuthorization, includeBody);+ distribution, errOrigin, key, viewerAuthorization, includeBody, ViewerRange.NONE);} else {String forwardUri = CloudFrontRequestRouter.resolveForwardUri(errOrigin.getOriginPath(), errNormalized, null);page = fetchFromCustomOrigin(diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/ByteRange.java b/src/main/java/io/github/hectorvent/floci/services/s3/ByteRange.javanew file mode 100644index 0000000000..1da7a0e8d4--- /dev/null+++ b/src/main/java/io/github/hectorvent/floci/services/s3/ByteRange.java@@ -0,0 +1,142 @@+package io.github.hectorvent.floci.services.s3;++import java.io.InputStream;+import java.time.Instant;+import java.time.ZonedDateTime;+import java.time.format.DateTimeFormatter;+import java.time.format.DateTimeParseException;+import java.time.temporal.ChronoUnit;+import java.util.regex.Matcher;+import java.util.regex.Pattern;++/**+ * One byte range of an object, from a {@code Range} header that asks for a single range, as RFC 9110+ * section 14.1.2 defines it. {@code last} is inclusive. S3 serves one range per GET request.+ */+public record ByteRange(long first, long last) {++ /**+ * The message of S3's {@code InvalidRange} error as S3 sends it, which is not the wording of the+ * error code table's description.+ */+ public static final String NOT_SATISFIABLE_MESSAGE = "The requested range is not satisfiable";++ private static final String UNIT = "bytes=";+ private static final Pattern SPEC = Pattern.compile("(\\d*)-(\\d*)");++ /**+ * What a {@code Range} header asks of an object: a part of it, the whole of it (no header, a+ * unit other than bytes, or a suffix of an empty object), a range that starts past its end, or+ * something that is not one valid range, such as several ranges or a last byte before the first.+ */+ public enum Outcome { PART, WHOLE, NOT_SATISFIABLE, INVALID }++ /** A resolved {@code Range} header; {@code range} is set only for {@link Outcome#PART}. */+ public record Resolution(Outcome outcome, ByteRange range) {+ }++ public ByteRange {+ if (first < 0 || last < first) {+ throw new IllegalArgumentException("Not a byte range: " + first + "-" + last);+ }+ }++ public long length() {+ return last - first + 1;+ }++ /** The {@code Content-Range} value of this range of an object of {@code size} bytes. */+ public String contentRange(long size) {+ return "bytes " + first + "-" + last + "/" + size;+ }++ /** The {@code Content-Range} value of a 416 response for an object of {@code size} bytes. */+ public static String unsatisfiedContentRange(long size) {+ return "bytes */" + size;+ }++ /**+ * Whether a {@code Range} header asks for bytes, the one unit S3 serves. Range unit names are+ * case-insensitive (RFC 9110, section 14.1).+ */+ public static boolean inBytes(String header) {+ return header != null && header.regionMatches(true, 0, UNIT, 0, UNIT.length());+ }++ /**+ * Whether a request's {@code Range} applies to an object, given the request's {@code If-Range}:+ * always without one, and otherwise when its validator still describes the object, a quoted+ * entity tag equal to the object's ETag, or an HTTP-date equal to its Last-Modified.+ * Any other validator, a weak or malformed one included, means the whole object is served.+ */+ public static boolean rangeApplies(String ifRange, String etag, Instant lastModified) {+ if (ifRange == null) {+ return true;+ }+ String validator = ifRange.trim();+ if (validator.startsWith("\"")) {+ return validator.length() > 1 && validator.endsWith("\"") && validator.equals(etag);+ }+ Instant date;+ try {+ date = ZonedDateTime.parse(validator, DateTimeFormatter.RFC_1123_DATE_TIME).toInstant();+ } catch (DateTimeParseException expected) {+ // Neither a quoted entity tag nor an HTTP-date, such as a weak W/ tag: it cannot match.+ return false;+ }+ // RFC 9110 section 13.1.5 matches a date exactly, not as the "not modified since" of+ // If-Unmodified-Since, so a later date gets the whole object. Last-Modified is sent to the+ // second, so it is compared to the second.+ return lastModified != null && lastModified.truncatedTo(ChronoUnit.SECONDS).equals(date);+ }++ /** This range's bytes, read from {@code whole}, a stream over the whole object, as they are needed. */+ public InputStream slice(InputStream whole) {+ return new ByteRangeInputStream(whole, first, length());+ }++ /** Resolves a {@code Range} header against an object of {@code size} bytes. */+ public static Resolution resolve(String header, long size) {+ if (!inBytes(header)) {+ return new Resolution(Outcome.WHOLE, null);+ }+ Matcher matcher = SPEC.matcher(header.substring(UNIT.length()).trim());+ if (!matcher.matches() || (matcher.group(1).isEmpty() && matcher.group(2).isEmpty())) {+ return new Resolution(Outcome.INVALID, null);+ }+ long first;+ long last;+ if (matcher.group(1).isEmpty()) {+ long suffix = position(matcher.group(2));+ if (suffix == 0) {+ return new Resolution(Outcome.NOT_SATISFIABLE, null);+ }+ if (size == 0) {+ return new Resolution(Outcome.WHOLE, null);+ }+ first = Math.max(0, size - suffix);+ last = size - 1;+ } else {+ first = position(matcher.group(1));+ last = matcher.group(2).isEmpty() ? Long.MAX_VALUE : position(matcher.group(2));+ if (last < first) {+ return new Resolution(Outcome.INVALID, null);+ }+ last = Math.min(last, size - 1);+ }+ if (first >= size) {+ return new Resolution(Outcome.NOT_SATISFIABLE, null);+ }+ return new Resolution(Outcome.PART, new ByteRange(first, last));+ }++ private static long position(String digits) {+ try {+ return Long.parseLong(digits);+ } catch (NumberFormatException ignored) {+ // The pattern admits digits alone, so only a position too large for a long gets here, and+ // that is past the end of any object.+ return Long.MAX_VALUE;+ }+ }+}diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/CopyRangeInputStream.java b/src/main/java/io/github/hectorvent/floci/services/s3/ByteRangeInputStream.javasimilarity index 65%rename from src/main/java/io/github/hectorvent/floci/services/s3/CopyRangeInputStream.javarename to src/main/java/io/github/hectorvent/floci/services/s3/ByteRangeInputStream.javaindex 1567e7ead1..cad0bc7f65 100644--- a/src/main/java/io/github/hectorvent/floci/services/s3/CopyRangeInputStream.java+++ b/src/main/java/io/github/hectorvent/floci/services/s3/ByteRangeInputStream.java@@ -1,28 +1,26 @@package io.github.hectorvent.floci.services.s3;-import io.github.hectorvent.floci.services.s3.S3Service.CopySourceRange;-import java.io.EOFException;import java.io.IOException;import java.io.InputStream;/**- * The bytes of one copy-source range, read from a stream over the whole source as they are needed.- * The first read skips to the range's first byte, so nothing of the source is touched until the- * part is about to be written, and the stream ends after the range's last byte. A source that ends+ * The bytes of one range of an object, read from a stream over the whole object as they are needed.+ * The first read skips to the range's first byte, so nothing of the object is touched until the+ * range is about to be written, and the stream ends after the range's last byte. An object that ends* before the range does fails the read.*/-final class CopyRangeInputStream extends InputStream {+final class ByteRangeInputStream extends InputStream {private final InputStream source;private final long first;private long remaining;private boolean positioned;- CopyRangeInputStream(InputStream source, CopySourceRange range) {+ ByteRangeInputStream(InputStream source, long first, long length) {this.source = source;- this.first = range.first();- this.remaining = range.length();+ this.first = first;+ this.remaining = length;}@Override@@ -46,7 +44,7 @@ final class CopyRangeInputStream extends InputStream {}int read = source.read(buffer, offset, (int) Math.min(length, remaining));if (read < 0) {- throw new EOFException("Copy source ended " + remaining + " bytes before the end of the requested range");+ throw new EOFException("Object ended " + remaining + " bytes before the end of the requested range");}remaining -= read;return read;diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilter.java b/src/main/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilter.javaindex bab4a0276f..0d2f06498a 100644--- a/src/main/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilter.java+++ b/src/main/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilter.java@@ -360,7 +360,9 @@ public class PreSignedUrlFilter implements ContainerRequestFilter {return requestHeaderNames.stream().map(name -> name.toLowerCase(Locale.ROOT))- .filter(name -> CHECKSUM_HEADERS_REQUIRING_SIGNATURE.contains(name) || name.startsWith("x-amz-meta-"))+ .filter(name -> CHECKSUM_HEADERS_REQUIRING_SIGNATURE.contains(name) || name.startsWith("x-amz-meta-")+ // A signed Range makes S3 require the If-Range that qualifies it to be signed too.+ || ("if-range".equals(name) && normalizedSignedHeaders.contains("range"))).filter(name -> !normalizedSignedHeaders.contains(name)).distinct().sorted()diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java b/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.javaindex 13fd04c37f..6abf8373b7 100644--- a/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java+++ b/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java@@ -56,10 +56,8 @@ import javax.xml.stream.XMLStreamReader;import java.io.ByteArrayInputStream;import java.io.ByteArrayOutputStream;import java.io.Closeable;-import java.io.EOFException;import java.io.IOException;import java.io.InputStream;-import java.io.OutputStream;import java.io.UncheckedIOException;import java.net.URLDecoder;import java.nio.charset.StandardCharsets;@@ -1050,8 +1048,10 @@ public class S3Controller {}boolean includeChecksum = "ENABLED".equalsIgnoreCase(resolveHeaderOrQueryParam(checksumMode, uriInfo, "x-amz-checksum-mode"));- if (rangeHeader != null && rangeHeader.startsWith("bytes=")) {- Response rangeResponse = handleRangeRequest(obj, read.body(), rangeHeader, overrides, includeChecksum);+ if (ByteRange.inBytes(rangeHeader) && ByteRange.rangeApplies(+ httpHeaders.getHeaderString("If-Range"), obj.getETag(), obj.getLastModified())) {+ Response rangeResponse = handleRangeRequest(+ bucket, key, obj, read.body(), rangeHeader, overrides, includeChecksum);bodyHandedOff = rangeResponse.getEntity() instanceof StreamingOutput;return rangeResponse;}@@ -1117,57 +1117,33 @@ public class S3Controller {return resp;}- private Response handleRangeRequest(S3Object obj, InputStream body, String rangeHeader,- ResponseHeaderOverrides overrides,+ private Response handleRangeRequest(String bucket, String key, S3Object obj, InputStream body,+ String rangeHeader, ResponseHeaderOverrides overrides,boolean includeChecksum) {long totalSize = obj.getSize();- String rangeSpec = rangeHeader.substring("bytes=".length()).trim();-- long start, end;- try {- int dash = rangeSpec.indexOf('-');- if (dash < 0) {- return invalidRangeResponse(totalSize);- }- String before = rangeSpec.substring(0, dash);- String after = rangeSpec.substring(dash + 1);- if (before.isEmpty() && after.isEmpty()) {- return invalidRangeResponse(totalSize);- }- if (before.isEmpty()) {- long suffix = Long.parseLong(after);- if (suffix <= 0) {- return invalidRangeResponse(totalSize);- }- start = Math.max(0, totalSize - suffix);- end = totalSize - 1;- } else {- start = Long.parseLong(before);- end = after.isEmpty() ? totalSize - 1 : Math.min(Long.parseLong(after), totalSize - 1);- }- } catch (NumberFormatException e) {- return invalidRangeResponse(totalSize);+ ByteRange.Resolution resolution = ByteRange.resolve(rangeHeader, totalSize);+ if (resolution.outcome() == ByteRange.Outcome.WHOLE) {+ emitCloudTrailEvent("GetObject", bucket, key, 0L, totalSize, null, null);+ return fullObjectResponse(obj, body, overrides, includeChecksum);}-- if (start < 0 || start >= totalSize || start > end) {- if (totalSize == 0 && rangeSpec.startsWith("-")) {- return fullObjectResponse(obj, body, overrides, includeChecksum);- }- return invalidRangeResponse(totalSize);+ if (resolution.outcome() != ByteRange.Outcome.PART) {+ emitCloudTrailEvent("GetObject", bucket, key, 0L, 0L,+ "InvalidRange", ByteRange.NOT_SATISFIABLE_MESSAGE);+ return invalidRangeResponse(rangeHeader, totalSize);}- long length = end - start + 1;+ ByteRange range = resolution.range();+ emitCloudTrailEvent("GetObject", bucket, key, 0L, range.length(), null, null);StreamingOutput stream = output -> {- try (InputStream in = body) {- in.skipNBytes(start);- copyExactly(in, output, length);+ try (InputStream in = range.slice(body)) {+ in.transferTo(output);}};var resp = Response.status(206).entity(stream).header("Content-Type", overrides.contentType() != null ? overrides.contentType() : obj.getContentType())- .header("Content-Length", length)- .header("Content-Range", "bytes " + start + "-" + end + "/" + totalSize)+ .header("Content-Length", range.length())+ .header("Content-Range", range.contentRange(totalSize)).header("ETag", obj.getETag()).header("Last-Modified", RFC_822.format(obj.getLastModified())).header("Accept-Ranges", "bytes");@@ -1183,19 +1159,6 @@ public class S3Controller {return fullObjectResponse(read.object(), read.body(), overrides, false);}- private static void copyExactly(InputStream in, OutputStream out, long length) throws IOException {- byte[] buffer = new byte[64 * 1024];- long remaining = length;- while (remaining > 0) {- int n = in.read(buffer, 0, (int) Math.min(buffer.length, remaining));- if (n < 0) {- throw new EOFException("S3 object ended " + remaining + " bytes before the requested range");- }- out.write(buffer, 0, n);- remaining -= n;- }- }-private static void closeQuietly(Closeable closeable) {try {closeable.close();@@ -1204,19 +1167,21 @@ public class S3Controller {}}- private Response invalidRangeResponse(long totalSize) {+ private Response invalidRangeResponse(String rangeHeader, long totalSize) {String xml = new XmlBuilder().raw("<?xml version=\"1.0\" encoding=\"UTF-8\"?>").start("Error").elem("Code", "InvalidRange")- .elem("Message", "The requested range is not satisfiable.")+ .elem("Message", ByteRange.NOT_SATISFIABLE_MESSAGE)+ .elem("RangeRequested", rangeHeader)+ .elem("ActualObjectSize", totalSize).elem("RequestId", java.util.UUID.randomUUID().toString()).end("Error").build();return Response.status(416).entity(xml).type(MediaType.APPLICATION_XML)- .header("Content-Range", "bytes */" + totalSize)+ .header("Content-Range", ByteRange.unsatisfiedContentRange(totalSize)).build();}@@ -1281,16 +1246,35 @@ public class S3Controller {return xmlErrorResponse(new AwsException("InvalidRequest","Request specific response headers cannot be used for anonymous GET requests.", 400));}+ // "If the Range is satisfiable, only the ContentLength is affected in the response. If the+ // Range is not satisfiable, S3 returns a 416." A Range that is not one valid range is ignored.+ long contentLength = obj.getSize();+ boolean ranged = false;+ String rangeHeader = httpHeaders.getHeaderString("Range");+ if (ByteRange.inBytes(rangeHeader) && ByteRange.rangeApplies(+ httpHeaders.getHeaderString("If-Range"), obj.getETag(), obj.getLastModified())) {+ ByteRange.Resolution resolution = ByteRange.resolve(rangeHeader, obj.getSize());+ if (resolution.outcome() == ByteRange.Outcome.PART) {+ contentLength = resolution.range().length();+ ranged = true;+ } else if (resolution.outcome() == ByteRange.Outcome.NOT_SATISFIABLE) {+ emitCloudTrailEvent("HeadObject", bucket, key, 0L, 0L,+ "InvalidRange", ByteRange.NOT_SATISFIABLE_MESSAGE);+ return headOnlyResponse(invalidRangeResponse(rangeHeader, obj.getSize()));+ }+ }var resp = Response.ok().header("Content-Type", overrides.contentType() != null ? overrides.contentType() : obj.getContentType())- .header("Content-Length", obj.getSize())+ .header("Content-Length", contentLength).header("ETag", obj.getETag()).header("Last-Modified", RFC_822.format(obj.getLastModified())).header("Accept-Ranges", "bytes");if (obj.getVersionId() != null) {resp.header("x-amz-version-id", obj.getVersionId());}- boolean includeChecksum = "ENABLED".equalsIgnoreCase(resolveHeaderOrQueryParam(checksumMode, uriInfo, "x-amz-checksum-mode"));+ // As on a ranged GetObject, the stored checksum covers the whole object, not the range.+ boolean includeChecksum = !ranged+ && "ENABLED".equalsIgnoreCase(resolveHeaderOrQueryParam(checksumMode, uriInfo, "x-amz-checksum-mode"));appendObjectHeaders(resp, obj, overrides, includeChecksum);emitCloudTrailEvent("HeadObject", bucket, key, 0L, obj.getSize(), null, null);return resp.build();diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.javaindex f64c627cd6..656f7b9c4f 100644--- a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java+++ b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java@@ -4060,7 +4060,7 @@ public class S3Service implements Resettable, ResourceProvider {"Your proposed upload exceeds the maximum allowed object size.", 400);}return storePart(destBucket, destKey, uploadId, partNumber,- new CopyRangeInputStream(read.body(), range), UploadChecksums.NONE,+ new ByteRangeInputStream(read.body(), range.first(), range.length()), UploadChecksums.NONE,sseCustomerHeaders.algorithm(), sseCustomerHeaders.key(), sseCustomerHeaders.keyMd5()).getETag();} catch (IOException e) {diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontCustomOriginServingTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontCustomOriginServingTest.javaindex 18197a5cfa..db2e634790 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontCustomOriginServingTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontCustomOriginServingTest.java@@ -34,6 +34,7 @@ import java.security.KeyPairGenerator;import java.security.Signature;import java.time.Duration;import java.time.Instant;+import java.util.Arrays;import java.util.Base64;import java.util.LinkedHashMap;import java.util.List;@@ -424,6 +425,27 @@ class CloudFrontCustomOriginServingTest {});}+ @Test+ void forwardsARangeGetWhateverTheCachePolicyAndPassesThePartialResponse() throws Exception {+ byte[] video = randomBytes(1000);+ AtomicReference<ReceivedRequest> received = new AtomicReference<>();+ startRangeOrigin(video, received);+ // CachingDisabled forwards no viewer headers, but CloudFront still forwards a range GET.+ Distribution created = cloudFrontService.createDistribution(+ customOriginDistribution(policyBehavior(CACHING_DISABLED, null)), Map.of());++ Response response = given().header("Host", created.getDomainName())+ .header("Range", "bytes=100-199").header("If-Range", "\"v1\"")+ .when().get("/video.bin");++ assertEquals(206, response.statusCode());+ assertEquals("bytes 100-199/1000", response.header("Content-Range"));+ assertEquals("100", response.header("Content-Length"));+ assertArrayEquals(Arrays.copyOfRange(video, 100, 200), response.asByteArray());+ assertEquals("bytes=100-199", received.get().header("Range"));+ assertEquals("\"v1\"", received.get().header("If-Range"));+ }+@Testvoid forwardsTheViewerAuthorizationOnWriteMethodsOnly() throws Exception {AtomicReference<ReceivedRequest> received = new AtomicReference<>();@@ -826,6 +848,30 @@ class CloudFrontCustomOriginServingTest {originServer.start();}+ /** An origin that answers a GET for {@code body} with the one {@code bytes=first-last} range asked for. */+ private void startRangeOrigin(byte[] body, AtomicReference<ReceivedRequest> received) throws IOException {+ originServer = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);+ originServer.createContext("/", exchange -> {+ received.set(new ReceivedRequest(exchange.getRequestMethod(), exchange.getRequestURI().getRawPath(),+ exchange.getRequestURI().getRawQuery(), exchange.getRequestHeaders(), new byte[0]));+ exchange.getResponseHeaders().add("Content-Type", "application/octet-stream");+ String range = exchange.getRequestHeaders().getFirst("Range");+ if (range == null) {+ exchange.sendResponseHeaders(200, body.length);+ exchange.getResponseBody().write(body);+ } else {+ String[] bounds = range.substring("bytes=".length()).split("-");+ int first = Integer.parseInt(bounds[0]);+ int last = Integer.parseInt(bounds[1]);+ exchange.getResponseHeaders().add("Content-Range", "bytes " + first + "-" + last + "/" + body.length);+ exchange.sendResponseHeaders(206, last - first + 1);+ exchange.getResponseBody().write(body, first, last - first + 1);+ }+ exchange.close();+ });+ originServer.start();+ }+/*** POSTs {@code body} with {@code Transfer-Encoding: chunked} over a raw connection, since an HTTP* client library buffers a stream it is given and sends its length instead.diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontDistributionServingTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontDistributionServingTest.javaindex ffa319f7c1..8bc6bda761 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontDistributionServingTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontDistributionServingTest.java@@ -52,6 +52,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue;class CloudFrontDistributionServingTest {private static final String REGION = "us-east-1";+ private static final String ALPHABET = "abcdefghijklmnopqrstuvwxyz";@InjectS3Service s3Service;@@ -283,6 +284,86 @@ class CloudFrontDistributionServingTest {.header("Content-Length", equalTo(Integer.toString(body.getBytes(StandardCharsets.UTF_8).length)));}+ @Test+ void servesTheByteRangeAViewerAsksForFromAnS3Object() {+ Distribution dist = alphabetDistribution();++ assertRange(dist, "bytes=2-5", "bytes 2-5/26", "cdef");+ assertRange(dist, "bytes=20-", "bytes 20-25/26", "uvwxyz");+ assertRange(dist, "bytes=-3", "bytes 23-25/26", "xyz");+ assertRange(dist, "bytes=-100", "bytes 0-25/26", ALPHABET);+ assertRange(dist, "bytes=24-100", "bytes 24-25/26", "yz");+ assertRange(dist, "Bytes=2-5", "bytes 2-5/26", "cdef");+ // HeadObject: "If the Range is satisfiable, only the ContentLength is affected in the response."+ given().header("Host", dist.getDomainName()).header("Range", "bytes=2-5").when().head("/alphabet.txt")+ .then().statusCode(200)+ .header("Content-Length", equalTo("4"))+ .header("Content-Range", nullValue());+ }++ @Test+ void servesTheWholeS3ObjectWhenItsIfRangeNoLongerMatches() {+ Distribution dist = alphabetDistribution();+ String etag = given().header("Host", dist.getDomainName()).when().head("/alphabet.txt")+ .then().extract().header("ETag");++ given().header("Host", dist.getDomainName()).header("Range", "bytes=2-5").header("If-Range", etag)+ .when().get("/alphabet.txt")+ .then().statusCode(206)+ .body(equalTo("cdef"));+ given().header("Host", dist.getDomainName()).header("Range", "bytes=2-5").header("If-Range", "\"stale\"")+ .when().get("/alphabet.txt")+ .then().statusCode(200)+ .header("Content-Range", nullValue())+ .body(equalTo(ALPHABET));+ }++ @Test+ void answersARangePastTheEndOfAnS3ObjectWith416() {+ Distribution dist = alphabetDistribution();++ for (String range : List.of("bytes=26-", "bytes=100-200", "bytes=-0")) {+ given().header("Host", dist.getDomainName()).header("Range", range).when().get("/alphabet.txt")+ .then().statusCode(416)+ .header("Content-Range", equalTo("bytes */26"))+ .body(equalTo("The requested range is not satisfiable"));+ given().header("Host", dist.getDomainName()).header("Range", range).when().head("/alphabet.txt")+ .then().statusCode(416);+ }+ }++ @Test+ void servesTheWholeS3ObjectForARangeHeaderCloudFrontDoesNotAccept() {+ Distribution dist = alphabetDistribution();++ for (String range : List.of("bytes=5-2", "bytes=0-1,4-5", "bytes=abc", "bytes=-", "items=0-1")) {+ given().header("Host", dist.getDomainName()).header("Range", range).when().get("/alphabet.txt")+ .then().statusCode(200)+ .header("Content-Length", equalTo("26"))+ .header("Content-Range", nullValue())+ .body(equalTo(ALPHABET));+ }+ }++ @Test+ void aViewerRangeIsNotAppliedToTheErrorPageServedInstead() {+ String suffix = suffix();+ String bucket = "cf-range-error-page-" + suffix;+ createBucket(bucket);+ putObject(bucket, "index.html", "INDEX-" + suffix, "text/html");+ DistributionConfig cfg = new DistributionConfig();+ cfg.setEnabled(true);+ cfg.setOrigins(List.of(s3Origin("only-origin", bucket)));+ cfg.setDefaultCacheBehavior(defaultBehavior("only-origin"));+ cfg.setCustomErrorResponses(List.of(customError(404, 200, "/index.html")));+ Distribution dist = cloudFrontService.createDistribution(distribution(cfg), Map.of());++ given().header("Host", dist.getDomainName()).header("Range", "bytes=0-1").when().get("/missing/route")+ .then().statusCode(200)+ .header("Content-Range", nullValue())+ .body(equalTo("INDEX-" + suffix));+ }+@Testvoid returnsErrorPageOriginStatusWhenCustomErrorPageIsMissing() {// AWS: if the configured custom error page is itself unavailable, CloudFront returns the@@ -1605,6 +1686,27 @@ class CloudFrontDistributionServingTest {return new SignerResources(publicKey.getId(), keyGroup.getId());}+ /** A distribution in front of a bucket holding {@code alphabet.txt}, the 26 letters. */+ private Distribution alphabetDistribution() {+ String bucket = "cf-range-" + suffix();+ createBucket(bucket);+ putObject(bucket, "alphabet.txt", ALPHABET, "text/plain");+ DistributionConfig cfg = new DistributionConfig();+ cfg.setEnabled(true);+ cfg.setOrigins(List.of(s3Origin("only-origin", bucket)));+ cfg.setDefaultCacheBehavior(defaultBehavior("only-origin"));+ return cloudFrontService.createDistribution(distribution(cfg), Map.of());+ }++ private static void assertRange(Distribution dist, String range, String contentRange, String body) {+ given().header("Host", dist.getDomainName()).header("Range", range).when().get("/alphabet.txt")+ .then().statusCode(206)+ .header("Content-Range", equalTo(contentRange))+ .header("Content-Length", equalTo(Integer.toString(body.length())))+ .header("Accept-Ranges", equalTo("bytes"))+ .body(equalTo(body));+ }+private void createBucket(String bucket) {s3Service.createBucket(bucket, REGION);}diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilderTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilderTest.javaindex c67950f304..766786c0b6 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilderTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilderTest.java@@ -267,6 +267,22 @@ class CloudFrontOriginRequestBuilderTest {viewer("GET", null, "Content-Type", "application/json"), nothing), "Content-Type"));}+ @Test+ void aRangeReadReachesTheOriginWhateverThePolicy() {+ Forwarding nothing = Forwarding.policies(null, null, false);+ for (String method : List.of("GET", "HEAD")) {+ OriginRequest read = build(viewer(method, null, "Range", "bytes=0-99", "If-Range", "\"v1\""), nothing);+ assertEquals("bytes=0-99", value(read, "Range"), method);+ assertEquals("\"v1\"", value(read, "If-Range"), method);+ }++ OriginRequest post = build(viewer("POST", null, "Range", "bytes=0-99"), nothing);+ assertNull(value(post, "Range"), "a write follows the forwarding settings");++ assertNull(value(build(viewer("GET", null, "If-Range", "\"v1\""), nothing), "If-Range"),+ "an If-Range without a Range qualifies nothing");+ }+@Testvoid hopByHopAndCloudFrontOwnedHeadersAreNeverForwarded() {OriginRequest request = build(viewer("POST", null,diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontS3DiskOriginServingTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontS3DiskOriginServingTest.javaindex f6b539d186..b1fb090d7e 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontS3DiskOriginServingTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontS3DiskOriginServingTest.java@@ -13,6 +13,7 @@ import jakarta.inject.Inject;import org.junit.jupiter.api.Test;import java.nio.charset.StandardCharsets;+import java.util.Arrays;import java.util.LinkedHashMap;import java.util.List;import java.util.Map;@@ -62,6 +63,12 @@ class CloudFrontS3DiskOriginServingTest {given().header("Host", dist.getDomainName()).when().head("/video.bin").then().statusCode(200).header("Content-Length", equalTo(Integer.toString(body.length)));+ // A range from the middle of the object file.+ Response part = given().header("Host", dist.getDomainName())+ .header("Range", "bytes=1048576-1048675").when().get("/video.bin");+ assertEquals(206, part.statusCode());+ assertEquals("bytes 1048576-1048675/" + body.length, part.header("Content-Range"));+ assertArrayEquals(Arrays.copyOfRange(body, 1048576, 1048676), part.asByteArray());// The single-page-app fallback serves its error page through the same stream.given().header("Host", dist.getDomainName()).when().get("/missing/route").then().statusCode(200)diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudtrail/CloudTrailIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudtrail/CloudTrailIntegrationTest.javaindex ed230d130f..06e58b6eb0 100644--- a/src/test/java/io/github/hectorvent/floci/services/cloudtrail/CloudTrailIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/cloudtrail/CloudTrailIntegrationTest.java@@ -101,6 +101,16 @@ class CloudTrailIntegrationTest {headObject(sourceBucket, "documents/hello.txt", 200);listObjects(sourceBucket);getObject(sourceBucket, "documents/missing.txt", 404);+ // Ranged reads are recorded too, a range that starts past the end of the object as InvalidRange.+ given().header("Range", "bytes=0-4")+ .when().get("/" + sourceBucket + "/documents/hello.txt")+ .then().statusCode(206);+ given().header("Range", "bytes=100-")+ .when().get("/" + sourceBucket + "/documents/hello.txt")+ .then().statusCode(416);+ given().header("Range", "bytes=100-")+ .when().head("/" + sourceBucket + "/documents/hello.txt")+ .then().statusCode(416);deleteObject(sourceBucket, "documents/hello.txt");// 6. Force the writer to flush — bypasses the scheduled cadence.@@ -145,6 +155,7 @@ class CloudTrailIntegrationTest {+ ":\n" + new String(json));boolean sawPut = false, sawGet = false, sawDelete = false, sawNoSuchKey = false;+ boolean sawRangedGet = false, sawGetInvalidRange = false, sawHeadInvalidRange = false;for (JsonNode rec : records) {assertEquals("1.11", rec.path("eventVersion").asText(),"Bad eventVersion: " + rec);@@ -164,11 +175,22 @@ class CloudTrailIntegrationTest {if ("GetObject".equals(name) && rec.path("errorCode").isMissingNode()) sawGet = true;if ("DeleteObject".equals(name)) sawDelete = true;if ("NoSuchKey".equals(rec.path("errorCode").asText(null))) sawNoSuchKey = true;+ if ("GetObject".equals(name) && rec.path("errorCode").isMissingNode()+ && rec.path("additionalEventData").path("bytesTransferredOut").asLong() == 5) {+ sawRangedGet = true;+ }+ if ("InvalidRange".equals(rec.path("errorCode").asText(null))) {+ sawGetInvalidRange |= "GetObject".equals(name);+ sawHeadInvalidRange |= "HeadObject".equals(name);+ }}assertTrue(sawPut, "Expected a PutObject record");assertTrue(sawGet, "Expected a successful GetObject record");assertTrue(sawDelete, "Expected a DeleteObject record");assertTrue(sawNoSuchKey, "Expected a GetObject NoSuchKey record");+ assertTrue(sawRangedGet, "Expected a GetObject record of the 5 bytes of a ranged read");+ assertTrue(sawGetInvalidRange, "Expected a GetObject InvalidRange record");+ assertTrue(sawHeadInvalidRange, "Expected a HeadObject InvalidRange record");}@Testdiff --git a/src/test/java/io/github/hectorvent/floci/services/s3/ByteRangeTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/ByteRangeTest.javanew file mode 100644index 0000000000..8cfff2acdd--- /dev/null+++ b/src/test/java/io/github/hectorvent/floci/services/s3/ByteRangeTest.java@@ -0,0 +1,134 @@+package io.github.hectorvent.floci.services.s3;++import io.github.hectorvent.floci.services.s3.ByteRange.Outcome;+import io.github.hectorvent.floci.services.s3.ByteRange.Resolution;+import org.junit.jupiter.api.Test;++import java.io.ByteArrayInputStream;+import java.io.IOException;+import java.io.InputStream;+import java.nio.charset.StandardCharsets;+import java.time.Instant;+import java.util.List;++import static org.junit.jupiter.api.Assertions.assertArrayEquals;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertNull;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;++class ByteRangeTest {++ @Test+ void aSingleRangeResolvesToItsBytes() {+ assertPart("bytes=0-9", 100, 0, 9);+ assertPart("bytes=90-", 100, 90, 99);+ assertPart("bytes=-10", 100, 90, 99);+ assertPart("bytes= 5-6 ", 100, 5, 6);+ }++ @Test+ void theRangeUnitMatchesInAnyCase() {+ assertPart("Bytes=0-9", 100, 0, 9);+ assertPart("BYTES=90-", 100, 90, 99);+ assertOutcome(Outcome.WHOLE, "byte=0-9", 100);+ }++ @Test+ void aByteRangeRunsForwardFromAFirstByteThatExists() {+ assertThrows(IllegalArgumentException.class, () -> new ByteRange(5, 2));+ assertThrows(IllegalArgumentException.class, () -> new ByteRange(-1, 2));+ assertEquals(1, new ByteRange(7, 7).length());+ assertEquals("bytes */100", ByteRange.unsatisfiedContentRange(100));+ }++ @Test+ void aRangeRunningPastTheEndStopsAtTheLastByte() {+ assertPart("bytes=95-200", 100, 95, 99);+ assertPart("bytes=-500", 100, 0, 99);+ assertPart("bytes=0-99999999999999999999", 100, 0, 99);+ assertPart("bytes=-99999999999999999999", 100, 0, 99);+ }++ @Test+ void offsetsPastTwoGibibytesResolve() {+ long size = 3L * 1024 * 1024 * 1024;+ ByteRange range = assertPart("bytes=3000000000-3000000009", size, 3_000_000_000L, 3_000_000_009L);++ assertEquals(10, range.length());+ assertEquals("bytes 3000000000-3000000009/" + size, range.contentRange(size));+ }++ @Test+ void aRangeStartingPastTheEndIsNotSatisfiable() {+ for (String header : List.of("bytes=100-", "bytes=100-200", "bytes=-0", "bytes=99999999999999999999-")) {+ assertOutcome(Outcome.NOT_SATISFIABLE, header, 100);+ }+ assertOutcome(Outcome.NOT_SATISFIABLE, "bytes=0-", 0);+ assertOutcome(Outcome.NOT_SATISFIABLE, "bytes=0-0", 0);+ }++ @Test+ void anythingButOneValidRangeIsInvalid() {+ for (String header : List.of("bytes=5-2", "bytes=0-1,4-5", "bytes=abc", "bytes=-", "bytes=1",+ "bytes=--5", "bytes=+1-2", "bytes=")) {+ assertOutcome(Outcome.INVALID, header, 100);+ }+ }++ @Test+ void noHeaderAnotherUnitOrASuffixOfAnEmptyObjectIsTheWholeObject() {+ assertOutcome(Outcome.WHOLE, null, 100);+ assertOutcome(Outcome.WHOLE, "items=0-1", 100);+ assertOutcome(Outcome.WHOLE, "bytes=-5", 0);+ }++ @Test+ void anIfRangeThatStillDescribesTheObjectLetsTheRangeApply() {+ Instant modified = Instant.parse("2015-10-21T07:28:00.250Z");++ assertTrue(ByteRange.rangeApplies(null, "\"abc\"", modified));+ assertTrue(ByteRange.rangeApplies("\"abc\"", "\"abc\"", modified));+ assertTrue(ByteRange.rangeApplies("\"abc-2\"", "\"abc-2\"", modified), "a multipart ETag, whole");+ assertTrue(ByteRange.rangeApplies("Wed, 21 Oct 2015 07:28:00 GMT", "\"abc\"", modified),+ "Last-Modified, to the second");+ }++ @Test+ void anIfRangeThatNoLongerDescribesTheObjectServesItWhole() {+ Instant modified = Instant.parse("2015-10-21T07:28:00.250Z");++ assertFalse(ByteRange.rangeApplies("\"old\"", "\"abc\"", modified));+ assertFalse(ByteRange.rangeApplies("\"2\"", "\"abc-2\"", modified), "a multipart suffix alone");+ assertFalse(ByteRange.rangeApplies("Tue, 20 Oct 2015 07:28:00 GMT", "\"abc\"", modified));+ assertFalse(ByteRange.rangeApplies("Thu, 22 Oct 2015 07:28:00 GMT", "\"abc\"", modified),+ "a date later than Last-Modified");+ for (String malformed : List.of("W/\"abc\"", "abc", "\"", "2015-10-21T07:28:00Z", "")) {+ assertFalse(ByteRange.rangeApplies(malformed, "\"abc\"", modified), malformed);+ }+ }++ @Test+ void aSliceReadsTheRangeFromAStreamOverTheWholeObject() throws IOException {+ byte[] object = "abcdefghijklmnopqrstuvwxyz".getBytes(StandardCharsets.US_ASCII);+ ByteRange range = assertPart("bytes=2-5", object.length, 2, 5);++ try (InputStream slice = range.slice(new ByteArrayInputStream(object))) {+ assertArrayEquals("cdef".getBytes(StandardCharsets.US_ASCII), slice.readAllBytes());+ }+ }++ private static ByteRange assertPart(String header, long size, long first, long last) {+ Resolution resolution = ByteRange.resolve(header, size);+ assertEquals(Outcome.PART, resolution.outcome(), header);+ assertEquals(new ByteRange(first, last), resolution.range(), header);+ return resolution.range();+ }++ private static void assertOutcome(Outcome outcome, String header, long size) {+ Resolution resolution = ByteRange.resolve(header, size);+ assertEquals(outcome, resolution.outcome(), header + " of " + size + " bytes");+ assertNull(resolution.range(), header);+ }+}diff --git a/src/test/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilterTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilterTest.javaindex 784ebd46e9..3fcc4df5bd 100644--- a/src/test/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilterTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilterTest.java@@ -201,6 +201,23 @@ class PreSignedUrlFilterTest {"host;x-amz-meta-signed"));}+ @Test+ void requiresIfRangeSignedWhenRangeIsSigned() {+ assertEquals(+ List.of("if-range"),+ PreSignedUrlFilter.unsignedHeadersRequiringSignature(+ Set.of("host", "Range", "If-Range"), "host;range"));+ assertEquals(+ List.of(),+ PreSignedUrlFilter.unsignedHeadersRequiringSignature(+ Set.of("host", "range", "if-range"), "host;range;if-range"));+ assertEquals(+ List.of(),+ PreSignedUrlFilter.unsignedHeadersRequiringSignature(+ Set.of("host", "range", "if-range"), "host"),+ "an unsigned Range leaves If-Range unsigned too");+ }+@Testvoid ignoresHeadersOutsideChecksumFamily() {assertEquals(diff --git a/src/test/java/io/github/hectorvent/floci/services/s3/S3CopySourceRangeTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/S3CopySourceRangeTest.javaindex 73694132b1..89a5f24475 100644--- a/src/test/java/io/github/hectorvent/floci/services/s3/S3CopySourceRangeTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/s3/S3CopySourceRangeTest.java@@ -95,7 +95,8 @@ class S3CopySourceRangeTest {void aRangeSkipsToAnOffsetPastTwoGibibytes() throws IOException {CopySourceRange range = new CopySourceRange(2_500_000_000L, 2_500_000_015L);- byte[] data = new CopyRangeInputStream(patternStream(3L * 1024 * 1024 * 1024), range).readAllBytes();+ byte[] data = new ByteRangeInputStream(patternStream(3L * 1024 * 1024 * 1024), range.first(), range.length())+ .readAllBytes();assertEquals(16, data.length);for (int i = 0; i < data.length; i++) {@@ -119,8 +120,7 @@ class S3CopySourceRangeTest {byte[] data;try (FileChannel channel = FileChannel.open(file, StandardOpenOption.READ)) {- data = new CopyRangeInputStream(Channels.newInputStream(channel),- new CopySourceRange(offset, offset + written.length - 1)).readAllBytes();+ data = new ByteRangeInputStream(Channels.newInputStream(channel), offset, written.length).readAllBytes();}assertArrayEquals(written, data);@@ -129,9 +129,9 @@ class S3CopySourceRangeTest {@Testvoid aRangeFailsWhenTheSourceEndsBeforeIt() {assertThrows(IOException.class,- () -> new CopyRangeInputStream(patternStream(10), new CopySourceRange(5, 14)).readAllBytes());+ () -> new ByteRangeInputStream(patternStream(10), 5, 10).readAllBytes());assertThrows(IOException.class,- () -> new CopyRangeInputStream(patternStream(10), new CopySourceRange(12, 14)).readAllBytes(),+ () -> new ByteRangeInputStream(patternStream(10), 12, 3).readAllBytes(),"a source that ends before the range starts");}@@ -139,7 +139,7 @@ class S3CopySourceRangeTest {void aRangeEndsAfterItsLastByteWhateverTheReadSize() throws IOException {CopySourceRange range = new CopySourceRange(3, 1002);for (int readSize : List.of(1, 7, 1000, 4096)) {- try (InputStream in = new CopyRangeInputStream(patternStream(5000), range)) {+ try (InputStream in = new ByteRangeInputStream(patternStream(5000), range.first(), range.length())) {ByteArrayOutputStream copied = new ByteArrayOutputStream();byte[] buffer = new byte[readSize];for (int read = in.read(buffer); read >= 0; read = in.read(buffer)) {diff --git a/src/test/java/io/github/hectorvent/floci/services/s3/S3IntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/S3IntegrationTest.javaindex e2ab433a05..7b5b3ecea2 100644--- a/src/test/java/io/github/hectorvent/floci/services/s3/S3IntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/s3/S3IntegrationTest.java@@ -1031,6 +1031,108 @@ class S3IntegrationTest {given().delete("/checksum-mode-head-bucket");}+ @Test+ @Order(123)+ void getObjectWithARangeUnitInAnyCase() {+ putGreeting("range-unit-bucket");+ try {+ // Range unit names are case-insensitive (RFC 9110, section 14.1).+ given()+ .header("Range", "Bytes=0-4")+ .when()+ .get("/range-unit-bucket/greeting.txt")+ .then()+ .statusCode(206)+ .header("Content-Range", equalTo("bytes 0-4/20"))+ .body(equalTo("Hello"));+ } finally {+ deleteGreeting("range-unit-bucket");+ }+ }++ @Test+ @Order(124)+ void getObjectWithIfRangeServesTheRangeOnlyWhileItsValidatorMatches() {+ putGreeting("if-range-bucket");+ try {+ String etag = given().when().head("/if-range-bucket/greeting.txt").then().extract().header("ETag");+ String lastModified = given().when().head("/if-range-bucket/greeting.txt").then().extract()+ .header("Last-Modified");++ for (String current : List.of(etag, lastModified)) {+ given()+ .header("Range", "bytes=0-4")+ .header("If-Range", current)+ .when()+ .get("/if-range-bucket/greeting.txt")+ .then()+ .statusCode(206)+ .header("Content-Range", equalTo("bytes 0-4/20"))+ .body(equalTo("Hello"));+ }+ given()+ .header("Range", "bytes=0-4")+ .header("If-Range", "\"stale\"")+ .when()+ .get("/if-range-bucket/greeting.txt")+ .then()+ .statusCode(200)+ .header("Content-Range", nullValue())+ .header("Content-Length", equalTo("20"));+ } finally {+ deleteGreeting("if-range-bucket");+ }+ }++ @Test+ @Order(125)+ void headObjectWithRangeReportsTheRangeLengthOr416() {+ putGreeting("head-range-bucket");+ try {+ // "If the Range is satisfiable, only the ContentLength is affected in the response. If the+ // Range is not satisfiable, S3 returns a 416".+ given()+ .header("Range", "bytes=0-4")+ .header("x-amz-checksum-mode", "ENABLED")+ .when()+ .head("/head-range-bucket/greeting.txt")+ .then()+ .statusCode(200)+ .header("Content-Length", equalTo("5"))+ .header("Content-Range", nullValue())+ .header("x-amz-checksum-crc64nvme", nullValue())+ .header("x-amz-checksum-type", nullValue());+ given()+ .header("Range", "bytes=0-4")+ .header("If-Range", "\"stale\"")+ .header("x-amz-checksum-mode", "ENABLED")+ .when()+ .head("/head-range-bucket/greeting.txt")+ .then()+ .statusCode(200)+ .header("Content-Length", equalTo("20"))+ .header("x-amz-checksum-crc64nvme", notNullValue());+ given()+ .header("Range", "bytes=50-")+ .when()+ .head("/head-range-bucket/greeting.txt")+ .then()+ .statusCode(416);+ // A Range that is not one valid range is not one that cannot be satisfied, so it is ignored.+ for (String invalid : List.of("bytes=0-1,4-5", "bytes=5-2", "bytes=abc")) {+ given()+ .header("Range", invalid)+ .when()+ .head("/head-range-bucket/greeting.txt")+ .then()+ .statusCode(200)+ .header("Content-Length", equalTo("20"));+ }+ } finally {+ deleteGreeting("head-range-bucket");+ }+ }+@Test@Order(27)void getObjectWithFullRange() {@@ -1082,7 +1184,10 @@ class S3IntegrationTest {.then().statusCode(416).header("Content-Range", equalTo("bytes */20"))- .body(containsString("InvalidRange"));+ .body(hasXPath("/Error/Code", equalTo("InvalidRange")))+ .body(hasXPath("/Error/Message", equalTo("The requested range is not satisfiable")))+ .body(hasXPath("/Error/RangeRequested", equalTo("bytes=50-100")))+ .body(hasXPath("/Error/ActualObjectSize", equalTo("20")));}@Test@@ -3199,6 +3304,17 @@ class S3IntegrationTest {}}+ /** Creates {@code bucket} holding {@code greeting.txt}, the 20 bytes "Hello World from S3!". */+ private static void putGreeting(String bucket) {+ given().when().put("/" + bucket).then().statusCode(200);+ given().body("Hello World from S3!").when().put("/" + bucket + "/greeting.txt").then().statusCode(200);+ }++ private static void deleteGreeting(String bucket) {+ given().delete("/" + bucket + "/greeting.txt");+ given().delete("/" + bucket);+ }+private static byte[] alphabetBytes(int size) {byte[] bytes = new byte[size];for (int i = 0; i < size; i++) {