Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

fix(cloudfront): serve viewer Range GETs (#5377)

Priamo Germosenff20750d9123
diff --git a/docs/services/cloudfront.md b/docs/services/cloudfront.md
index f5ee9caf74..f58b563669 100644
--- a/docs/services/cloudfront.md
+++ b/docs/services/cloudfront.md
@@ -196,7 +196,8 @@ PUT, PATCH and DELETE).
union of both). `HeaderBehavior` `none`, `whitelist`, `allViewer`,
`allViewerAndWhitelistCloudFront` and `allExcept`, `CookieBehavior` and `QueryStringBehavior`
`none`, `whitelist`, `allExcept` and `all` are evaluated. A cache policy that selects nothing and no
- origin request policy forwards no viewer headers, cookies or query strings, as on AWS.
+ origin request policy forwards no viewer headers, cookies or query strings, as on AWS, apart from
+ the headers below that CloudFront forwards whatever the policies.
- A cache behavior without a cache policy uses its legacy `ForwardedValues`: `QueryString=true`
forwards every query string (`QueryStringCacheKeys` only narrows the cache key), cookies follow
`Forward` `none`, `whitelist` (with `*` and `?` wildcards) or `all`, and viewer headers are
@@ -209,11 +210,13 @@ PUT, PATCH and DELETE).
forwarded, appends the viewer address to `X-Forwarded-For`, adds a
`Via: <viewer HTTP version> <distribution domain> (CloudFront)` hop and an `X-Amz-Cf-Id`.
`Authorization` is always forwarded on POST, PUT, PATCH and DELETE, and on OPTIONS unless OPTIONS
- is a cached method; `Content-Type` travels with a request body. Cache policy compression settings
- normalize `Accept-Encoding` to `br,gzip`, `gzip`, `br` or `identity`. Forwarded cookies are sorted
- by name, and cookies that are not `name=value` or whose name starts with `$` are dropped.
- Hop-by-hop headers, `Expect`, `X-Edge-*`, `X-Real-IP`, `X-Forwarded-Proto` and viewer-supplied
- `CloudFront-*` headers are never forwarded.
+ is a cached method; `Content-Type` travels with a request body; a GET or HEAD with a `Range`
+ carries the viewer's `Range` and `If-Range`, since CloudFront forwards a range GET to the origin,
+ and a HEAD can then report the range's length. Cache policy compression settings normalize
+ `Accept-Encoding` to `br,gzip`, `gzip`, `br` or `identity`.
+ Forwarded cookies are sorted by name, and cookies that are not `name=value` or whose name starts
+ with `$` are dropped. Hop-by-hop headers, `Expect`, `X-Edge-*`, `X-Real-IP`, `X-Forwarded-Proto`
+ and viewer-supplied `CloudFront-*` headers are never forwarded.
- Of the CloudFront request headers, `CloudFront-Forwarded-Proto`, `CloudFront-Viewer-Address` and
`CloudFront-Viewer-Http-Version` are generated when a policy names them, or when an `allExcept`
header behavior does not exclude them (AWS documents Managed-AllViewerExceptHostHeader as adding
@@ -225,11 +228,22 @@ PUT, PATCH and DELETE).
`CachingOptimized` and `CachingOptimizedForUncompressedObjects` but not `Amplify`,
`Elemental-MediaPackage` or the `UseOriginCacheControlHeaders` policies.
- In-process S3 origins do not apply forwarding settings: viewer query strings, cookies and headers
- do not change the S3 read.
+ do not change the S3 read, apart from the viewer's `Range` and `If-Range` on a GET or HEAD.
- Origin responses are streamed to the viewer, from in-process S3 origins and custom origins alike,
so a large object is never held in memory. A custom origin's `Content-Length` is passed on, and a
response it sends chunked stays chunked. Floci does not cache, so it serves objects larger than
50 GB the way CloudFront does with caching disabled; with caching enabled, CloudFront refuses them.
+- Range GETs: an S3 origin serves the one byte range a viewer's `Range` asks for as 206 with its
+ `Content-Range`, and answers 416 when the range starts past the end of the object. Any other
+ `Range` gets the whole object: one CloudFront does not accept (ranges out of order or overlapping,
+ or a negative value), several ranges, which S3 does not serve in one request, or a last byte
+ before the first, as does a `Range` whose `If-Range` no longer describes the object, which S3
+ answers with the whole object. A HEAD is answered as S3's HeadObject answers a `Range`: a
+ satisfiable range changes only its `Content-Length`, and one past the end gets 416. A custom
+ origin gets the viewer's `Range` and `If-Range` on every GET or HEAD with a `Range`, whatever the
+ forwarding settings, and its answer (206, the whole object, or 416) is passed on. Floci does not
+ cache, so unlike CloudFront it cannot serve a later range request from an object the origin sent
+ whole. A custom error page is served whole, whatever the `Range` of the request it replaces.
- Origin `Set-Cookie` headers always reach the viewer. With legacy `Forward=none`, AWS strips them
from the response; Floci does not.
- Custom origins that resolve to loopback, private, link-local, carrier-grade NAT, or other non-routable addresses are rejected by default. Development-only private origins must be explicitly allowlisted by exact hostname.
diff --git a/docs/services/s3.md b/docs/services/s3.md
index 490d59d4cb..886611b29b 100644
--- a/docs/services/s3.md
+++ b/docs/services/s3.md
@@ -106,6 +106,10 @@ Redirect-all and advanced routing-rule configurations are not implemented.
Suffix ranges against empty objects return an empty `200` response, matching AWS behavior used by clients that issue tail reads.
+`HeadObject` honors `Range` as the API Reference describes: a satisfiable range changes `Content-Length` to the length of the range, and one that cannot be satisfied returns `416`, while a `Range` that is not one valid range, such as several ranges or a last byte before the first, is ignored. Like a ranged `GetObject`, it leaves out the whole-object checksum headers.
+
+An `If-Range` header makes the range conditional, so a resumed download never mixes two versions of an object. The range is served while the validator still describes the object: a quoted ETag equal to the object's (a multipart ETag compared whole, `-N` suffix included), or an HTTP-date equal to its `Last-Modified`, an exact match as RFC 9110 section 13.1.5 requires, so a later date does not qualify. Otherwise, including a weak (`W/`) or malformed validator, `GetObject` returns the whole object with `200`. A presigned URL that signs `Range` must also sign an `If-Range` the request carries, or the request fails with `403 AccessDenied` naming it in `HeadersNotSigned`.
+
## S3 Select
`SelectObjectContent` runs SQL queries directly against S3 objects without downloading the entire file. Floci supports CSV, JSON Lines, JSON arrays, and Parquet inputs. The SQL dialect follows [AWS S3 Select SQL reference](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-select-sql-reference.html).
diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilder.java b/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilder.java
index 819a7221e6..8d55c03b0f 100644
--- a/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilder.java
+++ b/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilder.java
@@ -28,6 +28,10 @@ import java.util.concurrent.ThreadLocalRandom;
* {@code User-Agent} is forwarded, an {@code X-Forwarded-For} ending in the viewer address,
* CloudFront's {@code Via} hop and an {@code X-Amz-Cf-Id}. {@code Host} is the origin's own
* name unless the viewer's {@code Host} is forwarded.</li>
+ * <li>Whatever the policies, {@code Authorization} goes with the methods CloudFront does not cache,
+ * {@code Content-Type} with a request body, and the viewer's {@code Range} and
+ * {@code If-Range} with a GET or HEAD that has a {@code Range}, since CloudFront forwards a
+ * range GET to the origin.</li>
* </ul>
*
* <p>This class holds no state and does no I/O. Origin custom headers are applied afterwards by the
@@ -113,6 +117,8 @@ final class CloudFrontOriginRequestBuilder {
Set<String> connectionTokens = connectionTokens(viewer.headers());
boolean forwardsAuthorization = BODY_METHODS.contains(viewer.method())
|| ("OPTIONS".equals(viewer.method()) && !forwarding.optionsCached());
+ boolean rangeRead = ("GET".equals(viewer.method()) || "HEAD".equals(viewer.method()))
+ && viewer.headers().stream().anyMatch(header -> "range".equals(normalize(header.name())));
List<Header> headers = new ArrayList<>();
List<String> cookieHeaders = new ArrayList<>();
@@ -151,6 +157,14 @@ final class CloudFrontOriginRequestBuilder {
headers.add(header);
}
}
+ case "range", "if-range" -> {
+ // CloudFront forwards a range GET to the origin whatever the cache settings, so the
+ // origin can answer with the range alone. A HEAD gets them too, so it can report the
+ // range's length as S3's HeadObject does. If-Range only qualifies a Range.
+ if (rangeRead || selections.forwardsHeader(name)) {
+ headers.add(header);
+ }
+ }
default -> {
if (selections.forwardsHeader(name)) {
headers.add(header);
diff --git a/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontServingController.java b/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontServingController.java
index 37551b01b8..b67111045e 100644
--- a/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontServingController.java
+++ b/src/main/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontServingController.java
@@ -8,6 +8,7 @@ import io.github.hectorvent.floci.services.cloudfront.model.Distribution;
import io.github.hectorvent.floci.services.cloudfront.model.DistributionConfig;
import io.github.hectorvent.floci.services.cloudfront.model.Origin;
import io.github.hectorvent.floci.services.cloudfront.model.OriginAccessControl;
+import io.github.hectorvent.floci.services.s3.ByteRange;
import io.github.hectorvent.floci.services.s3.S3Service;
import io.github.hectorvent.floci.services.s3.model.S3Object;
import io.quarkus.vertx.http.runtime.CurrentVertxRequest;
@@ -473,8 +474,9 @@ public class CloudFrontServingController {
}
String key = CloudFrontRequestRouter.resolveOriginKey(
origin.getOriginPath(), decodedViewerPath, config.getDefaultRootObject());
- return fetchFromS3(
- distribution, origin, key, viewerAuthorization, !"HEAD".equals(method));
+ HttpServerRequest request = currentVertxRequest.getCurrent().request();
+ return fetchFromS3(distribution, origin, key, viewerAuthorization, !"HEAD".equals(method),
+ new ViewerRange(request.getHeader("Range"), request.getHeader("If-Range")));
}
String forwardUri = CloudFrontRequestRouter.resolveForwardUri(
origin.getOriginPath(), rawViewerPath, config.getDefaultRootObject());
@@ -565,7 +567,8 @@ public class CloudFrontServingController {
Origin origin,
String key,
String viewerAuthorization,
- boolean includeBody) {
+ boolean includeBody,
+ ViewerRange viewerRange) {
String bucket = CloudFrontRequestRouter.bucketFromS3Domain(origin.getDomainName());
if (bucket == null) {
return OriginResponse.error(502, "Could not determine S3 bucket for origin.");
@@ -577,14 +580,9 @@ public class CloudFrontServingController {
if (includeBody) {
// The body is streamed to the viewer, so an object of any size is served without
// being read into the heap; the metadata and the stream come from one snapshot.
- S3Service.ObjectRead read = s3Service.openObject(bucket, key, null);
- S3Object obj = read.object();
- response = new OriginResponse(
- 200, contentType(obj), null, read.body(), obj.getSize(), s3ObjectHeaders(obj));
+ response = s3ObjectResponse(s3Service.openObject(bucket, key, null), viewerRange);
} else {
- S3Object meta = s3Service.headObject(bucket, key);
- response = new OriginResponse(
- 200, contentType(meta), null, meta.getSize(), s3ObjectHeaders(meta));
+ response = s3HeadResponse(s3Service.headObject(bucket, key), viewerRange);
}
} catch (AwsException e) {
response = OriginResponse.error(e.getHttpStatus(), e.getMessage());
@@ -598,6 +596,69 @@ public class CloudFrontServingController {
}
}
+ /**
+ * The GET of an S3 object, answered as S3 answers the viewer's {@code Range}: the one range it
+ * asks for, or 416 when that range starts past the end of the object. CloudFront serves the
+ * whole object, as for no {@code Range} at all, when the header is not one valid range, which
+ * covers several ranges, since S3 serves one range per request, and when an {@code If-Range}
+ * no longer describes the object.
+ */
+ private static OriginResponse s3ObjectResponse(S3Service.ObjectRead read, ViewerRange viewerRange) {
+ S3Object obj = read.object();
+ ByteRange.Resolution resolution = resolveRange(viewerRange, obj);
+ return switch (resolution.outcome()) {
+ case PART -> {
+ ByteRange range = resolution.range();
+ Map<String, List<String>> headers = s3ObjectHeaders(obj);
+ putHeader(headers, "Content-Range", range.contentRange(obj.getSize()));
+ yield new OriginResponse(206, contentType(obj), null, range.slice(read.body()),
+ range.length(), headers);
+ }
+ case NOT_SATISFIABLE -> {
+ closeUnsent(read);
+ yield rangeNotSatisfiable(obj.getSize());
+ }
+ case WHOLE, INVALID -> new OriginResponse(
+ 200, contentType(obj), null, read.body(), obj.getSize(), s3ObjectHeaders(obj));
+ };
+ }
+
+ /**
+ * The HEAD of an S3 object, answered as HeadObject answers a {@code Range}: "If the Range is
+ * satisfiable, only the ContentLength is affected in the response. If the Range is not
+ * satisfiable, S3 returns a 416". A header that is not one valid range is ignored, as for a GET.
+ */
+ private static OriginResponse s3HeadResponse(S3Object meta, ViewerRange viewerRange) {
+ ByteRange.Resolution resolution = resolveRange(viewerRange, meta);
+ return switch (resolution.outcome()) {
+ case PART -> new OriginResponse(
+ 200, contentType(meta), null, resolution.range().length(), s3ObjectHeaders(meta));
+ case NOT_SATISFIABLE -> rangeNotSatisfiable(meta.getSize());
+ case WHOLE, INVALID -> new OriginResponse(
+ 200, contentType(meta), null, meta.getSize(), s3ObjectHeaders(meta));
+ };
+ }
+
+ /** The viewer's {@code Range} resolved against an object, unless its {@code If-Range} no longer matches it. */
+ private static ByteRange.Resolution resolveRange(ViewerRange viewerRange, S3Object obj) {
+ String rangeHeader = ByteRange.rangeApplies(viewerRange.ifRange(), obj.getETag(), obj.getLastModified())
+ ? viewerRange.range() : null;
+ return ByteRange.resolve(rangeHeader, obj.getSize());
+ }
+
+ private static OriginResponse rangeNotSatisfiable(long size) {
+ return OriginResponse.error(416, ByteRange.NOT_SATISFIABLE_MESSAGE)
+ .withHeaders(Map.of("Content-Range", List.of(ByteRange.unsatisfiedContentRange(size))));
+ }
+
+ private static void closeUnsent(S3Service.ObjectRead read) {
+ try {
+ read.close();
+ } catch (IOException e) {
+ LOG.debugv("Could not close an unsent S3 object: {0}", e.getMessage());
+ }
+ }
+
private void authorizeS3OriginRead(
Distribution distribution,
Origin origin,
@@ -759,6 +820,11 @@ public class CloudFrontServingController {
record ViewerBody(InputStream stream, long length) {
}
+ /** The viewer's {@code Range} and the {@code If-Range} that qualifies it, either one possibly absent. */
+ private record ViewerRange(String range, String ifRange) {
+ static final ViewerRange NONE = new ViewerRange(null, null);
+ }
+
private static String originAccessIdentityId(Origin origin) {
Map<String, String> config = origin.getS3OriginConfig();
String value = config != null ? config.get("OriginAccessIdentity") : null;
@@ -911,8 +977,9 @@ public class CloudFrontServingController {
OriginResponse page;
if (CloudFrontRequestRouter.isS3Origin(errOrigin)) {
String key = CloudFrontRequestRouter.resolveOriginKey(errOrigin.getOriginPath(), errNormalized, null);
+ // The viewer's Range is for the object it asked for, not for the page served instead.
page = fetchFromS3(
- distribution, errOrigin, key, viewerAuthorization, includeBody);
+ distribution, errOrigin, key, viewerAuthorization, includeBody, ViewerRange.NONE);
} else {
String forwardUri = CloudFrontRequestRouter.resolveForwardUri(errOrigin.getOriginPath(), errNormalized, null);
page = fetchFromCustomOrigin(
diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/ByteRange.java b/src/main/java/io/github/hectorvent/floci/services/s3/ByteRange.java
new file mode 100644
index 0000000000..1da7a0e8d4
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/s3/ByteRange.java
@@ -0,0 +1,142 @@
+package io.github.hectorvent.floci.services.s3;
+
+import java.io.InputStream;
+import java.time.Instant;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.time.format.DateTimeParseException;
+import java.time.temporal.ChronoUnit;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * One byte range of an object, from a {@code Range} header that asks for a single range, as RFC 9110
+ * section 14.1.2 defines it. {@code last} is inclusive. S3 serves one range per GET request.
+ */
+public record ByteRange(long first, long last) {
+
+ /**
+ * The message of S3's {@code InvalidRange} error as S3 sends it, which is not the wording of the
+ * error code table's description.
+ */
+ public static final String NOT_SATISFIABLE_MESSAGE = "The requested range is not satisfiable";
+
+ private static final String UNIT = "bytes=";
+ private static final Pattern SPEC = Pattern.compile("(\\d*)-(\\d*)");
+
+ /**
+ * What a {@code Range} header asks of an object: a part of it, the whole of it (no header, a
+ * unit other than bytes, or a suffix of an empty object), a range that starts past its end, or
+ * something that is not one valid range, such as several ranges or a last byte before the first.
+ */
+ public enum Outcome { PART, WHOLE, NOT_SATISFIABLE, INVALID }
+
+ /** A resolved {@code Range} header; {@code range} is set only for {@link Outcome#PART}. */
+ public record Resolution(Outcome outcome, ByteRange range) {
+ }
+
+ public ByteRange {
+ if (first < 0 || last < first) {
+ throw new IllegalArgumentException("Not a byte range: " + first + "-" + last);
+ }
+ }
+
+ public long length() {
+ return last - first + 1;
+ }
+
+ /** The {@code Content-Range} value of this range of an object of {@code size} bytes. */
+ public String contentRange(long size) {
+ return "bytes " + first + "-" + last + "/" + size;
+ }
+
+ /** The {@code Content-Range} value of a 416 response for an object of {@code size} bytes. */
+ public static String unsatisfiedContentRange(long size) {
+ return "bytes */" + size;
+ }
+
+ /**
+ * Whether a {@code Range} header asks for bytes, the one unit S3 serves. Range unit names are
+ * case-insensitive (RFC 9110, section 14.1).
+ */
+ public static boolean inBytes(String header) {
+ return header != null && header.regionMatches(true, 0, UNIT, 0, UNIT.length());
+ }
+
+ /**
+ * Whether a request's {@code Range} applies to an object, given the request's {@code If-Range}:
+ * always without one, and otherwise when its validator still describes the object, a quoted
+ * entity tag equal to the object's ETag, or an HTTP-date equal to its Last-Modified.
+ * Any other validator, a weak or malformed one included, means the whole object is served.
+ */
+ public static boolean rangeApplies(String ifRange, String etag, Instant lastModified) {
+ if (ifRange == null) {
+ return true;
+ }
+ String validator = ifRange.trim();
+ if (validator.startsWith("\"")) {
+ return validator.length() > 1 && validator.endsWith("\"") && validator.equals(etag);
+ }
+ Instant date;
+ try {
+ date = ZonedDateTime.parse(validator, DateTimeFormatter.RFC_1123_DATE_TIME).toInstant();
+ } catch (DateTimeParseException expected) {
+ // Neither a quoted entity tag nor an HTTP-date, such as a weak W/ tag: it cannot match.
+ return false;
+ }
+ // RFC 9110 section 13.1.5 matches a date exactly, not as the "not modified since" of
+ // If-Unmodified-Since, so a later date gets the whole object. Last-Modified is sent to the
+ // second, so it is compared to the second.
+ return lastModified != null && lastModified.truncatedTo(ChronoUnit.SECONDS).equals(date);
+ }
+
+ /** This range's bytes, read from {@code whole}, a stream over the whole object, as they are needed. */
+ public InputStream slice(InputStream whole) {
+ return new ByteRangeInputStream(whole, first, length());
+ }
+
+ /** Resolves a {@code Range} header against an object of {@code size} bytes. */
+ public static Resolution resolve(String header, long size) {
+ if (!inBytes(header)) {
+ return new Resolution(Outcome.WHOLE, null);
+ }
+ Matcher matcher = SPEC.matcher(header.substring(UNIT.length()).trim());
+ if (!matcher.matches() || (matcher.group(1).isEmpty() && matcher.group(2).isEmpty())) {
+ return new Resolution(Outcome.INVALID, null);
+ }
+ long first;
+ long last;
+ if (matcher.group(1).isEmpty()) {
+ long suffix = position(matcher.group(2));
+ if (suffix == 0) {
+ return new Resolution(Outcome.NOT_SATISFIABLE, null);
+ }
+ if (size == 0) {
+ return new Resolution(Outcome.WHOLE, null);
+ }
+ first = Math.max(0, size - suffix);
+ last = size - 1;
+ } else {
+ first = position(matcher.group(1));
+ last = matcher.group(2).isEmpty() ? Long.MAX_VALUE : position(matcher.group(2));
+ if (last < first) {
+ return new Resolution(Outcome.INVALID, null);
+ }
+ last = Math.min(last, size - 1);
+ }
+ if (first >= size) {
+ return new Resolution(Outcome.NOT_SATISFIABLE, null);
+ }
+ return new Resolution(Outcome.PART, new ByteRange(first, last));
+ }
+
+ private static long position(String digits) {
+ try {
+ return Long.parseLong(digits);
+ } catch (NumberFormatException ignored) {
+ // The pattern admits digits alone, so only a position too large for a long gets here, and
+ // that is past the end of any object.
+ return Long.MAX_VALUE;
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/CopyRangeInputStream.java b/src/main/java/io/github/hectorvent/floci/services/s3/ByteRangeInputStream.java
similarity index 65%
rename from src/main/java/io/github/hectorvent/floci/services/s3/CopyRangeInputStream.java
rename to src/main/java/io/github/hectorvent/floci/services/s3/ByteRangeInputStream.java
index 1567e7ead1..cad0bc7f65 100644
--- a/src/main/java/io/github/hectorvent/floci/services/s3/CopyRangeInputStream.java
+++ b/src/main/java/io/github/hectorvent/floci/services/s3/ByteRangeInputStream.java
@@ -1,28 +1,26 @@
package io.github.hectorvent.floci.services.s3;
-import io.github.hectorvent.floci.services.s3.S3Service.CopySourceRange;
-
import java.io.EOFException;
import java.io.IOException;
import java.io.InputStream;
/**
- * The bytes of one copy-source range, read from a stream over the whole source as they are needed.
- * The first read skips to the range's first byte, so nothing of the source is touched until the
- * part is about to be written, and the stream ends after the range's last byte. A source that ends
+ * The bytes of one range of an object, read from a stream over the whole object as they are needed.
+ * The first read skips to the range's first byte, so nothing of the object is touched until the
+ * range is about to be written, and the stream ends after the range's last byte. An object that ends
* before the range does fails the read.
*/
-final class CopyRangeInputStream extends InputStream {
+final class ByteRangeInputStream extends InputStream {
private final InputStream source;
private final long first;
private long remaining;
private boolean positioned;
- CopyRangeInputStream(InputStream source, CopySourceRange range) {
+ ByteRangeInputStream(InputStream source, long first, long length) {
this.source = source;
- this.first = range.first();
- this.remaining = range.length();
+ this.first = first;
+ this.remaining = length;
}
@Override
@@ -46,7 +44,7 @@ final class CopyRangeInputStream extends InputStream {
}
int read = source.read(buffer, offset, (int) Math.min(length, remaining));
if (read < 0) {
- throw new EOFException("Copy source ended " + remaining + " bytes before the end of the requested range");
+ throw new EOFException("Object ended " + remaining + " bytes before the end of the requested range");
}
remaining -= read;
return read;
diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilter.java b/src/main/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilter.java
index bab4a0276f..0d2f06498a 100644
--- a/src/main/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilter.java
+++ b/src/main/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilter.java
@@ -360,7 +360,9 @@ public class PreSignedUrlFilter implements ContainerRequestFilter {
return requestHeaderNames.stream()
.map(name -> name.toLowerCase(Locale.ROOT))
- .filter(name -> CHECKSUM_HEADERS_REQUIRING_SIGNATURE.contains(name) || name.startsWith("x-amz-meta-"))
+ .filter(name -> CHECKSUM_HEADERS_REQUIRING_SIGNATURE.contains(name) || name.startsWith("x-amz-meta-")
+ // A signed Range makes S3 require the If-Range that qualifies it to be signed too.
+ || ("if-range".equals(name) && normalizedSignedHeaders.contains("range")))
.filter(name -> !normalizedSignedHeaders.contains(name))
.distinct()
.sorted()
diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java b/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java
index 13fd04c37f..6abf8373b7 100644
--- a/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java
+++ b/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java
@@ -56,10 +56,8 @@ import javax.xml.stream.XMLStreamReader;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.Closeable;
-import java.io.EOFException;
import java.io.IOException;
import java.io.InputStream;
-import java.io.OutputStream;
import java.io.UncheckedIOException;
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
@@ -1050,8 +1048,10 @@ public class S3Controller {
}
boolean includeChecksum = "ENABLED".equalsIgnoreCase(resolveHeaderOrQueryParam(checksumMode, uriInfo, "x-amz-checksum-mode"));
- if (rangeHeader != null && rangeHeader.startsWith("bytes=")) {
- Response rangeResponse = handleRangeRequest(obj, read.body(), rangeHeader, overrides, includeChecksum);
+ if (ByteRange.inBytes(rangeHeader) && ByteRange.rangeApplies(
+ httpHeaders.getHeaderString("If-Range"), obj.getETag(), obj.getLastModified())) {
+ Response rangeResponse = handleRangeRequest(
+ bucket, key, obj, read.body(), rangeHeader, overrides, includeChecksum);
bodyHandedOff = rangeResponse.getEntity() instanceof StreamingOutput;
return rangeResponse;
}
@@ -1117,57 +1117,33 @@ public class S3Controller {
return resp;
}
- private Response handleRangeRequest(S3Object obj, InputStream body, String rangeHeader,
- ResponseHeaderOverrides overrides,
+ private Response handleRangeRequest(String bucket, String key, S3Object obj, InputStream body,
+ String rangeHeader, ResponseHeaderOverrides overrides,
boolean includeChecksum) {
long totalSize = obj.getSize();
- String rangeSpec = rangeHeader.substring("bytes=".length()).trim();
-
- long start, end;
- try {
- int dash = rangeSpec.indexOf('-');
- if (dash < 0) {
- return invalidRangeResponse(totalSize);
- }
- String before = rangeSpec.substring(0, dash);
- String after = rangeSpec.substring(dash + 1);
- if (before.isEmpty() && after.isEmpty()) {
- return invalidRangeResponse(totalSize);
- }
- if (before.isEmpty()) {
- long suffix = Long.parseLong(after);
- if (suffix <= 0) {
- return invalidRangeResponse(totalSize);
- }
- start = Math.max(0, totalSize - suffix);
- end = totalSize - 1;
- } else {
- start = Long.parseLong(before);
- end = after.isEmpty() ? totalSize - 1 : Math.min(Long.parseLong(after), totalSize - 1);
- }
- } catch (NumberFormatException e) {
- return invalidRangeResponse(totalSize);
+ ByteRange.Resolution resolution = ByteRange.resolve(rangeHeader, totalSize);
+ if (resolution.outcome() == ByteRange.Outcome.WHOLE) {
+ emitCloudTrailEvent("GetObject", bucket, key, 0L, totalSize, null, null);
+ return fullObjectResponse(obj, body, overrides, includeChecksum);
}
-
- if (start < 0 || start >= totalSize || start > end) {
- if (totalSize == 0 && rangeSpec.startsWith("-")) {
- return fullObjectResponse(obj, body, overrides, includeChecksum);
- }
- return invalidRangeResponse(totalSize);
+ if (resolution.outcome() != ByteRange.Outcome.PART) {
+ emitCloudTrailEvent("GetObject", bucket, key, 0L, 0L,
+ "InvalidRange", ByteRange.NOT_SATISFIABLE_MESSAGE);
+ return invalidRangeResponse(rangeHeader, totalSize);
}
- long length = end - start + 1;
+ ByteRange range = resolution.range();
+ emitCloudTrailEvent("GetObject", bucket, key, 0L, range.length(), null, null);
StreamingOutput stream = output -> {
- try (InputStream in = body) {
- in.skipNBytes(start);
- copyExactly(in, output, length);
+ try (InputStream in = range.slice(body)) {
+ in.transferTo(output);
}
};
var resp = Response.status(206)
.entity(stream)
.header("Content-Type", overrides.contentType() != null ? overrides.contentType() : obj.getContentType())
- .header("Content-Length", length)
- .header("Content-Range", "bytes " + start + "-" + end + "/" + totalSize)
+ .header("Content-Length", range.length())
+ .header("Content-Range", range.contentRange(totalSize))
.header("ETag", obj.getETag())
.header("Last-Modified", RFC_822.format(obj.getLastModified()))
.header("Accept-Ranges", "bytes");
@@ -1183,19 +1159,6 @@ public class S3Controller {
return fullObjectResponse(read.object(), read.body(), overrides, false);
}
- private static void copyExactly(InputStream in, OutputStream out, long length) throws IOException {
- byte[] buffer = new byte[64 * 1024];
- long remaining = length;
- while (remaining > 0) {
- int n = in.read(buffer, 0, (int) Math.min(buffer.length, remaining));
- if (n < 0) {
- throw new EOFException("S3 object ended " + remaining + " bytes before the requested range");
- }
- out.write(buffer, 0, n);
- remaining -= n;
- }
- }
-
private static void closeQuietly(Closeable closeable) {
try {
closeable.close();
@@ -1204,19 +1167,21 @@ public class S3Controller {
}
}
- private Response invalidRangeResponse(long totalSize) {
+ private Response invalidRangeResponse(String rangeHeader, long totalSize) {
String xml = new XmlBuilder()
.raw("<?xml version=\"1.0\" encoding=\"UTF-8\"?>")
.start("Error")
.elem("Code", "InvalidRange")
- .elem("Message", "The requested range is not satisfiable.")
+ .elem("Message", ByteRange.NOT_SATISFIABLE_MESSAGE)
+ .elem("RangeRequested", rangeHeader)
+ .elem("ActualObjectSize", totalSize)
.elem("RequestId", java.util.UUID.randomUUID().toString())
.end("Error")
.build();
return Response.status(416)
.entity(xml)
.type(MediaType.APPLICATION_XML)
- .header("Content-Range", "bytes */" + totalSize)
+ .header("Content-Range", ByteRange.unsatisfiedContentRange(totalSize))
.build();
}
@@ -1281,16 +1246,35 @@ public class S3Controller {
return xmlErrorResponse(new AwsException("InvalidRequest",
"Request specific response headers cannot be used for anonymous GET requests.", 400));
}
+ // "If the Range is satisfiable, only the ContentLength is affected in the response. If the
+ // Range is not satisfiable, S3 returns a 416." A Range that is not one valid range is ignored.
+ long contentLength = obj.getSize();
+ boolean ranged = false;
+ String rangeHeader = httpHeaders.getHeaderString("Range");
+ if (ByteRange.inBytes(rangeHeader) && ByteRange.rangeApplies(
+ httpHeaders.getHeaderString("If-Range"), obj.getETag(), obj.getLastModified())) {
+ ByteRange.Resolution resolution = ByteRange.resolve(rangeHeader, obj.getSize());
+ if (resolution.outcome() == ByteRange.Outcome.PART) {
+ contentLength = resolution.range().length();
+ ranged = true;
+ } else if (resolution.outcome() == ByteRange.Outcome.NOT_SATISFIABLE) {
+ emitCloudTrailEvent("HeadObject", bucket, key, 0L, 0L,
+ "InvalidRange", ByteRange.NOT_SATISFIABLE_MESSAGE);
+ return headOnlyResponse(invalidRangeResponse(rangeHeader, obj.getSize()));
+ }
+ }
var resp = Response.ok()
.header("Content-Type", overrides.contentType() != null ? overrides.contentType() : obj.getContentType())
- .header("Content-Length", obj.getSize())
+ .header("Content-Length", contentLength)
.header("ETag", obj.getETag())
.header("Last-Modified", RFC_822.format(obj.getLastModified()))
.header("Accept-Ranges", "bytes");
if (obj.getVersionId() != null) {
resp.header("x-amz-version-id", obj.getVersionId());
}
- boolean includeChecksum = "ENABLED".equalsIgnoreCase(resolveHeaderOrQueryParam(checksumMode, uriInfo, "x-amz-checksum-mode"));
+ // As on a ranged GetObject, the stored checksum covers the whole object, not the range.
+ boolean includeChecksum = !ranged
+ && "ENABLED".equalsIgnoreCase(resolveHeaderOrQueryParam(checksumMode, uriInfo, "x-amz-checksum-mode"));
appendObjectHeaders(resp, obj, overrides, includeChecksum);
emitCloudTrailEvent("HeadObject", bucket, key, 0L, obj.getSize(), null, null);
return resp.build();
diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java
index f64c627cd6..656f7b9c4f 100644
--- a/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java
+++ b/src/main/java/io/github/hectorvent/floci/services/s3/S3Service.java
@@ -4060,7 +4060,7 @@ public class S3Service implements Resettable, ResourceProvider {
"Your proposed upload exceeds the maximum allowed object size.", 400);
}
return storePart(destBucket, destKey, uploadId, partNumber,
- new CopyRangeInputStream(read.body(), range), UploadChecksums.NONE,
+ new ByteRangeInputStream(read.body(), range.first(), range.length()), UploadChecksums.NONE,
sseCustomerHeaders.algorithm(), sseCustomerHeaders.key(), sseCustomerHeaders.keyMd5())
.getETag();
} catch (IOException e) {
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontCustomOriginServingTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontCustomOriginServingTest.java
index 18197a5cfa..db2e634790 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontCustomOriginServingTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontCustomOriginServingTest.java
@@ -34,6 +34,7 @@ import java.security.KeyPairGenerator;
import java.security.Signature;
import java.time.Duration;
import java.time.Instant;
+import java.util.Arrays;
import java.util.Base64;
import java.util.LinkedHashMap;
import java.util.List;
@@ -424,6 +425,27 @@ class CloudFrontCustomOriginServingTest {
});
}
+ @Test
+ void forwardsARangeGetWhateverTheCachePolicyAndPassesThePartialResponse() throws Exception {
+ byte[] video = randomBytes(1000);
+ AtomicReference<ReceivedRequest> received = new AtomicReference<>();
+ startRangeOrigin(video, received);
+ // CachingDisabled forwards no viewer headers, but CloudFront still forwards a range GET.
+ Distribution created = cloudFrontService.createDistribution(
+ customOriginDistribution(policyBehavior(CACHING_DISABLED, null)), Map.of());
+
+ Response response = given().header("Host", created.getDomainName())
+ .header("Range", "bytes=100-199").header("If-Range", "\"v1\"")
+ .when().get("/video.bin");
+
+ assertEquals(206, response.statusCode());
+ assertEquals("bytes 100-199/1000", response.header("Content-Range"));
+ assertEquals("100", response.header("Content-Length"));
+ assertArrayEquals(Arrays.copyOfRange(video, 100, 200), response.asByteArray());
+ assertEquals("bytes=100-199", received.get().header("Range"));
+ assertEquals("\"v1\"", received.get().header("If-Range"));
+ }
+
@Test
void forwardsTheViewerAuthorizationOnWriteMethodsOnly() throws Exception {
AtomicReference<ReceivedRequest> received = new AtomicReference<>();
@@ -826,6 +848,30 @@ class CloudFrontCustomOriginServingTest {
originServer.start();
}
+ /** An origin that answers a GET for {@code body} with the one {@code bytes=first-last} range asked for. */
+ private void startRangeOrigin(byte[] body, AtomicReference<ReceivedRequest> received) throws IOException {
+ originServer = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
+ originServer.createContext("/", exchange -> {
+ received.set(new ReceivedRequest(exchange.getRequestMethod(), exchange.getRequestURI().getRawPath(),
+ exchange.getRequestURI().getRawQuery(), exchange.getRequestHeaders(), new byte[0]));
+ exchange.getResponseHeaders().add("Content-Type", "application/octet-stream");
+ String range = exchange.getRequestHeaders().getFirst("Range");
+ if (range == null) {
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ } else {
+ String[] bounds = range.substring("bytes=".length()).split("-");
+ int first = Integer.parseInt(bounds[0]);
+ int last = Integer.parseInt(bounds[1]);
+ exchange.getResponseHeaders().add("Content-Range", "bytes " + first + "-" + last + "/" + body.length);
+ exchange.sendResponseHeaders(206, last - first + 1);
+ exchange.getResponseBody().write(body, first, last - first + 1);
+ }
+ exchange.close();
+ });
+ originServer.start();
+ }
+
/**
* POSTs {@code body} with {@code Transfer-Encoding: chunked} over a raw connection, since an HTTP
* client library buffers a stream it is given and sends its length instead.
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontDistributionServingTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontDistributionServingTest.java
index ffa319f7c1..8bc6bda761 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontDistributionServingTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontDistributionServingTest.java
@@ -52,6 +52,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
class CloudFrontDistributionServingTest {
private static final String REGION = "us-east-1";
+ private static final String ALPHABET = "abcdefghijklmnopqrstuvwxyz";
@Inject
S3Service s3Service;
@@ -283,6 +284,86 @@ class CloudFrontDistributionServingTest {
.header("Content-Length", equalTo(Integer.toString(body.getBytes(StandardCharsets.UTF_8).length)));
}
+ @Test
+ void servesTheByteRangeAViewerAsksForFromAnS3Object() {
+ Distribution dist = alphabetDistribution();
+
+ assertRange(dist, "bytes=2-5", "bytes 2-5/26", "cdef");
+ assertRange(dist, "bytes=20-", "bytes 20-25/26", "uvwxyz");
+ assertRange(dist, "bytes=-3", "bytes 23-25/26", "xyz");
+ assertRange(dist, "bytes=-100", "bytes 0-25/26", ALPHABET);
+ assertRange(dist, "bytes=24-100", "bytes 24-25/26", "yz");
+ assertRange(dist, "Bytes=2-5", "bytes 2-5/26", "cdef");
+ // HeadObject: "If the Range is satisfiable, only the ContentLength is affected in the response."
+ given().header("Host", dist.getDomainName()).header("Range", "bytes=2-5").when().head("/alphabet.txt")
+ .then().statusCode(200)
+ .header("Content-Length", equalTo("4"))
+ .header("Content-Range", nullValue());
+ }
+
+ @Test
+ void servesTheWholeS3ObjectWhenItsIfRangeNoLongerMatches() {
+ Distribution dist = alphabetDistribution();
+ String etag = given().header("Host", dist.getDomainName()).when().head("/alphabet.txt")
+ .then().extract().header("ETag");
+
+ given().header("Host", dist.getDomainName()).header("Range", "bytes=2-5").header("If-Range", etag)
+ .when().get("/alphabet.txt")
+ .then().statusCode(206)
+ .body(equalTo("cdef"));
+ given().header("Host", dist.getDomainName()).header("Range", "bytes=2-5").header("If-Range", "\"stale\"")
+ .when().get("/alphabet.txt")
+ .then().statusCode(200)
+ .header("Content-Range", nullValue())
+ .body(equalTo(ALPHABET));
+ }
+
+ @Test
+ void answersARangePastTheEndOfAnS3ObjectWith416() {
+ Distribution dist = alphabetDistribution();
+
+ for (String range : List.of("bytes=26-", "bytes=100-200", "bytes=-0")) {
+ given().header("Host", dist.getDomainName()).header("Range", range).when().get("/alphabet.txt")
+ .then().statusCode(416)
+ .header("Content-Range", equalTo("bytes */26"))
+ .body(equalTo("The requested range is not satisfiable"));
+ given().header("Host", dist.getDomainName()).header("Range", range).when().head("/alphabet.txt")
+ .then().statusCode(416);
+ }
+ }
+
+ @Test
+ void servesTheWholeS3ObjectForARangeHeaderCloudFrontDoesNotAccept() {
+ Distribution dist = alphabetDistribution();
+
+ for (String range : List.of("bytes=5-2", "bytes=0-1,4-5", "bytes=abc", "bytes=-", "items=0-1")) {
+ given().header("Host", dist.getDomainName()).header("Range", range).when().get("/alphabet.txt")
+ .then().statusCode(200)
+ .header("Content-Length", equalTo("26"))
+ .header("Content-Range", nullValue())
+ .body(equalTo(ALPHABET));
+ }
+ }
+
+ @Test
+ void aViewerRangeIsNotAppliedToTheErrorPageServedInstead() {
+ String suffix = suffix();
+ String bucket = "cf-range-error-page-" + suffix;
+ createBucket(bucket);
+ putObject(bucket, "index.html", "INDEX-" + suffix, "text/html");
+ DistributionConfig cfg = new DistributionConfig();
+ cfg.setEnabled(true);
+ cfg.setOrigins(List.of(s3Origin("only-origin", bucket)));
+ cfg.setDefaultCacheBehavior(defaultBehavior("only-origin"));
+ cfg.setCustomErrorResponses(List.of(customError(404, 200, "/index.html")));
+ Distribution dist = cloudFrontService.createDistribution(distribution(cfg), Map.of());
+
+ given().header("Host", dist.getDomainName()).header("Range", "bytes=0-1").when().get("/missing/route")
+ .then().statusCode(200)
+ .header("Content-Range", nullValue())
+ .body(equalTo("INDEX-" + suffix));
+ }
+
@Test
void returnsErrorPageOriginStatusWhenCustomErrorPageIsMissing() {
// AWS: if the configured custom error page is itself unavailable, CloudFront returns the
@@ -1605,6 +1686,27 @@ class CloudFrontDistributionServingTest {
return new SignerResources(publicKey.getId(), keyGroup.getId());
}
+ /** A distribution in front of a bucket holding {@code alphabet.txt}, the 26 letters. */
+ private Distribution alphabetDistribution() {
+ String bucket = "cf-range-" + suffix();
+ createBucket(bucket);
+ putObject(bucket, "alphabet.txt", ALPHABET, "text/plain");
+ DistributionConfig cfg = new DistributionConfig();
+ cfg.setEnabled(true);
+ cfg.setOrigins(List.of(s3Origin("only-origin", bucket)));
+ cfg.setDefaultCacheBehavior(defaultBehavior("only-origin"));
+ return cloudFrontService.createDistribution(distribution(cfg), Map.of());
+ }
+
+ private static void assertRange(Distribution dist, String range, String contentRange, String body) {
+ given().header("Host", dist.getDomainName()).header("Range", range).when().get("/alphabet.txt")
+ .then().statusCode(206)
+ .header("Content-Range", equalTo(contentRange))
+ .header("Content-Length", equalTo(Integer.toString(body.length())))
+ .header("Accept-Ranges", equalTo("bytes"))
+ .body(equalTo(body));
+ }
+
private void createBucket(String bucket) {
s3Service.createBucket(bucket, REGION);
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilderTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilderTest.java
index c67950f304..766786c0b6 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilderTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontOriginRequestBuilderTest.java
@@ -267,6 +267,22 @@ class CloudFrontOriginRequestBuilderTest {
viewer("GET", null, "Content-Type", "application/json"), nothing), "Content-Type"));
}
+ @Test
+ void aRangeReadReachesTheOriginWhateverThePolicy() {
+ Forwarding nothing = Forwarding.policies(null, null, false);
+ for (String method : List.of("GET", "HEAD")) {
+ OriginRequest read = build(viewer(method, null, "Range", "bytes=0-99", "If-Range", "\"v1\""), nothing);
+ assertEquals("bytes=0-99", value(read, "Range"), method);
+ assertEquals("\"v1\"", value(read, "If-Range"), method);
+ }
+
+ OriginRequest post = build(viewer("POST", null, "Range", "bytes=0-99"), nothing);
+ assertNull(value(post, "Range"), "a write follows the forwarding settings");
+
+ assertNull(value(build(viewer("GET", null, "If-Range", "\"v1\""), nothing), "If-Range"),
+ "an If-Range without a Range qualifies nothing");
+ }
+
@Test
void hopByHopAndCloudFrontOwnedHeadersAreNeverForwarded() {
OriginRequest request = build(viewer("POST", null,
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontS3DiskOriginServingTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontS3DiskOriginServingTest.java
index f6b539d186..b1fb090d7e 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontS3DiskOriginServingTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudfront/CloudFrontS3DiskOriginServingTest.java
@@ -13,6 +13,7 @@ import jakarta.inject.Inject;
import org.junit.jupiter.api.Test;
import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -62,6 +63,12 @@ class CloudFrontS3DiskOriginServingTest {
given().header("Host", dist.getDomainName()).when().head("/video.bin")
.then().statusCode(200)
.header("Content-Length", equalTo(Integer.toString(body.length)));
+ // A range from the middle of the object file.
+ Response part = given().header("Host", dist.getDomainName())
+ .header("Range", "bytes=1048576-1048675").when().get("/video.bin");
+ assertEquals(206, part.statusCode());
+ assertEquals("bytes 1048576-1048675/" + body.length, part.header("Content-Range"));
+ assertArrayEquals(Arrays.copyOfRange(body, 1048576, 1048676), part.asByteArray());
// The single-page-app fallback serves its error page through the same stream.
given().header("Host", dist.getDomainName()).when().get("/missing/route")
.then().statusCode(200)
diff --git a/src/test/java/io/github/hectorvent/floci/services/cloudtrail/CloudTrailIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/cloudtrail/CloudTrailIntegrationTest.java
index ed230d130f..06e58b6eb0 100644
--- a/src/test/java/io/github/hectorvent/floci/services/cloudtrail/CloudTrailIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/cloudtrail/CloudTrailIntegrationTest.java
@@ -101,6 +101,16 @@ class CloudTrailIntegrationTest {
headObject(sourceBucket, "documents/hello.txt", 200);
listObjects(sourceBucket);
getObject(sourceBucket, "documents/missing.txt", 404);
+ // Ranged reads are recorded too, a range that starts past the end of the object as InvalidRange.
+ given().header("Range", "bytes=0-4")
+ .when().get("/" + sourceBucket + "/documents/hello.txt")
+ .then().statusCode(206);
+ given().header("Range", "bytes=100-")
+ .when().get("/" + sourceBucket + "/documents/hello.txt")
+ .then().statusCode(416);
+ given().header("Range", "bytes=100-")
+ .when().head("/" + sourceBucket + "/documents/hello.txt")
+ .then().statusCode(416);
deleteObject(sourceBucket, "documents/hello.txt");
// 6. Force the writer to flush — bypasses the scheduled cadence.
@@ -145,6 +155,7 @@ class CloudTrailIntegrationTest {
+ ":\n" + new String(json));
boolean sawPut = false, sawGet = false, sawDelete = false, sawNoSuchKey = false;
+ boolean sawRangedGet = false, sawGetInvalidRange = false, sawHeadInvalidRange = false;
for (JsonNode rec : records) {
assertEquals("1.11", rec.path("eventVersion").asText(),
"Bad eventVersion: " + rec);
@@ -164,11 +175,22 @@ class CloudTrailIntegrationTest {
if ("GetObject".equals(name) && rec.path("errorCode").isMissingNode()) sawGet = true;
if ("DeleteObject".equals(name)) sawDelete = true;
if ("NoSuchKey".equals(rec.path("errorCode").asText(null))) sawNoSuchKey = true;
+ if ("GetObject".equals(name) && rec.path("errorCode").isMissingNode()
+ && rec.path("additionalEventData").path("bytesTransferredOut").asLong() == 5) {
+ sawRangedGet = true;
+ }
+ if ("InvalidRange".equals(rec.path("errorCode").asText(null))) {
+ sawGetInvalidRange |= "GetObject".equals(name);
+ sawHeadInvalidRange |= "HeadObject".equals(name);
+ }
}
assertTrue(sawPut, "Expected a PutObject record");
assertTrue(sawGet, "Expected a successful GetObject record");
assertTrue(sawDelete, "Expected a DeleteObject record");
assertTrue(sawNoSuchKey, "Expected a GetObject NoSuchKey record");
+ assertTrue(sawRangedGet, "Expected a GetObject record of the 5 bytes of a ranged read");
+ assertTrue(sawGetInvalidRange, "Expected a GetObject InvalidRange record");
+ assertTrue(sawHeadInvalidRange, "Expected a HeadObject InvalidRange record");
}
@Test
diff --git a/src/test/java/io/github/hectorvent/floci/services/s3/ByteRangeTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/ByteRangeTest.java
new file mode 100644
index 0000000000..8cfff2acdd
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/s3/ByteRangeTest.java
@@ -0,0 +1,134 @@
+package io.github.hectorvent.floci.services.s3;
+
+import io.github.hectorvent.floci.services.s3.ByteRange.Outcome;
+import io.github.hectorvent.floci.services.s3.ByteRange.Resolution;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class ByteRangeTest {
+
+ @Test
+ void aSingleRangeResolvesToItsBytes() {
+ assertPart("bytes=0-9", 100, 0, 9);
+ assertPart("bytes=90-", 100, 90, 99);
+ assertPart("bytes=-10", 100, 90, 99);
+ assertPart("bytes= 5-6 ", 100, 5, 6);
+ }
+
+ @Test
+ void theRangeUnitMatchesInAnyCase() {
+ assertPart("Bytes=0-9", 100, 0, 9);
+ assertPart("BYTES=90-", 100, 90, 99);
+ assertOutcome(Outcome.WHOLE, "byte=0-9", 100);
+ }
+
+ @Test
+ void aByteRangeRunsForwardFromAFirstByteThatExists() {
+ assertThrows(IllegalArgumentException.class, () -> new ByteRange(5, 2));
+ assertThrows(IllegalArgumentException.class, () -> new ByteRange(-1, 2));
+ assertEquals(1, new ByteRange(7, 7).length());
+ assertEquals("bytes */100", ByteRange.unsatisfiedContentRange(100));
+ }
+
+ @Test
+ void aRangeRunningPastTheEndStopsAtTheLastByte() {
+ assertPart("bytes=95-200", 100, 95, 99);
+ assertPart("bytes=-500", 100, 0, 99);
+ assertPart("bytes=0-99999999999999999999", 100, 0, 99);
+ assertPart("bytes=-99999999999999999999", 100, 0, 99);
+ }
+
+ @Test
+ void offsetsPastTwoGibibytesResolve() {
+ long size = 3L * 1024 * 1024 * 1024;
+ ByteRange range = assertPart("bytes=3000000000-3000000009", size, 3_000_000_000L, 3_000_000_009L);
+
+ assertEquals(10, range.length());
+ assertEquals("bytes 3000000000-3000000009/" + size, range.contentRange(size));
+ }
+
+ @Test
+ void aRangeStartingPastTheEndIsNotSatisfiable() {
+ for (String header : List.of("bytes=100-", "bytes=100-200", "bytes=-0", "bytes=99999999999999999999-")) {
+ assertOutcome(Outcome.NOT_SATISFIABLE, header, 100);
+ }
+ assertOutcome(Outcome.NOT_SATISFIABLE, "bytes=0-", 0);
+ assertOutcome(Outcome.NOT_SATISFIABLE, "bytes=0-0", 0);
+ }
+
+ @Test
+ void anythingButOneValidRangeIsInvalid() {
+ for (String header : List.of("bytes=5-2", "bytes=0-1,4-5", "bytes=abc", "bytes=-", "bytes=1",
+ "bytes=--5", "bytes=+1-2", "bytes=")) {
+ assertOutcome(Outcome.INVALID, header, 100);
+ }
+ }
+
+ @Test
+ void noHeaderAnotherUnitOrASuffixOfAnEmptyObjectIsTheWholeObject() {
+ assertOutcome(Outcome.WHOLE, null, 100);
+ assertOutcome(Outcome.WHOLE, "items=0-1", 100);
+ assertOutcome(Outcome.WHOLE, "bytes=-5", 0);
+ }
+
+ @Test
+ void anIfRangeThatStillDescribesTheObjectLetsTheRangeApply() {
+ Instant modified = Instant.parse("2015-10-21T07:28:00.250Z");
+
+ assertTrue(ByteRange.rangeApplies(null, "\"abc\"", modified));
+ assertTrue(ByteRange.rangeApplies("\"abc\"", "\"abc\"", modified));
+ assertTrue(ByteRange.rangeApplies("\"abc-2\"", "\"abc-2\"", modified), "a multipart ETag, whole");
+ assertTrue(ByteRange.rangeApplies("Wed, 21 Oct 2015 07:28:00 GMT", "\"abc\"", modified),
+ "Last-Modified, to the second");
+ }
+
+ @Test
+ void anIfRangeThatNoLongerDescribesTheObjectServesItWhole() {
+ Instant modified = Instant.parse("2015-10-21T07:28:00.250Z");
+
+ assertFalse(ByteRange.rangeApplies("\"old\"", "\"abc\"", modified));
+ assertFalse(ByteRange.rangeApplies("\"2\"", "\"abc-2\"", modified), "a multipart suffix alone");
+ assertFalse(ByteRange.rangeApplies("Tue, 20 Oct 2015 07:28:00 GMT", "\"abc\"", modified));
+ assertFalse(ByteRange.rangeApplies("Thu, 22 Oct 2015 07:28:00 GMT", "\"abc\"", modified),
+ "a date later than Last-Modified");
+ for (String malformed : List.of("W/\"abc\"", "abc", "\"", "2015-10-21T07:28:00Z", "")) {
+ assertFalse(ByteRange.rangeApplies(malformed, "\"abc\"", modified), malformed);
+ }
+ }
+
+ @Test
+ void aSliceReadsTheRangeFromAStreamOverTheWholeObject() throws IOException {
+ byte[] object = "abcdefghijklmnopqrstuvwxyz".getBytes(StandardCharsets.US_ASCII);
+ ByteRange range = assertPart("bytes=2-5", object.length, 2, 5);
+
+ try (InputStream slice = range.slice(new ByteArrayInputStream(object))) {
+ assertArrayEquals("cdef".getBytes(StandardCharsets.US_ASCII), slice.readAllBytes());
+ }
+ }
+
+ private static ByteRange assertPart(String header, long size, long first, long last) {
+ Resolution resolution = ByteRange.resolve(header, size);
+ assertEquals(Outcome.PART, resolution.outcome(), header);
+ assertEquals(new ByteRange(first, last), resolution.range(), header);
+ return resolution.range();
+ }
+
+ private static void assertOutcome(Outcome outcome, String header, long size) {
+ Resolution resolution = ByteRange.resolve(header, size);
+ assertEquals(outcome, resolution.outcome(), header + " of " + size + " bytes");
+ assertNull(resolution.range(), header);
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilterTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilterTest.java
index 784ebd46e9..3fcc4df5bd 100644
--- a/src/test/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilterTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/s3/PreSignedUrlFilterTest.java
@@ -201,6 +201,23 @@ class PreSignedUrlFilterTest {
"host;x-amz-meta-signed"));
}
+ @Test
+ void requiresIfRangeSignedWhenRangeIsSigned() {
+ assertEquals(
+ List.of("if-range"),
+ PreSignedUrlFilter.unsignedHeadersRequiringSignature(
+ Set.of("host", "Range", "If-Range"), "host;range"));
+ assertEquals(
+ List.of(),
+ PreSignedUrlFilter.unsignedHeadersRequiringSignature(
+ Set.of("host", "range", "if-range"), "host;range;if-range"));
+ assertEquals(
+ List.of(),
+ PreSignedUrlFilter.unsignedHeadersRequiringSignature(
+ Set.of("host", "range", "if-range"), "host"),
+ "an unsigned Range leaves If-Range unsigned too");
+ }
+
@Test
void ignoresHeadersOutsideChecksumFamily() {
assertEquals(
diff --git a/src/test/java/io/github/hectorvent/floci/services/s3/S3CopySourceRangeTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/S3CopySourceRangeTest.java
index 73694132b1..89a5f24475 100644
--- a/src/test/java/io/github/hectorvent/floci/services/s3/S3CopySourceRangeTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/s3/S3CopySourceRangeTest.java
@@ -95,7 +95,8 @@ class S3CopySourceRangeTest {
void aRangeSkipsToAnOffsetPastTwoGibibytes() throws IOException {
CopySourceRange range = new CopySourceRange(2_500_000_000L, 2_500_000_015L);
- byte[] data = new CopyRangeInputStream(patternStream(3L * 1024 * 1024 * 1024), range).readAllBytes();
+ byte[] data = new ByteRangeInputStream(patternStream(3L * 1024 * 1024 * 1024), range.first(), range.length())
+ .readAllBytes();
assertEquals(16, data.length);
for (int i = 0; i < data.length; i++) {
@@ -119,8 +120,7 @@ class S3CopySourceRangeTest {
byte[] data;
try (FileChannel channel = FileChannel.open(file, StandardOpenOption.READ)) {
- data = new CopyRangeInputStream(Channels.newInputStream(channel),
- new CopySourceRange(offset, offset + written.length - 1)).readAllBytes();
+ data = new ByteRangeInputStream(Channels.newInputStream(channel), offset, written.length).readAllBytes();
}
assertArrayEquals(written, data);
@@ -129,9 +129,9 @@ class S3CopySourceRangeTest {
@Test
void aRangeFailsWhenTheSourceEndsBeforeIt() {
assertThrows(IOException.class,
- () -> new CopyRangeInputStream(patternStream(10), new CopySourceRange(5, 14)).readAllBytes());
+ () -> new ByteRangeInputStream(patternStream(10), 5, 10).readAllBytes());
assertThrows(IOException.class,
- () -> new CopyRangeInputStream(patternStream(10), new CopySourceRange(12, 14)).readAllBytes(),
+ () -> new ByteRangeInputStream(patternStream(10), 12, 3).readAllBytes(),
"a source that ends before the range starts");
}
@@ -139,7 +139,7 @@ class S3CopySourceRangeTest {
void aRangeEndsAfterItsLastByteWhateverTheReadSize() throws IOException {
CopySourceRange range = new CopySourceRange(3, 1002);
for (int readSize : List.of(1, 7, 1000, 4096)) {
- try (InputStream in = new CopyRangeInputStream(patternStream(5000), range)) {
+ try (InputStream in = new ByteRangeInputStream(patternStream(5000), range.first(), range.length())) {
ByteArrayOutputStream copied = new ByteArrayOutputStream();
byte[] buffer = new byte[readSize];
for (int read = in.read(buffer); read >= 0; read = in.read(buffer)) {
diff --git a/src/test/java/io/github/hectorvent/floci/services/s3/S3IntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/s3/S3IntegrationTest.java
index e2ab433a05..7b5b3ecea2 100644
--- a/src/test/java/io/github/hectorvent/floci/services/s3/S3IntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/s3/S3IntegrationTest.java
@@ -1031,6 +1031,108 @@ class S3IntegrationTest {
given().delete("/checksum-mode-head-bucket");
}
+ @Test
+ @Order(123)
+ void getObjectWithARangeUnitInAnyCase() {
+ putGreeting("range-unit-bucket");
+ try {
+ // Range unit names are case-insensitive (RFC 9110, section 14.1).
+ given()
+ .header("Range", "Bytes=0-4")
+ .when()
+ .get("/range-unit-bucket/greeting.txt")
+ .then()
+ .statusCode(206)
+ .header("Content-Range", equalTo("bytes 0-4/20"))
+ .body(equalTo("Hello"));
+ } finally {
+ deleteGreeting("range-unit-bucket");
+ }
+ }
+
+ @Test
+ @Order(124)
+ void getObjectWithIfRangeServesTheRangeOnlyWhileItsValidatorMatches() {
+ putGreeting("if-range-bucket");
+ try {
+ String etag = given().when().head("/if-range-bucket/greeting.txt").then().extract().header("ETag");
+ String lastModified = given().when().head("/if-range-bucket/greeting.txt").then().extract()
+ .header("Last-Modified");
+
+ for (String current : List.of(etag, lastModified)) {
+ given()
+ .header("Range", "bytes=0-4")
+ .header("If-Range", current)
+ .when()
+ .get("/if-range-bucket/greeting.txt")
+ .then()
+ .statusCode(206)
+ .header("Content-Range", equalTo("bytes 0-4/20"))
+ .body(equalTo("Hello"));
+ }
+ given()
+ .header("Range", "bytes=0-4")
+ .header("If-Range", "\"stale\"")
+ .when()
+ .get("/if-range-bucket/greeting.txt")
+ .then()
+ .statusCode(200)
+ .header("Content-Range", nullValue())
+ .header("Content-Length", equalTo("20"));
+ } finally {
+ deleteGreeting("if-range-bucket");
+ }
+ }
+
+ @Test
+ @Order(125)
+ void headObjectWithRangeReportsTheRangeLengthOr416() {
+ putGreeting("head-range-bucket");
+ try {
+ // "If the Range is satisfiable, only the ContentLength is affected in the response. If the
+ // Range is not satisfiable, S3 returns a 416".
+ given()
+ .header("Range", "bytes=0-4")
+ .header("x-amz-checksum-mode", "ENABLED")
+ .when()
+ .head("/head-range-bucket/greeting.txt")
+ .then()
+ .statusCode(200)
+ .header("Content-Length", equalTo("5"))
+ .header("Content-Range", nullValue())
+ .header("x-amz-checksum-crc64nvme", nullValue())
+ .header("x-amz-checksum-type", nullValue());
+ given()
+ .header("Range", "bytes=0-4")
+ .header("If-Range", "\"stale\"")
+ .header("x-amz-checksum-mode", "ENABLED")
+ .when()
+ .head("/head-range-bucket/greeting.txt")
+ .then()
+ .statusCode(200)
+ .header("Content-Length", equalTo("20"))
+ .header("x-amz-checksum-crc64nvme", notNullValue());
+ given()
+ .header("Range", "bytes=50-")
+ .when()
+ .head("/head-range-bucket/greeting.txt")
+ .then()
+ .statusCode(416);
+ // A Range that is not one valid range is not one that cannot be satisfied, so it is ignored.
+ for (String invalid : List.of("bytes=0-1,4-5", "bytes=5-2", "bytes=abc")) {
+ given()
+ .header("Range", invalid)
+ .when()
+ .head("/head-range-bucket/greeting.txt")
+ .then()
+ .statusCode(200)
+ .header("Content-Length", equalTo("20"));
+ }
+ } finally {
+ deleteGreeting("head-range-bucket");
+ }
+ }
+
@Test
@Order(27)
void getObjectWithFullRange() {
@@ -1082,7 +1184,10 @@ class S3IntegrationTest {
.then()
.statusCode(416)
.header("Content-Range", equalTo("bytes */20"))
- .body(containsString("InvalidRange"));
+ .body(hasXPath("/Error/Code", equalTo("InvalidRange")))
+ .body(hasXPath("/Error/Message", equalTo("The requested range is not satisfiable")))
+ .body(hasXPath("/Error/RangeRequested", equalTo("bytes=50-100")))
+ .body(hasXPath("/Error/ActualObjectSize", equalTo("20")));
}
@Test
@@ -3199,6 +3304,17 @@ class S3IntegrationTest {
}
}
+ /** Creates {@code bucket} holding {@code greeting.txt}, the 20 bytes "Hello World from S3!". */
+ private static void putGreeting(String bucket) {
+ given().when().put("/" + bucket).then().statusCode(200);
+ given().body("Hello World from S3!").when().put("/" + bucket + "/greeting.txt").then().statusCode(200);
+ }
+
+ private static void deleteGreeting(String bucket) {
+ given().delete("/" + bucket + "/greeting.txt");
+ given().delete("/" + bucket);
+ }
+
private static byte[] alphabetBytes(int size) {
byte[] bytes = new byte[size];
for (int i = 0; i < size; i++) {