Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(codeartifact): bridge GetPackageVersionAsset to Maven, npm, and pypi sidecars (#5019)

Priamo Germosen583b2baf7f9e
diff --git a/docs/services/codeartifact.md b/docs/services/codeartifact.md
index 92e265f8aa..c10b529bba 100644
--- a/docs/services/codeartifact.md
+++ b/docs/services/codeartifact.md
@@ -264,16 +264,19 @@ state.
itself has no JSON index support to pass through. Low practical risk in itself, since PEP 691
requires clients to accept HTML as a fallback and pip/uv both do, but it is a real gap against
AWS's documented surface, not just a quirk of the sidecar.
-- **`GetPackageVersionAsset` and `DescribePackageVersion` can't see Maven-, npm-, or pypi-uploaded
- artifacts.** Those three formats' repository endpoints are deliberately metadata-free passthroughs
- straight to their sidecar container (Reposilite/Verdaccio/pypiserver) and never create the
- generic-format package-version record these two actions depend on
- (`CodeArtifactService.getPackageVersionAsset`/`describePackageVersion` only ever look up
- `packageVersions.getForAccount`, the record only the `generic` format's `PublishPackageVersion`
- flow creates). Real CodeArtifact's `GetPackageVersionAsset` works uniformly across every format.
- The artifact is really there and downloadable through that format's own wire endpoint
- (`mvn dependency:get`, `npm install`, `pip install`, or a direct `curl`/`twine`); only the
- CodeArtifact-native asset-get/describe calls 404 for it.
+- **`DescribePackageVersion` can't see Maven-, npm-, or pypi-uploaded package versions.** Those
+ three formats' repository endpoints are deliberately metadata-free passthroughs straight to
+ their sidecar container (Reposilite/Verdaccio/pypiserver) and never create the generic-format
+ package-version record this action depends on (`CodeArtifactService.describePackageVersion`
+ only ever looks up `packageVersions.getForAccount`, the record only the `generic` format's
+ `PublishPackageVersion` flow creates). `GetPackageVersionAsset` no longer has this gap: it
+ bridges to each format's own sidecar (via a new `RepositorySidecarManager.fetchPackageVersionAsset`
+ method each sidecar client implements) when there's no generic-format record, so it now works
+ uniformly across formats the way real CodeArtifact's does. `DescribePackageVersion` is a harder
+ problem for the same bridge: it describes every asset and the status of a whole version, not one
+ named file, and none of these sidecars expose that shape cheaply (Reposilite has no per-GAV
+ metadata beyond file listings; Verdaccio's own registry API could answer it for npm but Maven and
+ pypi have nothing equivalent), so it remains unbridged and still 404s for these three formats.
See the [CodeArtifact API Reference](https://docs.aws.amazon.com/codeartifact/latest/APIReference/Welcome.html).
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java
index 11c8b490a9..ef41a876a8 100644
--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java
@@ -674,6 +674,17 @@ public class CodeArtifactService implements Resettable {
requireNonBlank(version, "packageVersion");
requireNonBlank(assetName, "asset");
String owner = effectiveOwner(domainOwner);
+
+ // Maven, npm, and pypi are metadata-free passthroughs straight to their own sidecar
+ // (ReposiliteSidecarClient/VerdaccioSidecarManager/PypiserverSidecarManager): publish for
+ // these formats never goes through publishPackageVersion (it only ever accepts "generic"),
+ // so packageVersions can never hold a record for them. Bridge straight to the sidecar
+ // that actually has the asset instead of always 404ing here.
+ if (CONTAINER_BACKED_FORMATS.contains(format)) {
+ return getContainerBackedPackageVersionAsset(region, domain, owner, repository, format, namespace,
+ packageName, version, assetName, packageVersionRevision);
+ }
+
String key = packageVersionKey(region, domain, repository, format, namespace, packageName, version);
CodeArtifactPackageVersion pv = packageVersions.getForAccount(owner, key)
.orElseThrow(() -> notFound("Package version '" + version + "' of package '" + packageName
@@ -695,6 +706,53 @@ public class CodeArtifactService implements Resettable {
return new PackageVersionAssetResult(asset, pv.getRevision());
}
+ /**
+ * There's no stored revision for a sidecar-backed format to compare against, so one is
+ * synthesized from the version coordinate itself (this repository's own sidecar container id,
+ * plus namespace/package/version), not the fetched asset's own bytes: a real revision is
+ * shared by every asset of one package version (a Maven version's JAR and POM alike), so
+ * hashing per-asset content would give a client that described the version once, then tried
+ * to fetch a second asset at that same revision, a spurious not-found on the second call even
+ * though both assets genuinely exist. Keying on the sidecar container id, not just the bare
+ * namespace/package/version strings, keeps two different repositories (even two with the
+ * identical coordinates) from ever synthesizing the same revision.
+ *
+ * <p>Doesn't change when a new asset is published to the version afterward, unlike a real
+ * stored revision: tracking that would mean enumerating every asset a sidecar holds for a
+ * version on every call, the same capability {@code DescribePackageVersion} would need and
+ * doesn't have either for these formats (see the Known limitations doc). Accepted for the
+ * same reason: nothing before this change ever let a client obtain a revision for these
+ * formats in the first place (not even a stale one), so this isn't a regression against any
+ * previously working staleness check.
+ */
+ private PackageVersionAssetResult getContainerBackedPackageVersionAsset(String region, String domain,
+ String owner, String repository, String format, String namespace, String packageName, String version,
+ String assetName, String packageVersionRevision) {
+ if ("maven".equals(format) && (namespace == null || namespace.isBlank())) {
+ throw validation("namespace is required when requesting an asset from a maven package version.");
+ }
+ String repoId = ensureFormatContainerId(format, region, domain, owner, repository);
+ String revisionKey = repoId + "/" + (namespace == null ? "" : namespace) + "/" + packageName + "/" + version;
+ String revision = sha256Hex(revisionKey.getBytes(StandardCharsets.UTF_8));
+ if (packageVersionRevision != null && !packageVersionRevision.equals(revision)) {
+ throw notFound("Package version '" + version + "' was not found at revision '"
+ + packageVersionRevision + "'.", version, "package-version");
+ }
+ RepositorySidecarManager manager = sidecarRegistry.forFormat(format)
+ .orElseThrow(() -> notFound("Package version '" + version + "' of package '" + packageName
+ + "' was not found.", version, "package-version"));
+ byte[] content = manager.fetchPackageVersionAsset(repoId, domain, repository, namespace, packageName,
+ version, assetName)
+ .orElseThrow(() -> notFound("Asset '" + assetName + "' was not found on package version '"
+ + version + "'.", assetName, "asset"));
+ PackageAsset asset = new PackageAsset();
+ asset.setName(assetName);
+ asset.setSize(content.length);
+ asset.setContent(content);
+ asset.setHashes(computeHashes(content));
+ return new PackageVersionAssetResult(asset, revision);
+ }
+
// ------------------------------------------------------------ authorization
public AuthorizationToken getAuthorizationToken(String region, String domain, String domainOwner,
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManager.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManager.java
index deadd17d08..202405e200 100644
--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManager.java
@@ -13,7 +13,17 @@ import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import org.jboss.logging.Logger;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.net.http.HttpResponse.BodyHandlers;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
import java.util.List;
+import java.util.Locale;
+import java.util.Optional;
+import java.util.regex.Pattern;
/**
* Lazily starts and manages one pypiserver container per CodeArtifact repository backing the
@@ -47,11 +57,13 @@ public class PypiserverSidecarManager implements RepositorySidecarManager, Conta
private static final int PYPISERVER_PORT = 8080;
private static final String HEALTH_PATH = "/health";
private static final String PACKAGES_DIR = "/data/packages";
+ private static final Pattern SIMPLE_INDEX_HREF = Pattern.compile("href=\"([^\"]*)\"");
private final ContainerBuilder containerBuilder;
private final ContainerLifecycleManager lifecycleManager;
private final EmulatorConfig config;
private final PerKeyContainerPool pool;
+ private final HttpClient httpClient;
@Inject
public PypiserverSidecarManager(ContainerBuilder containerBuilder, ContainerLifecycleManager lifecycleManager,
@@ -60,6 +72,7 @@ public class PypiserverSidecarManager implements RepositorySidecarManager, Conta
this.lifecycleManager = lifecycleManager;
this.config = config;
this.pool = new PerKeyContainerPool(lifecycleManager, HEALTH_PATH);
+ this.httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
}
@Override
@@ -83,6 +96,254 @@ public class PypiserverSidecarManager implements RepositorySidecarManager, Conta
pool.stopContainer(pypiRepositoryId);
}
+ /**
+ * PyPI packages have no namespace (AWS: "Python ... package versions do not have a
+ * corresponding component"). pypiserver serves every file it holds flatly at
+ * {@code /packages/<filename>} regardless of which package or version it belongs to,
+ * confirmed against a live container, so both are verified before serving anything: that
+ * {@code packageName}'s own simple index genuinely lists {@code assetName}, and that
+ * {@code assetName} itself embeds {@code version} the way every real PyPI filename does
+ * (PEP 427/517 naming) rather than trusting the caller's {@code version} argument on its own.
+ *
+ * <p>{@code assetName} is percent-encoded via {@link #encodeSegment} before joining it to
+ * the fixed {@code /packages/} prefix: without that, a {@code /} inside it would be
+ * indistinguishable from a real path separator and could splice in extra path segments.
+ */
+ @Override
+ public Optional<byte[]> fetchPackageVersionAsset(String repositoryContainerId, String domain, String repository,
+ String namespace, String packageName, String version, String assetName) {
+ if (!filenameEmbedsVersion(assetName, packageName, version)) {
+ return Optional.empty();
+ }
+ String baseUrl = ensureReady(repositoryContainerId, null);
+ if (!packageIndexListsAsset(baseUrl, packageName, assetName)) {
+ return Optional.empty();
+ }
+ URI uri = combine(URI.create(baseUrl), "/packages/" + encodeSegment(assetName));
+ HttpRequest request = HttpRequest.newBuilder(uri).timeout(Duration.ofSeconds(10)).GET().build();
+ HttpResponse<byte[]> response;
+ try {
+ response = httpClient.send(request, BodyHandlers.ofByteArray());
+ } catch (Exception e) {
+ throw new IllegalStateException("Could not reach the pypiserver sidecar to fetch " + assetName, e);
+ }
+ if (response.statusCode() == 404) {
+ return Optional.empty();
+ }
+ if (response.statusCode() != 200) {
+ throw new IllegalStateException("Could not fetch " + assetName + " from the pypiserver sidecar: "
+ + "upstream returned " + response.statusCode());
+ }
+ return Optional.of(response.body());
+ }
+
+ /**
+ * Combines {@code base}'s scheme, authority, and own path prefix (this manager's own {@code
+ * baseUrl} is always a bare container address today with no path, but the sibling Reposilite
+ * client's equivalent helper does need this for its externally configurable URL, so this
+ * stays consistent with it) with an already percent-encoded {@code encodedPath}, built from
+ * {@link #encodeSegment}-encoded, caller-supplied segments. {@code encodedPath} must be fully
+ * encoded going in: this hands the combined string straight to {@link URI#create}, which
+ * only parses, rather than running it through a constructor that would try to quote it a
+ * second time and corrupt any {@code %} it finds (double-encoding, e.g. turning a deliberate
+ * {@code %2F} into {@code %252F}).
+ */
+ private static URI combine(URI base, String encodedPath) {
+ // getRawPath(), not getPath(): if baseUrl's own path carries any encoding, it must
+ // survive untouched rather than being decoded and re-encoded alongside the new path.
+ // getRawAuthority() below is the same reasoning applied to userinfo/host.
+ String prefix = base.getRawPath();
+ if (prefix == null || prefix.equals("/")) {
+ prefix = "";
+ } else if (prefix.endsWith("/")) {
+ prefix = prefix.substring(0, prefix.length() - 1);
+ }
+ return URI.create(base.getScheme() + "://" + base.getRawAuthority() + prefix + encodedPath);
+ }
+
+ /**
+ * Percent-encodes a single untrusted path segment so it can never be mistaken for a
+ * structural separator or a dot-segment. Every byte outside the unreserved set is escaped,
+ * including {@code /} (otherwise legal inside a path, and so otherwise indistinguishable
+ * from one of the real separators this class inserts around it). A segment that is exactly
+ * {@code .} or {@code ..} gets its dots escaped too: those are themselves otherwise-legal
+ * unreserved characters, but a bare {@code ..} between two real {@code /} separators is a
+ * dot-segment that something in front of the sidecar could resolve by walking back out of
+ * the directory this method intends to confine the request to.
+ */
+ private static String encodeSegment(String segment) {
+ if (segment.equals(".") || segment.equals("..")) {
+ return segment.replace(".", "%2E");
+ }
+ StringBuilder encoded = new StringBuilder();
+ for (byte b : segment.getBytes(StandardCharsets.UTF_8)) {
+ int v = b & 0xFF;
+ char c = (char) v;
+ if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')
+ || c == '-' || c == '_' || c == '~' || c == '.') {
+ encoded.append(c);
+ } else {
+ encoded.append('%').append(String.format("%02X", v));
+ }
+ }
+ return encoded.toString();
+ }
+
+ /**
+ * PEP 503 normalization: lowercase, runs of {@code -_.} collapsed to one {@code -}, same as
+ * {@code CodeArtifactPypiController}'s own pre-upload index check. {@code -} and {@code _}
+ * collapsing to the same character here is what lets this match a filename's own escaped
+ * name portion (wheels, and modern sdists, replace a name's internal {@code -}/{@code .}
+ * with {@code _}) against the caller-supplied, unescaped {@code packageName}.
+ */
+ private static String normalizePackageName(String packageName) {
+ return packageName.toLowerCase(Locale.ROOT).replaceAll("[-_.]+", "-");
+ }
+
+ private boolean packageIndexListsAsset(String baseUrl, String packageName, String assetName) {
+ String normalized = normalizePackageName(packageName);
+ URI uri = combine(URI.create(baseUrl), "/simple/" + encodeSegment(normalized) + "/");
+ HttpRequest request = HttpRequest.newBuilder(uri).timeout(Duration.ofSeconds(10)).GET().build();
+ HttpResponse<String> response;
+ try {
+ response = httpClient.send(request, BodyHandlers.ofString());
+ } catch (Exception e) {
+ throw new IllegalStateException("Could not reach the pypiserver sidecar to look up " + packageName, e);
+ }
+ if (response.statusCode() == 404) {
+ return false;
+ }
+ if (response.statusCode() != 200) {
+ throw new IllegalStateException("Could not look up " + packageName + " on the pypiserver sidecar: "
+ + "upstream returned " + response.statusCode());
+ }
+ // An exact entry match, not a substring search: a shorter real filename that happens to
+ // be a substring of a different, longer real filename must not pass this check.
+ return SIMPLE_INDEX_HREF.matcher(response.body()).results()
+ .map(match -> {
+ String href = match.group(1);
+ String filename = href.substring(href.lastIndexOf('/') + 1);
+ int fragment = filename.indexOf('#');
+ return fragment < 0 ? filename : filename.substring(0, fragment);
+ })
+ .anyMatch(assetName::equals);
+ }
+
+ /**
+ * PEP 440 multi-letter keywords that can introduce a continuation after a {@code -} or
+ * {@code .}, per the {@code packaging} library's own {@code VERSION_PATTERN}: {@code
+ * alpha}/{@code beta}/{@code preview}/{@code pre}/{@code rc} are all pre-release aliases,
+ * {@code post}/{@code rev} are post-release aliases, {@code dev} is the dev-release keyword.
+ * The matching single-letter aliases ({@code a}, {@code b}, {@code c}, {@code r}) are handled
+ * separately in {@link #versionBoundaryEndsAt}, since a bare letter needs a following digit
+ * to be treated as a continuation rather than an unrelated filename extension starting there.
+ */
+ private static final List<String> PEP440_CONTINUATION_PREFIXES =
+ List.of("alpha", "beta", "preview", "pre", "post", "rev", "dev", "rc");
+
+ /**
+ * Real PyPI filenames always embed their own version right after a {@code -} (PEP 427/517
+ * naming: {@code <name>-<version>.<ext>} for an sdist, {@code <name>-<version>-<tags>.whl}
+ * for a wheel), so this is checked directly against the filename rather than trusted from the
+ * caller's {@code version} argument alone, which a real file from a different version of the
+ * same package would otherwise sail straight through.
+ *
+ * <p>Checks every {@code -<version>} occurrence and requires both that what follows it
+ * actually ends the version field, handled by {@link #versionBoundaryEndsAt} (requesting
+ * version {@code 1.0} against a real {@code pkg-1.0.1-py3-none-any.whl} contains the literal
+ * text {@code -1.0.}, even though the file's actual version is {@code 1.0.1}, not {@code
+ * 1.0}), and that what precedes it is exactly {@code packageName}, normalized the same way
+ * the simple-index lookup normalizes it. Without that second check, a package name that
+ * itself contains a run of digits could produce a false match inside the name portion of the
+ * filename instead of the real version field: a package literally named {@code foo-1.0-bar}
+ * at version {@code 2.0} produces {@code foo-1.0-bar-2.0.tar.gz}, and the text {@code -1.0-}
+ * inside that name would otherwise look like a complete, valid version {@code 1.0} to a
+ * caller who asked for the wrong one.
+ */
+ private static boolean filenameEmbedsVersion(String assetName, String packageName, String version) {
+ String normalizedPackageName = normalizePackageName(packageName);
+ String marker = "-" + version;
+ for (int idx = assetName.indexOf(marker); idx >= 0; idx = assetName.indexOf(marker, idx + 1)) {
+ String prefix = assetName.substring(0, idx);
+ if (normalizePackageName(prefix).equals(normalizedPackageName)
+ && versionBoundaryEndsAt(assetName, idx + marker.length())) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * {@code after} is where a candidate version match ends; true when the version field genuinely
+ * ends there rather than continuing as a longer PEP 440 version. A {@code +} never does: PEP
+ * 440 gives it exactly one meaning, introducing a non-empty local-version segment, so whatever
+ * immediately follows it is always part of a longer, different version (e.g. {@code 1.0+linux}
+ * is not {@code 1.0}), never an unrelated filename extension starting there.
+ *
+ * <p>A {@code -} or a {@code .} might not end it either, and both need the same continuation
+ * check: PEP 440 allows either character to separate a release segment from a following
+ * {@code <release segment>} ({@code 1.0.1}), a pre/post/dev-release suffix (every keyword in
+ * {@link #PEP440_CONTINUATION_PREFIXES}, plus the single-letter aliases checked below), or,
+ * only after a {@code -} specifically, an implicit post-release with no keyword at all
+ * ({@code 1.0-1} means {@code 1.0.post1}) <em>unless</em> {@link #isWheelBuildTag} says the
+ * digits are actually a PEP 427 wheel build tag instead, which looks identical but does end
+ * the version right there. All of those start right after the matched text the same way, so
+ * both separators are checked for and rejected as real continuations.
+ */
+ private static boolean versionBoundaryEndsAt(String assetName, int after) {
+ if (after == assetName.length()) {
+ return true;
+ }
+ char next = assetName.charAt(after);
+ if (next == '+' || (next != '-' && next != '.')) {
+ return false;
+ }
+ String remainder = assetName.substring(after + 1);
+ if (remainder.isEmpty()) {
+ return true;
+ }
+ if (Character.isDigit(remainder.charAt(0))) {
+ return next == '-' && isWheelBuildTag(assetName, remainder);
+ }
+ for (String continuation : PEP440_CONTINUATION_PREFIXES) {
+ if (remainder.startsWith(continuation)) {
+ return false;
+ }
+ }
+ // Single-letter PEP 440 aliases (a/alpha, b/beta, c/rc, r/rev/post) need a following
+ // digit to count as a continuation; without one, a bare letter is far more likely to be
+ // the start of an unrelated filename extension than a pre/post-release marker.
+ char first = remainder.charAt(0);
+ boolean singleLetterAlias = first == 'a' || first == 'b' || first == 'c' || first == 'r';
+ return !(singleLetterAlias && remainder.length() > 1 && Character.isDigit(remainder.charAt(1)));
+ }
+
+ /**
+ * {@code remainder} is known to start with a digit, right after a {@code -}. PEP 440's
+ * implicit, keyword-less post-release numeral ({@code 1.0-1} meaning {@code 1.0.post1}) and
+ * PEP 427's wheel build tag ({@code digit+ letter*}, placed between the version and the
+ * python/abi/platform tags, e.g. {@code 1.0-1-py3-none-any.whl}) look identical at this
+ * point: both are a bare run of digits (optionally followed by letters) right after the
+ * version. They're told apart by what comes after that run: a build tag is always followed
+ * by another {@code -} introducing the next wheel tag, since wheels always have those three
+ * trailing tags, whereas a post-release numeral is followed directly by the file extension.
+ * Only a {@code .whl} filename can have a build tag at all; an sdist's {@code -<digits>} has
+ * no other valid meaning than the post-release numeral.
+ */
+ private static boolean isWheelBuildTag(String assetName, String remainder) {
+ if (!assetName.endsWith(".whl")) {
+ return false;
+ }
+ int i = 0;
+ while (i < remainder.length() && Character.isDigit(remainder.charAt(i))) {
+ i++;
+ }
+ while (i < remainder.length() && Character.isLetter(remainder.charAt(i))) {
+ i++;
+ }
+ return i < remainder.length() && remainder.charAt(i) == '-';
+ }
+
private PerKeyContainerPool.StartedContainer startContainer(String pypiRepositoryId) {
String image = config.services().codeartifact().pypiImage();
String containerName = ContainerStorageHelper.dockerName(config, "floci-pypiserver-" + pypiRepositoryId);
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClient.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClient.java
index 1123a5033c..a22081221f 100644
--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClient.java
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClient.java
@@ -11,11 +11,13 @@ import org.jboss.logging.Logger;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
+import java.net.URISyntaxException;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpRequest.BodyPublishers;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
+import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.Arrays;
import java.util.Optional;
@@ -80,6 +82,46 @@ public class ReposiliteSidecarClient implements RepositorySidecarManager {
releaseRepository(repositoryContainerId);
}
+ /**
+ * {@code namespace} is the Maven groupId; the caller is responsible for requiring it, since
+ * every real Maven coordinate has one. Builds the standard Maven repository layout path
+ * (groupId with dots as slashes, then artifactId, version, and the asset's own filename).
+ *
+ * <p>Unlike {@link #fetchArtifact(String, String)}, which the real wire-protocol GET route
+ * uses with a single caller-supplied path that is legitimately allowed to contain {@code /}
+ * (it's the whole relative GAV path), every one of this method's four coordinates is its own
+ * independent, untrusted value that is only supposed to be one path segment. Concatenating
+ * them with {@code +} the way the wire-protocol route concatenates its one path would let a
+ * {@code /} inside, say, {@code assetName} splice in extra path segments, or a value of
+ * exactly {@code ..} walk back out of the version directory it's supposed to be confined to.
+ * Each coordinate is therefore percent-encoded on its own via {@link #encodeSegment}, which
+ * escapes both of those, before being joined with the real {@code /} separators this method
+ * supplies itself.
+ */
+ @Override
+ public Optional<byte[]> fetchPackageVersionAsset(String repositoryContainerId, String domain, String repository,
+ String namespace, String packageName, String version, String assetName) {
+ ensureRepository(repositoryContainerId);
+ String baseUrl = manager.ensureReady();
+ StringBuilder path = new StringBuilder("/").append(encodeSegment(repositoryContainerId));
+ for (String namespacePart : namespace.split("\\.", -1)) {
+ path.append('/').append(encodeSegment(namespacePart));
+ }
+ path.append('/').append(encodeSegment(packageName))
+ .append('/').append(encodeSegment(version))
+ .append('/').append(encodeSegment(assetName));
+ HttpRequest request = HttpRequest.newBuilder(combine(URI.create(baseUrl), path.toString()))
+ .timeout(Duration.ofSeconds(30))
+ .header("Authorization", manager.basicAuthHeader())
+ .GET()
+ .build();
+ HttpResponse<byte[]> response = send(request, BodyHandlers.ofByteArray());
+ if (response.statusCode() != 200) {
+ return Optional.empty();
+ }
+ return Optional.of(response.body());
+ }
+
/** Ensures a Reposilite repository named {@code repoId} exists, creating it if not. */
public void ensureRepository(String repoId) {
String baseUrl = manager.ensureReady();
@@ -365,13 +407,94 @@ public class ReposiliteSidecarClient implements RepositorySidecarManager {
}
}
+ /**
+ * {@code gav} isn't necessarily a trusted constant: a caller-supplied asset name can reach
+ * here (via {@link #fetchPackageVersionAsset}'s bridge from {@code GetPackageVersionAsset}, or
+ * the raw wire-protocol GAV path segment itself), so building the URI with a plain string
+ * concatenation would let a {@code #} truncate the path at a URI fragment, or a {@code ?}
+ * reinterpret the rest as a query string, silently fetching a different, unintended path
+ * instead of the one actually requested. The three-argument {@link URI} constructor percent-
+ * encodes whatever isn't already valid inside a path while leaving the real {@code /}
+ * separators alone, unlike {@link URI#create}.
+ */
private HttpRequest.Builder authenticated(String baseUrl, String repoId, String gav) {
return HttpRequest.newBuilder()
- .uri(URI.create(baseUrl + "/" + repoId + "/" + gav))
+ .uri(requestUri(baseUrl, "/" + repoId + "/" + gav))
.timeout(Duration.ofSeconds(30))
.header("Authorization", manager.basicAuthHeader());
}
+ /**
+ * {@code baseUrl} isn't always just scheme and authority: {@code FLOCI_SERVICES_CODEARTIFACT_
+ * MAVEN_URL} can point at a pre-configured Reposilite instance behind its own path prefix, so
+ * that prefix is carried over, and carried over with whatever encoding it was configured with
+ * untouched, rather than dropped or re-encoded. {@code path} can separately carry
+ * caller-supplied text (a package or asset name), so it's not safe to hand straight to
+ * {@link URI#create} either: a raw string concatenation would let a {@code #} in that text
+ * truncate the request at a URI fragment, or a {@code ?} reinterpret the rest as a query
+ * string, silently fetching a different path than the one actually requested.
+ */
+ private static URI requestUri(String baseUrl, String path) {
+ try {
+ String encodedPath = new URI(null, null, path, null, null).getRawPath();
+ return combine(URI.create(baseUrl), encodedPath);
+ } catch (URISyntaxException e) {
+ throw new IllegalStateException("Could not build a request URI for " + path, e);
+ }
+ }
+
+ /**
+ * Combines {@code base}'s scheme, authority, and own path prefix with an already
+ * percent-encoded {@code encodedPath}. {@code encodedPath} must be fully encoded going in:
+ * this method hands the combined string straight to {@link URI#create}, which only parses,
+ * rather than running it through a constructor that would try to quote it a second time and
+ * corrupt any {@code %} it finds (double-encoding, e.g. turning a deliberate {@code %2F} into
+ * {@code %252F}).
+ */
+ private static URI combine(URI base, String encodedPath) {
+ // getRawPath(), not getPath(): the configured prefix's own encoding (e.g. a literal %2F
+ // standing for an escaped slash within one opaque segment) must survive untouched. Decoding
+ // it and handing it to the URI constructor alongside the new path would re-encode the whole
+ // thing together and could turn that %2F into a literal /, changing what the prefix means to
+ // whatever sits in front of the sidecar. getRawAuthority() below is the same reasoning
+ // applied to userinfo/host, e.g. a percent-encoded character inside HTTP Basic credentials.
+ String prefix = base.getRawPath();
+ if (prefix == null || prefix.equals("/")) {
+ prefix = "";
+ } else if (prefix.endsWith("/")) {
+ prefix = prefix.substring(0, prefix.length() - 1);
+ }
+ return URI.create(base.getScheme() + "://" + base.getRawAuthority() + prefix + encodedPath);
+ }
+
+ /**
+ * Percent-encodes a single untrusted path segment so it can never be mistaken for a
+ * structural separator or a dot-segment. Every byte outside the unreserved set is escaped,
+ * including {@code /} (which the generic path-quoting {@link URI} constructor otherwise
+ * leaves alone, since a real {@code /} is legal inside a path). A segment that is exactly
+ * {@code .} or {@code ..} gets its dots escaped too: those are themselves otherwise-legal
+ * unreserved characters, but a bare {@code ..} between two real {@code /} separators is a
+ * dot-segment that something in front of the sidecar could resolve by walking back out of
+ * the directory this method intends to confine the request to.
+ */
+ private static String encodeSegment(String segment) {
+ if (segment.equals(".") || segment.equals("..")) {
+ return segment.replace(".", "%2E");
+ }
+ StringBuilder encoded = new StringBuilder();
+ for (byte b : segment.getBytes(StandardCharsets.UTF_8)) {
+ int v = b & 0xFF;
+ char c = (char) v;
+ if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')
+ || c == '-' || c == '_' || c == '~' || c == '.') {
+ encoded.append(c);
+ } else {
+ encoded.append('%').append(String.format("%02X", v));
+ }
+ }
+ return encoded.toString();
+ }
+
private <T> HttpResponse<T> send(HttpRequest request, HttpResponse.BodyHandler<T> bodyHandler) {
try {
return httpClient.send(request, bodyHandler);
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/RepositorySidecarManager.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/RepositorySidecarManager.java
index 68eb70acbf..dbbb7c0734 100644
--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/RepositorySidecarManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/RepositorySidecarManager.java
@@ -1,5 +1,7 @@
package io.github.hectorvent.floci.services.codeartifact;
+import java.util.Optional;
+
/**
* Common contract for a CodeArtifact sidecar backing one package format: get one repository ready
* to serve that format, and release whatever backing storage it holds once the repository is
@@ -39,4 +41,19 @@ public interface RepositorySidecarManager {
* format is a no-op, not an error.
*/
void release(String repositoryContainerId);
+
+ /**
+ * Fetches the bytes of one named asset already published to this sidecar's own backing
+ * storage, for bridging {@code GetPackageVersionAsset} when there is no generic-format record
+ * to serve it from: Maven, npm, and pypi are metadata-free passthroughs straight to their
+ * sidecar, so nothing in {@code CodeArtifactService}'s own package-version store ever records
+ * what they hold. {@code domain}/{@code repository} exist only for a sidecar whose
+ * {@link #ensureReady} needs a public URL to self-start correctly the same way a real client's
+ * first request would; most implementations ignore them. {@code namespace} is {@code null}
+ * for a format or package that has none. Returns empty both when the asset genuinely isn't
+ * there and when the repository was never started for this format, since an unstarted sidecar
+ * has published nothing either way.
+ */
+ Optional<byte[]> fetchPackageVersionAsset(String repositoryContainerId, String domain, String repository,
+ String namespace, String packageName, String version, String assetName);
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/VerdaccioSidecarManager.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/VerdaccioSidecarManager.java
index 4fbc109fbc..c16acc5522 100644
--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/VerdaccioSidecarManager.java
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/VerdaccioSidecarManager.java
@@ -1,5 +1,7 @@
package io.github.hectorvent.floci.services.codeartifact;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.ContainerTeardown;
import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
@@ -18,7 +20,14 @@ import org.jboss.logging.Logger;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.net.http.HttpResponse.BodyHandlers;
import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Optional;
/**
* Lazily starts and manages one Verdaccio container per CodeArtifact repository backing the
@@ -79,14 +88,18 @@ public class VerdaccioSidecarManager implements RepositorySidecarManager, Contai
private final ContainerLifecycleManager lifecycleManager;
private final EmulatorConfig config;
private final PerKeyContainerPool pool;
+ private final HttpClient httpClient;
+ private final ObjectMapper mapper;
@Inject
public VerdaccioSidecarManager(ContainerBuilder containerBuilder, ContainerLifecycleManager lifecycleManager,
- EmulatorConfig config) {
+ EmulatorConfig config, ObjectMapper mapper) {
this.containerBuilder = containerBuilder;
this.lifecycleManager = lifecycleManager;
this.config = config;
this.pool = new PerKeyContainerPool(lifecycleManager, HEALTH_PATH);
+ this.httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ this.mapper = mapper;
}
@Override
@@ -117,6 +130,142 @@ public class VerdaccioSidecarManager implements RepositorySidecarManager, Contai
pool.stopContainer(npmRepositoryId);
}
+ /**
+ * {@code namespace} is the npm scope without its leading {@code @}, or {@code null} for an
+ * unscoped package; real npm (and Verdaccio, confirmed against a live container) serves every
+ * version's tarball at {@code <scoped-or-plain-name>/-/<filename>}, with the scope, if any,
+ * still carrying its {@code @} in that path even though the filename itself never includes it.
+ *
+ * <p>The tarball path alone is keyed by filename, not version: a package's own metadata is
+ * checked first (its {@code versions.<version>.dist.tarball} entry) to confirm {@code version}
+ * genuinely maps to {@code assetName} before fetching it, rather than serving whatever file
+ * that name happens to resolve to under a version the caller never actually published.
+ *
+ * <p>{@code namespace}, {@code packageName}, and {@code assetName} are each their own
+ * independently caller-supplied value, only meant to be one path segment; {@code
+ * packagePath} and the final {@code /-/<assetName>} suffix supply the real {@code /}
+ * separators themselves, around each value already percent-encoded on its own via {@link
+ * #encodeSegment}. Without that, a {@code /} inside, say, {@code assetName} would be
+ * indistinguishable from one of those real separators and could splice in extra path
+ * segments.
+ */
+ @Override
+ public Optional<byte[]> fetchPackageVersionAsset(String repositoryContainerId, String domain, String repository,
+ String namespace, String packageName, String version, String assetName) {
+ String publicUrl = config.effectiveBaseUrl() + "/codeartifact/npm/" + domain + "/" + repository + "/";
+ String baseUrl = ensureReady(repositoryContainerId, publicUrl);
+ String packagePath = packagePath(namespace, packageName);
+ if (!versionHasAsset(baseUrl, packagePath, version, assetName)) {
+ return Optional.empty();
+ }
+ URI uri = combine(URI.create(baseUrl), "/" + packagePath + "/-/" + encodeSegment(assetName));
+ HttpRequest request = HttpRequest.newBuilder(uri).timeout(Duration.ofSeconds(10)).GET().build();
+ HttpResponse<byte[]> response;
+ try {
+ response = httpClient.send(request, BodyHandlers.ofByteArray());
+ } catch (Exception e) {
+ throw new IllegalStateException("Could not reach the Verdaccio sidecar to fetch " + packagePath, e);
+ }
+ if (response.statusCode() == 404) {
+ return Optional.empty();
+ }
+ if (response.statusCode() != 200) {
+ throw new IllegalStateException("Could not fetch " + packagePath + "/-/" + assetName
+ + " from the Verdaccio sidecar: upstream returned " + response.statusCode());
+ }
+ return Optional.of(response.body());
+ }
+
+ private static String packagePath(String namespace, String packageName) {
+ return namespace == null
+ ? encodeSegment(packageName)
+ : "@" + encodeSegment(namespace) + "/" + encodeSegment(packageName);
+ }
+
+ private boolean versionHasAsset(String baseUrl, String packagePath, String version, String assetName) {
+ HttpRequest request = HttpRequest.newBuilder(combine(URI.create(baseUrl), "/" + packagePath))
+ .timeout(Duration.ofSeconds(10)).GET().build();
+ HttpResponse<String> response;
+ try {
+ response = httpClient.send(request, BodyHandlers.ofString());
+ } catch (Exception e) {
+ throw new IllegalStateException("Could not reach the Verdaccio sidecar to look up " + packagePath, e);
+ }
+ if (response.statusCode() == 404) {
+ return false;
+ }
+ if (response.statusCode() != 200) {
+ throw new IllegalStateException("Could not look up " + packagePath + " on the Verdaccio sidecar: "
+ + "upstream returned " + response.statusCode());
+ }
+ JsonNode versionNode;
+ try {
+ versionNode = mapper.readTree(response.body()).path("versions").path(version);
+ } catch (Exception e) {
+ throw new IllegalStateException("Could not parse Verdaccio's package metadata for " + packagePath, e);
+ }
+ if (versionNode.isMissingNode()) {
+ return false;
+ }
+ String tarballUrl = versionNode.path("dist").path("tarball").asText("");
+ String tarballFilename = tarballUrl.substring(tarballUrl.lastIndexOf('/') + 1);
+ return assetName.equals(tarballFilename);
+ }
+
+ /**
+ * Combines {@code base}'s scheme, authority, and own path prefix (this manager's own {@code
+ * baseUrl} is always a bare container address today with no path, but the sibling Reposilite
+ * client's equivalent helper does need this for its externally configurable URL, so this
+ * stays consistent with it) with an already percent-encoded {@code encodedPath}, built from
+ * {@link #encodeSegment}-encoded, caller-supplied segments. {@code encodedPath} must be fully
+ * encoded going in: this hands the combined string straight to {@link URI#create}, which
+ * only parses, rather than running it through a constructor that would try to quote it a
+ * second time and corrupt any {@code %} it finds (double-encoding, e.g. turning a deliberate
+ * {@code %2F} into {@code %252F}).
+ */
+ private static URI combine(URI base, String encodedPath) {
+ // getRawPath(), not getPath(): if baseUrl's own path carries any encoding (an escaped
+ // slash within one opaque segment, say), it must survive untouched rather than being
+ // decoded and re-encoded alongside the new path, which could change what it means to
+ // whatever sits in front of the sidecar. getRawAuthority() below is the same reasoning
+ // applied to userinfo/host.
+ String prefix = base.getRawPath();
+ if (prefix == null || prefix.equals("/")) {
+ prefix = "";
+ } else if (prefix.endsWith("/")) {
+ prefix = prefix.substring(0, prefix.length() - 1);
+ }
+ return URI.create(base.getScheme() + "://" + base.getRawAuthority() + prefix + encodedPath);
+ }
+
+ /**
+ * Percent-encodes a single untrusted path segment so it can never be mistaken for a
+ * structural separator or a dot-segment. Every byte outside the unreserved set is escaped,
+ * including {@code /} (otherwise legal inside a path, and so otherwise indistinguishable
+ * from one of the real separators this class inserts around it). A segment that is exactly
+ * {@code .} or {@code ..} gets its dots escaped too: those are themselves otherwise-legal
+ * unreserved characters, but a bare {@code ..} between two real {@code /} separators is a
+ * dot-segment that something in front of the sidecar could resolve by walking back out of
+ * the directory this method intends to confine the request to.
+ */
+ private static String encodeSegment(String segment) {
+ if (segment.equals(".") || segment.equals("..")) {
+ return segment.replace(".", "%2E");
+ }
+ StringBuilder encoded = new StringBuilder();
+ for (byte b : segment.getBytes(StandardCharsets.UTF_8)) {
+ int v = b & 0xFF;
+ char c = (char) v;
+ if ((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9')
+ || c == '-' || c == '_' || c == '~' || c == '.') {
+ encoded.append(c);
+ } else {
+ encoded.append('%').append(String.format("%02X", v));
+ }
+ }
+ return encoded.toString();
+ }
+
private PerKeyContainerPool.StartedContainer startContainer(String npmRepositoryId, String publicUrl) {
String image = config.services().codeartifact().npmImage();
String containerName = ContainerStorageHelper.dockerName(config, "floci-verdaccio-" + npmRepositoryId);
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenDockerIntegrationTest.java
index 518df921ad..166c76f471 100644
--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenDockerIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenDockerIntegrationTest.java
@@ -25,6 +25,7 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import static io.restassured.RestAssured.given;
+import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.startsWith;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -358,6 +359,44 @@ class CodeArtifactMavenDockerIntegrationTest {
assertEquals(repositoryCountBeforeDelete - 1, reposiliteSettingsRepositoryCount());
}
+ /**
+ * The real-world gap this closes: {@code GetPackageVersionAsset} used to 404 for every
+ * Maven-deployed artifact since Floci's generic-format package-version store never had a
+ * record for it (Maven publishing goes straight to Reposilite, bypassing it entirely). Proves
+ * the JSON API now bridges to the same artifact {@link #deployThenFetchRoundTripsTheExactBytes}
+ * already confirmed is really there, through the real sidecar, not a mock.
+ */
+ @Test
+ @Order(12)
+ void getPackageVersionAssetBridgesToTheRealReposiliteSidecar() {
+ byte[] fetched = given().header("Authorization", AUTH)
+ .get("/v1/package/version/asset?domain=" + DOMAIN + "&repository=" + REPO + "&format=maven"
+ + "&namespace=com.example&package=spike&version=1.0.0&asset=spike-1.0.0.jar")
+ .then().statusCode(200)
+ .header("X-AssetName", equalTo("spike-1.0.0.jar"))
+ .header("X-PackageVersion", equalTo("1.0.0"))
+ .extract().asByteArray();
+ assertEquals("real-jar-bytes", new String(fetched, StandardCharsets.UTF_8));
+ }
+
+ @Test
+ @Order(12)
+ void getPackageVersionAssetRequiresANamespaceForMaven() {
+ given().header("Authorization", AUTH)
+ .get("/v1/package/version/asset?domain=" + DOMAIN + "&repository=" + REPO + "&format=maven"
+ + "&package=spike&version=1.0.0&asset=spike-1.0.0.jar")
+ .then().statusCode(400);
+ }
+
+ @Test
+ @Order(12)
+ void getPackageVersionAssetReturns404ForAMavenAssetThatWasNeverDeployed() {
+ given().header("Authorization", AUTH)
+ .get("/v1/package/version/asset?domain=" + DOMAIN + "&repository=" + REPO + "&format=maven"
+ + "&namespace=com.example&package=spike&version=1.0.0&asset=does-not-exist.jar")
+ .then().statusCode(404);
+ }
+
/**
* Reads the real Reposilite instance's own {@code maven} settings domain directly (the same
* shared container {@link ReposiliteSidecarManager} started for every test in this class).
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDockerIntegrationTest.java
index 0fa3317f2c..e17ca59a51 100644
--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDockerIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDockerIntegrationTest.java
@@ -28,6 +28,7 @@ import java.util.concurrent.atomic.AtomicInteger;
import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.anyOf;
+import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.startsWith;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
@@ -284,6 +285,34 @@ class CodeArtifactNpmDockerIntegrationTest {
.then().statusCode(200);
}
+ /**
+ * The real-world gap this closes: {@code GetPackageVersionAsset} used to 404 for every
+ * npm-published package since Floci's generic-format package-version store never had a
+ * record for it (npm publishing goes straight to Verdaccio, bypassing it entirely). Proves the
+ * JSON API now bridges to the same tarball {@link #publishThenFetchRoundTripsTheRealNpmEnvelope}
+ * already confirmed is really there, through the real sidecar, not a mock.
+ */
+ @Test
+ @Order(9)
+ void getPackageVersionAssetBridgesToTheRealVerdaccioSidecar() throws IOException {
+ byte[] fetched = given().header("Authorization", AUTH)
+ .get("/v1/package/version/asset?domain=" + DOMAIN + "&repository=" + REPO + "&format=npm"
+ + "&package=" + PACKAGE_NAME + "&version=1.0.0&asset=" + PACKAGE_NAME + "-1.0.0.tgz")
+ .then().statusCode(200)
+ .header("X-AssetName", equalTo(PACKAGE_NAME + "-1.0.0.tgz"))
+ .extract().asByteArray();
+ assertArrayEquals(expectedTarballBytes, fetched);
+ }
+
+ @Test
+ @Order(9)
+ void getPackageVersionAssetReturns404ForAnNpmAssetThatWasNeverPublished() {
+ given().header("Authorization", AUTH)
+ .get("/v1/package/version/asset?domain=" + DOMAIN + "&repository=" + REPO + "&format=npm"
+ + "&package=" + PACKAGE_NAME + "&version=1.0.0&asset=does-not-exist.tgz")
+ .then().statusCode(404);
+ }
+
/**
* Proves the actual mechanism, not just that {@code VerdaccioSidecarManager} has a method
* named right: {@code ContainerTeardowns.stopAll} runs every {@code ContainerTeardown} on
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java
index adc1925ee5..2cd1dd7fcd 100644
--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java
@@ -20,6 +20,7 @@ import java.util.concurrent.atomic.AtomicInteger;
import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.startsWith;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -317,6 +318,35 @@ class CodeArtifactPypiDockerIntegrationTest {
.then().statusCode(200);
}
+ /**
+ * The real-world gap this closes: {@code GetPackageVersionAsset} used to 404 for every
+ * pypi-published file since Floci's generic-format package-version store never had a record
+ * for it (pypi publishing goes straight to pypiserver, bypassing it entirely). Proves the JSON
+ * API now bridges to the same file {@link #uploadThenDownloadRoundTripsTheExactBytesThroughTheSimpleIndex}
+ * already confirmed is really there, through the real sidecar, not a mock. No namespace: pypi
+ * packages don't have one.
+ */
+ @Test
+ @Order(12)
+ void getPackageVersionAssetBridgesToTheRealPypiserverSidecar() {
+ byte[] fetched = given().header("Authorization", AUTH)
+ .get("/v1/package/version/asset?domain=" + DOMAIN + "&repository=" + REPO + "&format=pypi"
+ + "&package=" + PACKAGE_NAME + "&version=1.0.0&asset=" + FILENAME)
+ .then().statusCode(200)
+ .header("X-AssetName", equalTo(FILENAME))
+ .extract().asByteArray();
+ assertEquals("real-sdist-bytes", new String(fetched, StandardCharsets.UTF_8));
+ }
+
+ @Test
+ @Order(12)
+ void getPackageVersionAssetReturns404ForAPypiAssetThatWasNeverUploaded() {
+ given().header("Authorization", AUTH)
+ .get("/v1/package/version/asset?domain=" + DOMAIN + "&repository=" + REPO + "&format=pypi"
+ + "&package=" + PACKAGE_NAME + "&version=1.0.0&asset=does-not-exist.tar.gz")
+ .then().statusCode(404);
+ }
+
/**
* Proves the actual mechanism, not just that {@code PypiserverSidecarManager} has a method
* named right: {@code ContainerTeardowns.stopAll} runs every {@code ContainerTeardown} on
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java
index 9cede4acc8..c457e02e10 100644
--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java
@@ -36,6 +36,8 @@ import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.ArgumentMatchers.isNull;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
@@ -51,6 +53,7 @@ class CodeArtifactServiceTest {
private AccountAwareStorageBackend<CodeArtifactRepository> repoStore;
private VerdaccioSidecarManager verdaccioManager;
private ReposiliteSidecarClient reposiliteClient;
+ private PypiserverSidecarManager pypiserverManager;
@BeforeEach
void setUp() {
@@ -72,8 +75,11 @@ class CodeArtifactServiceTest {
when(verdaccioManager.format()).thenReturn("npm");
reposiliteClient = mock(ReposiliteSidecarClient.class);
when(reposiliteClient.format()).thenReturn("maven");
+ pypiserverManager = mock(PypiserverSidecarManager.class);
+ when(pypiserverManager.format()).thenReturn("pypi");
service = new CodeArtifactService(domainStore, repoStore, packageVersionStore, regionResolver, config,
- true, null, new CodeArtifactSidecarRegistry(List.of(verdaccioManager, reposiliteClient)));
+ true, null, new CodeArtifactSidecarRegistry(
+ List.of(verdaccioManager, reposiliteClient, pypiserverManager)));
}
// -------------------------------------------------------------- domains
@@ -1027,6 +1033,138 @@ class CodeArtifactServiceTest {
assertEquals("package-version", wrongRevision.getExtendedData().get("resourceType"));
}
+ @Test
+ void getPackageVersionAssetBridgesToReposiliteForMavenSinceNoGenericRecordCanExist() {
+ service.createDomain(REGION, "dom", null, Map.of());
+ CodeArtifactRepository created =
+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());
+ String repoId = created.getSidecarContainerIds().get("maven");
+ byte[] content = "jar bytes".getBytes(StandardCharsets.UTF_8);
+ when(reposiliteClient.fetchPackageVersionAsset(eq(repoId), eq("dom"), eq("repo"), eq("com.example"),
+ eq("my-artifact"), eq("1.0.0"), eq("my-artifact-1.0.0.jar"))).thenReturn(Optional.of(content));
+
+ PackageVersionAssetResult result = service.getPackageVersionAsset(REGION, "dom", null, "repo", "maven",
+ "com.example", "my-artifact", "1.0.0", "my-artifact-1.0.0.jar", null);
+
+ assertEquals("jar bytes", new String(result.asset().getContent(), StandardCharsets.UTF_8));
+ assertEquals(sha256Hex((repoId + "/com.example/my-artifact/1.0.0").getBytes(StandardCharsets.UTF_8)),
+ result.packageVersionRevision());
+ }
+
+ @Test
+ void getPackageVersionAssetSynthesizesADifferentRevisionForTheSameCoordinatesInADifferentRepository() {
+ service.createDomain(REGION, "dom", null, Map.of());
+ service.createRepository(REGION, "dom", null, "repo-a", null, null, Map.of());
+ service.createRepository(REGION, "dom", null, "repo-b", null, null, Map.of());
+ byte[] content = "jar bytes".getBytes(StandardCharsets.UTF_8);
+ when(reposiliteClient.fetchPackageVersionAsset(anyString(), eq("dom"), anyString(), eq("com.example"),
+ eq("my-artifact"), eq("1.0.0"), eq("my-artifact-1.0.0.jar"))).thenReturn(Optional.of(content));
+
+ PackageVersionAssetResult fromA = service.getPackageVersionAsset(REGION, "dom", null, "repo-a", "maven",
+ "com.example", "my-artifact", "1.0.0", "my-artifact-1.0.0.jar", null);
+ PackageVersionAssetResult fromB = service.getPackageVersionAsset(REGION, "dom", null, "repo-b", "maven",
+ "com.example", "my-artifact", "1.0.0", "my-artifact-1.0.0.jar", null);
+
+ assertTrue(!fromA.packageVersionRevision().equals(fromB.packageVersionRevision()),
+ "identical coordinates in two different repositories must not synthesize the same revision");
+ }
+
+ @Test
+ void getPackageVersionAssetSynthesizesTheSameRevisionForEveryAssetOfOneMavenVersion() {
+ service.createDomain(REGION, "dom", null, Map.of());
+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());
+ when(reposiliteClient.fetchPackageVersionAsset(anyString(), eq("dom"), eq("repo"), eq("com.example"),
+ eq("my-artifact"), eq("1.0.0"), eq("my-artifact-1.0.0.jar")))
+ .thenReturn(Optional.of("jar bytes".getBytes(StandardCharsets.UTF_8)));
+ when(reposiliteClient.fetchPackageVersionAsset(anyString(), eq("dom"), eq("repo"), eq("com.example"),
+ eq("my-artifact"), eq("1.0.0"), eq("my-artifact-1.0.0.pom")))
+ .thenReturn(Optional.of("pom bytes".getBytes(StandardCharsets.UTF_8)));
+
+ PackageVersionAssetResult jar = service.getPackageVersionAsset(REGION, "dom", null, "repo", "maven",
+ "com.example", "my-artifact", "1.0.0", "my-artifact-1.0.0.jar", null);
+ String jarRevision = jar.packageVersionRevision();
+
+ // A client that described the version once and now fetches a second asset pinned to that
+ // same revision must not get a spurious not-found just because this asset's own bytes
+ // differ from the first one's.
+ PackageVersionAssetResult pom = service.getPackageVersionAsset(REGION, "dom", null, "repo", "maven",
+ "com.example", "my-artifact", "1.0.0", "my-artifact-1.0.0.pom", jarRevision);
+
+ assertEquals(jarRevision, pom.packageVersionRevision());
+ }
+
+ @Test
+ void getPackageVersionAssetRequiresNamespaceForMaven() {
+ service.createDomain(REGION, "dom", null, Map.of());
+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());
+
+ AwsException e = assertThrows(AwsException.class, () -> service.getPackageVersionAsset(REGION, "dom", null,
+ "repo", "maven", null, "my-artifact", "1.0.0", "my-artifact-1.0.0.jar", null));
+ assertEquals("ValidationException", e.getErrorCode());
+ }
+
+ @Test
+ void getPackageVersionAssetBridgesToVerdaccioForNpm() {
+ service.createDomain(REGION, "dom", null, Map.of());
+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());
+ byte[] content = "tarball bytes".getBytes(StandardCharsets.UTF_8);
+ when(verdaccioManager.fetchPackageVersionAsset(anyString(), eq("dom"), eq("repo"), eq("myscope"),
+ eq("my-pkg"), eq("1.0.0"), eq("my-pkg-1.0.0.tgz"))).thenReturn(Optional.of(content));
+
+ PackageVersionAssetResult result = service.getPackageVersionAsset(REGION, "dom", null, "repo", "npm",
+ "myscope", "my-pkg", "1.0.0", "my-pkg-1.0.0.tgz", null);
+
+ assertEquals("tarball bytes", new String(result.asset().getContent(), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void getPackageVersionAssetBridgesToPypiserver() {
+ service.createDomain(REGION, "dom", null, Map.of());
+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());
+ byte[] content = "wheel bytes".getBytes(StandardCharsets.UTF_8);
+ when(pypiserverManager.fetchPackageVersionAsset(anyString(), eq("dom"), eq("repo"), isNull(),
+ eq("my-pkg"), eq("1.0.0"), eq("my_pkg-1.0.0-py3-none-any.whl"))).thenReturn(Optional.of(content));
+
+ PackageVersionAssetResult result = service.getPackageVersionAsset(REGION, "dom", null, "repo", "pypi",
+ null, "my-pkg", "1.0.0", "my_pkg-1.0.0-py3-none-any.whl", null);
+
+ assertEquals("wheel bytes", new String(result.asset().getContent(), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void getPackageVersionAssetReturns404WhenTheSidecarHasNoSuchAssetForAContainerBackedFormat() {
+ service.createDomain(REGION, "dom", null, Map.of());
+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());
+ when(pypiserverManager.fetchPackageVersionAsset(anyString(), eq("dom"), eq("repo"), isNull(),
+ eq("my-pkg"), eq("1.0.0"), eq("missing.whl"))).thenReturn(Optional.empty());
+
+ AwsException e = assertThrows(AwsException.class, () -> service.getPackageVersionAsset(REGION, "dom", null,
+ "repo", "pypi", null, "my-pkg", "1.0.0", "missing.whl", null));
+ assertEquals("ResourceNotFoundException", e.getErrorCode());
+ assertEquals("missing.whl", e.getExtendedData().get("resourceId"));
+ assertEquals("asset", e.getExtendedData().get("resourceType"));
+ }
+
+ @Test
+ void getPackageVersionAssetValidatesTheCallerProvidedRevisionForAContainerBackedFormatToo() {
+ service.createDomain(REGION, "dom", null, Map.of());
+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());
+ byte[] content = "wheel bytes".getBytes(StandardCharsets.UTF_8);
+ when(pypiserverManager.fetchPackageVersionAsset(anyString(), eq("dom"), eq("repo"), isNull(),
+ eq("my-pkg"), eq("1.0.0"), eq("my_pkg-1.0.0.whl"))).thenReturn(Optional.of(content));
+ String realRevision = service.getPackageVersionAsset(REGION, "dom", null, "repo", "pypi", null, "my-pkg",
+ "1.0.0", "my_pkg-1.0.0.whl", null).packageVersionRevision();
+
+ PackageVersionAssetResult matching = service.getPackageVersionAsset(REGION, "dom", null, "repo", "pypi",
+ null, "my-pkg", "1.0.0", "my_pkg-1.0.0.whl", realRevision);
+ assertEquals("wheel bytes", new String(matching.asset().getContent(), StandardCharsets.UTF_8));
+
+ AwsException wrongRevision = assertThrows(AwsException.class, () -> service.getPackageVersionAsset(REGION,
+ "dom", null, "repo", "pypi", null, "my-pkg", "1.0.0", "my_pkg-1.0.0.whl", "not-the-real-revision"));
+ assertEquals("ResourceNotFoundException", wrongRevision.getErrorCode());
+ assertEquals("package-version", wrongRevision.getExtendedData().get("resourceType"));
+ }
+
@Test
void concurrentUnfinishedPublishesOfDifferentAssetsBothPersist() throws InterruptedException {
service.createDomain(REGION, "dom", null, Map.of());
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManagerTest.java
index 24406cf5c8..80277a30bf 100644
--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManagerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManagerTest.java
@@ -1,16 +1,26 @@
package io.github.hectorvent.floci.services.codeartifact;
+import com.sun.net.httpserver.HttpServer;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.ContainerTeardown;
import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
import io.github.hectorvent.floci.core.common.docker.PerKeyContainerPool;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Field;
+import java.net.InetSocketAddress;
+import java.nio.charset.StandardCharsets;
+import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicReference;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoInteractions;
@@ -27,16 +37,368 @@ class PypiserverSidecarManagerTest {
private final ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
private final EmulatorConfig config = mock(EmulatorConfig.class);
+ private HttpServer backend;
+ private String backendUrl;
+
+ @BeforeEach
+ void setUpBackend() throws Exception {
+ backend = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
+ backend.createContext("/health", exchange -> {
+ exchange.sendResponseHeaders(200, -1);
+ exchange.close();
+ });
+ backend.start();
+ backendUrl = "http://127.0.0.1:" + backend.getAddress().getPort();
+ }
+
+ @AfterEach
+ void tearDownBackend() {
+ backend.stop(0);
+ }
+
@Test
void implementsContainerTeardownSoStateResetAndNukeCanFindIt() {
assertInstanceOf(ContainerTeardown.class, manager());
}
+ @Test
+ void fetchPackageVersionAssetGetsTheFlatPackagesPath() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ respondWithSimpleIndex("demo-pkg", "demo_pkg-1.0.0.tar.gz");
+ backend.createContext("/packages/demo_pkg-1.0.0.tar.gz", exchange -> {
+ byte[] body = "sdist bytes".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "demo-pkg",
+ "1.0.0", "demo_pkg-1.0.0.tar.gz");
+
+ assertTrue(result.isPresent());
+ assertEquals("sdist bytes", new String(result.get(), StandardCharsets.UTF_8));
+ }
+
+ /**
+ * The fix for a real finding: the filename alone was trusted as proof of version, so a caller
+ * could request a different (even nonexistent) version and still get back a real file that
+ * actually belongs to a version it never named.
+ */
+ @Test
+ void fetchPackageVersionAssetReturnsEmptyWhenTheRequestedVersionDoesNotMatchTheFilename() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ respondWithSimpleIndex("demo-pkg", "demo_pkg-1.0.0.tar.gz");
+ backend.createContext("/packages/demo_pkg-1.0.0.tar.gz", exchange -> {
+ byte[] body = "should never be served".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "demo-pkg",
+ "9.9.9", "demo_pkg-1.0.0.tar.gz");
+
+ assertTrue(result.isEmpty());
+ }
+
+ /**
+ * The fix for a real finding: a plain substring check on {@code "-" + version} would wrongly
+ * accept this, since {@code pkg-1.0.1-...} literally contains the text {@code -1.0.} even
+ * though the file's real version is {@code 1.0.1}, not {@code 1.0}.
+ */
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfTheFilenamesRealVersion() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ respondWithSimpleIndex("demo-pkg", "demo_pkg-1.0.1-py3-none-any.whl");
+ backend.createContext("/packages/demo_pkg-1.0.1-py3-none-any.whl", exchange -> {
+ byte[] body = "should never be served".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "demo-pkg",
+ "1.0", "demo_pkg-1.0.1-py3-none-any.whl");
+
+ assertTrue(result.isEmpty());
+ }
+
+ /**
+ * The fix for a real finding on top of the previous one: a PEP 440 pre/post/dev-release
+ * suffix also starts right after the matched version text with a letter, not a digit, so the
+ * first boundary fix (reject only when followed by another digit) still wrongly accepted
+ * {@code 1.0.post1}, {@code 1.0.dev1}, and {@code 1.0.rc1} as if they were plain {@code 1.0}.
+ */
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfAPostReleaseSuffix() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0.post1");
+ }
+
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfADevReleaseSuffix() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0.dev1");
+ }
+
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfAReleaseCandidateSuffix() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0.rc1");
+ }
+
+ /**
+ * The fix for a real finding: {@code +} introduces a PEP 440 local-version segment, so
+ * whatever follows it is always part of a longer, different version. {@code 1.0+linux} is
+ * not {@code 1.0}, but a {@code +} used to be treated the same as a wheel tag's {@code -} and
+ * accepted as ending the version field right there.
+ */
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfALocalVersionSegment() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0+linux");
+ }
+
+ /**
+ * The fix for a real finding: PEP 440 allows a bare {@code -<digits>} right after a release
+ * segment, with no keyword at all, as a legacy, implicit way of writing a post-release
+ * ({@code 1.0-1} means {@code 1.0.post1}). The old check treated every {@code -} right after
+ * a matched version as an unconditional boundary, so this legacy form wasn't recognized as a
+ * continuation.
+ */
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfAnImplicitPostRelease() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0-1");
+ }
+
+ /**
+ * Same finding, the explicit-keyword form of a legacy post-release separator: PEP 440 also
+ * allows {@code -} (not just {@code .}) directly before {@code post}/{@code dev}/{@code rc}.
+ */
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfAHyphenSeparatedPostRelease() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0-post1");
+ }
+
+ /**
+ * The fix for a real finding: {@code pre}, {@code preview}, and {@code c} are all PEP 440
+ * aliases for {@code rc}, and weren't in the continuation list, so a filename using one of
+ * them could sail through as if it were the plain requested version.
+ */
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfAPreReleaseAlias() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0.pre1");
+ }
+
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfAPreviewReleaseAlias() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0.preview1");
+ }
+
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfAShorthandReleaseCandidateAlias() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0.c1");
+ }
+
+ /**
+ * Same finding, the post-release side: {@code r} is a PEP 440 alias for {@code post}/{@code
+ * rev}, wasn't recognized either, and sits one character shorter than {@code rev} so it needs
+ * its own coverage.
+ */
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatIsOnlyAPrefixOfAShorthandPostReleaseAlias() throws Exception {
+ assertRequestedVersionRejectedAgainstRealVersion("1.0-r1");
+ }
+
+ /**
+ * The fix for a real finding on the opposite side of the same area: a PEP 427 wheel build tag
+ * ({@code digit+ letter*}, placed between the version and the python/abi/platform tags) looks
+ * exactly like PEP 440's implicit post-release numeral at the point this check inspects, but
+ * actually ends the version right there rather than continuing it. {@code demo_pkg-1.0-1-py3-
+ * none-any.whl} is version {@code 1.0} with build tag {@code 1}, not version {@code 1.0.post1}.
+ */
+ @Test
+ void fetchPackageVersionAssetAcceptsAWheelBuildTagRatherThanTreatingItAsAPostRelease() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ respondWithSimpleIndex("demo-pkg", "demo_pkg-1.0-1-py3-none-any.whl");
+ backend.createContext("/packages/demo_pkg-1.0-1-py3-none-any.whl", exchange -> {
+ byte[] body = "wheel bytes".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "demo-pkg",
+ "1.0", "demo_pkg-1.0-1-py3-none-any.whl");
+
+ assertTrue(result.isPresent());
+ assertEquals("wheel bytes", new String(result.get(), StandardCharsets.UTF_8));
+ }
+
+ /**
+ * The fix for a real finding: a package name can itself contain a run of digits that looks
+ * like a version, and a legacy sdist filename (pre-dating the PEP 625 escaping modern
+ * {@code demo_pkg-1.0.0.tar.gz}-style filenames in this test class use) keeps the project
+ * name's own hyphens literal rather than underscoring them. A package literally named {@code
+ * foo-1.0-bar} at version {@code 2.0} then produces the filename {@code
+ * foo-1.0-bar-2.0.tar.gz}; the old check only looked at what followed a {@code -<version>}
+ * match, so the {@code -1.0-} inside the name portion looked like a complete, valid version
+ * {@code 1.0} on its own, serving the version-2.0 asset for a request asking for 1.0. Fixed
+ * by also requiring that what precedes the match is exactly the requested package name,
+ * normalized the same way the simple-index lookup normalizes it; this filenameEmbedsVersion
+ * check now fails before any request is even sent.
+ */
+ @Test
+ void fetchPackageVersionAssetRejectsAVersionThatOnlyMatchesInsideThePackageNamePortion() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "foo-1.0-bar",
+ "1.0", "foo-1.0-bar-2.0.tar.gz");
+
+ assertTrue(result.isEmpty());
+ }
+
+ private void assertRequestedVersionRejectedAgainstRealVersion(String realVersion) throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ String filename = "demo_pkg-" + realVersion + ".tar.gz";
+ respondWithSimpleIndex("demo-pkg", filename);
+ backend.createContext("/packages/" + filename, exchange -> {
+ byte[] body = "should never be served".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "demo-pkg",
+ "1.0", filename);
+
+ assertTrue(result.isEmpty(), "version 1.0 must not match " + filename);
+ }
+
+ /**
+ * The fix for a real finding: the package-index check used to be a plain substring search,
+ * which a shorter real filename that happens to be a substring of a different, longer real
+ * filename could pass incorrectly.
+ */
+ @Test
+ void fetchPackageVersionAssetRequiresAnExactIndexEntryNotASubstringMatch() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ // The index genuinely lists a longer filename that contains the requested, shorter one as
+ // a substring; the shorter one was never actually published on its own.
+ respondWithSimpleIndex("demo-pkg", "demo_pkg-1.0.0.tar.gz.asc");
+ backend.createContext("/packages/demo_pkg-1.0.0.tar.gz", exchange -> {
+ byte[] body = "should never be served".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "demo-pkg",
+ "1.0.0", "demo_pkg-1.0.0.tar.gz");
+
+ assertTrue(result.isEmpty());
+ }
+
+ @Test
+ void fetchPackageVersionAssetReturnsEmptyWhenThePackageWasNeverUploaded() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ backend.createContext("/simple/demo-pkg/", exchange -> {
+ exchange.sendResponseHeaders(404, -1);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "demo-pkg",
+ "1.0.0", "missing.whl");
+
+ assertTrue(result.isEmpty());
+ }
+
+ /**
+ * The fix for a real finding: pypiserver serves every file it holds flatly, regardless of
+ * which package it belongs to, so without checking the named package's own simple index first,
+ * a caller could name a real file that belongs to a different package in the same repository
+ * and get it back under the wrong package's name.
+ */
+ @Test
+ void fetchPackageVersionAssetReturnsEmptyWhenTheFileBelongsToADifferentPackage() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ respondWithSimpleIndex("demo-pkg", "demo_pkg-1.0.0.tar.gz");
+ backend.createContext("/packages/other_pkg-1.0.0.tar.gz", exchange -> {
+ byte[] body = "should never be served".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "demo-pkg",
+ "1.0.0", "other_pkg-1.0.0.tar.gz");
+
+ assertTrue(result.isEmpty());
+ }
+
+ @Test
+ void fetchPackageVersionAssetThrowsRatherThanReportingNotFoundOnASidecarServerError() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ backend.createContext("/simple/demo-pkg/", exchange -> {
+ exchange.sendResponseHeaders(500, -1);
+ exchange.close();
+ });
+
+ assertThrows(IllegalStateException.class, () -> manager.fetchPackageVersionAsset("pypi-repo-1", "dom",
+ "repo", null, "demo-pkg", "1.0.0", "demo_pkg-1.0.0.tar.gz"));
+ }
+
+ /**
+ * The fix for a real finding: the normalized package name used to be spliced straight into
+ * {@code /simple/<normalized>/}. A {@code /} inside {@code packageName} would then be
+ * indistinguishable from the real separators around it and could splice in extra path
+ * segments. Fixed by percent-encoding the normalized name via {@code encodeSegment} before
+ * it's placed in the path, so a {@code /} inside it survives only as a literal {@code %2F}
+ * within that one segment. {@code packageName} is {@code a/../b}, not a cleaner {@code ../..}:
+ * the PEP 503 normalization this method already applies collapses runs of {@code -_.} to a
+ * single {@code -} first, which would otherwise quietly absorb a pure dot-segment attempt
+ * before it ever reached the encoding this test is checking.
+ */
+ @Test
+ void fetchPackageVersionAssetTreatsASlashInThePackageNameAsLiteralNotAStructuralSeparator() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", backendUrl);
+ AtomicReference<String> observedRawPath = new AtomicReference<>();
+ backend.createContext("/", exchange -> {
+ observedRawPath.set(exchange.getRequestURI().getRawPath());
+ exchange.sendResponseHeaders(404, -1);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("pypi-repo-1", "dom", "repo", null, "a/../b",
+ "1.0.0", "a/-/b-1.0.0.tar.gz");
+
+ assertTrue(result.isEmpty());
+ assertEquals("/simple/a%2F-%2Fb/", observedRawPath.get());
+ }
+
+ private void respondWithSimpleIndex(String packageName, String filename) {
+ String html = "<a href=\"/packages/" + filename + "\">" + filename + "</a>";
+ backend.createContext("/simple/" + packageName + "/", exchange -> {
+ byte[] bytes = html.getBytes(StandardCharsets.UTF_8);
+ exchange.getResponseHeaders().set("Content-Type", "text/html");
+ exchange.sendResponseHeaders(200, bytes.length);
+ exchange.getResponseBody().write(bytes);
+ exchange.close();
+ });
+ }
+
@Test
void stopManagedContainersStopsEveryContainerThePoolIsTracking() throws Exception {
PypiserverSidecarManager manager = manager();
- seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1");
- seedPooledContainer(manager, "pypi-repo-2", "tracked-pypiserver-2");
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1", "http://127.0.0.1:1");
+ seedPooledContainer(manager, "pypi-repo-2", "tracked-pypiserver-2", "http://127.0.0.1:1");
manager.stopManagedContainers();
@@ -58,8 +420,8 @@ class PypiserverSidecarManagerTest {
}
@SuppressWarnings("unchecked")
- private static void seedPooledContainer(PypiserverSidecarManager manager, String key, String containerId)
- throws Exception {
+ private static void seedPooledContainer(PypiserverSidecarManager manager, String key, String containerId,
+ String url) throws Exception {
Field poolField = PypiserverSidecarManager.class.getDeclaredField("pool");
poolField.setAccessible(true);
PerKeyContainerPool pool = (PerKeyContainerPool) poolField.get(manager);
@@ -68,6 +430,6 @@ class PypiserverSidecarManagerTest {
containersField.setAccessible(true);
ConcurrentHashMap<String, PerKeyContainerPool.StartedContainer> containers =
(ConcurrentHashMap<String, PerKeyContainerPool.StartedContainer>) containersField.get(pool);
- containers.put(key, new PerKeyContainerPool.StartedContainer(containerId, "http://127.0.0.1:1"));
+ containers.put(key, new PerKeyContainerPool.StartedContainer(containerId, url));
}
}
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClientTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClientTest.java
index dfb4f34892..336ce72fc0 100644
--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClientTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClientTest.java
@@ -10,7 +10,9 @@ import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.io.OutputStream;
+import java.lang.reflect.Method;
import java.net.InetSocketAddress;
+import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
@@ -23,6 +25,7 @@ import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasSize;
import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.nullValue;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -358,6 +361,197 @@ class ReposiliteSidecarClientTest {
assertTrue(artifact.isEmpty());
}
+ @Test
+ void fetchPackageVersionAssetBuildsTheGavPathFromTheGroupIdArtifactIdVersionAndFilename() {
+ settingsBody.set("{\"repositories\":[{\"id\":\"dom--repo\",\"visibility\":\"PUBLIC\",\"redeployment\":false}]}");
+ server.createContext("/api/settings/domain/maven", exchange -> respond(exchange, 200, settingsBody.get()));
+ server.createContext("/dom--repo/com/example/nested/a/1.0/a-1.0.jar", exchange -> respond(exchange, 200, "jar-bytes"));
+
+ Optional<byte[]> result = client.fetchPackageVersionAsset("dom--repo", "dom", "repo", "com.example.nested",
+ "a", "1.0", "a-1.0.jar");
+
+ assertTrue(result.isPresent());
+ assertArrayEquals("jar-bytes".getBytes(StandardCharsets.UTF_8), result.get());
+ }
+
+ /**
+ * The fix for a real finding: building the request URI with a plain string concatenation
+ * would let a {@code #} in a caller-supplied asset name get interpreted as a URI fragment and
+ * silently dropped from the actual wire request, truncating it to a shorter, real filename.
+ * Checks the path the backend actually receives, not just the outcome: {@code com.sun.net.
+ * httpserver.HttpServer}'s own context matching turns out to be looser than a real HTTP
+ * server's (it matches a registered {@code .../b-1.0.jar} against a request path of
+ * {@code .../b-1.0.jar#forged} too), so asserting on the response alone wouldn't actually
+ * distinguish the fixed behavior from the bug this is checking for.
+ */
+ @Test
+ void fetchPackageVersionAssetDoesNotTreatAHashInTheAssetNameAsAUriFragment() {
+ settingsBody.set("{\"repositories\":[{\"id\":\"dom--repo\",\"visibility\":\"PUBLIC\",\"redeployment\":false}]}");
+ server.createContext("/api/settings/domain/maven", exchange -> respond(exchange, 200, settingsBody.get()));
+ AtomicReference<String> observedPath = new AtomicReference<>();
+ server.createContext("/dom--repo/com/example/b/1.0/b-1.0.jar", exchange -> {
+ observedPath.set(exchange.getRequestURI().getPath());
+ respond(exchange, 404, "");
+ });
+
+ client.fetchPackageVersionAsset("dom--repo", "dom", "repo", "com.example", "b", "1.0", "b-1.0.jar#forged");
+
+ assertThat(observedPath.get(), equalTo("/dom--repo/com/example/b/1.0/b-1.0.jar#forged"));
+ }
+
+ @Test
+ void fetchPackageVersionAssetDoesNotTreatAQuestionMarkInTheAssetNameAsAQueryString() {
+ settingsBody.set("{\"repositories\":[{\"id\":\"dom--repo\",\"visibility\":\"PUBLIC\",\"redeployment\":false}]}");
+ server.createContext("/api/settings/domain/maven", exchange -> respond(exchange, 200, settingsBody.get()));
+ AtomicReference<String> observedPath = new AtomicReference<>();
+ AtomicReference<String> observedQuery = new AtomicReference<>();
+ server.createContext("/dom--repo/com/example/b/1.0/b-1.0.jar", exchange -> {
+ observedPath.set(exchange.getRequestURI().getPath());
+ observedQuery.set(exchange.getRequestURI().getQuery());
+ respond(exchange, 404, "");
+ });
+
+ client.fetchPackageVersionAsset("dom--repo", "dom", "repo", "com.example", "b", "1.0", "b-1.0.jar?evil=1");
+
+ assertThat(observedPath.get(), equalTo("/dom--repo/com/example/b/1.0/b-1.0.jar?evil=1"));
+ assertThat(observedQuery.get(), is(nullValue()));
+ }
+
+ @Test
+ void fetchPackageVersionAssetReturnsEmptyWhenTheBackendHasNoSuchFile() {
+ settingsBody.set("{\"repositories\":[{\"id\":\"dom--repo\",\"visibility\":\"PUBLIC\",\"redeployment\":false}]}");
+ server.createContext("/api/settings/domain/maven", exchange -> respond(exchange, 200, settingsBody.get()));
+ server.createContext("/dom--repo/com/example/a/1.0/missing.jar", exchange -> respond(exchange, 404, ""));
+
+ Optional<byte[]> result = client.fetchPackageVersionAsset("dom--repo", "dom", "repo", "com.example", "a",
+ "1.0", "missing.jar");
+
+ assertTrue(result.isEmpty());
+ }
+
+ /**
+ * The fix for a real finding: building the request URI from only the base URL's scheme and
+ * authority, dropping any path it has, would break a pre-configured
+ * {@code FLOCI_SERVICES_CODEARTIFACT_MAVEN_URL} that points at Reposilite behind its own path
+ * prefix. Setup still used the full configured URL in that case, but every subsequent GET/PUT
+ * would go to the server root instead and fail.
+ */
+ @Test
+ void fetchPackageVersionAssetPreservesABaseUrlPathPrefix() {
+ when(manager.ensureReady()).thenReturn("http://127.0.0.1:" + server.getAddress().getPort() + "/reposilite-prefix");
+ settingsBody.set("{\"repositories\":[{\"id\":\"dom--repo\",\"visibility\":\"PUBLIC\",\"redeployment\":false}]}");
+ server.createContext("/reposilite-prefix/api/settings/domain/maven",
+ exchange -> respond(exchange, 200, settingsBody.get()));
+ server.createContext("/reposilite-prefix/dom--repo/com/example/a/1.0/a-1.0.jar",
+ exchange -> respond(exchange, 200, "jar-bytes"));
+
+ Optional<byte[]> result = client.fetchPackageVersionAsset("dom--repo", "dom", "repo", "com.example", "a",
+ "1.0", "a-1.0.jar");
+
+ assertTrue(result.isPresent());
+ assertArrayEquals("jar-bytes".getBytes(StandardCharsets.UTF_8), result.get());
+ }
+
+ /**
+ * The fix for a real finding on top of the previous one: decoding the configured prefix with
+ * {@code getPath()} and handing it to the URI constructor alongside the new path would
+ * re-encode the whole thing together, turning a deliberately-escaped {@code %2F} (one opaque
+ * segment containing a literal slash) back into a real {@code /}. Something in front of the
+ * sidecar that treats those two differently would then see a different path than the one that
+ * was actually configured. Checks the raw path the backend receives on the wire, since by the
+ * time {@code com.sun.net.httpserver.HttpServer} hands back a matched context, its own
+ * {@code getPath()} would already show the decoded form either way.
+ */
+ @Test
+ void fetchPackageVersionAssetPreservesAPercentEncodedCharacterInTheBaseUrlPrefix() {
+ when(manager.ensureReady())
+ .thenReturn("http://127.0.0.1:" + server.getAddress().getPort() + "/reposilite%2Fv1");
+ settingsBody.set("{\"repositories\":[{\"id\":\"dom--repo\",\"visibility\":\"PUBLIC\",\"redeployment\":false}]}");
+ server.createContext("/reposilite/v1/api/settings/domain/maven",
+ exchange -> respond(exchange, 200, settingsBody.get()));
+ AtomicReference<String> observedRawPath = new AtomicReference<>();
+ server.createContext("/reposilite/v1/dom--repo/com/example/a/1.0/a-1.0.jar", exchange -> {
+ observedRawPath.set(exchange.getRequestURI().getRawPath());
+ respond(exchange, 200, "jar-bytes");
+ });
+
+ Optional<byte[]> result = client.fetchPackageVersionAsset("dom--repo", "dom", "repo", "com.example", "a",
+ "1.0", "a-1.0.jar");
+
+ assertTrue(result.isPresent());
+ assertThat(observedRawPath.get(), equalTo("/reposilite%2Fv1/dom--repo/com/example/a/1.0/a-1.0.jar"));
+ }
+
+ /**
+ * Same class of bug as the path-prefix case above, but for the authority component: a
+ * percent-encoded character within userinfo (e.g. {@code user%2Ftenant@host}) must survive
+ * untouched. {@code getAuthority()} decodes it before the URI is rebuilt, which would send
+ * the request to a different authority than the one actually configured. Unlike the path
+ * case, this isn't observable over the wire through a live HTTP call (userinfo never reaches
+ * the server), so this calls the private helper directly via reflection.
+ */
+ @Test
+ void requestUriPreservesAPercentEncodedCharacterInTheBaseUrlAuthority() throws Exception {
+ Method requestUri = ReposiliteSidecarClient.class.getDeclaredMethod("requestUri", String.class, String.class);
+ requestUri.setAccessible(true);
+
+ URI uri = (URI) requestUri.invoke(null, "http://user%[email protected]", "/dom--repo/a-1.0.jar");
+
+ assertThat(uri.getRawAuthority(), equalTo("user%[email protected]"));
+ }
+
+ /**
+ * The fix for a real finding: {@code fetchPackageVersionAsset} used to build its GAV path
+ * with a plain {@code +} concatenation of four independently caller-supplied coordinates. A
+ * {@code /} inside {@code assetName} would then be indistinguishable from one of the real
+ * separators between groupId/artifactId/version/filename, letting it splice in extra path
+ * segments. Fixed by percent-encoding each coordinate on its own before joining them, so a
+ * {@code /} inside one of them survives only as a literal {@code %2F} within one segment.
+ */
+ @Test
+ void fetchPackageVersionAssetTreatsASlashInTheAssetNameAsLiteralNotAStructuralSeparator() {
+ settingsBody.set("{\"repositories\":[{\"id\":\"dom--repo\",\"visibility\":\"PUBLIC\",\"redeployment\":false}]}");
+ server.createContext("/api/settings/domain/maven", exchange -> respond(exchange, 200, settingsBody.get()));
+ AtomicReference<String> observedRawPath = new AtomicReference<>();
+ server.createContext("/", exchange -> {
+ if (exchange.getRequestURI().getRawPath().startsWith("/dom--repo/")) {
+ observedRawPath.set(exchange.getRequestURI().getRawPath());
+ }
+ respond(exchange, 404, "");
+ });
+
+ Optional<byte[]> result = client.fetchPackageVersionAsset("dom--repo", "dom", "repo", "com.example", "a",
+ "1.0", "../other.jar");
+
+ assertTrue(result.isEmpty());
+ assertThat(observedRawPath.get(), equalTo("/dom--repo/com/example/a/1.0/..%2Fother.jar"));
+ }
+
+ /**
+ * Same finding, the other half of it: a coordinate that is exactly {@code ..} (no slash at
+ * all, so the fix above wouldn't catch it on its own) would sit as a real dot-segment between
+ * two of this method's own {@code /} separators. Fixed by also escaping the dots themselves
+ * whenever a coordinate's entire value is {@code .} or {@code ..}.
+ */
+ @Test
+ void fetchPackageVersionAssetTreatsADotDotVersionAsLiteralNotADotSegment() {
+ settingsBody.set("{\"repositories\":[{\"id\":\"dom--repo\",\"visibility\":\"PUBLIC\",\"redeployment\":false}]}");
+ server.createContext("/api/settings/domain/maven", exchange -> respond(exchange, 200, settingsBody.get()));
+ AtomicReference<String> observedRawPath = new AtomicReference<>();
+ server.createContext("/", exchange -> {
+ if (exchange.getRequestURI().getRawPath().startsWith("/dom--repo/")) {
+ observedRawPath.set(exchange.getRequestURI().getRawPath());
+ }
+ respond(exchange, 404, "");
+ });
+
+ Optional<byte[]> result = client.fetchPackageVersionAsset("dom--repo", "dom", "repo", "com.example", "a",
+ "..", "a-1.0.jar");
+
+ assertTrue(result.isEmpty());
+ assertThat(observedRawPath.get(), equalTo("/dom--repo/com/example/a/%2E%2E/a-1.0.jar"));
+ }
+
@Test
void artifactExistsReflectsAHeadResponse() {
server.createContext("/dom--repo/present.jar", exchange -> respond(exchange, 200, ""));
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/VerdaccioSidecarManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/VerdaccioSidecarManagerTest.java
index a5d3ca7f3f..752f3f8276 100644
--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/VerdaccioSidecarManagerTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/VerdaccioSidecarManagerTest.java
@@ -1,16 +1,27 @@
package io.github.hectorvent.floci.services.codeartifact;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.sun.net.httpserver.HttpServer;
import io.github.hectorvent.floci.config.EmulatorConfig;
import io.github.hectorvent.floci.core.common.ContainerTeardown;
import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
import io.github.hectorvent.floci.core.common.docker.PerKeyContainerPool;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Field;
+import java.net.InetSocketAddress;
+import java.nio.charset.StandardCharsets;
+import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicReference;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoInteractions;
@@ -33,16 +44,191 @@ class VerdaccioSidecarManagerTest {
private final ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
private final EmulatorConfig config = mock(EmulatorConfig.class);
+ private HttpServer backend;
+ private String backendUrl;
+
+ @BeforeEach
+ void setUpBackend() throws Exception {
+ backend = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
+ backend.createContext("/-/ping", exchange -> {
+ exchange.sendResponseHeaders(200, -1);
+ exchange.close();
+ });
+ backend.start();
+ backendUrl = "http://127.0.0.1:" + backend.getAddress().getPort();
+ }
+
+ @AfterEach
+ void tearDownBackend() {
+ backend.stop(0);
+ }
+
@Test
void implementsContainerTeardownSoStateResetAndNukeCanFindIt() {
assertInstanceOf(ContainerTeardown.class, manager());
}
+ @Test
+ void fetchPackageVersionAssetGetsTheUnscopedTarballPath() throws Exception {
+ VerdaccioSidecarManager manager = manager();
+ seedPooledContainer(manager, "npm-repo-1", "tracked-verdaccio-1", backendUrl);
+ respondWithMetadata("/demo-pkg", "1.0.0", "/demo-pkg/-/demo-pkg-1.0.0.tgz");
+ backend.createContext("/demo-pkg/-/demo-pkg-1.0.0.tgz", exchange -> {
+ byte[] body = "tarball bytes".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("npm-repo-1", "dom", "repo", null, "demo-pkg",
+ "1.0.0", "demo-pkg-1.0.0.tgz");
+
+ assertTrue(result.isPresent());
+ assertEquals("tarball bytes", new String(result.get(), StandardCharsets.UTF_8));
+ }
+
+ /**
+ * Confirmed against a live Verdaccio container: a scoped package's tarball path keeps the
+ * {@code @scope/} segment in the URL even though the filename itself never includes it
+ * (unlike the local tarball the npm CLI builds when packing, which does).
+ */
+ @Test
+ void fetchPackageVersionAssetGetsTheScopedTarballPathWithTheFilenameUnprefixed() throws Exception {
+ VerdaccioSidecarManager manager = manager();
+ seedPooledContainer(manager, "npm-repo-1", "tracked-verdaccio-1", backendUrl);
+ respondWithMetadata("/@myscope/demo-pkg", "2.0.0", "/@myscope/demo-pkg/-/demo-pkg-2.0.0.tgz");
+ backend.createContext("/@myscope/demo-pkg/-/demo-pkg-2.0.0.tgz", exchange -> {
+ byte[] body = "scoped tarball bytes".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("npm-repo-1", "dom", "repo", "myscope",
+ "demo-pkg", "2.0.0", "demo-pkg-2.0.0.tgz");
+
+ assertTrue(result.isPresent());
+ assertEquals("scoped tarball bytes", new String(result.get(), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void fetchPackageVersionAssetReturnsEmptyWhenThePackageWasNeverPublished() throws Exception {
+ VerdaccioSidecarManager manager = manager();
+ seedPooledContainer(manager, "npm-repo-1", "tracked-verdaccio-1", backendUrl);
+ backend.createContext("/demo-pkg", exchange -> {
+ exchange.sendResponseHeaders(404, -1);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("npm-repo-1", "dom", "repo", null, "demo-pkg",
+ "1.0.0", "missing.tgz");
+
+ assertTrue(result.isEmpty());
+ }
+
+ /**
+ * The fix for a real finding: without checking the package's own metadata first, a caller
+ * naming a real tarball filename under a version that was never actually published to it
+ * would get that file's bytes back anyway, since the tarball route is keyed by filename alone.
+ */
+ @Test
+ void fetchPackageVersionAssetReturnsEmptyWhenTheVersionWasNeverPublishedEvenIfTheFilenameExists() throws Exception {
+ VerdaccioSidecarManager manager = manager();
+ seedPooledContainer(manager, "npm-repo-1", "tracked-verdaccio-1", backendUrl);
+ respondWithMetadata("/demo-pkg", "1.0.0", "/demo-pkg/-/demo-pkg-1.0.0.tgz");
+ backend.createContext("/demo-pkg/-/demo-pkg-9.9.9.tgz", exchange -> {
+ byte[] body = "should never be served".getBytes(StandardCharsets.UTF_8);
+ exchange.sendResponseHeaders(200, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("npm-repo-1", "dom", "repo", null, "demo-pkg",
+ "9.9.9", "demo-pkg-9.9.9.tgz");
+
+ assertTrue(result.isEmpty());
+ }
+
+ @Test
+ void fetchPackageVersionAssetThrowsRatherThanReportingNotFoundOnASidecarServerError() throws Exception {
+ VerdaccioSidecarManager manager = manager();
+ seedPooledContainer(manager, "npm-repo-1", "tracked-verdaccio-1", backendUrl);
+ backend.createContext("/demo-pkg", exchange -> {
+ exchange.sendResponseHeaders(500, -1);
+ exchange.close();
+ });
+
+ assertThrows(IllegalStateException.class, () -> manager.fetchPackageVersionAsset("npm-repo-1", "dom", "repo",
+ null, "demo-pkg", "1.0.0", "demo-pkg-1.0.0.tgz"));
+ }
+
+ /**
+ * The fix for a real finding: {@code packagePath} used to splice the npm scope straight into
+ * the URL. A {@code /} inside {@code namespace} (the scope) would then be indistinguishable
+ * from one of the real separators between the scope, the package name, and {@code -}, letting
+ * it splice in extra path segments. Fixed by percent-encoding each coordinate on its own
+ * before joining them, so a {@code /} inside one of them survives only as a literal
+ * {@code %2F} within one segment.
+ */
+ @Test
+ void fetchPackageVersionAssetTreatsASlashInTheNamespaceAsLiteralNotAStructuralSeparator() throws Exception {
+ VerdaccioSidecarManager manager = manager();
+ seedPooledContainer(manager, "npm-repo-1", "tracked-verdaccio-1", backendUrl);
+ AtomicReference<String> observedRawPath = new AtomicReference<>();
+ backend.createContext("/", exchange -> {
+ observedRawPath.set(exchange.getRequestURI().getRawPath());
+ exchange.sendResponseHeaders(404, -1);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("npm-repo-1", "dom", "repo", "../../escape",
+ "demo-pkg", "1.0.0", "demo-pkg-1.0.0.tgz");
+
+ assertTrue(result.isEmpty());
+ assertEquals("/@..%2F..%2Fescape/demo-pkg", observedRawPath.get());
+ }
+
+ /**
+ * Same finding, the other half of it: a coordinate that is exactly {@code ..} (no slash at
+ * all, so the fix above wouldn't catch it on its own) would sit as a real dot-segment between
+ * two of this method's own {@code /} separators. Fixed by also escaping the dots themselves
+ * whenever a coordinate's entire value is {@code .} or {@code ..}.
+ */
+ @Test
+ void fetchPackageVersionAssetTreatsADotDotPackageNameAsLiteralNotADotSegment() throws Exception {
+ VerdaccioSidecarManager manager = manager();
+ seedPooledContainer(manager, "npm-repo-1", "tracked-verdaccio-1", backendUrl);
+ AtomicReference<String> observedRawPath = new AtomicReference<>();
+ backend.createContext("/", exchange -> {
+ observedRawPath.set(exchange.getRequestURI().getRawPath());
+ exchange.sendResponseHeaders(404, -1);
+ exchange.close();
+ });
+
+ Optional<byte[]> result = manager.fetchPackageVersionAsset("npm-repo-1", "dom", "repo", null, "..",
+ "1.0.0", "demo-pkg-1.0.0.tgz");
+
+ assertTrue(result.isEmpty());
+ assertEquals("/%2E%2E", observedRawPath.get());
+ }
+
+ private void respondWithMetadata(String packagePath, String version, String tarballPath) {
+ String body = "{\"versions\":{\"" + version + "\":{\"dist\":{\"tarball\":\"http://ignored" + tarballPath
+ + "\"}}}}";
+ backend.createContext(packagePath, exchange -> {
+ byte[] bytes = body.getBytes(StandardCharsets.UTF_8);
+ exchange.getResponseHeaders().set("Content-Type", "application/json");
+ exchange.sendResponseHeaders(200, bytes.length);
+ exchange.getResponseBody().write(bytes);
+ exchange.close();
+ });
+ }
+
@Test
void stopManagedContainersStopsEveryContainerThePoolIsTracking() throws Exception {
VerdaccioSidecarManager manager = manager();
- seedPooledContainer(manager, "npm-repo-1", "tracked-verdaccio-1");
- seedPooledContainer(manager, "npm-repo-2", "tracked-verdaccio-2");
+ seedPooledContainer(manager, "npm-repo-1", "tracked-verdaccio-1", "http://127.0.0.1:1");
+ seedPooledContainer(manager, "npm-repo-2", "tracked-verdaccio-2", "http://127.0.0.1:1");
manager.stopManagedContainers();
@@ -60,12 +246,12 @@ class VerdaccioSidecarManagerTest {
}
private VerdaccioSidecarManager manager() {
- return new VerdaccioSidecarManager(containerBuilder, lifecycleManager, config);
+ return new VerdaccioSidecarManager(containerBuilder, lifecycleManager, config, new ObjectMapper());
}
@SuppressWarnings("unchecked")
- private static void seedPooledContainer(VerdaccioSidecarManager manager, String key, String containerId)
- throws Exception {
+ private static void seedPooledContainer(VerdaccioSidecarManager manager, String key, String containerId,
+ String url) throws Exception {
Field poolField = VerdaccioSidecarManager.class.getDeclaredField("pool");
poolField.setAccessible(true);
PerKeyContainerPool pool = (PerKeyContainerPool) poolField.get(manager);
@@ -74,6 +260,6 @@ class VerdaccioSidecarManagerTest {
containersField.setAccessible(true);
ConcurrentHashMap<String, PerKeyContainerPool.StartedContainer> containers =
(ConcurrentHashMap<String, PerKeyContainerPool.StartedContainer>) containersField.get(pool);
- containers.put(key, new PerKeyContainerPool.StartedContainer(containerId, "http://127.0.0.1:1"));
+ containers.put(key, new PerKeyContainerPool.StartedContainer(containerId, url));
}
}