Skip to content

armgit / floci

Public Imported from GitHub
0

Light, fluffy, and always free - The AWS Local Emulator alternative

feat(codeartifact): add PyPI format support via pypiserver sidecar (#4998)

Priamo Germosenfca7d87f2d58
diff --git a/docs/services/codeartifact.md b/docs/services/codeartifact.md
index 58128c4ab4..92e265f8aa 100644
--- a/docs/services/codeartifact.md
+++ b/docs/services/codeartifact.md
@@ -7,10 +7,10 @@
Floci supports the CodeArtifact control plane: domains, repositories, resource policies, tags,
and public upstream (external) connections. Package publish/fetch through the CodeArtifact API
itself is implemented for the `generic` format only, matching AWS's own restriction that
-`PublishPackageVersion` accepts only `generic`. The `maven` and `npm` formats are each served
-through their own real package-manager-protocol proxy (`mvn`/Gradle and `npm`/`yarn`/`pnpm`
-publish and resolve all work against the URL `GetRepositoryEndpoint` returns); the remaining
-formats (PyPI, NuGet, etc.) have no real proxy behind them yet.
+`PublishPackageVersion` accepts only `generic`. The `maven`, `npm`, and `pypi` formats are each
+served through their own real package-manager-protocol proxy (`mvn`/Gradle, `npm`/`yarn`/`pnpm`,
+and `pip`/`twine` publish and resolve all work against the URL `GetRepositoryEndpoint` returns);
+the remaining formats (NuGet, etc.) have no real proxy behind them yet.
## Supported Actions
@@ -141,6 +141,45 @@ client-unreachable address; without this, `npm install` would try to fetch the t
from an address it cannot reach. One config knob, `FLOCI_SERVICES_CODEARTIFACT_NPM_IMAGE`, pins
the image version.
+## The pypi repository endpoint
+
+`GetRepositoryEndpoint` for `format=pypi` returns `http://localhost:4566/codeartifact/pypi/<domain>/<repository>/`.
+Real pip and twine can install from and publish to that URL directly; it speaks the real PyPI
+simple-repository protocol (simple index, package download, twine's multipart upload), proxied
+straight through to a real [pypiserver](https://pypi.org/project/pypiserver/) instance rather than
+reimplemented. Every request needs a token from `GetAuthorizationToken` scoped to the domain being
+accessed, sent as HTTP Basic with the token as the password (any username) - AWS documents `pip`'s
+index URL embedding `aws:$CODEARTIFACT_AUTH_TOKEN@` and twine's `.pypirc`/environment variables as
+`username=aws, password=<token>`
+([configure pip](https://docs.aws.amazon.com/codeartifact/latest/ug/python-configure-pip.html),
+[configure twine](https://docs.aws.amazon.com/codeartifact/latest/ug/python-configure-twine.html)),
+the same Basic convention Maven's HTTP wagon uses, not npm's Bearer-only `_authToken`; a Bearer
+token is also accepted, matching npm, since accepting both costs nothing. A missing, invalid,
+expired, or wrong-domain token gets a 401 challenging `Basic`.
+
+Like Verdaccio, pypiserver has no native concept of multiple named indexes inside one instance, so
+each CodeArtifact repository gets its own pypiserver container instead of sharing one, started
+lazily on first use and identified internally by a fresh id generated at `CreateRepository` time,
+so a repository deleted and recreated under the same name never inherits the previous one's
+packages. `DeleteRepository` stops and removes that repository's container immediately, the same
+as npm. Unlike Verdaccio, no public-URL environment variable is needed: pypiserver's simple-index
+responses link to package files with a root-relative path (`/packages/<file>`), which pip and
+twine resolve against whatever host they actually connected to, not an address the container
+returns itself.
+
+Unlike Reposilite (`redeployment: false`) and Verdaccio (which rejects a duplicate publish
+natively, matching real npm), pypiserver has no overwrite protection of its own: publishing the
+same package name, version, and filename twice with different content silently replaces the file
+(confirmed by hand against a live instance with both twine and a raw multipart upload). Floci's
+proxy closes this gap itself, matching AWS's own documented behavior for every format
+("Overwriting package assets" in CodeArtifact's packages overview): before forwarding an upload, it
+checks the target repository's own simple index for the exact filename being uploaded; if the
+filename doesn't exist yet, the upload proceeds, if it exists with byte-identical content the
+upload succeeds without touching pypiserver again (idempotent, so a client's retry after a dropped
+response is never a spurious conflict), and only a name collision with genuinely different content
+gets rejected with 409. One config knob, `FLOCI_SERVICES_CODEARTIFACT_PYPI_IMAGE`, pins the image
+version.
+
## AWS-compatible failures
Domain and repository names, tags, pagination, duplicate names, missing upstreams, policy-revision
@@ -175,34 +214,66 @@ state.
`DisposePackageVersions`, and `UpdatePackageVersionsStatus` don't exist yet; the only way to
move a version from `Unfinished` to `Published` today is a follow-up `PublishPackageVersion`
call that omits the `unfinished` flag.
-- **The 5 GB asset size quota is nominal.** `PublishPackageVersion` and the Maven repository
- endpoint both receive the request body as a single byte array before Floci ever checks its
- length, so a request already large enough to exhaust available heap fails before the quota check
- runs. The rejection (`ServiceQuotaExceededException` from `PublishPackageVersion`, HTTP 413 from
- the Maven endpoint) is correct for anything that does fit in memory; it is not itself a streaming
- size limit.
-- **Maven artifacts and npm packages do not survive a Floci restart, even under persistent
- storage.** The Reposilite instance and every per-repository Verdaccio container have no volume
- attached and are removed on shutdown along with everything published to them. CodeArtifact
- repository/domain metadata (including each format's stored sidecar container id) survives a
- restart the same way any other Floci state does under persistent storage mode; the artifacts and
- packages themselves do not, so the first request after a restart re-provisions empty storage and
- returns 404 for anything published before the restart.
+- **The 5 GB asset size quota is nominal.** `PublishPackageVersion` and the Maven and pypi
+ repository endpoints all receive the request body as a single byte array before Floci ever
+ checks its length, so a request already large enough to exhaust available heap fails before the
+ quota check runs. The rejection (`ServiceQuotaExceededException` from `PublishPackageVersion`,
+ HTTP 413 from the Maven and pypi endpoints) is correct for anything that does fit in memory; it
+ is not itself a streaming size limit.
+- **Maven artifacts, npm packages, and pypi packages do not survive a Floci restart, even under
+ persistent storage.** The Reposilite instance and every per-repository Verdaccio and pypiserver
+ container have no volume attached and are removed on shutdown along with everything published to
+ them. CodeArtifact repository/domain metadata (including each format's stored sidecar container
+ id) survives a restart the same way any other Floci state does under persistent storage mode; the
+ artifacts and packages themselves do not, so the first request after a restart re-provisions
+ empty storage and returns 404 for anything published before the restart.
- **`GetAuthorizationToken` tokens are not revocable and are not tied to any IAM identity.** Real
CodeArtifact tokens are scoped to the calling principal's permissions; Floci's are scoped only to
the domain named in the request; anyone who obtains one keeps the same domain-scoped access for
its full lifetime.
-- **Neither the Maven nor the npm repository endpoint resolves through upstream repositories or
+- **The Maven, npm, and pypi sidecar containers have no authentication of their own, and anything
+ else sharing their Docker network can reach them directly on their container port.** Real
+ CodeArtifact's token check happens once, at Floci's own proxy layer (`CodeArtifactMavenController`/
+ the npm data plane/`CodeArtifactPypiController`), before a request ever reaches Reposilite,
+ Verdaccio, or pypiserver; those containers' own auth is deliberately disabled since the proxy is
+ meant to be the only path in. Binding the published host port to loopback only keeps an outside
+ host off that port; it does not stop another container already on the same configured Docker
+ network (for example a Lambda, Batch, or ECS workload container) from reaching the sidecar on its
+ container-internal address, bypassing the proxy's token check entirely. Closing this for real
+ would mean isolating each sidecar on its own Docker network reachable only from Floci's own
+ process, a change that would need to apply uniformly across all three sidecar-backed formats, not
+ a one-off fix to any single proxy.
+- **Neither the Maven, npm, nor pypi repository endpoint resolves through upstream repositories or
external connections.** On real CodeArtifact, a repository with another repository configured as
an upstream (`UpdateRepository`'s `upstreams`) or with an `AssociateExternalConnection` to a
- public repository (`public:maven-central`, `public:npmjs`, etc.) serves packages from those
- sources too, not just its own. Floci's proxies only ever look up the repository's own backing
- storage: a package that exists solely in an upstream, or only through an external connection,
- returns 404 through a repository that has it configured as one.
+ public repository (`public:maven-central`, `public:npmjs`, `public:pypi`, etc.) serves packages
+ from those sources too, not just its own. Floci's proxies only ever look up the repository's own
+ backing storage: a package that exists solely in an upstream, or only through an external
+ connection, returns 404 through a repository that has it configured as one; the pypi proxy's
+ pypiserver container is also started with `--disable-fallback` specifically so it never silently
+ redirects a missing package to the real, public PyPI.
- **The npm repository endpoint does not enforce the 5 GB asset size quota.** Unlike
`PublishPackageVersion` and the Maven endpoint, it streams the request straight through to the
backing Verdaccio container rather than buffering it first, so there is nowhere in the request
path to check a byte count against the quota before forwarding it.
+- **The pypi repository endpoint only serves the HTML Simple Repository API, not the PEP 691/700
+ JSON variant.** AWS documents both: pip/uv can request
+ `application/vnd.pypi.simple.v1+json` from `/simple/<project>/` and get a JSON index with
+ per-file hashes, `requires-python`, and `upload-time`. Floci's proxy only ever forwards
+ pypiserver's native HTML response regardless of the client's `Accept` header, since pypiserver
+ itself has no JSON index support to pass through. Low practical risk in itself, since PEP 691
+ requires clients to accept HTML as a fallback and pip/uv both do, but it is a real gap against
+ AWS's documented surface, not just a quirk of the sidecar.
+- **`GetPackageVersionAsset` and `DescribePackageVersion` can't see Maven-, npm-, or pypi-uploaded
+ artifacts.** Those three formats' repository endpoints are deliberately metadata-free passthroughs
+ straight to their sidecar container (Reposilite/Verdaccio/pypiserver) and never create the
+ generic-format package-version record these two actions depend on
+ (`CodeArtifactService.getPackageVersionAsset`/`describePackageVersion` only ever look up
+ `packageVersions.getForAccount`, the record only the `generic` format's `PublishPackageVersion`
+ flow creates). Real CodeArtifact's `GetPackageVersionAsset` works uniformly across every format.
+ The artifact is really there and downloadable through that format's own wire endpoint
+ (`mvn dependency:get`, `npm install`, `pip install`, or a direct `curl`/`twine`); only the
+ CodeArtifact-native asset-get/describe calls 404 for it.
See the [CodeArtifact API Reference](https://docs.aws.amazon.com/codeartifact/latest/APIReference/Welcome.html).
@@ -215,3 +286,4 @@ See the [CodeArtifact API Reference](https://docs.aws.amazon.com/codeartifact/la
| `FLOCI_SERVICES_CODEARTIFACT_MAVEN_URL` | unset | When set, use this URL and skip Reposilite container management |
| `FLOCI_SERVICES_CODEARTIFACT_MAVEN_TOKEN` | unset | `name:secret` access token for a pre-configured `MAVEN_URL` |
| `FLOCI_SERVICES_CODEARTIFACT_NPM_IMAGE` | `verdaccio/verdaccio:6.10.4` | Verdaccio image used to serve the `npm` format |
+| `FLOCI_SERVICES_CODEARTIFACT_PYPI_IMAGE` | `pypiserver/pypiserver:v2.4.2` | pypiserver image used to serve the `pypi` format |
diff --git a/docs/services/index.md b/docs/services/index.md
index d49be6ea62..c3e8a875fa 100644
--- a/docs/services/index.md
+++ b/docs/services/index.md
@@ -121,7 +121,7 @@ Operation counts are exact. For dispatch-table services (Query and JSON 1.1) eac
| [AWS Backup](backup.md) | `/backup-vaults/*`, `/backup/plans/*`, `/backup-jobs/*`, `/supported-resource-types` | REST JSON | 20 |
| [AWS FIS](fis.md) | `/experimentTemplates/*`, `/experiments/*`, `/actions/*`, `/targetResourceTypes/*`, `/safetyLevers/*`, `/tags/*` | REST JSON | 26 |
| [CodeGuru Reviewer](codegurureviewer.md) | `/associations`, `/associations/{associationArn}`, `/tags/*` | REST JSON | 7 |
-| [CodeArtifact](codeartifact.md) | `/v1/domain*`, `/v1/repository*`, `/v1/package/version*`, `/v1/tag*`, `/v1/authorization-token`, `/codeartifact/maven/*`, `/codeartifact/npm/*` | REST JSON | 26 |
+| [CodeArtifact](codeartifact.md) | `/v1/domain*`, `/v1/repository*`, `/v1/package/version*`, `/v1/tag*`, `/v1/authorization-token`, `/codeartifact/maven/*`, `/codeartifact/npm/*`, `/codeartifact/pypi/*` | REST JSON | 26 |
| [CloudFront](cloudfront.md) | `/2020-05-31/distribution/*`, `/2020-05-31/cache-policy/*`, `/2020-05-31/function/*` | REST XML | 50 |
| [Route53](route53.md) | `/2013-04-01/hostedzone/*`, `/2013-04-01/healthcheck/*`, `/2013-04-01/change/*` | REST XML | 25 |
| [Route 53 Resolver](route53resolver.md) | `POST /` + `X-Amz-Target: Route53Resolver.*` | JSON 1.1 | 18 |
diff --git a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
index 20e1b9acb2..7b749edb00 100644
--- a/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
+++ b/src/main/java/io/github/hectorvent/floci/config/EmulatorConfig.java
@@ -876,6 +876,15 @@ public interface EmulatorConfig {
*/
@WithDefault("verdaccio/verdaccio:6.10.4")
String npmImage();
+
+ /**
+ * Image used for the per-repository pypiserver container backing the {@code pypi} format.
+ * Same reasoning as {@link #npmImage()}: pypiserver has no native concept of multiple
+ * named indexes inside one instance either, so this gets one container per CodeArtifact
+ * repository too, and there is no single external instance a URL override could name.
+ */
+ @WithDefault("pypiserver/pypiserver:v2.4.2")
+ String pypiImage();
}
interface ConnectServiceConfig {
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/MultipartFormParser.java b/src/main/java/io/github/hectorvent/floci/core/common/MultipartFormParser.java
new file mode 100644
index 0000000000..2c5fb1ee04
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/core/common/MultipartFormParser.java
@@ -0,0 +1,187 @@
+package io.github.hectorvent.floci.core.common;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Parses a {@code multipart/form-data} body into its plain text fields and, at most, one file
+ * part, without RESTEasy's {@code MultipartFormDataInput}: that reads reflectively in a way that
+ * needs its own native-image registration per caller, where this is a handful of byte-array
+ * scans. Pulled out of {@code S3Controller}'s presigned-POST handling once
+ * {@code CodeArtifactPypiController} needed the identical parsing for a twine upload's own
+ * multipart body.
+ */
+public final class MultipartFormParser {
+
+ private MultipartFormParser() {
+ }
+
+ /** One named field's value, or the one file part carrying a {@code filename} parameter. */
+ public record FilePart(String fieldName, String filename, String contentType, byte[] content) {}
+
+ /** {@code fields} excludes the file part; {@code file} is empty when the body carried none. */
+ public record ParsedForm(Map<String, String> fields, Optional<FilePart> file) {}
+
+ /** The {@code boundary} parameter of a {@code multipart/form-data} Content-Type, if present. */
+ public static Optional<String> extractBoundary(String contentType) {
+ if (contentType == null) {
+ return Optional.empty();
+ }
+ for (String part : contentType.split(";")) {
+ String trimmed = part.trim();
+ if (trimmed.toLowerCase(Locale.ROOT).startsWith("boundary=")) {
+ String boundary = trimmed.substring("boundary=".length()).trim();
+ if (boundary.startsWith("\"") && boundary.endsWith("\"")) {
+ boundary = boundary.substring(1, boundary.length() - 1);
+ }
+ return Optional.of(boundary);
+ }
+ }
+ return Optional.empty();
+ }
+
+ public static ParsedForm parse(byte[] body, String boundary) {
+ Map<String, String> fields = new LinkedHashMap<>();
+ FilePart file = null;
+
+ byte[] boundaryBytes = ("--" + boundary).getBytes(StandardCharsets.UTF_8);
+ for (byte[] part : splitParts(body, boundaryBytes)) {
+ int headerEnd = indexOfDoubleNewline(part);
+ if (headerEnd < 0) {
+ continue;
+ }
+ String headers = new String(part, 0, headerEnd, StandardCharsets.UTF_8);
+ int bodyStart = headerEnd + 4; // skip \r\n\r\n
+ byte[] partBody = Arrays.copyOfRange(part, bodyStart, part.length);
+
+ // Trim the trailing CRLF the boundary split leaves on every part body.
+ if (partBody.length >= 2
+ && partBody[partBody.length - 2] == '\r'
+ && partBody[partBody.length - 1] == '\n') {
+ partBody = Arrays.copyOf(partBody, partBody.length - 2);
+ }
+
+ String disposition = extractHeaderValue(headers, "Content-Disposition");
+ if (disposition == null) {
+ continue;
+ }
+ String fieldName = extractDispositionParam(disposition, "name");
+ if (fieldName == null) {
+ continue;
+ }
+
+ String filename = extractDispositionParam(disposition, "filename");
+ if (filename != null) {
+ String partContentType = extractHeaderValue(headers, "Content-Type");
+ file = new FilePart(fieldName, filename,
+ partContentType == null ? null : partContentType.trim(), partBody);
+ } else {
+ fields.put(fieldName, new String(partBody, StandardCharsets.UTF_8));
+ }
+ }
+ return new ParsedForm(fields, Optional.ofNullable(file));
+ }
+
+ private static List<byte[]> splitParts(byte[] body, byte[] boundary) {
+ List<byte[]> parts = new ArrayList<>();
+ int pos = indexOf(body, boundary, 0);
+ if (pos < 0) {
+ return parts;
+ }
+ pos += boundary.length;
+ if (pos < body.length - 1 && body[pos] == '-' && body[pos + 1] == '-') {
+ return parts; // closing boundary immediately, no parts
+ }
+ if (pos < body.length - 1 && body[pos] == '\r' && body[pos + 1] == '\n') {
+ pos += 2;
+ }
+
+ while (pos < body.length) {
+ int nextBoundary = indexOf(body, boundary, pos);
+ if (nextBoundary < 0) {
+ break;
+ }
+ parts.add(Arrays.copyOfRange(body, pos, nextBoundary));
+ pos = nextBoundary + boundary.length;
+ if (pos < body.length - 1 && body[pos] == '-' && body[pos + 1] == '-') {
+ break;
+ }
+ if (pos < body.length - 1 && body[pos] == '\r' && body[pos + 1] == '\n') {
+ pos += 2;
+ }
+ }
+ return parts;
+ }
+
+ private static int indexOf(byte[] data, byte[] pattern, int fromIndex) {
+ outer:
+ for (int i = fromIndex; i <= data.length - pattern.length; i++) {
+ for (int j = 0; j < pattern.length; j++) {
+ if (data[i + j] != pattern[j]) {
+ continue outer;
+ }
+ }
+ return i;
+ }
+ return -1;
+ }
+
+ private static int indexOfDoubleNewline(byte[] data) {
+ for (int i = 0; i < data.length - 3; i++) {
+ if (data[i] == '\r' && data[i + 1] == '\n' && data[i + 2] == '\r' && data[i + 3] == '\n') {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ private static String extractHeaderValue(String headers, String headerName) {
+ String lowerHeaders = headers.toLowerCase(Locale.ROOT);
+ String lowerName = headerName.toLowerCase(Locale.ROOT) + ":";
+ int idx = lowerHeaders.indexOf(lowerName);
+ if (idx < 0) {
+ return null;
+ }
+ int valueStart = idx + lowerName.length();
+ int lineEnd = headers.indexOf('\r', valueStart);
+ if (lineEnd < 0) {
+ lineEnd = headers.indexOf('\n', valueStart);
+ }
+ if (lineEnd < 0) {
+ lineEnd = headers.length();
+ }
+ return headers.substring(valueStart, lineEnd).trim();
+ }
+
+ private static String extractDispositionParam(String disposition, String paramName) {
+ String search = paramName + "=";
+ int idx = disposition.indexOf(search);
+ if (idx < 0) {
+ return null;
+ }
+ int valueStart = idx + search.length();
+ if (valueStart >= disposition.length()) {
+ return null;
+ }
+ if (disposition.charAt(valueStart) == '"') {
+ valueStart++;
+ int valueEnd = disposition.indexOf('"', valueStart);
+ if (valueEnd < 0) {
+ return disposition.substring(valueStart);
+ }
+ return disposition.substring(valueStart, valueEnd);
+ } else {
+ int valueEnd = disposition.indexOf(';', valueStart);
+ if (valueEnd < 0) {
+ valueEnd = disposition.length();
+ }
+ return disposition.substring(valueStart, valueEnd).trim();
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/core/common/ResolvedServiceCatalog.java b/src/main/java/io/github/hectorvent/floci/core/common/ResolvedServiceCatalog.java
index 41892d388e..48e5bbf4b1 100644
--- a/src/main/java/io/github/hectorvent/floci/core/common/ResolvedServiceCatalog.java
+++ b/src/main/java/io/github/hectorvent/floci/core/common/ResolvedServiceCatalog.java
@@ -9,6 +9,8 @@ import io.github.hectorvent.floci.services.appconfig.AppConfigDataController;
import io.github.hectorvent.floci.services.batch.BatchController;
import io.github.hectorvent.floci.services.bedrock.BedrockController;
import io.github.hectorvent.floci.services.bedrockruntime.BedrockRuntimeController;
+import io.github.hectorvent.floci.services.codeartifact.CodeArtifactMavenController;
+import io.github.hectorvent.floci.services.codeartifact.CodeArtifactPypiController;
import io.github.hectorvent.floci.services.cognito.CognitoOAuthController;
import io.github.hectorvent.floci.services.cognito.CognitoWellKnownController;
import io.github.hectorvent.floci.services.eks.EksController;
@@ -743,7 +745,9 @@ public class ResolvedServiceCatalog {
"codeartifact", config.storage().mode(), 5000L, null, ServiceProtocol.REST_JSON,
protocols(ServiceProtocol.REST_JSON),
Set.of(), Set.of("codeartifact"), Set.of(),
- Set.of(io.github.hectorvent.floci.services.codeartifact.CodeArtifactController.class)),
+ Set.of(io.github.hectorvent.floci.services.codeartifact.CodeArtifactController.class,
+ CodeArtifactMavenController.class,
+ CodeArtifactPypiController.class)),
descriptor("verifiedpermissions", "verifiedpermissions", config.services().verifiedpermissions().enabled(), true,
"verifiedpermissions", config.storage().mode(), 5000L, null, ServiceProtocol.JSON,
protocols(ServiceProtocol.JSON),
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenController.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenController.java
index 635f5d6926..bb73185e8a 100644
--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenController.java
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenController.java
@@ -18,10 +18,6 @@ import jakarta.ws.rs.core.HttpHeaders;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.core.Response;
-import java.nio.charset.StandardCharsets;
-import java.util.Base64;
-import java.util.Optional;
-
/**
* Serves the real Maven repository wire protocol behind the URL {@code GetRepositoryEndpoint}
* returns for the {@code maven} format: plain GET/PUT/HEAD over a GAV path, not a CodeArtifact
@@ -43,9 +39,6 @@ import java.util.Optional;
@Path("/codeartifact/maven/{domain}/{repository}/{gav:.+}")
public class CodeArtifactMavenController {
- private static final String BEARER_PREFIX = "Bearer ";
- private static final String BASIC_PREFIX = "Basic ";
-
private final CodeArtifactService service;
private final ReposiliteSidecarClient reposilite;
@@ -96,7 +89,7 @@ public class CodeArtifactMavenController {
}
private String requireRepositoryId(HttpHeaders headers, String domain, String repository) {
- String token = extractToken(headers);
+ String token = CodeArtifactTokenExtractor.extractToken(headers);
AuthorizationTokenScope scope = service.resolveAuthorizationToken(token, domain)
.orElseThrow(() -> new NotAuthorizedException("Basic realm=\"floci-codeartifact\""));
String repoId;
@@ -108,33 +101,4 @@ public class CodeArtifactMavenController {
reposilite.ensureRepository(repoId);
return repoId;
}
-
- /**
- * {@code null} when neither scheme is present, malformed, or the Basic credentials have no
- * password field; {@link CodeArtifactService#resolveAuthorizationToken} treats a null token
- * the same as any other invalid one.
- */
- private static String extractToken(HttpHeaders headers) {
- String authorization = headers.getHeaderString(HttpHeaders.AUTHORIZATION);
- if (authorization == null) {
- return null;
- }
- if (authorization.startsWith(BEARER_PREFIX)) {
- return authorization.substring(BEARER_PREFIX.length());
- }
- if (authorization.startsWith(BASIC_PREFIX)) {
- return passwordFromBasicCredentials(authorization.substring(BASIC_PREFIX.length()));
- }
- return null;
- }
-
- private static String passwordFromBasicCredentials(String base64Credentials) {
- try {
- String decoded = new String(Base64.getDecoder().decode(base64Credentials), StandardCharsets.UTF_8);
- int separator = decoded.indexOf(':');
- return separator >= 0 ? decoded.substring(separator + 1) : null;
- } catch (IllegalArgumentException e) {
- return null;
- }
- }
}
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiController.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiController.java
new file mode 100644
index 0000000000..3f129ca385
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiController.java
@@ -0,0 +1,284 @@
+package io.github.hectorvent.floci.services.codeartifact;
+
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.core.common.MultipartFormParser;
+import io.github.hectorvent.floci.services.codeartifact.CodeArtifactService.AuthorizationTokenScope;
+import jakarta.inject.Inject;
+import jakarta.ws.rs.Consumes;
+import jakarta.ws.rs.GET;
+import jakarta.ws.rs.NotAuthorizedException;
+import jakarta.ws.rs.NotFoundException;
+import jakarta.ws.rs.POST;
+import jakarta.ws.rs.Path;
+import jakarta.ws.rs.PathParam;
+import jakarta.ws.rs.Produces;
+import jakarta.ws.rs.core.Context;
+import jakarta.ws.rs.core.HttpHeaders;
+import jakarta.ws.rs.core.MediaType;
+import jakarta.ws.rs.core.Response;
+
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpRequest.BodyPublishers;
+import java.net.http.HttpResponse;
+import java.net.http.HttpResponse.BodyHandlers;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Locale;
+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Serves the real PyPI simple-repository wire protocol behind the URL
+ * {@code GetRepositoryEndpoint} returns for the {@code pypi} format: the same plain GET/POST
+ * pip and twine speak against real PyPI, not a CodeArtifact API action. Backed by one pypiserver
+ * container per repository, provisioned on first use.
+ *
+ * <p>Requires the bearer token real CodeArtifact requires here, obtained from
+ * {@code GetAuthorizationToken} and scoped to the domain being accessed. AWS documents pip and
+ * twine both authenticating with HTTP Basic ({@code username=aws, password=<token>}, verified
+ * against AWS's own pip and twine configuration docs), the same convention Maven's HTTP wagon
+ * uses, not npm's Bearer-only {@code _authToken}; {@link CodeArtifactTokenExtractor} accepts
+ * either scheme.
+ *
+ * <p>A pip/twine request carries no SigV4 {@code Authorization} header, so there is no account or
+ * Region to resolve from it the way the CodeArtifact JSON API does; the token itself supplies
+ * both, since it was issued under a specific account and Region.
+ *
+ * <p>Unlike Maven and npm, this rejects a re-upload of a file pypiserver has already published
+ * itself: pypiserver has no immutability check of its own (confirmed by uploading the same
+ * name/version/filename twice against a real instance; the second upload silently replaces the
+ * first), unlike Reposilite ({@code redeployment: false}) and Verdaccio (rejects natively,
+ * matching real npm). A pre-upload check against pypiserver's own simple index and the existing
+ * file's own bytes is what gives PyPI the same immutability guarantee the other two formats
+ * already get for free from their sidecars, including AWS's own documented exception to it:
+ * republishing an asset whose content is byte-identical to what is already there succeeds
+ * idempotently rather than conflicting (verified against AWS's CodeArtifact packages-overview
+ * docs, "Overwriting package assets": only a same-name upload with <i>different</i> content
+ * returns 409).
+ */
+@Path("/codeartifact/pypi/{domain}/{repository}")
+public class CodeArtifactPypiController {
+
+ private static final String SIMPLE_PATH = "/simple/";
+
+ private final CodeArtifactService service;
+ private final PypiserverSidecarManager pypiserver;
+ private final HttpClient httpClient;
+ private final ConcurrentHashMap<String, Object> uploadLocks = new ConcurrentHashMap<>();
+
+ @Inject
+ public CodeArtifactPypiController(CodeArtifactService service, PypiserverSidecarManager pypiserver) {
+ this(service, pypiserver, HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build());
+ }
+
+ CodeArtifactPypiController(CodeArtifactService service, PypiserverSidecarManager pypiserver,
+ HttpClient httpClient) {
+ this.service = service;
+ this.pypiserver = pypiserver;
+ this.httpClient = httpClient;
+ }
+
+ private record RepositoryHandle(String repoId, String baseUrl, String pathPrefix) {
+ }
+
+ // No route for the bare "/simple/" project-list endpoint: AWS's CodeArtifact User Guide
+ // documents it as unsupported ("Unsupported PyPI APIs", only /simple/<project>/ is served).
+
+ /**
+ * pypiserver's own simple-index HTML links to package files with a root-relative
+ * {@code href="/packages/<file>"}, resolved by pip against whatever host it actually
+ * connected to: this proxy's own host, not pypiserver's. Left unrewritten, that resolves to
+ * {@code /packages/<file>} at the proxy's root, not under this repository's
+ * {@code /codeartifact/pypi/<domain>/<repository>/} prefix where {@link #downloadFile} is
+ * actually routed, so pip would 404 trying to install anything the index lists. Verified
+ * against a real pypiserver response before writing this, not assumed.
+ */
+ @GET
+ @Path("simple/{package}/")
+ @Produces(MediaType.TEXT_HTML)
+ public Response packageIndex(@Context HttpHeaders headers, @PathParam("domain") String domain,
+ @PathParam("repository") String repository,
+ @PathParam("package") String packageName) {
+ RepositoryHandle handle = requireRepository(headers, domain, repository);
+ Response response = proxyGet(handle.baseUrl(), SIMPLE_PATH + packageName + "/");
+ if (response.getStatus() != 200) {
+ return response;
+ }
+ String html = new String((byte[]) response.getEntity(), StandardCharsets.UTF_8);
+ String rewritten = html.replace("href=\"/packages/", "href=\"" + handle.pathPrefix() + "/packages/");
+ return Response.status(200).entity(rewritten.getBytes(StandardCharsets.UTF_8)).type(MediaType.TEXT_HTML)
+ .build();
+ }
+
+ @GET
+ @Path("packages/{filename}")
+ @Produces(MediaType.WILDCARD)
+ public Response downloadFile(@Context HttpHeaders headers, @PathParam("domain") String domain,
+ @PathParam("repository") String repository,
+ @PathParam("filename") String filename) {
+ String baseUrl = requireRepository(headers, domain, repository).baseUrl();
+ return proxyGet(baseUrl, "/packages/" + filename);
+ }
+
+ /**
+ * Handles a twine upload: {@code POST} to the repository root with a
+ * {@code multipart/form-data} body whose {@code content} part is the package file and whose
+ * {@code name}/{@code version} fields identify it. When the exact filename being uploaded is
+ * already published, compares its existing bytes to the new upload: byte-identical content
+ * succeeds without forwarding anything (idempotent, matching AWS's own documented behavior for
+ * a retried publish), while different content is rejected with 409 before ever reaching
+ * pypiserver, since pypiserver has no immutability check of its own.
+ *
+ * <p>The existence check and the forwarded publish run inside a lock keyed by this
+ * repository's id: unlike Reposilite ({@code redeployment: false}), pypiserver has no atomic
+ * backstop of its own, so without this lock two uploads racing the same new filename could
+ * both observe "not published yet" and both succeed, with pypiserver silently keeping whichever
+ * write landed last instead of either one being rejected.
+ */
+ @POST
+ @Consumes(MediaType.WILDCARD)
+ public Response upload(@Context HttpHeaders headers, @PathParam("domain") String domain,
+ @PathParam("repository") String repository, byte[] body) {
+ byte[] content = body != null ? body : new byte[0];
+ // Same 5 GB AWS quota PublishPackageVersion enforces, and just as nominal: RESTEasy
+ // Reactive has already buffered the full body into this byte[] before this method runs,
+ // so this rejects anything that made it this far rather than bounding memory use upfront.
+ if (content.length > CodeArtifactService.MAX_ASSET_FILE_SIZE_BYTES) {
+ return Response.status(413).build();
+ }
+ RepositoryHandle handle = requireRepository(headers, domain, repository);
+
+ String contentType = headers.getHeaderString(HttpHeaders.CONTENT_TYPE);
+ String boundary = MultipartFormParser.extractBoundary(contentType).orElse(null);
+ if (boundary == null) {
+ return Response.status(400).entity("Could not determine multipart boundary from Content-Type.").build();
+ }
+ MultipartFormParser.ParsedForm form = MultipartFormParser.parse(content, boundary);
+ String packageName = form.fields().get("name");
+ Optional<MultipartFormParser.FilePart> file = form.file();
+ if (packageName == null || packageName.isBlank() || file.isEmpty()) {
+ return Response.status(400).entity("Upload must include name and content fields.").build();
+ }
+
+ Object lock = uploadLocks.computeIfAbsent(handle.repoId(), k -> new Object());
+ synchronized (lock) {
+ Optional<byte[]> existing = existingFileContent(handle.baseUrl(), packageName, file.get().filename());
+ if (existing.isPresent()) {
+ if (Arrays.equals(existing.get(), file.get().content())) {
+ return Response.status(200).build();
+ }
+ return Response.status(409)
+ .entity("File already exists. See "
+ + "https://docs.aws.amazon.com/codeartifact/latest/ug/PyPI-format.html")
+ .build();
+ }
+
+ return forwardUpload(handle.baseUrl(), contentType, content);
+ }
+ }
+
+ /**
+ * {@code empty()} when {@code filename} isn't listed in the package's simple index yet, its
+ * actual bytes otherwise. pypiserver normalizes a package name for its own simple-index path
+ * the same way real PyPI does (PEP 503: lowercase, runs of {@code -_.} collapsed to one
+ * {@code -}); querying with the raw, as-uploaded name for a package whose normalized form
+ * differs would 404 even though the package exists.
+ *
+ * <p>A connectivity failure here throws rather than returning {@code empty()}: treating "the
+ * check couldn't complete" the same as "nothing published yet" would let an upload with
+ * different content through to pypiserver (which has no conflict check of its own) during
+ * exactly the moment this proxy's own protection is unavailable, the opposite of fail-safe.
+ */
+ private Optional<byte[]> existingFileContent(String baseUrl, String packageName, String filename) {
+ String normalized = normalizePackageName(packageName);
+ HttpRequest indexRequest = HttpRequest.newBuilder(URI.create(baseUrl + SIMPLE_PATH + normalized + "/"))
+ .timeout(Duration.ofSeconds(5)).GET().build();
+ HttpResponse<String> indexResponse;
+ try {
+ indexResponse = httpClient.send(indexRequest, BodyHandlers.ofString());
+ } catch (Exception e) {
+ throw new IllegalStateException(
+ "Could not check pypi package " + packageName + " for an existing " + filename
+ + " before publishing", e);
+ }
+ if (indexResponse.statusCode() == 404) {
+ return Optional.empty();
+ }
+ if (indexResponse.statusCode() != 200) {
+ throw new IllegalStateException(
+ "Could not check pypi package " + packageName + " for an existing " + filename
+ + ": upstream returned " + indexResponse.statusCode());
+ }
+ if (!indexResponse.body().contains(filename)) {
+ return Optional.empty();
+ }
+
+ HttpRequest fileRequest = HttpRequest.newBuilder(URI.create(baseUrl + "/packages/" + filename))
+ .timeout(Duration.ofSeconds(10)).GET().build();
+ HttpResponse<byte[]> fileResponse;
+ try {
+ fileResponse = httpClient.send(fileRequest, BodyHandlers.ofByteArray());
+ } catch (Exception e) {
+ throw new IllegalStateException(
+ "Could not fetch the existing pypi file " + filename + " to check for a content match", e);
+ }
+ if (fileResponse.statusCode() != 200) {
+ throw new IllegalStateException(
+ "Could not fetch the existing pypi file " + filename + " to check for a content match: "
+ + "upstream returned " + fileResponse.statusCode());
+ }
+ return Optional.of(fileResponse.body());
+ }
+
+ private static String normalizePackageName(String name) {
+ return name.toLowerCase(Locale.ROOT).replaceAll("[-_.]+", "-");
+ }
+
+ private Response forwardUpload(String baseUrl, String contentType, byte[] content) {
+ HttpRequest request = HttpRequest.newBuilder(URI.create(baseUrl + "/"))
+ .timeout(Duration.ofSeconds(30))
+ .header("Content-Type", contentType)
+ .POST(BodyPublishers.ofByteArray(content))
+ .build();
+ HttpResponse<Void> response;
+ try {
+ response = httpClient.send(request, BodyHandlers.discarding());
+ } catch (Exception e) {
+ throw new IllegalStateException("Could not reach the pypi sidecar to publish this file", e);
+ }
+ return Response.status(response.statusCode()).build();
+ }
+
+ private Response proxyGet(String baseUrl, String path) {
+ HttpRequest request = HttpRequest.newBuilder(URI.create(baseUrl + path))
+ .timeout(Duration.ofSeconds(10)).GET().build();
+ HttpResponse<byte[]> response;
+ try {
+ response = httpClient.send(request, BodyHandlers.ofByteArray());
+ } catch (Exception e) {
+ throw new IllegalStateException("Could not reach the pypi sidecar at " + path, e);
+ }
+ if (response.statusCode() == 404) {
+ return Response.status(404).build();
+ }
+ String contentType = response.headers().firstValue("Content-Type").orElse("application/octet-stream");
+ return Response.status(response.statusCode()).entity(response.body()).type(contentType).build();
+ }
+
+ private RepositoryHandle requireRepository(HttpHeaders headers, String domain, String repository) {
+ String token = CodeArtifactTokenExtractor.extractToken(headers);
+ AuthorizationTokenScope scope = service.resolveAuthorizationToken(token, domain)
+ .orElseThrow(() -> new NotAuthorizedException("Basic realm=\"floci-codeartifact\""));
+ String repoId;
+ try {
+ repoId = service.ensureFormatContainerId("pypi", scope.region(), domain, scope.owner(), repository);
+ } catch (AwsException e) {
+ throw new NotFoundException();
+ }
+ String baseUrl = pypiserver.ensureReady(repoId, null);
+ return new RepositoryHandle(repoId, baseUrl, "/codeartifact/pypi/" + domain + "/" + repository);
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java
index 9ad97fa76e..11c8b490a9 100644
--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java
@@ -73,7 +73,7 @@ public class CodeArtifactService implements Resettable {
* and the backfill in {@link #ensureFormatContainerId} already handle it, including for
* repositories created before that format's proxy existed.
*/
- private static final List<String> CONTAINER_BACKED_FORMATS = List.of("maven", "npm");
+ private static final List<String> CONTAINER_BACKED_FORMATS = List.of("maven", "npm", "pypi");
private static final Logger LOG = Logger.getLogger(CodeArtifactService.class);
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactTokenExtractor.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactTokenExtractor.java
new file mode 100644
index 0000000000..28048a316d
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactTokenExtractor.java
@@ -0,0 +1,57 @@
+package io.github.hectorvent.floci.services.codeartifact;
+
+import jakarta.ws.rs.core.HttpHeaders;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+
+/**
+ * Extracts a CodeArtifact authorization token from an incoming request's {@code Authorization}
+ * header, accepting either scheme real package-manager clients use: {@code Bearer <token>}
+ * directly, or HTTP Basic with the token as the password field and any username. AWS documents
+ * only Basic for Maven's {@code settings.xml} and pip/twine's {@code username=aws} convention;
+ * accepting Bearer too costs nothing and matches what a Bearer-only format like npm sends.
+ *
+ * <p>Pulled out of {@code CodeArtifactMavenController} once {@code CodeArtifactPypiController}
+ * needed the identical extraction: pip and twine authenticate exactly the way Maven's HTTP wagon
+ * does (verified against AWS's own pip and twine configuration docs, both showing
+ * {@code username=aws, password=<token>}), not the Bearer-only convention npm's {@code _authToken}
+ * uses.
+ */
+final class CodeArtifactTokenExtractor {
+
+ private static final String BEARER_PREFIX = "Bearer ";
+ private static final String BASIC_PREFIX = "Basic ";
+
+ private CodeArtifactTokenExtractor() {
+ }
+
+ /**
+ * {@code null} when neither scheme is present, malformed, or the Basic credentials have no
+ * password field; {@link CodeArtifactService#resolveAuthorizationToken} treats a null token
+ * the same as any other invalid one.
+ */
+ static String extractToken(HttpHeaders headers) {
+ String authorization = headers.getHeaderString(HttpHeaders.AUTHORIZATION);
+ if (authorization == null) {
+ return null;
+ }
+ if (authorization.startsWith(BEARER_PREFIX)) {
+ return authorization.substring(BEARER_PREFIX.length());
+ }
+ if (authorization.startsWith(BASIC_PREFIX)) {
+ return passwordFromBasicCredentials(authorization.substring(BASIC_PREFIX.length()));
+ }
+ return null;
+ }
+
+ private static String passwordFromBasicCredentials(String base64Credentials) {
+ try {
+ String decoded = new String(Base64.getDecoder().decode(base64Credentials), StandardCharsets.UTF_8);
+ int separator = decoded.indexOf(':');
+ return separator >= 0 ? decoded.substring(separator + 1) : null;
+ } catch (IllegalArgumentException e) {
+ return null;
+ }
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManager.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManager.java
new file mode 100644
index 0000000000..deadd17d08
--- /dev/null
+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManager.java
@@ -0,0 +1,131 @@
+package io.github.hectorvent.floci.services.codeartifact;
+
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.ContainerTeardown;
+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.ContainerInfo;
+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager.EndpointInfo;
+import io.github.hectorvent.floci.core.common.docker.ContainerSpec;
+import io.github.hectorvent.floci.core.common.docker.ContainerStorageHelper;
+import io.github.hectorvent.floci.core.common.docker.PerKeyContainerPool;
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+import org.jboss.logging.Logger;
+
+import java.util.List;
+
+/**
+ * Lazily starts and manages one pypiserver container per CodeArtifact repository backing the
+ * {@code pypi} format. Like Verdaccio (npm), pypiserver has no native concept of multiple named
+ * indexes inside one instance, so pypi repositories each get their own container instead of
+ * sharing one the way Reposilite's Maven repositories do. The per-key container lifecycle (map of
+ * running containers, per-key start lock, health poll, restart-on-unhealthy, stop-all on
+ * shutdown/reset) is generic and lives in {@link PerKeyContainerPool}; this class only knows how
+ * to build and configure a pypiserver container specifically.
+ *
+ * <p>Each container is configured with authentication disabled (real CodeArtifact authorization
+ * happens once at Floci's proxy layer, {@code CodeArtifactPypiController}, before a request ever
+ * reaches this container) and {@code --disable-fallback}, matching the Maven and npm proxies' own
+ * deliberate choice not to resolve upstream repositories or external connections: pypiserver
+ * otherwise redirects a package missing from its local index to the real, public PyPI.
+ *
+ * <p>Unlike Verdaccio, this needs no public-URL environment variable: pypiserver's simple-index
+ * responses link to package files with a root-relative path ({@code /packages/<file>}), which pip
+ * and twine resolve against whatever host they actually connected to, not an address the
+ * container returns itself. There is nothing here for an internal, client-unreachable address to
+ * leak into. That root-relative path still needs rewriting to this repository's own
+ * {@code /codeartifact/pypi/<domain>/<repository>} prefix before a client sees it, since the
+ * download route lives there, not at the proxy's bare root; {@code CodeArtifactPypiController}
+ * does that rewrite, not this class.
+ */
+@ApplicationScoped
+public class PypiserverSidecarManager implements RepositorySidecarManager, ContainerTeardown {
+
+ private static final Logger LOG = Logger.getLogger(PypiserverSidecarManager.class);
+ private static final String FORMAT = "pypi";
+ private static final int PYPISERVER_PORT = 8080;
+ private static final String HEALTH_PATH = "/health";
+ private static final String PACKAGES_DIR = "/data/packages";
+
+ private final ContainerBuilder containerBuilder;
+ private final ContainerLifecycleManager lifecycleManager;
+ private final EmulatorConfig config;
+ private final PerKeyContainerPool pool;
+
+ @Inject
+ public PypiserverSidecarManager(ContainerBuilder containerBuilder, ContainerLifecycleManager lifecycleManager,
+ EmulatorConfig config) {
+ this.containerBuilder = containerBuilder;
+ this.lifecycleManager = lifecycleManager;
+ this.config = config;
+ this.pool = new PerKeyContainerPool(lifecycleManager, HEALTH_PATH);
+ }
+
+ @Override
+ public String format() {
+ return FORMAT;
+ }
+
+ /**
+ * Base URL of a ready pypiserver instance for this pypi repository, starting its container if
+ * this is the first use. {@code publicUrl} is unused: see the class javadoc for why pypiserver
+ * needs no self-referential URL rewritten.
+ */
+ @Override
+ public String ensureReady(String pypiRepositoryId, String publicUrl) {
+ return pool.ensureReady(pypiRepositoryId, () -> startContainer(pypiRepositoryId));
+ }
+
+ /** Stops and removes the container for one pypi repository, if one was ever started. */
+ @Override
+ public void release(String pypiRepositoryId) {
+ pool.stopContainer(pypiRepositoryId);
+ }
+
+ private PerKeyContainerPool.StartedContainer startContainer(String pypiRepositoryId) {
+ String image = config.services().codeartifact().pypiImage();
+ String containerName = ContainerStorageHelper.dockerName(config, "floci-pypiserver-" + pypiRepositoryId);
+ lifecycleManager.removeIfExists(containerName);
+
+ // Loopback-only: this container authenticates nothing on its own (auth disabled below), so
+ // the Bearer check in CodeArtifactPypiController is the only thing standing between a
+ // client and the backing storage. Publishing this to every interface would let anyone who
+ // can reach the host bypass that check entirely, the same reasoning as Verdaccio's binding.
+ ContainerSpec spec = containerBuilder.newContainer(image)
+ .withName(containerName)
+ .withLoopbackPortBinding(PYPISERVER_PORT, 0)
+ .withDockerNetwork(config.services().dockerNetwork())
+ .withEmbeddedDns()
+ .withLogRotation()
+ .withCmd(List.of(
+ "run",
+ "-p", String.valueOf(PYPISERVER_PORT),
+ // Disables pypiserver's own auth entirely (real CodeArtifact authorization
+ // is already enforced before a request reaches this container) and its
+ // default redirect-to-real-PyPI for a package missing from the local index,
+ // matching the Maven and npm proxies' own choice not to resolve upstreams.
+ "-a", ".",
+ "-P", ".",
+ "--disable-fallback",
+ "--health-endpoint", HEALTH_PATH,
+ PACKAGES_DIR))
+ .build();
+ String containerId = lifecycleManager.create(spec);
+ try {
+ ContainerInfo info = lifecycleManager.startCreated(containerId, spec);
+ EndpointInfo endpoint = info.getEndpoint(PYPISERVER_PORT);
+ String url = "http://" + endpoint;
+ LOG.infov("pypiserver sidecar for pypi repository {0} is ready at {1}", pypiRepositoryId, url);
+ return new PerKeyContainerPool.StartedContainer(containerId, url);
+ } catch (RuntimeException e) {
+ lifecycleManager.stopAndRemove(containerId, null);
+ throw e;
+ }
+ }
+
+ @Override
+ public void stopManagedContainers() {
+ pool.stopAll();
+ }
+}
diff --git a/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java b/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java
index d61b07bca5..d6f7506ce5 100644
--- a/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java
+++ b/src/main/java/io/github/hectorvent/floci/services/s3/S3Controller.java
@@ -9,6 +9,7 @@ import io.github.hectorvent.floci.core.common.AwsException;
import io.github.hectorvent.floci.core.common.AwsNamespaces;
import io.github.hectorvent.floci.core.common.AwsRegions;
import io.github.hectorvent.floci.core.common.IamEnforcementFilter;
+import io.github.hectorvent.floci.core.common.MultipartFormParser;
import io.github.hectorvent.floci.core.common.XmlBuilder;
import io.github.hectorvent.floci.core.common.XmlParser;
import io.github.hectorvent.floci.core.common.RegionResolver;
@@ -3444,55 +3445,14 @@ public class S3Controller {
}
private Response doHandlePresignedPost(String bucket, String contentType, byte[] body) {
- String boundary = extractBoundary(contentType);
- if (boundary == null) {
- throw new AwsException("InvalidArgument",
- "Could not determine multipart boundary from Content-Type.", 400);
- }
-
- Map<String, String> fields = new LinkedHashMap<>();
- byte[] fileData = null;
- String fileContentType = null;
+ String boundary = MultipartFormParser.extractBoundary(contentType).orElseThrow(() ->
+ new AwsException("InvalidArgument",
+ "Could not determine multipart boundary from Content-Type.", 400));
- byte[] boundaryBytes = ("--" + boundary).getBytes(StandardCharsets.UTF_8);
- List<byte[]> parts = splitMultipartParts(body, boundaryBytes);
-
- for (byte[] part : parts) {
- int headerEnd = indexOfDoubleNewline(part);
- if (headerEnd < 0) {
- continue;
- }
- String headers = new String(part, 0, headerEnd, StandardCharsets.UTF_8);
- int bodyStart = headerEnd + 4; // skip \r\n\r\n
- byte[] partBody = Arrays.copyOfRange(part, bodyStart, part.length);
-
- // Trim trailing \r\n from part body
- if (partBody.length >= 2
- && partBody[partBody.length - 2] == '\r'
- && partBody[partBody.length - 1] == '\n') {
- partBody = Arrays.copyOf(partBody, partBody.length - 2);
- }
-
- String disposition = extractHeaderValue(headers, "Content-Disposition");
- if (disposition == null) {
- continue;
- }
- String fieldName = extractDispositionParam(disposition, "name");
- if (fieldName == null) {
- continue;
- }
-
- String filename = extractDispositionParam(disposition, "filename");
- if (filename != null) {
- fileData = partBody;
- String partContentType = extractHeaderValue(headers, "Content-Type");
- if (partContentType != null) {
- fileContentType = partContentType.trim();
- }
- } else {
- fields.put(fieldName, new String(partBody, StandardCharsets.UTF_8));
- }
- }
+ MultipartFormParser.ParsedForm form = MultipartFormParser.parse(body, boundary);
+ Map<String, String> fields = form.fields();
+ byte[] fileData = form.file().map(MultipartFormParser.FilePart::content).orElse(null);
+ String fileContentType = form.file().map(MultipartFormParser.FilePart::contentType).orElse(null);
String key = fields.get("key");
if (key == null || key.isEmpty()) {
@@ -3768,123 +3728,6 @@ public class S3Controller {
return fields.get(fieldName);
}
- private static String extractBoundary(String contentType) {
- if (contentType == null) {
- return null;
- }
- for (String part : contentType.split(";")) {
- String trimmed = part.trim();
- if (trimmed.toLowerCase(Locale.ROOT).startsWith("boundary=")) {
- String boundary = trimmed.substring("boundary=".length()).trim();
- if (boundary.startsWith("\"") && boundary.endsWith("\"")) {
- boundary = boundary.substring(1, boundary.length() - 1);
- }
- return boundary;
- }
- }
- return null;
- }
-
- private static List<byte[]> splitMultipartParts(byte[] body, byte[] boundary) {
- java.util.ArrayList<byte[]> parts = new java.util.ArrayList<>();
- int pos = indexOf(body, boundary, 0);
- if (pos < 0) {
- return parts;
- }
- // Skip past the first boundary line
- pos += boundary.length;
- // Skip the CRLF or -- after boundary
- if (pos < body.length - 1 && body[pos] == '-' && body[pos + 1] == '-') {
- return parts; // closing boundary immediately
- }
- if (pos < body.length - 1 && body[pos] == '\r' && body[pos + 1] == '\n') {
- pos += 2;
- }
-
- while (pos < body.length) {
- int nextBoundary = indexOf(body, boundary, pos);
- if (nextBoundary < 0) {
- break;
- }
- parts.add(Arrays.copyOfRange(body, pos, nextBoundary));
- pos = nextBoundary + boundary.length;
- // Check for closing boundary --
- if (pos < body.length - 1 && body[pos] == '-' && body[pos + 1] == '-') {
- break;
- }
- // Skip CRLF after boundary
- if (pos < body.length - 1 && body[pos] == '\r' && body[pos + 1] == '\n') {
- pos += 2;
- }
- }
- return parts;
- }
-
- private static int indexOf(byte[] data, byte[] pattern, int fromIndex) {
- outer:
- for (int i = fromIndex; i <= data.length - pattern.length; i++) {
- for (int j = 0; j < pattern.length; j++) {
- if (data[i + j] != pattern[j]) {
- continue outer;
- }
- }
- return i;
- }
- return -1;
- }
-
- private static int indexOfDoubleNewline(byte[] data) {
- for (int i = 0; i < data.length - 3; i++) {
- if (data[i] == '\r' && data[i + 1] == '\n' && data[i + 2] == '\r' && data[i + 3] == '\n') {
- return i;
- }
- }
- return -1;
- }
-
- private static String extractHeaderValue(String headers, String headerName) {
- String lowerHeaders = headers.toLowerCase(Locale.ROOT);
- String lowerName = headerName.toLowerCase(Locale.ROOT) + ":";
- int idx = lowerHeaders.indexOf(lowerName);
- if (idx < 0) {
- return null;
- }
- int valueStart = idx + lowerName.length();
- int lineEnd = headers.indexOf('\r', valueStart);
- if (lineEnd < 0) {
- lineEnd = headers.indexOf('\n', valueStart);
- }
- if (lineEnd < 0) {
- lineEnd = headers.length();
- }
- return headers.substring(valueStart, lineEnd).trim();
- }
-
- private static String extractDispositionParam(String disposition, String paramName) {
- String search = paramName + "=";
- int idx = disposition.indexOf(search);
- if (idx < 0) {
- return null;
- }
- int valueStart = idx + search.length();
- if (valueStart >= disposition.length()) {
- return null;
- }
- if (disposition.charAt(valueStart) == '"') {
- valueStart++;
- int valueEnd = disposition.indexOf('"', valueStart);
- if (valueEnd < 0) {
- return disposition.substring(valueStart);
- }
- return disposition.substring(valueStart, valueEnd);
- } else {
- int valueEnd = disposition.indexOf(';', valueStart);
- if (valueEnd < 0) {
- valueEnd = disposition.length();
- }
- return disposition.substring(valueStart, valueEnd).trim();
- }
- }
private static final int MAX_INLINE_TAGS = 10;
private static final int MAX_INLINE_TAGGING_HEADER_BYTES = 8 * 1024;
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 7b6e7f65c5..f2e43b0114 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -928,6 +928,7 @@ floci:
# maven-url: http://floci-reposilite:8080 # set to skip Reposilite sidecar container management
maven-image: "dzikoysk/reposilite:3.6.3"
npm-image: "verdaccio/verdaccio:6.10.4"
+ pypi-image: "pypiserver/pypiserver:v2.4.2"
marketplace:
enabled: true
dms:
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/MultipartFormParserTest.java b/src/test/java/io/github/hectorvent/floci/core/common/MultipartFormParserTest.java
new file mode 100644
index 0000000000..036810fbc2
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/common/MultipartFormParserTest.java
@@ -0,0 +1,118 @@
+package io.github.hectorvent.floci.core.common;
+
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class MultipartFormParserTest {
+
+ private static final String BOUNDARY = "boundary123";
+
+ @Test
+ void extractBoundaryReadsTheParameterRegardlessOfQuoting() {
+ assertEquals(Optional.of("abc"), MultipartFormParser.extractBoundary("multipart/form-data; boundary=abc"));
+ assertEquals(Optional.of("abc"),
+ MultipartFormParser.extractBoundary("multipart/form-data; boundary=\"abc\""));
+ assertEquals(Optional.empty(), MultipartFormParser.extractBoundary("application/json"));
+ assertEquals(Optional.empty(), MultipartFormParser.extractBoundary(null));
+ }
+
+ @Test
+ void parseSeparatesTextFieldsFromTheFilePart() {
+ byte[] body = multipart(
+ field("name", "requests"),
+ field("version", "2.34.2"),
+ file("content", "requests-2.34.2-py3-none-any.whl", "application/octet-stream",
+ "fake wheel bytes"));
+
+ MultipartFormParser.ParsedForm form = MultipartFormParser.parse(body, BOUNDARY);
+
+ assertEquals("requests", form.fields().get("name"));
+ assertEquals("2.34.2", form.fields().get("version"));
+ assertFalse(form.fields().containsKey("content"));
+ assertTrue(form.file().isPresent());
+ MultipartFormParser.FilePart file = form.file().get();
+ assertEquals("content", file.fieldName());
+ assertEquals("requests-2.34.2-py3-none-any.whl", file.filename());
+ assertEquals("application/octet-stream", file.contentType());
+ assertEquals("fake wheel bytes", new String(file.content(), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void parseReturnsAnEmptyFileWhenNoPartCarriesAFilename() {
+ byte[] body = multipart(field("key", "value"));
+
+ MultipartFormParser.ParsedForm form = MultipartFormParser.parse(body, BOUNDARY);
+
+ assertEquals("value", form.fields().get("key"));
+ assertTrue(form.file().isEmpty());
+ }
+
+ @Test
+ void parsePreservesBinaryContentThatContainsCrlfBytes() {
+ byte[] binary = {0x50, 0x4B, 0x03, 0x04, '\r', '\n', 0x00, (byte) 0xFF};
+ byte[] body = multipart(filePart("content", "binary.bin", "application/octet-stream", binary));
+
+ MultipartFormParser.ParsedForm form = MultipartFormParser.parse(body, BOUNDARY);
+
+ assertTrue(form.file().isPresent());
+ assertArrayEqualsExactly(binary, form.file().get().content());
+ }
+
+ @Test
+ void parseOnAnEmptyOrMalformedBodyReturnsNoFieldsAndNoFile() {
+ MultipartFormParser.ParsedForm form = MultipartFormParser.parse(new byte[0], BOUNDARY);
+
+ assertTrue(form.fields().isEmpty());
+ assertTrue(form.file().isEmpty());
+ }
+
+ private static void assertArrayEqualsExactly(byte[] expected, byte[] actual) {
+ assertEquals(expected.length, actual.length);
+ for (int i = 0; i < expected.length; i++) {
+ assertEquals(expected[i], actual[i], "byte at index " + i);
+ }
+ }
+
+ private static byte[] multipart(byte[]... parts) {
+ StringBuilder header = new StringBuilder();
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ try {
+ for (byte[] part : parts) {
+ out.write(("--" + BOUNDARY + "\r\n").getBytes(StandardCharsets.UTF_8));
+ out.write(part);
+ out.write("\r\n".getBytes(StandardCharsets.UTF_8));
+ }
+ out.write(("--" + BOUNDARY + "--\r\n").getBytes(StandardCharsets.UTF_8));
+ } catch (IOException e) {
+ throw new IllegalStateException(e);
+ }
+ return out.toByteArray();
+ }
+
+ private static byte[] field(String name, String value) {
+ String part = "Content-Disposition: form-data; name=\"" + name + "\"\r\n\r\n" + value;
+ return part.getBytes(StandardCharsets.UTF_8);
+ }
+
+ private static byte[] file(String fieldName, String filename, String contentType, String content) {
+ return filePart(fieldName, filename, contentType, content.getBytes(StandardCharsets.UTF_8));
+ }
+
+ private static byte[] filePart(String fieldName, String filename, String contentType, byte[] content) {
+ String header = "Content-Disposition: form-data; name=\"" + fieldName + "\"; filename=\"" + filename + "\"\r\n"
+ + "Content-Type: " + contentType + "\r\n\r\n";
+ byte[] headerBytes = header.getBytes(StandardCharsets.UTF_8);
+ byte[] result = new byte[headerBytes.length + content.length];
+ System.arraycopy(headerBytes, 0, result, 0, headerBytes.length);
+ System.arraycopy(content, 0, result, headerBytes.length, content.length);
+ return result;
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/core/common/ServiceEnablementIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/core/common/ServiceEnablementIntegrationTest.java
index 24de93b4ce..19e5df4aa8 100644
--- a/src/test/java/io/github/hectorvent/floci/core/common/ServiceEnablementIntegrationTest.java
+++ b/src/test/java/io/github/hectorvent/floci/core/common/ServiceEnablementIntegrationTest.java
@@ -170,6 +170,49 @@ class ServiceEnablementIntegrationTest {
.body("message", equalTo("Service fis is not enabled."));
}
+ /**
+ * {@code CodeArtifactMavenController} and {@code CodeArtifactPypiController} serve a real
+ * package-manager wire protocol, not a CodeArtifact JSON API action, so nothing about a
+ * request to either carries a SigV4 signature or an {@code X-Amz-Target} for
+ * {@code ServiceEnabledFilter} to resolve a service from the normal ways. Both must still be
+ * registered in the {@code codeartifact} descriptor's own resource-class set, matched by JAX-RS
+ * routing alone: without that registration a disabled codeartifact service does not stop
+ * either from reaching its own token check and responding with a 401 instead of ever saying
+ * the service itself is unavailable. Caught after CodeArtifactPypiController needed the exact
+ * same registration CodeArtifactMavenController had been missing.
+ */
+ @Test
+ void mavenWireProtocolRequestsAreRejectedWhenServiceDisabled() {
+ given()
+ .header("Authorization", "Bearer whatever")
+ .when()
+ .get("/codeartifact/maven/some-domain/some-repo/some/path/1.0/file.jar")
+ .then()
+ .statusCode(400)
+ .contentType("application/json")
+ .header("X-Amzn-Errortype", "ServiceNotAvailableException")
+ .body("__type", equalTo("ServiceNotAvailableException"))
+ .body("message", equalTo("Service codeartifact is not enabled."));
+ }
+
+ @Test
+ void pypiWireProtocolRequestsAreRejectedWhenServiceDisabled() {
+ // Not the bare "/simple/" root: AWS doesn't support that path either (only
+ // /simple/<project>/ is), so Floci has no route for it at all, and a request to a
+ // genuinely unrouted path 404s before this filter ever runs. "/simple/<project>/" is a
+ // real, routed endpoint, so it still demonstrates the service-disabled rejection.
+ given()
+ .header("Authorization", "Bearer whatever")
+ .when()
+ .get("/codeartifact/pypi/some-domain/some-repo/simple/some-pkg/")
+ .then()
+ .statusCode(400)
+ .contentType("application/json")
+ .header("X-Amzn-Errortype", "ServiceNotAvailableException")
+ .body("__type", equalTo("ServiceNotAvailableException"))
+ .body("message", equalTo("Service codeartifact is not enabled."));
+ }
+
@Test
void signedRdsDataExecuteRequestsReturnJsonWhenServiceDisabled() {
given()
@@ -203,6 +246,7 @@ class ServiceEnablementIntegrationTest {
public Map<String, String> getConfigOverrides() {
return Map.of(
"floci.services.acm.enabled", "false",
+ "floci.services.codeartifact.enabled", "false",
"floci.services.dynamodb.enabled", "false",
"floci.services.ecs.enabled", "false",
"floci.services.fis.enabled", "false",
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiControllerTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiControllerTest.java
new file mode 100644
index 0000000000..866d144a91
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiControllerTest.java
@@ -0,0 +1,326 @@
+package io.github.hectorvent.floci.services.codeartifact;
+
+import com.sun.net.httpserver.HttpServer;
+import io.github.hectorvent.floci.core.common.AwsException;
+import io.github.hectorvent.floci.services.codeartifact.CodeArtifactService.AuthorizationTokenScope;
+import jakarta.ws.rs.NotAuthorizedException;
+import jakarta.ws.rs.NotFoundException;
+import jakarta.ws.rs.core.HttpHeaders;
+import jakarta.ws.rs.core.Response;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.net.InetSocketAddress;
+import java.net.http.HttpClient;
+import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Exercises {@link CodeArtifactPypiController} against a fake pypiserver backend (a real
+ * {@link HttpServer}, not a mock of the JDK {@link HttpClient}, since that client's request/response
+ * types are effectively unmockable) so the proxy-passthrough, auth-resolution, and pre-upload
+ * overwrite-rejection logic all run for real. {@link CodeArtifactService} and
+ * {@link PypiserverSidecarManager} are mocked: what they do is already covered by
+ * {@code CodeArtifactServiceTest} and {@code PypiserverSidecarManagerTest}, what matters here is how
+ * this controller uses their results.
+ */
+class CodeArtifactPypiControllerTest {
+
+ private static final String DOMAIN = "dom";
+ private static final String REPOSITORY = "repo";
+ private static final String REGION = "us-east-1";
+ private static final String OWNER = "000000000000";
+
+ private HttpServer backend;
+ private String backendUrl;
+ private CodeArtifactService service;
+ private PypiserverSidecarManager pypiserver;
+ private CodeArtifactPypiController controller;
+
+ @BeforeEach
+ void setUp() throws Exception {
+ backend = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
+ backend.start();
+ backendUrl = "http://127.0.0.1:" + backend.getAddress().getPort();
+
+ service = mock(CodeArtifactService.class);
+ pypiserver = mock(PypiserverSidecarManager.class);
+ when(service.resolveAuthorizationToken(anyString(), eq(DOMAIN)))
+ .thenReturn(Optional.of(new AuthorizationTokenScope(OWNER, REGION)));
+ when(service.ensureFormatContainerId(eq("pypi"), eq(REGION), eq(DOMAIN), eq(OWNER), eq(REPOSITORY)))
+ .thenReturn("pypi-repo-id");
+ when(pypiserver.ensureReady(eq("pypi-repo-id"), any())).thenReturn(backendUrl);
+
+ HttpClient httpClient = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build();
+ controller = new CodeArtifactPypiController(service, pypiserver, httpClient);
+ }
+
+ @AfterEach
+ void tearDown() {
+ backend.stop(0);
+ }
+
+ @Test
+ void requireBaseUrlRejectsAMissingOrUnresolvableToken() {
+ when(service.resolveAuthorizationToken(anyString(), eq(DOMAIN))).thenReturn(Optional.empty());
+
+ assertThrows(NotAuthorizedException.class,
+ () -> controller.packageIndex(headers("Bearer bogus"), DOMAIN, REPOSITORY, "my-pkg"));
+ }
+
+ @Test
+ void requireBaseUrlTranslatesAMissingRepositoryIntoNotFound() {
+ when(service.ensureFormatContainerId(eq("pypi"), eq(REGION), eq(DOMAIN), eq(OWNER), eq(REPOSITORY)))
+ .thenThrow(new AwsException("ResourceNotFoundException", "no such repository", 404));
+
+ assertThrows(NotFoundException.class,
+ () -> controller.packageIndex(headers("Bearer good"), DOMAIN, REPOSITORY, "my-pkg"));
+ }
+
+ @Test
+ void packageIndexRewritesRootRelativePackageLinksToThisRepositorysPathPrefix() {
+ // Real pypiserver response shape, confirmed against a live container: a root-relative
+ // href, not the bare filename a naive fixture might use.
+ backend.createContext("/simple/my-pkg/", exchange -> respond(exchange, 200, "text/html",
+ "<a href=\"/packages/my-pkg-1.0.tar.gz#sha256=abc\">my-pkg-1.0.tar.gz</a>"));
+
+ Response response = controller.packageIndex(headers("Bearer good"), DOMAIN, REPOSITORY, "my-pkg");
+
+ assertEquals(200, response.getStatus());
+ String body = new String((byte[]) response.getEntity(), StandardCharsets.UTF_8);
+ assertTrue(body.contains("href=\"/codeartifact/pypi/" + DOMAIN + "/" + REPOSITORY
+ + "/packages/my-pkg-1.0.tar.gz#sha256=abc\""), body);
+ }
+
+ @Test
+ void downloadFileProxiesRawBytesFromTheBackend() {
+ byte[] content = "package bytes".getBytes(StandardCharsets.UTF_8);
+ backend.createContext("/packages/my-pkg-1.0.tar.gz",
+ exchange -> respond(exchange, 200, "application/octet-stream", content));
+
+ Response response = controller.downloadFile(headers("Bearer good"), DOMAIN, REPOSITORY, "my-pkg-1.0.tar.gz");
+
+ assertEquals(200, response.getStatus());
+ assertEquals("package bytes", new String((byte[]) response.getEntity(), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void downloadFileReturns404WhenTheBackendHasNoSuchFile() {
+ backend.createContext("/packages/missing.tar.gz", exchange -> respond(exchange, 404, "text/plain", ""));
+
+ Response response = controller.downloadFile(headers("Bearer good"), DOMAIN, REPOSITORY, "missing.tar.gz");
+
+ assertEquals(404, response.getStatus());
+ }
+
+ @Test
+ void uploadRejectsABodyWithNoDeterminableBoundary() {
+ Response response = controller.upload(headers("Bearer good", "application/octet-stream"), DOMAIN, REPOSITORY,
+ "not multipart".getBytes(StandardCharsets.UTF_8));
+
+ assertEquals(400, response.getStatus());
+ }
+
+ @Test
+ void uploadRejectsAMultipartBodyMissingNameOrContent() {
+ byte[] body = multipart("boundary123", "version", "1.0.0");
+
+ Response response = controller.upload(
+ headers("Bearer good", "multipart/form-data; boundary=boundary123"), DOMAIN, REPOSITORY, body);
+
+ assertEquals(400, response.getStatus());
+ }
+
+ @Test
+ void uploadRejectsAFileAlreadyListedInTheBackendsSimpleIndex() {
+ backend.createContext("/simple/my-pkg/",
+ exchange -> respond(exchange, 200, "text/html", "<a href=\"my_pkg-1.0.tar.gz\">my_pkg-1.0.tar.gz</a>"));
+ AtomicReference<Boolean> uploadReceived = new AtomicReference<>(false);
+ backend.createContext("/", exchange -> {
+ if ("POST".equals(exchange.getRequestMethod())) {
+ uploadReceived.set(true);
+ }
+ respond(exchange, 200, "text/plain", "");
+ });
+ byte[] body = multipartWithFile("boundary123", "my-pkg", "my_pkg-1.0.tar.gz", "new content");
+
+ Response response = controller.upload(
+ headers("Bearer good", "multipart/form-data; boundary=boundary123"), DOMAIN, REPOSITORY, body);
+
+ assertEquals(409, response.getStatus());
+ assertEquals(false, uploadReceived.get());
+ }
+
+ @Test
+ void uploadOfByteIdenticalContentSucceedsIdempotentlyWithoutForwardingToTheBackend() {
+ String sameContent = "same content";
+ backend.createContext("/simple/my-pkg/",
+ exchange -> respond(exchange, 200, "text/html", "<a href=\"my_pkg-1.0.tar.gz\">my_pkg-1.0.tar.gz</a>"));
+ backend.createContext("/packages/my_pkg-1.0.tar.gz",
+ exchange -> respond(exchange, 200, "application/octet-stream", sameContent));
+ AtomicReference<Boolean> uploadReceived = new AtomicReference<>(false);
+ backend.createContext("/", exchange -> {
+ if ("POST".equals(exchange.getRequestMethod())) {
+ uploadReceived.set(true);
+ }
+ respond(exchange, 200, "text/plain", "");
+ });
+ byte[] body = multipartWithFile("boundary123", "my-pkg", "my_pkg-1.0.tar.gz", sameContent);
+
+ Response response = controller.upload(
+ headers("Bearer good", "multipart/form-data; boundary=boundary123"), DOMAIN, REPOSITORY, body);
+
+ assertEquals(200, response.getStatus());
+ assertEquals(false, uploadReceived.get());
+ }
+
+ @Test
+ void uploadForwardsANewFileToTheBackendWhenNothingConflicts() {
+ backend.createContext("/simple/my-pkg/", exchange -> respond(exchange, 404, "text/plain", ""));
+ AtomicReference<String> receivedBody = new AtomicReference<>();
+ backend.createContext("/", exchange -> {
+ if ("POST".equals(exchange.getRequestMethod())) {
+ receivedBody.set(new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8));
+ }
+ respond(exchange, 200, "text/plain", "");
+ });
+ byte[] body = multipartWithFile("boundary123", "my-pkg", "my_pkg-1.0.tar.gz", "new content");
+
+ Response response = controller.upload(
+ headers("Bearer good", "multipart/form-data; boundary=boundary123"), DOMAIN, REPOSITORY, body);
+
+ assertEquals(200, response.getStatus());
+ assertTrue(receivedBody.get().contains("new content"));
+ }
+
+ @Test
+ void uploadFailsClosedWhenTheExistenceCheckCannotReachTheBackend() {
+ // Nothing listens on this port: the existence check's own connect attempt fails, which
+ // must reject the upload rather than silently treating the failure as "not published yet."
+ when(pypiserver.ensureReady(eq("pypi-repo-id"), any())).thenReturn("http://127.0.0.1:1");
+ byte[] body = multipartWithFile("boundary123", "my-pkg", "my_pkg-1.0.tar.gz", "content");
+
+ assertThrows(IllegalStateException.class, () -> controller.upload(
+ headers("Bearer good", "multipart/form-data; boundary=boundary123"), DOMAIN, REPOSITORY, body));
+ }
+
+ @Test
+ void concurrentUploadsOfDifferentContentForTheSameNewFilenameAreSerialized() throws Exception {
+ CountDownLatch holdFirstCheck = new CountDownLatch(1);
+ CountDownLatch firstCheckEntered = new CountDownLatch(1);
+ AtomicInteger indexHits = new AtomicInteger(0);
+ AtomicBoolean published = new AtomicBoolean(false);
+
+ backend.createContext("/simple/my-pkg/", exchange -> {
+ if (indexHits.incrementAndGet() == 1) {
+ firstCheckEntered.countDown();
+ try {
+ assertTrue(holdFirstCheck.await(5, TimeUnit.SECONDS), "test never released the first check");
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ if (published.get()) {
+ respond(exchange, 200, "text/html", "<a href=\"/packages/my_pkg-1.0.tar.gz\">my_pkg-1.0.tar.gz</a>");
+ } else {
+ respond(exchange, 404, "text/plain", "");
+ }
+ });
+ backend.createContext("/packages/my_pkg-1.0.tar.gz",
+ exchange -> respond(exchange, 200, "application/octet-stream", "content-A"));
+ backend.createContext("/", exchange -> {
+ if ("POST".equals(exchange.getRequestMethod())) {
+ published.set(true);
+ }
+ respond(exchange, 200, "text/plain", "");
+ });
+
+ ExecutorService pool = Executors.newFixedThreadPool(2);
+ try {
+ Future<Response> first = pool.submit(() -> controller.upload(
+ headers("Bearer good", "multipart/form-data; boundary=boundary123"), DOMAIN, REPOSITORY,
+ multipartWithFile("boundary123", "my-pkg", "my_pkg-1.0.tar.gz", "content-A")));
+ assertTrue(firstCheckEntered.await(5, TimeUnit.SECONDS), "first upload never started its check");
+
+ Future<Response> second = pool.submit(() -> controller.upload(
+ headers("Bearer good", "multipart/form-data; boundary=boundary123"), DOMAIN, REPOSITORY,
+ multipartWithFile("boundary123", "my-pkg", "my_pkg-1.0.tar.gz", "content-B")));
+
+ // The second upload must block on the same repository's lock, held by the first, and
+ // so cannot even reach the backend yet: without the lock this would race ahead and
+ // both uploads could observe "not published yet" before either one's POST lands.
+ assertThrows(TimeoutException.class, () -> second.get(500, TimeUnit.MILLISECONDS));
+ assertEquals(1, indexHits.get(), "second upload must not have reached the backend yet");
+
+ holdFirstCheck.countDown();
+ assertEquals(200, first.get(5, TimeUnit.SECONDS).getStatus());
+ assertEquals(409, second.get(5, TimeUnit.SECONDS).getStatus());
+ } finally {
+ pool.shutdownNow();
+ }
+ }
+
+ private static HttpHeaders headers(String authorization) {
+ return headers(authorization, "text/plain");
+ }
+
+ private static HttpHeaders headers(String authorization, String contentType) {
+ HttpHeaders headers = mock(HttpHeaders.class);
+ when(headers.getHeaderString("Authorization")).thenReturn(authorization);
+ when(headers.getHeaderString(HttpHeaders.CONTENT_TYPE)).thenReturn(contentType);
+ return headers;
+ }
+
+ private static byte[] multipart(String boundary, String fieldName, String value) {
+ String body = "--" + boundary + "\r\n"
+ + "Content-Disposition: form-data; name=\"" + fieldName + "\"\r\n\r\n"
+ + value + "\r\n"
+ + "--" + boundary + "--\r\n";
+ return body.getBytes(StandardCharsets.UTF_8);
+ }
+
+ private static byte[] multipartWithFile(String boundary, String packageName, String filename, String content) {
+ String body = "--" + boundary + "\r\n"
+ + "Content-Disposition: form-data; name=\"name\"\r\n\r\n"
+ + packageName + "\r\n"
+ + "--" + boundary + "\r\n"
+ + "Content-Disposition: form-data; name=\"content\"; filename=\"" + filename + "\"\r\n"
+ + "Content-Type: application/octet-stream\r\n\r\n"
+ + content + "\r\n"
+ + "--" + boundary + "--\r\n";
+ return body.getBytes(StandardCharsets.UTF_8);
+ }
+
+ private static void respond(com.sun.net.httpserver.HttpExchange exchange, int status, String contentType,
+ String body) throws java.io.IOException {
+ respond(exchange, status, contentType, body.getBytes(StandardCharsets.UTF_8));
+ }
+
+ private static void respond(com.sun.net.httpserver.HttpExchange exchange, int status, String contentType,
+ byte[] body) throws java.io.IOException {
+ exchange.getResponseHeaders().add("Content-Type", contentType);
+ exchange.sendResponseHeaders(status, body.length);
+ exchange.getResponseBody().write(body);
+ exchange.close();
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java
new file mode 100644
index 0000000000..adc1925ee5
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java
@@ -0,0 +1,402 @@
+package io.github.hectorvent.floci.services.codeartifact;
+
+import io.github.hectorvent.floci.testing.RestAssuredJsonUtils;
+import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.TestProfile;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.MethodOrderer;
+import org.junit.jupiter.api.Order;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestMethodOrder;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static io.restassured.RestAssured.given;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.startsWith;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Runs a real PyPI-shaped upload/download round trip against a live, per-repository pypiserver
+ * container, started by {@link PypiserverSidecarManager} exactly as in production. The upload body
+ * is a hand-built multipart form matching the fields a real {@code twine upload} sends (verified
+ * against a live pypiserver instance and real twine by hand during design), not a captured fixture:
+ * unlike npm's envelope, the parsing itself ({@link io.github.hectorvent.floci.core.common.MultipartFormParser})
+ * is already covered by its own unit tests and by {@code CodeArtifactPypiControllerTest}, so what
+ * this class needs to prove is the real container's behavior, not the parser's fidelity to one
+ * exact client's byte-for-byte output.
+ */
+@QuarkusTest
+@TestProfile(CodeArtifactPypiSidecarProfile.class)
+@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
+class CodeArtifactPypiDockerIntegrationTest {
+
+ private static final String AUTH = "AWS4-HMAC-SHA256 Credential=AKID/20260904/us-east-1/codeartifact/aws4_request";
+ private static final String DOMAIN = "pypi-sidecar-domain";
+ private static final String REPO = "pypi-sidecar-repo";
+ private static final String PACKAGE_NAME = "floci-pypiserver-spike";
+ // Real sdist filenames use underscores where the package name uses hyphens; exercising this
+ // exact mismatch is what proves the PEP 503 normalization in the overwrite-check actually runs,
+ // not just a name that happens to already match its own normalized form.
+ private static final String FILENAME = "floci_pypiserver_spike-1.0.0.tar.gz";
+
+ private static String bearerToken;
+
+ @BeforeAll
+ static void setUp() {
+ CodeArtifactPypiSidecarProfile.requireDockerAndTheSidecarImage();
+ RestAssuredJsonUtils.configureAwsContentTypes();
+ }
+
+ @Test
+ @Order(0)
+ void createDomainRepositoryAndAuthorizationToken() {
+ given().contentType("application/json").header("Authorization", AUTH).body("{}")
+ .post("/v1/domain?domain=" + DOMAIN)
+ .then().statusCode(200);
+ given().contentType("application/json").header("Authorization", AUTH).body("{}")
+ .post("/v1/repository?domain=" + DOMAIN + "&repository=" + REPO)
+ .then().statusCode(200);
+
+ bearerToken = given().header("Authorization", AUTH)
+ .post("/v1/authorization-token?domain=" + DOMAIN)
+ .then().statusCode(200)
+ .extract().jsonPath().getString("authorizationToken");
+ }
+
+ @Test
+ @Order(1)
+ void uploadThenDownloadRoundTripsTheExactBytesThroughTheSimpleIndex() {
+ byte[] content = "real-sdist-bytes".getBytes(StandardCharsets.UTF_8);
+
+ given().header("Authorization", "Bearer " + bearerToken)
+ .contentType("multipart/form-data; boundary=boundary123")
+ .body(twineUploadBody("boundary123", PACKAGE_NAME, FILENAME, content))
+ .post("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/")
+ .then().statusCode(200);
+
+ String indexHtml = given().header("Authorization", "Bearer " + bearerToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/simple/" + PACKAGE_NAME + "/")
+ .then().statusCode(200)
+ .extract().asString();
+ assertTrue(indexHtml.contains(FILENAME), "simple index must list the uploaded file: " + indexHtml);
+
+ byte[] fetched = given().header("Authorization", "Bearer " + bearerToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/packages/" + FILENAME)
+ .then().statusCode(200)
+ .extract().asByteArray();
+ assertEquals(new String(content, StandardCharsets.UTF_8), new String(fetched, StandardCharsets.UTF_8));
+ }
+
+ /**
+ * AWS's own CodeArtifact docs ("Overwriting package assets" in packages-overview.html) say a
+ * republish of an asset whose content is byte-identical to what is already published succeeds
+ * because the operation is idempotent; only different content gets the 409 the next test
+ * covers. A naive "is this filename already listed" check would reject both cases alike, which
+ * would make twine's own retry-after-a-dropped-response behavior fail spuriously.
+ */
+ @Test
+ @Order(2)
+ void reuploadingTheExactSameFileSucceedsIdempotently() {
+ given().header("Authorization", "Bearer " + bearerToken)
+ .contentType("multipart/form-data; boundary=boundary123")
+ .body(twineUploadBody("boundary123", PACKAGE_NAME, FILENAME, "real-sdist-bytes".getBytes(StandardCharsets.UTF_8)))
+ .post("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/")
+ .then().statusCode(200);
+
+ byte[] fetched = given().header("Authorization", "Bearer " + bearerToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/packages/" + FILENAME)
+ .then().statusCode(200)
+ .extract().asByteArray();
+ assertEquals("real-sdist-bytes", new String(fetched, StandardCharsets.UTF_8));
+ }
+
+ @Test
+ @Order(3)
+ void reuploadingTheSameFileIsRejectedAndTheOriginalContentSurvives() {
+ byte[] differentContent = "different-sdist-bytes".getBytes(StandardCharsets.UTF_8);
+
+ given().header("Authorization", "Bearer " + bearerToken)
+ .contentType("multipart/form-data; boundary=boundary123")
+ .body(twineUploadBody("boundary123", PACKAGE_NAME, FILENAME, differentContent))
+ .post("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/")
+ .then().statusCode(409);
+
+ byte[] fetched = given().header("Authorization", "Bearer " + bearerToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/packages/" + FILENAME)
+ .then().statusCode(200)
+ .extract().asByteArray();
+ assertEquals("real-sdist-bytes", new String(fetched, StandardCharsets.UTF_8));
+ }
+
+ @Test
+ @Order(4)
+ void missingPackageAndMissingRepositoryAreNotFound() {
+ given().header("Authorization", "Bearer " + bearerToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/simple/does-not-exist/")
+ .then().statusCode(404);
+
+ given().header("Authorization", "Bearer " + bearerToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/no-such-repo/simple/does-not-exist/")
+ .then().statusCode(404);
+ }
+
+ @Test
+ @Order(5)
+ void repositoriesAreIsolatedFromEachOther() {
+ given().contentType("application/json").header("Authorization", AUTH).body("{}")
+ .post("/v1/repository?domain=" + DOMAIN + "&repository=other-repo")
+ .then().statusCode(200);
+
+ given().header("Authorization", "Bearer " + bearerToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/other-repo/simple/" + PACKAGE_NAME + "/")
+ .then().statusCode(404);
+ }
+
+ @Test
+ @Order(6)
+ void missingOrWrongDomainTokensAreUnauthorized() {
+ // /simple/<package>/, not the bare /simple/ root: AWS doesn't support that path either,
+ // so there's no route for it at all, and an unrouted path 404s before auth ever runs.
+ // Challenges as Basic, not Bearer: real pip and twine both authenticate with HTTP Basic
+ // (username=aws, password=<token>), so this is the scheme a retry would actually honor.
+ given().get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/simple/" + PACKAGE_NAME + "/")
+ .then().statusCode(401)
+ .header("WWW-Authenticate", startsWith("Basic"));
+
+ given().header("Authorization", "Bearer not-a-real-token")
+ .get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/simple/" + PACKAGE_NAME + "/")
+ .then().statusCode(401);
+
+ given().contentType("application/json").header("Authorization", AUTH).body("{}")
+ .post("/v1/domain?domain=other-pypi-token-domain")
+ .then().statusCode(200);
+ String otherDomainToken = given().header("Authorization", AUTH)
+ .post("/v1/authorization-token?domain=other-pypi-token-domain")
+ .then().statusCode(200)
+ .extract().jsonPath().getString("authorizationToken");
+
+ given().header("Authorization", "Bearer " + otherDomainToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/simple/" + PACKAGE_NAME + "/")
+ .then().statusCode(401);
+ }
+
+ @Test
+ @Order(7)
+ void aRealPipClientsBasicAuthCredentialsAreAcceptedWithTheTokenAsThePassword() {
+ String basic = "Basic " + Base64.getEncoder().encodeToString(("aws:" + bearerToken).getBytes(StandardCharsets.UTF_8));
+
+ given().header("Authorization", basic)
+ .get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/simple/" + PACKAGE_NAME + "/")
+ .then().statusCode(200)
+ .body(containsString(FILENAME));
+ }
+
+ @Test
+ @Order(8)
+ void concurrentFirstUseOfANewRepositoryOnlyStartsOneContainer() throws InterruptedException {
+ given().contentType("application/json").header("Authorization", AUTH).body("{}")
+ .post("/v1/repository?domain=" + DOMAIN + "&repository=concurrent-repo")
+ .then().statusCode(200);
+
+ int attempts = 8;
+ ExecutorService pool = Executors.newFixedThreadPool(attempts);
+ CountDownLatch ready = new CountDownLatch(attempts);
+ CountDownLatch go = new CountDownLatch(1);
+ AtomicInteger notFoundResponses = new AtomicInteger();
+ try {
+ for (int i = 0; i < attempts; i++) {
+ pool.submit(() -> {
+ ready.countDown();
+ try {
+ go.await();
+ given().header("Authorization", "Bearer " + bearerToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/concurrent-repo/simple/does-not-exist/")
+ .then().statusCode(404);
+ notFoundResponses.incrementAndGet();
+ } catch (InterruptedException ignored) {
+ Thread.currentThread().interrupt();
+ }
+ });
+ }
+ ready.await(5, TimeUnit.SECONDS);
+ go.countDown();
+ } finally {
+ pool.shutdown();
+ assertTrue(pool.awaitTermination(30, TimeUnit.SECONDS));
+ }
+ assertEquals(attempts, notFoundResponses.get());
+ }
+
+ @Test
+ @Order(9)
+ void recreatingASameNamedRepositoryDoesNotInheritThePreviousOnesPackages() {
+ given().contentType("application/json").header("Authorization", AUTH).body("{}")
+ .post("/v1/repository?domain=" + DOMAIN + "&repository=reused-name")
+ .then().statusCode(200);
+ given().header("Authorization", "Bearer " + bearerToken)
+ .contentType("multipart/form-data; boundary=boundary123")
+ .body(twineUploadBody("boundary123", PACKAGE_NAME, FILENAME, "first-generation".getBytes(StandardCharsets.UTF_8)))
+ .post("/codeartifact/pypi/" + DOMAIN + "/reused-name/")
+ .then().statusCode(200);
+
+ given().header("Authorization", AUTH)
+ .delete("/v1/repository?domain=" + DOMAIN + "&repository=reused-name")
+ .then().statusCode(200);
+ given().contentType("application/json").header("Authorization", AUTH).body("{}")
+ .post("/v1/repository?domain=" + DOMAIN + "&repository=reused-name")
+ .then().statusCode(200);
+
+ given().header("Authorization", "Bearer " + bearerToken)
+ .get("/codeartifact/pypi/" + DOMAIN + "/reused-name/simple/" + PACKAGE_NAME + "/")
+ .then().statusCode(404);
+ }
+
+ @Test
+ @Order(10)
+ void aDomainCreatedInANonDefaultRegionIsServedThroughTheTokensOwnRegion() {
+ String nonDefaultRegionAuth =
+ "AWS4-HMAC-SHA256 Credential=AKID/20260904/us-west-2/codeartifact/aws4_request";
+ String domain = "pypi-sidecar-non-default-region-domain";
+
+ given().contentType("application/json").header("Authorization", nonDefaultRegionAuth).body("{}")
+ .post("/v1/domain?domain=" + domain)
+ .then().statusCode(200);
+ given().contentType("application/json").header("Authorization", nonDefaultRegionAuth).body("{}")
+ .post("/v1/repository?domain=" + domain + "&repository=" + REPO)
+ .then().statusCode(200);
+ String token = given().header("Authorization", nonDefaultRegionAuth)
+ .post("/v1/authorization-token?domain=" + domain)
+ .then().statusCode(200)
+ .extract().jsonPath().getString("authorizationToken");
+
+ given().header("Authorization", "Bearer " + token)
+ .contentType("multipart/form-data; boundary=boundary123")
+ .body(twineUploadBody("boundary123", PACKAGE_NAME, FILENAME, "non-default-region-bytes".getBytes(StandardCharsets.UTF_8)))
+ .post("/codeartifact/pypi/" + domain + "/" + REPO + "/")
+ .then().statusCode(200);
+
+ given().header("Authorization", "Bearer " + token)
+ .get("/codeartifact/pypi/" + domain + "/" + REPO + "/simple/" + PACKAGE_NAME + "/")
+ .then().statusCode(200);
+ }
+
+ @Test
+ @Order(11)
+ void aDomainCreatedUnderANonDefaultAccountIsServedThroughTheTokensOwnAccount() {
+ String otherAccountAuth = "AWS4-HMAC-SHA256 Credential=111122223333/20260904/us-east-1/codeartifact/aws4_request";
+ String domain = "pypi-sidecar-cross-account-domain";
+
+ given().contentType("application/json").header("Authorization", otherAccountAuth).body("{}")
+ .post("/v1/domain?domain=" + domain)
+ .then().statusCode(200);
+ given().contentType("application/json").header("Authorization", otherAccountAuth).body("{}")
+ .post("/v1/repository?domain=" + domain + "&repository=" + REPO)
+ .then().statusCode(200);
+ String token = given().header("Authorization", otherAccountAuth)
+ .post("/v1/authorization-token?domain=" + domain)
+ .then().statusCode(200)
+ .extract().jsonPath().getString("authorizationToken");
+
+ given().header("Authorization", "Bearer " + token)
+ .contentType("multipart/form-data; boundary=boundary123")
+ .body(twineUploadBody("boundary123", PACKAGE_NAME, FILENAME, "cross-account-bytes".getBytes(StandardCharsets.UTF_8)))
+ .post("/codeartifact/pypi/" + domain + "/" + REPO + "/")
+ .then().statusCode(200);
+
+ given().header("Authorization", "Bearer " + token)
+ .get("/codeartifact/pypi/" + domain + "/" + REPO + "/simple/" + PACKAGE_NAME + "/")
+ .then().statusCode(200);
+ }
+
+ /**
+ * Proves the actual mechanism, not just that {@code PypiserverSidecarManager} has a method
+ * named right: {@code ContainerTeardowns.stopAll} runs every {@code ContainerTeardown} on
+ * {@code /state/reset}, and this is what stops a live container.
+ *
+ * <p>Ordered before every other test, for the same reason {@code CodeArtifactNpmDockerIntegrationTest}'s
+ * equivalent reset test runs first: a reset stops every container this process's pool is
+ * tracking, not just the one this test starts, so running it after the other tests would sweep
+ * up their containers too and break the baseline comparison for a reason unrelated to whether
+ * this test's own container actually stopped.
+ */
+ @Test
+ @Order(-1)
+ void stateResetStopsTheRepositorysPypiserverContainer() throws Exception {
+ int baseline = runningPypiserverTestContainerCount();
+
+ String domain = "pypi-sidecar-reset-domain";
+ given().contentType("application/json").header("Authorization", AUTH).body("{}")
+ .post("/v1/domain?domain=" + domain)
+ .then().statusCode(200);
+ given().contentType("application/json").header("Authorization", AUTH).body("{}")
+ .post("/v1/repository?domain=" + domain + "&repository=reset-repo")
+ .then().statusCode(200);
+ String token = given().header("Authorization", AUTH)
+ .post("/v1/authorization-token?domain=" + domain)
+ .then().statusCode(200)
+ .extract().jsonPath().getString("authorizationToken");
+ // A 404 read still starts the container; only a missing token skips it.
+ given().header("Authorization", "Bearer " + token)
+ .get("/codeartifact/pypi/" + domain + "/reset-repo/simple/does-not-exist/")
+ .then().statusCode(404);
+ assertEquals(baseline + 1, runningPypiserverTestContainerCount(),
+ "expected reset-repo's pypiserver container to be running before reset");
+
+ given().post("/_floci/state/reset").then().statusCode(200);
+
+ assertEquals(baseline, runningPypiserverTestContainerCount(),
+ "state reset must stop reset-repo's pypiserver container");
+ }
+
+ private static int runningPypiserverTestContainerCount() throws IOException, InterruptedException {
+ Process process = new ProcessBuilder("docker", "ps", "-q",
+ "--filter", "name=floci-aws-codeartifact-pypi-test-pypiserver-")
+ .redirectErrorStream(true).start();
+ String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
+ process.waitFor();
+ return output.isBlank() ? 0 : (int) output.lines().count();
+ }
+
+ /**
+ * The fields a real {@code twine upload} sends (verified by hand against a live pypiserver
+ * instance during design); pypiserver, and this controller's own parsing, only look at
+ * {@code name} and the {@code content} file part, so the rest are included for realism only.
+ */
+ private static byte[] twineUploadBody(String boundary, String packageName, String filename, byte[] content) {
+ String preamble = "--" + boundary + "\r\n"
+ + "Content-Disposition: form-data; name=\":action\"\r\n\r\n"
+ + "file_upload\r\n"
+ + "--" + boundary + "\r\n"
+ + "Content-Disposition: form-data; name=\"protocol_version\"\r\n\r\n"
+ + "1\r\n"
+ + "--" + boundary + "\r\n"
+ + "Content-Disposition: form-data; name=\"name\"\r\n\r\n"
+ + packageName + "\r\n"
+ + "--" + boundary + "\r\n"
+ + "Content-Disposition: form-data; name=\"version\"\r\n\r\n"
+ + "1.0.0\r\n"
+ + "--" + boundary + "\r\n"
+ + "Content-Disposition: form-data; name=\"filetype\"\r\n\r\n"
+ + "sdist\r\n"
+ + "--" + boundary + "\r\n"
+ + "Content-Disposition: form-data; name=\"content\"; filename=\"" + filename + "\"\r\n"
+ + "Content-Type: application/octet-stream\r\n\r\n";
+ String closing = "\r\n--" + boundary + "--\r\n";
+ byte[] preambleBytes = preamble.getBytes(StandardCharsets.UTF_8);
+ byte[] closingBytes = closing.getBytes(StandardCharsets.UTF_8);
+ byte[] body = new byte[preambleBytes.length + content.length + closingBytes.length];
+ System.arraycopy(preambleBytes, 0, body, 0, preambleBytes.length);
+ System.arraycopy(content, 0, body, preambleBytes.length, content.length);
+ System.arraycopy(closingBytes, 0, body, preambleBytes.length + content.length, closingBytes.length);
+ return body;
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiSidecarProfile.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiSidecarProfile.java
new file mode 100644
index 0000000000..d247a98e41
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiSidecarProfile.java
@@ -0,0 +1,52 @@
+package io.github.hectorvent.floci.services.codeartifact;
+
+import io.quarkus.test.junit.QuarkusTestProfile;
+import org.eclipse.microprofile.config.ConfigProvider;
+import org.junit.jupiter.api.Assumptions;
+
+import java.util.Map;
+
+/**
+ * Shared by every CodeArtifact integration test that needs a real pypiserver sidecar container,
+ * started by {@link PypiserverSidecarManager} exactly as in production. Namespaces containers as
+ * {@code floci-codeartifact-pypi-test-*} so the manager's remove-before-start never touches a
+ * developer's real running Floci. Needs Docker; skipped without it. Locally it is also skipped
+ * when the pinned image is not present, so a developer without registry access is not stuck; in
+ * CI a missing image is a failure, since a vanished tag must not pass silently.
+ */
+public class CodeArtifactPypiSidecarProfile implements QuarkusTestProfile {
+
+ @Override
+ public Map<String, String> getConfigOverrides() {
+ return Map.of("floci.docker.resource-namespace", "codeartifact-pypi-test");
+ }
+
+ public static void requireDockerAndTheSidecarImage() {
+ Assumptions.assumeTrue(isDockerAvailable(), "Docker daemon must be available for CodeArtifact pypi "
+ + "sidecar integration tests");
+ String image = ConfigProvider.getConfig().getValue("floci.services.codeartifact.pypi-image", String.class);
+ Assumptions.assumeTrue(imageUsable(image), "pypiserver sidecar image " + image + " is not present locally");
+ }
+
+ private static boolean isDockerAvailable() {
+ return run("docker", "version", "--format", "{{.Server.Version}}");
+ }
+
+ private static boolean imageUsable(String image) {
+ if ("true".equals(System.getenv("CI"))) {
+ return true;
+ }
+ return run("docker", "image", "inspect", image);
+ }
+
+ private static boolean run(String... command) {
+ try {
+ Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
+ return process.waitFor() == 0;
+ } catch (Exception ignored) {
+ // Docker (or the image) isn't available; the caller turns this into a skipped test,
+ // not a failure, so there is nothing more useful to log here.
+ return false;
+ }
+ }
+}
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java
index 1270056b76..9cede4acc8 100644
--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java
@@ -300,7 +300,7 @@ class CodeArtifactServiceTest {
service.createDomain(REGION, "dom", null, Map.of());
CodeArtifactRepository created = service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());
- assertEquals(Set.of("maven", "npm"), created.getSidecarContainerIds().keySet());
+ assertEquals(Set.of("maven", "npm", "pypi"), created.getSidecarContainerIds().keySet());
created.getSidecarContainerIds().values()
.forEach(id -> assertTrue(id != null && !id.isBlank()));
}
@@ -313,7 +313,7 @@ class CodeArtifactServiceTest {
service.deleteRepository(REGION, "dom", null, "repo");
CodeArtifactRepository recreated = service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());
- for (String format : Set.of("maven", "npm")) {
+ for (String format : Set.of("maven", "npm", "pypi")) {
assertTrue(
!first.getSidecarContainerIds().get(format).equals(recreated.getSidecarContainerIds().get(format)),
"a recreated repository must never reuse the previous one's container id for a format, "
@@ -506,6 +506,9 @@ class CodeArtifactServiceTest {
String endpoint = service.getRepositoryEndpoint(REGION, "dom", null, "repo", "npm", null);
assertEquals("http://localhost:4566/codeartifact/npm/dom/repo/", endpoint);
+ String pypiEndpoint = service.getRepositoryEndpoint(REGION, "dom", null, "repo", "pypi", null);
+ assertEquals("http://localhost:4566/codeartifact/pypi/dom/repo/", pypiEndpoint);
+
AwsException e = assertThrows(AwsException.class,
() -> service.getRepositoryEndpoint(REGION, "dom", null, "repo", "not-a-format", null));
assertEquals("ValidationException", e.getErrorCode());
diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManagerTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManagerTest.java
new file mode 100644
index 0000000000..24406cf5c8
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/PypiserverSidecarManagerTest.java
@@ -0,0 +1,73 @@
+package io.github.hectorvent.floci.services.codeartifact;
+
+import io.github.hectorvent.floci.config.EmulatorConfig;
+import io.github.hectorvent.floci.core.common.ContainerTeardown;
+import io.github.hectorvent.floci.core.common.docker.ContainerBuilder;
+import io.github.hectorvent.floci.core.common.docker.ContainerLifecycleManager;
+import io.github.hectorvent.floci.core.common.docker.PerKeyContainerPool;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoInteractions;
+
+/**
+ * Same reasoning as {@code VerdaccioSidecarManagerTest}: {@link PypiserverSidecarManager}'s own
+ * {@link PerKeyContainerPool#stopAll()} behavior is already covered by {@code PerKeyContainerPoolTest},
+ * so what matters here is the wiring, that the manager genuinely implements {@link ContainerTeardown}
+ * and hands the pool container ids it actually started.
+ */
+class PypiserverSidecarManagerTest {
+
+ private final ContainerBuilder containerBuilder = mock(ContainerBuilder.class);
+ private final ContainerLifecycleManager lifecycleManager = mock(ContainerLifecycleManager.class);
+ private final EmulatorConfig config = mock(EmulatorConfig.class);
+
+ @Test
+ void implementsContainerTeardownSoStateResetAndNukeCanFindIt() {
+ assertInstanceOf(ContainerTeardown.class, manager());
+ }
+
+ @Test
+ void stopManagedContainersStopsEveryContainerThePoolIsTracking() throws Exception {
+ PypiserverSidecarManager manager = manager();
+ seedPooledContainer(manager, "pypi-repo-1", "tracked-pypiserver-1");
+ seedPooledContainer(manager, "pypi-repo-2", "tracked-pypiserver-2");
+
+ manager.stopManagedContainers();
+
+ verify(lifecycleManager).stopAndRemove("tracked-pypiserver-1", null);
+ verify(lifecycleManager).stopAndRemove("tracked-pypiserver-2", null);
+ }
+
+ @Test
+ void stopManagedContainersIsANoOpWhenNothingWasEverStarted() {
+ PypiserverSidecarManager manager = manager();
+
+ manager.stopManagedContainers();
+
+ verifyNoInteractions(lifecycleManager);
+ }
+
+ private PypiserverSidecarManager manager() {
+ return new PypiserverSidecarManager(containerBuilder, lifecycleManager, config);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static void seedPooledContainer(PypiserverSidecarManager manager, String key, String containerId)
+ throws Exception {
+ Field poolField = PypiserverSidecarManager.class.getDeclaredField("pool");
+ poolField.setAccessible(true);
+ PerKeyContainerPool pool = (PerKeyContainerPool) poolField.get(manager);
+
+ Field containersField = PerKeyContainerPool.class.getDeclaredField("containers");
+ containersField.setAccessible(true);
+ ConcurrentHashMap<String, PerKeyContainerPool.StartedContainer> containers =
+ (ConcurrentHashMap<String, PerKeyContainerPool.StartedContainer>) containersField.get(pool);
+ containers.put(key, new PerKeyContainerPool.StartedContainer(containerId, "http://127.0.0.1:1"));
+ }
+}