Light, fluffy, and always free - The AWS Local Emulator alternative
feat(codeartifact): compute DescribeDomain's assetSizeBytes for every format (#5484)
Priamo Germosen6b0d0bc37803
diff --git a/docs/services/codeartifact.md b/docs/services/codeartifact.mdindex 62eca7b5f6..55bac0ac63 100644--- a/docs/services/codeartifact.md+++ b/docs/services/codeartifact.md@@ -19,7 +19,7 @@ the remaining formats (NuGet, etc.) have no real proxy behind them yet.| --- | --- || `CreateDomain` | Creates a domain (max 10 per account per Region), optionally with a KMS encryption key and initial tags. || `DeleteDomain` | Deletes a domain; fails with `ConflictException` while it still contains repositories, and with `ResourceNotFoundException` for a missing domain (as AWS does, although the API reference does not list it). |-| `DescribeDomain` | Returns a domain's full description, including its repository count. |+| `DescribeDomain` | Returns a domain's full description, including its repository count and the total size of its assets (generic, Maven, npm, and pypi; see Known limitations). || `ListDomains` | Lists domain summaries for the account and Region, paginated. || `GetAuthorizationToken` | Issues a bearer token scoped to a domain, valid for 0 (12 hours) or 900-43200 seconds, required by the `maven` and `npm` repository endpoints. || `PutDomainPermissionsPolicy` | Attaches or replaces a domain's resource policy, versioned by `policyRevision`. |@@ -294,6 +294,35 @@ state.yet check this setting. `upstream` is stored and reported faithfully for every format but isnever enforced for any of them, since Floci has no upstream ingestion path to block in the firstplace.+- **`DescribeDomain`'s `assetSizeBytes` only counts what was published after this field started+ being computed.** Generic-format bytes are summed live from the still-existing+ `PublishPackageVersion` records, so they're always current. Maven, npm, and pypi have no such+ record to sum (the same gap `DescribePackageVersion` has), so each proxy reports its own+ successfully-written byte count directly; a package published through one of those three formats+ before this accounting existed was never reported and is not retroactively counted. A genuine+ conflict or an idempotent no-op retry (any of the three formats' own duplicate-publish+ protection) correctly adds nothing either way. Both `DeleteRepository` and `DeletePackage` clean+ up every format's own records and byte counts (and, for `DeleteRepository`, the per-version+ asset content underneath them), so a repository or package recreated under the same name never+ inherits a predecessor's bytes. Maven/npm/pypi byte counts live only in memory, the same as the+ sidecar content they describe (see the restart limitation below), not through `StorageFactory`+ like every durable store here: the content these bytes count does not survive a restart either,+ so neither does the count.+- **A whole-package `npm unpublish` is accounted for; unpublishing a single version of a+ multi-version package is not.** The pinned Verdaccio image is configured to honor `npm+ unpublish` natively (`unpublish: $all`), bypassing `DeletePackage` entirely; the npm data plane+ detects that one specific wire shape (`DELETE .../-rev/<revision>`) and cleans up this service's+ own byte count and deletion fence once Verdaccio confirms it. Removing a single version from a+ multi-version package is a different, harder-to-detect shape (a `PUT` carrying an updated+ packument, indistinguishable at the wire level from an ordinary republish), and is not detected;+ since this accounting is per-package, not per-version, there is no narrower count a single+ version's removal could subtract from in the first place.+- **A `maven-metadata.xml` upload for a versioned snapshot (`.../<artifact>/<version>/+ maven-metadata.xml`) is counted under a package name built from its version string, not its real+ artifact name.** Every real `mvn deploy` writes a group/artifact-level `maven-metadata.xml`+ too, which this correctly attributes to the real artifact; only the rarer, version-scoped+ snapshot form gets a package name this way. The bytes are still counted toward the domain total+ either way, just under an internal accounting bucket nothing else ever looks up by name.See the [CodeArtifact API Reference](https://docs.aws.amazon.com/codeartifact/latest/APIReference/Welcome.html).diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactController.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactController.javaindex 155212318b..0adc31ec61 100644--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactController.java+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactController.java@@ -443,7 +443,7 @@ public class CodeArtifactController {private ObjectNode domainDescription(DomainView view) {ObjectNode node = objectMapper.createObjectNode();node.put("arn", view.domain().getArn());- node.put("assetSizeBytes", 0L);+ node.put("assetSizeBytes", view.assetSizeBytes());node.put("createdTime", view.domain().getCreatedTime());node.put("encryptionKey", view.domain().getEncryptionKey());node.put("name", view.domain().getName());diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenController.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenController.javaindex bb73185e8a..42719cbe12 100644--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenController.java+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenController.java@@ -18,6 +18,9 @@ import jakarta.ws.rs.core.HttpHeaders;import jakarta.ws.rs.core.MediaType;import jakarta.ws.rs.core.Response;+import java.util.Arrays;+import java.util.Optional;+/*** Serves the real Maven repository wire protocol behind the URL {@code GetRepositoryEndpoint}* returns for the {@code maven} format: plain GET/PUT/HEAD over a GAV path, not a CodeArtifact@@ -60,16 +63,65 @@ public class CodeArtifactMavenController {if (content.length > CodeArtifactService.MAX_ASSET_FILE_SIZE_BYTES) {return Response.status(413).build();}- String repoId = requireRepositoryId(headers, domain, repository);- int status = reposilite.deployArtifact(repoId, gav, content);- return Response.status(status).build();+ MavenRequestScope scope = requireScope(headers, domain, repository);+ Optional<String[]> coordinate = parseNamespaceAndPackageName(gav);+ // Snapshotted before the real write, not after: see+ // CodeArtifactService#currentContainerBackedPackageFence's own javadoc for why the order+ // matters.+ Optional<Long> fenceSnapshot = coordinate.map(c -> service.currentContainerBackedPackageFence(+ scope.region(), domain, scope.owner(), repository, "maven", c[0], c[1]));+ ReposiliteSidecarClient.DeployOutcome outcome = reposilite.deployArtifact(scope.repoId(), gav, content);+ if (outcome.wroteNewContent()) {+ coordinate.ifPresent(c -> service.recordContainerBackedAssetBytes(scope.region(), domain, scope.owner(),+ repository, "maven", c[0], c[1], content.length, fenceSnapshot.get()));+ }+ return Response.status(outcome.status()).build();+ }++ /**+ * Splits a GAV path ({@code <groupId-as-slashes>/<artifactId>/<version>/<filename>}, the real+ * Maven repository layout) into CodeArtifact's Maven namespace (dot-joined groupId) and+ * package name (artifactId), the inverse of what {@code ReposiliteSidecarClient}'s own+ * namespace/packageName-aware methods already build. Only ever used for byte accounting, a+ * side channel that must never fail a publish that would otherwise succeed, so a path with too+ * few segments to be a well-formed GAV is reported as unknown rather than thrown.+ *+ * <p>{@code maven-metadata.xml} (and its {@code .sha1}/{@code .md5}/{@code .sha256} checksum+ * siblings) needs its own rule, not the ordinary one below: every real {@code mvn deploy} also+ * PUTs one straight to the group/artifact directory, with no version segment at all, e.g.+ * {@code com/example/myartifact/maven-metadata.xml}. That path has the same segment count as+ * the shortest real versioned GAV ({@code group/artifact/version/filename}), so segment count+ * alone can't tell the two apart; the filename itself can, and already does, via+ * {@link #isMavenMetadataFile}. Once that is known, the artifactId is always the segment+ * directly before it, no version segment in between, so this takes one fewer trailing segment+ * than the ordinary rule does. A versioned snapshot's own {@code .../<version>/+ * maven-metadata.xml} is the one real path this still gets slightly wrong, reading the version+ * string itself as the package name instead of skipping over it; accepted rather than chased+ * further, since this is a byte count summed at the domain level only (never read back by+ * package), and this one rare case still adds its real byte count to the right total under a+ * harmless, if misnamed, bucket nothing else will ever look up by that name.+ */+ private static Optional<String[]> parseNamespaceAndPackageName(String gav) {+ String[] segments = gav.split("/");+ int packageNameIndex = isMavenMetadataFile(segments[segments.length - 1])+ ? segments.length - 2 : segments.length - 3;+ if (packageNameIndex < 1) {+ return Optional.empty();+ }+ String packageName = segments[packageNameIndex];+ String namespace = String.join(".", Arrays.asList(segments).subList(0, packageNameIndex));+ return Optional.of(new String[] {namespace, packageName});+ }++ private static boolean isMavenMetadataFile(String filename) {+ return filename.equals("maven-metadata.xml") || filename.startsWith("maven-metadata.xml.");}@GET@Produces(MediaType.WILDCARD)public Response fetch(@Context HttpHeaders headers, @PathParam("domain") String domain,@PathParam("repository") String repository, @PathParam("gav") String gav) {- String repoId = requireRepositoryId(headers, domain, repository);+ String repoId = requireScope(headers, domain, repository).repoId();return reposilite.fetchArtifact(repoId, gav).map(this::toResponse).orElseGet(() -> Response.status(Response.Status.NOT_FOUND).build());@@ -78,7 +130,7 @@ public class CodeArtifactMavenController {@HEADpublic Response head(@Context HttpHeaders headers, @PathParam("domain") String domain,@PathParam("repository") String repository, @PathParam("gav") String gav) {- String repoId = requireRepositoryId(headers, domain, repository);+ String repoId = requireScope(headers, domain, repository).repoId();return reposilite.artifactExists(repoId, gav)? Response.ok().build(): Response.status(Response.Status.NOT_FOUND).build();@@ -88,7 +140,9 @@ public class CodeArtifactMavenController {return Response.ok(artifact.content(), artifact.contentType()).build();}- private String requireRepositoryId(HttpHeaders headers, String domain, String repository) {+ private record MavenRequestScope(String repoId, String region, String owner) {}++ private MavenRequestScope requireScope(HttpHeaders headers, String domain, String repository) {String token = CodeArtifactTokenExtractor.extractToken(headers);AuthorizationTokenScope scope = service.resolveAuthorizationToken(token, domain).orElseThrow(() -> new NotAuthorizedException("Basic realm=\"floci-codeartifact\""));@@ -99,6 +153,6 @@ public class CodeArtifactMavenController {throw new NotFoundException();}reposilite.ensureRepository(repoId);- return repoId;+ return new MavenRequestScope(repoId, scope.region(), scope.owner());}}diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDataPlane.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDataPlane.javaindex ac61697d7a..bbe73b5521 100644--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDataPlane.java+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDataPlane.java@@ -39,6 +39,7 @@ import java.util.Optional;import java.util.Set;import java.util.concurrent.atomic.AtomicBoolean;import java.util.concurrent.atomic.AtomicReference;+import java.util.function.IntConsumer;/*** Proxies the real npm registry protocol through Floci to the per-repository Verdaccio container@@ -133,7 +134,7 @@ public class CodeArtifactNpmDataPlane {AtomicReference<PreflightGate.Waiter> queued = new AtomicReference<>();context.response().closeHandler(ignored -> preflights.abandon(queued.get()));PreflightGate.Waiter waiter = preflights.acquire(() -> eventLoop.runOnContext(ignored ->- handlePublish(context, npmRequest, npmRepositoryId)));+ handlePublish(context, npmRequest, npmRepositoryId, scope.get())));if (waiter == null) {context.request().resume();context.response().putHeader("Retry-After", "1").setStatusCode(429).end();@@ -145,6 +146,9 @@ public class CodeArtifactNpmDataPlane {context.request().pause();String repoId = npmRepositoryId;+ AuthorizationTokenScope tokenScope = scope.get();+ Optional<PackageIdentity> unpublishTarget = nativeUnpublishTarget(context.request().method(),+ npmRequest.rest());vertx.<String>executeBlocking(promise -> promise.complete(ensureBackendReady(repoId, npmRequest))).onComplete(result -> {if (result.failed()) {@@ -152,7 +156,14 @@ public class CodeArtifactNpmDataPlane {context.response().setStatusCode(503).end();return;}- proxy(context, result.result(), npmRequest.rest() + querySuffix(context));+ proxy(context, result.result(), npmRequest.rest() + querySuffix(context), status ->+ unpublishTarget.ifPresent(identity -> {+ if (status == 200 || status == 201) {+ service.forgetContainerBackedPackage(tokenScope.region(), npmRequest.domain(),+ tokenScope.owner(), npmRequest.repository(), "npm", identity.namespace(),+ identity.packageName());+ }+ }));});}@@ -171,6 +182,34 @@ public class CodeArtifactNpmDataPlane {return method == HttpMethod.PUT && rest.length() > 1 && rest.indexOf('/', 1) < 0;}+ /**+ * A real {@code npm unpublish <pkg> --force} (whole package, not one version) is a+ * {@code DELETE} straight to {@code <packagePath>/-rev/<revision>}, the exact mechanism+ * {@code VerdaccioSidecarClient.deletePackage} itself uses internally. The pinned Verdaccio+ * image is configured with {@code unpublish: $all}, so it honors this natively; this proxy+ * only ever forwards it (see {@link #handle}'s own fallback to {@link #proxy}), with no path+ * through {@link CodeArtifactService#deletePackage} at all. Detected here purely so that same+ * fallback can still clean up this service's own accounting afterward, via+ * {@link CodeArtifactService#forgetContainerBackedPackage}, once Verdaccio's response+ * confirms the delete actually happened.+ *+ * <p>Unpublishing a single version of a multi-version package is a different, harder-to-detect+ * shape (a {@code PUT} carrying an updated packument with that version removed, the same+ * method and the same bare-package path a real publish uses), not handled here: this+ * service's own accounting is per-package, not per-version, so there is no narrower count a+ * single version's removal could subtract from in the first place.+ */+ private static Optional<PackageIdentity> nativeUnpublishTarget(HttpMethod method, String rest) {+ if (method != HttpMethod.DELETE) {+ return Optional.empty();+ }+ int revIndex = rest.indexOf("/-rev/");+ if (revIndex < 1) {+ return Optional.empty();+ }+ return Optional.of(packageIdentityFromPath(rest.substring(0, revIndex)));+ }+/*** Real npm registries and the pinned Verdaccio image both reject a republish of an existing* version with 409, even for byte-identical content. The AWS CodeArtifact User Guide states the@@ -180,7 +219,8 @@ public class CodeArtifactNpmDataPlane {* stream, so its size never costs heap. Only when every version and dist-tag matches does this* answer 200 without reaching Verdaccio; anything else forwards the spooled body unchanged.*/- private void handlePublish(RoutingContext context, NpmRequest npmRequest, String npmRepositoryId) {+ private void handlePublish(RoutingContext context, NpmRequest npmRequest, String npmRepositoryId,+ AuthorizationTokenScope scope) {AtomicBoolean finished = new AtomicBoolean();AtomicReference<String> spool = new AtomicReference<>();AtomicReference<AsyncFile> reading = new AtomicReference<>();@@ -232,15 +272,18 @@ public class CodeArtifactNpmDataPlane {return;}String backendBaseUrl = readyResult.result();- vertx.<Boolean>executeBlocking(promise ->- promise.complete(confirmedRepublish(backendBaseUrl, npmRequest, path)),+ vertx.<RepublishCheck>executeBlocking(promise ->+ promise.complete(checkRepublish(backendBaseUrl, npmRequest, path)),false)- .onComplete(matchResult -> {+ .onComplete(checkResult -> {if (context.response().closed()) {finish.run();return;}- if (matchResult.succeeded() && matchResult.result()) {+ RepublishCheck check = checkResult.succeeded() ? checkResult.result()+ : new RepublishCheck(false, 0L,+ packageIdentityFromPath(npmRequest.rest()));+ if (check.confirmed()) {// "ok" is a message string in a real Verdaccio response// (e.g. "created new package"), not a boolean; matching// that shape in case any client inspects it for display@@ -252,6 +295,17 @@ public class CodeArtifactNpmDataPlane {finish.run();return;}+ // Snapshotted before the real write (the forwarded+ // request below hasn't even been issued yet), not+ // after: see CodeArtifactService#+ // currentContainerBackedPackageFence's own javadoc for+ // why the order matters, and why npm needs it most of+ // all three formats, with this call's own response+ // arriving on a delayed callback rather than inline.+ long fenceSnapshot = service.currentContainerBackedPackageFence(+ scope.region(), npmRequest.domain(), scope.owner(),+ npmRequest.repository(), "npm", check.identity().namespace(),+ check.identity().packageName());vertx.fileSystem().open(path, new OpenOptions().setRead(true)).onComplete(reopened -> {if (reopened.failed()) {@@ -264,7 +318,9 @@ public class CodeArtifactNpmDataPlane {context.response().closeHandler(ignored -> finish.run());forwardSpooled(context, backendBaseUrl,npmRequest.rest() + querySuffix(context),- reopened.result());+ reopened.result(), scope, npmRequest.domain(),+ npmRequest.repository(), check.identity(),+ check.totalAttachmentBytes(), fenceSnapshot);});});});@@ -346,13 +402,19 @@ public class CodeArtifactNpmDataPlane {}/**- * {@code true} only when every version in the publish envelope already exists on Verdaccio with- * the same integrity, every dist-tag the envelope sends already points where it says, and every- * attachment in the envelope is one of those versions' tarballs. Client bytes are checked against- * the integrity the client itself claims, so a mismatch between the two falls through too.- * Anything unreadable or inconsistent is not confirmable and forwards unchanged.+ * {@code confirmed} is {@code true} only when every version in the publish envelope already+ * exists on Verdaccio with the same integrity, every dist-tag the envelope sends already points+ * where it says, and every attachment in the envelope is one of those versions' tarballs.+ * Client bytes are checked against the integrity the client itself claims, so a mismatch+ * between the two falls through too. Anything unreadable or inconsistent is not confirmable and+ * forwards unchanged. {@code totalAttachmentBytes} is the real decoded byte count of every+ * attachment the envelope carries, regardless of {@code confirmed}: the caller only ever uses+ * it when forwarding an unconfirmed publish actually succeeds, since a confirmed republish+ * writes no new bytes at all.*/- private boolean confirmedRepublish(String backendBaseUrl, NpmRequest npmRequest, String spoolPath) {+ private record RepublishCheck(boolean confirmed, long totalAttachmentBytes, PackageIdentity identity) {}++ private RepublishCheck checkRepublish(String backendBaseUrl, NpmRequest npmRequest, String spoolPath) {PackageIdentity identity = packageIdentityFromPath(npmRequest.rest());NpmPublishEnvelopeScanner.Envelope envelope;try (InputStream in = new BufferedInputStream(Files.newInputStream(Path.of(spoolPath)))) {@@ -360,8 +422,11 @@ public class CodeArtifactNpmDataPlane {} catch (IOException e) {LOG.debugv(e, "Publish for {0}/{1} is not a confirmable republish: {2}",npmRequest.domain(), npmRequest.repository(), e.getMessage());- return false;+ return new RepublishCheck(false, 0L, identity);}+ long totalAttachmentBytes = envelope.attachments().values().stream()+ .mapToLong(NpmPublishEnvelopeScanner.Attachment::size)+ .sum();Optional<JsonNode> storedDocument;try {storedDocument = verdaccioClient.fetchPackageDocument(backendBaseUrl, identity.namespace(),@@ -369,25 +434,25 @@ public class CodeArtifactNpmDataPlane {} catch (IllegalStateException e) {LOG.warnv(e, "Could not check {0}/{1} for an existing match before publishing: {2}",npmRequest.domain(), npmRequest.repository(), e.getMessage());- return false;+ return new RepublishCheck(false, totalAttachmentBytes, identity);}if (storedDocument.isEmpty() || !isConfirmedRepublish(envelope, storedDocument.get(), identity)) {- return false;+ return new RepublishCheck(false, totalAttachmentBytes, identity);}for (NpmPublishEnvelopeScanner.Version version : envelope.versions().values()) {try {Optional<String> stored = verdaccioClient.storedTarballIntegrity(backendBaseUrl,identity.namespace(), identity.packageName(), tarballFilename(version.tarball()));if (stored.isEmpty() || !stored.get().equals(version.integrity())) {- return false;+ return new RepublishCheck(false, totalAttachmentBytes, identity);}} catch (IllegalStateException e) {LOG.warnv(e, "Could not verify the stored tarball of {0}/{1} before publishing: {2}",npmRequest.domain(), npmRequest.repository(), e.getMessage());- return false;+ return new RepublishCheck(false, totalAttachmentBytes, identity);}}- return true;+ return new RepublishCheck(true, totalAttachmentBytes, identity);}private static boolean isConfirmedRepublish(NpmPublishEnvelopeScanner.Envelope envelope, JsonNode stored,@@ -417,12 +482,13 @@ public class CodeArtifactNpmDataPlane {|| !sent.integrity().equals(storedVersion.path("dist").path("integrity").asText(""))) {return false;}- if (!sent.integrity().equals(envelope.attachmentIntegrity().get(filename))) {+ NpmPublishEnvelopeScanner.Attachment attachment = envelope.attachments().get(filename);+ if (attachment == null || !sent.integrity().equals(attachment.integrity())) {return false;}tarballs.add(filename);}- for (String attachment : envelope.attachmentIntegrity().keySet()) {+ for (String attachment : envelope.attachments().keySet()) {if (!tarballs.contains(attachment)) {return false;}@@ -462,8 +528,19 @@ public class CodeArtifactNpmDataPlane {}}+ /**+ * {@code identity}/{@code totalAttachmentBytes} are only used to record real published bytes+ * once Verdaccio's own response confirms the publish actually succeeded (200 or 201): a+ * forward that Verdaccio itself rejects (409 on a genuine conflict, or anything else) writes+ * nothing, and must not be counted as though it had. {@code totalAttachmentBytes} is already 0+ * when the envelope couldn't be parsed (see {@link #checkRepublish}), so this is a no-op in+ * that case regardless of what Verdaccio answers. {@code fenceSnapshot} is whatever {@link+ * CodeArtifactService#currentContainerBackedPackageFence} returned at the caller's own earlier+ * snapshot point, before this request was ever forwarded.+ */private void forwardSpooled(RoutingContext context, String backendBaseUrl, String backendPath,- ReadStream<Buffer> body) {+ ReadStream<Buffer> body, AuthorizationTokenScope scope, String domain, String repository,+ PackageIdentity identity, long totalAttachmentBytes, long fenceSnapshot) {URI backend = URI.create(backendBaseUrl);RequestOptions options = new RequestOptions().setHost(backend.getHost())@@ -482,14 +559,43 @@ public class CodeArtifactNpmDataPlane {respondUnlessGone(context, 502);return;}- copyResponseHeaders(context, upstreamResponse.result());- upstreamResponse.result().pipeTo(context.response());+ HttpClientResponse response = upstreamResponse.result();+ copyResponseHeaders(context, response);+ response.pipeTo(context.response());+ // Piped on first, not after: the client is never made to wait on the call below.+ // That call is fast and in-memory-only now (no sidecar round trip of its own,+ // unlike the one this replaced), but it still runs off the event loop, in+ // executeBlocking, on principle: CodeArtifactService is a plain CDI bean Vert.x+ // knows nothing about, so nothing here can guarantee it stays that way forever,+ // and the cost of getting this wrong (stalling every request on this event loop,+ // not just this one) is high enough that "it happens to be cheap today" is not a+ // reason to call a synchronized service method straight from a response callback.+ if (totalAttachmentBytes > 0 && (response.statusCode() == 200 || response.statusCode() == 201)) {+ vertx.<Void>executeBlocking(promise -> {+ service.recordContainerBackedAssetBytes(scope.region(), domain, scope.owner(), repository,+ "npm", identity.namespace(), identity.packageName(), totalAttachmentBytes,+ fenceSnapshot);+ promise.complete();+ }, false).onFailure(e -> LOG.warnv(e, "Could not record npm asset bytes for {0}/{1} package "+ + "''{2}'': {3}", domain, repository, identity.packageName(), e.getMessage()));+ }});clientReq.send(body).onFailure(ignored -> respondUnlessGone(context, 502));});}private void proxy(RoutingContext context, String backendBaseUrl, String backendPath) {+ proxy(context, backendBaseUrl, backendPath, status -> { });+ }++ /**+ * {@code onSuccessStatus} runs only once Verdaccio's own response is fully known, for a+ * caller that needs to react to what it actually was (a native {@code npm unpublish}+ * cleaning up this service's own accounting only once Verdaccio really did remove the+ * package, not before); every other caller passes a no-op.+ */+ private void proxy(RoutingContext context, String backendBaseUrl, String backendPath,+ IntConsumer onSuccessStatus) {URI backend = URI.create(backendBaseUrl);RequestOptions options = new RequestOptions().setHost(backend.getHost())@@ -511,8 +617,10 @@ public class CodeArtifactNpmDataPlane {}return;}- copyResponseHeaders(context, upstreamResponse.result());- upstreamResponse.result().pipeTo(context.response());+ HttpClientResponse response = upstreamResponse.result();+ copyResponseHeaders(context, response);+ response.pipeTo(context.response());+ onSuccessStatus.accept(response.statusCode());});clientReq.send(context.request()).onFailure(ignored -> {if (!context.response().ended()) {diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiController.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiController.javaindex 096562c6fc..dff1c48ac9 100644--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiController.java+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiController.java@@ -85,7 +85,8 @@ public class CodeArtifactPypiController {this.httpClient = httpClient;}- private record RepositoryHandle(String repoId, String baseUrl, String pathPrefix) {+ private record RepositoryHandle(String repoId, String baseUrl, String pathPrefix, String region,+ String owner) {}// No route for the bare "/simple/" project-list endpoint: AWS's CodeArtifact User Guide@@ -181,7 +182,17 @@ public class CodeArtifactPypiController {.build();}- return forwardUpload(handle.baseUrl(), contentType, content);+ // Snapshotted before the real write, not after: see+ // CodeArtifactService#currentContainerBackedPackageFence's own javadoc for why the+ // order matters.+ long fenceSnapshot = service.currentContainerBackedPackageFence(handle.region(), domain, handle.owner(),+ repository, "pypi", null, packageName);+ Response response = forwardUpload(handle.baseUrl(), contentType, content);+ if (response.getStatus() == 200 || response.getStatus() == 201) {+ service.recordContainerBackedAssetBytes(handle.region(), domain, handle.owner(), repository,+ "pypi", null, packageName, file.get().content().length, fenceSnapshot);+ }+ return response;}}@@ -314,6 +325,7 @@ public class CodeArtifactPypiController {throw new NotFoundException();}String baseUrl = pypiserver.ensureReady(repoId, null);- return new RepositoryHandle(repoId, baseUrl, "/codeartifact/pypi/" + domain + "/" + repository);+ return new RepositoryHandle(repoId, baseUrl, "/codeartifact/pypi/" + domain + "/" + repository,+ scope.region(), scope.owner());}}diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.javaindex 8b9342a3cc..dbf6d0f5c5 100644--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactService.java@@ -41,11 +41,13 @@ import java.util.ArrayList;import java.util.Comparator;import java.util.LinkedHashMap;import java.util.List;+import java.util.Locale;import java.util.Map;import java.util.Optional;import java.util.Set;import java.util.UUID;import java.util.concurrent.ConcurrentHashMap;+import java.util.concurrent.atomic.AtomicLong;import java.util.regex.Pattern;import java.util.stream.Stream;@@ -101,7 +103,7 @@ public class CodeArtifactService implements Resettable {private static final long DEFAULT_TOKEN_DURATION_SECONDS = 43200;private static final int TOKEN_RANDOM_BYTES = 32;- public record DomainView(CodeArtifactDomain domain, int repositoryCount) {}+ public record DomainView(CodeArtifactDomain domain, int repositoryCount, long assetSizeBytes) {}public record ResourcePolicy(String resourceArn, String revision, String document) {}public record PublishPackageVersionResult(CodeArtifactPackageVersion packageVersion, PackageAsset asset) {}public record PackageVersionAssetResult(PackageAsset asset, String packageVersionRevision) {}@@ -125,6 +127,23 @@ public class CodeArtifactService implements Resettable {// Bearer tokens are ephemeral security material, not durable emulator state: kept in memory// only, the same way IamService keeps STS session credentials, regardless of storage mode.private final ConcurrentHashMap<String, AuthorizationTokenRecord> authorizationTokens = new ConcurrentHashMap<>();+ // A race guard, not emulator state a caller can ever observe directly: never persisted, and+ // never needs to survive a restart (an in-flight race can't span one either). See+ // currentContainerBackedPackageFence's own javadoc for what this protects against.+ private final ConcurrentHashMap<String, Long> containerBackedPackageDeletionFence = new ConcurrentHashMap<>();+ private final AtomicLong containerBackedPackageDeletionCounter = new AtomicLong();+ // Maven, npm, and pypi publishes go straight to their own sidecar and never create a+ // packageVersions record (see CONTAINER_BACKED_FORMATS), so DescribeDomain's assetSizeBytes+ // can't see their bytes by summing packageVersions the way it does for generic. Each format's+ // own proxy reports a successful, non-idempotent publish's size here instead, keyed exactly+ // like packageOriginConfigs (region/domain/repository/format/namespace/package, no version).+ // In memory only, like the fence above, not through StorageFactory: the Maven/npm/pypi+ // content these bytes count lives in the sidecar containers themselves, which do not survive+ // a Floci restart either (no volume attached, removed on shutdown; see this service's own+ // Known limitations doc). Persisting the count but not the content it describes would leave+ // a stale total surviving a restart for bytes that no longer exist anywhere, with nothing to+ // ever naturally clean it up; kept in memory instead, so a restart loses both together.+ private final ConcurrentHashMap<String, Long> containerBackedAssetBytes = new ConcurrentHashMap<>();@Injectpublic CodeArtifactService(StorageFactory storageFactory, RegionResolver regionResolver, EmulatorConfig config,@@ -194,7 +213,7 @@ public class CodeArtifactService implements Resettable {d.setCreatedTime(Instant.now().getEpochSecond());d.setTags(validateTags(tags, Map.of(), domain, "domain"));domains.putForAccount(owner, key, d);- return new DomainView(d, 0);+ return new DomainView(d, 0, 0L);}/**@@ -211,13 +230,14 @@ public class CodeArtifactService implements Resettable {+ "until they are deleted.", domain, "domain");}domains.deleteForAccount(owner, key);- return new DomainView(d, 0);+ return new DomainView(d, 0, 0L);}public DomainView describeDomain(String region, String domain, String domainOwner) {String owner = effectiveOwner(domainOwner);CodeArtifactDomain d = requireDomain(owner, domainKey(region, domain), domain);- return new DomainView(d, repositoryCountForDomain(owner, region, domain));+ return new DomainView(d, repositoryCountForDomain(owner, region, domain),+ assetSizeBytesForDomain(owner, region, domain));}public PaginatedResult<DomainView> listDomains(String region, Integer maxResults, String nextToken) {@@ -225,8 +245,11 @@ public class CodeArtifactService implements Resettable {List<CodeArtifactDomain> matching = domains.scanForAccount(owner, k -> k.startsWith(region + "::"));PaginatedResult<CodeArtifactDomain> page = Pagination.paginate(matching, CodeArtifactDomain::getName,maxResults, nextToken, 100, 1000, "ValidationException");+ // DomainSummary (what ListDomains actually returns) has no assetSizeBytes field at all on+ // real AWS, unlike DomainDescription; 0 here is never observed by a caller, same as+ // repositoryCount would be if DomainSummary carried one.List<DomainView> views = page.items().stream()- .map(d -> new DomainView(d, repositoryCountForDomain(owner, region, d.getName())))+ .map(d -> new DomainView(d, repositoryCountForDomain(owner, region, d.getName()), 0L)).toList();return new PaginatedResult<>(views, page.nextToken());}@@ -340,6 +363,32 @@ public class CodeArtifactService implements Resettable {packageOriginConfigs.keysForAccount(owner).stream().filter(k -> k.startsWith(originPrefix)).forEach(k -> packageOriginConfigs.deleteForAccount(owner, k));+ // Keyed the same way: without this, a repository recreated under the same name would+ // inherit the previous one's Maven/npm/pypi byte counts into DescribeDomain's total.+ String ownerScopedOriginPrefix = ownerScopedKey(owner, originPrefix);+ containerBackedAssetBytes.keySet().stream()+ .filter(k -> k.startsWith(ownerScopedOriginPrefix))+ .toList()+ .forEach(containerBackedAssetBytes::remove);+ // packageVersions (generic-format metadata and asset bytes) is keyed the same way too, and+ // was never cleaned up here before assetSizeBytesForDomain existed to notice: without+ // this, a repository recreated under this same name would inherit every generic package+ // version the previous one ever had, resurfacing through DescribePackageVersion,+ // GetPackageVersionAsset, and DescribeDomain's own total alike. Deletes each version's own+ // asset content too, the same way deletePackage already does for one package at a time.+ for (CodeArtifactPackageVersion pv : packageVersions.scanForAccount(owner, k -> k.startsWith(originPrefix))) {+ String versionKey = packageVersionKey(region, domain, repository, pv.getFormat(), pv.getNamespace(),+ pv.getPackageName(), pv.getVersion());+ packageVersions.deleteForAccount(owner, versionKey);+ for (PackageAsset asset : pv.getAssets().values()) {+ try {+ deleteAssetContent(owner, versionKey, asset.getName());+ } catch (UncheckedIOException e) {+ LOG.warnv(e, "Could not delete CodeArtifact asset file for {0}/{1}: {2}", versionKey,+ asset.getName(), e.getMessage());+ }+ }+ }return r;}@@ -833,9 +882,19 @@ public class CodeArtifactService implements Resettable {String originKey = packageKey(region, domain, repository, format, namespace, packageName);Optional<PackageOriginConfig> originConfig = packageOriginConfigs.getForAccount(owner, originKey);boolean hadRealContent;+ String accountingKey = packageKey(region, domain, repository, format, namespace,+ accountingPackageName(format, packageName));if (CONTAINER_BACKED_FORMATS.contains(format)) {hadRealContent = deleteContainerBackedPackageIfPresent(region, domain, owner, repository, format,namespace, packageName);+ if (hadRealContent) {+ // Bumped only on an actual deletion, never a no-op (nothing there to begin with):+ // see currentContainerBackedPackageFence's own javadoc for what this protects+ // against, and why it can replace a sidecar round trip entirely rather than just+ // relocate one.+ containerBackedPackageDeletionFence.put(ownerScopedKey(owner, accountingKey),+ containerBackedPackageDeletionCounter.incrementAndGet());+ }} else {List<CodeArtifactPackageVersion> versions = genericPackageVersions(region, domain, owner, repository,format, namespace, packageName);@@ -869,6 +928,12 @@ public class CodeArtifactService implements Resettable {// same coordinates must 404 rather than finding a zero-version package the deleted config// left behind.packageOriginConfigs.deleteForAccount(owner, originKey);+ // Harmless no-op for generic (nothing is ever recorded there under this key; its bytes+ // live in packageVersions instead, already gone via the version-deleting branch above).+ // Uses accountingKey, not originKey: for pypi the two can differ (see+ // accountingPackageName's own javadoc), and the byte count was written under whichever+ // one actually matches what pypiserver itself treats as this package's identity.+ containerBackedAssetBytes.remove(ownerScopedKey(owner, accountingKey));String publishRestriction = originConfig.map(PackageOriginConfig::getPublishRestriction).orElse("ALLOW");String upstreamRestriction = originConfig.map(PackageOriginConfig::getUpstreamRestriction).orElse("BLOCK");return new PackageDescription(format, namespace, packageName, publishRestriction, upstreamRestriction);@@ -1131,6 +1196,8 @@ public class CodeArtifactService implements Resettable {repositories.clear();packageVersions.clear();packageOriginConfigs.clear();+ containerBackedAssetBytes.clear();+ containerBackedPackageDeletionFence.clear();authorizationTokens.clear();if (inMemory) {memoryAssetStore.clear();@@ -1177,6 +1244,202 @@ public class CodeArtifactService implements Resettable {return repositories.scanForAccount(owner, k -> k.startsWith(region + "::" + domain + "::")).size();}+ /**+ * Sums every stored {@link PackageAsset}'s size across the domain's repositories, plus+ * whatever each repository's Maven/npm/pypi proxy has reported through+ * {@link #recordContainerBackedAssetBytes}, for {@code DescribeDomain}'s {@code+ * assetSizeBytes}.+ *+ * <p>Scoped per-repository (via each live repository's own key prefix) rather than one scan+ * over the whole domain: a defense-in-depth match for the per-repository prefix every read+ * path here already uses, not a load-bearing one by itself. {@link #deleteRepository} deletes+ * a repository's {@code packageVersions} records (and their asset content) and+ * {@code containerBackedAssetBytes} outright, by that same prefix, specifically so a+ * repository recreated under the same name can never inherit either; this method's own+ * per-repository scoping means that even a future gap in that cleanup would still only risk+ * double-scanning live data, never resurrecting a deleted repository's orphaned bytes into a+ * same-named replacement's total.+ */+ private long assetSizeBytesForDomain(String owner, String region, String domain) {+ long total = 0L;+ for (CodeArtifactRepository r : repositories.scanForAccount(owner, k -> k.startsWith(region + "::" + domain+ + "::"))) {+ String repositoryPrefix = region + "::" + domain + "::" + r.getName() + "::";+ for (CodeArtifactPackageVersion pv : packageVersions.scanForAccount(owner,+ k -> k.startsWith(repositoryPrefix))) {+ for (PackageAsset asset : pv.getAssets().values()) {+ total += asset.getSize();+ }+ }+ String ownerScopedRepositoryPrefix = ownerScopedKey(owner, repositoryPrefix);+ for (Map.Entry<String, Long> entry : containerBackedAssetBytes.entrySet()) {+ if (entry.getKey().startsWith(ownerScopedRepositoryPrefix)) {+ total += entry.getValue();+ }+ }+ }+ return total;+ }++ /**+ * A snapshot each format proxy takes right before issuing its real write to the sidecar+ * (Reposilite/Verdaccio/pypiserver), to pass into {@link #recordContainerBackedAssetBytes}+ * once that write (and the proxy's own decision that it actually wrote something new)+ * completes. Comparing the two there detects not just that the package existed when the+ * write was issued, but that no {@link #deletePackage} has completed in between: a package+ * deleted and republished under the same name while an npm publish's accounting runs on a+ * delayed callback (its own Vert.x response arriving well after this snapshot was taken, long+ * after Maven's or pypi's near-immediate inline accounting would have run) must not have the+ * first publish's bytes misattributed to the second. {@code deletePackage}'s container-backed+ * branch bumps the counter behind this snapshot on an actual deletion; so does+ * {@link #forgetContainerBackedPackage}, for the one other way a container-backed package's+ * real content can disappear: a real npm client's native {@code npm unpublish}, which the+ * pinned Verdaccio image (configured with {@code unpublish: $all}) honors on its own wire+ * protocol without ever calling {@code deletePackage} (confirmed: {@code+ * CodeArtifactNpmDataPlane} forwards it straight through, detecting only enough of its shape+ * to call {@code forgetContainerBackedPackage} afterward). Maven and pypi have no such native+ * delete of their own to worry about. A snapshot that still matches at record time proves+ * neither of those two paths has removed whatever now answers to this name. Cheap and+ * lock-free (a {@code ConcurrentHashMap} read), unlike the sidecar round trip this replaces,+ * so it never costs a proxy anything to call this inline, even on a Vert.x event-loop thread.+ */+ public long currentContainerBackedPackageFence(String region, String domain, String domainOwner,+ String repository, String format, String namespace, String packageName) {+ String owner = effectiveOwner(domainOwner);+ String key = packageKey(region, domain, repository, format, namespace, accountingPackageName(format,+ packageName));+ return containerBackedPackageDeletionFence.getOrDefault(ownerScopedKey(owner, key), 0L);+ }++ /**+ * Adds {@code bytes} to the running total CodeArtifact's own storage tracks for one+ * Maven/npm/pypi package, independent of whatever that sidecar could report back later: none+ * of the three expose an enumeration cheap enough for {@link #assetSizeBytesForDomain} to+ * query live (the same reason {@link #describePackageVersion} can't bridge to them either).+ * Called by each format's own proxy only once it has already decided, in its own terms, that a+ * real write just happened: never for an idempotent short-circuit (nothing new was written) or+ * a rejected conflict (Reposilite's {@code redeployment: false}, Verdaccio's native reject, or+ * this emulator's own pypi preflight), since neither actually adds any bytes.+ * {@code packageFenceSnapshot} is whatever {@link #currentContainerBackedPackageFence} returned+ * when the proxy called it, right before issuing that write.+ *+ * <p>{@code namespace}/{@code packageName} here come straight from the wire protocol (a Maven+ * GAV path segment, an npm scope/name, a twine form field), never through+ * {@link #validatePackageCoordinates}/{@link #validatePackageToken} the way every JSON-API+ * package operation's own namespace/packageName already is. {@link #escapeKeySegment}'s whole+ * collision-safety argument rests on no namespace/package/version that reaches it ever+ * containing a raw {@code '#'}; silently skipping one that does (rather than escaping or+ * rejecting outright) keeps that invariant true for this one new path into+ * {@link #packageKey} without turning a client's publish, which already succeeded against the+ * real sidecar, into a thrown exception over what is only a best-effort accounting side+ * channel.+ *+ * <p>Unlike every other mutating method here, the caller's own "does this repository still+ * exist" check ({@code ensureFormatContainerId}, inside each proxy's own {@code requireScope}/+ * {@code requireRepository} path) ran in a separate, already-released {@code synchronized}+ * call, before the real HTTP round trip to the sidecar that this call reports the result of.+ * A {@code DeleteRepository} landing in that window, after the sidecar write already+ * succeeded but before this call runs, would otherwise resurrect a key {@code deleteRepository}+ * already cleaned up: invisible to {@code DescribeDomain} while the repository stays deleted+ * (its domain-sum loop only walks live repositories), but inherited the moment a repository is+ * recreated under the same name, since this key is name-based with no fresh id the way {@code+ * sidecarContainerIds} deliberately gets one. Re-checking existence here, inside the same+ * {@code synchronized} call that does the write, closes that window the same way every other+ * method's single-call check-then-act already does.+ *+ * <p>The repository check above does not catch the same window opening for+ * {@link #deletePackage} instead of {@link #deleteRepository}: the repository stays alive,+ * only the one package underneath it is gone, and {@link #assetSizeBytesForDomain}'s+ * per-repository scan would then count a phantom entry for a package that no longer exists, or+ * misattribute it to whatever was republished under the same name. The fence comparison below+ * closes that window precisely, with no sidecar round trip (and so no lock-held network call,+ * and no event-loop-blocking call from an async proxy like npm's) needed to do it.+ */+ public synchronized void recordContainerBackedAssetBytes(String region, String domain, String domainOwner,+ String repository, String format, String namespace, String packageName, long bytes,+ long packageFenceSnapshot) {+ if (!isValidPackageToken(packageName) || (namespace != null && !isValidPackageToken(namespace))) {+ LOG.warnv("Skipping container-backed asset byte accounting for {0}/{1} package ''{2}'': namespace or "+ + "package name contains a character ('#', '/', or whitespace) a real CodeArtifact package "+ + "coordinate cannot have.", domain, repository, packageName);+ return;+ }+ String owner = effectiveOwner(domainOwner);+ if (repositories.getForAccount(owner, repositoryKey(region, domain, repository)).isEmpty()) {+ LOG.warnv("Skipping container-backed asset byte accounting for {0}/{1}: the repository was deleted "+ + "while this publish was in flight.", domain, repository);+ return;+ }+ String key = packageKey(region, domain, repository, format, namespace, accountingPackageName(format,+ packageName));+ long currentFence = containerBackedPackageDeletionFence.getOrDefault(ownerScopedKey(owner, key), 0L);+ if (currentFence != packageFenceSnapshot) {+ LOG.warnv("Skipping container-backed asset byte accounting for {0}/{1} package ''{2}'': a "+ + "DeletePackage removed it from the {3} sidecar (possibly replaced by a different publish) "+ + "while this publish was in flight.", domain, repository, packageName, format);+ return;+ }+ String ownerScopedKey = ownerScopedKey(owner, key);+ long existing = containerBackedAssetBytes.getOrDefault(ownerScopedKey, 0L);+ containerBackedAssetBytes.put(ownerScopedKey, existing + bytes);+ }++ /**+ * Cleans up this service's own byte accounting and deletion fence for one container-backed+ * package after something other than {@link #deletePackage} removed it directly from the+ * sidecar: specifically, a real npm client's native {@code npm unpublish}, which the pinned+ * Verdaccio image (configured with {@code unpublish: $all}) honors on its own wire protocol,+ * bypassing {@link #deletePackage} entirely (confirmed: {@code CodeArtifactNpmDataPlane}+ * forwards any request that is not a publish straight through). {@link #deletePackage}'s own+ * container-backed branch already does both of these same two things as part of deleting the+ * real sidecar content too; this is that same bookkeeping on its own, for when the sidecar+ * content is already gone by a path {@link #deletePackage} never saw.+ *+ * <p>{@code namespace}/{@code packageName} come straight from the wire protocol here too (the+ * unpublish request's own path), so this guards the same escape-collision invariant+ * {@link #recordContainerBackedAssetBytes} does, the same way and for the same reason.+ */+ public synchronized void forgetContainerBackedPackage(String region, String domain, String domainOwner,+ String repository, String format, String namespace, String packageName) {+ if (!isValidPackageToken(packageName) || (namespace != null && !isValidPackageToken(namespace))) {+ LOG.warnv("Ignoring a native unpublish of {0}/{1} package ''{2}'': namespace or package name contains "+ + "a character ('#', '/', or whitespace) a real CodeArtifact package coordinate cannot have.",+ domain, repository, packageName);+ return;+ }+ String owner = effectiveOwner(domainOwner);+ String key = packageKey(region, domain, repository, format, namespace, accountingPackageName(format,+ packageName));+ String ownerScopedKey = ownerScopedKey(owner, key);+ containerBackedAssetBytes.remove(ownerScopedKey);+ containerBackedPackageDeletionFence.put(ownerScopedKey, containerBackedPackageDeletionCounter.incrementAndGet());+ }++ /**+ * The fence and {@code containerBackedAssetBytes} are both process-wide plain+ * {@code ConcurrentHashMap}s, not account-scoped storage the way every durable store here is,+ * so the account has to be folded into the key explicitly rather than relying on an+ * {@code AccountAwareStorageBackend} to keep two accounts' identical-looking package keys+ * apart.+ */+ private static String ownerScopedKey(String owner, String packageKey) {+ return owner + "::" + packageKey;+ }++ /**+ * The same PEP 503 project-name normalization {@code PypiserverSidecarClient} applies before+ * every lookup against the real pypiserver sidecar (lowercase, runs of {@code -_.} collapsed+ * to one {@code -}), applied here too, only for pypi's own accounting/fence key: pypiserver+ * treats case- and punctuation-equivalent spellings as the same package, so an upload spelled+ * one way and a later {@code DeletePackage} spelled another must still address the same byte+ * count and the same fence entry, the way they already address the same sidecar package.+ * {@code packageOriginConfigs} deliberately keeps using the caller's raw spelling unchanged;+ * that mismatch is pre-existing and unrelated to the byte accounting this method guards.+ */+ private static String accountingPackageName(String format, String packageName) {+ return "pypi".equals(format) ? packageName.toLowerCase(Locale.ROOT).replaceAll("[-_.]+", "-") : packageName;+ }+private int domainCountForAccount(String owner, String region) {return domains.scanForAccount(owner, k -> k.startsWith(region + "::")).size();}@@ -1293,11 +1556,15 @@ public class CodeArtifactService implements Resettable {}private static void validatePackageToken(String field, String value) {- if (value == null || value.isEmpty() || value.length() > 255 || !PACKAGE_TOKEN.matcher(value).matches()) {+ if (!isValidPackageToken(value)) {throw validation(field + " must be 1-255 characters with no '#', '/', or whitespace.");}}+ private static boolean isValidPackageToken(String value) {+ return value != null && !value.isEmpty() && value.length() <= 255 && PACKAGE_TOKEN.matcher(value).matches();+ }+/*** The format/package/namespace coordinate validation shared by every package-level operation* that takes them. {@code action} is a present-participle phrase ("describing", "setting thediff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/NpmPublishEnvelopeScanner.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/NpmPublishEnvelopeScanner.javaindex 364f971db8..1ef27b8d9e 100644--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/NpmPublishEnvelopeScanner.java+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/NpmPublishEnvelopeScanner.java@@ -26,8 +26,11 @@ final class NpmPublishEnvelopeScanner {record Version(String tarball, String integrity) {}+ /** {@code size} is the attachment's real, decoded byte count, counted alongside its digest. */+ record Attachment(String integrity, long size) {}+record Envelope(String name, Map<String, String> distTags, Map<String, Version> versions,- Map<String, String> attachmentIntegrity) {}+ Map<String, Attachment> attachments) {}private static final JsonFactory FACTORY = JsonFactory.builder().enable(StreamReadFeature.STRICT_DUPLICATE_DETECTION)@@ -47,7 +50,7 @@ final class NpmPublishEnvelopeScanner {private String name;private final Map<String, String> distTags = new HashMap<>();private final Map<String, Map<String, String>> versionFields = new HashMap<>();- private final Map<String, String> attachmentIntegrity = new HashMap<>();+ private final Map<String, Attachment> attachments = new HashMap<>();private NpmPublishEnvelopeScanner(JsonParser parser) {this.parser = parser;@@ -74,7 +77,7 @@ final class NpmPublishEnvelopeScanner {case "_attachments" -> forEachField(filename ->forEachField(attachmentField -> {if (attachmentField.equals("data")) {- attachmentIntegrity.put(filename, hashBase64());+ attachments.put(filename, hashBase64());} else {parser.skipChildren();}@@ -125,15 +128,41 @@ final class NpmPublishEnvelopeScanner {return parser.getText();}- private String hashBase64() throws IOException {+ /**+ * {@link JsonParser#readBinaryValue} already resolves a JSON escape sequence within the+ * base64 text before decoding it (confirmed directly: a validly-escaped, valid-length+ * attachment decodes correctly, digest and byte count both exactly right), so this never+ * needs a separate unescaping path of its own.+ */+ private Attachment hashBase64() throws IOException {requireToken(JsonToken.VALUE_STRING);MessageDigest digest = sha512();- try (OutputStream sink = new DigestOutputStream(OutputStream.nullOutputStream(), digest)) {+ CountingOutputStream counted = new CountingOutputStream();+ try (OutputStream sink = new DigestOutputStream(counted, digest)) {parser.readBinaryValue(sink);} catch (IllegalArgumentException e) {throw invalid("attachment data is not plain base64");}- return "sha512-" + Base64.getEncoder().encodeToString(digest.digest());+ return new Attachment("sha512-" + Base64.getEncoder().encodeToString(digest.digest()), counted.count());+ }++ /** Discards every byte written to it, counting them as it goes. */+ private static final class CountingOutputStream extends OutputStream {+ private long count;++ @Override+ public void write(int b) {+ count++;+ }++ @Override+ public void write(byte[] b, int off, int len) {+ count += len;+ }++ long count() {+ return count;+ }}private void requireToken(JsonToken expected) throws IOException {@@ -148,7 +177,7 @@ final class NpmPublishEnvelopeScanner {Map<String, String> fields = entry.getValue();versions.put(entry.getKey(), new Version(fields.get("tarball"), fields.get("integrity")));}- return new Envelope(name, distTags, versions, attachmentIntegrity);+ return new Envelope(name, distTags, versions, attachments);}private static MessageDigest sha512() {diff --git a/src/main/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClient.java b/src/main/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClient.javaindex 18f4e8ffde..3423d3797a 100644--- a/src/main/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClient.java+++ b/src/main/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClient.java@@ -336,6 +336,14 @@ public class ReposiliteSidecarClient implements RepositorySidecarManager {}}+ /**+ * {@code wroteNewContent} is {@code false} for the idempotent-match short-circuit (nothing was+ * actually sent to Reposilite) and {@code true} only when the real {@code PUT} below got back+ * a success status; a genuine conflict (Reposilite's own {@code redeployment: false} rejection)+ * or any other non-2xx status wrote nothing either, despite having been attempted.+ */+ public record DeployOutcome(int status, boolean wroteNewContent) {}+/*** Deploys {@code content} to {@code repoId}'s {@code gav} path, returning the HTTP status.*@@ -352,15 +360,16 @@ public class ReposiliteSidecarClient implements RepositorySidecarManager {* the deploy outright, since Reposilite's own redeployment check is still there to catch a* genuine conflict either way.*/- public int deployArtifact(String repoId, String gav, byte[] content) {+ public DeployOutcome deployArtifact(String repoId, String gav, byte[] content) {String baseUrl = manager.ensureReady();if (existingArtifactMatches(baseUrl, repoId, gav, content)) {- return 200;+ return new DeployOutcome(200, false);}HttpRequest request = authenticated(baseUrl, repoId, gav).PUT(BodyPublishers.ofByteArray(content)).build();- return send(request, BodyHandlers.discarding()).statusCode();+ int status = send(request, BodyHandlers.discarding()).statusCode();+ return new DeployOutcome(status, status == 200 || status == 201);}/**diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactAssetPersistenceTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactAssetPersistenceTest.javaindex e889701e81..a71fed2ad7 100644--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactAssetPersistenceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactAssetPersistenceTest.java@@ -87,6 +87,30 @@ class CodeArtifactAssetPersistenceTest {assertEquals(published.packageVersion().getRevision(), result.packageVersionRevision());}+ /**+ * The fix for a real finding (hectorvent, PR #5484): {@code containerBackedAssetBytes} used+ * to be wired through {@code StorageFactory} like every other durable store here, but the+ * Maven/npm/pypi content it counts lives only in the sidecar containers themselves, which do+ * not survive a Floci restart either (no volume attached, removed on shutdown; see this+ * service's own Known limitations doc). A persisted count surviving a restart the content+ * itself does not would leave a stale total with nothing to ever clean it up.+ */+ @Test+ void containerBackedAssetBytesDoesNotSurviveRestart(@TempDir Path dir) {+ CodeArtifactService first = newService(dir);+ first.createDomain(REGION, "dom", null, Map.of());+ first.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ long fence = first.currentContainerBackedPackageFence(REGION, "dom", null, "repo", "maven", "com.example",+ "my-pkg");+ first.recordContainerBackedAssetBytes(REGION, "dom", null, "repo", "maven", "com.example", "my-pkg", 4096L,+ fence);+ assertEquals(4096L, first.describeDomain(REGION, "dom", null).assetSizeBytes());++ CodeArtifactService restarted = newService(dir);++ assertEquals(0L, restarted.describeDomain(REGION, "dom", null).assetSizeBytes());+ }+/*** {@code packageOriginConfigs} is wired through {@code StorageFactory} the same way* {@code domains}/{@code repositories}/{@code packageVersions} are, but that similarity haddiff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenDockerIntegrationTest.javaindex 26db2d46c2..a2ef801cba 100644--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenDockerIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactMavenDockerIntegrationTest.java@@ -438,6 +438,62 @@ class CodeArtifactMavenDockerIntegrationTest {.then().statusCode(404).body("__type", equalTo("ResourceNotFoundException"));}+ /**+ * Asserts the delta rather than an absolute total: this class shares one domain across every+ * ordered test, so other artifacts already contribute to the running total by this point.+ * {@code assetSizeBytes} used to be hardcoded to {@code 0} regardless of what was actually+ * published; this is the real Reposilite round trip proving a Maven deploy now counts.+ */+ @Test+ @Order(14)+ void describeDomainAssetSizeBytesIncreasesByExactlyWhatWasDeployed() {+ long before = given().header("Authorization", AUTH)+ .get("/v1/domain?domain=" + DOMAIN)+ .then().statusCode(200)+ .extract().jsonPath().getLong("domain.assetSizeBytes");++ byte[] content = "asset-size-accounting-bytes".getBytes(StandardCharsets.UTF_8);+ given().header("Authorization", "Bearer " + bearerToken).body(content)+ .put("/codeartifact/maven/" + DOMAIN + "/" + REPO + "/com/example/sizecheck/1.0.0/sizecheck-1.0.0.jar")+ .then().statusCode(200);++ long after = given().header("Authorization", AUTH)+ .get("/v1/domain?domain=" + DOMAIN)+ .then().statusCode(200)+ .extract().jsonPath().getLong("domain.assetSizeBytes");++ assertEquals(before + content.length, after);+ }++ /**+ * {@code mvn deploy} always PUTs a {@code maven-metadata.xml} straight to the group/artifact+ * directory too, with no version segment: {@code com/example/metacheck/maven-metadata.xml} is+ * only one segment shorter than a real versioned asset path, the same four segments as the+ * shortest real GAV ({@code group/artifact/version/filename}). It must not be mistaken for+ * one and misattributed one groupId component too shallow; it is still a real, successful+ * write, so its bytes still belong in the total.+ */+ @Test+ @Order(15)+ void describeDomainAssetSizeBytesCountsAGroupLevelMavenMetadataUpload() {+ long before = given().header("Authorization", AUTH)+ .get("/v1/domain?domain=" + DOMAIN)+ .then().statusCode(200)+ .extract().jsonPath().getLong("domain.assetSizeBytes");++ byte[] metadata = "<metadata><groupId>com.example</groupId></metadata>".getBytes(StandardCharsets.UTF_8);+ given().header("Authorization", "Bearer " + bearerToken).body(metadata)+ .put("/codeartifact/maven/" + DOMAIN + "/" + REPO + "/com/example/metacheck/maven-metadata.xml")+ .then().statusCode(200);++ long after = given().header("Authorization", AUTH)+ .get("/v1/domain?domain=" + DOMAIN)+ .then().statusCode(200)+ .extract().jsonPath().getLong("domain.assetSizeBytes");++ assertEquals(before + metadata.length, after);+ }+/*** Reads the real Reposilite instance's own {@code maven} settings domain directly (the same* shared container {@link ReposiliteSidecarManager} started for every test in this class).diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDockerIntegrationTest.javaindex 615df40cc9..a2da209ab8 100644--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDockerIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactNpmDockerIntegrationTest.java@@ -98,11 +98,18 @@ class CodeArtifactNpmDockerIntegrationTest {@Test@Order(1)void publishThenFetchRoundTripsTheRealNpmEnvelope() throws IOException {+ long assetSizeBytesBeforePublish = describeDomainAssetSizeBytes();+given().header("Authorization", "Bearer " + bearerToken).contentType("application/json").body(publishFixture).put("/codeartifact/npm/" + DOMAIN + "/" + REPO + "/" + PACKAGE_NAME).then().statusCode(anyOf201Or200());+ // assetSizeBytes used to be hardcoded to 0 regardless of what was actually published; this+ // is the real Verdaccio round trip proving a real npm publish now counts, by exactly the+ // tarball's own decoded byte count (not the envelope's base64-and-JSON-wrapped size).+ assertEquals(assetSizeBytesBeforePublish + expectedTarballBytes.length, describeDomainAssetSizeBytes());+String metadataBody = given().header("Authorization", "Bearer " + bearerToken).get("/codeartifact/npm/" + DOMAIN + "/" + REPO + "/" + PACKAGE_NAME).then().statusCode(200)@@ -139,10 +146,23 @@ class CodeArtifactNpmDockerIntegrationTest {@Test@Order(2)void republishingTheSameEnvelopeASecondTimeIsStillIdempotent() {+ long assetSizeBytesBeforeRepublish = describeDomainAssetSizeBytes();+given().header("Authorization", "Bearer " + bearerToken).contentType("application/json").body(publishFixture).put("/codeartifact/npm/" + DOMAIN + "/" + REPO + "/" + PACKAGE_NAME).then().statusCode(anyOf201Or200());++ // The confirmed-republish short-circuit never reaches Verdaccio, so no new bytes were+ // written; assetSizeBytes must not count this tarball a second time.+ assertEquals(assetSizeBytesBeforeRepublish, describeDomainAssetSizeBytes());+ }++ private static long describeDomainAssetSizeBytes() {+ return given().header("Authorization", AUTH)+ .get("/v1/domain?domain=" + DOMAIN)+ .then().statusCode(200)+ .extract().jsonPath().getLong("domain.assetSizeBytes");}@Test@@ -373,6 +393,48 @@ class CodeArtifactNpmDockerIntegrationTest {.then().statusCode(404).body("__type", equalTo("ResourceNotFoundException"));}+ /**+ * The fix for a real finding (hectorvent, PR #5484): the pinned Verdaccio image is configured+ * with {@code unpublish: $all}, so a real npm client's native {@code DELETE} straight to the+ * packument ({@code npm unpublish <pkg> --force}) bypasses {@code DeletePackage} entirely and+ * goes straight through this proxy's own transparent passthrough for any non-publish request.+ * Without detecting this one wire shape, the byte count this package's publish added would+ * survive the unpublish forever, with nothing to ever clean it up ({@code DeletePackage} never+ * runs; {@code DeleteRepository} only cleans it up if the whole repository is later deleted+ * too). A throwaway package, not {@link #PACKAGE_NAME}, for the same reason as the test above.+ */+ @Test+ @Order(12)+ void nativeNpmUnpublishCleansUpTheByteCountToo() {+ String packageName = "unpublish-me-pkg";+ byte[] tarball = "unpublish throwaway tarball".getBytes(StandardCharsets.UTF_8);+ String envelope = "{\"name\":\"" + packageName + "\",\"versions\":{\"1.0.0\":{\"name\":\"" + packageName+ + "\",\"version\":\"1.0.0\",\"dist\":{\"tarball\":\"http://ignored/" + packageName + "-1.0.0.tgz\""+ + "}}},\"_attachments\":{\"" + packageName + "-1.0.0.tgz\":{\"content_type\":"+ + "\"application/octet-stream\",\"data\":\""+ + Base64.getEncoder().encodeToString(tarball) + "\",\"length\":" + tarball.length + "}}}";++ long assetSizeBytesBeforePublish = describeDomainAssetSizeBytes();+ given().header("Authorization", "Bearer " + bearerToken).contentType("application/json").body(envelope)+ .put("/codeartifact/npm/" + DOMAIN + "/" + REPO + "/" + packageName)+ .then().statusCode(anyOf201Or200());+ assertEquals(assetSizeBytesBeforePublish + tarball.length, describeDomainAssetSizeBytes());++ String rev = given().header("Authorization", "Bearer " + bearerToken)+ .get("/codeartifact/npm/" + DOMAIN + "/" + REPO + "/" + packageName)+ .then().statusCode(200)+ .extract().jsonPath().getString("_rev");++ given().header("Authorization", "Bearer " + bearerToken)+ .delete("/codeartifact/npm/" + DOMAIN + "/" + REPO + "/" + packageName + "/-rev/" + rev)+ .then().statusCode(anyOf201Or200());++ given().header("Authorization", "Bearer " + bearerToken)+ .get("/codeartifact/npm/" + DOMAIN + "/" + REPO + "/" + packageName)+ .then().statusCode(404);+ assertEquals(assetSizeBytesBeforePublish, describeDomainAssetSizeBytes());+ }+/*** Proves the actual mechanism, not just that {@code VerdaccioSidecarManager} has a method* named right: {@code ContainerTeardowns.stopAll} runs every {@code ContainerTeardown} ondiff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.javaindex 6c34820347..80567c6b78 100644--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactPypiDockerIntegrationTest.java@@ -77,6 +77,7 @@ class CodeArtifactPypiDockerIntegrationTest {@Order(1)void uploadThenDownloadRoundTripsTheExactBytesThroughTheSimpleIndex() {byte[] content = "real-sdist-bytes".getBytes(StandardCharsets.UTF_8);+ long assetSizeBytesBeforeUpload = describeDomainAssetSizeBytes();given().header("Authorization", "Bearer " + bearerToken).contentType("multipart/form-data; boundary=boundary123")@@ -84,6 +85,10 @@ class CodeArtifactPypiDockerIntegrationTest {.post("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/").then().statusCode(200);+ // assetSizeBytes used to be hardcoded to 0 regardless of what was actually published; this+ // is the real pypiserver round trip proving a real twine upload now counts.+ assertEquals(assetSizeBytesBeforeUpload + content.length, describeDomainAssetSizeBytes());+String indexHtml = given().header("Authorization", "Bearer " + bearerToken).get("/codeartifact/pypi/" + DOMAIN + "/" + REPO + "/simple/" + PACKAGE_NAME + "/").then().statusCode(200)@@ -112,6 +117,8 @@ class CodeArtifactPypiDockerIntegrationTest {@Test@Order(2)void reuploadingTheExactSameFileSucceedsIdempotently() {+ long assetSizeBytesBeforeReupload = describeDomainAssetSizeBytes();+given().header("Authorization", "Bearer " + bearerToken).contentType("multipart/form-data; boundary=boundary123").body(twineUploadBody("boundary123", PACKAGE_NAME, FILENAME, "real-sdist-bytes".getBytes(StandardCharsets.UTF_8)))@@ -123,6 +130,16 @@ class CodeArtifactPypiDockerIntegrationTest {.then().statusCode(200).extract().asByteArray();assertEquals("real-sdist-bytes", new String(fetched, StandardCharsets.UTF_8));+ // The idempotent-match short-circuit never forwards to pypiserver, so no new bytes were+ // written; assetSizeBytes must not count this file a second time.+ assertEquals(assetSizeBytesBeforeReupload, describeDomainAssetSizeBytes());+ }++ private static long describeDomainAssetSizeBytes() {+ return given().header("Authorization", AUTH)+ .get("/v1/domain?domain=" + DOMAIN)+ .then().statusCode(200)+ .extract().jsonPath().getLong("domain.assetSizeBytes");}@Testdiff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.javaindex 366c6e81d1..e5afef1329 100644--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/CodeArtifactServiceTest.java@@ -210,6 +210,336 @@ class CodeArtifactServiceTest {assertEquals(2, service.describeDomain(REGION, "counted", null).repositoryCount());}+ @Test+ void createAndDeleteDomainReportZeroAssetSizeBytes() {+ DomainView created = service.createDomain(REGION, "fresh", null, Map.of());+ assertEquals(0L, created.assetSizeBytes());+ DomainView deleted = service.deleteDomain(REGION, "fresh", null);+ assertEquals(0L, deleted.assetSizeBytes());+ }++ @Test+ void describeDomainAssetSizeBytesSumsGenericAssetsAcrossRepositoriesAndVersions() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo-a", null, null, Map.of());+ service.createRepository(REGION, "dom", null, "repo-b", null, null, Map.of());+ byte[] first = "first-package".getBytes(StandardCharsets.UTF_8);+ byte[] second = "second-package-longer".getBytes(StandardCharsets.UTF_8);+ byte[] third = "third".getBytes(StandardCharsets.UTF_8);+ service.publishPackageVersion(REGION, "dom", null, "repo-a", "generic", "ns", "pkg-a", "1.0.0", "a.txt",+ Digests.sha256Hex(first), "false", first);+ service.publishPackageVersion(REGION, "dom", null, "repo-a", "generic", "ns", "pkg-a", "2.0.0", "a.txt",+ Digests.sha256Hex(second), "false", second);+ service.publishPackageVersion(REGION, "dom", null, "repo-b", "generic", "ns", "pkg-b", "1.0.0", "a.txt",+ Digests.sha256Hex(third), "false", third);++ long expected = first.length + second.length + third.length;+ assertEquals(expected, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ @Test+ void describeDomainAssetSizeBytesDropsByExactlyWhatDeletePackageRemoves() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ byte[] kept = "kept-package".getBytes(StandardCharsets.UTF_8);+ byte[] removed = "removed-package-content".getBytes(StandardCharsets.UTF_8);+ service.publishPackageVersion(REGION, "dom", null, "repo", "generic", "ns", "kept", "1.0.0", "a.txt",+ Digests.sha256Hex(kept), "false", kept);+ service.publishPackageVersion(REGION, "dom", null, "repo", "generic", "ns", "removed", "1.0.0", "a.txt",+ Digests.sha256Hex(removed), "false", removed);+ assertEquals(kept.length + removed.length, service.describeDomain(REGION, "dom", null).assetSizeBytes());++ service.deletePackage(REGION, "dom", null, "repo", "generic", "ns", "removed");++ assertEquals(kept.length, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * {@code deleteRepository} does not clean up a generic package's {@code packageVersions}+ * records or asset bytes today, so they are only unreachable through the normal API, not+ * actually gone. Without scoping the sum to each still-live repository's own key prefix, the+ * domain total would keep counting a deleted repository's orphaned bytes forever.+ */+ @Test+ void describeDomainAssetSizeBytesExcludesADeletedRepositorysOrphanedBytes() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo-a", null, null, Map.of());+ service.createRepository(REGION, "dom", null, "repo-b", null, null, Map.of());+ byte[] inDeletedRepo = "goes-away-with-repo-a".getBytes(StandardCharsets.UTF_8);+ byte[] inLiveRepo = "stays-in-repo-b".getBytes(StandardCharsets.UTF_8);+ service.publishPackageVersion(REGION, "dom", null, "repo-a", "generic", "ns", "pkg-a", "1.0.0", "a.txt",+ Digests.sha256Hex(inDeletedRepo), "false", inDeletedRepo);+ service.publishPackageVersion(REGION, "dom", null, "repo-b", "generic", "ns", "pkg-b", "1.0.0", "a.txt",+ Digests.sha256Hex(inLiveRepo), "false", inLiveRepo);++ service.deleteRepository(REGION, "dom", null, "repo-a");++ assertEquals(inLiveRepo.length, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * A Maven/npm/pypi package merely existing on its sidecar (confirmed here via a stubbed+ * {@code packageExists}) contributes nothing on its own: {@code CodeArtifactService} never+ * queries a sidecar for size, only the running total each format's own proxy explicitly+ * reports through {@link CodeArtifactService#recordContainerBackedAssetBytes}, which nothing+ * in this test ever calls. The next tests below exercise that path directly.+ */+ @Test+ void aContainerBackedPackageExistingWithNoRecordedBytesContributesNothing() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ byte[] generic = "generic-package".getBytes(StandardCharsets.UTF_8);+ service.publishPackageVersion(REGION, "dom", null, "repo", "generic", "ns", "generic-pkg", "1.0.0", "a.txt",+ Digests.sha256Hex(generic), "false", generic);+ when(reposiliteClient.packageExists(anyString(), eq("dom"), eq("repo"), eq("com.example"), eq("maven-pkg")))+ .thenReturn(true);++ assertEquals(generic.length, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * Mirrors exactly what each real proxy does: snapshot {@code currentContainerBackedPackageFence}+ * before the (here, simulated) real write, then report it alongside the byte count, the way+ * {@code CodeArtifactMavenController}/the npm data plane/{@code CodeArtifactPypiController}+ * all do for real.+ */+ private void recordBytes(String domain, String repository, String format, String namespace, String packageName,+ long bytes) {+ long fence = service.currentContainerBackedPackageFence(REGION, domain, null, repository, format, namespace,+ packageName);+ service.recordContainerBackedAssetBytes(REGION, domain, null, repository, format, namespace, packageName,+ bytes, fence);+ }++ /**+ * {@code recordContainerBackedAssetBytes} is what each of the three sidecar-backed proxies+ * calls after a real (non-idempotent, non-rejected) write; this is the unit-level proof that+ * it feeds {@code assetSizeBytes} correctly across all three formats plus generic at once,+ * accumulating repeated calls for the same package the way a multi-file Maven deploy+ * (jar + pom as separate PUTs) needs.+ */+ @Test+ void describeDomainAssetSizeBytesSumsContainerBackedBytesAcrossEveryFormat() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ byte[] generic = "generic".getBytes(StandardCharsets.UTF_8);+ service.publishPackageVersion(REGION, "dom", null, "repo", "generic", "ns", "generic-pkg", "1.0.0", "a.txt",+ Digests.sha256Hex(generic), "false", generic);+ recordBytes("dom", "repo", "maven", "com.example", "maven-pkg", 100L);+ recordBytes("dom", "repo", "maven", "com.example", "maven-pkg", 50L);+ recordBytes("dom", "repo", "npm", null, "npm-pkg", 200L);+ recordBytes("dom", "repo", "pypi", null, "pypi-pkg", 300L);++ long expected = generic.length + 100L + 50L + 200L + 300L;+ assertEquals(expected, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * Unlike every JSON-API package operation's own namespace/packageName, the wire-protocol+ * proxies (Maven's GAV path, npm's scope/name, twine's form field) never validate theirs+ * before calling this: a crafted namespace spelling out another namespace's already-escaped+ * form (here, raw {@code "a#cb"} vs. legitimate {@code "a:b"}, see+ * {@code newEscapedKeysNeverCollideWithAPreExistingRawKeyThatHappenedToContainABackslash} for+ * the same invariant proven on the JSON API side) must not be allowed to land in+ * {@code containerBackedAssetBytes} under the legitimate package's own key at all.+ */+ @Test+ void recordContainerBackedAssetBytesIgnoresANamespaceContainingTheEscapeCharacter() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ recordBytes("dom", "repo", "maven", "a:b", "legit", 10L);++ recordBytes("dom", "repo", "maven", "a#cb", "legit", 999L);++ assertEquals(10L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ @Test+ void recordContainerBackedAssetBytesIgnoresAPackageNameContainingTheEscapeCharacter() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());++ recordBytes("dom", "repo", "pypi", null, "bad#name", 999L);++ assertEquals(0L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ @Test+ void describeDomainAssetSizeBytesDropsWhenDeletePackageRemovesAContainerBackedPackage() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ when(reposiliteClient.packageExists(anyString(), eq("dom"), eq("repo"), eq("com.example"), eq("removed")))+ .thenReturn(true);+ recordBytes("dom", "repo", "maven", "com.example", "kept", 40L);+ recordBytes("dom", "repo", "maven", "com.example", "removed", 60L);+ assertEquals(100L, service.describeDomain(REGION, "dom", null).assetSizeBytes());++ service.deletePackage(REGION, "dom", null, "repo", "maven", "com.example", "removed");++ assertEquals(40L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * Same orphan-exclusion property as generic (see the backslash/collision test above), proven+ * here for the sidecar ledger: {@code deleteRepository} actually does clean this one up (unlike+ * {@code packageVersions}), but the sum is scoped per live repository regardless, so this holds+ * either way.+ */+ @Test+ void describeDomainAssetSizeBytesExcludesADeletedRepositorysContainerBackedBytes() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo-a", null, null, Map.of());+ service.createRepository(REGION, "dom", null, "repo-b", null, null, Map.of());+ recordBytes("dom", "repo-a", "npm", null, "pkg-a", 70L);+ recordBytes("dom", "repo-b", "npm", null, "pkg-b", 30L);++ service.deleteRepository(REGION, "dom", null, "repo-a");++ assertEquals(30L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * {@code assetSizeBytesForDomain} scans generic {@code packageVersions} by each live+ * repository's name-based prefix, not by a freshly minted id, so a repository recreated under+ * this same name would otherwise sum right back into any orphaned record+ * {@code deleteRepository} failed to clean up: unlike {@code containerBackedAssetBytes}, there+ * was never any cleanup of {@code packageVersions} on {@code deleteRepository} before this,+ * since nothing before {@code assetSizeBytes} existed ever looked an old repository's+ * generic-format records up by name alone the way this domain-wide sum does.+ */+ @Test+ void describeDomainAssetSizeBytesDoesNotResurrectAGenericPackagesBytesAfterRepositoryRecreation() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ byte[] content = "generic content".getBytes(StandardCharsets.UTF_8);+ service.publishPackageVersion(REGION, "dom", null, "repo", "generic", "ns", "generic-pkg", "1.0.0", "a.txt",+ Digests.sha256Hex(content), "false", content);+ assertEquals(content.length, service.describeDomain(REGION, "dom", null).assetSizeBytes());++ service.deleteRepository(REGION, "dom", null, "repo");+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());++ assertEquals(0L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * Reproduces the one window where a real publish and a real {@code DeleteRepository} could+ * interleave: the sidecar-backed proxies check the repository exists in one {@code+ * synchronized} call ({@code ensureFormatContainerId}), then do the real HTTP write to the+ * sidecar with no lock held at all, and only then call this method in a second, separate+ * {@code synchronized} call. A {@code DeleteRepository} landing in that gap would otherwise+ * resurrect a key it just cleaned up. Calling {@code recordContainerBackedAssetBytes} after+ * {@code deleteRepository} reproduces exactly that interleaving deterministically (both methods+ * are themselves {@code synchronized}, so there is no actual race to simulate, only the order+ * the two calls land in): the resurrected entry must not survive, and in particular must not+ * be inherited by a same-named repository created afterward, the same invariant+ * {@code deleteRepository}'s own cleanup already protects against for every ordering but this+ * one.+ */+ @Test+ void recordContainerBackedAssetBytesAfterTheRepositoryIsAlreadyDeletedIsIgnored() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ long fence = service.currentContainerBackedPackageFence(REGION, "dom", null, "repo", "maven", "com.example",+ "late");+ service.deleteRepository(REGION, "dom", null, "repo");++ service.recordContainerBackedAssetBytes(REGION, "dom", null, "repo", "maven", "com.example", "late", 999L,+ fence);++ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ assertEquals(0L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * The same window, but for {@code DeletePackage} instead of {@code DeleteRepository}: the+ * repository stays alive, only the package underneath it is gone, so a resurrected entry+ * would be counted by {@code DescribeDomain} immediately rather than only after a repository+ * is recreated. Snapshotting the fence before the delete, exactly like a real proxy snapshots+ * it before its real write, reproduces exactly the interleaving a delayed accounting callback+ * racing a concurrent {@code DeletePackage} would see.+ */+ @Test+ void recordContainerBackedAssetBytesAfterAConcurrentDeletePackageIsIgnored() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ long fence = service.currentContainerBackedPackageFence(REGION, "dom", null, "repo", "maven", "com.example",+ "late");+ when(reposiliteClient.packageExists(anyString(), eq("dom"), eq("repo"), eq("com.example"), eq("late")))+ .thenReturn(true);+ service.deletePackage(REGION, "dom", null, "repo", "maven", "com.example", "late");++ service.recordContainerBackedAssetBytes(REGION, "dom", null, "repo", "maven", "com.example", "late", 999L,+ fence);++ assertEquals(0L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * The harder version of the same race: the package is deleted {@code and} a different publish+ * recreates it under the same name, both completing before the first publish's own delayed+ * accounting call runs. A sidecar existence check alone could not tell these two packages+ * apart (the name exists either way); the fence can, since only an actual deletion bumps it.+ */+ @Test+ void recordContainerBackedAssetBytesAfterDeleteAndRepublishUnderTheSameNameIsIgnored() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ long fence = service.currentContainerBackedPackageFence(REGION, "dom", null, "repo", "maven", "com.example",+ "late");+ when(reposiliteClient.packageExists(anyString(), eq("dom"), eq("repo"), eq("com.example"), eq("late")))+ .thenReturn(true);+ service.deletePackage(REGION, "dom", null, "repo", "maven", "com.example", "late");+ // The republish: a different publish, same coordinates, after the first one's snapshot.+ recordBytes("dom", "repo", "maven", "com.example", "late", 10L);++ service.recordContainerBackedAssetBytes(REGION, "dom", null, "repo", "maven", "com.example", "late", 999L,+ fence);++ assertEquals(10L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }++ /**+ * {@code currentContainerBackedPackageFence}/{@code recordContainerBackedAssetBytes} never+ * touch a sidecar at all, so there is no failure mode left to fail open on: the fence+ * comparison is a plain in-memory read, always available, closing the same race the sidecar+ * round trip this replaced used to close, at the cost of a thrown exception whenever it+ * couldn't reach the sidecar.+ */+ @Test+ void recordContainerBackedAssetBytesNeverNeedsTheSidecarAtAll() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ when(reposiliteClient.packageExists(any(), any(), any(), any(), any()))+ .thenThrow(new IllegalStateException("sidecar unreachable"));++ recordBytes("dom", "repo", "maven", "com.example", "never-asked", 55L);++ assertEquals(55L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ verify(reposiliteClient, never()).packageExists(any(), any(), any(), any(), any());+ }++ /**+ * {@code packageOriginConfigs} keeps the caller's raw spelling (pre-existing, unrelated to byte+ * accounting); {@code containerBackedAssetBytes}/the fence use pypiserver's own PEP 503+ * normalization instead, so an upload spelled one way and a {@code DeletePackage} spelled a+ * case/punctuation-equivalent way still address the one byte count pypiserver itself would+ * treat as a single package.+ */+ @Test+ void pypiAccountingNormalizesThePackageNameLikePypiserverItself() {+ service.createDomain(REGION, "dom", null, Map.of());+ service.createRepository(REGION, "dom", null, "repo", null, null, Map.of());+ recordBytes("dom", "repo", "pypi", null, "My.Package", 42L);+ assertEquals(42L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ when(pypiserverClient.packageExists(anyString(), eq("dom"), eq("repo"), isNull(), eq("my_package")))+ .thenReturn(true);++ service.deletePackage(REGION, "dom", null, "repo", "pypi", null, "my_package");++ assertEquals(0L, service.describeDomain(REGION, "dom", null).assetSizeBytes());+ }+@Testvoid concurrentCreateDomainWithSameNameOnlyOneWins() throws InterruptedException {int attempts = 8;diff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/NpmPublishEnvelopeScannerTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/NpmPublishEnvelopeScannerTest.javaindex d9fefabcdf..9c404b0c4d 100644--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/NpmPublishEnvelopeScannerTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/NpmPublishEnvelopeScannerTest.java@@ -33,7 +33,7 @@ class NpmPublishEnvelopeScannerTest {NpmPublishEnvelopeScanner.Version version = envelope.versions().get("1.0.0");assertEquals(raw.at("/versions/1.0.0/dist/integrity").asText(), version.integrity());String filename = raw.at("/versions/1.0.0/dist/tarball").asText().replaceAll(".*/", "");- assertEquals(version.integrity(), envelope.attachmentIntegrity().get(filename));+ assertEquals(version.integrity(), envelope.attachments().get(filename).integrity());}@Test@@ -55,7 +55,8 @@ class NpmPublishEnvelopeScannerTest {NpmPublishEnvelopeScanner.Envelope envelope = scan(json);assertEquals("sha512-" + Base64.getEncoder().encodeToString(sha512(tarball)),- envelope.attachmentIntegrity().get("a.tgz"));+ envelope.attachments().get("a.tgz").integrity());+ assertEquals(tarball.length, envelope.attachments().get("a.tgz").size());}@Test@@ -68,7 +69,8 @@ class NpmPublishEnvelopeScannerTest {NpmPublishEnvelopeScanner.Envelope envelope = scan(json);assertEquals("sha512-" + Base64.getEncoder().encodeToString(sha512(tarball)),- envelope.attachmentIntegrity().get("big.tgz"));+ envelope.attachments().get("big.tgz").integrity());+ assertEquals(tarball.length, envelope.attachments().get("big.tgz").size());}@Test@@ -110,9 +112,43 @@ class NpmPublishEnvelopeScannerTest {assertInvalid("{\"_attachments\":{\"a.tgz\":{\"data\":\"abcde\"}}}");}+ /**+ * A real finding turned out not to be one: a review raised whether a JSON-escaped attachment+ * (most plausibly {@code \/}, the one base64 character JSON permits escaping, though no real+ * npm client's own {@code JSON.stringify} ever actually produces one) would be silently+ * undercounted here. Checked directly rather than assumed either way: {@code+ * JsonParser#readBinaryValue} already resolves the escape before decoding, so a validly+ * escaped, valid-length attachment decodes correctly, digest and byte count both exactly+ * right. {@code 0xFF 0xFF 0xFF} is a deliberately chosen fixture: its base64 encoding is+ * exactly {@code "////"}, four real {@code /} characters with no padding, so every character+ * of the escaped payload below actually exercises the escape-resolution path, not just one+ * character buried among ordinary ones.+ */+ @Test+ void attachmentDataContainingAnEscapedSlashIsDecodedCorrectly() throws Exception {+ byte[] tarball = {(byte) 0xFF, (byte) 0xFF, (byte) 0xFF};+ String base64 = Base64.getEncoder().encodeToString(tarball);+ assertEquals("////", base64);+ String escaped = base64.replace("/", "\\/");+ String json = "{\"_attachments\":{\"a.tgz\":{\"data\":\"" + escaped + "\"}}}";++ NpmPublishEnvelopeScanner.Envelope envelope = scan(json);++ assertEquals("sha512-" + Base64.getEncoder().encodeToString(sha512(tarball)),+ envelope.attachments().get("a.tgz").integrity());+ assertEquals(tarball.length, envelope.attachments().get("a.tgz").size());+ }++ /**+ * An escape sequence present is not, on its own, a pass: decoded, this is {@code "a/bcd"},+ * 5 characters, not a multiple of 4, so still genuinely invalid base64 regardless of the+ * escape. (This is also what the older, differently-reasoned version of this test actually+ * exercised: {@code "ab\/cd"} decodes to {@code "ab/cd"}, 5 characters, rejected for the same+ * length reason, not because it contained an escape at all.)+ */@Test- void attachmentDataContainingAnEscapeIsRejected() {- assertInvalid("{\"_attachments\":{\"a.tgz\":{\"data\":\"ab\\/cd\"}}}");+ void attachmentDataContainingAnEscapeButStillInvalidBase64IsRejected() {+ assertInvalid("{\"_attachments\":{\"a.tgz\":{\"data\":\"a\\/bcd\"}}}");}@Testdiff --git a/src/test/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClientTest.java b/src/test/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClientTest.javaindex 2209619551..47e5f595ad 100644--- a/src/test/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClientTest.java+++ b/src/test/java/io/github/hectorvent/floci/services/codeartifact/ReposiliteSidecarClientTest.java@@ -262,9 +262,11 @@ class ReposiliteSidecarClientTest {respond(exchange, 200, "");});- int status = client.deployArtifact("dom--repo", "com/example/a/1.0/a-1.0.jar", "hello".getBytes(StandardCharsets.UTF_8));+ ReposiliteSidecarClient.DeployOutcome outcome = client.deployArtifact("dom--repo",+ "com/example/a/1.0/a-1.0.jar", "hello".getBytes(StandardCharsets.UTF_8));- assertThat(status, is(200));+ assertThat(outcome.status(), is(200));+ assertTrue(outcome.wroteNewContent());assertArrayEquals("hello".getBytes(StandardCharsets.UTF_8), received.get());assertThat(authHeader.get(), equalTo("Basic dGVzdDp0ZXN0"));}@@ -278,9 +280,11 @@ class ReposiliteSidecarClientTest {respond(exchange, 200, "hello");});- int status = client.deployArtifact("dom--repo", "com/example/a/1.0/a-1.0.jar", content);+ ReposiliteSidecarClient.DeployOutcome outcome = client.deployArtifact("dom--repo",+ "com/example/a/1.0/a-1.0.jar", content);- assertThat(status, is(200));+ assertThat(outcome.status(), is(200));+ assertFalse(outcome.wroteNewContent());assertThat(methodsSeen, equalTo(List.of("GET")));}@@ -300,10 +304,11 @@ class ReposiliteSidecarClientTest {}});- int status = client.deployArtifact("dom--repo", "com/example/a/1.0/a-1.0.jar",- "hello".getBytes(StandardCharsets.UTF_8));+ ReposiliteSidecarClient.DeployOutcome outcome = client.deployArtifact("dom--repo",+ "com/example/a/1.0/a-1.0.jar", "hello".getBytes(StandardCharsets.UTF_8));- assertThat(status, is(409));+ assertThat(outcome.status(), is(409));+ assertFalse(outcome.wroteNewContent());assertThat(methodsSeen, equalTo(List.of("GET", "PUT")));assertArrayEquals("hello".getBytes(StandardCharsets.UTF_8), putBody.get());}@@ -328,10 +333,11 @@ class ReposiliteSidecarClientTest {}});- int status = client.deployArtifact("dom--repo", "com/example/a/1.0/a-1.0.jar",- "hello".getBytes(StandardCharsets.UTF_8));+ ReposiliteSidecarClient.DeployOutcome outcome = client.deployArtifact("dom--repo",+ "com/example/a/1.0/a-1.0.jar", "hello".getBytes(StandardCharsets.UTF_8));- assertThat(status, is(200));+ assertThat(outcome.status(), is(200));+ assertTrue(outcome.wroteNewContent());assertTrue(methodsSeen.stream().allMatch(m -> m.equals("GET") || m.equals("PUT")));assertThat(methodsSeen.get(methodsSeen.size() - 1), equalTo("PUT"));assertArrayEquals("hello".getBytes(StandardCharsets.UTF_8), putBody.get());