Skip to content

🐍 Geometric Computer Vision Library for Spatial AI

fix(augmentation): an empty batch takes the configured output size under any gate (#4429) (#6272)

Dmytro Mishkinac22ebd26f67
diff --git a/changelog.d/5288.fixed.md b/changelog.d/5288.fixed.md
index c719b6c80..3b795a942 100644
--- a/changelog.d/5288.fixed.md
+++ b/changelog.d/5288.fixed.md
@@ -1,4 +1,3 @@
`Resize` with an integer size, `LongestMaxSize` and `SmallestMaxSize` now return an empty batch with the
-resized spatial shape for a `(0, C, H, W)` input at `p=1` instead of raising `KeyError: 'output_size'`. With
-`p < 1` the empty batch keeps the input's spatial shape, as it does for `Resize` with a tuple size and the other
-shape-changing augmentations (#4429).
+resized spatial shape for a `(0, C, H, W)` input, whatever `p` is, instead of raising `KeyError: 'output_size'`
+(#4429).
diff --git a/changelog.d/5620.breaking.md b/changelog.d/5620.breaking.md
index f23f75145..8242ae620 100644
--- a/changelog.d/5620.breaking.md
+++ b/changelog.d/5620.breaking.md
@@ -4,8 +4,8 @@ to return the transformed size: for a `(0, 3, 6, 8)` forward, `Resize((5, 7)).in
gives `(0, 3, 6, 8)`. This covers `Resize`, `LongestMaxSize`, `SmallestMaxSize`, `CenterCrop`, `RandomResizedCrop`
and resample-mode `RandomCrop`, directly and through `AugmentationSequential.inverse`. Two empty-batch parameters
change with it. `RandomCrop` now records the unpadded input as `forward_input_shape`; it used to record the padded
-canvas, which an empty batch cannot undo because its `padding_size` has no rows. In a container, a stage after an
-always-applied (`p=1`) `RandomCrop` or `Resize` (including `LongestMaxSize` and `SmallestMaxSize`) now records its
+canvas, which an empty batch cannot undo because its `padding_size` has no rows. In a container, a stage after a
+`RandomCrop` or `Resize` (including `LongestMaxSize` and `SmallestMaxSize`), whatever its `p`, now records its
own input size; it used to record the size before that crop or resize. Empty `RandomCrop` parameters saved before
this change still replay but restore the padded canvas, for example `(0, 1, 12, 12)` instead of `(0, 1, 6, 8)` with
`padding=(1, 2, 3, 4)`: regenerate them. Non-empty batches are unchanged. Part of #4429.
diff --git a/changelog.d/5647.fixed.md b/changelog.d/5647.fixed.md
index bc0207992..b57e7447b 100644
--- a/changelog.d/5647.fixed.md
+++ b/changelog.d/5647.fixed.md
@@ -1,5 +1,4 @@
`ImageSequential`, `AugmentationSequential` and `VideoSequential` no longer raise `IndexError: index 0 is out of
-bounds for dimension 0 with size 0` on an empty batch when they contain a `CenterCrop`. A stage after an
-always-applied `CenterCrop` records the crop size, as for a non-empty batch, so a later `CenterCrop` larger than
-that size raises the same `AssertionError` as for a non-empty batch. With `p < 1` an empty batch keeps its input
-size, as its forward pass does (#4429).
+bounds for dimension 0 with size 0` on an empty batch when they contain a `CenterCrop`. A stage after the
+`CenterCrop` records the crop size, whatever its `p`, as its forward pass does, so a later `CenterCrop` larger than
+that size raises the same `AssertionError` as for a non-empty batch (#4429).
diff --git a/changelog.d/5706.breaking.md b/changelog.d/5706.breaking.md
index f88a032f8..d095d6e7d 100644
--- a/changelog.d/5706.breaking.md
+++ b/changelog.d/5706.breaking.md
@@ -1,8 +1,8 @@
-On an empty batch, an always-applied (`p=1`) shape-changing 2D augmentation now gives a mask the height and width it
+On an empty batch, a shape-changing 2D augmentation now gives a mask the height and width it
gives the image. It used to return the mask at its input size: `AugmentationSequential(Resize((4, 5)),
data_keys=["input", "mask"])` on a `(0, 3, 8, 10)` image and a `(0, 1, 8, 10)` mask gave a `(0, 3, 4, 5)` image next
to a `(0, 1, 8, 10)` mask, and passing that pair to `AugmentationSequential.inverse` raised `ValueError`. The mask is
now `(0, 1, 4, 5)` and the inverse restores `(0, 1, 8, 10)`. This covers `Resize`, `LongestMaxSize`,
`SmallestMaxSize`, `CenterCrop`, `RandomCrop`, `RandomResizedCrop` and `PadTo`, in `AugmentationSequential` (also with
-dictionary inputs and inside `VideoSequential`) and in a direct `transform_masks` call. With `p < 1` an empty image and
-mask both keep their input size, as before. Non-empty batches are unchanged. Part of #4429.
+dictionary inputs and inside `VideoSequential`) and in a direct `transform_masks` call. This holds
+for any `p`: an empty image and mask both take the configured size. Non-empty batches are unchanged. Part of #4429.
diff --git a/changelog.d/6272.fixed.md b/changelog.d/6272.fixed.md
new file mode 100644
index 000000000..f5841db00
--- /dev/null
+++ b/changelog.d/6272.fixed.md
@@ -0,0 +1,6 @@
+An empty batch through a shape-changing augmentation with `p < 1` now comes back at the configured output size, as
+with `p=1`, instead of at the input size: `RandomCrop((4, 4), p=0.5)` on a `(0, 3, 6, 8)` batch gives
+`(0, 3, 4, 4)`, as in 0.8.3. This covers `RandomCrop`, `CenterCrop`, `RandomResizedCrop`, `Resize`,
+`LongestMaxSize`, `SmallestMaxSize`, `RandomCrop3D` and `CenterCrop3D`, the masks that follow the image, and the
+sizes `AugmentationSequential` records for its later stages and `inverse`. A non-empty batch that the gate skips
+entirely still keeps its input size (#4429).
diff --git a/kornia/augmentation/base.py b/kornia/augmentation/base.py
index d4f4afbc1..86566fdfa 100644
--- a/kornia/augmentation/base.py
+++ b/kornia/augmentation/base.py
@@ -397,13 +397,16 @@ class _AugmentationBase(_BasicAugmentationBase):
fullgraph-friendly). Shape-changing augmentations (e.g. crop/resize) whose branches
differ in spatial size fall back to a Python branch on ``to_apply.any()``, which is
not onnx-exportable. Such a fallback cannot select rows, so a ``to_apply`` that mixes applied and skipped
- rows raises ``ValueError`` instead of transforming the skipped rows too.
+ rows raises ``ValueError`` instead of transforming the skipped rows too. An empty gate has no row to skip
+ and takes the transformed branch, so an empty batch gets the configured output size whatever ``p`` is.
"""
if transformed.shape == not_transformed.shape and transformed.shape[0] == to_apply.shape[0]:
to_apply_expanded = _to_device_nonblocking(
to_apply.view(-1, *([1] * (len(transformed.shape) - 1))), transformed.device
)
return torch.where(to_apply_expanded, transformed, not_transformed)
+ if to_apply.numel() == 0:
+ return transformed
apply_any = bool(to_apply.any())
if apply_any and not bool(to_apply.all()) and transformed.shape[1:] != not_transformed.shape[1:]:
raise _mixed_gate_shape_error(not_transformed.shape[1:], transformed.shape[1:])
@@ -496,12 +499,8 @@ class _AugmentationBase(_BasicAugmentationBase):
output_transformed = self.apply_transform_mask(in_tensor, params, flags, transform=transform)
output_not_transformed = self.apply_non_transform_mask(in_tensor, params, flags, transform=transform)
- # With an empty, statically always-applied batch, the image path takes the transformed shape.
- # An empty gate has no rows to blend, so keep the mask on that same canvas (#4429).
- if self.p == 1.0 and self.p_batch == 1.0 and batch_prob.numel() == 0:
- output = output_transformed
- else:
- output = self._blend_by_prob(output_transformed, output_not_transformed, to_apply)
+ # An empty gate takes the transformed branch, so the mask lands on the image's canvas (#4429).
+ output = self._blend_by_prob(output_transformed, output_not_transformed, to_apply)
if not self.keepdim:
return output
diff --git a/kornia/augmentation/container/image.py b/kornia/augmentation/container/image.py
index 48e25f624..e7f88a48f 100644
--- a/kornia/augmentation/container/image.py
+++ b/kornia/augmentation/container/image.py
@@ -505,10 +505,9 @@ def _get_new_batch_shape(param: ParamItem, batch_shape: torch.Size, module: Opti
Augmentations that change the spatial size provide `output_size` with shape (B, spatial_dims).
`PadTo` and `RandomMosaic(output_size=...)` provide none, so their configured size is used; with
`crop_if_exceeds=False`, `PadTo` keeps a dimension that is already larger.
- Empty crops and resizes have no sampled
- output-size rows, so their configured size is used only when the image forward path
- always applies the transform. Other probability branches retain the current empty-gate
- fallback tracked in #4429.
+ Empty crops and resizes have no sampled output-size rows, so their configured size is used,
+ whatever their probability: an empty gate takes the transformed branch, as the image
+ forward does (#4429).
"""
data = param.data
@@ -559,10 +558,10 @@ def _get_new_batch_shape(param: ParamItem, batch_shape: torch.Size, module: Opti
and isinstance(module, _AugmentationBase)
and ("output_size" in data or isinstance(module, (K.RandomCrop, K.CenterCrop, K.RandomResizedCrop, K.Resize)))
):
- # Match the image forward's unconditional path. An empty gate selects the original
- # canvas in the other probability branches, including Resize's p_batch setting (#4429).
+ # Match the image forward: an empty gate takes the transformed branch under any probability,
+ # including Resize's p_batch setting (#4429).
size = module.flags.get("size")
- if module.p != 1.0 or module.p_batch != 1.0 or size is None:
+ if size is None:
return batch_shape
if isinstance(module, K.Resize) and isinstance(size, int):
size = _side_to_image_size(size, batch_shape[-1] / batch_shape[-2], module.flags["side"])
diff --git a/tests/augmentation/_3d/test_random_crop_3d.py b/tests/augmentation/_3d/test_random_crop_3d.py
index 5689a8384..b34713b0d 100644
--- a/tests/augmentation/_3d/test_random_crop_3d.py
+++ b/tests/augmentation/_3d/test_random_crop_3d.py
@@ -331,15 +331,15 @@ class TestRandomCrop3DEmptyValidation(BaseTester):
def test_legal_empty_crop_keeps_shape_4429(
self, size, padding, pad_if_needed, probability, in_sequence, device, dtype
):
- """Empty crops validate against the padded canvas and preserve the skip shape."""
+ """Empty crops validate against the padded canvas and take the crop size under any probability."""
volume = torch.empty(0, 1, 4, 5, 6, device=device, dtype=dtype)
crop = RandomCrop3D(size, padding=padding, pad_if_needed=pad_if_needed, p=probability)
augmentation = AugmentationSequential(crop) if in_sequence else crop
output = augmentation(volume)
- expected_size = size if probability == 1.0 else (4, 5, 6)
- assert output.shape == (0, 1, *expected_size)
+ # An empty batch has no sample for the gate to skip (#4429).
+ assert output.shape == (0, 1, *size)
assert output.device == volume.device
assert output.dtype == volume.dtype
diff --git a/tests/augmentation/container/test_augmentation_sequential.py b/tests/augmentation/container/test_augmentation_sequential.py
index 42b22aa87..ba778db84 100644
--- a/tests/augmentation/container/test_augmentation_sequential.py
+++ b/tests/augmentation/container/test_augmentation_sequential.py
@@ -2551,13 +2551,14 @@ class TestConventionAugmentationSequential(BaseTester):
)
@pytest.mark.parametrize("p", [0.5, 1.0])
def test_empty_batch_mask_round_trip_follows_image_4429(self, make_op, p, device, dtype):
- # The mask takes the image's canvas under every gate, and the inverse restores both. With p < 1 the empty
- # image keeps its input size (the blend fallback tracked in #4429), so only the parity is pinned here.
+ # An empty batch takes the configured size under every gate (#4429): the mask follows the image onto that
+ # canvas, and the inverse restores both.
image = torch.empty(0, 3, 8, 10, device=device, dtype=dtype)
mask = torch.empty(0, 1, 8, 10, device=device, dtype=torch.int64)
seq = K.AugmentationSequential(make_op(p), data_keys=["input", "mask"])
out_image, out_mask = seq(image, mask)
- assert out_mask.shape == (0, 1, *out_image.shape[-2:])
+ assert out_image.shape == (0, 3, 4, 5)
+ assert out_mask.shape == (0, 1, 4, 5)
restored_image, restored_mask = seq.inverse(out_image, out_mask)
assert restored_image.shape == image.shape
assert restored_mask.shape == mask.shape
@@ -3091,7 +3092,8 @@ class TestCrop3DShapePropagation(BaseTester):
handle.remove()
self.assert_close(actual, expected)
saved = deepcopy(seq._params)
- canvas = (4, 5, 6) if p == 1.0 else (8, 10, 12)
+ # A skipped crop keeps its input canvas; an empty batch has no sample to skip (#4429).
+ canvas = (4, 5, 6) if p == 1.0 or batch == 0 else (8, 10, 12)
assert seen == [torch.Size((batch, 1, *canvas))]
assert tuple(saved[1].data["forward_input_shape"]) == seen[0]
assert saved[1].data["output_size"].shape == (batch, 3)
diff --git a/tests/augmentation/test_base.py b/tests/augmentation/test_base.py
index ad39b2fd6..d5a94e00a 100644
--- a/tests/augmentation/test_base.py
+++ b/tests/augmentation/test_base.py
@@ -1253,6 +1253,35 @@ class TestConventionAugmentationBase2D(BaseTester):
empty = torch.rand(0, 3, 6, 8, device=device, dtype=dtype)
assert augmentation()(empty).shape == shape
+ @pytest.mark.parametrize(
+ ("augmentation", "input_shape", "output_size"),
+ [
+ pytest.param(lambda p: K.RandomCrop((4, 6), p=p), (3, 6, 8), (4, 6), id="RandomCrop"),
+ pytest.param(lambda p: K.RandomCrop((4, 6), padding=1, p=p), (3, 6, 8), (4, 6), id="RandomCrop-padding"),
+ pytest.param(lambda p: K.CenterCrop((4, 6), p=p), (3, 6, 8), (4, 6), id="CenterCrop"),
+ pytest.param(lambda p: K.RandomResizedCrop((4, 4), p=p), (3, 6, 8), (4, 4), id="RandomResizedCrop"),
+ pytest.param(lambda p: K.Resize((4, 5), p=p), (3, 6, 8), (4, 5), id="Resize-tuple"),
+ pytest.param(lambda p: K.Resize(10, side="long", p=p), (3, 6, 8), (7, 10), id="Resize-int-long"),
+ pytest.param(lambda p: K.LongestMaxSize(16, p=p), (3, 6, 8), (12, 16), id="LongestMaxSize"),
+ pytest.param(lambda p: K.SmallestMaxSize(12, p=p), (3, 6, 8), (12, 16), id="SmallestMaxSize"),
+ pytest.param(lambda p: K.RandomCrop3D((4, 4, 4), p=p), (1, 6, 8, 10), (4, 4, 4), id="RandomCrop3D"),
+ pytest.param(lambda p: K.CenterCrop3D((4, 4, 4), p=p), (1, 6, 8, 10), (4, 4, 4), id="CenterCrop3D"),
+ ],
+ )
+ @pytest.mark.parametrize("p", [0.0, 0.5, 1.0])
+ def test_zero_batch_takes_configured_size_under_any_gate_4429(
+ self, augmentation, input_shape, output_size, p, device, dtype
+ ):
+ # An empty batch has no sample for the gate to skip, so a shape-changing augmentation returns its configured
+ # output size whatever ``p`` is (#4429), as 0.8.3 did for the 2D crops. A non-empty batch the gate skips
+ # entirely still comes back at its input size.
+ channels = input_shape[0]
+ aug = augmentation(p)
+ empty = torch.rand(0, *input_shape, device=device, dtype=dtype)
+ assert aug(empty).shape == (0, channels, *output_size)
+ skipped = torch.rand(2, *input_shape, device=device, dtype=dtype)
+ assert augmentation(0.0)(skipped).shape == skipped.shape
+
_AUGMENTATIONS = [K.RandomEqualize, K.RandomEqualize3D, K.RandomPosterize, K.RandomSharpness]
diff --git a/tests/augmentation/test_empty_inverse.py b/tests/augmentation/test_empty_inverse.py
index 74a236a54..bd0a18a74 100644
--- a/tests/augmentation/test_empty_inverse.py
+++ b/tests/augmentation/test_empty_inverse.py
@@ -152,10 +152,10 @@ class TestEmptyGeometricInverse(BaseTester):
output = sequence(image)
saved_params = deepcopy(sequence._params)
- # With p < 1 an empty batch keeps its input size, because the blend falls back to the skipped
- # branch (tracked in #4429). This pins the current forward, which the tracked shapes must follow.
- crop_size = (4, 5) if crop_p == 1.0 else (height, width)
- output_size = (3, 4) if resize_p == 1.0 else crop_size
+ # An empty batch has no sample to skip, so every stage takes its configured output size whatever its
+ # probability (#4429), as on 0.8.3 for the crops; the tracked shapes must follow that forward.
+ crop_size = (4, 5)
+ output_size = (3, 4)
assert saved_params[0].data["forward_input_shape"].tolist() == [0, 3, height, width]
assert saved_params[1].data["forward_input_shape"].tolist() == [0, 3, *crop_size]
assert output.shape == (0, 3, *output_size)
@@ -373,10 +373,8 @@ class TestEmptyGeometricInverse(BaseTester):
pytest.skip("The forward resampling kernel does not support this device/dtype's backward pass.")
image = torch.empty(0, 3, 6, 8, device=device, dtype=dtype, requires_grad=True)
output = aug(image)
- # With p < 1 an empty batch keeps its input size (blend fallback, tracked in #4429); the inverse
- # must undo whichever size the forward produced.
- expected_size = output_size if p == 1.0 else (6, 8)
- assert output.shape == (0, 3, *expected_size)
+ # An empty batch takes the configured output size whatever ``p`` is (#4429); the inverse undoes it.
+ assert output.shape == (0, 3, *output_size)
restored = aug.inverse(output)
assert restored.shape == image.shape
assert restored.device == image.device
@@ -519,9 +517,8 @@ class TestEmptyCenterResizedCropHistory(BaseTester):
for key in standalone:
self.assert_close(params[0].data[key], standalone[key], rtol=0, atol=0)
assert params[0].data["forward_input_shape"].tolist() == [0, 3, *input_size]
- # Follow the current empty-gate blend, not a desired probability specification (#4429).
- expected_size = crop_size if p == 1.0 else input_size
- assert params[1].data["forward_input_shape"].tolist() == [0, 3, *expected_size]
+ # An empty gate takes the crop's configured size whatever ``p`` is, as the image forward does (#4429).
+ assert params[1].data["forward_input_shape"].tolist() == [0, 3, *crop_size]
if crop_type is K.CenterCrop:
assert params[0].data["output_size"].shape == (0, 2)
else: