π Geometric Computer Vision Library for Spatial AI
Python artificial-intelligencecomputer-visiondeep-learninghacktoberfestimage-processingmachine-learningneural-networkpythonpytorchroboticsspatial-ai
fix(augmentation): an empty batch takes the configured output size under any gate (#4429) (#6272)
Dmytro Mishkinac22ebd26f67
diff --git a/changelog.d/5288.fixed.md b/changelog.d/5288.fixed.mdindex c719b6c80..3b795a942 100644--- a/changelog.d/5288.fixed.md+++ b/changelog.d/5288.fixed.md@@ -1,4 +1,3 @@`Resize` with an integer size, `LongestMaxSize` and `SmallestMaxSize` now return an empty batch with the-resized spatial shape for a `(0, C, H, W)` input at `p=1` instead of raising `KeyError: 'output_size'`. With-`p < 1` the empty batch keeps the input's spatial shape, as it does for `Resize` with a tuple size and the other-shape-changing augmentations (#4429).+resized spatial shape for a `(0, C, H, W)` input, whatever `p` is, instead of raising `KeyError: 'output_size'`+(#4429).diff --git a/changelog.d/5620.breaking.md b/changelog.d/5620.breaking.mdindex f23f75145..8242ae620 100644--- a/changelog.d/5620.breaking.md+++ b/changelog.d/5620.breaking.md@@ -4,8 +4,8 @@ to return the transformed size: for a `(0, 3, 6, 8)` forward, `Resize((5, 7)).ingives `(0, 3, 6, 8)`. This covers `Resize`, `LongestMaxSize`, `SmallestMaxSize`, `CenterCrop`, `RandomResizedCrop`and resample-mode `RandomCrop`, directly and through `AugmentationSequential.inverse`. Two empty-batch parameterschange with it. `RandomCrop` now records the unpadded input as `forward_input_shape`; it used to record the padded-canvas, which an empty batch cannot undo because its `padding_size` has no rows. In a container, a stage after an-always-applied (`p=1`) `RandomCrop` or `Resize` (including `LongestMaxSize` and `SmallestMaxSize`) now records its+canvas, which an empty batch cannot undo because its `padding_size` has no rows. In a container, a stage after a+`RandomCrop` or `Resize` (including `LongestMaxSize` and `SmallestMaxSize`), whatever its `p`, now records itsown input size; it used to record the size before that crop or resize. Empty `RandomCrop` parameters saved beforethis change still replay but restore the padded canvas, for example `(0, 1, 12, 12)` instead of `(0, 1, 6, 8)` with`padding=(1, 2, 3, 4)`: regenerate them. Non-empty batches are unchanged. Part of #4429.diff --git a/changelog.d/5647.fixed.md b/changelog.d/5647.fixed.mdindex bc0207992..b57e7447b 100644--- a/changelog.d/5647.fixed.md+++ b/changelog.d/5647.fixed.md@@ -1,5 +1,4 @@`ImageSequential`, `AugmentationSequential` and `VideoSequential` no longer raise `IndexError: index 0 is out of-bounds for dimension 0 with size 0` on an empty batch when they contain a `CenterCrop`. A stage after an-always-applied `CenterCrop` records the crop size, as for a non-empty batch, so a later `CenterCrop` larger than-that size raises the same `AssertionError` as for a non-empty batch. With `p < 1` an empty batch keeps its input-size, as its forward pass does (#4429).+bounds for dimension 0 with size 0` on an empty batch when they contain a `CenterCrop`. A stage after the+`CenterCrop` records the crop size, whatever its `p`, as its forward pass does, so a later `CenterCrop` larger than+that size raises the same `AssertionError` as for a non-empty batch (#4429).diff --git a/changelog.d/5706.breaking.md b/changelog.d/5706.breaking.mdindex f88a032f8..d095d6e7d 100644--- a/changelog.d/5706.breaking.md+++ b/changelog.d/5706.breaking.md@@ -1,8 +1,8 @@-On an empty batch, an always-applied (`p=1`) shape-changing 2D augmentation now gives a mask the height and width it+On an empty batch, a shape-changing 2D augmentation now gives a mask the height and width itgives the image. It used to return the mask at its input size: `AugmentationSequential(Resize((4, 5)),data_keys=["input", "mask"])` on a `(0, 3, 8, 10)` image and a `(0, 1, 8, 10)` mask gave a `(0, 3, 4, 5)` image nextto a `(0, 1, 8, 10)` mask, and passing that pair to `AugmentationSequential.inverse` raised `ValueError`. The mask isnow `(0, 1, 4, 5)` and the inverse restores `(0, 1, 8, 10)`. This covers `Resize`, `LongestMaxSize`,`SmallestMaxSize`, `CenterCrop`, `RandomCrop`, `RandomResizedCrop` and `PadTo`, in `AugmentationSequential` (also with-dictionary inputs and inside `VideoSequential`) and in a direct `transform_masks` call. With `p < 1` an empty image and-mask both keep their input size, as before. Non-empty batches are unchanged. Part of #4429.+dictionary inputs and inside `VideoSequential`) and in a direct `transform_masks` call. This holds+for any `p`: an empty image and mask both take the configured size. Non-empty batches are unchanged. Part of #4429.diff --git a/changelog.d/6272.fixed.md b/changelog.d/6272.fixed.mdnew file mode 100644index 000000000..f5841db00--- /dev/null+++ b/changelog.d/6272.fixed.md@@ -0,0 +1,6 @@+An empty batch through a shape-changing augmentation with `p < 1` now comes back at the configured output size, as+with `p=1`, instead of at the input size: `RandomCrop((4, 4), p=0.5)` on a `(0, 3, 6, 8)` batch gives+`(0, 3, 4, 4)`, as in 0.8.3. This covers `RandomCrop`, `CenterCrop`, `RandomResizedCrop`, `Resize`,+`LongestMaxSize`, `SmallestMaxSize`, `RandomCrop3D` and `CenterCrop3D`, the masks that follow the image, and the+sizes `AugmentationSequential` records for its later stages and `inverse`. A non-empty batch that the gate skips+entirely still keeps its input size (#4429).diff --git a/kornia/augmentation/base.py b/kornia/augmentation/base.pyindex d4f4afbc1..86566fdfa 100644--- a/kornia/augmentation/base.py+++ b/kornia/augmentation/base.py@@ -397,13 +397,16 @@ class _AugmentationBase(_BasicAugmentationBase):fullgraph-friendly). Shape-changing augmentations (e.g. crop/resize) whose branchesdiffer in spatial size fall back to a Python branch on ``to_apply.any()``, which isnot onnx-exportable. Such a fallback cannot select rows, so a ``to_apply`` that mixes applied and skipped- rows raises ``ValueError`` instead of transforming the skipped rows too.+ rows raises ``ValueError`` instead of transforming the skipped rows too. An empty gate has no row to skip+ and takes the transformed branch, so an empty batch gets the configured output size whatever ``p`` is."""if transformed.shape == not_transformed.shape and transformed.shape[0] == to_apply.shape[0]:to_apply_expanded = _to_device_nonblocking(to_apply.view(-1, *([1] * (len(transformed.shape) - 1))), transformed.device)return torch.where(to_apply_expanded, transformed, not_transformed)+ if to_apply.numel() == 0:+ return transformedapply_any = bool(to_apply.any())if apply_any and not bool(to_apply.all()) and transformed.shape[1:] != not_transformed.shape[1:]:raise _mixed_gate_shape_error(not_transformed.shape[1:], transformed.shape[1:])@@ -496,12 +499,8 @@ class _AugmentationBase(_BasicAugmentationBase):output_transformed = self.apply_transform_mask(in_tensor, params, flags, transform=transform)output_not_transformed = self.apply_non_transform_mask(in_tensor, params, flags, transform=transform)- # With an empty, statically always-applied batch, the image path takes the transformed shape.- # An empty gate has no rows to blend, so keep the mask on that same canvas (#4429).- if self.p == 1.0 and self.p_batch == 1.0 and batch_prob.numel() == 0:- output = output_transformed- else:- output = self._blend_by_prob(output_transformed, output_not_transformed, to_apply)+ # An empty gate takes the transformed branch, so the mask lands on the image's canvas (#4429).+ output = self._blend_by_prob(output_transformed, output_not_transformed, to_apply)if not self.keepdim:return outputdiff --git a/kornia/augmentation/container/image.py b/kornia/augmentation/container/image.pyindex 48e25f624..e7f88a48f 100644--- a/kornia/augmentation/container/image.py+++ b/kornia/augmentation/container/image.py@@ -505,10 +505,9 @@ def _get_new_batch_shape(param: ParamItem, batch_shape: torch.Size, module: OptiAugmentations that change the spatial size provide `output_size` with shape (B, spatial_dims).`PadTo` and `RandomMosaic(output_size=...)` provide none, so their configured size is used; with`crop_if_exceeds=False`, `PadTo` keeps a dimension that is already larger.- Empty crops and resizes have no sampled- output-size rows, so their configured size is used only when the image forward path- always applies the transform. Other probability branches retain the current empty-gate- fallback tracked in #4429.+ Empty crops and resizes have no sampled output-size rows, so their configured size is used,+ whatever their probability: an empty gate takes the transformed branch, as the image+ forward does (#4429)."""data = param.data@@ -559,10 +558,10 @@ def _get_new_batch_shape(param: ParamItem, batch_shape: torch.Size, module: Optiand isinstance(module, _AugmentationBase)and ("output_size" in data or isinstance(module, (K.RandomCrop, K.CenterCrop, K.RandomResizedCrop, K.Resize)))):- # Match the image forward's unconditional path. An empty gate selects the original- # canvas in the other probability branches, including Resize's p_batch setting (#4429).+ # Match the image forward: an empty gate takes the transformed branch under any probability,+ # including Resize's p_batch setting (#4429).size = module.flags.get("size")- if module.p != 1.0 or module.p_batch != 1.0 or size is None:+ if size is None:return batch_shapeif isinstance(module, K.Resize) and isinstance(size, int):size = _side_to_image_size(size, batch_shape[-1] / batch_shape[-2], module.flags["side"])diff --git a/tests/augmentation/_3d/test_random_crop_3d.py b/tests/augmentation/_3d/test_random_crop_3d.pyindex 5689a8384..b34713b0d 100644--- a/tests/augmentation/_3d/test_random_crop_3d.py+++ b/tests/augmentation/_3d/test_random_crop_3d.py@@ -331,15 +331,15 @@ class TestRandomCrop3DEmptyValidation(BaseTester):def test_legal_empty_crop_keeps_shape_4429(self, size, padding, pad_if_needed, probability, in_sequence, device, dtype):- """Empty crops validate against the padded canvas and preserve the skip shape."""+ """Empty crops validate against the padded canvas and take the crop size under any probability."""volume = torch.empty(0, 1, 4, 5, 6, device=device, dtype=dtype)crop = RandomCrop3D(size, padding=padding, pad_if_needed=pad_if_needed, p=probability)augmentation = AugmentationSequential(crop) if in_sequence else cropoutput = augmentation(volume)- expected_size = size if probability == 1.0 else (4, 5, 6)- assert output.shape == (0, 1, *expected_size)+ # An empty batch has no sample for the gate to skip (#4429).+ assert output.shape == (0, 1, *size)assert output.device == volume.deviceassert output.dtype == volume.dtypediff --git a/tests/augmentation/container/test_augmentation_sequential.py b/tests/augmentation/container/test_augmentation_sequential.pyindex 42b22aa87..ba778db84 100644--- a/tests/augmentation/container/test_augmentation_sequential.py+++ b/tests/augmentation/container/test_augmentation_sequential.py@@ -2551,13 +2551,14 @@ class TestConventionAugmentationSequential(BaseTester):)@pytest.mark.parametrize("p", [0.5, 1.0])def test_empty_batch_mask_round_trip_follows_image_4429(self, make_op, p, device, dtype):- # The mask takes the image's canvas under every gate, and the inverse restores both. With p < 1 the empty- # image keeps its input size (the blend fallback tracked in #4429), so only the parity is pinned here.+ # An empty batch takes the configured size under every gate (#4429): the mask follows the image onto that+ # canvas, and the inverse restores both.image = torch.empty(0, 3, 8, 10, device=device, dtype=dtype)mask = torch.empty(0, 1, 8, 10, device=device, dtype=torch.int64)seq = K.AugmentationSequential(make_op(p), data_keys=["input", "mask"])out_image, out_mask = seq(image, mask)- assert out_mask.shape == (0, 1, *out_image.shape[-2:])+ assert out_image.shape == (0, 3, 4, 5)+ assert out_mask.shape == (0, 1, 4, 5)restored_image, restored_mask = seq.inverse(out_image, out_mask)assert restored_image.shape == image.shapeassert restored_mask.shape == mask.shape@@ -3091,7 +3092,8 @@ class TestCrop3DShapePropagation(BaseTester):handle.remove()self.assert_close(actual, expected)saved = deepcopy(seq._params)- canvas = (4, 5, 6) if p == 1.0 else (8, 10, 12)+ # A skipped crop keeps its input canvas; an empty batch has no sample to skip (#4429).+ canvas = (4, 5, 6) if p == 1.0 or batch == 0 else (8, 10, 12)assert seen == [torch.Size((batch, 1, *canvas))]assert tuple(saved[1].data["forward_input_shape"]) == seen[0]assert saved[1].data["output_size"].shape == (batch, 3)diff --git a/tests/augmentation/test_base.py b/tests/augmentation/test_base.pyindex ad39b2fd6..d5a94e00a 100644--- a/tests/augmentation/test_base.py+++ b/tests/augmentation/test_base.py@@ -1253,6 +1253,35 @@ class TestConventionAugmentationBase2D(BaseTester):empty = torch.rand(0, 3, 6, 8, device=device, dtype=dtype)assert augmentation()(empty).shape == shape+ @pytest.mark.parametrize(+ ("augmentation", "input_shape", "output_size"),+ [+ pytest.param(lambda p: K.RandomCrop((4, 6), p=p), (3, 6, 8), (4, 6), id="RandomCrop"),+ pytest.param(lambda p: K.RandomCrop((4, 6), padding=1, p=p), (3, 6, 8), (4, 6), id="RandomCrop-padding"),+ pytest.param(lambda p: K.CenterCrop((4, 6), p=p), (3, 6, 8), (4, 6), id="CenterCrop"),+ pytest.param(lambda p: K.RandomResizedCrop((4, 4), p=p), (3, 6, 8), (4, 4), id="RandomResizedCrop"),+ pytest.param(lambda p: K.Resize((4, 5), p=p), (3, 6, 8), (4, 5), id="Resize-tuple"),+ pytest.param(lambda p: K.Resize(10, side="long", p=p), (3, 6, 8), (7, 10), id="Resize-int-long"),+ pytest.param(lambda p: K.LongestMaxSize(16, p=p), (3, 6, 8), (12, 16), id="LongestMaxSize"),+ pytest.param(lambda p: K.SmallestMaxSize(12, p=p), (3, 6, 8), (12, 16), id="SmallestMaxSize"),+ pytest.param(lambda p: K.RandomCrop3D((4, 4, 4), p=p), (1, 6, 8, 10), (4, 4, 4), id="RandomCrop3D"),+ pytest.param(lambda p: K.CenterCrop3D((4, 4, 4), p=p), (1, 6, 8, 10), (4, 4, 4), id="CenterCrop3D"),+ ],+ )+ @pytest.mark.parametrize("p", [0.0, 0.5, 1.0])+ def test_zero_batch_takes_configured_size_under_any_gate_4429(+ self, augmentation, input_shape, output_size, p, device, dtype+ ):+ # An empty batch has no sample for the gate to skip, so a shape-changing augmentation returns its configured+ # output size whatever ``p`` is (#4429), as 0.8.3 did for the 2D crops. A non-empty batch the gate skips+ # entirely still comes back at its input size.+ channels = input_shape[0]+ aug = augmentation(p)+ empty = torch.rand(0, *input_shape, device=device, dtype=dtype)+ assert aug(empty).shape == (0, channels, *output_size)+ skipped = torch.rand(2, *input_shape, device=device, dtype=dtype)+ assert augmentation(0.0)(skipped).shape == skipped.shape+_AUGMENTATIONS = [K.RandomEqualize, K.RandomEqualize3D, K.RandomPosterize, K.RandomSharpness]diff --git a/tests/augmentation/test_empty_inverse.py b/tests/augmentation/test_empty_inverse.pyindex 74a236a54..bd0a18a74 100644--- a/tests/augmentation/test_empty_inverse.py+++ b/tests/augmentation/test_empty_inverse.py@@ -152,10 +152,10 @@ class TestEmptyGeometricInverse(BaseTester):output = sequence(image)saved_params = deepcopy(sequence._params)- # With p < 1 an empty batch keeps its input size, because the blend falls back to the skipped- # branch (tracked in #4429). This pins the current forward, which the tracked shapes must follow.- crop_size = (4, 5) if crop_p == 1.0 else (height, width)- output_size = (3, 4) if resize_p == 1.0 else crop_size+ # An empty batch has no sample to skip, so every stage takes its configured output size whatever its+ # probability (#4429), as on 0.8.3 for the crops; the tracked shapes must follow that forward.+ crop_size = (4, 5)+ output_size = (3, 4)assert saved_params[0].data["forward_input_shape"].tolist() == [0, 3, height, width]assert saved_params[1].data["forward_input_shape"].tolist() == [0, 3, *crop_size]assert output.shape == (0, 3, *output_size)@@ -373,10 +373,8 @@ class TestEmptyGeometricInverse(BaseTester):pytest.skip("The forward resampling kernel does not support this device/dtype's backward pass.")image = torch.empty(0, 3, 6, 8, device=device, dtype=dtype, requires_grad=True)output = aug(image)- # With p < 1 an empty batch keeps its input size (blend fallback, tracked in #4429); the inverse- # must undo whichever size the forward produced.- expected_size = output_size if p == 1.0 else (6, 8)- assert output.shape == (0, 3, *expected_size)+ # An empty batch takes the configured output size whatever ``p`` is (#4429); the inverse undoes it.+ assert output.shape == (0, 3, *output_size)restored = aug.inverse(output)assert restored.shape == image.shapeassert restored.device == image.device@@ -519,9 +517,8 @@ class TestEmptyCenterResizedCropHistory(BaseTester):for key in standalone:self.assert_close(params[0].data[key], standalone[key], rtol=0, atol=0)assert params[0].data["forward_input_shape"].tolist() == [0, 3, *input_size]- # Follow the current empty-gate blend, not a desired probability specification (#4429).- expected_size = crop_size if p == 1.0 else input_size- assert params[1].data["forward_input_shape"].tolist() == [0, 3, *expected_size]+ # An empty gate takes the crop's configured size whatever ``p`` is, as the image forward does (#4429).+ assert params[1].data["forward_input_shape"].tolist() == [0, 3, *crop_size]if crop_type is K.CenterCrop:assert params[0].data["output_size"].shape == (0, 2)else: