Skip to content

🐍 Geometric Computer Vision Library for Spatial AI

fix(augmentation): run auto-augment operations with a magnitude after .to(device) (#6593)

Dmytro Mishkin8a48e199c65a
diff --git a/changelog.d/6593.breaking.md b/changelog.d/6593.breaking.md
new file mode 100644
index 000000000..da55aaf02
--- /dev/null
+++ b/changelog.d/6593.breaking.md
@@ -0,0 +1 @@
+`OperationBase.magnitude_range` used to be a plain attribute: assigning to it changed the bounds the magnitude is clamped to, and for an `nn.Parameter` range `nn.Module` also registered the parameter under that name. It is now a read-only property that reads the wrapped generator's current range, so assigning to it raises `AttributeError` (`KeyError` for an `nn.Parameter`); pass the range when building the operation instead. For an `nn.Parameter` range, `state_dict()` and `named_parameters()` no longer have a `magnitude_range` entry, and `parameters()` yields the range last rather than first, so an optimizer state saved before this change no longer matches and has to be rebuilt. State dicts and pickles saved before this change still load. A range assigned to `magnitude_range` that differs from the generator's is dropped without a warning, and the generator's range applies. In the other direction, on an older kornia a state dict with an `nn.Parameter` range saved now needs `strict=False`, and a pickled operation with a learnable magnitude or a pickled `RandAugment` raises `AttributeError`.
diff --git a/changelog.d/6593.fixed.md b/changelog.d/6593.fixed.md
new file mode 100644
index 000000000..0f907eb65
--- /dev/null
+++ b/changelog.d/6593.fixed.md
@@ -0,0 +1 @@
+Auto-augment operations with a magnitude (`Brightness`, `Rotate`, `ShearX`, `Posterize`, `TranslateX` and the rest) now run after `.to(device)`: `magnitude_range` reads the generator's current range, and the magnitude's factor is placed on the magnitude's device. `RandAugment` computes its magnitudes on the range's device, so a seeded `RandAugment` moved to an accelerator draws the signs of `rotate` and `shear` from that device's generator, and `.to(torch.float16)` or `.to(torch.bfloat16)` computes its magnitudes from the range in that dtype, a relative change of at most 2.6e-3.
diff --git a/kornia/augmentation/_2d/geometric/translate.py b/kornia/augmentation/_2d/geometric/translate.py
index a4e8a7aa4..842eae15f 100644
--- a/kornia/augmentation/_2d/geometric/translate.py
+++ b/kornia/augmentation/_2d/geometric/translate.py
@@ -107,7 +107,14 @@ class RandomTranslate(GeometricAugmentationBase2D):
def compute_transformation(
self, input: torch.Tensor, params: Dict[str, torch.Tensor], flags: Dict[str, Any]
) -> torch.Tensor:
- translations = torch.stack([params["translate_x"], params["translate_y"]], dim=-1)
+ translate_x, translate_y = params["translate_x"], params["translate_y"]
+ if translate_x.device != translate_y.device:
+ # A learned magnitude sits with its operation, and the generator's other draw stays where it was drawn. With
+ # both on one device, the stack below is moved once rather than each component: the shared form costs this
+ # method 16 to 19 % on CUDA (one extra copy, about 20 us; RTX 4090, torch 2.15).
+ translate_x = _as_tensor_nonblocking(translate_x, input.device, input.dtype)
+ translate_y = _as_tensor_nonblocking(translate_y, input.device, input.dtype)
+ translations = torch.stack([translate_x, translate_y], dim=-1)
return get_translation_matrix2d(_as_tensor_nonblocking(translations, input.device, input.dtype))
def apply_transform(
diff --git a/kornia/augmentation/auto/operations/base.py b/kornia/augmentation/auto/operations/base.py
index 7463b3d4d..650298aca 100644
--- a/kornia/augmentation/auto/operations/base.py
+++ b/kornia/augmentation/auto/operations/base.py
@@ -15,7 +15,7 @@
# limitations under the License.
#
-from typing import Callable, Dict, List, Optional, Tuple, TypeVar
+from typing import Any, Callable, Dict, List, Optional, Tuple, TypeVar, Union
import torch
from torch import nn
@@ -62,11 +62,15 @@ class OperationBase(nn.Module):
no part in sampling and receives no gradient: the gate is drawn from the wrapped augmentation's own
``p`` and ``p_batch`` as a hard ``0`` or ``1``. ``magnitude``, where the operation has one, is clamped to
the wrapped generator's range and receives a gradient where the wrapped augmentation is differentiable
- in it; ``forward_parameters`` substitutes it into the wrapped augmentation's draw.
+ in it; ``forward_parameters`` substitutes it into the wrapped augmentation's draw. The substituted factor
+ is on the device of the magnitude, while the other parameters stay where the generator returns them.
- ``forward`` linearly blends the wrapped output with the input using ``batch_prob``. The wrapped
augmentation transforms every row whose gate is above ``0``, so the same supplied fractional
``batch_prob`` produces the same soft blend whatever the wrapped ``p`` or ``p_batch``. Below ``p = 1`` the
wrapped augmentation skips a row with gate ``0``, so that row's matrix stays the identity.
+ - ``magnitude_range`` is read-only and reads the wrapped generator on each access. A range that is an
+ ``nn.Parameter`` belongs to the generator, so ``state_dict()`` and ``named_parameters()`` have no
+ ``magnitude_range`` entry for it; a checkpoint or pickle that still has the former entry loads.
- a symmetric magnitude applies the magnitude mapping first and then a random sign per row, so a mapping
that quantizes to zero stays zero (``Posterize`` maps ``0.5`` to ``0`` bits with ``magnitude_range=(0, 8)``).
- the concrete classes in ``kornia.augmentation.auto.operations.ops`` wrap public 2D augmentations and
@@ -107,6 +111,21 @@ class OperationBase(nn.Module):
self.symmetric_megnitude = symmetric_megnitude
self._magnitude_fn = self._init_magnitude_fn(magnitude_fn)
+ def __setstate__(self, state: Dict[str, Any]) -> None:
+ super().__setstate__(state)
+ # A module pickled when ``magnitude_range`` was an attribute brings it back, and ``_parameters`` would go on
+ # demanding the key below from every later load. The property reads the generator's range, which is the same one
+ # unless the attribute was assigned; an assigned range is dropped.
+ self._parameters.pop("magnitude_range", None)
+ self.__dict__.pop("magnitude_range", None)
+
+ def _load_from_state_dict(self, state_dict: Dict[str, Any], prefix: str, *args: Any, **kwargs: Any) -> None:
+ # ``magnitude_range`` was an attribute, so ``nn.Module`` registered a parameter range under that name as well.
+ # Checkpoints from then hold the same values under the generator's own key, unless a parameter was assigned to
+ # the attribute. The generator's key is the one that loads.
+ state_dict.pop(f"{prefix}magnitude_range", None)
+ super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
+
def _init_magnitude_fn(
self, magnitude_fn: Optional[Callable[[torch.Tensor], torch.Tensor]]
) -> Callable[[torch.Tensor], torch.Tensor]:
@@ -128,13 +147,12 @@ class OperationBase(nn.Module):
if initial_magnitude is None:
self._factor_name = None
self._magnitude = None
- self.magnitude_range = None
else:
self._factor_name = initial_magnitude[0][0]
- if self.op._param_generator is not None:
- self.magnitude_range = getattr(self.op._param_generator, self._factor_name)
- else:
+ if self.op._param_generator is None:
raise ValueError(f"No valid magnitude `{self._factor_name}` found in `{self.op._param_generator}`.")
+ # Reject a name the generator does not have now, rather than at the first call of ``magnitude_range``.
+ getattr(self.op._param_generator, self._factor_name)
self._magnitude = None
if initial_magnitude[0][1] is not None:
@@ -161,7 +179,11 @@ class OperationBase(nn.Module):
if mag is not None:
if self._factor_name is None:
raise RuntimeError("No factor found in the params while `mag` is provided.")
- params[self._factor_name] = params[self._factor_name].zero_() + mag
+ # The generator returns its draw on its own device (the CPU by default, even after ``.to()``), so build the
+ # replacement on the device of ``mag``: this copies nothing and waits for nothing.
+ draw = params[self._factor_name]
+ device = mag.device if isinstance(mag, torch.Tensor) else draw.device
+ params[self._factor_name] = torch.zeros_like(draw, device=device) + mag
if self._factor_name is not None:
params[self._factor_name] = self._magnitude_fn(params[self._factor_name])
@@ -206,6 +228,21 @@ class OperationBase(nn.Module):
return self.op.transform_matrix
return None
+ @property
+ def magnitude_range(self) -> Optional[Union[torch.Tensor, Tuple[float, float]]]:
+ """Return the wrapped generator's range for the magnitude, as it holds it at this moment.
+
+ It is read on every access rather than stored: ``.to()`` replaces the tensors of the generator, so a reference
+ kept here would go on pointing at the tensor from before the move.
+
+ Returns:
+ A tensor of shape ``(2,)`` on the device the generator keeps it on, or the ``(low, high)`` pair of a
+ generator that keeps the constructor's numbers. ``None`` when this operation has no magnitude.
+ """
+ if self._factor_name is None:
+ return None
+ return getattr(self.op._param_generator, self._factor_name)
+
@property
def magnitude(self) -> Optional[torch.Tensor]:
"""Return the learned magnitude value for the operation.
@@ -218,9 +255,13 @@ class OperationBase(nn.Module):
if self._magnitude is None:
return None
mag = self._magnitude
- if self.magnitude_range is not None:
- return mag.clamp(*self.magnitude_range)
- return mag
+ bounds = self.magnitude_range
+ if bounds is None:
+ return mag
+ # ``set_rng_device_and_dtype`` can move a range without moving this parameter.
+ if isinstance(bounds, torch.Tensor):
+ bounds = _to_device_nonblocking(bounds, mag.device)
+ return mag.clamp(*bounds)
@property
def probability(self) -> torch.Tensor:
diff --git a/kornia/augmentation/auto/rand_augment/rand_augment.py b/kornia/augmentation/auto/rand_augment/rand_augment.py
index 9c6d6b2df..d842b6887 100644
--- a/kornia/augmentation/auto/rand_augment/rand_augment.py
+++ b/kornia/augmentation/auto/rand_augment/rand_augment.py
@@ -24,6 +24,7 @@ from kornia.augmentation.auto.base import SUBPOLICY_CONFIG, PolicyAugmentBase
from kornia.augmentation.auto.operations import OperationBase
from kornia.augmentation.auto.operations.policy import PolicySequential
from kornia.augmentation.container.params import ParamItem
+from kornia.core.utils import _to_device_nonblocking
from . import ops
@@ -176,12 +177,14 @@ class RandAugment(PolicyAugmentBase):
op = cast(PolicySequential, module)[0]
op = cast(OperationBase, op)
mag = None
- if op.magnitude_range is not None:
- minval, maxval = op.magnitude_range
- if op._factor_name == "bits_factor":
- mag = (1 - m) * float(maxval - minval) + minval
- else:
- mag = m * float(maxval - minval) + minval
+ magnitude_range = op.magnitude_range
+ if magnitude_range is not None:
+ minval, maxval = magnitude_range
+ scale = 1 - m if op._factor_name == "bits_factor" else m
+ # A moved policy has its range on the accelerator: stay there rather than read the range on the host.
+ if isinstance(minval, torch.Tensor):
+ scale = _to_device_nonblocking(scale, minval.device)
+ mag = scale * (maxval - minval) + minval
mod_param = op.forward_parameters(batch_shape, mag=mag)
# Compose it
diff --git a/testing/augmentation/sync_registry.py b/testing/augmentation/sync_registry.py
index 59cc24b94..f8190d713 100644
--- a/testing/augmentation/sync_registry.py
+++ b/testing/augmentation/sync_registry.py
@@ -420,7 +420,9 @@ INHERENT_SITES: dict[str, str] = {
# Thin wrappers whose synchronizations belong to their caller. A site is the deepest kornia frame that is not one of
# them, so that INHERENT_SITES names the caller, which is what decides whether the synchronization is inherent.
-PASS_THROUGH: frozenset[str] = frozenset({"core/utils.py:_torch_inverse_cast", "core/utils.py:_torch_solve_cast"})
+PASS_THROUGH: frozenset[str] = frozenset(
+ {"core/utils.py:_to_device_nonblocking", "core/utils.py:_torch_inverse_cast", "core/utils.py:_torch_solve_cast"}
+)
def _inputs_2d(device: torch.device, dtype: torch.dtype) -> dict[str, torch.Tensor]:
diff --git a/tests/augmentation/test_auto_operation.py b/tests/augmentation/test_auto_operation.py
index 951604cf3..0c153b2ea 100644
--- a/tests/augmentation/test_auto_operation.py
+++ b/tests/augmentation/test_auto_operation.py
@@ -17,13 +17,15 @@
import copy
import inspect
+import pickle
from typing import List
import pytest
import torch
+import kornia.augmentation as K
from kornia.augmentation.auto.autoaugment import AutoAugment
-from kornia.augmentation.auto.operations import OperationBase, ops
+from kornia.augmentation.auto.operations import OperationBase, PolicySequential, ops
from kornia.augmentation.auto.rand_augment.rand_augment import RandAugment
from kornia.augmentation.auto.rand_augment.rand_augment import default_policy as randaug_config
from kornia.augmentation.auto.trivial_augment import TrivialAugment
@@ -296,9 +298,7 @@ def test_operation_preserves_input_dtype(device, low_dtype):
# input to float32. float32 / float64 inputs are unaffected (promotion goes
# up), which is why this only shows below float32. Covers RandAugment /
# AutoAugment / TrivialAugment / AugMix, which all route through this method.
- # dtype preservation is device-independent; check it on CPU (the auto-augment
- # RNG state of a freshly built ``_find_all_ops()`` op does not fully follow a
- # post-hoc ``.to(non-cpu)``).
+ # dtype preservation is device-independent; check it on CPU.
if torch.device(device).type != "cpu":
pytest.skip("dtype-preservation check runs on CPU")
checked = 0
@@ -579,3 +579,345 @@ def test_policy_brightness_uses_multiplicative_factor_6253(policy, device, dtype
factor = params[0].data[0].data["brightness_factor"].to(image)
expected = (image * factor[:, None, None, None]).clamp(0, 1)
assert_close(aug(image, params=params), expected)
+
+
+# Every wrapper with a magnitude. ``Posterize`` truncates it with ``long`` and ``Solarize`` compares against it, so
+# their output carries no gradient for it on any device.
+_MAGNITUDE_OPS = [
+ op
+ for _, op in inspect.getmembers(ops, inspect.isclass)
+ if issubclass(op, OperationBase)
+ and op is not OperationBase
+ and "initial_magnitude" in inspect.signature(op).parameters
+]
+_NOT_DIFFERENTIABLE_IN_MAGNITUDE = (ops.Posterize, ops.Solarize)
+_DIFFERENTIABLE_OPS = [op for op in _MAGNITUDE_OPS if op not in _NOT_DIFFERENTIABLE_IN_MAGNITUDE]
+_MAGNITUDE_OP_IDS = [op.__name__ for op in _MAGNITUDE_OPS]
+# One wrapper per place a generator keeps its range: a buffer, an attribute ``make_samplers`` rebuilds, a raw tuple.
+_RANGE_KINDS = [ops.Brightness, ops.ShearX, ops.Posterize, ops.TranslateX]
+
+
+def _round_trip_pickle(module):
+ return pickle.loads(pickle.dumps(module)) # noqa: S301
+
+
+def _assert_magnitude_gradient(output, operation):
+ (gradient,) = torch.autograd.grad(output.square().sum(), operation._magnitude)
+ assert torch.isfinite(gradient).all(), type(operation).__name__
+ assert gradient.abs().sum() > 0, type(operation).__name__
+
+
+class TestOperationAfterTo6297(BaseTester):
+ """``.to()`` moves a wrapper's magnitude, range and generator together.
+
+ The range was a plain attribute holding the tensor from before the move, and the generators hand their draws back
+ on the CPU after one. A run on the CPU alone cannot fail most of these, so they need ``--device`` of an accelerator.
+ """
+
+ @pytest.mark.parametrize("operation", _MAGNITUDE_OPS, ids=_MAGNITUDE_OP_IDS)
+ def test_forward_parameters_puts_the_factor_with_the_magnitude(self, operation, device, dtype):
+ moved = operation(initial_probability=1.0).to(device=device, dtype=dtype)
+ params = moved.forward_parameters(torch.Size([2, 3, 16, 16]))
+ assert params[moved._factor_name].device == moved._magnitude.device == device
+ assert params[moved._factor_name].shape == (2,)
+
+ @pytest.mark.parametrize("probability", [1.0, 0.5])
+ @pytest.mark.parametrize("operation", _MAGNITUDE_OPS, ids=_MAGNITUDE_OP_IDS)
+ def test_forward_and_replay(self, operation, probability, device, dtype):
+ moved = operation(initial_probability=probability).to(device=device, dtype=dtype)
+ image = torch.rand(2, 3, 16, 16, device=device, dtype=dtype)
+ assert moved(image).device == device
+ params = moved.forward_parameters(image.shape)
+ self.assert_close(moved(image, params=params), moved(image, params=params))
+
+ @pytest.mark.parametrize("operation", _DIFFERENTIABLE_OPS, ids=[op.__name__ for op in _DIFFERENTIABLE_OPS])
+ def test_magnitude_keeps_its_gradient(self, operation, device, dtype):
+ moved = operation(initial_probability=1.0).to(device=device, dtype=dtype)
+ image = torch.rand(2, 3, 16, 16, device=device, dtype=dtype)
+ _assert_magnitude_gradient(moved(image), moved)
+ params = moved.forward_parameters(image.shape)
+ _assert_magnitude_gradient(moved(image, params=params), moved)
+
+ @pytest.mark.parametrize("operation", _MAGNITUDE_OPS, ids=_MAGNITUDE_OP_IDS)
+ def test_magnitude_stays_clamped_to_the_range(self, operation, device, dtype):
+ moved = operation(initial_probability=1.0).to(device=device, dtype=dtype)
+ low, high = moved.magnitude_range
+ with torch.no_grad():
+ moved._magnitude.fill_(float(high) + 1.0)
+ self.assert_close(moved.magnitude, torch.full((1,), float(high), device=device, dtype=dtype))
+ with torch.no_grad():
+ moved._magnitude.fill_(float(low) - 1.0)
+ self.assert_close(moved.magnitude, torch.full((1,), float(low), device=device, dtype=dtype))
+
+ def test_magnitude_on_the_device_with_the_operation_elsewhere(self, device, dtype):
+ # ``mag`` overrides the draw of a wrapper left where it was built, so the factor follows ``mag``.
+ operation = ops.Brightness(None, 1.0)
+ image = torch.rand(2, 3, 16, 16, device=device, dtype=dtype)
+ mag = torch.tensor(0.5, device=device, dtype=dtype, requires_grad=True)
+ params = operation.forward_parameters(image.shape, mag=mag)
+ assert params["brightness_factor"].device == device
+ (gradient,) = torch.autograd.grad(operation(image, params=params).sum(), mag)
+ assert gradient.abs() > 0
+
+ @pytest.mark.parametrize("operation", _RANGE_KINDS, ids=[op.__name__ for op in _RANGE_KINDS])
+ def test_generator_moved_on_its_own(self, operation, device, dtype):
+ # ``set_rng_device_and_dtype`` moves the draws and, for ``ShearGenerator``, the range they come from, but not
+ # the magnitude of the wrapper. The clamp has to work across that.
+ wrapper = operation(initial_probability=1.0)
+ wrapper.op.set_rng_device_and_dtype(device, dtype)
+ assert wrapper.magnitude.device == wrapper._magnitude.device
+ wrapper.forward_parameters(torch.Size([2, 3, 16, 16]))
+
+ def test_policy_sequential_keeps_every_magnitude_gradient(self, device, dtype):
+ # ``Posterize`` runs first: its output has a gradient of zero, which would cut the graph of what runs before it.
+ learnable = (ops.Brightness, ops.ShearX, ops.TranslateY, ops.Rotate)
+ policy = PolicySequential(
+ ops.Posterize(initial_probability=1.0), *[op(initial_probability=1.0) for op in learnable]
+ ).to(device=device, dtype=dtype)
+ image = torch.rand(2, 3, 16, 16, device=device, dtype=dtype)
+ params = policy.forward_parameters(image.shape)
+ assert len(params) == len(learnable) + 1
+ self.assert_close(policy(image, params=params), policy(image, params=params))
+ output = policy(image)
+ assert output.device == device
+ magnitudes = [operation._magnitude for operation in list(policy.children())[1:]]
+ gradients = torch.autograd.grad(output.square().sum(), magnitudes)
+ for gradient in gradients:
+ assert torch.isfinite(gradient).all()
+ assert gradient.abs().sum() > 0
+
+ @pytest.mark.parametrize(
+ "make_policy",
+ [
+ pytest.param(lambda: RandAugment(n=len(randaug_config), m=15), id="RandAugment"),
+ pytest.param(TrivialAugment, id="TrivialAugment"),
+ pytest.param(AutoAugment, id="AutoAugment"),
+ ],
+ )
+ def test_builtin_policy(self, make_policy, device, dtype):
+ moved = make_policy().to(device=device, dtype=dtype)
+ image = torch.rand(2, 3, 16, 16, device=device, dtype=dtype)
+ output = moved(image)
+ assert output.device == device
+ self.assert_close(moved(image, params=moved._params), output)
+ params = moved.forward_parameters(image.shape)
+ self.assert_close(moved(image, params=params), moved(image, params=params))
+
+ @pytest.mark.parametrize(
+ "make_policy",
+ [
+ pytest.param(lambda: RandAugment(n=3, m=15), id="RandAugment"),
+ pytest.param(TrivialAugment, id="TrivialAugment"),
+ pytest.param(AutoAugment, id="AutoAugment"),
+ ],
+ )
+ def test_builtin_policy_in_a_container(self, make_policy, device, dtype):
+ _test_sequential(make_policy().to(device=device, dtype=dtype), device=device, dtype=dtype)
+
+ def test_transformation_matrix_of_a_policy_with_magnitudes(self, device, dtype):
+ policy = PolicySequential(
+ ops.TranslateX(initial_probability=1.0),
+ ops.ShearY(initial_probability=1.0),
+ ops.Rotate(initial_probability=1.0),
+ ).to(device=device, dtype=dtype)
+ image = torch.rand(2, 3, 16, 16, device=device, dtype=dtype)
+ params = policy.forward_parameters(image.shape)
+ policy(image, params=params)
+ recomputed = policy.get_transformation_matrix(image, params=params, recompute=True)
+ self.assert_close(policy.transform_matrix, recomputed)
+
+ @pytest.mark.parametrize("candidate", randaug_config, ids=[subpolicy[0][0] for subpolicy in randaug_config])
+ def test_rand_augment_magnitude_matches_the_unmoved_policy(self, candidate, device, dtype):
+ # A draw selects ``n`` candidates, so go through each. The range of a moved policy is on the device: reading it
+ # has to neither mix devices nor change the magnitude ``m`` selects from it.
+ moved = RandAugment(n=1, m=10, policy=[candidate]).to(device=device, dtype=dtype)
+ image = torch.rand(2, 3, 16, 16, device=device, dtype=dtype)
+ params = moved.forward_parameters(image.shape)
+ self.assert_close(moved(image, params=params), moved(image, params=params))
+ factor = moved[0][0]._factor_name
+ if factor is None:
+ return
+ unmoved = RandAugment(n=1, m=10, policy=[candidate]).forward_parameters(image.shape)
+ self.assert_close(
+ params[0].data[0].data[factor].abs().to(device=device, dtype=dtype),
+ unmoved[0].data[0].data[factor].abs().to(device=device, dtype=dtype),
+ )
+
+ @pytest.mark.parametrize("copier", [copy.deepcopy, _round_trip_pickle], ids=["deepcopy", "pickle"])
+ @pytest.mark.parametrize("operation", _RANGE_KINDS, ids=[op.__name__ for op in _RANGE_KINDS])
+ def test_copy_of_a_moved_operation_owns_its_magnitude(self, operation, copier, device, dtype):
+ moved = operation(initial_probability=1.0).to(device=device, dtype=dtype)
+ image = torch.rand(2, 3, 16, 16, device=device, dtype=dtype)
+ moved(image)
+ clone = copier(moved)
+ assert clone._magnitude is not moved._magnitude
+ assert clone._magnitude.device == device
+ assert clone.magnitude_range is getattr(clone.op._param_generator, clone._factor_name)
+ output = clone(image)
+ if operation in _NOT_DIFFERENTIABLE_IN_MAGNITUDE:
+ return
+ _assert_magnitude_gradient(output, clone)
+ assert moved._magnitude.grad is None
+
+ def test_state_dict_round_trip_after_to(self, device, dtype):
+ source = ops.Brightness(initial_probability=1.0, magnitude_range=(0.4, 0.6)).to(device=device, dtype=dtype)
+ loaded = ops.Brightness(initial_probability=1.0, magnitude_range=(0.2, 1.8)).to(device=device, dtype=dtype)
+ loaded.load_state_dict(source.state_dict(), strict=True)
+ self.assert_close(loaded.magnitude_range, source.magnitude_range)
+ self.assert_close(loaded.magnitude, source.magnitude)
+ loaded(torch.rand(2, 3, 16, 16, device=device, dtype=dtype))
+
+ @pytest.mark.device_agnostic
+ @pytest.mark.parametrize("operation", _MAGNITUDE_OPS, ids=_MAGNITUDE_OP_IDS)
+ def test_range_follows_a_dtype_move(self, operation):
+ # What the CPU can see of the stale range: ``.to()`` replaces the tensors of the generator, on any device.
+ moved = operation(initial_probability=1.0).to(torch.float64)
+ assert moved.magnitude_range is getattr(moved.op._param_generator, moved._factor_name)
+ assert moved.magnitude.dtype == torch.float64
+
+ @pytest.mark.device_agnostic
+ @pytest.mark.parametrize("operation", _MAGNITUDE_OPS, ids=_MAGNITUDE_OP_IDS)
+ def test_factor_follows_a_magnitude_on_another_device(self, operation):
+ # What the CPU can see of the draw's placement: a magnitude on ``meta`` meets a draw on the CPU.
+ wrapper = operation(None, 1.0)
+ params = wrapper.forward_parameters(torch.Size([2, 3, 8, 8]), mag=torch.zeros(1, device="meta"))
+ factor = params[wrapper._factor_name]
+ assert factor.device.type == "meta"
+ assert factor.shape == (2,)
+
+ @pytest.mark.device_agnostic
+ @pytest.mark.parametrize("operation", _MAGNITUDE_OPS, ids=_MAGNITUDE_OP_IDS)
+ def test_a_number_still_replaces_the_draw(self, operation):
+ # The annotation says tensor, but a plain number has always worked, as it has no device to follow.
+ wrapper = operation(None, 1.0, symmetric_megnitude=False)
+ factor = wrapper.forward_parameters(torch.Size([2, 3, 8, 8]), mag=0.25)[wrapper._factor_name]
+ assert factor.shape == (2,)
+
+ @pytest.mark.device_agnostic
+ def test_factor_name_the_generator_lacks_raises_at_construction(self):
+ with pytest.raises(AttributeError, match="nonsense"):
+ OperationBase(K.RandomRotation(30.0), initial_magnitude=[("nonsense", 1.0)])
+
+ @pytest.mark.device_agnostic
+ def test_state_dict_keys_stay_the_same(self):
+ # Checkpoints embed these names, so following a range that is not a parameter must not add one. A parameter
+ # range did have a key of its own, which the tests below cover.
+ common = {"_magnitude", "_probability", "temperature"}
+ assert set(ops.Brightness().state_dict()) == common | {"op._param_generator.brightness_factor"}
+ assert set(ops.ShearX().state_dict()) == common
+ assert set(ops.Posterize().state_dict()) == common
+ assert set(ops.TranslateX().state_dict()) == common
+
+ @pytest.mark.device_agnostic
+ def test_deepcopy_with_a_learnable_shear_range(self):
+ # ``ShearGenerator`` derives the range in ``make_samplers``, so with a parameter it holds autograd history. A
+ # copy of the generator rebuilds it (#6541), and the wrapper must not keep a reference of its own.
+ wrapper = OperationBase(
+ K.RandomShear(torch.nn.Parameter(torch.tensor([0.0, 0.3])), p=1.0, align_corners=True),
+ initial_magnitude=[("shear_x", 0.1)],
+ )
+ clone = copy.deepcopy(wrapper)
+ assert clone.magnitude_range is clone.op._param_generator.shear_x
+ assert clone.magnitude_range is not wrapper.magnitude_range
+
+
+def _wrapper_with_learnable_range(low=0.0, high=30.0):
+ return OperationBase(
+ K.RandomRotation(torch.nn.Parameter(torch.tensor([low, high])), p=1.0), initial_magnitude=[("degrees", 5.0)]
+ )
+
+
+# The ``state_dict`` of ``_wrapper_with_learnable_range`` before #6297. ``magnitude_range`` was an attribute, so
+# ``nn.Module`` registered a parameter range on the wrapper too, and checkpoints hold that alias beside the generator's.
+_OLD_LEARNABLE_RANGE_KEYS = (
+ "magnitude_range",
+ "_magnitude",
+ "_probability",
+ "temperature",
+ "op._param_generator.degrees",
+)
+
+
+def _old_learnable_range_state_dict(prefix=""):
+ return {
+ f"{prefix}magnitude_range": torch.tensor([0.0, 20.0]),
+ f"{prefix}_magnitude": torch.tensor([7.0]),
+ f"{prefix}_probability": torch.tensor([0.5]),
+ f"{prefix}temperature": torch.tensor([0.1]),
+ f"{prefix}op._param_generator.degrees": torch.tensor([0.0, 20.0]),
+ }
+
+
+class TestLearnableMagnitudeRangeCheckpoint6297(BaseTester):
+ def test_old_state_dict_loads_strictly(self, device, dtype):
+ old = _old_learnable_range_state_dict()
+ assert tuple(old) == _OLD_LEARNABLE_RANGE_KEYS
+ wrapper = _wrapper_with_learnable_range().to(device=device, dtype=dtype)
+ wrapper.load_state_dict(old, strict=True)
+ expected = torch.tensor([0.0, 20.0], device=device, dtype=dtype)
+ self.assert_close(wrapper.magnitude_range, expected)
+ self.assert_close(wrapper.op._param_generator.degrees, expected)
+ self.assert_close(wrapper._magnitude, torch.tensor([7.0], device=device, dtype=dtype))
+ assert "magnitude_range" in old, "the caller's dict stays as it was"
+
+ def test_old_state_dict_loads_strictly_inside_a_policy(self, device, dtype):
+ policy = PolicySequential(_wrapper_with_learnable_range()).to(device=device, dtype=dtype)
+ name = next(iter(policy._modules))
+ policy.load_state_dict(_old_learnable_range_state_dict(f"{name}."), strict=True)
+ self.assert_close(policy[0].magnitude_range, torch.tensor([0.0, 20.0], device=device, dtype=dtype))
+
+ @pytest.mark.device_agnostic
+ def test_only_the_old_key_is_forgiven(self):
+ wrapper = _wrapper_with_learnable_range()
+ unexpected = {**_old_learnable_range_state_dict(), "nonsense": torch.zeros(1)}
+ with pytest.raises(RuntimeError, match=r'Unexpected key.*"nonsense"') as error:
+ wrapper.load_state_dict(unexpected, strict=True)
+ assert "magnitude_range" not in str(error.value)
+ missing = _old_learnable_range_state_dict()
+ del missing["op._param_generator.degrees"]
+ with pytest.raises(RuntimeError, match=r"Missing key.*op\._param_generator\.degrees"):
+ wrapper.load_state_dict(missing, strict=True)
+
+ def test_state_dict_round_trip(self, device, dtype):
+ source = _wrapper_with_learnable_range(0.0, 20.0).to(device=device, dtype=dtype)
+ state = source.state_dict()
+ assert set(state) == {"_magnitude", "_probability", "temperature", "op._param_generator.degrees"}
+ loaded = _wrapper_with_learnable_range().to(device=device, dtype=dtype)
+ loaded.load_state_dict(state, strict=True)
+ self.assert_close(loaded.magnitude_range, source.magnitude_range)
+
+ def test_optimizer_on_the_moved_wrapper_holds_each_parameter_once(self, device, dtype):
+ wrapper = _wrapper_with_learnable_range().to(device=device, dtype=dtype)
+ optimizer = torch.optim.SGD(wrapper.parameters(), lr=0.1)
+ held = [parameter for group in optimizer.param_groups for parameter in group["params"]]
+ assert {id(parameter) for parameter in held} == {
+ id(wrapper._magnitude),
+ id(wrapper._probability),
+ id(wrapper.magnitude_range),
+ }
+ assert len(held) == 3
+ assert all(parameter.device == device for parameter in held)
+ wrapper(torch.rand(2, 3, 16, 16, device=device, dtype=dtype)).square().sum().backward()
+ before = wrapper._magnitude.detach().clone()
+ optimizer.step()
+ assert not torch.equal(wrapper._magnitude, before)
+
+ @pytest.mark.device_agnostic
+ @pytest.mark.parametrize("learnable", [True, False], ids=["parameter", "buffer"])
+ def test_wrapper_pickled_with_the_old_attribute(self, learnable):
+ # A whole module saved before #6297 brings the attribute back with it: in ``_parameters`` for a parameter range,
+ # where it would demand the old key from every later load, and otherwise as a stale tensor in ``__dict__``.
+ wrapper = _wrapper_with_learnable_range() if learnable else ops.Brightness()
+ generator_range = wrapper.magnitude_range
+ state = dict(wrapper.__dict__)
+ if learnable:
+ state["_parameters"] = {"magnitude_range": generator_range, **state["_parameters"]}
+ else:
+ state["magnitude_range"] = generator_range.clone()
+ restored = type(wrapper).__new__(type(wrapper))
+ restored.__setstate__(state)
+ assert "magnitude_range" not in restored.__dict__
+ assert restored.magnitude_range is generator_range
+ assert "magnitude_range" not in restored.state_dict()
+ assert "magnitude_range" not in dict(restored.named_parameters())
+ restored.load_state_dict(wrapper.state_dict(), strict=True)
diff --git a/tests/augmentation/test_host_sync.py b/tests/augmentation/test_host_sync.py
index 87bb7d0e6..089287533 100644
--- a/tests/augmentation/test_host_sync.py
+++ b/tests/augmentation/test_host_sync.py
@@ -68,6 +68,17 @@ _RANGES_ON_CUDA = {
"RandomRotation": lambda r: K.RandomRotation(r(-10.0, 10.0), p=1.0),
"RandomGamma": lambda r: K.RandomGamma(r(0.8, 1.2), r(0.9, 1.1), p=1.0),
}
+# Auto operations and policies moved to CUDA as a whole (#6297). The range they read from their generator is on CUDA,
+# where reading its values on the host would wait for the GPU.
+_MOVED_AUTO = {
+ "Brightness": lambda: KAuto.operations.ops.Brightness(initial_probability=1.0),
+ "Posterize": lambda: KAuto.operations.ops.Posterize(initial_probability=1.0),
+ "ShearX": lambda: KAuto.operations.ops.ShearX(initial_probability=1.0),
+ "TranslateX": lambda: KAuto.operations.ops.TranslateX(initial_probability=1.0),
+ "RandAugment-rotate": lambda: KAuto.RandAugment(n=1, m=10, policy=[[("rotate", -30.0, 30.0)]]),
+ "RandAugment-shear_x": lambda: KAuto.RandAugment(n=1, m=10, policy=[[("shear_x", -0.3, 0.3)]]),
+ "RandAugment-posterize": lambda: KAuto.RandAugment(n=1, m=10, policy=[[("posterize", 4.0, 8.0)]]),
+}
# Host reads that only happen because the caller moved the parameters to CUDA: the crop window and the padding set
# the sizes of Python slices.
_REPLAY_DEVICE_READS = {
@@ -158,6 +169,18 @@ class TestAugmentationHostSync(BaseTester):
outputs[0].sum().backward()
assert all(p.grad is not None for p in op.parameters())
+ @pytest.mark.parametrize("name", sorted(_MOVED_AUTO))
+ def test_moved_auto_operation_adds_no_sync_of_its_own(self, name, device, dtype) -> None:
+ if device.type != "cuda":
+ pytest.skip("CUDA host synchronizations need a CUDA device")
+ module = _MOVED_AUTO[name]().to(device=device, dtype=dtype)
+ x = torch.rand(4, 3, 32, 32, device=device, dtype=dtype)
+ module(x)
+ torch.cuda.synchronize()
+ sites = sync_sites(lambda: module(x))
+ # The generators return their draws on the CPU (see the conventions page), which is not the wrapper's doing.
+ assert not [site for site in sites if site.startswith("augmentation/auto/")], dict(sites)
+
@pytest.mark.parametrize("same_on_batch", [False, True])
@pytest.mark.parametrize("p", [0.0, 0.5, 1.0])
@pytest.mark.parametrize("name", sorted(_GATED))
diff --git a/tests/core/test_onnx_mixin.py b/tests/core/test_onnx_mixin.py
index ef195634d..c9cfcb203 100644
--- a/tests/core/test_onnx_mixin.py
+++ b/tests/core/test_onnx_mixin.py
@@ -124,7 +124,7 @@ _NOT_CALL_STATE = {
"ImageSequential.disable_features": (("_disable_features",), "a property setter"),
"SequentialBase.keepdim": (("_keepdim",), "a property setter"),
"SequentialBase.same_on_batch": (("_same_on_batch",), "a property setter"),
- "OperationBase._init_magnitude": (("_factor_name", "_magnitude", "magnitude_range"), "a constructor helper"),
+ "OperationBase._init_magnitude": (("_factor_name", "_magnitude"), "a constructor helper"),
"TransformMatrixMinIn._parse_transformation_matrix_mode": (("_transformation_matrix_arg",), "a constructor helper"),
"ImageSequentialBase.transform_inputs": (
("_sequence_matrices",),