Skip to content

🐍 Geometric Computer Vision Library for Spatial AI

12.6 KB
Raw Blame
Base Classes
============

.. meta::
   :description: The Base Classes module in Kornia provides foundational classes for creating new image transformations. It supports rigid (e.g., affine) and non-rigid (e.g., cut-out) augmentations, with predefined routines for sampling, applying, and reversing transformations.

.. currentmodule:: kornia.augmentation

These are the base classes for creating a new transform on top of the predefined routine of `kornia.augmentation`.
Any given augmentation can be classified as either rigid (e.g. affine transformations that
manipulate images with a standard transformation matrix) or non-rigid (e.g. cutting out a random area). At the
image level, Kornia provides `GeometricAugmentationBase2D` for rigid transformations that modify the geometric
location of image pixels, `IntensityAugmentationBase2D` for transformations that preserve pixel locations, and the
generic `AugmentationBase2D`, which allows more freedom for customized augmentation design.

The Base-Class Hierarchy
------------------------

The bases form a chain, each layer adding one concern and shared by several subclasses β€” so pick the
*shallowest* base that already does what you need:

.. code-block:: text

   nn.Module
   └─ _BasicAugmentationBase          parameter sampling + the forward skeleton
      β”œβ”€ _AugmentationBase            dispatch to image / mask / box / keypoint / class data keys
      β”‚  β”œβ”€ AugmentationBase2D        2D tensor validation  (subclass for a fully custom 2D op)
      β”‚  β”‚  └─ RigidAffineAugmentationBase2D     transform-matrix machinery
      β”‚  β”‚     β”œβ”€ IntensityAugmentationBase2D    intensity ops β€” override apply_transform
      β”‚  β”‚     └─ GeometricAugmentationBase2D    warp ops β€” also override compute_transformation
      β”‚  └─ AugmentationBase3D … (the 3D mirror of the 2D chain)
      └─ MixAugmentationBaseV2        mix ops (MixUp / CutMix) β€” bypass the per-key dispatch

Each level is a distinct, reused axis (sampling, data-key dispatch, 2D vs 3D, rigid-matrix vs free-form,
intensity vs geometric); the four ``*Base2D`` classes are public API that external code subclasses.
For a custom augmentation, subclass ``IntensityAugmentationBase2D`` or ``GeometricAugmentationBase2D``.

The Predefined Augmentation Routine
-----------------------------------

Kornia augmentations generally follow a `sample-apply` routine.

- `sample`: Kornia aims at flexible tensor-level augmentations that augment every image in a batch with
  different parameters and probabilities. The sampling step first draws a set of random
  parameters. The sampled augmentation state is then stored in the ``_params`` attribute of the augmentation,
  for replay. Application-time draws can require additional RNG control; see the reproducibility notes below.
- `apply`: with the generated (or user-provided) parameters, the augmentation is performed accordingly.
  Apart from transforming image tensors, Kornia also supports inverse operations that revert the transform,
  and transforms of other data modalities (`data keys` in Kornia) such as masks, keypoints, and bounding boxes.
  These features depend on the concrete operation and its data-key handlers. `AugmentationSequential` passes the
  matrix of the call's parameters to the mask, box and keypoint handlers of geometric children and of custom
  `RigidAffineAugmentationBase2D` subclasses, recomputing it from replayed ``params`` that the child did not last
  apply to an image or that were edited since. An in-place edit made inside ``torch.inference_mode`` is not seen,
  because inference tensors have no version counter; replace the value instead.
  `RandomElasticTransform`, `RandomThinPlateSpline` and `RandomFisheye` have no matrix, so a forward call through
  a pipeline that contains one refuses keypoint and box data keys. The container does not call the box and
  keypoint handlers of a custom child that subclasses `AugmentationBase2D` directly, so those keys come back
  unchanged; subclass `GeometricAugmentationBase2D` to move them
  (`#4420 <https://github.com/kornia/kornia/issues/4420>`_).

Custom Augmentation Classes
---------------------------

`IntensityAugmentationBase2D` supplies an identity matrix and default passthrough handlers for most annotations.
Subclasses can override these defaults: `RandomErasing` zero-fills the erased mask region. Direct and container
calls pass masks, boxes, and keypoints through unchanged unless a subclass overrides
`apply_transform_mask`, `apply_transform_box`, or `apply_transform_keypoint`.
`GeometricAugmentationBase2D` supplies the dispatch used for geometric coordinate transformations.

For a geometric operation, implement `compute_transformation` and `apply_transform`. Supporting image inversion
also requires `inverse_transform`; the base provides matrix inversion.

.. autoclass:: GeometricAugmentationBase2D

   .. automethod:: compute_transformation
   .. automethod:: apply_transform
   .. automethod:: compute_inverse_transformation
   .. automethod:: inverse_transform

For `IntensityAugmentationBase2D`, the user only needs to override `apply_transform`.

.. autoclass:: IntensityAugmentationBase2D

   .. automethod:: apply_transform

The most common case is a pixel-wise augmentation with a random per-sample parameter. Subclass
`IntensityAugmentationBase2D`, declare a parameter generator in ``__init__``, and read the sampled
value in `apply_transform`:

.. code-block:: python

   from typing import Any, Dict, Optional

   import torch
   from torch import Tensor

   from kornia.augmentation import IntensityAugmentationBase2D
   from kornia.augmentation import random_generator as rg

   class RandomAddValue(IntensityAugmentationBase2D):
       """Add a per-sample value drawn uniformly from ``add_range``."""

       def __init__(self, add_range=(0.0, 0.2), same_on_batch=False, p=1.0, keepdim=False):
           super().__init__(p=p, same_on_batch=same_on_batch, keepdim=keepdim)
           # A PlainUniformGenerator sampler is a 4-tuple ``(range, name, center, bound)``:
           # sample a value inside ``range`` and expose it as ``params[name]``. ``center`` and
           # ``bound`` (``None`` here) are optional constraints for centered/bounded ranges.
           self._param_generator = rg.PlainUniformGenerator((add_range, "add", None, None))

       def apply_transform(
           self, input: Tensor, params: Dict[str, Tensor], flags: Dict[str, Any],
           transform: Optional[Tensor] = None,
       ) -> Tensor:
           # ``params["add"]`` has shape ``(B,)`` β€” reshape to broadcast over C, H, W.
           add = params["add"].to(input).view(-1, 1, 1, 1)
           return input + add

   aug = RandomAddValue((0.0, 0.2), p=1.0)
   out = aug(torch.rand(4, 3, 32, 32))                       # a different value per sample
   again = aug(torch.rand(4, 3, 32, 32), params=aug._params)  # reuse the recorded draw

Static (non-random) configuration goes in ``self.flags`` (a plain dict), read from the ``flags``
argument of `apply_transform`. A custom augmentation works standalone and inside
`AugmentationSequential` with no extra wiring.

For a rigid **geometric** augmentation, implement `compute_transformation` to return the
``(B, 3, 3)`` matrix and `apply_transform` to implement the corresponding image operation.
The geometric base propagates the matrix to boxes and keypoints; mask processing uses its mask handler.
The following example only illustrates the method signatures, not an invertible geometric warp:

.. code-block:: python

   from typing import Any, Dict, Optional

   import torch
   from torch import Tensor

   from kornia.augmentation import GeometricAugmentationBase2D
   from kornia.augmentation import random_generator as rg
   from kornia.core.ops import eye_like

   class MyRandomTransform(GeometricAugmentationBase2D):

      def __init__(
         self,
         factor=(0., 1.),
         same_on_batch: bool = False,
         p: float = 1.0,
         keepdim: bool = False,
      ) -> None:
         super().__init__(p=p, same_on_batch=same_on_batch, keepdim=keepdim)
         self._param_generator = rg.PlainUniformGenerator((factor, "factor", None, None))

      def compute_transformation(self, input, params, flags):
         # return the (B, 3, 3) transform matrix for this augmentation
         # identity shown only to illustrate the required matrix shape
         return eye_like(3, input)

      def apply_transform(
         self, input: Tensor, params: Dict[str, Tensor], flags: Dict[str, Any], transform: Optional[Tensor] = None
      ) -> Tensor:
         factor = params["factor"].to(input).view(-1, 1, 1, 1)
         return input * factor

For non-rigid augmentations, the user may implement the `apply_transform*` and `apply_non_transform*` APIs
as needed. Specifically, `apply_transform*` applies to the elements of a batch that are selected for augmentation,
while `apply_non_transform*` applies to the elements that are skipped. For example, a crop operation changes the size
of the selected elements, so the skipped elements must be resized as well to keep the whole batch tensor at one size.

.. autoclass:: AugmentationBase2D

   .. automethod:: apply_transform
   .. automethod:: apply_non_transform
   .. automethod:: apply_transform_mask
   .. automethod:: apply_non_transform_mask
   .. automethod:: apply_transform_box
   .. automethod:: apply_non_transform_box
   .. automethod:: apply_transform_keypoint
   .. automethod:: apply_non_transform_keypoint
   .. automethod:: apply_transform_class
   .. automethod:: apply_non_transform_class

`RigidAffineAugmentationBase2D` sits between `AugmentationBase2D` and the two rigid bases. It adds the
transform-matrix machinery β€” `compute_transformation` and the `transform_matrix` attribute β€” but no `inverse`.

.. autoclass:: RigidAffineAugmentationBase2D

   .. automethod:: compute_transformation

The 3D bases provide analogous shape, matrix and data-key machinery; see :class:`AugmentationBase3D` for their
contract, including the missing inverse.

.. autoclass:: AugmentationBase3D

.. autoclass:: RigidAffineAugmentationBase3D

.. autoclass:: GeometricAugmentationBase3D

.. autoclass:: IntensityAugmentationBase3D

Mix augmentations derive from `MixAugmentationBaseV2` and have their own forward and data-key contracts.
Some combine different samples; `RandomJigsaw` rearranges patches within each image. Label and box handling
are class-specific: `RandomMixUpV2` accepts class labels, while `RandomMosaic` accepts boxes but does not
implement class-label transforms. `RandomTransplantation` and `RandomTransplantation3D` require a segmentation
mask; a mask-only call needs ``data_keys=["mask"]``. The 3D transplantation class also inherits
`AugmentationBase3D`.

.. autoclass:: MixAugmentationBaseV2

Some Further Notes
------------------

Probabilities
^^^^^^^^^^^^^
:class:`AugmentationBase2D` states the ``p`` / ``p_batch`` model. Every augmentation honours ``p_batch``, and it can
be set as an attribute (``aug.p_batch = 0.5``). Only `RandomHorizontalFlip`, `RandomVerticalFlip`,
`RandomTransplantation` and `RandomTransplantation3D` also take it in the constructor. A concrete constructor can map
its public ``p`` to either gate: `RandomMixUpV2` gates the batch and `RandomJigsaw` individual samples. When
``0 < p_batch < 1``, the base draws the batch Bernoulli before the per-sample gate; endpoints skip that draw, and
``p=1.0, p_batch=0.0`` selects no sample.

With static ``p=1.0, p_batch=1.0`` every sample is transformed: a ``batch_prob`` put into ``params`` is not consulted
by the image, the transformation matrix, the labels or ``inverse``, in eager and compiled runs alike
(`#5585 <https://github.com/kornia/kornia/issues/5585>`_). Pass ``p < 1`` to skip samples by hand.

Random Generators
^^^^^^^^^^^^^^^^^
To get an automatically generated ``__repr__`` that lists all custom parameters, implement
``_param_generator`` by inheriting from ``RandomGeneratorBase`` to generate the random parameters, and
put all static parameters inside ``self.flags``. You can take advantage of ``PlainUniformGenerator`` to
generate simple uniform parameters with less boilerplate code.

Random Reproducibility
^^^^^^^^^^^^^^^^^^^^^^
See :class:`AugmentationBase2D` for where parameters are sampled and what ``params=`` replays, and
:doc:`/get-started/conventions` for seeding, worker seeds and consumption order.

Serialization
^^^^^^^^^^^^^
- Several constructors that accept ``nn.Parameter`` ranges propagate gradients to them.
- A ``kornia.augmentation.auto`` policy pickles when every operation wrapper in it does, and a wrapper does when
  its magnitude mapping is picklable; the built-in mappings are.
- :class:`AugmentationBase2D` covers the range buffers in ``state_dict()``, and
  :class:`RigidAffineAugmentationBase2D` what a lazily built matrix keeps.