Skip to content

🐍 Geometric Computer Vision Library for Spatial AI

docs(geometry): rewrite camera, calibration, subpix, boxes, Lie groups and the rest of kornia.geometry in the choose-then-use style (#6487)

Dmytro Mishkin18361a533c28
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 3ae1bedb1..d83fcdefe 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -216,9 +216,9 @@ A docstring is how a reader, human or agent, decides which function to call, so
What `tests/test_docstring_style.py` checks:
- every package of `kornia` is assigned to in scope or excluded (global);
-- a converted module exports no function or class without a docstring;
+- a converted module has no function or class, exported by an `__all__` or documented on a docs page, without a docstring;
- each public function and class of a converted module has a `See Also:` whose entries carry an explanation and name a kornia object, with no self-link or private target; an exception needs a non-empty reason, and an ``Alias of :func:`target`.`` or ``Deprecated: use :func:`target`.`` title line replaces the `See Also:`;
-- every Python-domain object role (`:func:`, `:class:`, `:meth:`, `:attr:`, `:data:`, `:const:`, `:exc:`, `:obj:`) in a converted object's docstring resolves as Sphinx resolves it. It must name either an object registered by the `.rst` pages under `docs/source`, or a torch, Python or NumPy object in the cached intersphinx inventories (`docs/source/_intersphinx`). Registered objects include the data attributes autodoc documents: enum members and dataclass fields under `:undoc-members:`, and variables with a `#:` comment or a docstring. An intersphinx object must be of a type the role links to: a function for `:func:`, a method for `:meth:`, an attribute or property for `:attr:`, and nothing for `:const:`. Only `:func:` and `:meth:` drop a trailing `()`. Roles in a catalog module docstring must import;
+- every Python-domain object role (`:func:`, `:class:`, `:meth:`, `:attr:`, `:data:`, `:const:`, `:exc:`, `:obj:`) in a converted object's docstring, and in those of the methods, properties and attributes autodoc documents under a converted class, resolves as Sphinx resolves it. It must name either an object registered by the `.rst` pages under `docs/source`, or a torch, Python or NumPy object in the cached intersphinx inventories (`docs/source/_intersphinx`). Registered objects include the data attributes autodoc documents: enum members and dataclass fields under `:undoc-members:`, and variables with a `#:` comment or a docstring. An intersphinx object must be of a type the role links to: a function for `:func:`, a method for `:meth:`, an attribute or property for `:attr:`, and nothing for `:const:`. Only `:func:` and `:meth:` drop a trailing `()`. Roles in a catalog module docstring must import;
- no `kornia/**/*.py` file or `docs/source` page writes a shape tuple as `:math:`, comments included (global); a tuple that reads as a shape but is a value is listed with what it is in `MATH_NON_SHAPES` (`testing/docstring_rollout.py`), and an entry that no longer occurs fails;
- a catalog namespace's module docstring names every export of its `__all__` inside a role or a double-backtick literal, where a qualified name must be a path to that export (`torch.nn.functional.normalize` does not catalog `kornia.enhance.normalize`); the same word in plain prose or in single backticks does not count.
diff --git a/docs/source/geometry.bbox.rst b/docs/source/geometry.bbox.rst
index b0a1340a0..ae38e3836 100644
--- a/docs/source/geometry.bbox.rst
+++ b/docs/source/geometry.bbox.rst
@@ -4,8 +4,5 @@ kornia.geometry.bbox
.. meta::
:description: The kornia.geometry.bbox module in Kornia provides useful functions for manipulating 2D and 3D bounding boxes. It includes tools for creating, transforming, and operating on bounding boxes, making it easier to handle spatial data in computer vision tasks.
-Functional API for 2D and 3D bounding boxes represented as corner tensors of shape ``(B, 4, 2)`` and ``(B, 8, 3)``.
-For an object-oriented interface with format conversions, see :doc:`geometry.boxes`.
-
.. automodule:: kornia.geometry.bbox
:members:
diff --git a/docs/source/geometry.boxes.rst b/docs/source/geometry.boxes.rst
index 429246982..9aee0d320 100644
--- a/docs/source/geometry.boxes.rst
+++ b/docs/source/geometry.boxes.rst
@@ -4,9 +4,5 @@ kornia.geometry.boxes
.. meta::
:description: The kornia.geometry.boxes module in Kornia provides essential functions for manipulating 2D and 3D bounding boxes. It offers utilities for the creation, transformation, and handling of bounding boxes, which are crucial for spatial data processing in computer vision applications.
-Object-oriented API for 2D and 3D bounding boxes: the :class:`~kornia.geometry.boxes.Boxes` and
-:class:`~kornia.geometry.boxes.Boxes3D` classes wrap a tensor of boxes and provide format conversions
-(``xyxy``, ``xywh``, vertices), transformations and padding.
-
.. automodule:: kornia.geometry.boxes
:members:
diff --git a/docs/source/geometry.calibration.rst b/docs/source/geometry.calibration.rst
index 7baa3d403..6ae818496 100644
--- a/docs/source/geometry.calibration.rst
+++ b/docs/source/geometry.calibration.rst
@@ -6,7 +6,12 @@ kornia.geometry.calibration
.. currentmodule:: kornia.geometry.calibration
-Module with useful functionalities for camera calibration.
+Lens distortion of pixel coordinates with OpenCV's Brown-Conrady model, whose coefficient vector has 4, 5, 8, 12 or
+14 elements, and camera pose from 3D-2D correspondences. :func:`distort_points` adds the distortion to ideal pixels,
+:func:`undistort_points` removes it from measured pixels and :func:`undistort_image` from a whole image,
+:func:`tilt_projection` builds the sensor-tilt step of the last two coefficients, and :func:`solve_pnp_dlt`
+estimates a world-to-camera pose. For the Kannala-Brandt fisheye model, on normalized points, see
+:doc:`geometry.camera`. The model these functions share is derived below.
The pinhole model is an ideal projection model that does not consider lens distortion when projecting a 3D point :math:`(X, Y, Z)` onto the image plane. To model the distortion of a 2D pixel point :math:`(u,v)` projected with the linear pinhole model, we first need to estimate the normalized 2D point coordinates :math:`(\bar{u}, \bar{v})`. For that, we can use the calibration matrix :math:`\mathbf{K}` with the following expression
diff --git a/docs/source/geometry.camera.perspective.rst b/docs/source/geometry.camera.perspective.rst
index 7e7d3af55..7e9cd42e7 100644
--- a/docs/source/geometry.camera.perspective.rst
+++ b/docs/source/geometry.camera.perspective.rst
@@ -6,5 +6,11 @@ Perspective Camera
.. currentmodule:: kornia.geometry.camera.perspective
+:func:`project_points` maps camera-frame 3D points to pixels with a ``(*, 3, 3)`` ``K`` and no extrinsics, and
+:func:`unproject_points` maps pixels at given depths back to camera-frame points. For world points, use
+:class:`~kornia.geometry.camera.pinhole.PinholeCamera`; for every pixel of a depth map,
+:func:`~kornia.geometry.depth.depth_to_3d_v2`; to compose the projection with a lens model, the normalized-plane
+functions of :doc:`geometry.camera`.
+
.. autofunction:: project_points
.. autofunction:: unproject_points
diff --git a/docs/source/geometry.camera.pinhole.rst b/docs/source/geometry.camera.pinhole.rst
index d483769e5..57e23f004 100644
--- a/docs/source/geometry.camera.pinhole.rst
+++ b/docs/source/geometry.camera.pinhole.rst
@@ -6,7 +6,10 @@ Pinhole Camera
.. currentmodule:: kornia.geometry.camera.pinhole
-This module contains the functions and data structures needed to describe the projection of a 3D scene onto a 2D image plane.
+:class:`PinholeCamera` holds a batch of cameras, with ``(B, 4, 4)`` intrinsics and world-to-camera extrinsics, and
+projects world points to pixels and back. :func:`pixel2cam` and :func:`cam2pixel` are the two halves of a dense
+reprojection with the same ``(B, 4, 4)`` matrices: a depth map to camera-frame points, and those to the pixels of a
+camera. For camera-frame points and a ``3x3`` ``K``, see :doc:`perspective <geometry.camera.perspective>`.
In computer vision, we can map between the 3D world and a 2D image using *projective geometry*. The module implements the simplest camera model, the **pinhole camera**, which is the most basic model for general projective cameras from the finite cameras group.
@@ -51,7 +54,7 @@ where:
The camera rotation and translation are expressed in terms of a Euclidean coordinate frame known as the *world coordinate system*. These terms are usually expressed by the joint rotation-translation matrix :math:`[R|t]`, also known as the extrinsic matrix. It describes the camera pose around a static scene and transforms the coordinates of a 3D point :math:`(X,Y,Z)` from the *world coordinate system* to the *camera coordinate system*.
The :class:`PinholeCamera` expects the *intrinsic matrices* and the *extrinsic matrices*
-to be of shape `(B, 4, 4)` such that each *intrinsic matrix* has the following format:
+to be of shape ``(B, 4, 4)`` such that each *intrinsic matrix* has the following format:
.. math::
\begin{bmatrix}
diff --git a/docs/source/geometry.camera.rst b/docs/source/geometry.camera.rst
index dd3222d7e..ac92a7b59 100644
--- a/docs/source/geometry.camera.rst
+++ b/docs/source/geometry.camera.rst
@@ -6,9 +6,23 @@ kornia.geometry.camera
.. currentmodule:: kornia.geometry.camera
+Camera models that map 3D points to pixels and back. Points are in the camera frame, +X right, +Y down and +Z forward,
+and pixels are ``(u, v)`` = (column, row) with integer pixel centres; :doc:`get-started/camera-conventions` compares
+these conventions with other libraries. This page holds the building blocks of a camera model, which take image-plane
+coordinates before the intrinsics (normalized ``z = 1`` coordinates for a perspective camera) and flat parameter
+vectors: a projection to the image plane, then the intrinsics, with or without a fisheye distortion. For pixels from a
+``3x3`` ``K`` in one call, see :doc:`perspective <geometry.camera.perspective>`; for cameras with extrinsics,
+:doc:`pinhole <geometry.camera.pinhole>`; for a rectified stereo rig, :doc:`stereo camera <geometry.camera.stereo>`; for
+OpenCV's Brown-Conrady distortion of pixel coordinates, :doc:`geometry.calibration`.
+
Projections
-----------
+Camera-frame points to the image plane and back, without intrinsics: :func:`project_points_z1` divides by ``z``, onto
+the normalized ``z = 1`` plane (perspective), and :func:`project_points_orthographic` drops it.
+:func:`dx_project_points_z1` returns the ``(..., 2, 3)`` Jacobian of the perspective projection, and
+:func:`dx_project_points_orthographic` the derivative ``du/dx`` alone.
+
.. autofunction:: project_points_z1
.. autofunction:: unproject_points_z1
.. autofunction:: dx_project_points_z1
@@ -20,6 +34,10 @@ Projections
Distortion
----------
+Normalized points to pixels and back with a flat parameter vector: the affine model ``[fx, fy, cx, cy]``, which
+applies the intrinsics and no lens distortion, and the Kannala-Brandt fisheye model
+``[fx, fy, cx, cy, k0, k1, k2, k3]``, each with its inverse and its Jacobian with respect to the point.
+
.. autofunction:: distort_points_affine
.. autofunction:: undistort_points_affine
.. autofunction:: dx_distort_points_affine
diff --git a/docs/source/geometry.camera.stereo.rst b/docs/source/geometry.camera.stereo.rst
index 5beaec57f..d06052410 100644
--- a/docs/source/geometry.camera.stereo.rst
+++ b/docs/source/geometry.camera.stereo.rst
@@ -6,7 +6,13 @@ Stereo Camera
.. currentmodule:: kornia.geometry.camera.stereo
-This module provides the :class:`StereoCamera` class, which contains functionality for working with a horizontal stereo camera setup.
+:class:`StereoCamera` takes the two rectified projection matrices of a horizontal stereo rig and reprojects a
+channels-last ``(B, H, W, 1)`` disparity map to ``(B, H, W, 3)`` points in the left rectified camera frame;
+:func:`reproject_disparity_to_3D` does the same from a ``Q`` matrix, as ``cv2.reprojectImageTo3D`` does. Both raise
+:class:`StereoException` for the invalid rigs, disparity maps and ``Q`` matrices its docstring lists. A rig or ``Q``
+batch that is neither 1 nor the disparity batch raises ``ValueError``, and left and right matrices with different batch
+sizes, other than a right batch of 1, raise ``RuntimeError``. For a depth map rather than points, use
+:func:`~kornia.geometry.depth.depth_from_disparity`.
The horizontal stereo camera setup is assumed to be calibrated and rectified such that the setup can be described by two camera matrices:
@@ -180,3 +186,6 @@ scope, and coordinate and gradient behavior.
.. automethod:: __init__
.. autofunction:: reproject_disparity_to_3D
+
+.. autoexception:: StereoException
+ :show-inheritance:
diff --git a/docs/source/geometry.grid.rst b/docs/source/geometry.grid.rst
index 34021c2d3..0759d6a0d 100644
--- a/docs/source/geometry.grid.rst
+++ b/docs/source/geometry.grid.rst
@@ -6,5 +6,13 @@ kornia.geometry.grid
.. currentmodule:: kornia.geometry.grid
+Grids of pixel and voxel coordinates. :func:`create_meshgrid` returns the ``(1, H, W, 2)`` grid of ``(x, y)``
+coordinates of an image and :func:`create_meshgrid3d` the ``(1, D, H, W, 3)`` grid of ``(d, x, y)`` coordinates of a
+volume, normalized to :math:`[-1, 1]` by default or in pixels. The normalized 2D grid is the identity grid of
+:func:`torch.nn.functional.grid_sample` called with the same ``align_corners``, ``True`` by default. The 3D grid is
+corner-aligned only and depth first, while ``grid_sample`` reads ``(x, y, z)``: pass ``grid[..., [1, 2, 0]]`` with
+``align_corners=True``. See :ref:`Coordinates and sizes <coordinate-conventions>` for the pixel-centre and
+``align_corners`` conventions.
+
.. autofunction:: create_meshgrid
.. autofunction:: create_meshgrid3d
diff --git a/docs/source/geometry.keypoints.rst b/docs/source/geometry.keypoints.rst
index 2747a7902..6075ec707 100644
--- a/docs/source/geometry.keypoints.rst
+++ b/docs/source/geometry.keypoints.rst
@@ -4,9 +4,12 @@ kornia.geometry.keypoints
.. meta::
:description: The kornia.geometry.keypoints module provides the Keypoints container for 2D (x, y) keypoints, which kornia.augmentation transforms together with the images, and the Keypoints3D container for 3D (x, y, z) points.
-Object-oriented API for keypoints: :class:`~kornia.geometry.keypoints.Keypoints` wraps a tensor of 2D ``(x, y)``
-points and transforms and pads it; :class:`~kornia.geometry.keypoints.Keypoints3D` validates and stores 3D
-``(x, y, z)`` points and transforms them.
+Keypoint containers. :class:`~kornia.geometry.keypoints.Keypoints` wraps an ``(N, 2)`` or ``(B, N, 2)`` tensor of
+``(x, y)`` pixel coordinates, transforms it by ``3 x 3`` matrices and pads it with its image, as
+:class:`~kornia.augmentation.container.AugmentationSequential` does with its ``'keypoints'`` data.
+:class:`~kornia.geometry.keypoints.Keypoints3D` wraps an ``(N, 3)`` or ``(B, N, 3)`` tensor of ``(x, y, z)`` points
+and transforms it by ``4 x 4`` matrices. Pixel ``(0, 0)`` is centred at ``(0, 0)``; see
+:ref:`Coordinates and sizes <coordinate-conventions>`.
.. autoclass:: kornia.geometry.keypoints.Keypoints
:members:
diff --git a/docs/source/geometry.liegroup.rst b/docs/source/geometry.liegroup.rst
index 07192a916..2ef03b258 100644
--- a/docs/source/geometry.liegroup.rst
+++ b/docs/source/geometry.liegroup.rst
@@ -41,7 +41,9 @@ Lie group to its Lie algebra through the following facts:
dimension :math:`d` is the number of degrees of freedom of :math:`M`.
#. The exponential map, :math:`\exp : m \rightarrow M`, exactly converts elements of the Lie algebra into elements of the group.
- The :math:`\log` map is the inverse operation.
+ The :math:`\log` map is the inverse operation. In kornia ``log`` is principal: ``exp(log(g))`` gives back ``g`` for a
+ unit rotation, while ``log(exp(v))`` gives back ``v`` only while the rotation angle of ``v`` is below :math:`\pi`;
+ above it, ``log`` returns the equivalent vector whose rotation angle is at most :math:`\pi`.
.. image:: data/lie_ops.png
:alt: The exponential and logarithmic maps between a Lie group and its Lie algebra
@@ -51,6 +53,14 @@ Reference: `A micro Lie theory for state estimation in robotics <https://arxiv.o
Classes
-------
+Pick the class by dimension and by whether the motion has a translation: :class:`So2` and :class:`Se2` in 2D,
+:class:`So3` and :class:`Se3` in 3D. ``a * b`` composes as ``a.matrix() @ b.matrix()``, so ``b`` acts first, and
+``g * p`` maps points as ``R p + t``. Tangent vectors put the rotation last, in radians: ``(v_x, v_y, theta)`` for
+``Se2`` and ``[upsilon, omega]`` for ``Se3``. :ref:`Rotations and rigid motions <rotation-conventions>` compares these
+conventions with scipy, Sophus and Eigen. For a pose that names its frames, see
+:class:`~kornia.geometry.pose.NamedPose`; for quaternion arithmetic and averaging,
+:class:`~kornia.geometry.quaternion.Quaternion`.
+
.. autoclass:: So3
:members:
:special-members:
diff --git a/docs/source/geometry.linalg.rst b/docs/source/geometry.linalg.rst
index 3a936359a..839b84277 100644
--- a/docs/source/geometry.linalg.rst
+++ b/docs/source/geometry.linalg.rst
@@ -6,6 +6,12 @@ kornia.geometry.linalg
.. currentmodule:: kornia.geometry.linalg
+Homogeneous transforms and the operations around them. :func:`compose_transformations` chains two ``4 x 4``
+transforms, :func:`inverse_transformation` inverts a rigid one and :func:`relative_transformation` gives the transform
+between two poses expressed in one frame; ``trans_ab`` maps points in frame ``b`` to frame ``a``.
+:func:`transform_points` applies a ``(D + 1, D + 1)`` matrix to ``(..., N, D)`` points. :func:`euclidean_distance` and
+:func:`point_line_distance` measure distances, and :func:`batched_dot_product` and :func:`batched_squared_norm` (also
+exported as :func:`squared_norm`) reduce the last dimension.
.. autofunction:: relative_transformation
.. autofunction:: compose_transformations
diff --git a/docs/source/geometry.line.rst b/docs/source/geometry.line.rst
index af901c77a..3d49053bb 100644
--- a/docs/source/geometry.line.rst
+++ b/docs/source/geometry.line.rst
@@ -6,6 +6,15 @@ kornia.geometry.line
.. currentmodule:: kornia.geometry.line
+Lines in any dimension and planes in 3D, as objects. :class:`ParametrizedLine` is the line :math:`o + t d` from an
+origin and a direction, or through two points, with the points along it, the distances and projections of points and
+the intersection with a plane; :func:`fit_line` fits one to each of ``B`` point sets by total least squares, optionally
+weighted. :class:`~kornia.geometry.plane.Hyperplane` is the plane :math:`n \cdot x + d = 0` from a normal and an
+offset, a normal and a point, or three points, with signed distances and projections;
+:func:`~kornia.geometry.plane.fit_plane` fits one to 3D points. The distances and projections are Euclidean for a unit
+direction or normal, which the two-point and three-point constructors and the fits return. For 2D lines given as
+coefficients :math:`ax + by + c = 0`, use :func:`~kornia.geometry.linalg.point_line_distance`.
+
.. autoclass:: ParametrizedLine
:members:
:special-members: __init__
diff --git a/docs/source/geometry.pointcloud.rst b/docs/source/geometry.pointcloud.rst
index 35b9092a8..a52e05c3a 100644
--- a/docs/source/geometry.pointcloud.rst
+++ b/docs/source/geometry.pointcloud.rst
@@ -6,6 +6,11 @@ kornia.geometry.pointcloud
.. currentmodule:: kornia.geometry.pointcloud
+Point clouds to and from PLY files. :func:`save_pointcloud_ply` and :func:`save_pointcloud_ply_binary` write the
+``x``, ``y`` and ``z`` of a channels-last ``(*, 3)`` tensor as ``double`` vertices, as text or as little-endian
+binary; :func:`load_pointcloud_ply` and :func:`load_pointcloud_ply_binary` read the vertex coordinates of an ASCII or a
+binary file, skipping normals, colours and faces, into a float32 ``(N, 3)`` tensor.
+
.. autofunction:: save_pointcloud_ply
.. autofunction:: load_pointcloud_ply
.. autofunction:: save_pointcloud_ply_binary
diff --git a/docs/source/geometry.quaternion.rst b/docs/source/geometry.quaternion.rst
index 309e7c956..4166cb222 100644
--- a/docs/source/geometry.quaternion.rst
+++ b/docs/source/geometry.quaternion.rst
@@ -6,6 +6,12 @@ kornia.geometry.quaternion
.. currentmodule:: kornia.geometry.quaternion
+:class:`Quaternion` holds ``(*, 4)`` quaternions in ``(w, x, y, z)`` order, multiplies them by the Hamilton product and
+converts them to and from rotation matrices, rotation vectors and Euler angles; :func:`average_quaternions` returns the
+mean rotation of many. A quaternion does not act on points: :class:`~kornia.geometry.liegroup.So3` wraps one to rotate
+points and adds ``exp``, ``log`` and Jacobians, and :class:`~kornia.geometry.liegroup.Se3` adds a translation. For the
+same conversions on plain tensors, see :doc:`geometry.conversions`.
+
Module state
------------
diff --git a/docs/source/geometry.rst b/docs/source/geometry.rst
index b051c6f6a..2f8660cf9 100644
--- a/docs/source/geometry.rst
+++ b/docs/source/geometry.rst
@@ -15,8 +15,9 @@ This module, the original core of the library, groups its submodules by task:
- Apply a homogeneous transformation to points, compose, invert or relate ``4 x 4`` transformations, and measure
distances of points to points and lines: :doc:`linalg <geometry.linalg>`. Grids of pixel coordinates:
:doc:`grid <geometry.grid>`.
-- Camera models, and the projection and unprojection of points: :doc:`camera <geometry.camera>`; lens distortion and
- pose from 2D-3D correspondences: :doc:`calibration <geometry.calibration>`.
+- Camera models, the projection and unprojection of points, and the Kannala-Brandt fisheye and affine distortion
+ models on normalized points: :doc:`camera <geometry.camera>`; OpenCV (Brown-Conrady) lens distortion and
+ undistortion of pixels and images, and pose from 2D-3D correspondences: :doc:`calibration <geometry.calibration>`.
- 3D points and surface normals from depth maps, and images warped by depth: :doc:`depth <geometry.depth>`.
- Two-view geometry, fundamental and essential matrices, relative pose and triangulation:
:doc:`epipolar <geometry.epipolar>`; homographies from points or line segments:
diff --git a/docs/source/geometry.subpix.rst b/docs/source/geometry.subpix.rst
index 8652f05e1..045457d58 100644
--- a/docs/source/geometry.subpix.rst
+++ b/docs/source/geometry.subpix.rst
@@ -6,7 +6,14 @@ kornia.geometry.subpix
.. currentmodule:: kornia.geometry.subpix
-Functions and modules to extract coordinates with sub-pixel accuracy, for example to refine keypoint locations.
+Sub-pixel locations from heatmaps and scale-space volumes. :func:`spatial_soft_argmax2d` returns one ``(x, y)`` per
+heatmap channel, the softmax-weighted mean over the whole map, and :func:`conv_soft_argmax2d` and
+:func:`conv_soft_argmax3d` one per sliding window. :func:`nms2d` and :func:`nms3d` keep the strict local maxima, as
+the suppressed map or a boolean mask, and :func:`nms3d_minmax` returns two boolean masks of a volume, its strict local
+maxima and minima. :func:`conv_quad_interp3d` and :func:`iterative_quad_interp3d` refine the maxima of a scale-space
+volume by a quadratic fit, with :class:`AdaptiveQuadInterp3d` picking the faster one for the device. Coordinates are
+``(x, y)`` in 2D and ``(d, x, y)`` in 3D; see :ref:`Coordinates and sizes <coordinate-conventions>`. Each class under
+Modules except :class:`AdaptiveQuadInterp3d` stores the arguments of its function form.
Convolutional
-------------
diff --git a/docs/source/geometry.vector.rst b/docs/source/geometry.vector.rst
index 4aa04b7f1..a41c142d2 100644
--- a/docs/source/geometry.vector.rst
+++ b/docs/source/geometry.vector.rst
@@ -6,6 +6,12 @@ kornia.geometry.vector
.. currentmodule:: kornia.geometry.vector
+Tensor wrappers for 3D vectors ``(*, 3)``, 2D vectors ``(*, 2)`` and scalars, which
+:class:`~kornia.geometry.plane.Hyperplane`, the Lie groups of ``kornia.geometry.liegroup`` and the camera models of
+``kornia.sensors.camera`` take and return. :class:`Vector3` and :class:`Vector2` add coordinate access, dot products
+and normalisation, and :class:`Scalar` holds one value per vector. A wrapper does not copy its tensor and is not a
+``torch.Tensor``: torch functions and operators accept it, and kornia's tensor functions take ``v.data``.
+
.. autoclass:: Vector3
:members:
diff --git a/kornia/geometry/__init__.py b/kornia/geometry/__init__.py
index cd2c60b3e..e6b50380a 100644
--- a/kornia/geometry/__init__.py
+++ b/kornia/geometry/__init__.py
@@ -33,8 +33,9 @@ Which subpackage to use:
conventions: ``conversions``.
- Apply a homogeneous ``(D + 1, D + 1)`` transformation to points; compose, invert or relate ``4 x 4``
transformations; distances of points to points and lines: ``linalg``. Grids of pixel coordinates: ``grid``.
-- Camera models, and the projection and unprojection of points: ``camera``
- (:class:`~kornia.geometry.camera.pinhole.PinholeCamera`); lens distortion and pose from 2D-3D correspondences:
+- Camera models, the projection and unprojection of points, and the Kannala-Brandt fisheye and affine distortion
+ models on normalized points: ``camera`` (:class:`~kornia.geometry.camera.pinhole.PinholeCamera`); OpenCV
+ (Brown-Conrady) lens distortion and undistortion of pixels and images, and pose from 2D-3D correspondences:
``calibration``.
- 3D points and surface normals from depth maps, and images warped by depth: ``depth``.
- Two-view geometry, fundamental and essential matrices, relative pose and triangulation: ``epipolar``;
diff --git a/kornia/geometry/bbox.py b/kornia/geometry/bbox.py
index c10ee3cb2..9b0d038b4 100644
--- a/kornia/geometry/bbox.py
+++ b/kornia/geometry/bbox.py
@@ -15,6 +15,37 @@
# limitations under the License.
#
+"""Box functions on plain corner tensors: build, measure, validate, mask, transform and NMS.
+
+A 2D box is four ``(x, y)`` corners in clockwise order, top-left, top-right, bottom-right, bottom-left, so one box per
+image is ``(B, 4, 2)``. A 3D box is those four corners at the front followed by the same four at the back, ``(B, 8, 3)``
+with ``(x, y, z)`` corners. Corners are inclusive pixel centres: a box covering pixels ``0..9`` has corners at ``0`` and
+``9`` and width ``x_max - x_min + 1 = 10``, unlike the exclusive boxes of torchvision, COCO and albumentations
+(`#3934 <https://github.com/kornia/kornia/issues/3934>`_). The generators and the shape and mask functions share this
+convention; :func:`nms` takes exclusive ``(N, 4)`` ``xyxy`` boxes instead
+(`#4008 <https://github.com/kornia/kornia/issues/4008>`_), and :func:`transform_bbox` takes ``xyxy`` or ``xywh`` boxes,
+or corners, and adds no ``+1``. From an unbatched :class:`~kornia.geometry.boxes.Boxes`, pass its
+``to_tensor('vertices_plus')`` export; the ``'vertices'`` export reads one pixel larger
+(`#4009 <https://github.com/kornia/kornia/issues/4009>`_). See
+:doc:`Conventions & Pitfalls </get-started/conventions>` for the box formats of the rest of kornia.
+
+Which function to use:
+
+- Corners from a start corner and a size: :func:`bbox_generator` in 2D, :func:`bbox_generator3d` in 3D.
+- The size of axis-aligned corners: :func:`infer_bbox_shape` returns heights and widths, :func:`infer_bbox_shape3d`
+ depths, heights and widths.
+- Whether corners are laid out as a box: :func:`validate_bbox` returns ``False`` when they are not,
+ :func:`validate_bbox3d` raises ``AssertionError``.
+- A filled mask per box: :func:`bbox_to_mask` returns ``(B, H, W)``, :func:`bbox_to_mask3d` ``(B, 1, D, H, W)``.
+- Boxes moved by ``3 x 3`` matrices: :func:`transform_bbox` for ``xyxy`` or ``xywh`` tensors and corners;
+ :meth:`~kornia.geometry.boxes.Boxes.transform_boxes` moves all four corners of a box container.
+- The highest-scored of overlapping boxes: :func:`nms`, by IoU. The local maxima of a response map are
+ :func:`~kornia.geometry.subpix.nms2d` instead.
+- Boxes as objects with format conversions, several boxes per image, padding and filtering:
+ :class:`~kornia.geometry.boxes.Boxes` and :class:`~kornia.geometry.boxes.Boxes3D`. The image patch inside each box:
+ :func:`~kornia.geometry.transform.crop_and_resize`.
+"""
+
from __future__ import annotations
import math
@@ -55,13 +86,18 @@ _VALIDATE_BBOX3D_FALSE_MESSAGE = (
def validate_bbox(boxes: torch.Tensor) -> bool:
- """Validate whether a 2D box has matching top and bottom edge vectors.
+ """Check that 2D box corners have equal top and bottom edge vectors, returning a bool.
+
+ ``ok = validate_bbox(boxes)`` takes corners ``(B, 4, 2)`` or ``(B, N, 4, 2)`` and returns ``True`` when every
+ coordinate is finite and, in every box, the top edge (top-left to top-right) equals the bottom edge (bottom-left
+ to bottom-right) within ``1e-4`` on each axis. Otherwise it returns ``False``, without raising, and an eager call
+ warns with a ``FutureWarning``. It is a check of the corner layout only; for 3D corners, use
+ :func:`validate_bbox3d`, which raises instead.
Convention:
Vertices are inclusive, in clockwise top-left, top-right, bottom-right, bottom-left order (see
- :func:`infer_bbox_shape`). Accepts ``(B, 4, 2)`` and ``(B, N, 4, 2)``. Returns ``False``, without
- raising, for an invalid shape or when the top and bottom edge vectors differ by more than ``1e-4``. Right
- angles, positive area and direction are not checked, so any parallelogram in cyclic vertex order passes.
+ :func:`infer_bbox_shape`). Returns ``False`` for any other shape too. Right angles, positive area and
+ direction are not checked, so any parallelogram in cyclic vertex order passes. An empty batch passes.
.. warning::
Returning ``False`` is deprecated; a future release raises ``AssertionError``
@@ -73,6 +109,16 @@ def validate_bbox(boxes: torch.Tensor) -> bool:
following ``clockwise`` order: top-left, top-right, bottom-right, bottom-left. The coordinates must be in
the x, y order.
+ Returns:
+ ``True`` for valid boxes, ``False`` otherwise.
+
+ See Also:
+ - :func:`validate_bbox3d`: the check for ``(B, 8, 3)`` corners, which raises ``AssertionError`` instead of
+ returning ``False``.
+ - :func:`infer_bbox_shape`: the inclusive height and width of corners that pass.
+ - :meth:`~kornia.geometry.boxes.Boxes.from_tensor`: imports boxes into a container, rejecting non-finite
+ coordinates, and non-positive extents in the ``'xy*'`` modes, but not checking the corner layout.
+
"""
if not (len(boxes.shape) in [3, 4] and boxes.shape[-2:] == torch.Size([4, 2])):
if not torch.jit.is_scripting() and not torch.jit.is_tracing() and not is_compiling():
@@ -108,14 +154,21 @@ def validate_bbox(boxes: torch.Tensor) -> bool:
def validate_bbox3d(boxes: torch.Tensor) -> bool:
- r"""Validate that a 3D box has equal inclusive edge extents along each axis, raising when it does not.
+ r"""Check that 3D box corners measure one extent per axis, raising if not.
+
+ ``ok = validate_bbox3d(boxes)`` takes corners ``(B, 8, 3)`` or ``(B, N, 8, 3)`` and returns ``True`` when, in every
+ box, the ``x`` extents of the four left-to-right edges agree, the ``y`` extents of the two diagonals of the front
+ face and of the back face agree, and the ``z`` extents of the four front-to-back edges agree, to the default
+ tolerances of :func:`torch.isclose`; for a box with right angles these are its edge extents. Another shape or a
+ disagreement raises ``AssertionError``; a non-finite coordinate returns ``False``, with a ``FutureWarning`` on an
+ eager call. For 2D corners, use :func:`validate_bbox`, which returns ``False`` instead of raising.
Convention:
Vertices are inclusive, front-top-left, front-top-right, front-bottom-right, front-bottom-left, then the
- same four back vertices. Accepts ``(B, 8, 3)`` and ``(B, N, 8, 3)``; another shape, or unequal
- extents of the four edges parallel to an axis, raises ``AssertionError``. Right angles and positive
- extent are not checked. A non-finite coordinate returns ``False``. Under graph capture only the shape is
- checked.
+ same four back vertices. Right angles and positive extent are not checked as such: a box whose right face is
+ shifted in ``y`` raises, because its face diagonals then differ in ``y``, while one whose bottom face is
+ shifted in ``x`` or ``z``, whose back face is shifted in ``x`` or ``y``, or whose right face is shifted in
+ ``z`` passes. Under graph capture only the shape is checked.
.. warning::
Returning ``False`` is deprecated; a future release raises ``AssertionError``
@@ -131,6 +184,12 @@ def validate_bbox3d(boxes: torch.Tensor) -> bool:
``True``, or ``False`` when any coordinate is non-finite. Invalid input otherwise raises instead of
returning ``False``.
+ See Also:
+ - :func:`validate_bbox`: the check for ``(B, 4, 2)`` corners, which returns ``False`` instead of raising.
+ - :func:`infer_bbox_shape3d`: runs this check, then returns the depths, heights and widths of the corners.
+ - :meth:`~kornia.geometry.boxes.Boxes3D.from_tensor`: imports ``(N, 6)`` boxes into a container, rejecting
+ non-finite coordinates and non-positive extents.
+
"""
if _validate_bbox3d(boxes):
return True
@@ -184,7 +243,14 @@ def _validate_bbox3d(boxes: torch.Tensor) -> bool:
def infer_bbox_shape(boxes: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
- r"""Auto-infer the output sizes for the given 2D bounding boxes.
+ r"""Read the inclusive heights and widths of (N, 4, 2) box corners from fixed vertices.
+
+ ``heights, widths = infer_bbox_shape(boxes)`` takes ``(N, 4, 2)`` corners and returns two ``(N,)`` tensors in the
+ dtype of ``boxes``, the inclusive heights ``y_br - y_tl + 1`` and widths ``x_tr - x_tl + 1`` read from the
+ top-left, top-right and bottom-right vertices. It does not validate the corners. For ``(B, N, 4, 2)`` corners,
+ flatten them to ``(B * N, 4, 2)`` first; for corners in any order, such as a rotated box, use
+ :meth:`~kornia.geometry.boxes.Boxes.get_boxes_shape`, which measures the axis-aligned bounds; for 3D corners,
+ :func:`infer_bbox_shape3d`.
Convention:
Vertices are **inclusive**: a box covering pixels ``0..9`` has corners at ``0`` and ``9`` and width
@@ -212,6 +278,12 @@ def infer_bbox_shape(boxes: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
- Bounding box heights, shape of ``(N,)``.
- Bounding box widths, shape of ``(N,)``.
+ See Also:
+ - :func:`infer_bbox_shape3d`: the depths, heights and widths of ``(B, 8, 3)`` corners, which it validates first.
+ - :meth:`~kornia.geometry.boxes.Boxes.get_boxes_shape`: the inclusive size of the axis-aligned bounds of the
+ boxes in a container, for corners in any order.
+ - :func:`bbox_generator`: the inverse, corners from a start corner and this height and width.
+
Example:
>>> boxes = torch.tensor([[
... [1., 1.],
@@ -236,7 +308,15 @@ def infer_bbox_shape(boxes: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
def infer_bbox_shape3d(boxes: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
- r"""Auto-infer the output sizes for the given 3D bounding boxes.
+ r"""Read the inclusive depths, heights and widths of (B, 8, 3) box corners, validating them.
+
+ ``depths, heights, widths = infer_bbox_shape3d(boxes)`` takes ``(B, 8, 3)`` corners, four at the front and then
+ four at the back, and returns three ``(B,)`` tensors in the dtype of ``boxes``, the inclusive extent of each box
+ along ``z``, ``y`` and ``x``. It first checks the corners as :func:`validate_bbox3d` does and raises
+ ``AssertionError`` where that check finds extents that disagree, such as for a box whose right face is shifted in
+ ``y``, or a non-finite coordinate; another shape than ``(B, 8, 3)`` raises
+ :class:`~kornia.core.exceptions.ShapeError`. For ``(N, 4, 2)`` corners, use :func:`infer_bbox_shape`, which does
+ not validate.
Convention:
Vertices are inclusive (see :func:`infer_bbox_shape`), in :func:`validate_bbox3d` order. The result is
@@ -263,6 +343,12 @@ def infer_bbox_shape3d(boxes: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor,
- Bounding box heights, shape of ``(B,)``.
- Bounding box widths, shape of ``(B,)``.
+ See Also:
+ - :func:`infer_bbox_shape`: the heights and widths of ``(N, 4, 2)`` corners, without validation.
+ - :meth:`~kornia.geometry.boxes.Boxes3D.get_boxes_shape`: the same three sizes for the boxes of a container,
+ from the axis-aligned bounds of the corners.
+ - :func:`bbox_generator3d`: the inverse, corners from a start corner and these sizes.
+
Example:
>>> boxes = torch.tensor([[[ 0, 1, 2],
... [10, 1, 2],
@@ -312,16 +398,20 @@ def bbox_to_mask(
height: Optional[int] = None,
size: Optional[tuple[int, int]] = None,
) -> torch.Tensor:
- """Convert 2D bounding boxes to masks. Covered area is 1. and the remaining is 0.
+ """Draw (B, 4, 2) inclusive box corners as filled (B, H, W) masks.
+
+ ``mask = bbox_to_mask(boxes, size=(H, W))`` takes one box per image, ``(B, 4, 2)`` corners, and returns
+ ``(B, H, W)`` masks in the dtype of ``boxes``: ``1`` on each pixel ``(x, y)`` with ``xmin <= x <= xmax`` and
+ ``ymin <= y <= ymax``, read from the top-left and bottom-right corners, and ``0`` elsewhere, with no gradient path.
+ The bounds are compared inclusively and unrounded, and a box partly outside the image fills only its part inside.
+ For several boxes per image, use :meth:`~kornia.geometry.boxes.Boxes.to_mask`, which returns ``(N, H, W)`` for
+ unbatched boxes and ``(B, N, H, W)`` for batched ones; for volumes, :func:`bbox_to_mask3d`.
Convention:
The size is ``size=(height, width)``, as for :func:`bbox_to_mask3d` and
:meth:`~kornia.geometry.boxes.Boxes.to_mask`; the deprecated ``width, height`` arguments come in the
- opposite order. The mask is :math:`(B, height, width)`. Only vertices 0 (top-left) and 2
- (bottom-right) are read. A pixel is covered when ``xmin <= x <= xmax`` and ``ymin <= y <= ymax`` on the
- unrounded values, i.e. inclusively: pass the ``'vertices_plus'`` export of
- :class:`~kornia.geometry.boxes.Boxes`. The mask has the input dtype and no gradient path.
- ``(B, N, 4, 2)`` raises :class:`~kornia.core.exceptions.ShapeError`.
+ opposite order. Only vertices 0 (top-left) and 2 (bottom-right) are read. Pass the ``'vertices_plus'`` export
+ of :class:`~kornia.geometry.boxes.Boxes`. ``(B, N, 4, 2)`` raises :class:`~kornia.core.exceptions.ShapeError`.
.. warning::
The ``(width, height)`` form is deprecated in favour of ``size=(height, width)``:
@@ -339,15 +429,19 @@ def bbox_to_mask(
size: (height, width) of the masked image.
Returns:
- the output mask tensor, shape of :math:`(B, height, width)` and dtype of ``boxes``.
+ the output mask tensor, shape of ``(B, height, width)`` and dtype of ``boxes``.
Raises:
ShapeError: if ``boxes`` does not have shape ``(B, 4, 2)``.
ValueError: if ``size`` is passed together with ``width`` or ``height``.
TypeError: if neither ``size`` nor both ``width`` and ``height`` are given.
- Note:
- It is currently non-differentiable.
+ See Also:
+ - :func:`bbox_to_mask3d`: the 3D form, ``(B, 1, D, H, W)`` masks of ``(B, 8, 3)`` corners, whose bounds are
+ truncated toward zero.
+ - :meth:`~kornia.geometry.boxes.Boxes.to_mask`: masks of several boxes per image, ``(N, H, W)`` for unbatched
+ boxes and ``(B, N, H, W)`` for batched ones, filled over the rounded exclusive bounds.
+ - :func:`infer_bbox_shape`: the inclusive height and width of the same corners.
Note:
The pixel-position grid is built in ``float32`` regardless of ``boxes.dtype``, so
@@ -421,14 +515,26 @@ def bbox_to_mask(
def bbox_to_mask3d(boxes: torch.Tensor, size: tuple[int, int, int]) -> torch.Tensor:
- r"""Convert 3D bounding boxes to masks. Covered area is 1. and the remaining is 0.
+ r"""Draw (B, 8, 3) inclusive 3D box corners as filled (B, 1, D, H, W) masks.
+
+ ``mask = bbox_to_mask3d(boxes, (D, H, W))`` takes one box per volume, ``(B, 8, 3)`` corners, and returns
+ ``(B, 1, D, H, W)`` masks in the dtype of ``boxes``: ``1`` on the voxels inside each box and ``0`` elsewhere, with
+ no gradient path. It first checks the corners as :func:`validate_bbox3d` does and raises ``AssertionError`` where
+ that check finds extents that disagree, or for a non-finite coordinate. For several boxes per volume, use
+ :meth:`~kornia.geometry.boxes.Boxes3D.to_mask`; for images, :func:`bbox_to_mask`.
Convention:
- ``size`` is ``(depth, height, width)`` and the mask :math:`(B, 1, depth, height, width)` in the input
+ ``size`` is ``(depth, height, width)`` and the mask ``(B, 1, depth, height, width)`` in the input
dtype, with a channel axis that :func:`bbox_to_mask` does not have
- (:meth:`kornia.geometry.boxes.Boxes3D.to_mask` returns :math:`(N, depth, height, width)`). The bounds are
- truncated toward zero and compared inclusively: pass the ``'vertices_plus'`` export. There is no
- gradient path. ``(B, N, 8, 3)`` input raises :class:`~kornia.core.exceptions.ShapeError`.
+ (:meth:`kornia.geometry.boxes.Boxes3D.to_mask` returns ``(N, depth, height, width)`` for unbatched boxes and
+ ``(B, N, depth, height, width)`` for batched ones). The bounds are truncated toward zero and compared
+ inclusively: pass the ``'vertices_plus'`` export. ``(B, N, 8, 3)`` input raises
+ :class:`~kornia.core.exceptions.ShapeError`.
+
+ The bounds come from fixed vertices, ``x`` from the first two, ``y`` from the second and third and ``z``
+ from the first and fifth, and fill axis-aligned ranges. A sheared box that :func:`validate_bbox3d` accepts,
+ such as one whose back face is shifted in ``x``, is masked as the axis-aligned box of those vertices, not
+ rasterized.
.. warning::
For fractional coordinates the truncation differs from :func:`bbox_to_mask` and
@@ -444,7 +550,14 @@ def bbox_to_mask3d(boxes: torch.Tensor, size: tuple[int, int, int]) -> torch.Ten
size: depth, height and width of the masked image.
Returns:
- the output mask tensor, shape of :math:`(B, 1, depth, height, width)` and dtype of ``boxes``.
+ the output mask tensor, shape of ``(B, 1, depth, height, width)`` and dtype of ``boxes``.
+
+ See Also:
+ - :func:`bbox_to_mask`: the 2D form, ``(B, H, W)`` masks of ``(B, 4, 2)`` corners with no channel axis, whose
+ bounds are compared unrounded.
+ - :meth:`~kornia.geometry.boxes.Boxes3D.to_mask`: masks of several boxes per volume, ``(N, D, H, W)`` for
+ unbatched boxes and ``(B, N, D, H, W)`` for batched ones, filled over the rounded exclusive bounds.
+ - :func:`infer_bbox_shape3d`: the inclusive depth, height and width of the same corners.
Examples:
>>> boxes = torch.tensor([[
@@ -518,13 +631,21 @@ def bbox_to_mask3d(boxes: torch.Tensor, size: tuple[int, int, int]) -> torch.Ten
def bbox_generator(
x_start: torch.Tensor, y_start: torch.Tensor, width: torch.Tensor, height: torch.Tensor
) -> torch.Tensor:
- """Generate 2D bounding boxes according to the provided start coords, width and height.
+ """Build (B, 4, 2) inclusive box corners from top-left starts, widths and heights.
+
+ ``boxes = bbox_generator(x_start, y_start, width, height)`` takes ``(B,)`` or scalar tensors and returns
+ ``(B, 4, 2)`` corners in clockwise order from the top-left one, with the far corner at ``start + size - 1``, in
+ the dtype of the inputs and with a gradient path to them. These are the corners
+ :meth:`~kornia.geometry.boxes.Boxes.from_tensor` stores for the same boxes in ``'xywh'`` mode, which also
+ validates them and wraps them in a container; to crop the boxes from images, pass the result to
+ :func:`~kornia.geometry.transform.crop_and_resize`. For 3D boxes, use :func:`bbox_generator3d`.
Convention:
The far corner is at ``start + size - 1`` (inclusive, see :func:`infer_bbox_shape`), so
:func:`infer_bbox_shape` reads back ``width`` and ``height``; a zero size puts the far corner one before
- the start. A scalar input gives a batch of one. The four tensors must share dtype and device, otherwise
- ``AssertionError``; the output keeps a gradient path.
+ the start. Scalar inputs give a batch of one, and scalar sizes broadcast against ``(B,)`` starts; scalar
+ starts with ``(B,)`` sizes, ``B > 1``, raise ``RuntimeError``. The four tensors must share dtype and device,
+ otherwise ``AssertionError``.
Args:
x_start: a tensor containing the x coordinates of the bounding boxes to be extracted. Shape must be a scalar
@@ -537,6 +658,13 @@ def bbox_generator(
Returns:
the bounding box tensor, shape of ``(B, 4, 2)``.
+ See Also:
+ - :func:`bbox_generator3d`: the 3D form, ``(B, 8, 3)`` corners from a start corner and three sizes.
+ - :func:`infer_bbox_shape`: the inverse, the height and width of the corners.
+ - :meth:`~kornia.geometry.boxes.Boxes.from_tensor`: the same corners from ``'xywh'`` boxes, validated and kept
+ in a container with format conversions.
+ - :func:`~kornia.geometry.transform.crop_and_resize`: crops one such box per image and resizes it.
+
Examples:
>>> x_start = torch.tensor([0, 1])
>>> y_start = torch.tensor([1, 0])
@@ -596,11 +724,18 @@ def bbox_generator3d(
height: torch.Tensor,
depth: torch.Tensor,
) -> torch.Tensor:
- """Generate 3D bounding boxes according to the provided start coords, width, height and depth.
+ """Build (B, 8, 3) inclusive 3D box corners from start corners and sizes.
+
+ ``boxes = bbox_generator3d(x_start, y_start, z_start, width, height, depth)`` takes ``(B,)`` or scalar tensors
+ and returns ``(B, 8, 3)`` corners: four at the front, at ``z_start``, in clockwise order from the top-left one,
+ then the same four at the back, at ``z_start + depth - 1``. They are in the dtype of the inputs, with a gradient
+ path to them. These are the corners :meth:`~kornia.geometry.boxes.Boxes3D.from_tensor` stores for the same boxes
+ in ``'xyzwhd'`` mode. For 2D boxes, use :func:`bbox_generator`.
Convention:
- The four front vertices precede the four back vertices. A scalar input gives a batch of one. The six
- tensors must share dtype and device, otherwise ``AssertionError``; the output keeps a gradient path.
+ The four front vertices precede the four back vertices. Scalar inputs give a batch of one, and scalar sizes
+ broadcast against ``(B,)`` starts; scalar starts with ``(B,)`` sizes, ``B > 1``, raise ``RuntimeError``. The
+ six tensors must share dtype and device, otherwise ``AssertionError``.
The far corner is at ``start + size - 1`` (inclusive, see :func:`infer_bbox_shape3d`), so
:func:`infer_bbox_shape3d` reads back ``size`` on every axis, matching :func:`bbox_generator`.
@@ -619,6 +754,14 @@ def bbox_generator3d(
Returns:
the 3d bounding box tensor ``(B, 8, 3)``.
+ See Also:
+ - :func:`bbox_generator`: the 2D form, ``(B, 4, 2)`` corners from a start corner, a width and a height.
+ - :func:`infer_bbox_shape3d`: the inverse, the depth, height and width of the corners.
+ - :meth:`~kornia.geometry.boxes.Boxes3D.from_tensor`: the same corners from ``'xyzwhd'`` boxes, validated and
+ kept in a container with format conversions.
+ - :func:`~kornia.geometry.transform.crop_and_resize3d`: crops one such box per volume and resizes it; pass
+ ``align_corners=True`` there.
+
Examples:
>>> x_start = torch.tensor([0, 3])
>>> y_start = torch.tensor([1, 4])
@@ -685,12 +828,21 @@ def bbox_generator3d(
def transform_bbox(
trans_mat: torch.Tensor, boxes: torch.Tensor, mode: str = "xyxy", restore_coordinates: Optional[bool] = None
) -> torch.Tensor:
- r"""Apply a transformation matrix to a box or batch of boxes.
+ r"""Transform xyxy or xywh box tensors by 3x3 matrices through their two endpoints.
+
+ ``out = transform_bbox(trans_mat, boxes)`` takes ``(B, 3, 3)`` matrices and ``(N, 4)`` or ``(B, N, 4)`` boxes in
+ ``mode`` ``'xyxy'`` (the default) or ``'xywh'``, and returns the transformed boxes in the same mode and shape.
+ Only the two endpoints ``(xmin, ymin)`` and ``(xmax, ymax)`` are transformed, and by default sorted again so that
+ ``xmin <= xmax`` and ``ymin <= ymax``, so under a rotation or a shear the result is not the bounding box of the
+ transformed box. ``(B, N, 4, 2)`` corners are transformed vertex by vertex instead. For the bounding box of a
+ rotated box, use :meth:`~kornia.geometry.boxes.Boxes.transform_boxes`, which transforms all four corners, and
+ export it with :meth:`~kornia.geometry.boxes.Boxes.to_tensor`.
Convention:
``xyxy`` and ``xywh`` inputs are transformed through endpoint pairs. Coordinate
restoration sorts their endpoints after a flip; polygon vertices retain their
- transformed cyclic order.
+ transformed cyclic order. ``'xywh'`` boxes are converted with ``xmax = xmin + width``, without the
+ inclusive ``+1`` of the corner functions of this module. ``boxes`` is not modified.
Args:
trans_mat: The transformation matrix to be applied, with supported shape ``(B, 3, 3)``.
@@ -708,6 +860,13 @@ def transform_bbox(
Returns:
The transformed boxes in the specified mode.
+ See Also:
+ - :meth:`~kornia.geometry.boxes.Boxes.transform_boxes`: transforms all four corners of the boxes in a
+ container, whose ``to_tensor`` export is the bounding box of a rotated box.
+ - :func:`~kornia.geometry.linalg.transform_points`: the point transform applied to the endpoints and corners.
+ - :func:`~kornia.geometry.transform.warp_perspective`: warps the images by the same source-to-destination
+ matrix.
+
"""
if not isinstance(mode, str):
raise TypeError(f"Mode must be a string. Got {type(mode)}")
@@ -744,7 +903,14 @@ def transform_bbox(
def nms(boxes: torch.Tensor, scores: torch.Tensor, iou_threshold: float) -> torch.Tensor:
- """Perform non-maxima suppression (NMS) on tensor of bounding boxes according to the intersection-over-union (IoU).
+ """Suppress overlapping (N, 4) xyxy boxes by IoU, keeping the highest-scored ones.
+
+ ``keep = nms(boxes, scores, iou_threshold)`` takes ``N`` exclusive ``xyxy`` boxes ``(N, 4)`` and their scores
+ ``(N,)``, and returns the ``int64`` indices of the boxes it keeps, highest score first: it keeps the
+ highest-scored remaining box, drops every remaining box whose IoU with it exceeds ``iou_threshold``, and repeats
+ until no box remains. It takes one set of boxes per call; see the notes for how it compares with
+ ``torchvision.ops.nms``. To keep the local maxima of a response map rather than boxes, use
+ :func:`~kornia.geometry.subpix.nms2d`.
Convention:
Boxes use exclusive ``xyxy`` coordinates. IoU area is
@@ -772,15 +938,28 @@ def nms(boxes: torch.Tensor, scores: torch.Tensor, iou_threshold: float) -> torc
runs it eagerly, so it is outside the :ref:`torch.compile promise <export-support-promise>`.
Note:
- Two boxes whose union is empty, such as two zero-width boxes, have an IoU of ``0``, so neither suppresses the
- other, as in ``torchvision.ops.nms``. A box with a NaN coordinate has a NaN IoU with every other box, which
- is not above ``iou_threshold``: it neither suppresses nor is suppressed, as in ``torchvision.ops.nms``.
+ For ``float32`` or ``float64`` boxes with finite coordinates, distinct scores and ``iou_threshold >= 0`` it
+ keeps the same boxes, in the same order, as ``torchvision.ops.nms``. Equal scores are ordered by
+ :func:`torch.sort`, which is not stable, so with ties the order, and which of two overlapping tied boxes is
+ kept, can differ from torchvision. A box with a NaN coordinate has a NaN IoU with every other box, which is
+ not above ``iou_threshold``: it neither suppresses nor is suppressed, as in ``torchvision.ops.nms``.
+
+ Note:
+ Two boxes whose union is empty, such as two zero-width boxes, have an IoU of ``0``, so at an ``iou_threshold``
+ of at least ``0`` neither suppresses the other, as in ``torchvision.ops.nms``. With a negative threshold the
+ lower-scored one is dropped, where torchvision, whose IoU for them is NaN, keeps both.
.. warning::
This differs from the inclusive coordinate arithmetic used by several other
:mod:`kornia.geometry.bbox` operations and is documented in
`#4008 <https://github.com/kornia/kornia/issues/4008>`_.
+ See Also:
+ - :func:`~kornia.geometry.subpix.nms2d`: keeps the strict local maxima of a ``(B, C, H, W)`` response map, not
+ boxes.
+ - :meth:`~kornia.geometry.boxes.Boxes.to_tensor`: exports the boxes of a container as the exclusive
+ ``'xyxy'`` boxes this function takes.
+
Example:
>>> boxes = torch.tensor([
... [10., 10., 20., 20.],
diff --git a/kornia/geometry/boxes.py b/kornia/geometry/boxes.py
index 7dc8f8546..4d80acd27 100644
--- a/kornia/geometry/boxes.py
+++ b/kornia/geometry/boxes.py
@@ -15,6 +15,27 @@
# limitations under the License.
#
+"""Box containers: 2D, 3D and video boxes stored as inclusive corners, with conversions.
+
+:class:`Boxes` stores 2D boxes as four ``(x, y)`` corners each, ``(N, 4, 2)`` for one image or ``(B, N, 4, 2)`` for a
+batch; :class:`Boxes3D` stores 3D boxes as eight ``(x, y, z)`` corners each; :class:`VideoBoxes` stores the 2D boxes of
+``(B, T)`` video frames. :meth:`~kornia.geometry.boxes.Boxes.from_tensor` and
+:meth:`~kornia.geometry.boxes.Boxes3D.from_tensor` import boxes in one of several formats, ``'xyxy'`` and ``'xyzxyz'``
+by default, and ``to_tensor`` exports them in any format, always as the axis-aligned bounds of the stored corners. The
+stored corners are inclusive pixel centres, ``x_max - x_min + 1`` wide, as in :mod:`kornia.geometry.bbox`
+(`#3934 <https://github.com/kornia/kornia/issues/3934>`_): the formats ending in ``_plus`` are inclusive, the others
+exclusive. The ``Convention`` block of each class gives its formats, padded batches and validation.
+
+Which class to use:
+
+- 2D boxes of images, several per image, with padding, clamping, area filtering and masks: :class:`Boxes`.
+- 2D boxes of video frames, exported with the temporal axis, ``(B, T, N, 4)`` in ``'xyxy'``: :class:`VideoBoxes`.
+- 3D boxes in volumes: :class:`Boxes3D`.
+- One box per image as a plain ``(B, 4, 2)`` corner tensor, without a container: the functions of
+ :mod:`kornia.geometry.bbox`, such as :func:`~kornia.geometry.bbox.bbox_to_mask`.
+- Points instead of boxes: :class:`~kornia.geometry.keypoints.Keypoints`.
+"""
+
from __future__ import annotations
from typing import Any, Optional, Tuple, cast
@@ -207,7 +228,18 @@ def _boxes3d_to_polygons3d(
class Boxes:
- r"""2D boxes containing N or BxN boxes.
+ r"""Store 2D boxes as inclusive corners, converting to and from xyxy, xywh and vertices.
+
+ ``boxes = Boxes.from_tensor(xyxy, mode='xyxy')`` takes the ``(N, 4)`` boxes of one image or the ``(B, N, 4)`` boxes
+ of a batch (``(N, 4, 2)`` or ``(B, N, 4, 2)`` corners in the vertex modes, or a list of per-image tensors) and
+ stores them as inclusive corners; ``boxes.to_tensor('xywh')`` exports them in any mode, with the same batch layout.
+ In between, :meth:`transform_boxes` applies ``3 x 3`` matrices to all four corners, :meth:`to_mask` draws
+ ``(N, H, W)`` or ``(B, N, H, W)`` masks, :meth:`pad` shifts the corners by image padding, :meth:`clamp` limits
+ them to per-image bounds, :meth:`filter_boxes_by_area` removes boxes and :meth:`merge` appends others.
+ ``Boxes(corners)`` wraps inclusive ``'vertices_plus'`` corners as they are.
+ :class:`~kornia.augmentation.container.AugmentationSequential` accepts a :class:`Boxes` and transforms it with the
+ images. For one box per image as a plain ``(B, 4, 2)`` tensor, use the :mod:`kornia.geometry.bbox` functions; for
+ 3D boxes, :class:`Boxes3D`; for the boxes of video frames, :class:`VideoBoxes`.
Args:
boxes: 2D boxes, shape of ``(N, 4, 2)``, ``(B, N, 4, 2)`` or a list of ``(N, 4, 2)``.
@@ -233,6 +265,13 @@ class Boxes:
- :func:`~kornia.geometry.bbox.infer_bbox_shape` and :func:`~kornia.geometry.bbox.bbox_to_mask` read
their input as inclusive: pass them the ``'vertices_plus'`` export, unbatched.
:func:`~kornia.geometry.bbox.nms` takes exclusive ``xyxy``.
+ - The modes of :meth:`from_tensor` and :meth:`to_tensor`, case-insensitive: ``'xyxy'`` is
+ ``(xmin, ymin, xmax, ymax)`` with ``xmax`` and ``ymax`` one past the last pixel, ``'xyxy_plus'`` the same
+ on the last pixel, ``'xywh'`` is ``(xmin, ymin, width, height)``, ``'vertices'`` the corners with ``xmax``
+ and ``ymax`` one past the last pixel, and ``'vertices_plus'`` the stored corners.
+ ``from_tensor(torch.tensor([[[0., 0., 10., 20.]]]))`` stores ``[[0, 0], [9, 0], [9, 19], [0, 19]]``,
+ which ``to_tensor`` exports as ``[0, 0, 10, 20]`` in ``'xyxy'`` and ``'xywh'`` and ``[0, 0, 9, 19]`` in
+ ``'xyxy_plus'``.
- With ``validate_boxes=True``, every mode rejects a non-finite coordinate. The ``'xy*'`` modes raise
``ValueError`` and also reject non-positive extents. The vertex modes raise ``RuntimeError`` through
``torch._assert_async`` (by the next synchronization on CUDA, and on MPS from torch 2.13), so they stay
@@ -241,7 +280,18 @@ class Boxes:
- The constructor rejects an integer tensor unless ``raise_if_not_floating_point=False`` (a list is
checked by its first element's dtype); :meth:`from_tensor` casts integer input to ``float32``,
or ``float64`` when that is the default dtype.
- - :meth:`merge` and :meth:`index_put` are non-mutating by default.
+ - A list of per-image tensors, ``(N_i, 4)`` or ``(N_i, 4, 2)`` in :meth:`from_tensor` and ``(N_i, 4, 2)``
+ in the constructor, is zero-padded to one ``(B, max N_i, 4, 2)`` batch; :meth:`to_tensor` then returns a
+ list trimmed back to each image's boxes, or the padded tensor with ``as_padded_sequence=True``. Item
+ assignment and :meth:`index_put` do not update the padding counts, so the trimmed export can then drop
+ real boxes or keep padding (`#6283 <https://github.com/kornia/kornia/issues/6283>`_).
+ - :meth:`filter_boxes_by_area` removes boxes without changing the shape: it zeroes their corners and marks
+ them invalid, and every export of an invalid box is exact zeros, after later transforms too.
+ - :meth:`transform_boxes`, :meth:`translate`, :meth:`clamp`, :meth:`filter_boxes_by_area`, :meth:`merge`
+ and :meth:`index_put` are non-mutating by default and return a new object; with ``inplace=True`` they
+ update and return ``self``. :meth:`pad` and :meth:`unpad` always shift ``self``. The constructor wraps a
+ floating-point tensor without copying it, so item assignment and the in-place updates of :meth:`pad`,
+ :meth:`unpad`, :meth:`clamp`, :meth:`filter_boxes_by_area` and :meth:`index_put` reach the caller's tensor.
.. warning::
The inclusive ``+1`` differs from torchvision, COCO and albumentations:
@@ -260,6 +310,18 @@ class Boxes:
`#3934 <https://github.com/kornia/kornia/issues/3934>`_. ``trim`` was never implemented and is deprecated:
`#4017 <https://github.com/kornia/kornia/issues/4017>`_.
+ See Also:
+ - :class:`Boxes3D`: 3D boxes as eight corners, from ``(N, 6)`` boxes, with no list input, padding or
+ filtering.
+ - :class:`VideoBoxes`: the 2D boxes of ``(B, T)`` video frames, exported with the temporal axis.
+ - :func:`~kornia.geometry.bbox.bbox_to_mask`: the mask of one box per image of a plain ``(B, 4, 2)`` tensor,
+ with unrounded inclusive bounds.
+ - :func:`~kornia.geometry.bbox.transform_bbox`: transforms plain ``xyxy`` or ``xywh`` tensors through their two
+ endpoints only.
+ - :func:`~kornia.geometry.bbox.nms`: suppresses overlapping boxes of one image, given the ``'xyxy'`` export.
+ - :class:`~kornia.geometry.keypoints.Keypoints`: ``(x, y)`` points in the same pixel frame, with matching pad,
+ unpad and transform methods.
+
"""
def __init__(
@@ -825,8 +887,8 @@ class Boxes:
See the Convention block on :class:`~kornia.geometry.boxes.Boxes`.
Convention:
- The size is ``(height, width)`` and the mask :math:`(N, height, width)` or
- :math:`(B, N, height, width)` in the box dtype. The exclusive ``'xyxy'`` export is clamped to the
+ The size is ``(height, width)`` and the mask ``(N, height, width)`` or
+ ``(B, N, height, width)`` in the box dtype. The exclusive ``'xyxy'`` export is clamped to the
image, rounded to the nearest integer and filled over ``[xmin, xmax)``, ``[ymin, ymax)``. Padding
entries of a list-backed object give empty masks. A box tensor that requires grad raises
``RuntimeError``.
@@ -842,7 +904,7 @@ class Boxes:
width: width of the masked image/images.
Returns:
- the output mask tensor, shape of :math:`(N, height, width)` or :math:`(B, N, height, width)` and dtype of
+ the output mask tensor, shape of ``(N, height, width)`` or ``(B, N, height, width)`` and dtype of
:func:`Boxes.dtype` (it can be any floating point dtype).
Note:
@@ -1010,8 +1072,8 @@ class Boxes:
Returns:
Mode string used as the default by :meth:`to_tensor`, such as
- ``"xyxy"``, ``"xywh"``, ``"vertices"``, or their ``"_plus"``
- variants.
+ ``"xyxy"``, ``"xyxy_plus"``, ``"xywh"``, ``"vertices"`` or
+ ``"vertices_plus"``.
"""
return self._mode
@@ -1026,7 +1088,11 @@ class Boxes:
return self._data.dtype
def to(self, device: Optional[torch.device] = None, dtype: Optional[torch.dtype] = None) -> Boxes:
- """Like :func:`torch.nn.Module.to()` method."""
+ """Move or cast the stored tensor and return ``self``, as :meth:`torch.nn.Module.to` does.
+
+ The tensor is rebound, not converted in place: one obtained from :attr:`data` before the call keeps its device
+ and dtype. A ``dtype`` that is not floating point raises ``ValueError``.
+ """
# In torchscript, dtype is a int and not a class. https://github.com/pytorch/pytorch/issues/51941
if dtype is not None and not _is_floating_point_dtype(dtype):
raise ValueError("Boxes must be in floating point")
@@ -1053,10 +1119,10 @@ class Boxes:
"""Cast the stored box coordinates to a new dtype.
Args:
- dtype: Target floating-point dtype for the coordinate tensor.
+ dtype: Target dtype for the coordinate tensor. Unlike :meth:`to`, an integer dtype is not rejected.
Returns:
- ``self`` after converting :attr:`data` in place.
+ ``self``, rebound to the converted tensor; the tensor it wrapped before is not modified.
"""
self._data = self._data.type(dtype)
self._valid = self._valid.to(device=self._data.device)
@@ -1064,19 +1130,21 @@ class Boxes:
class VideoBoxes(Boxes):
- r"""2D boxes with an explicit temporal channel for video sequences.
+ r"""Store the 2D boxes of (B, T) video frames, restoring the temporal axis on export.
- Accepts and returns box corners for a batch of videos as
- ``(B, T, N, 4, 2)`` in ``vertices_plus`` mode. Internally the corners
- are stored flattened as :math:`(B \cdot T, N, 4, 2)`.
- :class:`~kornia.augmentation.AugmentationSequential` uses this wrapper when
- the pipeline contains a video sequential so that ``to_tensor`` restores the
- temporal axis after geometric transforms.
+ ``boxes = VideoBoxes.from_tensor(corners)`` takes inclusive ``'vertices_plus'`` corners ``(B, T, N, 4, 2)``, with
+ no mode or conversion, and stores them flattened as ``(B * T, N, 4, 2)``, so the methods it inherits from
+ :class:`Boxes` act on every frame; ``boxes.to_tensor('xyxy')`` exports ``(B, T, N, 4)``, with the temporal axis
+ restored, and ``boxes.to_tensor()`` the ``(B, T, N, 4, 2)`` corners.
+ :class:`~kornia.augmentation.container.AugmentationSequential` stores the boxes of a pipeline that contains a
+ :class:`~kornia.augmentation.container.VideoSequential` in this class. Convert boxes in another format to corners
+ first: ``Boxes.from_tensor(xyxy.flatten(0, 1)).data.unflatten(0, (B, T))`` converts ``(B, T, N, 4)`` ``xyxy``
+ boxes. For the boxes of single images, use :class:`Boxes`.
See the Convention block on :class:`~kornia.geometry.boxes.Boxes`.
Convention:
- - :meth:`from_tensor` stores ``(B, T, N, 4, 2)`` input unchanged as :math:`(B \cdot T, N, 4, 2)`
+ - :meth:`from_tensor` stores ``(B, T, N, 4, 2)`` input unchanged as ``(B * T, N, 4, 2)``
``'vertices_plus'`` data, with no mode or conversion; integer input is cast to ``float32``,
or ``float64`` when that is the default dtype. Another shape or a list raises ``ValueError``.
``validate_boxes=True`` rejects a non-finite coordinate as
@@ -1087,17 +1155,23 @@ class VideoBoxes(Boxes):
``(B, T, N, height, width)`` masks. :attr:`~Boxes.shape` and :attr:`~Boxes.data` stay the flattened
storage ``(B * T, N, 4, 2)``.
- Indexing selects the video batch axis while preserving the temporal axis. An integer index returns a
- one-video :class:`VideoBoxes` so :meth:`to_tensor` keeps the shape :math:`(1, T, \ldots)`. Index
+ one-video :class:`VideoBoxes` so :meth:`to_tensor` keeps the shape ``(1, T, ...)``. Index
assignment, ``video_boxes[key] = other``, writes the same whole videos.
- - A transformation matrix is :math:`(B \cdot T, 3, 3)`; a ``(3, 3)`` matrix raises ``ValueError``
+ - A transformation matrix is ``(B * T, 3, 3)``; a ``(3, 3)`` matrix raises ``ValueError``
unless :math:`B \cdot T = 1`.
- - :meth:`transform_boxes`, :meth:`translate`, :meth:`clamp`, :meth:`filter_boxes_by_area` and
- :meth:`merge` return a new :class:`VideoBoxes`; :meth:`pad`, :meth:`unpad`, :meth:`to` and
- :meth:`type` update ``self`` in place.
+ - :meth:`Boxes.transform_boxes`, :meth:`Boxes.translate`, :meth:`Boxes.clamp`,
+ :meth:`Boxes.filter_boxes_by_area` and :meth:`Boxes.merge` return a new :class:`VideoBoxes`;
+ :meth:`Boxes.pad`, :meth:`Boxes.unpad`, :meth:`Boxes.to` and :meth:`Boxes.type` update and return ``self``.
Attributes:
temporal_channel_size: Number of frames :math:`T` stored with the boxes.
+ See Also:
+ - :class:`Boxes`: the 2D boxes of single images, imported from any mode, with no temporal axis.
+ - :class:`Boxes3D`: boxes in a ``(D, H, W)`` volume, eight corners each, rather than boxes per frame.
+ - :class:`~kornia.augmentation.container.VideoSequential`: the video augmentation whose boxes
+ :class:`~kornia.augmentation.container.AugmentationSequential` keeps in this class.
+
"""
temporal_channel_size: int
@@ -1118,7 +1192,7 @@ class VideoBoxes(Boxes):
:meth:`~kornia.geometry.boxes.Boxes.from_tensor`.
Returns:
- :class:`VideoBoxes` with :attr:`temporal_channel_size` set to
+ :class:`VideoBoxes` with ``temporal_channel_size`` set to
``boxes.size(1)``.
Raises:
@@ -1187,12 +1261,12 @@ class VideoBoxes(Boxes):
changes the result of a list-backed container; a
:class:`VideoBoxes` built from a ``(B, T, N, 4, 2)`` tensor
is not list-backed, so the value makes no difference there. The
- keyword is accepted because the inherited :meth:`get_boxes_shape`
- and :meth:`to_mask` pass it.
+ keyword is accepted because the inherited :meth:`Boxes.get_boxes_shape`
+ and :meth:`Boxes.to_mask` pass it.
Returns:
- Tensor shaped :math:`(B, T, \ldots)` where :math:`T` is
- :attr:`temporal_channel_size`.
+ Tensor shaped ``(B, T, ...)`` where :math:`T` is
+ ``temporal_channel_size``.
"""
out = super().to_tensor(mode, as_padded_sequence=as_padded_sequence)
batch_size = self._data.shape[0] // self.temporal_channel_size
@@ -1206,7 +1280,7 @@ class VideoBoxes(Boxes):
Returns:
New :class:`VideoBoxes` with cloned tensor storage and the same
- :attr:`temporal_channel_size`, mode, and batch metadata.
+ ``temporal_channel_size``, mode, and batch metadata.
"""
obj = type(self)(self._data.clone(), False)
obj._mode = self._mode
@@ -1247,7 +1321,15 @@ class VideoBoxes(Boxes):
class Boxes3D:
- r"""3D boxes containing N or BxN boxes.
+ r"""Store 3D boxes as eight inclusive corners, converting to and from xyzxyz and xyzwhd.
+
+ ``boxes = Boxes3D.from_tensor(xyzxyz)`` takes ``(N, 6)`` or ``(B, N, 6)`` boxes in ``'xyzxyz'`` (the default),
+ ``'xyzxyz_plus'`` or ``'xyzwhd'`` mode and stores them as ``(N, 8, 3)`` or ``(B, N, 8, 3)`` inclusive corners;
+ ``boxes.to_tensor('xyzwhd')`` exports them in any of these modes or as corners. In between,
+ :meth:`transform_boxes` applies ``4 x 4`` matrices to all eight corners and :meth:`to_mask` draws
+ ``(N, D, H, W)`` or ``(B, N, D, H, W)`` masks. Unlike :class:`Boxes` it takes no list input and has no padding,
+ clamping, merging or area filtering. For one box per volume as a plain ``(B, 8, 3)`` tensor, use
+ :func:`~kornia.geometry.bbox.bbox_to_mask3d` and :func:`~kornia.geometry.bbox.infer_bbox_shape3d`.
Args:
boxes: 3D boxes, shape of ``(N,8,3)`` or ``(B,N,8,3)``. See below for more details.
@@ -1285,6 +1367,15 @@ class Boxes3D:
The :meth:`to_tensor` default differs from :meth:`Boxes.to_tensor`, which defaults to the stored mode:
`#4251 <https://github.com/kornia/kornia/issues/4251>`_.
+ See Also:
+ - :class:`Boxes`: 2D boxes as four corners, with list input, padding, clamping and area filtering.
+ - :class:`VideoBoxes`: the 2D boxes of ``(B, T)`` video frames, rather than boxes in a volume.
+ - :func:`~kornia.geometry.bbox.bbox_to_mask3d`: the ``(B, 1, D, H, W)`` mask of one box per volume of a plain
+ ``(B, 8, 3)`` tensor, with truncated bounds.
+ - :func:`~kornia.geometry.bbox.infer_bbox_shape3d`: the inclusive sizes of a plain ``(B, 8, 3)`` tensor.
+ - :class:`~kornia.geometry.keypoints.Keypoints3D`: ``(x, y, z)`` points, transformed by the same ``4 x 4``
+ matrices.
+
"""
def __init__(
@@ -1455,7 +1546,7 @@ class Boxes3D:
* 'xyzxyz': boxes are assumed to be in the format ``xmin, ymin, zmin, xmax, ymax, zmax`` where
``width = xmax - xmin``, ``height = ymax - ymin`` and ``depth = zmax - zmin``.
* 'xyzxyz_plus': similar to 'xyzxyz' mode but where box width, length and depth are defined as
- ``width = xmax - xmin + 1``, ``height = ymax - ymin + 1`` and ``depth = zmax - zmin + 1``.
+ ``width = xmax - xmin + 1``, ``height = ymax - ymin + 1`` and ``depth = zmax - zmin + 1``.
* 'xyzwhd': boxes are assumed to be in the format ``xmin, ymin, zmin, width, height, depth`` where
``width = xmax - xmin``, ``height = ymax - ymin`` and ``depth = zmax - zmin``.
* 'vertices': boxes are defined by their vertices points in the following ``clockwise`` order:
@@ -1476,7 +1567,8 @@ class Boxes3D:
Note:
The ``amin``/``amax`` reduction ties on every face of an axis-aligned box (4 vertices share each
extremum), so the gradient there is a subgradient shared among the tied vertices; it sums to one
- over them but need not match a finite-difference estimate such as :func:`torch.autograd.gradcheck`.
+ over them but need not match a finite-difference estimate such as
+ :func:`~torch.autograd.gradcheck.gradcheck`.
Examples:
>>> boxes_xyzxyz = torch.as_tensor([[0, 3, 6, 1, 4, 8], [5, 1, 3, 8, 4, 9]])
@@ -1522,8 +1614,8 @@ class Boxes3D:
See the Convention block on :class:`~kornia.geometry.boxes.Boxes3D`.
Convention:
- The size is ``(depth, height, width)`` and the mask :math:`(N, depth, height, width)` or
- :math:`(B, N, depth, height, width)` in the box dtype, with no channel axis. The exclusive
+ The size is ``(depth, height, width)`` and the mask ``(N, depth, height, width)`` or
+ ``(B, N, depth, height, width)`` in the box dtype, with no channel axis. The exclusive
``'xyzxyz'`` export is clamped to the volume, rounded to the nearest integer and filled over
half-open ranges. A box tensor that requires grad raises ``RuntimeError``.
@@ -1537,7 +1629,7 @@ class Boxes3D:
width: width of the masked image/images.
Returns:
- the output mask tensor, shape of :math:`(N, depth, height, width)` or :math:`(B, N, depth, height, width)`
+ the output mask tensor, shape of ``(N, depth, height, width)`` or ``(B, N, depth, height, width)``
and dtype of :func:`Boxes3D.dtype` (it can be any floating point dtype).
Note:
@@ -1684,8 +1776,9 @@ class Boxes3D:
"""Return the 3D box format remembered by this container.
Returns:
- Mode string describing how this container should be interpreted
- during tensor conversion.
+ The mode label given to :meth:`from_tensor` or the constructor; the new object that
+ :meth:`transform_boxes` returns is labelled ``'xyzxyz_plus'``. :meth:`to_tensor` does not read it
+ (`#4251 <https://github.com/kornia/kornia/issues/4251>`_).
"""
return self._mode
@@ -1700,7 +1793,11 @@ class Boxes3D:
return self._data.dtype
def to(self, device: Optional[torch.device] = None, dtype: Optional[torch.dtype] = None) -> Boxes3D:
- """Like :func:`torch.nn.Module.to()` method."""
+ """Move or cast the stored tensor and return ``self``, as :meth:`torch.nn.Module.to` does.
+
+ The tensor is rebound, not converted in place: one obtained from :attr:`data` before the call keeps its device
+ and dtype. A ``dtype`` that is not floating point raises ``ValueError``.
+ """
# In torchscript, dtype is a int and not a class. https://github.com/pytorch/pytorch/issues/51941
if dtype is not None and not _is_floating_point_dtype(dtype):
raise ValueError("Boxes must be in floating point")
diff --git a/kornia/geometry/calibration/__init__.py b/kornia/geometry/calibration/__init__.py
index a660560f0..33202f033 100644
--- a/kornia/geometry/calibration/__init__.py
+++ b/kornia/geometry/calibration/__init__.py
@@ -15,9 +15,23 @@
# limitations under the License.
#
-"""Kornia Geometry Calibration — Camera calibration utilities for Kornia.
+r"""Lens distortion of pixel coordinates with OpenCV's Brown-Conrady model, and camera pose from 3D-2D points.
-This subpackage provides functions for camera calibration, distortion, and undistortion.
+Image points are pixel coordinates ``(u, v)`` = (column, row) with integer pixel centres, and ``K`` is a ``(*, 3, 3)``
+pinhole matrix. ``dist`` is OpenCV's coefficient vector
+:math:`(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6[, s_1, s_2, s_3, s_4[, \tau_x, \tau_y]]]])` of 4, 5, 8, 12 or 14
+elements: radial, tangential, thin-prism and sensor-tilt terms, as ``cv2.projectPoints`` and
+``cv2.undistortPoints`` read it.
+
+Which function to use:
+
+- Add the distortion to ideal pixels: :func:`distort_points`. Remove it from measured pixels:
+ :func:`undistort_points`; from a whole image: :func:`undistort_image`.
+- The sensor-tilt matrix of the last two coefficients, or its inverse: :func:`tilt_projection`.
+- A world-to-camera pose from ``N >= 6`` world points and their pixels: :func:`solve_pnp_dlt`.
+- The Kannala-Brandt fisheye model, on normalized points and a flat parameter vector:
+ :func:`~kornia.geometry.camera.distort_points_kannala_brandt` and
+ :func:`~kornia.geometry.camera.undistort_points_kannala_brandt`.
"""
from .distort import distort_points, tilt_projection
diff --git a/kornia/geometry/calibration/distort.py b/kornia/geometry/calibration/distort.py
index b2d1ee54c..371216f40 100644
--- a/kornia/geometry/calibration/distort.py
+++ b/kornia/geometry/calibration/distort.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
+"""Add OpenCV's Brown-Conrady lens distortion to pixel coordinates, and build its sensor-tilt matrix."""
+
from typing import Optional
import torch
@@ -25,7 +27,14 @@ from kornia.core.check import KORNIA_CHECK_SHAPE
# Based on https://github.com/opencv/opencv/blob/master/modules/calib3d/src/distortion_model.hpp#L75
def tilt_projection(taux: torch.Tensor, tauy: torch.Tensor, return_inverse: bool = False) -> torch.Tensor:
- r"""Estimate the tilt projection matrix or the inverse tilt projection matrix.
+ r"""Build the 3x3 sensor-tilt matrix of OpenCV's distortion model, or its inverse.
+
+ ``T = tilt_projection(taux, tauy)`` takes the tilt angles in radians, the last two of the 14 coefficients of
+ OpenCV's distortion vector, and returns the projective ``(*, 3, 3)`` matrix that
+ :func:`~kornia.geometry.calibration.distort_points` applies to the distorted normalized point ``(x, y, 1)``
+ before dividing by the third component; ``return_inverse=True`` returns its inverse, which
+ :func:`~kornia.geometry.calibration.undistort_points` applies first. Those two functions call it themselves:
+ call it directly only to inspect or reuse the tilt step.
Convention:
- the rotation is ``R = Ry(tauy) @ Rx(taux)`` and ``Pz`` is built from the third column of ``R``;
@@ -45,6 +54,13 @@ def tilt_projection(taux: torch.Tensor, tauy: torch.Tensor, return_inverse: bool
True, with shape ``(3, 3)`` for scalar angles or ``(*, 3, 3)`` for non-scalar angles, where a
trailing singleton angle-component axis is omitted from :math:`*`.
+ Raises:
+ ValueError: if ``taux`` and ``tauy`` differ in shape.
+
+ See Also:
+ - :func:`~kornia.geometry.calibration.distort_points`: applies this matrix when given 14 coefficients.
+ - :func:`~kornia.geometry.calibration.undistort_points`: applies its inverse when given 14 coefficients.
+
"""
if taux.shape != tauy.shape:
raise ValueError(f"Shape of taux {taux.shape} and tauy {tauy.shape} do not match.")
@@ -106,7 +122,14 @@ def tilt_projection(taux: torch.Tensor, tauy: torch.Tensor, return_inverse: bool
def distort_points(
points: torch.Tensor, K: torch.Tensor, dist: torch.Tensor, new_K: Optional[torch.Tensor] = None
) -> torch.Tensor:
- r"""Distortion of a set of 2D points based on the lens distortion model.
+ r"""Distort pixel coordinates with OpenCV's Brown-Conrady model of 4 to 14 coefficients.
+
+ ``distorted = distort_points(points, K, dist)`` takes ideal pinhole pixels ``(*, N, 2)``, the intrinsics
+ ``(*, 3, 3)`` and OpenCV's coefficient vector ``(*, n)``, ``n`` one of 4, 5, 8, 12 or 14, and returns the pixels
+ ``(*, N, 2)`` where a lens with that distortion images the same rays: the values ``cv2.projectPoints`` gives for
+ the points' rays. To remove distortion from measured pixels, use
+ :func:`~kornia.geometry.calibration.undistort_points`; from a whole image,
+ :func:`~kornia.geometry.calibration.undistort_image`.
Radial :math:`(k_1, k_2, k_3, k_4, k_5, k_6)`,
tangential :math:`(p_1, p_2)`, thin prism :math:`(s_1, s_2, s_3, s_4)`, and tilt :math:`(\tau_x, \tau_y)`
@@ -123,14 +146,20 @@ def distort_points(
- ``new_K`` maps the incoming pixel onto the normalized plane and ``K`` maps the distorted point back
to pixels; ``new_K`` defaults to ``K``. :func:`~kornia.geometry.calibration.undistort_points` is the
inverse map, with the two intrinsics in the mirrored roles.
- - matching leading dimensions of ``points``, ``K`` and ``dist`` are preserved.
+ - matching leading dimensions of ``points``, ``K`` and ``dist`` are preserved, and the leading dimensions
+ ``*`` of the three broadcast by torch's rules: ``(N, 2)`` points with ``(B, 3, 3)`` cameras give
+ ``(B, N, 2)``.
+ - the skew entry ``K[0, 1]`` is ignored, as in
+ :func:`~kornia.geometry.conversions.normalize_points_with_intrinsics`.
+ - ``points`` must be ``(*, N, 2)``: a trailing size other than 2, or a single ``(2,)`` point, raises
+ :class:`~kornia.core.exceptions.ShapeError`.
Args:
points: Input image points with shape ``(*, N, 2)``.
K: Intrinsic camera matrix with shape ``(*, 3, 3)``.
dist: Distortion coefficients
:math:`(k_1,k_2,p_1,p_2[,k_3[,k_4,k_5,k_6[,s_1,s_2,s_3,s_4[,\tau_x,\tau_y]]]])`. This is
- a vector with 4, 5, 8, 12 or 14 elements with shape :math:`(*, n)`.
+ a vector with 4, 5, 8, 12 or 14 elements with shape ``(*, n)``.
new_K: Intrinsic camera matrix used to map the incoming ``points`` from pixels onto the normalized
:math:`z = 1` plane -- the opposite of what the argument order suggests, since ``K`` is the one
that maps the distorted normalized point back to pixels. By default it is the same as ``K``, in
@@ -140,6 +169,18 @@ def distort_points(
Returns:
Distorted 2D points with shape ``(*, N, 2)``.
+ See Also:
+ - :func:`~kornia.geometry.calibration.undistort_points`: the inverse, by fixed-point iteration, with ``K``
+ and ``new_K`` in mirrored roles.
+ - :func:`~kornia.geometry.calibration.undistort_image`: removes the distortion from an image by sampling
+ it at the positions this function returns.
+ - :func:`~kornia.geometry.calibration.tilt_projection`: the sensor-tilt matrix of the last two
+ coefficients.
+ - :func:`~kornia.geometry.camera.distort_points_kannala_brandt`: the fisheye model, from normalized points
+ and a flat 8-vector.
+ - :func:`~kornia.geometry.camera.distort_points_affine`: the intrinsics alone, from normalized points and a
+ flat ``[fx, fy, cx, cy]``.
+
Example:
>>> points = torch.rand(1, 1, 2)
>>> K = torch.eye(3)[None]
diff --git a/kornia/geometry/calibration/pnp.py b/kornia/geometry/calibration/pnp.py
index f31243502..683976d85 100644
--- a/kornia/geometry/calibration/pnp.py
+++ b/kornia/geometry/calibration/pnp.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
+"""Estimate a camera's world-to-camera pose from 3D-2D correspondences by the direct linear transform (PnP)."""
+
from typing import Optional, Tuple
import torch
@@ -72,16 +74,15 @@ def solve_pnp_dlt(
weights: Optional[torch.Tensor] = None,
svd_eps: float = 1e-4,
) -> torch.Tensor:
- r"""Attempt to solve the Perspective-n-Point (PnP) problem using Direct Linear Transform (DLT).
-
- Given a batch (where batch size is :math:`B`) of :math:`N` 3D points
- (where :math:`N \geq 6`) in the world space, a batch of :math:`N`
- corresponding 2D points in the image space and a batch of
- intrinsic matrices, this function tries to estimate a batch of
- world to camera transformation matrices.
+ r"""Estimate a camera's world-to-camera pose from 3D-2D correspondences by the DLT (PnP).
- This implementation needs at least 6 points (i.e. :math:`N \geq 6`) to
- provide solutions.
+ ``Rt = solve_pnp_dlt(world_points, img_points, K)`` takes ``B`` sets of :math:`N \geq 6` world points
+ ``(B, N, 3)``, their pixels ``(B, N, 2)`` and the intrinsics ``(B, 3, 3)``, and returns the world-to-camera
+ extrinsics ``[R | t]`` of each set, ``(B, 3, 4)``: the Hartley-normalized, optionally weighted, linear
+ (Direct Linear Transform) solution of the Perspective-n-Point problem, with ``R`` made orthonormal by a QR step.
+ It is not robust to outliers and does not refine the pose nonlinearly. For 2D-2D correspondences of a plane,
+ use :func:`~kornia.geometry.homography.find_homography_dlt`; for two views with no known 3D points,
+ :func:`~kornia.geometry.epipolar.find_essential`.
This function cannot be used if all the 3D world points (of any element
of the batch) lie on a line or if all the 3D world points (of any element
@@ -96,8 +97,8 @@ def solve_pnp_dlt(
- the returned ``(B, 3, 4)`` matrix is the **world-to-camera** ``[R | t]``: it maps a world point
into the camera frame (``X_cam = X_world + (1, 0, 0)`` gives ``t = (+1, 0, 0)``), not a
camera-to-world pose.
- - ``intrinsics`` is the ``(B, 3, 3)`` ``K``; the ``(B, 4, 4)`` matrix that a
- ``PinholeCamera`` stores is rejected.
+ - ``intrinsics`` is the ``(B, 3, 3)`` ``K``, or a ``(1, 3, 3)`` ``K`` shared by the batch; the
+ ``(B, 4, 4)`` matrix that a :class:`~kornia.geometry.camera.pinhole.PinholeCamera` stores is rejected.
- ``weights`` scales the two rows each point contributes to the homogeneous linear system, and the
normalization and the degeneracy check use the same weighting. A uniform ``weights`` leaves the answer
unchanged up to rounding, and a zero weight is the same as removing the point.
@@ -121,6 +122,16 @@ def solve_pnp_dlt(
A torch.Tensor with shape ``(B, 3, 4)`` representing the estimated world to
camera transformation matrices (also known as the extrinsic matrices).
+ See Also:
+ - :func:`~kornia.geometry.homography.find_homography_dlt`: the DLT for 2D-2D correspondences, a homography
+ rather than a pose.
+ - :func:`~kornia.geometry.epipolar.find_essential`: the relative pose of two views from 2D-2D
+ correspondences, without 3D points.
+ - :func:`~kornia.geometry.epipolar.projection_from_KRt`: the ``(B, 3, 4)`` projection matrix
+ ``K [R | t]`` of the result.
+ - :class:`~kornia.geometry.camera.pinhole.PinholeCamera`: a camera model that takes the result, padded to
+ ``(B, 4, 4)``, as its extrinsics.
+
Example:
>>> world_points = torch.tensor([[
... [ 5. , -5. , 0. ], [ 0. , 0. , 1.5],
diff --git a/kornia/geometry/calibration/undistort.py b/kornia/geometry/calibration/undistort.py
index 1ea70a01e..be5254098 100644
--- a/kornia/geometry/calibration/undistort.py
+++ b/kornia/geometry/calibration/undistort.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
+"""Remove OpenCV's Brown-Conrady lens distortion from pixel coordinates and from images."""
+
from __future__ import annotations
from typing import Optional
@@ -34,7 +36,17 @@ from .distort import distort_points, tilt_projection
def undistort_points(
points: torch.Tensor, K: torch.Tensor, dist: torch.Tensor, new_K: Optional[torch.Tensor] = None, num_iters: int = 5
) -> torch.Tensor:
- r"""Compensate for lens distortion a set of 2D image points.
+ r"""Remove Brown-Conrady lens distortion from pixel coordinates by fixed-point iteration.
+
+ ``points = undistort_points(distorted, K, dist)`` takes measured pixels ``(*, N, 2)``, the intrinsics
+ ``(*, 3, 3)`` and OpenCV's coefficient vector ``(*, n)``, ``n`` one of 4, 5, 8, 12 or 14, and returns the
+ ideal pinhole pixels ``(*, N, 2)`` after ``num_iters`` fixed-point steps; within the convergence region, with
+ the default 5 steps, the result matches ``cv2.undistortPoints(points, K, dist, P=P)`` with
+ ``P = K if new_K is None else new_K`` and no skew in ``P``: OpenCV applies ``P[0, 1]``, which this function
+ ignores, and without ``P`` it returns normalized coordinates. For normalized
+ coordinates, as :func:`~kornia.geometry.epipolar.find_essential` takes, pass the result to
+ :func:`~kornia.geometry.conversions.normalize_points_with_intrinsics`; for a whole image, use
+ :func:`~kornia.geometry.calibration.undistort_image`.
Radial :math:`(k_1, k_2, k_3, k_4, k_5, k_6)`,
tangential :math:`(p_1, p_2)`, thin prism :math:`(s_1, s_2, s_3, s_4)`, and tilt :math:`(\tau_x, \tau_y)`
@@ -49,7 +61,11 @@ def undistort_points(
- the inverse is a fixed-point iteration of ``num_iters`` steps, so the round trip through
:func:`~kornia.geometry.calibration.distort_points` closes only to the accuracy the iteration reaches;
within its convergence region more steps improve it down to the dtype's rounding.
- - matching leading dimensions of ``points``, ``K`` and ``dist`` are preserved.
+ - matching leading dimensions of ``points``, ``K`` and ``dist`` are preserved, and the leading dimensions
+ broadcast as in :func:`~kornia.geometry.calibration.distort_points`, the sensor-tilt step of 14
+ coefficients included.
+ - the skew entries ``K[0, 1]`` and ``new_K[0, 1]`` are ignored, as in
+ :func:`~kornia.geometry.conversions.normalize_points_with_intrinsics`.
.. warning::
The iteration has no convergence test and no valid-radius guard: outside the convergence region it can
@@ -61,7 +77,7 @@ def undistort_points(
K: Intrinsic camera matrix with shape ``(*, 3, 3)``.
dist: Distortion coefficients
:math:`(k_1,k_2,p_1,p_2[,k_3[,k_4,k_5,k_6[,s_1,s_2,s_3,s_4[,\tau_x,\tau_y]]]])`. This is
- a vector with 4, 5, 8, 12 or 14 elements with shape :math:`(*, n)`.
+ a vector with 4, 5, 8, 12 or 14 elements with shape ``(*, n)``.
new_K: Intrinsic camera matrix used to map the undistorted normalized point back to pixels, while ``K``
is the one that maps the incoming distorted ``points`` from pixels onto the normalized
:math:`z = 1` plane -- the mirror image of the two roles in
@@ -73,6 +89,18 @@ def undistort_points(
Returns:
Undistorted 2D points with shape ``(*, N, 2)``.
+ See Also:
+ - :func:`~kornia.geometry.calibration.distort_points`: the forward map, in closed form, with ``K`` and
+ ``new_K`` in mirrored roles.
+ - :func:`~kornia.geometry.calibration.undistort_image`: undistorts every pixel of an image instead of a
+ point list.
+ - :func:`~kornia.geometry.calibration.tilt_projection`: the inverse sensor-tilt matrix applied first with
+ 14 coefficients.
+ - :func:`~kornia.geometry.camera.undistort_points_kannala_brandt`: the fisheye inverse, from pixels to
+ normalized points.
+ - :func:`~kornia.geometry.camera.undistort_points_affine`: the intrinsics alone, from pixels to normalized
+ points, in closed form.
+
Example:
>>> _ = torch.manual_seed(0)
>>> x = torch.rand(1, 4, 2)
@@ -165,7 +193,14 @@ def undistort_points(
# Based on https://github.com/opencv/opencv/blob/master/modules/calib3d/src/undistort.dispatch.cpp#L287
def undistort_image(image: torch.Tensor, K: torch.Tensor, dist: torch.Tensor) -> torch.Tensor:
- r"""Compensate an image for lens distortion.
+ r"""Undistort an image with OpenCV's Brown-Conrady coefficients, keeping its camera matrix K.
+
+ ``out = undistort_image(image, K, dist)`` takes images ``(*, C, H, W)``, the intrinsics ``(*, 3, 3)`` and
+ OpenCV's coefficient vector ``(*, n)``, ``n`` one of 4, 5, 8, 12 or 14, and returns the image a camera with the
+ same ``K`` and no distortion would take, ``(*, C, H, W)``: each output pixel samples the input bilinearly at its
+ :func:`~kornia.geometry.calibration.distort_points` position, with zeros outside the input. There is no
+ ``new_K``, so the result keeps the input's size and ``K``, as ``cv2.undistort`` does by default. For point
+ coordinates, use :func:`~kornia.geometry.calibration.undistort_points`.
Radial :math:`(k_1, k_2, k_3, k_4, k_5, k_6)`,
tangential :math:`(p_1, p_2)`, thin prism :math:`(s_1, s_2, s_3, s_4)`, and tilt :math:`(\tau_x, \tau_y)`
@@ -173,9 +208,10 @@ def undistort_image(image: torch.Tensor, K: torch.Tensor, dist: torch.Tensor) ->
Convention:
- the leading dimensions of ``image`` (in front of ``C, H, W``), ``K`` (in front of :math:`3 \times 3`)
- and ``dist`` (in front of its ``n`` coefficients) must match exactly; the one exception is the
- legacy unbatched call, a ``(1, C, H, W)`` image with a ``(3, 3)`` ``K`` and an :math:`(n,)`
- ``dist``.
+ and ``dist`` (in front of its ``n`` coefficients) must match exactly, so an unbatched ``(C, H, W)``
+ image takes a ``(3, 3)`` ``K`` and an ``(n,)`` ``dist``; the one exception is the legacy unbatched
+ call, a ``(1, C, H, W)`` image with a ``(3, 3)`` ``K`` and an ``(n,)`` ``dist``.
+ - an image that is not floating point raises :class:`ValueError`.
- the sampling map applies :func:`~kornia.geometry.calibration.distort_points` to the integer pixel
centres (see :class:`~kornia.geometry.camera.pinhole.PinholeCamera`) and is resampled bilinearly with
a fixed ``align_corners=True``, so all-zero coefficients return the input only to rounding.
@@ -185,11 +221,17 @@ def undistort_image(image: torch.Tensor, K: torch.Tensor, dist: torch.Tensor) ->
K: Intrinsic camera matrix with shape ``(*, 3, 3)``.
dist: Distortion coefficients
:math:`(k_1,k_2,p_1,p_2[,k_3[,k_4,k_5,k_6[,s_1,s_2,s_3,s_4[,\tau_x,\tau_y]]]])`. This is
- a vector with 4, 5, 8, 12 or 14 elements with shape :math:`(*, n)`.
+ a vector with 4, 5, 8, 12 or 14 elements with shape ``(*, n)``.
Returns:
Undistorted image with shape ``(*, C, H, W)``.
+ See Also:
+ - :func:`~kornia.geometry.calibration.undistort_points`: undistorts a list of pixel coordinates instead of
+ an image.
+ - :func:`~kornia.geometry.calibration.distort_points`: the map whose positions the image is sampled at.
+ - :func:`~kornia.geometry.transform.remap`: the sampling step, to resample with other maps or options.
+
Example:
>>> img = torch.rand(1, 3, 5, 5)
>>> K = torch.eye(3)[None]
diff --git a/kornia/geometry/camera/__init__.py b/kornia/geometry/camera/__init__.py
index b3c6c6594..47e6d36bd 100644
--- a/kornia/geometry/camera/__init__.py
+++ b/kornia/geometry/camera/__init__.py
@@ -15,9 +15,34 @@
# limitations under the License.
#
-"""Kornia Geometry Camera — Camera models and camera-related utilities for Kornia.
+"""Camera models: project 3D points to pixels and back, through lenses and stereo rigs.
-This subpackage provides camera models, distortion, and projection functions.
+Points are in the camera frame, +X right, +Y down and +Z forward, unless a function takes extrinsics. Pixels are
+``(u, v)`` = (column, row) with integer pixel centres, so a centred principal point is ``cx = (W - 1) / 2``.
+Normalized coordinates ``(x / z, y / z)`` are where a camera ray meets the ``z = 1`` plane. Intrinsics come as a
+``(*, 3, 3)`` ``K``, as a flat ``[fx, fy, cx, cy, ...]`` vector, or, in :class:`PinholeCamera`, as ``(B, 4, 4)``
+matrices next to world-to-camera extrinsics. Depth is the camera-frame ``z`` unless a ``normalize`` flag reads it as
+the Euclidean ray length. The camera-conventions page of the documentation compares these with other libraries.
+
+Which function to use:
+
+- Camera-frame points to pixels with a ``3x3`` ``K``: :func:`project_points`; pixels at given depths back to
+ camera-frame points: :func:`unproject_points`.
+- World points to pixels and back through cameras with extrinsics: :class:`PinholeCamera`. A depth map to
+ camera-frame points, and those to another camera's pixels, with its ``(B, 4, 4)`` matrices: :func:`pixel2cam`
+ and :func:`cam2pixel`.
+- Camera-frame points to the image plane and back, without intrinsics, to compose a camera model:
+ :func:`project_points_z1` and :func:`unproject_points_z1` (perspective, onto the normalized ``z = 1`` plane),
+ :func:`project_points_orthographic` and :func:`unproject_points_orthographic` (dropping ``z``); their derivatives
+ :func:`dx_project_points_z1` (the ``(..., 2, 3)`` Jacobian) and :func:`dx_project_points_orthographic`
+ (``du/dx`` alone).
+- Normalized points to pixels and back with a flat parameter vector: :func:`distort_points_affine` and
+ :func:`undistort_points_affine` (``[fx, fy, cx, cy]``, no lens distortion), :func:`distort_points_kannala_brandt`
+ and :func:`undistort_points_kannala_brandt` (fisheye, eight parameters); their Jacobians
+ :func:`dx_distort_points_affine` and :func:`dx_distort_points_kannala_brandt`. For OpenCV's Brown-Conrady
+ coefficients on pixel coordinates, use :mod:`kornia.geometry.calibration`.
+- 3D points from a rectified stereo disparity map: :class:`StereoCamera`, from the two projection matrices, or
+ :func:`reproject_disparity_to_3D`, from a ``Q`` matrix.
"""
from .distortion_affine import distort_points_affine, dx_distort_points_affine, undistort_points_affine
diff --git a/kornia/geometry/camera/distortion_affine.py b/kornia/geometry/camera/distortion_affine.py
index 418d6baba..a5c23775e 100644
--- a/kornia/geometry/camera/distortion_affine.py
+++ b/kornia/geometry/camera/distortion_affine.py
@@ -15,7 +15,7 @@
# limitations under the License.
#
-"""nn.Module containing the affine distortion model."""
+"""The affine camera model: flat [fx, fy, cx, cy] intrinsics between normalized z = 1 points and pixels."""
# inspired by: https://github.com/farm-ng/sophus-rs/blob/main/src/sensor/affine.rs
import torch
@@ -24,7 +24,13 @@ from kornia.core.check import KORNIA_CHECK_SHAPE
def distort_points_affine(projected_points_in_camera_z1_plane: torch.Tensor, params: torch.Tensor) -> torch.Tensor:
- r"""Distort one or more points from the canonical z=1 plane into the camera frame.
+ r"""Apply flat [fx, fy, cx, cy] intrinsics to normalized z = 1 points, giving pixels.
+
+ ``uv = distort_points_affine(xy, params)`` takes points on the normalized ``z = 1`` plane ``(..., 2)`` and the
+ vector ``params = [fx, fy, cx, cy]``, ``(..., 4)``, and returns the pixels ``(fx x + cx, fy y + cy)`` as
+ ``(..., 2)``. Despite its name it adds no lens distortion: it is the intrinsics step of a camera model, the
+ one :func:`distort_points_kannala_brandt` applies after its fisheye distortion. For a ``(*, 3, 3)`` ``K``, use
+ :func:`~kornia.geometry.conversions.denormalize_points_with_intrinsics` instead.
.. math::
\begin{bmatrix} u \\ v \end{bmatrix} =
@@ -39,19 +45,32 @@ def distort_points_affine(projected_points_in_camera_z1_plane: torch.Tensor, par
- :func:`~kornia.geometry.calibration.distort_points` is the same camera parametrized by a
``(*, 3, 3)`` ``K`` and an OpenCV coefficient vector; it takes a **pixel** input instead.
- :func:`undistort_points_affine` is the closed-form inverse.
+ - the leading dimensions ``...`` of the points and of ``params`` broadcast by torch's rules: one ``(4,)``
+ vector serves every point, and per-camera ``params`` for ``(B, N, 2)`` points are ``(B, 1, 4)``, since
+ a ``(B, 4)`` vector lines up with the ``N`` axis.
Args:
- projected_points_in_camera_z1_plane: torch.Tensor representing the points to distort with shape (..., 2).
- params: torch.Tensor representing the parameters of the affine distortion model with shape (..., 4).
+ projected_points_in_camera_z1_plane: torch.Tensor representing the points to distort with shape
+ ``(..., 2)``.
+ params: torch.Tensor representing the parameters of the affine distortion model with shape ``(..., 4)``.
Returns:
- torch.Tensor representing the distorted points with shape (..., 2).
+ torch.Tensor representing the distorted points with shape ``(..., 2)``.
+
+ See Also:
+ - :func:`undistort_points_affine`: the closed-form inverse, pixels to normalized points.
+ - :func:`dx_distort_points_affine`: its ``(..., 2, 2)`` Jacobian with respect to the point.
+ - :func:`distort_points_kannala_brandt`: the same intrinsics after a fisheye distortion of the ray angle.
+ - :func:`~kornia.geometry.calibration.distort_points`: OpenCV's Brown-Conrady distortion with a ``3x3``
+ ``K``, from pixels to pixels.
+ - :func:`~kornia.geometry.conversions.denormalize_points_with_intrinsics`: the same map from a
+ ``(*, 3, 3)`` ``K``.
Example:
- >>> points = torch.tensor([319.5, 239.5]) # center of a 640x480 image
+ >>> points = torch.tensor([0.1, -0.2]) # on the normalized z = 1 plane
>>> params = torch.tensor([600., 600., 319.5, 239.5])
>>> distort_points_affine(points, params)
- tensor([192019.5000, 143939.5000])
+ tensor([379.5000, 119.5000])
"""
KORNIA_CHECK_SHAPE(projected_points_in_camera_z1_plane, ["*", "2"])
@@ -70,7 +89,12 @@ def distort_points_affine(projected_points_in_camera_z1_plane: torch.Tensor, par
def undistort_points_affine(distorted_points_in_camera: torch.Tensor, params: torch.Tensor) -> torch.Tensor:
- r"""Undistort one or more points from the camera frame into the canonical z=1 plane.
+ r"""Map pixels back to normalized z = 1 points with flat [fx, fy, cx, cy] intrinsics.
+
+ ``xy = undistort_points_affine(uv, params)`` takes pixels ``(..., 2)`` and the ``(..., 4)`` vector
+ ``[fx, fy, cx, cy]`` and returns ``((u - cx) / fx, (v - cy) / fy)`` as ``(..., 2)``, on the normalized
+ ``z = 1`` plane; it removes no lens distortion. For a ``(*, 3, 3)`` ``K``, use
+ :func:`~kornia.geometry.conversions.normalize_points_with_intrinsics` instead.
.. math::
\begin{bmatrix} x \\ y \end{bmatrix} =
@@ -84,13 +108,23 @@ def undistort_points_affine(distorted_points_in_camera: torch.Tensor, params: to
:func:`distort_points_affine` takes, and this function is its closed-form inverse.
- ``params`` values are not validated (a design choice shared with the other intrinsics consumers): a
zero ``fx`` or ``fy`` gives ``inf`` or ``nan`` on that axis.
+ - the leading dimensions of the points and of ``params`` broadcast as in :func:`distort_points_affine`.
Args:
- distorted_points_in_camera: torch.Tensor representing the points to undistort with shape (..., 2).
- params: torch.Tensor representing the parameters of the affine distortion model with shape (..., 4).
+ distorted_points_in_camera: torch.Tensor representing the points to undistort with shape ``(..., 2)``.
+ params: torch.Tensor representing the parameters of the affine distortion model with shape ``(..., 4)``.
Returns:
- torch.Tensor representing the undistorted points with shape (..., 2).
+ torch.Tensor representing the undistorted points with shape ``(..., 2)``.
+
+ See Also:
+ - :func:`distort_points_affine`: the forward map, normalized points to pixels.
+ - :func:`dx_distort_points_affine`: the Jacobian of the forward map.
+ - :func:`undistort_points_kannala_brandt`: the inverse of the fisheye model, found iteratively.
+ - :func:`~kornia.geometry.calibration.undistort_points`: removes OpenCV's Brown-Conrady distortion, from
+ pixels to pixels.
+ - :func:`~kornia.geometry.conversions.normalize_points_with_intrinsics`: the same map from a
+ ``(*, 3, 3)`` ``K``.
Example:
>>> points = torch.tensor([319.5, 239.5]) # center of a 640x480 image
@@ -115,7 +149,11 @@ def undistort_points_affine(distorted_points_in_camera: torch.Tensor, params: to
def dx_distort_points_affine(projected_points_in_camera_z1_plane: torch.Tensor, params: torch.Tensor) -> torch.Tensor:
- r"""Compute the Jacobian of the affine distortion with respect to the point.
+ r"""Compute the 2x2 Jacobian of the affine [fx, fy, cx, cy] model with respect to the point.
+
+ ``J = dx_distort_points_affine(xy, params)`` takes normalized points ``(..., 2)`` and the ``(..., 4)`` vector
+ ``[fx, fy, cx, cy]`` and returns the Jacobian of :func:`distort_points_affine` at each point, ``diag(fx, fy)``,
+ as ``(..., 2, 2)``. Chain it after :func:`dx_project_points_z1` for the Jacobian of a pinhole camera model.
.. math::
\frac{\partial (u, v)}{\partial (x, y)} =
@@ -129,15 +167,23 @@ def dx_distort_points_affine(projected_points_in_camera_z1_plane: torch.Tensor,
:func:`distort_points_affine`: ``N`` points with one set of params give ``N`` Jacobians.
Args:
- projected_points_in_camera_z1_plane: torch.Tensor representing the points to distort with shape (..., 2).
- params: torch.Tensor representing the parameters of the affine distortion model with shape (..., 4).
+ projected_points_in_camera_z1_plane: torch.Tensor representing the points to distort with shape
+ ``(..., 2)``.
+ params: torch.Tensor representing the parameters of the affine distortion model with shape ``(..., 4)``.
Returns:
torch.Tensor representing the Jacobian of the distortion with respect to the point
- with shape (..., 2, 2).
+ with shape ``(..., 2, 2)``.
+
+ See Also:
+ - :func:`distort_points_affine`: the map this differentiates.
+ - :func:`undistort_points_affine`: the inverse map; its Jacobian is ``diag(1 / fx, 1 / fy)``.
+ - :func:`dx_distort_points_kannala_brandt`: the Jacobian of the fisheye model, equal to this one at the
+ optical axis.
+ - :func:`dx_project_points_z1`: the Jacobian of the projection step before it.
Example:
- >>> points = torch.tensor([319.5, 239.5]) # center of a 640x480 image
+ >>> points = torch.tensor([0.1, -0.2]) # on the normalized z = 1 plane
>>> params = torch.tensor([600., 600., 319.5, 239.5])
>>> dx_distort_points_affine(points, params)
tensor([[600., 0.],
diff --git a/kornia/geometry/camera/distortion_kannala_brandt.py b/kornia/geometry/camera/distortion_kannala_brandt.py
index bbdd2c74a..33032228f 100644
--- a/kornia/geometry/camera/distortion_kannala_brandt.py
+++ b/kornia/geometry/camera/distortion_kannala_brandt.py
@@ -17,6 +17,8 @@
# inspired by: shttps://github.com/farm-ng/sophus-rs/blob/main/src/sensor/kannala_brandt.rs
+"""The Kannala-Brandt fisheye camera model between normalized z = 1 points and pixels, its inverse and Jacobian."""
+
import torch
from kornia.core.check import KORNIA_CHECK_SHAPE
@@ -67,7 +69,15 @@ def _distort_points_kannala_brandt_impl(
def distort_points_kannala_brandt(
projected_points_in_camera_z1_plane: torch.Tensor, params: torch.Tensor
) -> torch.Tensor:
- r"""Distort points from the canonical z=1 plane into the camera frame using the Kannala-Brandt model.
+ r"""Map normalized z = 1 points to fisheye pixels with the Kannala-Brandt model.
+
+ ``uv = distort_points_kannala_brandt(xy, params)`` takes points on the normalized ``z = 1`` plane ``(..., 2)``
+ and the vector ``params = [fx, fy, cx, cy, k0, k1, k2, k3]``, ``(..., 8)``, and returns their pixels
+ ``(..., 2)``: the angle :math:`\theta = \arctan r` of each point's ray from the optical axis, with ``r`` its
+ distance from the axis, becomes the radius
+ :math:`\theta (1 + k_0 \theta^2 + k_1 \theta^4 + k_2 \theta^6 + k_3 \theta^8)` before the intrinsics of
+ :func:`distort_points_affine`, as OpenCV's ``cv2.fisheye.distortPoints`` with zero skew. For OpenCV's
+ Brown-Conrady coefficients on pixel coordinates, use :func:`~kornia.geometry.calibration.distort_points`.
Convention:
- ``projected_points_in_camera_z1_plane`` is a point on the **normalized** :math:`z = 1` plane, not a
@@ -78,19 +88,31 @@ def distort_points_kannala_brandt(
``k3`` multiply :math:`\theta^2`, :math:`\theta^4`, :math:`\theta^6` and :math:`\theta^8` in the
fish-eye polynomial.
- :func:`undistort_points_kannala_brandt` is the inverse map.
+ - a point with squared radius ``<= 1e-8`` takes the affine branch, ``(fx x + cx, fy y + cy)``.
+ - the leading dimensions of the points and of ``params`` broadcast as in :func:`distort_points_affine`:
+ per-camera ``params`` for ``(B, N, 2)`` points are ``(B, 1, 8)``.
Args:
- projected_points_in_camera_z1_plane: torch.Tensor representing the points to distort with shape (..., 2).
- params: torch.Tensor representing the parameters of the Kannala-Brandt distortion model with shape (..., 8).
+ projected_points_in_camera_z1_plane: torch.Tensor representing the points to distort with shape
+ ``(..., 2)``.
+ params: torch.Tensor representing the parameters of the Kannala-Brandt distortion model with shape
+ ``(..., 8)``.
Returns:
- torch.Tensor representing the distorted points with shape (..., 2).
+ torch.Tensor representing the distorted points with shape ``(..., 2)``.
+
+ See Also:
+ - :func:`undistort_points_kannala_brandt`: the inverse, pixels to normalized points.
+ - :func:`dx_distort_points_kannala_brandt`: its ``(..., 2, 2)`` Jacobian with respect to the point.
+ - :func:`distort_points_affine`: the intrinsics alone, with no distortion.
+ - :func:`~kornia.geometry.calibration.distort_points`: OpenCV's Brown-Conrady distortion with a ``3x3``
+ ``K``, from pixels to pixels.
Example:
- >>> points = torch.tensor([319.5, 239.5]) # center of a 640x480 image
+ >>> points = torch.tensor([0.1, -0.2]) # on the normalized z = 1 plane
>>> params = torch.tensor([1000.0, 1000.0, 320.0, 280.0, 0.1, 0.01, 0.001, 0.0001])
>>> distort_points_kannala_brandt(points, params)
- tensor([1982.6832, 1526.3619])
+ tensor([418.8600, 82.2799])
"""
KORNIA_CHECK_SHAPE(projected_points_in_camera_z1_plane, ["*", "2"])
@@ -115,23 +137,45 @@ def distort_points_kannala_brandt(
def undistort_points_kannala_brandt(distorted_points_in_camera: torch.Tensor, params: torch.Tensor) -> torch.Tensor:
- r"""Undistort points from the camera frame into the canonical z=1 plane using the Kannala-Brandt model.
+ r"""Map fisheye pixels back to normalized z = 1 points by inverting the Kannala-Brandt model.
+
+ ``xy = undistort_points_kannala_brandt(uv, params)`` takes pixels ``(..., 2)`` and the ``(..., 8)`` vector of
+ :func:`distort_points_kannala_brandt` and returns the normalized points ``(..., 2)`` that it maps to them,
+ solving for the ray angle iteratively. The result matches OpenCV's ``cv2.fisheye.undistortPoints`` where the
+ distorted angle, the distance of the normalized distorted point from the axis, is below :math:`\pi / 2`, the
+ polynomial increases up to the solution, the solved ray angle is below :math:`\pi / 2` too, and both
+ iterations converge within their ten steps. Where the first two fail, or OpenCV's iteration does not converge,
+ OpenCV clamps the distorted angle to :math:`\pi / 2` or returns ``(-1e6, -1e6)`` for a point it cannot invert,
+ while this function returns its last iterate, unflagged. Where only the solved angle is past
+ :math:`\pi / 2`, the ray points behind the camera: OpenCV's signed tangent puts the point on the far side of
+ the centre, and this function's absolute tangent keeps it on the side of the pixel. For
+ OpenCV's Brown-Conrady coefficients on pixel coordinates, use
+ :func:`~kornia.geometry.calibration.undistort_points`.
Convention:
- ``distorted_points_in_camera`` is a **pixel** coordinate and the result is a point on the normalized
:math:`z = 1` plane; ``params`` is the same 8-element vector documented on
:func:`distort_points_kannala_brandt`.
- - the inverse is a fixed number of Gauss-Newton steps with no convergence test and no caller-visible
- step count, so the round trip through :func:`distort_points_kannala_brandt` closes only to the
- accuracy that iteration reaches.
+ - the inverse is a fixed number of Gauss-Newton steps (10) with no convergence test and no
+ caller-visible step count, so the round trip through :func:`distort_points_kannala_brandt` closes only
+ to the accuracy that iteration reaches.
- a zero distorted radius maps to the origin.
+ - the iteration runs in the dtype of ``params`` and the result takes the dtype of the points.
Args:
- distorted_points_in_camera: torch.Tensor representing the points to undistort with shape (..., 2).
- params: torch.Tensor representing the parameters of the Kannala-Brandt distortion model with shape (..., 8).
+ distorted_points_in_camera: torch.Tensor representing the points to undistort with shape ``(..., 2)``.
+ params: torch.Tensor representing the parameters of the Kannala-Brandt distortion model with shape
+ ``(..., 8)``.
Returns:
- torch.Tensor representing the undistorted points with shape (..., 2).
+ torch.Tensor representing the undistorted points with shape ``(..., 2)``.
+
+ See Also:
+ - :func:`distort_points_kannala_brandt`: the forward map, normalized points to pixels.
+ - :func:`dx_distort_points_kannala_brandt`: the Jacobian of the forward map.
+ - :func:`undistort_points_affine`: the closed-form inverse of the intrinsics alone.
+ - :func:`~kornia.geometry.calibration.undistort_points`: removes OpenCV's Brown-Conrady distortion, from
+ pixels to pixels.
Example:
>>> points = torch.tensor([319.5, 239.5]) # center of a 640x480 image
@@ -200,7 +244,11 @@ def undistort_points_kannala_brandt(distorted_points_in_camera: torch.Tensor, pa
def dx_distort_points_kannala_brandt(
projected_points_in_camera_z1_plane: torch.Tensor, params: torch.Tensor
) -> torch.Tensor:
- r"""Return the analytic Jacobian of the Kannala-Brandt distortion model.
+ r"""Compute the 2x2 Jacobian of the Kannala-Brandt model with respect to the point.
+
+ ``J = dx_distort_points_kannala_brandt(xy, params)`` takes normalized points ``(..., 2)`` and the ``(..., 8)``
+ vector of :func:`distort_points_kannala_brandt` and returns the Jacobian of that map at each point as
+ ``(..., 2, 2)``. Chain it after :func:`dx_project_points_z1` for the Jacobian of a fisheye camera model.
Convention:
- the result has shape ``(..., 2, 2)`` and is laid out like the Jacobian that
@@ -208,14 +256,23 @@ def dx_distort_points_kannala_brandt(
``(u, v)``, columns the input components ``(x, y)``.
- for squared radii ``<= 1e-8`` the forward function uses its affine branch, so the Jacobian there is
``diag(fx, fy)``.
+ - float16 and bfloat16 inputs are computed in float32 and rounded once to the promoted input dtype.
Args:
- projected_points_in_camera_z1_plane: torch.Tensor representing the points to distort with shape (..., 2).
- params: torch.Tensor representing the parameters of the Kannala-Brandt distortion model with shape (..., 8).
+ projected_points_in_camera_z1_plane: torch.Tensor representing the points to distort with shape
+ ``(..., 2)``.
+ params: torch.Tensor representing the parameters of the Kannala-Brandt distortion model with shape
+ ``(..., 8)``.
Returns:
torch.Tensor representing the derivative of the distortion with respect to the point
- with shape (..., 2, 2).
+ with shape ``(..., 2, 2)``.
+
+ See Also:
+ - :func:`distort_points_kannala_brandt`: the map this differentiates.
+ - :func:`undistort_points_kannala_brandt`: the inverse map.
+ - :func:`dx_distort_points_affine`: the Jacobian of the intrinsics alone, ``diag(fx, fy)``.
+ - :func:`dx_project_points_z1`: the Jacobian of the projection step before it.
Example:
>>> points = torch.tensor([1., 2.])
diff --git a/kornia/geometry/camera/perspective.py b/kornia/geometry/camera/perspective.py
index 2d9d25377..93b7520a7 100644
--- a/kornia/geometry/camera/perspective.py
+++ b/kornia/geometry/camera/perspective.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
+"""Perspective projection of camera-frame points to pixels with a 3x3 intrinsics matrix, and back at given depths."""
+
import torch
import torch.nn.functional as F
@@ -27,14 +29,23 @@ from kornia.geometry.conversions import (
def project_points(point_3d: torch.Tensor, camera_matrix: torch.Tensor) -> torch.Tensor:
- r"""Project a 3d point onto the 2d camera plane.
+ r"""Project camera-frame 3D points to pixels with a 3x3 intrinsics matrix K.
- See the Convention block on :class:`~kornia.geometry.camera.pinhole.PinholeCamera`.
+ ``uv = project_points(points, K)`` takes camera-frame points ``(*, 3)``, with at least two dimensions, and the
+ intrinsics ``(*, 3, 3)``, and returns their pixels ``(fx x / z + cx, fy y / z + cy)`` as ``(*, 2)``. ``K``
+ broadcasts against the points as in :func:`~kornia.geometry.conversions.denormalize_points_with_intrinsics`:
+ a ``(B, 3, 3)`` ``K`` pairs camera ``b`` with ``points[b]`` of ``(B, 3)`` or ``(B, N, 3)`` points; for
+ ``(B, H, W, 3)`` points pass ``K[:, None, None]``. For world points, project through the extrinsics with
+ :meth:`~kornia.geometry.camera.pinhole.PinholeCamera.project`; for a lens with Brown-Conrady distortion, pass
+ the result to :func:`~kornia.geometry.calibration.distort_points`.
Convention:
- ``point_3d`` is a **camera-frame** point and ``camera_matrix`` the ``(*, 3, 3)`` ``K``; the
- function takes no extrinsics, so it does not move between frames.
+ function takes no extrinsics, so it does not move between frames. Pixel centres are integers; see the
+ Convention block on :class:`~kornia.geometry.camera.pinhole.PinholeCamera`.
- the input must be at least rank 2: an unbatched ``(3,)`` point raises :class:`ValueError`.
+ - the skew entry ``K[0, 1]`` is ignored, as in
+ :func:`~kornia.geometry.conversions.denormalize_points_with_intrinsics`.
.. warning::
At ``abs(z) <= 1e-8`` the divide is skipped and ``K`` is applied to the undivided point, giving the
@@ -49,7 +60,16 @@ def project_points(point_3d: torch.Tensor, camera_matrix: torch.Tensor) -> torch
matrix. The tensor shape must be ``(*, 3, 3)``.
Returns:
- tensor of (u, v) cam coordinates with shape ``(*, 2)``.
+ tensor of (u, v) pixel coordinates with shape ``(*, 2)``.
+
+ See Also:
+ - :func:`unproject_points`: the reverse, pixels at given depths to camera-frame points.
+ - :meth:`~kornia.geometry.camera.pinhole.PinholeCamera.project`: world points, through ``(B, 4, 4)``
+ intrinsics and extrinsics.
+ - :func:`~kornia.geometry.camera.project_points_z1`: the divide alone, to the normalized ``z = 1`` plane
+ without intrinsics, with a plain divide at ``z = 0``.
+ - :func:`~kornia.geometry.camera.project_points_orthographic`: drops ``z`` instead of dividing by it.
+ - :func:`~kornia.geometry.calibration.distort_points`: adds Brown-Conrady lens distortion to the pixels.
Example:
>>> _ = torch.manual_seed(0)
@@ -70,18 +90,26 @@ def project_points(point_3d: torch.Tensor, camera_matrix: torch.Tensor) -> torch
def unproject_points(
point_2d: torch.Tensor, depth: torch.Tensor, camera_matrix: torch.Tensor, normalize: bool = False
) -> torch.Tensor:
- r"""Unproject a 2d point in 3d.
+ r"""Unproject pixels at given depths to camera-frame 3D points with a 3x3 matrix K.
- Transform coordinates in the pixel frame to the camera frame.
-
- See the Convention block on :class:`~kornia.geometry.camera.pinhole.PinholeCamera`.
+ ``points = unproject_points(uv, depth, K)`` takes pixels ``(*, 2)``, with at least two dimensions, their depths
+ ``(*, 1)`` and the intrinsics ``(*, 3, 3)``, and returns the camera-frame points
+ ``((u - cx) d / fx, (v - cy) d / fy, d)`` as ``(*, 3)``. ``K`` broadcasts against the pixels as in
+ :func:`~kornia.geometry.conversions.normalize_points_with_intrinsics`: a ``(B, 3, 3)`` ``K`` pairs camera ``b``
+ with ``uv[b]`` of ``(B, 2)`` or ``(B, N, 2)`` pixels; for ``(B, H, W, 2)`` pixels pass ``K[:, None, None]``.
+ For a whole depth map, use :func:`~kornia.geometry.depth.depth_to_3d_v2`; for world points,
+ :meth:`~kornia.geometry.camera.pinhole.PinholeCamera.unproject`.
Convention:
- ``point_2d`` is a pixel coordinate and the result is in the **camera** frame: the function takes no
extrinsics, so it cannot return world coordinates.
:meth:`~kornia.geometry.camera.pinhole.PinholeCamera.unproject` is the world-frame counterpart.
+ Pixel centres are integers; see the Convention block on
+ :class:`~kornia.geometry.camera.pinhole.PinholeCamera`.
- ``depth`` is the camera-frame ``z`` of the result. With ``normalize=True`` it is read as the
Euclidean ray length instead, so the result has that norm and a smaller ``z``.
+ - ``depth`` must be a tensor (else :class:`TypeError`) whose last dimension is 1 (else
+ :class:`ValueError`); its leading dimensions broadcast against the pixels'.
Args:
point_2d: tensor containing the 2d points to be projected to
@@ -97,6 +125,16 @@ def unproject_points(
Returns:
tensor of (x, y, z) camera coordinates with shape ``(*, 3)``.
+ See Also:
+ - :func:`project_points`: the reverse, camera-frame points to pixels.
+ - :meth:`~kornia.geometry.camera.pinhole.PinholeCamera.unproject`: world points, through ``(B, 4, 4)``
+ intrinsics and extrinsics.
+ - :func:`~kornia.geometry.depth.depth_to_3d_v2`: every pixel of a depth map, channels last.
+ - :func:`~kornia.geometry.camera.unproject_points_z1`: from the normalized ``z = 1`` plane, without
+ intrinsics.
+ - :func:`~kornia.geometry.calibration.undistort_points`: removes Brown-Conrady lens distortion from the
+ pixels, to call first.
+
Example:
>>> _ = torch.manual_seed(0)
>>> x = torch.rand(1, 2)
diff --git a/kornia/geometry/camera/pinhole.py b/kornia/geometry/camera/pinhole.py
index 34036fc6d..1dad92a78 100644
--- a/kornia/geometry/camera/pinhole.py
+++ b/kornia/geometry/camera/pinhole.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
+"""Pinhole cameras with extrinsics: project world points to pixels and back, and the dense helpers that use them."""
+
import warnings
from typing import Iterable, List, Union
@@ -35,15 +37,27 @@ def _embed_intrinsics(camera_matrix: torch.Tensor) -> torch.Tensor:
class PinholeCamera:
- r"""Class that represents a Pinhole Camera model.
+ r"""Model a batch of pinhole cameras with extrinsics: world points to pixels and back.
+
+ ``cam = PinholeCamera(intrinsics, extrinsics, height, width)`` stores ``B`` cameras: ``(B, 4, 4)`` intrinsics,
+ ``(B, 4, 4)`` world-to-camera extrinsics and the ``(B,)`` image height and width. ``uv = cam.project(points)``
+ takes world points ``(B, N, 3)``, camera ``b`` projecting ``points[b]``, and returns their pixels
+ ``(B, N, 2)``; ``points = cam.unproject(uv, depth)`` takes pixels ``(B, N, 2)`` and their camera-frame depths
+ ``(B, N, 1)`` and returns world points ``(B, N, 3)``. For camera-frame points and a ``3x3`` ``K`` without
+ extrinsics, use :func:`~kornia.geometry.camera.perspective.project_points` and
+ :func:`~kornia.geometry.camera.perspective.unproject_points` instead.
Convention:
- ``intrinsics`` is the ``(B, 4, 4)`` calibration matrix whose top-left :math:`3 \times 3` block is
``[[fx, 0, cx], [0, fy, cy], [0, 0, 1]]``, and ``extrinsics`` the ``(B, 4, 4)`` **world-to-camera**
- transform ``[R | t]`` (OpenCV / COLMAP semantics): :meth:`project` takes **world** points, computes
- ``K (R X + t)`` and returns pixels, while :meth:`unproject` takes pixels and a camera-frame depth and
- returns **world** points. The functional API takes a ``3x3`` ``K`` and no extrinsics, so it works in
- the **camera** frame.
+ transform ``[R | t]`` (OpenCV / COLMAP semantics): :meth:`project` computes ``K (R X + t)`` of
+ **world** points, and :meth:`unproject` returns **world** points. The functional API takes a ``3x3``
+ ``K`` and no extrinsics, so it works in the **camera** frame.
+ - with ``B > 1`` cameras the points of :meth:`project`, ``(B, N, 3)`` or ``(B, H, W, 3)``, and the pixels
+ of :meth:`unproject`, ``(B, N, 2)`` or ``(B, H, W, 2)``, need a leading axis of size ``B`` and at least
+ three dimensions: ``(N, 3)`` and ``(1, N, 3)`` points, or ``(N, 2)`` and ``(1, N, 2)`` pixels, raise
+ :class:`ValueError`. A single camera (``B = 1``) takes any ``(*, N, 3)`` points or ``(*, N, 2)`` pixels
+ with at least two dimensions.
- pixel coordinates are ``(u, v)`` = (column, row) with **integer pixel centres**: pixel ``(0, 0)`` is
centred at ``(0, 0)``, as :func:`~kornia.geometry.grid.create_meshgrid` enumerates, so a centred image
has ``cx = (W - 1) / 2``, ``cy = (H - 1) / 2``. COLMAP's half-pixel convention reports the same
@@ -78,6 +92,19 @@ class PinholeCamera:
We assume that the class attributes are in batch form in order to take
advantage of PyTorch parallelism to boost computing performance.
+ See Also:
+ - :func:`~kornia.geometry.camera.perspective.project_points`: camera-frame points and a ``(*, 3, 3)`` ``K``,
+ with no extrinsics.
+ - :func:`~kornia.geometry.camera.perspective.unproject_points`: pixels at given depths to camera-frame
+ points with a ``(*, 3, 3)`` ``K``.
+ - :func:`pixel2cam`, :func:`cam2pixel`: the two halves of a dense reprojection, a depth map to camera-frame
+ points and those to another camera's pixels, with this class's ``(B, 4, 4)`` matrices.
+ - :class:`~kornia.geometry.depth.DepthWarper`: warps an image between two of these cameras by depth.
+ - :class:`~kornia.geometry.camera.stereo.StereoCamera`: a rectified stereo pair, from its two ``(B, 3, 4)``
+ projection matrices.
+ - :class:`~kornia.sensors.camera.CameraModel`: the experimental camera-model API, with lens distortion and
+ flat parameter vectors but no extrinsics.
+
"""
def __init__(
@@ -136,7 +163,7 @@ class PinholeCamera:
r"""Return the device for camera buffers.
Returns:
- Union[str, torch.device, None] type
+ the ``torch.device`` of the stored intrinsics.
"""
return self._intrinsics.device
@@ -257,7 +284,7 @@ class PinholeCamera:
@tz.setter
def tz(self, value: Union[torch.Tensor, float]) -> None:
- r"""Set the y-coordinate of the translation vector with the given value."""
+ r"""Set the z-coordinate of the translation vector with the given value."""
self.extrinsics[..., 2, -1] = value
@property
@@ -401,10 +428,10 @@ class PinholeCamera:
Args:
point_3d: torch.Tensor containing the 3d points to be projected
to the camera plane. The shape of the torch.Tensor can be ``(*, 3)`` with at least two
- dimensions.
+ dimensions; with ``B > 1`` cameras, ``(B, *, N, 3)``, camera ``b`` projecting ``point_3d[b]``.
Returns:
- torch.Tensor of (u, v) cam coordinates with shape ``(*, 2)``.
+ torch.Tensor of (u, v) pixel coordinates with shape ``(*, 2)``.
Example:
>>> _ = torch.manual_seed(0)
@@ -433,7 +460,7 @@ class PinholeCamera:
Args:
point_2d: torch.Tensor containing the 2d to be projected to
world coordinates. The shape of the torch.Tensor can be ``(*, 2)`` with at least two
- dimensions.
+ dimensions; with ``B > 1`` cameras, ``(B, *, N, 2)``, camera ``b`` unprojecting ``point_2d[b]``.
depth: torch.Tensor containing the depth value of each 2d
points. The torch.Tensor shape must be equal to point2d ``(*, 1)``.
@@ -628,25 +655,42 @@ class PinholeCamerasList(PinholeCamera):
def pixel2cam(depth: torch.Tensor, intrinsics_inv: torch.Tensor, pixel_coords: torch.Tensor) -> torch.Tensor:
- r"""Transform coordinates in the pixel frame to the camera frame.
+ r"""Unproject a depth map to camera-frame points through a 4x4 inverse intrinsics matrix.
- See the Convention block on :class:`~kornia.geometry.camera.pinhole.PinholeCamera`.
+ ``points = pixel2cam(depth, intrinsics_inv, pixel_coords)`` takes a ``(B, 1, H, W)`` depth map, the
+ ``(B, 4, 4)`` inverse intrinsics that :meth:`PinholeCamera.intrinsics_inverse` returns and the homogeneous
+ pixel grid ``(B, H, W, 3)`` of ``(u, v, 1)``, and returns the camera-frame point ``depth * K^-1 (u, v, 1)`` of
+ every pixel as ``(B, H, W, 3)``. It is the first half of the dense warp of
+ :class:`~kornia.geometry.depth.DepthWarper`; :func:`cam2pixel` is the second. For a ``3x3`` ``K``, use
+ :func:`~kornia.geometry.depth.depth_to_3d_v2`, which builds the pixel grid itself.
Convention:
- ``intrinsics_inv`` is a ``(B, 4, 4)`` inverse calibration matrix (the
:class:`~kornia.geometry.camera.pinhole.PinholeCamera` layout, not a ``3x3`` ``K``), and ``depth`` is
- the camera-frame ``z`` at each pixel of the ``(u, v, 1)`` grid.
- - a ``depth`` that is not ``Bx1xHxW``, an ``intrinsics_inv`` that is not ``(B, 4, 4)`` or a
- ``pixel_coords`` that is not ``BxHxWx3`` raises :class:`ValueError`; that ``pixel_coords`` matches
- the ``H`` and ``W`` of ``depth`` is not checked.
+ the camera-frame ``z`` at each pixel of the ``(u, v, 1)`` grid. Pixel centres are integers; see the
+ Convention block on :class:`~kornia.geometry.camera.pinhole.PinholeCamera`.
+ - a ``depth`` that is not ``(B, 1, H, W)``, an ``intrinsics_inv`` that is not ``(B, 4, 4)`` or a
+ ``pixel_coords`` that is not ``(B, H, W, 3)`` raises :class:`ValueError`. That ``pixel_coords`` matches
+ the ``H`` and ``W`` of ``depth`` is not checked up front: the two broadcast, so a mismatch raises a
+ broadcasting ``RuntimeError`` unless one of the sizes is 1.
+ - the batch of ``intrinsics_inv`` is that of ``pixel_coords`` or 1, and ``depth`` broadcasts against the
+ result: one camera and one grid serve a batch of depth maps.
Args:
- depth: the source depth maps. Shape must be Bx1xHxW.
- intrinsics_inv: the inverse intrinsics camera matrix. Shape must be Bx4x4.
- pixel_coords: the grid with (u, v, 1) pixel coordinates. Shape must be BxHxWx3.
+ depth: the source depth maps. Shape must be ``(B, 1, H, W)``.
+ intrinsics_inv: the inverse intrinsics camera matrix. Shape must be ``(B, 4, 4)``.
+ pixel_coords: the grid with (u, v, 1) pixel coordinates. Shape must be ``(B, H, W, 3)``.
Returns:
- torch.Tensor of shape BxHxWx3 with (x, y, z) cam coordinates.
+ torch.Tensor of shape ``(B, H, W, 3)`` with (x, y, z) cam coordinates.
+
+ See Also:
+ - :func:`cam2pixel`: the second half of the warp, camera-frame points to the pixels of a camera.
+ - :class:`PinholeCamera`: the camera model whose ``(B, 4, 4)`` matrices this function takes.
+ - :func:`~kornia.geometry.depth.depth_to_3d_v2`: the same points from a ``3x3`` ``K``, without a grid
+ to pass.
+ - :func:`~kornia.geometry.camera.perspective.unproject_points`: selected pixels at given depths, with a
+ ``3x3`` ``K``.
"""
if not (len(depth.shape) == 4 and depth.shape[1] == 1):
@@ -664,14 +708,22 @@ def pixel2cam(depth: torch.Tensor, intrinsics_inv: torch.Tensor, pixel_coords: t
def cam2pixel(cam_coords_src: torch.Tensor, dst_proj_src: torch.Tensor, eps: float = 1e-12) -> torch.Tensor:
- r"""Transform coordinates in the camera frame to the pixel frame.
+ r"""Project a map of camera-frame points to pixels through a 4x4 projection matrix.
- See the Convention block on :class:`~kornia.geometry.camera.pinhole.PinholeCamera`.
+ ``uv = cam2pixel(cam_coords_src, dst_proj_src)`` takes a ``(B, H, W, 3)`` map of points in the source camera
+ frame and the ``(B, 4, 4)`` projection ``dst_proj_src``, which moves a source-frame point into the destination
+ camera frame and applies the destination intrinsics, and returns their ``(u, v)`` pixel coordinates in the
+ destination camera, ``(B, H, W, 2)``. It is the second half of the dense warp of
+ :class:`~kornia.geometry.depth.DepthWarper`, after :func:`pixel2cam`. For points already in a camera's frame
+ and a ``3x3`` ``K``, use :func:`~kornia.geometry.camera.perspective.project_points`.
Convention:
- ``dst_proj_src`` is a ``(B, 4, 4)`` projection matrix (the
:class:`~kornia.geometry.camera.pinhole.PinholeCamera` layout, not a ``3x3`` ``K``), and the result is
- ``(u, v)`` pixel coordinates in the destination frame.
+ ``(u, v)`` pixel coordinates in the destination frame, with integer pixel centres; see the Convention
+ block on :class:`~kornia.geometry.camera.pinhole.PinholeCamera`.
+ - the batch of ``dst_proj_src`` is that of ``cam_coords_src`` or 1. A point behind the camera is divided
+ like any other and is not flagged.
.. warning::
The perspective division is ``x / (z + eps)`` rather than a guarded divide, so ``z = 0`` gives
@@ -680,13 +732,20 @@ def cam2pixel(cam_coords_src: torch.Tensor, dst_proj_src: torch.Tensor, eps: flo
`#4267 <https://github.com/kornia/kornia/issues/4267>`_.
Args:
- cam_coords_src: (x, y, z) coordinates defined in the first camera coordinates system. Shape must be BxHxWx3.
+ cam_coords_src: (x, y, z) coordinates defined in the first camera coordinates system. Shape must be
+ ``(B, H, W, 3)``.
dst_proj_src: the projection matrix between the
- reference and the non reference camera frame. Shape must be Bx4x4.
+ reference and the non reference camera frame. Shape must be ``(B, 4, 4)``.
eps: small value to avoid division by zero error.
Returns:
- torch.Tensor of shape BxHxWx2 with (u, v) pixel coordinates.
+ torch.Tensor of shape ``(B, H, W, 2)`` with (u, v) pixel coordinates.
+
+ See Also:
+ - :func:`pixel2cam`: the first half of the warp, a depth map to camera-frame points.
+ - :class:`PinholeCamera`: the camera model whose ``(B, 4, 4)`` matrices compose ``dst_proj_src``.
+ - :func:`~kornia.geometry.camera.perspective.project_points`: camera-frame points to pixels with a
+ ``3x3`` ``K``, skipping the divide at ``z = 0`` instead of adding ``eps``.
"""
if not (len(cam_coords_src.shape) == 4 and cam_coords_src.shape[3] == 3):
diff --git a/kornia/geometry/camera/projection_orthographic.py b/kornia/geometry/camera/projection_orthographic.py
index 5cb0f3bb9..1788f88c4 100644
--- a/kornia/geometry/camera/projection_orthographic.py
+++ b/kornia/geometry/camera/projection_orthographic.py
@@ -15,7 +15,7 @@
# limitations under the License.
#
-"""Functions for orthographic projection."""
+"""Orthographic projection of camera-frame points: drop z, append it back, and the derivative."""
# inspired by: https://github.com/farm-ng/sophus-rs/blob/main/src/sensor/ortho_camera.rs
import torch
@@ -24,7 +24,12 @@ from kornia.core.check import KORNIA_CHECK_SHAPE
def project_points_orthographic(points_in_camera: torch.Tensor) -> torch.Tensor:
- r"""Project camera-frame points orthographically onto the image plane by dropping z: (x, y, z) -> (x, y).
+ r"""Project camera-frame points orthographically by dropping z: (x, y, z) -> (x, y).
+
+ ``uv = project_points_orthographic(points)`` takes camera-frame points ``(..., 3)`` and returns their
+ ``(x, y)`` as ``(..., 2)``, in the units of the points: no division and no intrinsics. Map them to pixels with
+ :func:`distort_points_affine`. For a perspective camera, use :func:`project_points_z1`, or
+ :func:`~kornia.geometry.camera.perspective.project_points` for pixels from a ``3x3`` ``K``.
.. math::
\begin{bmatrix} u \\ v \end{bmatrix} =
@@ -32,10 +37,16 @@ def project_points_orthographic(points_in_camera: torch.Tensor) -> torch.Tensor:
Args:
- points_in_camera: torch.Tensor representing the points to project.
+ points_in_camera: torch.Tensor representing the points to project with shape ``(..., 3)``.
Returns:
- torch.Tensor representing the projected points.
+ torch.Tensor representing the projected points with shape ``(..., 2)``.
+
+ See Also:
+ - :func:`unproject_points_orthographic`: the reverse, appending a ``z``.
+ - :func:`dx_project_points_orthographic`: its derivative ``du/dx``.
+ - :func:`project_points_z1`: the perspective counterpart, dividing by ``z``.
+ - :func:`~kornia.geometry.camera.perspective.project_points`: perspective pixels from a ``3x3`` ``K``.
Example:
>>> points = torch.tensor([1., 2., 3.])
@@ -48,7 +59,11 @@ def project_points_orthographic(points_in_camera: torch.Tensor) -> torch.Tensor:
def unproject_points_orthographic(points_in_camera: torch.Tensor, extension: torch.Tensor) -> torch.Tensor:
- r"""Unproject orthographic image-plane points into the camera frame by appending a z value: (u, v) -> (u, v, z).
+ r"""Unproject orthographic points by appending a z value: (u, v) -> (u, v, z).
+
+ ``points = unproject_points_orthographic(uv, z)`` takes points ``(..., 2)`` and one ``z`` per point and returns
+ ``(u, v, z)`` as ``(..., 3)``: ``u`` and ``v`` are not scaled. To scale normalized coordinates by a depth, as a
+ perspective camera does, use :func:`unproject_points_z1` instead.
.. math::
\begin{bmatrix} x \\ y \\ z \end{bmatrix} =
@@ -57,13 +72,22 @@ def unproject_points_orthographic(points_in_camera: torch.Tensor, extension: tor
Convention:
- ``extension`` becomes the ``z`` component of the result; both a ``(N,)`` and a ``(N, 1)`` extension
are accepted.
+ - ``extension`` must have the points' leading dimensions exactly: it does not broadcast, unlike in
+ :func:`unproject_points_z1`, so a ``(1,)`` extension for ``(N, 2)`` points raises ``RuntimeError``.
+ A same-rank extension must have a trailing size of 1: an ``(N, 2)`` extension for ``(N, 2)`` points
+ raises :class:`~kornia.core.exceptions.ShapeError`.
Args:
- points_in_camera: torch.Tensor representing the points to unproject with shape (..., 2).
- extension: torch.Tensor representing the extension of the points to unproject with shape (..., 1).
+ points_in_camera: torch.Tensor representing the points to unproject with shape ``(..., 2)``.
+ extension: torch.Tensor representing the extension of the points to unproject with shape ``(..., 1)``
+ or ``(...,)``.
Returns:
- torch.Tensor representing the unprojected points with shape (..., 3).
+ torch.Tensor representing the unprojected points with shape ``(..., 3)``.
+
+ See Also:
+ - :func:`project_points_orthographic`: the reverse, dropping ``z``.
+ - :func:`unproject_points_z1`: the perspective counterpart, scaling ``x`` and ``y`` by the depth.
Example:
>>> points = torch.tensor([1., 2.])
@@ -82,16 +106,26 @@ def unproject_points_orthographic(points_in_camera: torch.Tensor, extension: tor
def dx_project_points_orthographic(points_in_camera: torch.Tensor) -> torch.Tensor:
- r"""Compute the derivative of the orthographic x projection with respect to x, which is always 1.
+ r"""Return the derivative du/dx of the orthographic projection: ones, one per point.
+
+ ``d = dx_project_points_orthographic(points)`` takes camera-frame points ``(..., 3)`` and returns the
+ derivative of ``u`` with respect to ``x`` at each, which is always 1, as ``(..., 1)``. It is not the full
+ Jacobian: that of the orthographic projection is the constant ``[[1, 0, 0], [0, 1, 0]]``, while
+ :func:`dx_project_points_z1` returns the full ``(..., 2, 3)`` Jacobian of the perspective projection.
.. math::
\frac{\partial u}{\partial x} = 1
Args:
- points_in_camera: torch.Tensor representing the points to project.
+ points_in_camera: torch.Tensor representing the points to project with shape ``(..., 3)``.
Returns:
- torch.Tensor representing the derivative of the x projection with respect to the x coordinate.
+ torch.Tensor representing the derivative of the x projection with respect to the x coordinate, with shape
+ ``(..., 1)``.
+
+ See Also:
+ - :func:`project_points_orthographic`: the projection this differentiates.
+ - :func:`dx_project_points_z1`: the full ``(..., 2, 3)`` Jacobian of the perspective projection.
Example:
>>> points = torch.tensor([1., 2., 3.])
diff --git a/kornia/geometry/camera/projection_z1.py b/kornia/geometry/camera/projection_z1.py
index 3c01d6305..3fa13a336 100644
--- a/kornia/geometry/camera/projection_z1.py
+++ b/kornia/geometry/camera/projection_z1.py
@@ -15,7 +15,7 @@
# limitations under the License.
#
-"""Functions for the perspective projection of points onto the canonical z=1 plane."""
+"""Perspective projection of camera-frame points onto the normalized z = 1 plane, its inverse and its Jacobian."""
# inspired by: https://github.com/farm-ng/sophus-rs/blob/main/src/sensor/perspective_camera.rs
from __future__ import annotations
@@ -28,7 +28,12 @@ from kornia.core.check import KORNIA_CHECK_SHAPE
def project_points_z1(points_in_camera: torch.Tensor) -> torch.Tensor:
- r"""Project one or more points from the camera frame into the canonical z=1 plane through perspective division.
+ r"""Divide camera-frame points by z onto the normalized z = 1 plane, without intrinsics.
+
+ ``xy = project_points_z1(points)`` takes camera-frame points ``(..., 3)`` and returns ``(x / z, y / z)`` as
+ ``(..., 2)``: normalized coordinates, not pixels. Map them to pixels with :func:`distort_points_affine`, or
+ through a fisheye lens with :func:`distort_points_kannala_brandt`; for pixels from a ``3x3`` ``K`` in one call,
+ use :func:`~kornia.geometry.camera.perspective.project_points` instead.
.. math::
@@ -54,10 +59,17 @@ def project_points_z1(points_in_camera: torch.Tensor) -> torch.Tensor:
`#4267 <https://github.com/kornia/kornia/issues/4267>`_.
Args:
- points_in_camera: torch.Tensor representing the points to project with shape (..., 3).
+ points_in_camera: torch.Tensor representing the points to project with shape ``(..., 3)``.
Returns:
- torch.Tensor representing the projected points with shape (..., 2).
+ torch.Tensor representing the projected points with shape ``(..., 2)``.
+
+ See Also:
+ - :func:`unproject_points_z1`: the reverse, scaling a normalized point by a depth.
+ - :func:`dx_project_points_z1`: the ``(..., 2, 3)`` Jacobian of this projection.
+ - :func:`project_points_orthographic`: drops ``z`` instead of dividing by it.
+ - :func:`~kornia.geometry.camera.perspective.project_points`: pixels from a ``3x3`` ``K`` in one call,
+ with the divide skipped at ``abs(z) <= 1e-8``.
Example:
>>> points = torch.tensor([1., 2., 3.])
@@ -72,7 +84,12 @@ def project_points_z1(points_in_camera: torch.Tensor) -> torch.Tensor:
def unproject_points_z1(
points_in_cam_canonical: torch.Tensor, extension: Optional[torch.Tensor] = None
) -> torch.Tensor:
- r"""Unproject points on the canonical z=1 plane into the camera frame by scaling them by a depth (default 1).
+ r"""Scale normalized z = 1 points by a depth into camera-frame points, z = 1 by default.
+
+ ``points = unproject_points_z1(xy, depth)`` takes normalized coordinates ``(..., 2)`` and their camera-frame
+ depths ``d``, ``(...,)`` or ``(..., 1)``, and returns ``(x d, y d, d)`` as ``(..., 3)``; without a depth it
+ appends ``z = 1``. For pixels and a ``3x3`` ``K``, use
+ :func:`~kornia.geometry.camera.perspective.unproject_points` instead.
.. math::
\begin{bmatrix} x \\ y \\ z \end{bmatrix} =
@@ -83,16 +100,24 @@ def unproject_points_z1(
by it and it becomes the third component.
:meth:`~kornia.sensors.camera.projection_model.Z1Projection.unproject` names the same argument
``depth``. Both ``(...,)`` and ``(..., 1)`` extensions are accepted, and their leading dimensions
- broadcast against the points'. One depth scales both coordinates: any other trailing size raises
- ``ShapeError``.
+ broadcast against the points'. An extension with one dimension fewer than the points is read as
+ ``(...,)``, any other as ``(..., 1)``: for ``(2, 5, 2)`` points a ``(5, 1)`` extension is read as
+ ``(...,)`` and raises, so pass ``(1, 5, 1)``. One depth scales both coordinates: any other trailing size
+ raises ``ShapeError``.
Args:
- points_in_cam_canonical: torch.Tensor representing the points to unproject with shape (..., 2).
+ points_in_cam_canonical: torch.Tensor representing the points to unproject with shape ``(..., 2)``.
extension: torch.Tensor representing the extension (depth) of the points to unproject with shape
- (..., 1) or (...), broadcastable against the points' leading dimensions. Defaults to unit depth.
+ ``(..., 1)`` or ``(...,)``, broadcastable against the points' leading dimensions. Defaults to unit
+ depth.
Returns:
- torch.Tensor representing the unprojected points with shape (..., 3).
+ torch.Tensor representing the unprojected points with shape ``(..., 3)``.
+
+ See Also:
+ - :func:`project_points_z1`: the reverse, dividing by ``z``.
+ - :func:`unproject_points_orthographic`: appends the depth without scaling ``x`` and ``y``.
+ - :func:`~kornia.geometry.camera.perspective.unproject_points`: from pixels, with a ``3x3`` ``K``.
Example:
>>> points = torch.tensor([1., 2.])
@@ -117,7 +142,13 @@ def unproject_points_z1(
def dx_project_points_z1(points_in_camera: torch.Tensor) -> torch.Tensor:
- r"""Compute the 2x3 Jacobian of the z=1 perspective projection with respect to the camera-frame point (x, y, z).
+ r"""Compute the 2x3 Jacobian of the z = 1 projection with respect to the camera-frame point.
+
+ ``J = dx_project_points_z1(points)`` takes camera-frame points ``(..., 3)`` and returns the Jacobian of
+ :func:`project_points_z1` at each, ``(..., 2, 3)``: the derivatives of ``(x / z, y / z)`` with respect to
+ ``(x, y, z)``. At ``z = 0`` its nonzero entries are infinite, or ``nan`` where ``x`` or ``y`` is 0. Chain it
+ with :func:`dx_distort_points_affine` or :func:`dx_distort_points_kannala_brandt` for the Jacobian of a whole
+ camera model.
.. math::
\frac{\partial \pi}{\partial x} =
@@ -138,11 +169,16 @@ def dx_project_points_z1(points_in_camera: torch.Tensor) -> torch.Tensor:
- the same ``z > 0`` precondition applies, and it is not validated either.
Args:
- points_in_camera: torch.Tensor representing the points to project with shape (..., 3).
+ points_in_camera: torch.Tensor representing the points to project with shape ``(..., 3)``.
Returns:
torch.Tensor representing the Jacobian of the projected point :math:`(u, v)` with respect to
- :math:`(x, y, z)`, with shape (..., 2, 3).
+ :math:`(x, y, z)`, with shape ``(..., 2, 3)``.
+
+ See Also:
+ - :func:`project_points_z1`: the projection this differentiates.
+ - :func:`dx_project_points_orthographic`: the orthographic counterpart, which returns ``du/dx`` alone.
+ - :func:`dx_distort_points_affine`: the Jacobian of the next step of a camera model, the intrinsics.
Example:
>>> points = torch.tensor([1., 2., 3.])
diff --git a/kornia/geometry/camera/stereo.py b/kornia/geometry/camera/stereo.py
index 4cadca32d..6a7a380bc 100644
--- a/kornia/geometry/camera/stereo.py
+++ b/kornia/geometry/camera/stereo.py
@@ -15,6 +15,25 @@
# limitations under the License.
#
+"""Rectified stereo: reproject disparity maps to 3D points, from the rig's projection matrices or from a Q matrix.
+
+A disparity map is channels-last, ``(B, H, W, 1)``, and holds ``d = u_left - u_right`` in pixels, positive for a
+point in front of the rig. Pixel ``(u, v)`` is (column, row) with integer pixel centres, as in
+``cv2.reprojectImageTo3D``, and the points are ``(B, H, W, 3)``.
+
+Which function to use:
+
+- A calibrated rig given by its two rectified ``(B, 3, 4)`` projection matrices (``P1`` and ``P2`` of
+ ``cv2.stereoRectify``): :class:`StereoCamera`, whose
+ :meth:`~kornia.geometry.camera.stereo.StereoCamera.reproject_disparity_to_3D` returns points in the left
+ rectified camera frame.
+- A ``(B, 4, 4)`` ``Q`` matrix from elsewhere: :func:`reproject_disparity_to_3D`.
+- The error both raise for the invalid rigs, disparity maps and ``Q`` matrices its docstring lists:
+ :class:`StereoException`. A rig or ``Q`` batch that is neither 1 nor the disparity batch raises ``ValueError``,
+ and left and right matrices with different batch sizes, other than a right batch of 1, raise ``RuntimeError``.
+- A depth map instead of points: :func:`~kornia.geometry.depth.depth_from_disparity`.
+"""
+
from typing import Any, Literal, overload
import torch
@@ -27,12 +46,27 @@ __all__ = ["StereoCamera", "StereoException", "reproject_disparity_to_3D"]
class StereoException(Exception):
- """Handle errors related to stereo camera calibration and rectification."""
+ """Signal an invalid stereo rig, disparity map or Q matrix, with a link to the stereo docs.
+
+ ``raise StereoException(msg)`` raises an exception whose message is ``msg`` followed by a link to the
+ :doc:`/geometry.camera.stereo` page. :class:`StereoCamera` raises it for projection matrices of the wrong
+ shape, on different devices or with different dtypes and, outside graph capture, for different intrinsics or a
+ baseline that is zero or points the wrong way; :func:`reproject_disparity_to_3D` raises it for a disparity map
+ or ``Q`` of the wrong type, shape or dtype. It derives from :class:`Exception` directly, so
+ ``except ValueError`` does not catch it; a ``Q`` batch that is neither 1 nor the disparity batch raises
+ :class:`ValueError` instead. Left and right projection matrices with different batch sizes, other than a right
+ batch of 1, raise a plain :class:`RuntimeError` from the constructor, not this exception.
+
+ See Also:
+ - :class:`StereoCamera`: the rig whose constructor checks raise it.
+ - :func:`reproject_disparity_to_3D`: the function whose input checks raise it.
+
+ """
def __init__(self, msg: str, *args: Any, **kwargs: Any) -> None:
- r"""Construct custom exception for the :module:`~kornia.geometry.camera.stereo` module.
+ r"""Construct custom exception for the :mod:`kornia.geometry.camera.stereo` module.
- Adds a general helper module redirecting the user to the proper documentation site.
+ Appends a link to the stereo documentation page to ``msg``.
Args:
msg: Custom message to add to the general message.
@@ -51,7 +85,18 @@ class StereoException(Exception):
class StereoCamera:
- """Represent a horizontal stereo camera setup for depth estimation.
+ """Model a rectified horizontal stereo rig and reproject its disparity maps to 3D points.
+
+ ``stereo = StereoCamera(P_left, P_right)`` takes the ``(B, 3, 4)`` rectified projection matrices of the left and
+ the right camera, such as ``P1`` and ``P2`` of ``cv2.stereoRectify``, and
+ ``points = stereo.reproject_disparity_to_3D(disparity)`` takes a channels-last ``(B, H, W, 1)`` disparity map
+ ``d = u_left - u_right``, in pixels, and returns the 3D point of every pixel in the left rectified camera
+ frame, ``(B, H, W, 3)``, in the units of the baseline: ``Z = fx * tx / d``. Where the homogeneous denominator
+ ``W`` satisfies ``abs(W) > 1e-8`` the result matches ``cv2.reprojectImageTo3D`` with OpenCV's ``Q``; elsewhere
+ a placeholder is returned. ``W = -fy * d`` on every rig the constructor accepts outside graph capture, so there
+ a disparity with ``abs(fy * d) <= 1e-8``, zero included, gets the placeholder (see the warning below). With a
+ ``Q`` matrix from elsewhere, call :func:`reproject_disparity_to_3D` instead; for a depth map rather than points,
+ :func:`~kornia.geometry.depth.depth_from_disparity`.
Convention:
- the two arguments are the **rectified projection matrices** of the left and the right camera, each of
@@ -68,21 +113,45 @@ class StereoCamera:
the result is ``X = (u - cx) Z / fx``, ``Y = (v - cy) Z / fy``, ``Z = fx * tx / disparity``.
- ``u`` is the **column** index and ``v`` the **row** index, as in ``cv2.reprojectImageTo3D``.
- a disparity map is channels-**last**, ``(B, H, W, 1)`` (a ``(B, 1, H, W)`` map is rejected),
- and the point cloud is ``(B, H, W, 3)``.
+ and the point cloud is ``(B, H, W, 3)``. On a rig with equal ``cx``, where ``abs(fy * d) > 1e-8``, a
+ positive disparity puts the point in front of the rig and a negative one behind it; a smaller disparity,
+ positive or not, gets the placeholder of the warning below. A rig with ``B = 1`` serves a batch of
+ disparity maps; any other batch must match the disparity batch, else :class:`ValueError`.
- pixel centres are integers; see the Convention block on
:class:`~kornia.geometry.camera.pinhole.PinholeCamera`.
+ - the constructor raises :class:`StereoException` for a matrix that is not ``(B, 3, 4)``, for matrices on
+ different devices or with different dtypes and, outside graph capture, for different intrinsics or a
+ zero or reversed baseline. Left and right matrices with different batch sizes raise a plain
+ :class:`RuntimeError` instead, except a right batch of 1, which is paired with every left matrix. The
+ method raises :class:`StereoException` for a disparity map that is not a floating ``(B, H, W, 1)``
+ tensor. An integer rig is accepted; its ``Q`` is in the default floating dtype.
.. warning::
A differing ``cx`` is **rejected**, although ``Q[3, 3] = fy * (cx_left - cx_right)`` exists for that
case, so it is zero on every rig the constructor accepts outside graph capture:
- `#4270 <https://github.com/kornia/kornia/issues/4270>`_. A zero disparity (a point at infinity) makes
- ``W = 0``; the divide is then skipped and a finite placeholder, behind the camera on a real rig, is
- returned. Request ``return_mask=True`` from :meth:`reproject_disparity_to_3D` to identify numerical
- reconstruction failures without changing those coordinates or their gradients.
+ `#4270 <https://github.com/kornia/kornia/issues/4270>`_. Under graph capture the check is skipped and
+ ``W = Q[3, 3] - fy * d``, so a nonzero disparity with ``fy * d = Q[3, 3]`` also makes ``W = 0``. On a
+ rig with equal ``cx``, a zero disparity (a point at infinity) makes ``W = -fy * d = 0``. Wherever
+ ``abs(W) <= 1e-8``, which includes small nonzero disparities such as ``d = 1e-9`` at ``fy = 1``, the divide
+ is skipped and a finite placeholder, behind the camera on a real rig, is returned, so ``d > 0`` alone does
+ not select points in front of the rig. Request
+ ``return_mask=True`` from :meth:`reproject_disparity_to_3D` to identify numerical reconstruction failures
+ without changing those coordinates or their gradients.
Args:
rectified_left_camera: The rectified left camera projection matrix of shape ``(B, 3, 4)``.
rectified_right_camera: The rectified right camera projection matrix of shape ``(B, 3, 4)``.
+
+ See Also:
+ - :func:`reproject_disparity_to_3D`: the same reprojection from a ``Q`` matrix instead of the two
+ projection matrices.
+ - :func:`~kornia.geometry.depth.depth_from_disparity`: a depth map, ``fx * tx / d`` at every pixel,
+ instead of points.
+ - :func:`~kornia.geometry.depth.depth_to_3d_v2`: points from a depth map; after
+ :func:`~kornia.geometry.depth.depth_from_disparity`, the same points up to the ``1e-8`` that function
+ adds to the disparity.
+ - :class:`~kornia.geometry.camera.pinhole.PinholeCamera`: a single camera with intrinsics and
+ world-to-camera extrinsics.
"""
def __init__(self, rectified_left_camera: torch.Tensor, rectified_right_camera: torch.Tensor) -> None:
@@ -417,7 +486,16 @@ def reproject_disparity_to_3D(
def reproject_disparity_to_3D(
disparity_tensor: torch.Tensor, Q_matrix: torch.Tensor, return_mask: bool = False
) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
- r"""Reproject the disparity torch.Tensor to a 3D point cloud.
+ r"""Reproject a disparity map to 3D points with a Q matrix, as cv2.reprojectImageTo3D does.
+
+ ``points = reproject_disparity_to_3D(disparity, Q)`` takes a channels-last ``(B, H, W, 1)`` disparity map and
+ ``Q`` matrices ``(B, 4, 4)`` or ``(1, 4, 4)``, and returns ``(X / W, Y / W, Z / W)`` of
+ ``Q (u, v, d, 1)`` for every pixel ``(u, v)`` = (column, row), ``(B, H, W, 3)``, in the frame ``Q`` defines.
+ Where ``abs(W) <= 1e-8``, as at a zero disparity when ``Q[3, 3] = 0``, it returns the undivided ``(X, Y, Z)``,
+ while OpenCV divides (infinities at ``W = 0``); see the Note. With ``return_mask=True`` it also returns a
+ ``(B, H, W, 1)`` bool mask of numerically valid points. For a rig given by its two rectified projection
+ matrices, :class:`StereoCamera` builds ``Q``; for a depth map rather than points, use
+ :func:`~kornia.geometry.depth.depth_from_disparity`.
See the Convention block on :class:`~kornia.geometry.camera.stereo.StereoCamera`.
@@ -433,6 +511,13 @@ def reproject_disparity_to_3D(
Raises:
ValueError: If the Q matrix batch size is neither 1 nor the disparity batch size.
+ StereoException: If ``disparity_tensor`` or ``Q_matrix`` is not a floating tensor of the documented shape.
+
+ See Also:
+ - :class:`StereoCamera`: builds ``Q`` from the two rectified projection matrices of a rig.
+ - :func:`~kornia.geometry.depth.depth_from_disparity`: a depth map instead of points, from a baseline and a
+ focal length.
+ - :func:`~kornia.geometry.depth.depth_to_3d_v2`: points from a depth map and a ``3x3`` ``K``.
Note:
A point is valid when all four homogeneous components are finite,
diff --git a/kornia/geometry/conversions.py b/kornia/geometry/conversions.py
index 1dc44c0d5..aace6472c 100644
--- a/kornia/geometry/conversions.py
+++ b/kornia/geometry/conversions.py
@@ -63,7 +63,8 @@ Which function to use:
:func:`ARKitQTVecs_to_ColmapQTVecs` and :func:`ColmapQTVecs_to_ARKitQTVecs`.
- Angles and polar coordinates: :func:`deg2rad`, :func:`rad2deg`, :func:`cart2pol`, :func:`pol2cart`.
- Rotations and poses as objects that compose and invert: :class:`~kornia.geometry.quaternion.Quaternion`,
- :class:`~kornia.geometry.liegroup.So3` and :class:`~kornia.geometry.liegroup.Se3`.
+ :class:`~kornia.geometry.liegroup.So2`, :class:`~kornia.geometry.liegroup.So3`,
+ :class:`~kornia.geometry.liegroup.Se2` and :class:`~kornia.geometry.liegroup.Se3`.
"""
from __future__ import annotations
@@ -546,6 +547,8 @@ def axis_angle_to_rotation_matrix(axis_angle: torch.Tensor) -> torch.Tensor:
- :func:`axis_angle_to_quaternion`: the ``(w, x, y, z)`` quaternion of the same rotation.
- :func:`quaternion_to_rotation_matrix`: the same matrix from a quaternion.
- :func:`angle_to_rotation_matrix`: a 2D ``(*, 2, 2)`` rotation from an angle in degrees.
+ - :class:`~kornia.geometry.liegroup.So3`: the same rotation as an object that composes, inverts and maps back
+ with ``log``; ``So3.exp(v).matrix()`` is this matrix.
Example:
>>> input = torch.tensor([[0., 0., 0.]])
@@ -1109,6 +1112,8 @@ def quaternion_to_axis_angle(quaternion: torch.Tensor) -> torch.Tensor:
- :func:`quaternion_exp_to_log`: the logarithm map, half this vector for ``w >= 0``; it tells ``q`` from
``-q``.
- :func:`rotation_matrix_to_axis_angle`: the same vector from a rotation matrix.
+ - :meth:`~kornia.geometry.quaternion.Quaternion.to_axis_angle`: the same vector of a
+ :class:`~kornia.geometry.quaternion.Quaternion` object.
Example:
>>> quaternion = torch.tensor((1., 0., 0., 0.))
@@ -1903,6 +1908,8 @@ def normalize_pixel_coordinates3d(
- :func:`denormalize_pixel_coordinates3d`: the reverse map, :math:`[-1, 1]` back to voxels.
- :func:`normalize_pixel_coordinates`: the 2D form, for ``(*, 2)`` coordinates in ``(x, y)`` order.
- :func:`normal_transform_pixel3d`: the same scaling as a ``(1, 4, 4)`` matrix on ``(x, y, z, 1)``.
+ - :func:`~kornia.geometry.grid.create_meshgrid3d`: the whole ``(1, D, H, W, 3)`` voxel grid, normalized the
+ same way by default.
"""
if pixel_coordinates.shape[-1] != 3:
@@ -2079,6 +2086,8 @@ def angle_to_rotation_matrix(angle: torch.Tensor) -> torch.Tensor:
turns about a centre, in the same sense.
- :func:`axis_angle_to_rotation_matrix`: a 3D ``(*, 3, 3)`` rotation from an axis-angle vector in radians.
- :func:`cart2pol`: the polar angle of a point, which this rotation decreases.
+ - :class:`~kornia.geometry.liegroup.So2`: a 2D rotation object built from radians, turning the other way:
+ its :meth:`~kornia.geometry.liegroup.So2.matrix` is the transpose of this matrix for the same angle.
Example:
>>> input = torch.rand(1, 3) # Nx3
diff --git a/kornia/geometry/depth.py b/kornia/geometry/depth.py
index f5ecc0fa5..965679ec0 100644
--- a/kornia/geometry/depth.py
+++ b/kornia/geometry/depth.py
@@ -201,6 +201,7 @@ def depth_to_3d_v2(
- :func:`depth_to_3d`: the same points channels first, ``(B, 3, H, W)``.
- :func:`unproject_meshgrid`: the per-pixel rays to pass as ``xyz_grid``.
- :func:`depth_to_normals`: per-pixel surface normals, from a ``(B, 1, H, W)`` depth.
+ - :func:`~kornia.geometry.camera.perspective.unproject_points`: the camera-frame point of selected pixels.
Example:
>>> depth = torch.rand(4, 4)
@@ -412,6 +413,8 @@ def depth_from_plane_equation(
See Also:
- :func:`~kornia.geometry.camera.perspective.unproject_points`: turn these pixels and depths into 3D points.
- :func:`depth_from_disparity`: a whole depth map from a rectified stereo disparity map, not from a plane.
+ - :class:`~kornia.geometry.plane.Hyperplane`: a plane object, :math:`n \cdot x + d = 0`, with the offset of
+ the opposite sign: for a batch of planes, pass ``plane.normal.data`` and ``-plane.offset.data[:, None]``.
"""
KORNIA_CHECK_SHAPE(plane_normals, ["B", "3"])
diff --git a/kornia/geometry/epipolar/_metrics.py b/kornia/geometry/epipolar/_metrics.py
index e84f7385a..fb976d2ab 100644
--- a/kornia/geometry/epipolar/_metrics.py
+++ b/kornia/geometry/epipolar/_metrics.py
@@ -432,6 +432,8 @@ def left_to_right_epipolar_distance(pts1: Tensor, pts2: Tensor, Fm: Tensor) -> T
- :func:`symmetrical_epipolar_distance`: both one-sided distances, squared and added.
- :func:`sampson_epipolar_distance`: the first-order geometric error over both images.
- :func:`~kornia.geometry.homography.oneway_transfer_error`: the image-2 error of a homography.
+ - :func:`~kornia.geometry.linalg.point_line_distance`: the distance from points to given lines, without a
+ fundamental matrix.
"""
KORNIA_CHECK_IS_TENSOR(pts1)
diff --git a/kornia/geometry/grid.py b/kornia/geometry/grid.py
index 52d4f390e..90e87736d 100644
--- a/kornia/geometry/grid.py
+++ b/kornia/geometry/grid.py
@@ -15,6 +15,20 @@
# limitations under the License.
#
+"""Grids of pixel and voxel coordinates, normalized to [-1, 1] for grid_sample or in pixels.
+
+Coordinates are ``(x, y)`` = (column, row) in 2D and ``(d, x, y)``, depth first, in 3D, with integer pixel centres;
+see :ref:`Coordinates and sizes <coordinate-conventions>`. A normalized grid maps the first and last pixel centres of
+each axis to ``-1`` and ``1`` by default, the ``align_corners=True`` convention of
+:func:`torch.nn.functional.grid_sample`.
+
+Which function to use:
+
+- The ``(1, H, W, 2)`` grid of ``(x, y)`` coordinates of an image: :func:`create_meshgrid`, which can also normalize
+ to the half-pixel ``align_corners=False`` convention.
+- The ``(1, D, H, W, 3)`` grid of ``(d, x, y)`` coordinates of a volume: :func:`create_meshgrid3d`, corner-aligned
+ only; reorder its last axis with ``grid[..., [1, 2, 0]]`` for ``grid_sample``.
+"""
from typing import Optional
@@ -33,7 +47,13 @@ def create_meshgrid(
dtype: Optional[torch.dtype] = None,
align_corners: bool = True,
) -> torch.Tensor:
- r"""Generate a (1, H, W, 2) grid of (x, y) pixel coordinates, normalized to [-1, 1] by default.
+ r"""Generate a (1, H, W, 2) grid of (x, y) pixel coordinates, in [-1, 1] by default.
+
+ ``grid = create_meshgrid(H, W)`` returns the ``(x, y)`` coordinates of every pixel of an ``H x W`` image as a
+ ``(1, H, W, 2)`` tensor, ``grid[0, i, j]`` holding column ``j`` and row ``i``: normalized to :math:`[-1, 1]` by
+ default, or in pixels, ``0`` to ``W - 1`` and ``H - 1``, with ``normalized_coordinates=False``. A normalized grid
+ is the identity sampling grid of :func:`torch.nn.functional.grid_sample` called with the same ``align_corners``.
+ For the ``(1, D, H, W, 3)`` grid of a volume, use :func:`create_meshgrid3d`.
Convention:
- Coordinates follow :ref:`Coordinates and sizes <coordinate-conventions>`. With
@@ -60,6 +80,17 @@ def create_meshgrid(
Return:
grid tensor with shape ``(1, H, W, 2)``.
+ See Also:
+ - :func:`create_meshgrid3d`: the ``(1, D, H, W, 3)`` grid of ``(d, x, y)`` voxel coordinates, corner-aligned
+ only.
+ - :func:`~kornia.geometry.conversions.normalize_pixel_coordinates`: normalizes given ``(x, y)`` pixel
+ coordinates the way the default grid is normalized.
+ - :func:`~kornia.geometry.transform.warp_grid`: applies homographies to this grid, without sampling.
+ - :func:`~kornia.geometry.transform.remap`: samples an image at an x map and a y map, such as the two
+ channels of a pixel grid plus a flow.
+ - :func:`~kornia.geometry.depth.unproject_meshgrid`: the camera ray through every pixel instead of its
+ coordinates.
+
Example:
>>> create_meshgrid(2, 2)
tensor([[[[-1., -1.],
@@ -158,7 +189,13 @@ def create_meshgrid3d(
device: Optional[torch.device] = None,
dtype: Optional[torch.dtype] = None,
) -> torch.Tensor:
- """Generate a (1, D, H, W, 3) grid of (d, x, y) voxel coordinates, normalized to [-1, 1] by default.
+ """Generate a (1, D, H, W, 3) grid of (d, x, y) voxel coordinates, in [-1, 1] by default.
+
+ ``grid = create_meshgrid3d(D, H, W)`` returns the ``(d, x, y)`` coordinates of every voxel of a ``D x H x W`` volume
+ as a ``(1, D, H, W, 3)`` tensor, ``grid[0, k, i, j]`` holding depth ``k``, column ``j`` and row ``i``: normalized
+ to :math:`[-1, 1]`, corner-aligned, by default, or in voxels with ``normalized_coordinates=False``. Depth comes
+ first, unlike the ``(x, y, z)`` that :func:`torch.nn.functional.grid_sample` reads. For the ``(1, H, W, 2)`` grid of
+ an image, use :func:`create_meshgrid`.
Convention:
- Coordinates follow :ref:`Coordinates and sizes <coordinate-conventions>`. A normalized grid is always
@@ -180,6 +217,14 @@ def create_meshgrid3d(
Return:
grid tensor with shape ``(1, D, H, W, 3)``.
+ See Also:
+ - :func:`create_meshgrid`: the ``(1, H, W, 2)`` grid of ``(x, y)`` pixel coordinates, with an
+ ``align_corners`` choice.
+ - :func:`~kornia.geometry.conversions.normalize_pixel_coordinates3d`: normalizes given ``(d, x, y)`` voxel
+ coordinates the way this grid is normalized.
+ - :func:`~kornia.geometry.transform.warp_grid3d`: applies ``4 x 4`` homographies to an ``(x, y, z)`` grid, so
+ reorder this one first.
+
"""
# See ``create_meshgrid``: ``arange`` keeps eager and captured pixel ramps identical at
# float16/bfloat16, keeps the output length dynamic under export, and reads the default dtype
diff --git a/kornia/geometry/homography.py b/kornia/geometry/homography.py
index ab7ad50e7..a4ef16c20 100644
--- a/kornia/geometry/homography.py
+++ b/kornia/geometry/homography.py
@@ -655,6 +655,8 @@ def find_homography_dlt(
scene.
- :func:`~kornia.geometry.solvers.find_affine`: an affine map, the special case with bottom row
``[0, 0, 1]``.
+ - :func:`~kornia.geometry.calibration.solve_pnp_dlt`: the DLT for 3D-2D correspondences, a camera pose
+ rather than a homography.
"""
device, dtype = _extract_device_dtype([points1, points2])
diff --git a/kornia/geometry/keypoints.py b/kornia/geometry/keypoints.py
index 954894353..4c102aade 100644
--- a/kornia/geometry/keypoints.py
+++ b/kornia/geometry/keypoints.py
@@ -15,6 +15,21 @@
# limitations under the License.
#
+"""Keypoint containers: 2D ``(x, y)`` and 3D ``(x, y, z)`` points that transform by matrices.
+
+A container wraps one floating-point tensor of points, ``(N, 2)`` or ``(B, N, 2)`` in 2D and ``(N, 3)`` or
+``(B, N, 3)`` in 3D, without copying it, and transforms it by homogeneous matrices with
+:func:`~kornia.geometry.linalg.transform_points`. 2D points are ``(x, y)`` pixel coordinates, ``x`` the column and ``y``
+the row, with pixel ``(0, 0)`` centred at ``(0, 0)``; see :ref:`Coordinates and sizes <coordinate-conventions>`.
+
+Which class to use:
+
+- 2D keypoints that follow an image through ``3 x 3`` transforms and padding, as
+ :class:`~kornia.augmentation.container.AugmentationSequential` does with its ``'keypoints'`` data: :class:`Keypoints`.
+- 3D points transformed by ``4 x 4`` matrices: :class:`Keypoints3D`.
+- Boxes rather than points: :class:`~kornia.geometry.boxes.Boxes`.
+"""
+
from typing import List, Optional, Tuple, Union, cast
import torch
@@ -35,7 +50,18 @@ def _merge_keypoint_list(keypoints: List[torch.torch.Tensor]) -> torch.torch.Ten
class Keypoints:
- r"""2D keypoints, stored as an ``(N, 2)`` or ``(B, N, 2)`` tensor of ``(x, y)`` points.
+ r"""Store (x, y) pixel keypoints, (N, 2) or (B, N, 2), to transform and pad with images.
+
+ ``kp = Keypoints(points)`` wraps a floating-point ``(N, 2)`` or ``(B, N, 2)`` tensor of ``(x, y)`` pixel
+ coordinates without copying it, and ``kp.transform_keypoints(M)`` maps the points by ``(3, 3)`` or ``(B, 3, 3)``
+ matrices and returns a new :class:`Keypoints`; ``kp.to_tensor()`` returns the stored tensor. A ``(B, 3, 3)``
+ batch of ``B > 1`` matrices pairs matrix ``b`` with ``points[b]`` and needs ``(B, N, 2)`` points. :meth:`pad` and
+ :meth:`unpad` shift the points by image padding.
+ :class:`~kornia.augmentation.container.AugmentationSequential` accepts a :class:`Keypoints` under its
+ ``'keypoints'`` data key and transforms it with the images, unless the pipeline contains a
+ :class:`~kornia.augmentation.container.VideoSequential`: such a pipeline raises ``TypeError`` for a
+ :class:`Keypoints` and takes ``VideoKeypoints`` or a ``(B, T, N, 2)`` tensor. To transform a plain tensor once, use
+ :func:`~kornia.geometry.linalg.transform_points`; for 3D points, :class:`Keypoints3D`.
Args:
keypoints: tensor of ``(N, 2)`` or ``(B, N, 2)`` coordinates. A list of tensors is not implemented
@@ -58,6 +84,12 @@ class Keypoints:
- Known defects: list input and ``to_tensor(as_padded_sequence=True)`` raise ``NotImplementedError``
(`#5023 <https://github.com/kornia/kornia/issues/5023>`_).
+ See Also:
+ - :class:`Keypoints3D`: ``(x, y, z)`` points, transformed by ``4 x 4`` matrices, with no padding.
+ - :func:`~kornia.geometry.linalg.transform_points`: the point transform this class applies, on a plain tensor.
+ - :class:`~kornia.geometry.boxes.Boxes`: boxes in the same pixel frame, with matching pad, unpad and transform
+ methods.
+
"""
def __init__(self, keypoints: torch.Tensor, raise_if_not_floating_point: bool = True) -> None:
@@ -377,9 +409,14 @@ class VideoKeypoints(Keypoints):
class Keypoints3D:
- """3D keypoints, stored as an ``(N, 3)`` or ``(B, N, 3)`` tensor of ``(x, y, z)`` points.
+ """Store (N, 3) or (B, N, 3) keypoints in (x, y, z), to transform by 4x4 matrices.
- The constructor validates the tensor as :class:`Keypoints` does and wraps a floating-point one without copying it.
+ ``kp = Keypoints3D(points)`` wraps a floating-point ``(N, 3)`` or ``(B, N, 3)`` tensor of ``(x, y, z)`` points
+ without copying it, and ``kp.transform_keypoints(M)`` maps them by ``(4, 4)`` or ``(B, 4, 4)`` matrices and returns
+ a new :class:`Keypoints3D`; ``kp.to_tensor()`` returns the stored tensor. Unlike :class:`Keypoints` it has no
+ padding, ``index_put``, ``type``, ``device`` or ``dtype``. To transform a plain tensor once, use
+ :func:`~kornia.geometry.linalg.transform_points`; for 2D image points, :class:`Keypoints`. The constructor validates
+ the tensor as :class:`Keypoints` does.
Args:
keypoints: tensor of ``(N, 3)`` or ``(B, N, 3)`` coordinates. A list of tensors is not implemented
@@ -394,6 +431,11 @@ class Keypoints3D:
:meth:`pad` and :meth:`unpad` are deprecated and also raise it
(`#5023 <https://github.com/kornia/kornia/issues/5023>`_).
+ See Also:
+ - :class:`Keypoints`: 2D ``(x, y)`` image points, transformed by ``3 x 3`` matrices and padded with the image.
+ - :func:`~kornia.geometry.linalg.transform_points`: the point transform this class applies, on a plain tensor.
+ - :class:`~kornia.geometry.boxes.Boxes3D`: 3D boxes, transformed by the same ``4 x 4`` matrices.
+
"""
def __init__(self, keypoints: torch.Tensor, raise_if_not_floating_point: bool = True) -> None:
@@ -509,8 +551,8 @@ class Keypoints3D:
"""Validate and wrap a tensor of 3D keypoint coordinates.
Args:
- keypoints: Tensor in ``(N, 3)`` or ``(B, N, 3)`` format,
- where the last dimension stores ``(x, y, z)``.
+ keypoints: Floating-point tensor in ``(N, 3)`` or ``(B, N, 3)`` format; an integer tensor raises
+ ``ValueError``. The last dimension stores ``(x, y, z)``.
Returns:
New :class:`Keypoints3D` instance containing the input coordinates.
diff --git a/kornia/geometry/liegroup/__init__.py b/kornia/geometry/liegroup/__init__.py
index b98a6eb52..8306770f5 100644
--- a/kornia/geometry/liegroup/__init__.py
+++ b/kornia/geometry/liegroup/__init__.py
@@ -15,9 +15,26 @@
# limitations under the License.
#
-"""Kornia Geometry Lie Group — Lie group operations for Kornia.
+r"""Lie groups for 2D and 3D rotations and rigid motions: composition, point maps, exp and log.
-This subpackage provides SE2, SE3, SO2, and related Lie group classes and utilities.
+Each class holds one group element or a batch of ``B`` as an ``nn.Module``, so ``state_dict()`` and ``.to()`` reach
+its tensors. ``a * b`` composes as ``a.matrix() @ b.matrix()``, so ``b`` acts first, and ``g * p`` maps points as
+:math:`R p + t`. ``exp`` maps a tangent vector to the group, and ``log`` maps back to the principal vector, whose
+rotation angle is at most :math:`\pi` in magnitude. Tangent vectors put the rotation last, in radians: the angle
+``theta`` for ``So2``, the rotation vector ``omega`` for ``So3``, ``(v_x, v_y, theta)`` for ``Se2`` and
+``[upsilon, omega]`` for ``Se3``; ``hat`` turns one into its matrix and ``vee`` reads it back. The conventions page
+of the documentation compares these with scipy, Sophus and Eigen.
+
+Which class to use:
+
+- A 2D rotation: :class:`So2`, stored as a complex number. A 2D rotation and translation: :class:`Se2`.
+- A 3D rotation: :class:`So3`, stored as a :class:`~kornia.geometry.quaternion.Quaternion`, with the Jacobians of
+ its ``exp``. A 3D rotation and translation: :class:`Se3`.
+- A rigid pose that names its source and destination frames and checks them when composing:
+ :class:`~kornia.geometry.pose.NamedPose`. Plain ``(N, 4, 4)`` matrices:
+ :func:`~kornia.geometry.linalg.compose_transformations`.
+- Quaternion arithmetic, interpolation and averaging, without acting on points:
+ :class:`~kornia.geometry.quaternion.Quaternion` and :func:`~kornia.geometry.quaternion.average_quaternions`.
"""
from .se2 import Se2
diff --git a/kornia/geometry/liegroup/se2.py b/kornia/geometry/liegroup/se2.py
index cf9fc9313..e27c3505e 100644
--- a/kornia/geometry/liegroup/se2.py
+++ b/kornia/geometry/liegroup/se2.py
@@ -80,15 +80,22 @@ def _check_se2_r_t_dtype_device(r: So2, t: torch.Tensor) -> None:
class Se2(nn.Module):
- r"""Base class to represent the Se2 group.
+ r"""Represent a 2D rigid motion, rotation and translation, as an SE(2) Lie group element.
- The SE(2) is the group of rigid body transformations about the origin of two-dimensional Euclidean
- space :math:`R^2` under the operation of composition.
+ ``g = Se2(rotation, t)`` pairs an :class:`So2` rotation with a translation ``t`` of shape ``(B, 2)``, or ``(2,)``
+ for an unbatched rotation; ``Se2.exp(v)`` builds one from tangent vectors :math:`v = (v_x, v_y, \theta)` of shape
+ ``(B, 3)``, :meth:`from_matrix` from ``(B, 3, 3)`` matrices, and ``Se2.identity(B)`` and ``Se2.random(B)`` build
+ ``B`` poses, or an unbatched one with ``batch_size=None``. ``g * p`` maps points as :math:`R p + t`, ``g1 * g2``
+ composes, ``g.inverse()`` inverts, ``g.log()`` returns the ``(B, 3)`` tangent vectors and ``g.matrix()`` the
+ ``(B, 3, 3)`` matrices. Unbatched, a pose drops the ``B`` and maps ``(2,)`` or ``(N, 2)`` points; a batch of ``B``
+ maps ``(B, 2)``, one point per pose, or a single ``(2,)`` point. For 3D, use :class:`Se3`; for a pose that names
+ its frames and checks them when composing, :class:`~kornia.geometry.pose.NamedPose`.
+
+ SE(2) is the group of rigid motions of :math:`R^2` under composition.
Convention:
- ``matrix()`` is the 3x3 :math:`[[R, t], [0, 1]]`. ``a * b`` is ``a.matrix() @ b.matrix()`` and ``g * p`` is
- :math:`R p + t`, as in :class:`~kornia.geometry.liegroup.Se3`, but an unbatched pose also transforms
- ``(N, 2)`` points and a batched one a single ``(2,)`` point. The rotation is an
+ :math:`R p + t`, as in :class:`~kornia.geometry.liegroup.Se3`. The rotation is an
:class:`~kornia.geometry.liegroup.So2`, whose storage and direction conventions apply.
- The tangent vector is :math:`(v_x, v_y, \theta)`, angle last: ``exp`` rotates by :math:`\theta` and
translates by :math:`V(\theta) (v_x, v_y)`, and ``log`` returns :math:`\theta` in :math:`[-\pi, \pi]`.
@@ -97,8 +104,9 @@ class Se2(nn.Module):
has the form :math:`[[a, -b], [b, a]]`, a rotation scaled by :math:`\sqrt{a^2 + b^2}`, and keeps the scale
as a non-unit ``z`` that ``log`` drops; it rejects a reflection. Graph capture (``torch.compile``,
``torch.export``) omits the value check and interprets the block as ``z = m00 + i m10``.
- - ``t`` is always a tensor registered as module state, whichever constructor built the pose; a ``Vector2``
- passed to the constructor is unwrapped. ``g * p`` returns a ``Vector2`` only when ``p`` is one.
+ - ``t`` is always a tensor, whichever constructor built the pose, and registered as module state outside
+ graph capture; a ``Vector2`` passed to the constructor is unwrapped. ``g * p`` returns a ``Vector2`` only
+ when ``p`` is one.
- ``hat`` returns the se(2) generator :math:`[[0, -\theta, v_x], [\theta, 0, v_y], [0, 0, 0]]` and ``vee``
reads it back, so ``matrix_exp(hat(v))`` equals ``exp(v).matrix()``.
- Module conversions follow :class:`~kornia.geometry.liegroup.So2`'s dtype policy for the rotation and
@@ -107,6 +115,12 @@ class Se2(nn.Module):
pose from bfloat16 values (``exp``, ``identity``, ``random``, ``from_matrix`` or ``trans*``) raises
``TypeCheckError``, as for ``So2``.
+ See Also:
+ - :class:`Se3`: the 3D rigid motion, with an :class:`So3` rotation and a ``(B, 3)`` translation.
+ - :class:`So2`: the rotation alone, without a translation.
+ - :class:`~kornia.geometry.pose.NamedPose`: wraps an ``Se2`` with the names of its source and destination
+ frames, and checks that they chain when composing.
+
Example:
>>> so2 = So2.identity(1)
>>> t = torch.ones((1, 2))
@@ -120,7 +134,7 @@ class Se2(nn.Module):
"""
def __init__(self, rotation: So2, translation: Vector2 | torch.Tensor) -> None:
- """Construct the base class.
+ """Construct an Se2 from an So2 rotation and a translation.
Internally represented by a torch.complex number `z` and a translation 2-vector.
@@ -131,7 +145,9 @@ class Se2(nn.Module):
a bfloat16 translation is also accepted beside a complex32, complex64 or complex128 rotation, which
is what ``.bfloat16()`` leaves (PyTorch has no complex bfloat16). Every method then reads the
translation at the rotation's precision, and ``inverse`` and ``g * g`` return a translation of the
- rotation's real dtype.
+ rotation's real dtype. It must be batched exactly when the rotation is (``ValueError`` otherwise), but
+ its batch size is not checked against the rotation's: a mismatch fails later, in ``matrix()``
+ (`#6482 <https://github.com/kornia/kornia/issues/6482>`_).
Raises:
TypeCheckError: if the translation is an integer, bool or complex tensor (an ``nn.Parameter`` too), or its
@@ -185,8 +201,9 @@ class Se2(nn.Module):
"""Compose two Se2 transformations, or transform points.
Args:
- right: the other Se2 transformation, or points: ``(2,)`` or ``(N, 2)`` for an unbatched pose,
- ``(B, 2)`` or ``(2,)`` for a batch of ``B`` (a batch of one also maps ``(N, 2)``).
+ right: the other Se2 transformation, or points as a tensor or a ``Vector2``: ``(2,)`` or ``(N, 2)``
+ for an unbatched pose, ``(B, 2)``, one point per pose, or a single ``(2,)`` point for a batch of
+ ``B`` (a batch of one also maps ``(N, 2)``).
Return:
The resulting Se2 transformation, or the transformed points.
@@ -229,10 +246,10 @@ class Se2(nn.Module):
@staticmethod
def exp(v: torch.Tensor) -> Se2:
- """Convert elements of lie algebra to elements of lie group.
+ r"""Convert elements of lie algebra to elements of lie group.
Args:
- v: vector of shape ``(B, 3)``.
+ v: vector :math:`(v_x, v_y, \theta)` of shape ``(B, 3)`` or ``(3,)``, angle last, in radians.
Example:
>>> v = torch.ones((1, 3))
diff --git a/kornia/geometry/liegroup/se3.py b/kornia/geometry/liegroup/se3.py
index 1c8d7d19c..d83b0b8fa 100644
--- a/kornia/geometry/liegroup/se3.py
+++ b/kornia/geometry/liegroup/se3.py
@@ -66,11 +66,23 @@ def _check_se3_r_t_dtype_device(r: Quaternion | So3, t: torch.Tensor) -> None:
class Se3(nn.Module):
- r"""Base class to represent the Se3 group.
-
- The SE(3) is the group of rigid body transformations about the origin of three-dimensional Euclidean
- space :math:`R^3` under the operation of composition.
- See more: https://ingmec.ual.es/~jlblanco/papers/jlblanco2010geometry3D_techrep.pdf
+ r"""Represent a 3D rigid motion, rotation and translation, as an SE(3) Lie group element.
+
+ ``g = Se3(rotation, t)`` pairs an :class:`So3` or :class:`~kornia.geometry.quaternion.Quaternion` rotation with a
+ translation ``t`` of shape ``(B, 3)``; ``Se3.exp(v)`` builds one from tangent vectors
+ :math:`v = [\upsilon, \omega]` of shape ``(B, 6)``, translation first, :meth:`from_matrix` from ``(B, 4, 4)``
+ matrices, :meth:`from_qxyz` from ``(B, 7)`` rows ``[qw, qx, qy, qz, x, y, z]``, and ``Se3.identity(B)`` and
+ ``Se3.random(B)`` build ``B`` poses, or an unbatched one with ``batch_size=None``. ``g1 * g2`` composes a batch
+ with a batch of the same size, a batch of one or an unbatched pose; ``g.inverse()`` inverts, ``g.log()`` returns
+ the ``(B, 6)`` tangent vectors and ``g.matrix()`` the ``(B, 4, 4)`` matrices; an unbatched pose drops the ``B``.
+ ``g * p`` maps points as :math:`R p + t`: ``(3,)`` or ``(N, 3)`` points for an unbatched pose, and ``(B, 3)``,
+ one point per pose, or a single ``(1, 3)`` point, not ``(3,)``, for a batch of ``B``; a batch of one also maps
+ ``(N, 3)``. For a pose that
+ names its frames and checks them when composing, use :class:`~kornia.geometry.pose.NamedPose`; for plain
+ ``(N, 4, 4)`` matrices, :func:`~kornia.geometry.linalg.compose_transformations`.
+
+ SE(3) is the group of rigid motions of :math:`R^3` under composition; see
+ https://ingmec.ual.es/~jlblanco/papers/jlblanco2010geometry3D_techrep.pdf.
Convention:
- The tangent vector is :math:`[\upsilon, \omega]`, translation first
@@ -82,8 +94,9 @@ class Se3(nn.Module):
- The rotation is an :class:`~kornia.geometry.liegroup.So3`, whose storage and point-shape conventions
apply: a non-unit quaternion stands for the rotation of its direction. ``from_matrix`` ignores the bottom
row.
- - ``t`` is always a tensor registered as module state, whichever constructor built the pose; a ``Vector3``
- passed to the constructor is unwrapped. ``g * p`` returns a ``Vector3`` only when ``p`` is one.
+ - ``t`` is always a tensor, whichever constructor built the pose, and registered as module state outside
+ graph capture; a ``Vector3`` passed to the constructor is unwrapped. ``g * p`` returns a ``Vector3`` only
+ when ``p`` is one.
- ``state_dict()``, ``load_state_dict()`` and module device/dtype conversions include both the rotation
and translation. Rotation state follows :class:`~kornia.geometry.quaternion.Quaternion`'s
buffer/parameter convention.
@@ -91,6 +104,14 @@ class Se3(nn.Module):
rounding :math:`R p + t` once; ``matrix()`` and ``inverse()`` take the rotation from
:class:`~kornia.geometry.liegroup.So3`, which does the same.
+ See Also:
+ - :class:`Se2`: the 2D rigid motion, with an :class:`So2` rotation and a ``(B, 2)`` translation.
+ - :class:`So3`: the rotation alone, without a translation.
+ - :class:`~kornia.geometry.pose.NamedPose`: wraps an ``Se3`` with the names of its source and destination
+ frames, and checks that they chain when composing.
+ - :func:`~kornia.geometry.linalg.compose_transformations`: the same composition on plain ``(N, 4, 4)``
+ matrices, without a group object.
+
Example:
>>> q = Quaternion.identity()
>>> s = Se3(q, torch.ones(3))
@@ -103,7 +124,7 @@ class Se3(nn.Module):
"""
def __init__(self, rotation: Quaternion | So3, translation: Vector3 | torch.Tensor) -> None:
- """Construct the base class.
+ """Construct an Se3 from a rotation and a translation.
Internally represented by an So3 rotation and a translation 3-vector.
@@ -111,7 +132,9 @@ class Se3(nn.Module):
rotation: So3 group encompassing a rotation, or a Quaternion to wrap in one; it is stored as given, and
a non-unit quaternion stands for the rotation of its direction.
translation: translation torch.Tensor with the shape of ``(B, 3)``, or a Vector3 wrapping one; the
- tensor is what gets stored. It must have the rotation's dtype and device.
+ tensor is what gets stored. It must have the rotation's dtype and device. Its batch is not checked
+ against the rotation's: a mismatch fails later, in ``matrix()``
+ (`#6482 <https://github.com/kornia/kornia/issues/6482>`_).
Raises:
TypeCheckError: if the translation is an integer, bool or complex tensor (an ``nn.Parameter`` too), or its
@@ -174,8 +197,9 @@ class Se3(nn.Module):
Both compute in float32 for float16/bfloat16 inputs and return the input dtype.
Args:
- right: the other Se3 transformation, or points as a tensor or ``Vector3``: ``(3,)`` or ``(N, 3)`` for
- an unbatched pose, ``(B, 3)`` for a batch of ``B`` (a batch of one also maps ``(N, 3)``).
+ right: the other Se3 transformation, unbatched, a batch of the same size or a batch of one, or points
+ as a tensor or ``Vector3``: ``(3,)`` or ``(N, 3)`` for an unbatched pose, and ``(B, 3)``, one point
+ per pose, or ``(1, 3)``, not ``(3,)``, for a batch of ``B`` (a batch of one also maps ``(N, 3)``).
Return:
The resulting Se3 transformation, or the transformed points.
@@ -227,10 +251,10 @@ class Se3(nn.Module):
@staticmethod
def exp(v: torch.Tensor) -> Se3:
- """Convert elements of lie algebra to elements of lie group.
+ r"""Convert elements of lie algebra to elements of lie group.
Args:
- v: vector of shape ``(B, 6)``.
+ v: vector :math:`[\upsilon, \omega]` of shape ``(B, 6)``, translation first, rotation vector last.
Example:
>>> v = torch.zeros((1, 6))
diff --git a/kornia/geometry/liegroup/so2.py b/kornia/geometry/liegroup/so2.py
index e1523f5a2..7ae29bfbf 100644
--- a/kornia/geometry/liegroup/so2.py
+++ b/kornia/geometry/liegroup/so2.py
@@ -46,16 +46,22 @@ def _complex_rotation(real: torch.Tensor, imag: torch.Tensor) -> torch.Tensor:
class So2(nn.Module):
- r"""Base class to represent the So2 group.
+ r"""Represent a 2D rotation as an SO(2) Lie group element stored as a complex number.
- The SO(2) is the group of all rotations about the origin of two-dimensional Euclidean space
- :math:`R^2` under the operation of composition.
- See more: https://en.wikipedia.org/wiki/Orthogonal_group#Special_orthogonal_group
+ ``r = So2.exp(theta)`` builds rotations from angles ``theta`` in radians, ``()`` or ``(B,)``, and ``So2(z)`` wraps
+ complex numbers ``z`` of shape ``()``, ``(B,)`` or ``(B, 1)``, read as ``(B,)``; ``So2.identity(B)`` and
+ ``So2.random(B)`` build ``B`` rotations, or an unbatched one with ``batch_size=None``. ``r * p`` rotates points
+ ``p`` as :math:`R p`, ``r1 * r2`` composes, ``r.inverse()`` inverts, ``r.log()`` returns the angles and
+ ``r.matrix()`` the ``(B, 2, 2)`` matrices (``(2, 2)`` unbatched). An unbatched rotation maps ``(2,)`` or
+ ``(N, 2)`` points, and a batch of ``B`` maps ``(B, 2)``, one point per rotation, or a single ``(2,)`` point. For a
+ rotation and a translation, use :class:`Se2`; for 3D rotations, :class:`So3`.
+
+ SO(2) is the group of rotations of :math:`R^2` about the origin under composition; see
+ https://en.wikipedia.org/wiki/Orthogonal_group#Special_orthogonal_group.
Convention:
- - Represents the rotation as a complex number ``z`` of shape :math:`()` or ``(B,)``; a ``(B, 1)``
- ``z`` or angle is accepted and read as ``(B,)``. For unit :math:`z = \cos\theta + i \sin\theta`,
- ``matrix()`` is :math:`[[\cos\theta, -\sin\theta], [\sin\theta, \cos\theta]]`. Non-unit ``z = a + i b``
+ - For unit :math:`z = \cos\theta + i \sin\theta`, ``matrix()`` is
+ :math:`[[\cos\theta, -\sin\theta], [\sin\theta, \cos\theta]]`. Non-unit ``z = a + i b``
is accepted and produces :math:`[[a, -b], [b, a]]`, which rotates and scales by :math:`|z|`. The complex
storage has no bfloat16 counterpart in PyTorch, so ``exp``, ``identity``, ``random`` and ``from_matrix``
raise ``TypeCheckError`` for bfloat16 values.
@@ -63,6 +69,8 @@ class So2(nn.Module):
displayed on y-down image axes. For a unit rotation, ``matrix()`` is the transpose of
:func:`~kornia.geometry.conversions.angle_to_rotation_matrix`, which takes degrees.
``log`` returns the angle in :math:`[-\pi, \pi]`, and ``adjoint()`` is the 2x2 identity.
+ - ``a * b`` composes like ``a.matrix() @ b.matrix()``, so ``b`` acts first; an unbatched rotation composes
+ with each rotation of a batch.
- ``hat`` returns the so(2) generator :math:`[[0, -\theta], [\theta, 0]]` and ``vee`` reads its ``[1, 0]``
entry, so ``matrix_exp(hat(theta))`` equals ``exp(theta).matrix()``.
- Module dtype conversions act on the real and imaginary components together: ``.float()`` and
@@ -72,6 +80,12 @@ class So2(nn.Module):
complex32 dtype. Since PyTorch has no complex bfloat16 dtype, ``.bfloat16()`` and ``.to(torch.bfloat16)``
keep the rotation's existing complex precision while following any requested device move.
+ See Also:
+ - :class:`So3`: the 3D rotation group, stored as a quaternion.
+ - :class:`Se2`: a 2D rotation followed by a translation, mapping points as :math:`R p + t`.
+ - :func:`~kornia.geometry.conversions.angle_to_rotation_matrix`: a ``(*, 2, 2)`` matrix from an angle in
+ degrees on a plain tensor, the transpose of :meth:`matrix` for the same angle.
+
Example:
>>> real = torch.tensor([0.6])
>>> imag = torch.tensor([0.8])
@@ -82,12 +96,10 @@ class So2(nn.Module):
"""
def __init__(self, z: torch.Tensor) -> None:
- """Construct the base class.
-
- Internally represented by torch.complex number `z`.
+ """Construct an So2 from complex numbers ``z``, stored as given.
Args:
- z: Complex number with the shape of ``(B,)`` or :math:`()`; ``(B, 1)`` is read as ``(B,)``.
+ z: Complex number with the shape of ``(B,)`` or ``()``; ``(B, 1)`` is read as ``(B,)``.
Raises:
TypeCheckError: if ``z`` is not a complex tensor (an integer, bool or real floating-point tensor).
@@ -156,8 +168,8 @@ class So2(nn.Module):
Args:
right: the other So2 transformation, or points as a tensor or a ``Vector2``: ``(2,)`` or ``(N, 2)``
- for an unbatched rotation, ``(B, 2)`` or ``(2,)`` for a batch of ``B`` (a batch of one also maps
- ``(N, 2)``).
+ for an unbatched rotation, ``(B, 2)``, one point per rotation, or a single ``(2,)`` point for a
+ batch of ``B`` (a batch of one also maps ``(N, 2)``).
Return:
The resulting So2 transformation, or the rotated points with the type of ``right``.
@@ -193,7 +205,7 @@ class So2(nn.Module):
@property
def z(self) -> torch.Tensor:
- """Return the underlying complex number of shape :math:`()` or ``(B,)``.
+ """Return the underlying complex number of shape ``()`` or ``(B,)``.
A ``(B, 1)`` ``z`` given to the constructor stays stored, and registered, as it was given, and is read
here as a ``(B,)`` view.
@@ -205,7 +217,7 @@ class So2(nn.Module):
"""Convert elements of lie algebra to elements of lie group.
Args:
- theta: angle in radians of shape ``(B,)`` or :math:`()`; ``(B, 1)`` is squeezed to ``(B,)``.
+ theta: angle in radians of shape ``(B,)`` or ``()``; ``(B, 1)`` is squeezed to ``(B,)``.
Raises:
TypeCheckError: if ``theta`` is bfloat16, which has no complex dtype in PyTorch.
@@ -246,7 +258,7 @@ class So2(nn.Module):
Its matrix exponential is the rotation :meth:`exp` returns.
Args:
- theta: angle in radians of shape ``(B,)`` or :math:`()`; ``(B, 1)`` is squeezed to ``(B,)``.
+ theta: angle in radians of shape ``(B,)`` or ``()``; ``(B, 1)`` is squeezed to ``(B,)``.
Example:
>>> theta = torch.tensor(3.1415/2)
@@ -427,7 +439,7 @@ class So2(nn.Module):
return cls.exp(theta)
def adjoint(self) -> torch.Tensor:
- """Return the adjoint matrix of shape ``(B, 2, 2)``.
+ """Return the adjoint matrix, the identity, of shape ``(B, 2, 2)`` (``(2, 2)`` unbatched).
Example:
>>> s = So2.identity()
diff --git a/kornia/geometry/liegroup/so3.py b/kornia/geometry/liegroup/so3.py
index fc790b68a..284376f91 100644
--- a/kornia/geometry/liegroup/so3.py
+++ b/kornia/geometry/liegroup/so3.py
@@ -117,13 +117,21 @@ def _check_points_layout(q: Quaternion, points: torch.Tensor) -> None:
class So3(nn.Module):
- r"""Base class to represent the So3 group.
-
- The SO(3) is the group of all rotations about the origin of three-dimensional Euclidean space
- :math:`R^3` under the operation of composition.
- See more: https://en.wikipedia.org/wiki/3D_rotation_group
-
- We internally represent the rotation by a :class:`~kornia.geometry.quaternion.Quaternion`.
+ r"""Represent a 3D rotation as an SO(3) Lie group element stored as a quaternion.
+
+ ``r = So3.exp(omega)`` builds rotations from rotation vectors ``omega`` of shape ``(B, 3)``, the axis times the
+ angle in radians; ``So3(q)`` wraps a ``(B, 4)`` :class:`~kornia.geometry.quaternion.Quaternion`,
+ :meth:`from_matrix` reads ``(B, 3, 3)`` matrices, and ``So3.identity(B)`` and ``So3.random(B)`` build ``B``
+ rotations, or an unbatched one with ``batch_size=None``. ``r1 * r2`` composes a batch with a batch of the same size,
+ a batch of one or an unbatched rotation; ``r.inverse()`` inverts, ``r.log()`` returns the ``(B, 3)`` rotation
+ vectors and ``r.matrix()`` the ``(B, 3, 3)`` matrices; an unbatched rotation drops the ``B``. ``r * p`` rotates
+ points as :math:`R p`: ``(3,)`` or ``(N, 3)`` points for an unbatched rotation, and ``(B, 3)``, one point per
+ rotation, or a single ``(1, 3)`` point, not ``(3,)``, for a batch of ``B``; a batch of one also maps ``(N, 3)``.
+ For a rotation and a translation, use :class:`Se3`; for
+ quaternion arithmetic, interpolation and Euler angles, :class:`~kornia.geometry.quaternion.Quaternion`.
+
+ SO(3) is the group of rotations of :math:`R^3` about the origin under composition; see
+ https://en.wikipedia.org/wiki/3D_rotation_group.
Convention:
- The tangent vector is the rotation vector, the axis times the angle in radians, and ``hat(v)`` is its
@@ -133,13 +141,14 @@ class So3(nn.Module):
:math:`\pi`, the same for ``q`` and ``-q`` below a half turn.
- ``a * b`` composes like ``a.matrix() @ b.matrix()``, so ``b`` acts first; ``s * p`` rotates points ``p``, a
tensor or a ``Vector3``, as :math:`R p`: ``(3,)`` or ``(N, 3)`` for an unbatched ``So3``, and ``(B, 3)``,
- one point per rotation, for a batch of ``B`` (a batch of one also maps ``(N, 3)``); other layouts raise
- ``ValueError``. ``adjoint()`` is :math:`R` itself.
+ one point per rotation, or ``(1, 3)``, not ``(3,)``, for a batch of ``B`` (a batch of one also maps
+ ``(N, 3)``); other layouts raise ``ValueError``. ``adjoint()`` is :math:`R` itself.
:ref:`Rotations and rigid motions <rotation-conventions>` compares these with scipy, Sophus and Eigen.
- A non-unit quaternion stands for the rotation of its direction :math:`q / |q|`, as for
:meth:`~kornia.geometry.quaternion.Quaternion.matrix`: ``matrix()`` and ``s * p`` normalise it where they
- use it, so ``matrix()`` is always a rotation and ``s * p`` keeps the norm of ``p``. ``q`` returns the
- quaternion as stored, which keeps an ``nn.Parameter`` registered and following optimiser steps.
+ use it, so ``matrix()`` is a rotation and ``s * p`` keeps the norm of ``p`` within the range limits stated
+ on :class:`~kornia.geometry.quaternion.Quaternion`. ``q`` returns the quaternion as stored, which keeps an
+ ``nn.Parameter`` registered and following optimiser steps.
- For a small :math:`\delta`,
:math:`\exp(\omega + \delta) \approx \exp(\omega) \exp(J_r \delta) = \exp(J_l \delta) \exp(\omega)`,
with :math:`J_r` = ``right_jacobian(omega)`` and :math:`J_l` = ``left_jacobian(omega)`` =
@@ -153,6 +162,14 @@ class So3(nn.Module):
- ``matrix()``, ``adjoint()``, ``a * b`` and ``s * p`` compute in float32 for float16/bfloat16 inputs and
return the input dtype, rounding once; float32 and float64 are evaluated in their own dtype.
+ See Also:
+ - :class:`So2`: the 2D rotation group, stored as a complex number.
+ - :class:`Se3`: a 3D rotation followed by a translation, mapping points as :math:`R p + t`.
+ - :class:`~kornia.geometry.quaternion.Quaternion`: the stored quaternion, with the Hamilton product,
+ ``slerp`` and Euler angles but no action on points.
+ - :func:`~kornia.geometry.conversions.axis_angle_to_rotation_matrix`: the matrix of ``So3.exp(v)`` in one
+ call on a plain ``(*, 3)`` tensor.
+
Example:
>>> q = Quaternion.identity()
>>> s = So3(q)
@@ -162,10 +179,10 @@ class So3(nn.Module):
"""
def __init__(self, q: Quaternion) -> None:
- """Construct the base class.
+ """Construct an So3 that stores the quaternion ``q`` as given.
Args:
- q: Quaternion with the shape of ``(B, 4)``.
+ q: Quaternion with the shape of ``(B, 4)`` or ``(4,)``.
Example:
>>> data = torch.tensor([[1., 0., 0., 0.], [0., 1., 0., 0.]])
@@ -200,8 +217,10 @@ class So3(nn.Module):
Both compute in float32 for float16/bfloat16 inputs and return the input dtype.
Args:
- right: the other So3 transformation, or points as a tensor or a ``Vector3``: ``(3,)`` or ``(N, 3)``
- for an unbatched rotation, ``(B, 3)`` for a batch of ``B`` (a batch of one also maps ``(N, 3)``).
+ right: the other So3 transformation, unbatched, a batch of the same size or a batch of one, or points
+ as a tensor or a ``Vector3``: ``(3,)`` or ``(N, 3)`` for an unbatched rotation, and ``(B, 3)``, one
+ point per rotation, or ``(1, 3)``, not ``(3,)``, for a batch of ``B`` (a batch of one also maps
+ ``(N, 3)``).
Return:
The resulting So3 transformation, or the rotated points of the type of ``right``.
@@ -223,7 +242,7 @@ class So3(nn.Module):
@property
def q(self) -> Quaternion:
- """Return the underlying data with shape ``(B,4)``."""
+ """Return the stored :class:`~kornia.geometry.quaternion.Quaternion`: ``(B, 4)``, or ``(4,)`` unbatched."""
return self._q
@staticmethod
@@ -338,7 +357,7 @@ class So3(nn.Module):
def matrix(self) -> torch.Tensor:
r"""Convert the quaternion to a rotation matrix of shape ``(B,3,3)``.
- The matrix is of the form:
+ The matrix is of the form, with :math:`(w, x, y, z)` the unit quaternion :math:`q / |q|`:
.. math::
\begin{bmatrix} 1-2y^2-2z^2 & 2xy-2zw & 2xz+2yw \\
diff --git a/kornia/geometry/linalg.py b/kornia/geometry/linalg.py
index 6a900ed52..f7cb20178 100644
--- a/kornia/geometry/linalg.py
+++ b/kornia/geometry/linalg.py
@@ -15,6 +15,26 @@
# limitations under the License.
#
+r"""Homogeneous 4x4 transforms and point transforms, with point-line distances and last-axis vector products.
+
+A transform ``trans_ab`` maps points in frame ``b`` to frame ``a``, :math:`p_a = T_a^{b} p_b`, so
+``compose_transformations(trans_01, trans_12)`` is ``trans_02`` and ``transform_points(trans_01, points_1)`` returns
+``points_0``. The ``4 x 4`` functions take ``(4, 4)`` or ``(N, 4, 4)`` inputs, two arguments with the same number of
+dimensions, and a batch of one broadcasts against a batch of ``N``. The vector functions reduce the last dimension and
+broadcast the leading ones.
+
+Which function to use:
+
+- Chain two transforms: :func:`compose_transformations`. Invert a rigid one: :func:`inverse_transformation`. The
+ transform between two poses given in one frame: :func:`relative_transformation`.
+- Apply a ``(D + 1, D + 1)`` homogeneous matrix, rigid, affine or projective, to ``(..., N, D)`` points:
+ :func:`transform_points`.
+- Distances: between paired points, :func:`euclidean_distance`; from 2D points to lines :math:`ax + by + c = 0`,
+ :func:`point_line_distance`.
+- Products over the last dimension: :func:`batched_dot_product`, and the squared norm :func:`batched_squared_norm`,
+ also exported as :func:`squared_norm`, the same function.
+"""
+
from __future__ import annotations
import torch
@@ -36,7 +56,14 @@ __all__ = [
def compose_transformations(trans_01: torch.Tensor, trans_12: torch.Tensor) -> torch.Tensor:
- r"""Compose two homogeneous transformations.
+ r"""Compose two 4x4 homogeneous transforms, trans_01 @ trans_12, into trans_02.
+
+ ``trans_02 = compose_transformations(trans_01, trans_12)`` takes two homogeneous transforms ``(N, 4, 4)`` or
+ ``(4, 4)`` and returns their product ``trans_01 @ trans_12``, the transform from frame 2 to frame 0, with the
+ broadcast shape. Both inputs need the same number of dimensions: a ``(4, 4)`` with an ``(N, 4, 4)`` raises
+ ``ValueError``, while a batch of one broadcasts against a batch of ``N``. For poses that name their frames and check
+ them when composing, use :class:`~kornia.geometry.pose.NamedPose`; to apply the result to points,
+ :func:`transform_points`.
.. math::
T_0^{2} = \begin{bmatrix} R_0^{1} R_1^{2} & R_0^{1} t_1^{2} + t_0^{1} \\
@@ -60,6 +87,13 @@ def compose_transformations(trans_01: torch.Tensor, trans_12: torch.Tensor) -> t
Returns:
the transformation between the two frames with shape ``(N, 4, 4)`` or ``(4, 4)``.
+ See Also:
+ - :func:`inverse_transformation`: the inverse of a rigid transform, to compose with.
+ - :func:`relative_transformation`: ``inverse_transformation(trans_01) @ trans_02`` in one call.
+ - :func:`transform_points`: applies a composed transform to points.
+ - :class:`~kornia.geometry.pose.NamedPose`: a pose that names its two frames and checks them when composing.
+ - :class:`~kornia.geometry.liegroup.Se3`: a rigid pose as a Lie group element, composed with ``*``.
+
Example::
>>> trans_01 = torch.eye(4) # 4x4
>>> trans_12 = torch.eye(4) # 4x4
@@ -101,7 +135,12 @@ def compose_transformations(trans_01: torch.Tensor, trans_12: torch.Tensor) -> t
def inverse_transformation(trans_12: torch.Tensor) -> torch.Tensor:
- r"""Invert a 4x4 rigid homogeneous transformation.
+ r"""Invert a 4x4 rigid transform in closed form by transposing its rotation block.
+
+ ``trans_21 = inverse_transformation(trans_12)`` takes a rigid transform :math:`[R|t]`, ``(N, 4, 4)`` or ``(4, 4)``,
+ and returns its inverse :math:`[R^\top | -R^\top t]` with the same shape, without a general matrix inversion. It
+ is the inverse only for a rigid input, as the convention below says; to invert and compose in one call, use
+ :func:`relative_transformation`.
:math:`T_1^{2} = \begin{bmatrix} R_1 & t_1 \\ \mathbf{0} & 1 \end{bmatrix}`
@@ -124,6 +163,12 @@ def inverse_transformation(trans_12: torch.Tensor) -> torch.Tensor:
Returns:
tensor with inverted transformations with shape ``(N, 4, 4)`` or ``(4, 4)``.
+ See Also:
+ - :func:`relative_transformation`: the inverse of one transform composed with another, in one call.
+ - :func:`compose_transformations`: chains transforms, such as this inverse with another one.
+ - :func:`~kornia.geometry.conversions.camtoworld_to_worldtocam_Rt`: the same rigid inverse for a rotation and
+ a translation given apart.
+
Example:
>>> trans_12 = torch.eye(4)[None] # Nx4x4: a rotation of 90 degrees about z and a translation
>>> trans_12[:, :3, :3] = torch.tensor([[0.0, -1.0, 0.0], [1.0, 0.0, 0.0], [0.0, 0.0, 1.0]])
@@ -154,7 +199,13 @@ def inverse_transformation(trans_12: torch.Tensor) -> torch.Tensor:
def relative_transformation(trans_01: torch.Tensor, trans_02: torch.Tensor) -> torch.Tensor:
- r"""Compute the relative homogeneous transformation from a reference transformation.
+ r"""Compute the relative 4x4 transform trans_12 = inv(trans_01) @ trans_02 of two rigid poses.
+
+ ``trans_12 = relative_transformation(trans_01, trans_02)`` takes two transforms into a common frame 0,
+ ``(N, 4, 4)`` or ``(4, 4)``, and returns the transform from frame 2 to frame 1 with the broadcast shape: for two
+ camera-to-world poses, the pose of camera 2 in the frame of camera 1. Both inputs need the same number of
+ dimensions, as in :func:`compose_transformations`. For world-to-camera extrinsics given as rotations and
+ translations, use :func:`~kornia.geometry.epipolar.relative_camera_motion`.
Given the reference :math:`T_0^{1} = \begin{bmatrix} R_1 & t_1 \\ \mathbf{0} & 1 \end{bmatrix}` and the
destination :math:`T_0^{2} = \begin{bmatrix} R_2 & t_2 \\ \mathbf{0} & 1 \end{bmatrix}`, the relative
@@ -185,6 +236,13 @@ def relative_transformation(trans_01: torch.Tensor, trans_02: torch.Tensor) -> t
Returns:
the relative transformation between the transformations with shape ``(N, 4, 4)`` or ``(4, 4)``.
+ See Also:
+ - :func:`inverse_transformation`: the rigid inverse this function applies to ``trans_01``, alone.
+ - :func:`compose_transformations`: the other way, ``trans_01 @ trans_12`` gives back ``trans_02``.
+ - :func:`~kornia.geometry.epipolar.relative_camera_motion`: the motion between two world-to-camera
+ extrinsics given as rotations and translations, :math:`E_2 E_1^{-1}`.
+ - :class:`~kornia.geometry.pose.NamedPose`: a pose that names its two frames.
+
Example::
>>> trans_01 = torch.eye(4) # 4x4
>>> trans_02 = torch.eye(4) # 4x4
@@ -239,26 +297,49 @@ def _matrix_batch_pairs_with_points(trans_batch: list[int], points_batch: list[i
def transform_points(trans_01: torch.Tensor, points_1: torch.Tensor) -> torch.Tensor:
- r"""Apply transformations to a set of points.
+ r"""Transform (..., N, D) Euclidean points by (D+1)x(D+1) homogeneous matrices, dividing by w.
+
+ ``points_0 = transform_points(trans_01, points_1)`` takes ``(D + 1, D + 1)`` or ``(B, D + 1, D + 1)`` homogeneous
+ matrices and Euclidean points ``(..., N, D)``, appends ``w = 1`` to each point, multiplies, divides by the
+ resulting ``w``, and returns the points ``(..., N, D)``. The matrix can be rigid, affine or projective: a
+ ``(B, 3, 3)`` homography for 2D points, a ``(B, 4, 4)`` pose for 3D points. A single matrix, ``(D + 1, D + 1)`` or
+ ``(1, D + 1, D + 1)``, applies to points of any leading shape; ``B > 1`` matrices pair matrix ``b`` with
+ ``points_1[b]``, so the points need a leading axis of size ``B``. To divide points that are already homogeneous by
+ their ``w``, use :func:`~kornia.geometry.conversions.convert_points_from_homogeneous`.
Convention:
- Integer and boolean operands are promoted, not rejected, because a transform moves integer coordinates
- (pixel indices from ``torch.nonzero`` or ``argmax``) to fractional ones. Integer points return in the
- floating dtype of ``trans_01``, or in the default floating dtype when both operands are integer, and an integer
- ``trans_01`` is applied in the dtype of the result. Floating points keep their dtype.
+ - ``(B, D + 1, D + 1)`` matrices with ``B > 1`` raise ``ValueError`` for ``(N, D)`` points, whatever ``N``,
+ and for ``(1, N, D)`` points: neither has a leading axis of size ``B``.
+ - A matrix batch with more than one leading axis pairs with the leading batch axes of the points, either as
+ it stands or once its size-1 axes are left out, as the ``trans_01`` argument describes; any other
+ multi-axis batch raises ``ValueError``.
+ - A point mapped to ``|w| <= 1e-8`` is returned undivided, as
+ :func:`~kornia.geometry.conversions.convert_points_from_homogeneous` does.
+ - Integer and boolean operands are promoted, not rejected, because a transform moves integer coordinates
+ (pixel indices from ``torch.nonzero`` or ``argmax``) to fractional ones. Integer points return in the
+ floating dtype of ``trans_01``, or in the default floating dtype when both operands are integer, and an
+ integer ``trans_01`` is applied in the dtype of the result. Floating points keep their dtype; the product
+ is computed in the dtype of ``trans_01``, so ``float64`` points under a ``float32`` matrix are transformed
+ in ``float32``.
Args:
trans_01: tensor for transformations of shape
``(D+1, D+1)`` or ``(B, D+1, D+1)``. A batch with several axes, ``(*, D+1, D+1)``, pairs with the
- leading batch axes of ``points_1`` in order once its size-1 axes are left out, so ``(B, 1, D+1, D+1)``
- and ``(1, B, D+1, D+1)`` both apply matrix ``b`` to ``points_1[b]``; any other multi-axis batch raises.
- points_1: tensor of points of shape ``(N, D)`` or ``(B, N, D)``.
+ leading batch axes of ``points_1`` in order, as it stands or once its size-1 axes are left out, so
+ ``(B, 1, D+1, D+1)`` and ``(1, B, D+1, D+1)`` both apply matrix ``b`` to ``points_1[b]``; any other
+ multi-axis batch raises.
+ points_1: tensor of points of shape ``(*, N, D)``: ``(N, D)``, ``(B, N, D)``, or with several leading
+ axes, such as ``(M, B, N, D)``, which pair with a multi-axis ``trans_01`` as described there.
Returns:
transformed points with the same shape as ``points_1``.
- Shape:
- - Output: same shape as ``points_1``.
+ See Also:
+ - :func:`~kornia.geometry.conversions.convert_points_from_homogeneous`: the division by ``w`` alone, for
+ points that are already homogeneous.
+ - :func:`compose_transformations`: chains two ``4 x 4`` transforms before they are applied.
+ - :func:`~kornia.geometry.transform.warp_grid`: applies homographies to a whole ``(1, H, W, 2)`` grid.
+ - :meth:`~kornia.geometry.pose.NamedPose.transform_points`: maps points by a pose that names its frames.
Examples:
>>> points_1 = torch.rand(2, 4, 3) # BxNx3
@@ -350,7 +431,14 @@ def _sqrt_or_zero(squared: torch.Tensor) -> torch.Tensor:
def point_line_distance(point: torch.Tensor, line: torch.Tensor, eps: float = 1e-9) -> torch.Tensor:
- r"""Return the distance from points to lines.
+ r"""Compute the distance from 2D points to lines ax + by + c = 0, at any scale of (a, b, c).
+
+ ``d = point_line_distance(points, lines)`` takes 2D points ``(*, N, 2)``, or homogeneous ``(*, N, 3)``, and lines
+ :math:`(a, b, c)` ``(*, N, 3)``, and returns the Euclidean distance from each point to its line, ``(*, N)``. The
+ leading dimensions broadcast, so ``(M, 1, 2)`` points and ``(1, N, 3)`` lines give all ``(M, N)`` distances. For
+ the distance from the points of one image to the epipolar lines of their matches, use
+ :func:`~kornia.geometry.epipolar.left_to_right_epipolar_distance`, which builds the lines from a fundamental
+ matrix.
Convention:
- ``line`` need not be normalised: the distance is :math:`|ax + by + c| / \|(a, b)\|` at every scale of the
@@ -367,6 +455,16 @@ def point_line_distance(point: torch.Tensor, line: torch.Tensor, eps: float = 1e
Returns:
the computed distance with shape ``(*, N)``.
+ Raises:
+ ValueError: if ``point`` does not end in 2 or 3 coordinates, or ``line`` does not end in 3.
+
+ See Also:
+ - :func:`~kornia.geometry.epipolar.left_to_right_epipolar_distance`: the distance from second-image points to
+ the epipolar lines of their matches, built from a fundamental matrix.
+ - :func:`euclidean_distance`: the distance between paired points instead of from a point to a line.
+ - :meth:`~kornia.geometry.line.ParametrizedLine.distance`: the distance to a line given by an origin and a
+ direction, in any dimension.
+
"""
KORNIA_CHECK_IS_TENSOR(point)
KORNIA_CHECK_IS_TENSOR(line)
@@ -401,19 +499,60 @@ def point_line_distance(point: torch.Tensor, line: torch.Tensor, eps: float = 1e
def batched_dot_product(x: torch.Tensor, y: torch.Tensor, keepdim: bool = False) -> torch.Tensor:
- """Return a batched version of .dot()."""
+ """Compute the dot product along the last dimension of two broadcast tensors.
+
+ ``d = batched_dot_product(x, y)`` takes ``x`` and ``y`` of shape ``(*, N)``, whose shapes broadcast, and returns
+ ``(x * y).sum(-1)``: the dot product of each pair of last-dimension vectors, with that dimension removed, or kept
+ as size 1 with ``keepdim=True``. For the squared norm of one tensor, use :func:`batched_squared_norm`.
+
+ Args:
+ x: first vectors, with shape ``(*, N)``.
+ y: second vectors, with shape ``(*, N)``.
+ keepdim: whether to keep the reduced dimension as size 1.
+
+ Returns:
+ the dot products, with shape ``(*)``, or ``(*, 1)`` with ``keepdim=True``.
+
+ See Also:
+ - :func:`batched_squared_norm`: the dot product of a tensor with itself.
+ - :func:`euclidean_distance`: the root of the summed squared differences of two point sets.
+
+ """
KORNIA_CHECK_SHAPE(x, ["*", "N"])
KORNIA_CHECK_SHAPE(y, ["*", "N"])
return (x * y).sum(-1, keepdim)
def batched_squared_norm(x: torch.Tensor, keepdim: bool = False) -> torch.Tensor:
- """Return the squared norm of a vector."""
+ """Compute the squared Euclidean norm along the last dimension of a tensor.
+
+ ``n2 = batched_squared_norm(x)`` takes ``(*, N)`` vectors and returns the sum of their squares over the last
+ dimension, ``batched_dot_product(x, x)``, with that dimension removed, or kept as size 1 with ``keepdim=True``.
+ ``squared_norm`` is the same function under another name. For the root, the norm of ``x - y``, use
+ :func:`euclidean_distance`.
+
+ Args:
+ x: vectors with shape ``(*, N)``.
+ keepdim: whether to keep the reduced dimension as size 1.
+
+ Returns:
+ the squared norms, with shape ``(*)``, or ``(*, 1)`` with ``keepdim=True``.
+
+ See Also:
+ - :func:`batched_dot_product`: the dot product of two tensors.
+ - :func:`euclidean_distance`: the unsquared norm of the difference of two point sets.
+
+ """
return batched_dot_product(x, x, keepdim)
def euclidean_distance(x: torch.Tensor, y: torch.Tensor, keepdim: bool = False, eps: float = 1e-6) -> torch.Tensor:
- """Compute the Euclidean distance between two set of n-dimensional points.
+ """Compute the Euclidean distance between paired points along the last dimension.
+
+ ``d = euclidean_distance(x, y)`` takes points ``x`` and ``y`` of shape ``(*, N)`` and returns the distance between
+ each pair, with the last dimension removed, or kept as size 1 with ``keepdim=True``. The points are paired, not
+ compared all against all: the leading dimensions broadcast, so pass ``x[:, None]`` and ``y[None]`` for an
+ ``(M, K)`` distance matrix, or use :func:`torch.cdist`. Coincident points give exactly ``0`` with a zero gradient.
More: https://en.wikipedia.org/wiki/Euclidean_distance
@@ -423,6 +562,14 @@ def euclidean_distance(x: torch.Tensor, y: torch.Tensor, keepdim: bool = False,
keepdim: whether to keep the dimension after reduction.
eps: unused; kept for backward compatibility. The result is the exact Euclidean distance.
+ Returns:
+ the distances, with shape ``(*)``, or ``(*, 1)`` with ``keepdim=True``.
+
+ See Also:
+ - :func:`batched_squared_norm`: the squared norm, e.g. of ``x - y`` for squared distances without the root.
+ - :func:`batched_dot_product`: the dot product along the last dimension.
+ - :func:`point_line_distance`: the distance from 2D points to lines.
+
"""
KORNIA_CHECK_SHAPE(x, ["*", "N"])
KORNIA_CHECK_SHAPE(y, ["*", "N"])
diff --git a/kornia/geometry/line.py b/kornia/geometry/line.py
index 6536e7b9e..4f8fe5cb9 100644
--- a/kornia/geometry/line.py
+++ b/kornia/geometry/line.py
@@ -15,6 +15,24 @@
# limitations under the License.
#
+r"""Lines in any dimension as origin plus direction: distances, plane intersection and fitting.
+
+A line is :math:`l(t) = o + t d`, with an origin ``o`` and a direction ``d`` of shape ``(D,)`` for one line or
+``(B, D)`` for ``B`` lines, in any dimension ``D``. The constructor stores the direction as given; the distances and
+the projection need it to be unit, which :meth:`~kornia.geometry.line.ParametrizedLine.through` and :func:`fit_line`
+return. ``kornia.geometry.ray.Ray`` is the same class as :class:`ParametrizedLine`.
+
+Which function to use:
+
+- A line from an origin and a direction, or through two points, with the points along it, the projection and distance
+ of points, and the intersection with a 3D plane: :class:`ParametrizedLine`.
+- The line that best fits each of ``B`` sets of points, by total least squares, optionally weighted: :func:`fit_line`.
+- A plane in 3D, and the plane that best fits 3D points: :class:`~kornia.geometry.plane.Hyperplane` and
+ :func:`~kornia.geometry.plane.fit_plane`.
+- The distance from 2D points to lines given as coefficients :math:`ax + by + c = 0`, without a line object:
+ :func:`~kornia.geometry.linalg.point_line_distance`.
+"""
+
# kornia.geometry.line module inspired by Eigen::geometry::ParametrizedLine
# https://gitlab.com/libeigen/eigen/-/blob/master/Eigen/src/Geometry/ParametrizedLine.h
import math
@@ -35,10 +53,17 @@ __all__ = ["ParametrizedLine", "fit_line"]
class ParametrizedLine(nn.Module):
- r"""Class that describes a parametrized line.
+ r"""Represent a line o + t d in any dimension, with distances and plane intersection.
+
+ ``line = ParametrizedLine(origin, direction)`` stores an origin and a direction of shape ``(D,)``, or ``(B, D)``
+ for ``B`` lines, in any dimension ``D``, as given; ``ParametrizedLine.through(p0, p1)`` builds the line from
+ ``p0`` towards ``p1`` with a unit direction. ``line.point_at(t)`` returns the points :math:`o + t d`; for a unit
+ direction, ``line.projection(p)`` returns the closest points on the line ``(..., D)`` and ``line.distance(p)``
+ the distances ``(...)`` of points ``(..., D)``; ``line.intersect(plane)`` returns where a 3D line meets a
+ :class:`~kornia.geometry.plane.Hyperplane`. To fit the line to many points, use :func:`fit_line`; for 2D lines
+ given as coefficients :math:`ax + by + c = 0`, :func:`~kornia.geometry.linalg.point_line_distance`.
- A parametrized line is defined by an origin point :math:`o` and a
- direction vector :math:`d` such that the line corresponds to the set
+ The line is the set
.. math::
@@ -49,16 +74,49 @@ class ParametrizedLine(nn.Module):
length. :meth:`through` and :func:`fit_line` return a unit direction; after :meth:`through`, ``t`` is the
Euclidean distance from ``p0``.
- :meth:`projection`, :meth:`squared_distance` and :meth:`distance` require a unit ``direction``.
+ - Points broadcast against the lines as torch broadcasts ``(..., D)`` against ``(B, D)``: the axis before
+ the coordinates pairs with the line batch. One line, or a batch of one, takes points of any leading shape;
+ a batch of ``B > 1`` takes ``(B, D)``, one point per line, a single ``(D,)`` point, or ``(N, B, D)`` for
+ ``N`` points per line. ``(B, N, D)`` is not ``N`` points per line: it raises unless ``N`` is ``B`` or 1.
+ With ``N = B``, line ``j`` meets ``point[:, j]``; with ``N = 1``, every line meets every point and the
+ result is ``(B, B)``.
- :meth:`intersect` returns ``(lambda, point)`` with ``point = point_at(lambda)``: ``lambda`` is in units of
- the stored direction, and the plane's normal need not be unit.
+ the stored direction, and the plane's normal need not be unit. It takes a 3D line. A batch of ``B`` lines
+ meets one plane, a batch of ``B`` planes or a batch of one, and one unbatched line or a batch of one meets a
+ batch of planes of any size.
+ - ``origin`` and ``direction`` are registered module state, so ``state_dict()`` and ``.to()`` include them.
+ Outside graph capture, a floating-point tensor that takes no part in autograd (no ``grad_fn`` and
+ ``requires_grad=False``) becomes an ``nn.Parameter``, so a line built from plain tensors, including the
+ result of :meth:`through` and :func:`fit_line` on such tensors, returns points and distances that require
+ grad. A caller's ``nn.Parameter`` is kept; any other floating-point tensor, such as a leaf that requires
+ grad or a tensor with autograd history, is stored as a buffer and keeps its autograd link; an integer or
+ bool tensor raises ``TypeCheckError``. ``copy.deepcopy`` of a line whose state carries autograd history
+ raises ``RuntimeError`` (`#4923 <https://github.com/kornia/kornia/issues/4923>`_).
+
+ See Also:
+ - :func:`fit_line`: fits the line to a set of points by total least squares, instead of taking an origin
+ and a direction.
+ - :class:`~kornia.geometry.plane.Hyperplane`: a plane in 3D, :math:`n \cdot x + d = 0`, the operand of
+ :meth:`intersect`.
+ - :func:`~kornia.geometry.linalg.point_line_distance`: the distance from 2D points to lines given as
+ coefficients :math:`(a, b, c)`, without a line object.
+
+ Example:
+ >>> line = ParametrizedLine.through(torch.tensor([0.0, 0.0]), torch.tensor([3.0, 4.0]))
+ >>> line.point_at(10.0)
+ tensor([6., 8.], grad_fn=<AddBackward0>)
+ >>> line.distance(torch.tensor([[4.0, 3.0], [-3.0, -4.0]]))
+ tensor([1.4000, 0.0000], grad_fn=<LinalgVectorNormBackward0>)
+
"""
def __init__(self, origin: torch.Tensor, direction: torch.Tensor) -> None:
"""Initialize a parametrized line of direction and origin.
Args:
- origin: any point on the line of any dimension.
- direction: the direction vector of the line, of the same dimension.
+ origin: a point on the line, ``(D,)`` or ``(B, D)``, in any dimension ``D``.
+ direction: the direction vector of the line, of the same shape; stored as given, neither normalised nor
+ checked against ``origin``.
Raises:
TypeCheckError: if ``origin`` or ``direction`` is an integer or bool tensor.
@@ -96,19 +154,25 @@ class ParametrizedLine(nn.Module):
return self._direction
def dim(self) -> int:
- """Return the dimension in which the line holds."""
+ """Return the dimension ``D`` of the space the line lies in, the size of the last axis of the direction."""
return self.direction.shape[-1]
@classmethod
def through(cls, p0: torch.Tensor, p1: torch.Tensor) -> "ParametrizedLine":
"""Construct a parametrized line going from a point :math:`p0` to :math:`p1`.
+ The origin is ``p0`` and the direction the unit vector from ``p0`` towards ``p1``, so ``point_at(t)`` is the
+ point at Euclidean distance ``t`` from ``p0``. The norm is floored at ``1e-12``: points closer than that pass
+ the check below but get a direction of length ``|p1 - p0| / 1e-12``, not a unit one, which scales
+ ``point_at`` and the projections and distances that assume a unit direction. The check that the points differ
+ is skipped under ``torch.compile`` and by ``disable_checks()``, like every kornia value check.
+
Args:
- p0: tensor with first point ``(B, D)`` where `D` is the point dimension.
- p1: tensor with second point ``(B, D)`` where `D` is the point dimension.
+ p0: tensor with first point ``(B, D)`` or ``(D,)``, where ``D`` is the point dimension.
+ p1: tensor with second point of the same shape.
Raises:
- ValueCheckError: if ``p0`` and ``p1`` coincide, so the line has no direction.
+ ValueCheckError: if ``p0`` and ``p1`` coincide in any row, so the line has no direction.
Example:
>>> p0 = torch.tensor([0.0, 0.0])
@@ -130,12 +194,12 @@ class ParametrizedLine(nn.Module):
"""Get the point at :math:`t` along this line.
Args:
- t: step along the line: a number or a 0-d tensor for every row, one step per row of shape
- ``(B,)`` or ``(B, 1)`` for a line of shape ``(B, D)``, or ``N`` steps of shape ``(N,)`` or
- ``(N, 1)`` for a line of shape ``(D,)``, giving ``(N, D)`` points.
+ t: step along the line, in units of the direction's length: a number or a 0-d tensor for every row,
+ one step per row of shape ``(B,)`` or ``(B, 1)`` for a line of shape ``(B, D)``, or ``N`` steps of
+ shape ``(N,)`` or ``(N, 1)`` for a line of shape ``(D,)``, giving ``(N, D)`` points.
Return:
- tensor with the point.
+ the points :math:`o + t d`, ``(D,)`` or ``(B, D)``, or ``(N, D)`` for an ``(N,)`` or ``(N, 1)`` step.
Example:
>>> p0 = torch.tensor([0.0, 0.0])
@@ -153,8 +217,14 @@ class ParametrizedLine(nn.Module):
def projection(self, point: torch.Tensor) -> torch.Tensor:
"""Return the projection of a point onto the line.
+ The projection is the closest point on the line only for a unit direction.
+
Args:
- point: the point to be projected.
+ point: the point to be projected, ``(..., D)``, broadcast against the lines as the class conventions
+ say.
+
+ Return:
+ the projected points, of the broadcast shape ``(..., D)``.
"""
return self.origin + batched_dot_product(self.direction, point - self.origin)[..., None] * self.direction
@@ -162,8 +232,13 @@ class ParametrizedLine(nn.Module):
def squared_distance(self, point: torch.Tensor) -> torch.Tensor:
"""Return the squared distance of a point to its projection onto the line.
+ It is the squared Euclidean distance to the line only for a unit direction.
+
Args:
- point: the point to calculate the distance onto the line.
+ point: the point to calculate the distance onto the line, ``(..., D)``.
+
+ Return:
+ the squared distances, of the broadcast leading shape ``(...)``.
"""
perp = self._perpendicular(point)
return torch.sum(perp * perp, dim=-1)
@@ -171,8 +246,13 @@ class ParametrizedLine(nn.Module):
def distance(self, point: torch.Tensor) -> torch.Tensor:
"""Return the distance of a point to its projection onto the line.
+ It is the Euclidean distance to the line only for a unit direction.
+
Args:
- point: the point to calculate the distance onto the line.
+ point: the point to calculate the distance onto the line, ``(..., D)``.
+
+ Return:
+ the distances, of the broadcast leading shape ``(...)``.
"""
return torch.linalg.vector_norm(self._perpendicular(point), dim=-1)
@@ -196,8 +276,10 @@ class ParametrizedLine(nn.Module):
eps: absolute threshold on ``|normal . direction|`` below which the line counts as parallel to the plane.
Return:
- - the lambda value used to compute the look at point.
- - the intersected point.
+ The pair ``(lambda, point)``. ``lambda`` is the line parameter of the intersection in units of the stored
+ direction, ``()`` for one unbatched line and one unbatched plane, and ``(B,)`` for the broadcast batch,
+ which for one unbatched line is the batch of the planes. ``point`` is the intersection
+ ``point_at(lambda)``, ``(3,)`` or ``(B, 3)`` with the same batch.
Note:
If the line is parallel to the plane (``|normal . direction| < eps``) there is no unique
@@ -347,7 +429,14 @@ def _reject_degenerate_line(points: torch.Tensor, weights: Optional[torch.Tensor
def fit_line(points: torch.Tensor, weights: Optional[torch.Tensor] = None) -> ParametrizedLine:
- r"""Fit a line from a set of points by total least squares.
+ r"""Fit a line to each of B point sets in any dimension by weighted total least squares.
+
+ ``line = fit_line(points, weights)`` takes ``B`` sets of ``N`` points ``(B, N, D)`` in any dimension ``D``, and
+ optional non-negative per-point weights ``(B, N)``, and returns a :class:`ParametrizedLine` whose origin and unit
+ direction are ``(B, D)``: for each set, the line through the centroid that minimises the sum of squared
+ perpendicular distances, both weighted when ``weights`` are given. A single set ``(N, D)`` raises ``ShapeError``:
+ pass ``points[None]``. Every point with a positive weight pulls on the fit, so down-weight outliers or remove them
+ first. For a plane through 3D points, use :func:`~kornia.geometry.plane.fit_plane` instead.
Convention:
- Returns a :class:`ParametrizedLine` (see its conventions) through the centroid of the points, weighted
@@ -356,20 +445,36 @@ def fit_line(points: torch.Tensor, weights: Optional[torch.Tensor] = None) -> Pa
- For 2-D points the direction is computed in closed form and has a non-negative x component, so an exactly
vertical line gets the direction ``(0, 1)`` up to rounding. For :math:`D \ge 3` it is the principal
direction of the scatter matrix, whose sign is not specified.
+ - The output has the promoted dtype of ``points`` and ``weights``; float16 and bfloat16 inputs are fitted in
+ float32 and the line is cast back.
Args:
points: tensor containing a batch of sets of n-dimensional points. The expected
shape of the tensor is ``(B, N, D)``.
weights: per-point weights, used in the centroid and in the fit. The expected
- shape of the tensor is ``(B, N)``.
+ shape of the tensor is ``(B, N)``; pass that shape. For ``N > 1`` a ``(B, 1)`` tensor is not rejected:
+ it broadcasts, skips the weight checks under Raises and in general gives a wrong line, wrong in a
+ different way for ``D = 2`` and ``D >= 3`` (`#6525 <https://github.com/kornia/kornia/issues/6525>`_).
Return:
The fitted line, with origin and direction of shape ``(B, D)``.
Raises:
+ TypeCheckError: if ``points`` or ``weights`` is not a tensor.
+ ShapeError: if ``points`` is not 3-D, or ``weights`` is not 2-D.
+ BaseError: if ``weights`` has another batch size than ``points``.
+ RuntimeError: if ``weights`` has another number of points than ``points``, other than one or zero, from torch.
+ IndexError: if ``weights`` has zero points, from torch.
ValueCheckError: if the points do not determine a line — fewer than two points,
all points identical, or (with weights) a zero weight sum, a negative weight, or all
- points with positive weight identical.
+ points with positive weight identical. Like every kornia value check, these are skipped under
+ ``torch.compile`` and by ``disable_checks()``.
+
+ See Also:
+ - :class:`ParametrizedLine`: the returned line, which can also be built from an origin and a direction or
+ through two points.
+ - :func:`~kornia.geometry.plane.fit_plane`: the same total least squares for a plane through 3D points,
+ unweighted, and for a set ``(N, 3)`` as well as a batch.
Example:
>>> points = torch.rand(2, 10, 3)
diff --git a/kornia/geometry/plane.py b/kornia/geometry/plane.py
index f983d149d..00b726a58 100644
--- a/kornia/geometry/plane.py
+++ b/kornia/geometry/plane.py
@@ -15,6 +15,24 @@
# limitations under the License.
#
+r"""Planes in 3D as n . x + d = 0: signed distances, projection and fitting.
+
+A plane is :math:`n \cdot x + d = 0`, with a normal ``n`` of shape ``(3,)``, or ``(B, 3)`` for ``B`` planes, held in a
+:class:`~kornia.geometry.vector.Vector3`, and an offset ``d`` of shape ``()`` or ``(B,)`` in a
+:class:`~kornia.geometry.vector.Scalar`. The signed distance :math:`n \cdot x + d` is positive on the side the normal
+points to, and a Euclidean distance only for a unit normal, which :meth:`~kornia.geometry.plane.Hyperplane.through`
+and :func:`fit_plane` return. :func:`~kornia.geometry.depth.depth_from_plane_equation` writes the plane as
+:math:`n \cdot X = d` instead.
+
+Which function to use:
+
+- A plane from a normal and an offset, a normal and a point, or three points, with the signed and absolute distances
+ and the projection of points: :class:`Hyperplane`.
+- The plane that best fits 3D points by total least squares: :func:`fit_plane`.
+- Where a line meets a plane: :meth:`~kornia.geometry.line.ParametrizedLine.intersect`.
+- The depth of given pixels on a known plane: :func:`~kornia.geometry.depth.depth_from_plane_equation`.
+"""
+
# kornia.geometry.plane module inspired by Eigen::geometry::Hyperplane
# https://gitlab.com/libeigen/eigen/-/blob/master/Eigen/src/Geometry/Hyperplane.h
@@ -34,7 +52,16 @@ __all__ = ["Hyperplane", "fit_plane"]
class Hyperplane(nn.Module):
- r"""Represent a plane in 3D space by its normal :math:`n` and offset :math:`d`.
+ r"""Represent 3D planes n . x + d = 0 by a normal and an offset, with signed distances.
+
+ ``plane = Hyperplane(Vector3(n), Scalar(d))`` stores the plane :math:`n \cdot x + d = 0` from a normal ``(3,)``
+ or ``(B, 3)`` in a :class:`~kornia.geometry.vector.Vector3` and an offset ``()`` or ``(B,)`` in a
+ :class:`~kornia.geometry.vector.Scalar`; plain tensors raise ``TypeCheckError``. :meth:`from_vector` builds it
+ from a normal and a point on it, and :meth:`through` from three points with a unit normal.
+ ``plane.signed_distance(p)`` returns :math:`n \cdot p + d` for points ``(..., 3)`` as a ``Scalar``,
+ :meth:`abs_distance` its absolute value, and :meth:`projection` the closest points as a ``Vector3`` for a unit
+ normal. To fit a plane to many points, use :func:`fit_plane`; to intersect it with a line,
+ :meth:`~kornia.geometry.line.ParametrizedLine.intersect`.
Convention:
- The plane is :math:`n \cdot x + d = 0`, and :meth:`signed_distance` is :math:`n \cdot x + d`: positive on
@@ -49,15 +76,38 @@ class Hyperplane(nn.Module):
:math:`p_0 \to p_1 \to p_2`. Swapping two points flips it.
- The normal and offset are registered module state under the keys ``_n`` and ``_d``, so ``state_dict()``,
``load_state_dict()`` and ``.to()`` include both tensors. As in
- :class:`~kornia.geometry.line.ParametrizedLine`, a floating-point tensor without autograd history becomes
- an ``nn.Parameter``: a plane built from plain tensors, including the result of :meth:`through` and
- :func:`fit_plane` on points without autograd history, has trainable parameters, and the distances and
- projections it returns require grad. A caller's ``nn.Parameter`` is kept, a tensor with autograd history
- keeps that history, and an integer tensor is stored as a buffer (ParametrizedLine rejects one).
+ :class:`~kornia.geometry.line.ParametrizedLine`, outside graph capture a floating-point tensor that takes
+ no part in autograd (no ``grad_fn`` and ``requires_grad=False``) becomes an ``nn.Parameter``: a plane built
+ from plain tensors, including the result of :meth:`through` and :func:`fit_plane` on points without
+ autograd history, has trainable parameters, and the distances and projections it returns require grad. A
+ caller's ``nn.Parameter`` is kept; any other floating-point tensor, such as a leaf that requires grad or a
+ tensor with autograd history, is stored as a buffer and keeps its autograd link; and an integer tensor is
+ stored as a buffer (ParametrizedLine rejects one).
+ ``copy.deepcopy`` of a plane whose state carries autograd history raises ``RuntimeError``
+ (`#4923 <https://github.com/kornia/kornia/issues/4923>`_).
+ - Points broadcast against the planes as torch broadcasts ``(..., 3)`` against ``(B, 3)``: the axis before
+ the coordinates pairs with the plane batch. One plane, or a batch of one, takes points of any leading
+ shape; a batch of ``B > 1`` takes ``(B, 3)``, one point per plane, a single ``(3,)`` point, or
+ ``(N, B, 3)`` for ``N`` points per plane. ``(B, N, 3)`` is not ``N`` points per plane: it raises unless
+ ``N`` is ``B`` or 1. With ``N = B``, plane ``j`` meets ``p[:, j]``; with ``N = 1``, every plane meets every
+ point and the result is ``(B, B)``.
Args:
n: The normal vector :math:`n`, a :class:`~kornia.geometry.vector.Vector3`.
d: The offset :math:`d`, a :class:`~kornia.geometry.vector.Scalar`.
+
+ See Also:
+ - :func:`fit_plane`: fits the plane to a set of 3D points by total least squares, instead of taking a normal
+ and an offset.
+ - :class:`~kornia.geometry.line.ParametrizedLine`: a line in any dimension; its
+ :meth:`~kornia.geometry.line.ParametrizedLine.intersect` returns where a 3D line meets this plane.
+ - :func:`~kornia.geometry.depth.depth_from_plane_equation`: the depth of pixels on a plane given as plain
+ tensors, with the offset of the opposite sign.
+
+ Example:
+ >>> plane = Hyperplane(Vector3(torch.tensor([0.0, 0.0, 1.0])), Scalar(torch.tensor(-2.0)))
+ >>> plane.signed_distance(torch.tensor([[1.0, 1.0, 5.0], [0.0, 0.0, 0.0]])).data
+ tensor([ 3., -2.], grad_fn=<AddBackward0>)
"""
def __init__(self, n: Vector3, d: Scalar) -> None:
@@ -132,11 +182,12 @@ class Hyperplane(nn.Module):
# https://gitlab.com/libeigen/eigen/-/blob/master/Eigen/src/Geometry/Hyperplane.h#L145
def signed_distance(self, p: Vector3 | torch.Tensor) -> Scalar:
- """Compute signed distances from points to the hyperplane.
+ r"""Compute signed distances from points to the hyperplane.
The sign is determined by the stored normal vector. Points in the
normal direction have positive values; points on the opposite side have
- negative values; points on the plane evaluate to zero.
+ negative values; points on the plane evaluate to zero. The value is
+ :math:`n \cdot p + d`, a Euclidean distance only for a unit normal.
Args:
p: Point or batch of points as a :class:`~kornia.geometry.vector.Vector3`
@@ -151,16 +202,16 @@ class Hyperplane(nn.Module):
# https://gitlab.com/libeigen/eigen/-/blob/master/Eigen/src/Geometry/Hyperplane.h#L154
def projection(self, p: Vector3 | torch.Tensor) -> Vector3:
- """Project points onto the hyperplane along the normal direction.
+ r"""Project points onto the hyperplane along the normal direction.
Args:
p: Point or batch of points as a :class:`~kornia.geometry.vector.Vector3`
or :class:`torch.Tensor`.
Returns:
- :class:`~kornia.geometry.vector.Vector3` containing the closest
- point on the hyperplane for each input point, preserving leading
- batch dimensions.
+ :class:`~kornia.geometry.vector.Vector3`, also for a tensor input, containing
+ :math:`p - (n \cdot p + d) n` for each input point, preserving leading
+ batch dimensions: the closest point on the hyperplane for a unit normal.
"""
dist = self.signed_distance(p)
return p - dist.data[..., None] * self.normal
@@ -172,7 +223,8 @@ class Hyperplane(nn.Module):
"""Create a hyperplane from a normal and one point on the plane.
Args:
- n: Normal vector :math:`n` defining the plane orientation.
+ n: Normal vector :math:`n` defining the plane orientation, a
+ :class:`~kornia.geometry.vector.Vector3` stored as given, not normalised.
e: Point :math:`e` that lies on the target plane.
Returns:
@@ -185,7 +237,7 @@ class Hyperplane(nn.Module):
@classmethod
def through(cls, p0: torch.Tensor, p1: torch.Tensor, p2: Optional[torch.Tensor] = None) -> "Hyperplane":
- """Construct the 3D plane through three points.
+ r"""Construct the 3D plane through three points.
Only the three-point form is supported: :class:`Hyperplane` stores its normal as a
:class:`~kornia.geometry.vector.Vector3`, so it cannot represent a 2D line, and calling
@@ -198,13 +250,15 @@ class Hyperplane(nn.Module):
``None`` only exists so that a two-point call fails with a clear error.
Returns:
- :class:`Hyperplane` passing through the three points.
+ :class:`Hyperplane` passing through the three points, with a unit normal along
+ :math:`(p_2 - p_0) \times (p_1 - p_0)`.
Raises:
BaseError: if ``p2`` is omitted, or if the points are not ``(..., 3)`` tensors of the
same shape.
ValueCheckError: if the three points are collinear (or coincide), so they do not
- determine a plane.
+ determine a plane. Like every kornia value check, it is skipped under ``torch.compile``
+ and by ``disable_checks()``.
"""
if p2 is None:
# Raised directly rather than through KORNIA_CHECK, so it still fires with checks disabled.
@@ -282,11 +336,21 @@ def _rank_tolerance(dtype: torch.dtype) -> float:
# TODO: factor to avoid duplicated from line.py
# https://github.com/strasdat/Sophus/blob/23.04-beta/cpp/sophus/geometry/fit_plane.h
def fit_plane(points: Vector3) -> Hyperplane:
- """Fit a plane from a set of points using SVD.
+ """Fit a 3D plane to a set of points by total least squares, through their centroid.
+
+ ``plane = fit_plane(points)`` takes ``N >= 3`` points ``(N, 3)``, or ``B`` sets of them ``(B, N, 3)``, as a
+ tensor or a :class:`~kornia.geometry.vector.Vector3`, and returns a :class:`Hyperplane` with a unit normal ``(3,)``
+ or ``(B, 3)`` and an offset ``()`` or ``(B,)``: the plane through the centroid that minimises the sum of squared
+ perpendicular distances to the points. Every point counts equally; there are no weights. For the plane through
+ exactly three points with an oriented normal, use :meth:`Hyperplane.through` instead; for a line,
+ :func:`~kornia.geometry.line.fit_line`.
Convention:
- Returns a :class:`Hyperplane` (see its conventions) through the centroid of the points, with a unit normal
- of unspecified sign. Each batch row is fitted on its own.
+ - Returns a :class:`Hyperplane` (see its conventions) through the centroid of the points, with a unit normal
+ of unspecified sign. Each batch row is fitted on its own.
+ - The points are collinear, and rejected, when the second singular value of the centred points is at most
+ ``8 * eps`` of the input dtype times the first, a test that does not depend on their scale.
+ - float16 and bfloat16 points are fitted in float32 and the plane is cast back to the input dtype.
Args:
points: a tensor or a :class:`~kornia.geometry.vector.Vector3` of 3D points, of shape ``(N, 3)`` or
@@ -296,8 +360,22 @@ def fit_plane(points: Vector3) -> Hyperplane:
The computed hyperplane object.
Raises:
- ValueCheckError: if fewer than three points are given, or the points are collinear,
- so they do not determine a plane.
+ TypeError: if ``points`` does not end in 3 coordinates.
+ ValueCheckError: if fewer than three points are given, the points are all identical, or they are
+ collinear, so they do not determine a plane. Like every kornia value check, these are skipped under
+ ``torch.compile`` and by ``disable_checks()``.
+
+ See Also:
+ - :class:`Hyperplane`: the returned plane; its :meth:`Hyperplane.through` takes exactly three points and
+ orients the normal by their order.
+ - :func:`~kornia.geometry.line.fit_line`: the same total least squares for a line, in any dimension and with
+ optional weights, for a batch ``(B, N, D)`` only.
+
+ Example:
+ >>> points = torch.tensor([[0.0, 0.0, 1.0], [1.0, 0.0, 1.0], [0.0, 1.0, 1.0], [1.0, 1.0, 1.0]])
+ >>> plane = fit_plane(points)
+ >>> plane.normal.data.abs(), plane.signed_distance(torch.tensor([0.0, 0.0, 3.0])).data.abs()
+ (tensor([0., 0., 1.], grad_fn=<AbsBackward0>), tensor(2., grad_fn=<AbsBackward0>))
"""
# TODO: fix to support more type check here
diff --git a/kornia/geometry/pointcloud.py b/kornia/geometry/pointcloud.py
index 7deef7c29..427056d84 100644
--- a/kornia/geometry/pointcloud.py
+++ b/kornia/geometry/pointcloud.py
@@ -15,6 +15,22 @@
# limitations under the License.
#
+"""Read and write point clouds as ASCII or binary PLY files of x, y, z vertices.
+
+The writers store one ``vertex`` per row of a channels-last ``(*, 3)`` tensor, with only the ``double`` properties
+``x``, ``y`` and ``z``. The readers parse the PLY header, read the ``x``, ``y`` and ``z`` of the ``vertex`` element
+whatever their scalar type, skip other vertex properties such as normals and colours and other elements such as
+faces (each reader says which ``list`` properties it rejects), and return a float32 ``(N, 3)`` tensor on the CPU.
+Every function takes a ``str`` or ``os.PathLike[str]`` ending in ``.ply``.
+
+Which function to use:
+
+- A text file: :func:`save_pointcloud_ply` writes one, :func:`load_pointcloud_ply` reads one.
+- A binary file: :func:`save_pointcloud_ply_binary` writes one in little-endian float64, 24 bytes per point, and
+ :func:`load_pointcloud_ply_binary` reads one in either byte order and with any PLY scalar type.
+- The points of a depth map, to write: :func:`~kornia.geometry.depth.depth_to_3d_v2` returns them channels-last.
+"""
+
from __future__ import annotations
import array
@@ -35,7 +51,13 @@ __all__ = [
def save_pointcloud_ply(filename: str | os.PathLike[str], pointcloud: torch.Tensor) -> None:
- r"""Save to disk a pointcloud in ASCII PLY format.
+ r"""Write a point cloud to an ASCII PLY file as double-precision x, y, z vertices.
+
+ ``save_pointcloud_ply(filename, pointcloud)`` writes the points of a channels-last tensor ``(*, 3)`` with at least
+ two dimensions, on any device, to a text PLY file, replacing any existing file, and returns ``None``. The file
+ holds the coordinates only: no normals, colours or faces. Read it back with :func:`load_pointcloud_ply`, which
+ returns float32, so float64 coordinates come back rounded to float32; for a binary file, use
+ :func:`save_pointcloud_ply_binary` instead.
Convention:
- The file holds one ``vertex`` per row of ``pointcloud.reshape(-1, 3)``, in order and with none dropped,
@@ -51,6 +73,16 @@ def save_pointcloud_ply(filename: str | os.PathLike[str], pointcloud: torch.Tens
filename: path of the ``.ply`` file to write, a ``str`` or ``os.PathLike[str]``.
pointcloud: tensor of shape ``(*, 3)`` with at least two dimensions, where the last dimension is the
point :math:`(X, Y, Z)`.
+
+ Raises:
+ TypeError: if ``filename`` is not a ``str`` or ``os.PathLike[str]`` ending in ``.ply``, if ``pointcloud`` is
+ not a tensor, or if it has fewer than two dimensions or a last dimension other than 3.
+
+ See Also:
+ - :func:`load_pointcloud_ply`: reads the file back, as a float32 ``(N, 3)`` tensor.
+ - :func:`save_pointcloud_ply_binary`: writes the same vertices as little-endian float64 instead of text.
+ - :func:`~kornia.geometry.depth.depth_to_3d_v2`: returns the channels-last ``(*, H, W, 3)`` points of a
+ depth map, ready to write.
"""
filename = os.fspath(filename)
if not (isinstance(filename, str) and filename.lower().endswith(".ply")):
@@ -93,7 +125,12 @@ def save_pointcloud_ply(filename: str | os.PathLike[str], pointcloud: torch.Tens
def save_pointcloud_ply_binary(filename: str | os.PathLike[str], pointcloud: torch.Tensor) -> None:
- r"""Save to disk a pointcloud in binary PLY format.
+ r"""Write a point cloud to a little-endian binary PLY file as float64 x, y, z vertices.
+
+ ``save_pointcloud_ply_binary(filename, pointcloud)`` writes the points of a channels-last tensor ``(*, 3)`` with
+ at least two dimensions, on any device, to a ``binary_little_endian`` PLY file, 24 bytes per point,
+ replacing any existing file, and returns ``None``. Read it back with :func:`load_pointcloud_ply_binary`; for a
+ text file, use :func:`save_pointcloud_ply` instead.
See the Convention block on :func:`save_pointcloud_ply`; the payload is little-endian float64.
@@ -101,6 +138,14 @@ def save_pointcloud_ply_binary(filename: str | os.PathLike[str], pointcloud: tor
filename: path of the ``.ply`` file to write, a ``str`` or ``os.PathLike[str]``.
pointcloud: tensor of shape ``(*, 3)`` with at least two dimensions, where the last dimension is the
point :math:`(X, Y, Z)`.
+
+ Raises:
+ TypeError: if ``filename`` is not a ``str`` or ``os.PathLike[str]`` ending in ``.ply``, if ``pointcloud`` is
+ not a tensor, or if it has fewer than two dimensions or a last dimension other than 3.
+
+ See Also:
+ - :func:`load_pointcloud_ply_binary`: reads the file back, as a float32 ``(N, 3)`` tensor.
+ - :func:`save_pointcloud_ply`: writes the same vertices as text instead.
"""
filename = os.fspath(filename)
if not (isinstance(filename, str) and filename.lower().endswith(".ply")):
@@ -281,7 +326,12 @@ def _check_ply_filename(filename: str) -> None:
def load_pointcloud_ply(filename: str | os.PathLike[str], header_size: Optional[int] = None) -> torch.Tensor:
- r"""Load from disk a pointcloud in ASCII PLY format.
+ r"""Read the x, y, z vertices of an ASCII PLY file into a float32 ``(N, 3)`` tensor.
+
+ ``points = load_pointcloud_ply(filename)`` reads an ASCII PLY file and returns the ``x``, ``y`` and ``z`` of its
+ ``N`` vertices as a float32 ``(N, 3)`` tensor on the CPU, so a float64 cloud written by
+ :func:`save_pointcloud_ply` comes back rounded to float32. For a binary PLY file, use
+ :func:`load_pointcloud_ply_binary` instead.
The header is parsed up to ``end_header``; the number of points comes from the
``element vertex N`` declaration and the coordinates from the ``x``, ``y`` and ``z``
@@ -291,11 +341,23 @@ def load_pointcloud_ply(filename: str | os.PathLike[str], header_size: Optional[
Args:
filename: path of an existing ``.ply`` file, a ``str`` or ``os.PathLike[str]``.
- header_size: deprecated and ignored; the header is parsed instead of skipped.
+ header_size: deprecated and ignored, with a ``DeprecationWarning`` when given; the header is parsed instead
+ of skipped.
Return:
tensor containing the loaded points with shape ``(N, 3)`` in ``float32``, where
:math:`N` is the declared vertex count.
+
+ Raises:
+ TypeError: if ``filename`` is not a ``str`` or ``os.PathLike[str]`` ending in ``.ply``.
+ ValueError: if the file does not exist or does not start with a well-formed PLY header; if it is a binary
+ PLY file; if it has no ``vertex`` element, no ``x``, ``y`` or ``z`` vertex property, or a ``list`` vertex
+ property; or if the payload ends early, a vertex has fewer values than declared properties, or a
+ coordinate is not a number.
+
+ See Also:
+ - :func:`save_pointcloud_ply`: writes the ASCII files this function reads.
+ - :func:`load_pointcloud_ply_binary`: reads a binary PLY file, little- or big-endian.
"""
filename = os.fspath(filename)
_check_ply_filename(filename)
@@ -341,7 +403,12 @@ def load_pointcloud_ply(filename: str | os.PathLike[str], header_size: Optional[
def load_pointcloud_ply_binary(filename: str | os.PathLike[str], header_size: Optional[int] = None) -> torch.Tensor:
- r"""Load from disk a pointcloud in binary PLY format.
+ r"""Read the x, y, z vertices of a big- or little-endian binary PLY file as ``(N, 3)``.
+
+ ``points = load_pointcloud_ply_binary(filename)`` reads a ``binary_little_endian`` or ``binary_big_endian`` PLY
+ file and returns the ``x``, ``y`` and ``z`` of its ``N`` vertices as a float32 ``(N, 3)`` tensor on the CPU, so a
+ float64 cloud written by :func:`save_pointcloud_ply_binary` comes back rounded to float32. For an ASCII PLY file,
+ use :func:`load_pointcloud_ply` instead.
The header is parsed up to ``end_header``; the number of points comes from the
``element vertex N`` declaration and the coordinates from the ``x``, ``y`` and ``z``
@@ -353,11 +420,23 @@ def load_pointcloud_ply_binary(filename: str | os.PathLike[str], header_size: Op
Args:
filename: path of an existing ``.ply`` file, a ``str`` or ``os.PathLike[str]``.
- header_size: deprecated and ignored; the header is parsed instead of skipped.
+ header_size: deprecated and ignored, with a ``DeprecationWarning`` when given; the header is parsed instead
+ of skipped.
Return:
tensor containing the loaded points with shape ``(N, 3)`` in ``float32``, where
:math:`N` is the declared vertex count.
+
+ Raises:
+ TypeError: if ``filename`` is not a ``str`` or ``os.PathLike[str]`` ending in ``.ply``.
+ ValueError: if the file does not exist or does not start with a well-formed PLY header; if it is an ASCII
+ PLY file; if it has no ``vertex`` element or no ``x``, ``y`` or ``z`` vertex property; if a ``list``
+ property is among the vertex properties or in an element before them; or if fewer bytes follow the
+ header than the vertices need.
+
+ See Also:
+ - :func:`save_pointcloud_ply_binary`: writes the little-endian files this function reads.
+ - :func:`load_pointcloud_ply`: reads an ASCII PLY file.
"""
filename = os.fspath(filename)
_check_ply_filename(filename)
diff --git a/kornia/geometry/quaternion.py b/kornia/geometry/quaternion.py
index c5a43f69e..187a38f9e 100644
--- a/kornia/geometry/quaternion.py
+++ b/kornia/geometry/quaternion.py
@@ -15,6 +15,30 @@
# limitations under the License.
#
+"""Quaternions in ``(w, x, y, z)`` order: Hamilton product, slerp, averaging and rotation conversions.
+
+A quaternion is stored real part first, with shape ``(*, 4)``, and ``q1 * q2`` is the Hamilton product, so
+``(q1 * q2).matrix()`` is ``q1.matrix() @ q2.matrix()``: the right operand acts first, as for the Lie groups of
+``kornia.geometry.liegroup``. ``q`` and ``-q`` are the same rotation. Nothing normalises the stored data; the
+conversions to a rotation read only its direction :math:`q / |q|`, within the limits stated on :class:`Quaternion`.
+
+Which function to use:
+
+- Quaternions as objects that multiply, invert, interpolate (``slerp``) and convert to and from rotation matrices,
+ rotation vectors and Euler angles: :class:`Quaternion`.
+- The mean rotation of many quaternions, optionally weighted, whatever their signs: :func:`average_quaternions`.
+- The same conversions on plain tensors, without an object:
+ :func:`~kornia.geometry.conversions.quaternion_to_rotation_matrix` and
+ :func:`~kornia.geometry.conversions.quaternion_to_axis_angle` from ``(*, 4)`` quaternions to ``(*, 3, 3)``
+ matrices and ``(*, 3)`` rotation vectors, and :func:`~kornia.geometry.conversions.rotation_matrix_to_quaternion`
+ and :func:`~kornia.geometry.conversions.axis_angle_to_quaternion` back;
+ :func:`~kornia.geometry.conversions.euler_from_quaternion` and
+ :func:`~kornia.geometry.conversions.quaternion_from_euler` for roll, pitch and yaw, with the quaternion as four
+ separate ``w``, ``x``, ``y``, ``z`` tensors.
+- A rotation that acts on points, with ``exp``, ``log`` and Jacobians: :class:`~kornia.geometry.liegroup.So3`; with a
+ translation: :class:`~kornia.geometry.liegroup.Se3`.
+"""
+
# kornia.geometry.quaternion module inspired by Eigen, Sophus-sympy, and PyQuaternion.
# https://github.com/strasdat/Sophus/blob/master/sympy/sophus/quaternion.py
# https://github.com/KieranWynn/pyquaternion/blob/master/pyquaternion/quaternion.py
@@ -39,10 +63,15 @@ from kornia.geometry.linalg import batched_dot_product
class Quaternion(nn.Module):
- r"""Base class to represent a Quaternion.
+ r"""Represent (w, x, y, z) quaternions: Hamilton product, slerp and rotation conversions.
- A quaternion is a four dimensional vector representation of a rotation transformation in 3d.
- See more: https://en.wikipedia.org/wiki/Quaternion
+ ``q = Quaternion(data)`` wraps a ``(*, 4)`` tensor of quaternions, real part first, without copying it;
+ :meth:`from_axis_angle`, :meth:`from_matrix`, :meth:`from_euler`, :meth:`identity` and :meth:`random` build one.
+ ``q1 * q2`` is the Hamilton product, ``q.inv()`` the inverse and ``q0.slerp(q1, t)`` the interpolation along the
+ shorter arc; ``q.matrix()`` returns the ``(*, 3, 3)`` rotation matrices and ``q.to_axis_angle()`` the ``(*, 3)``
+ rotation vectors of the direction :math:`q / |q|`. A quaternion does not act on points: for a rotation that maps
+ points and has ``exp``, ``log`` and Jacobians, use :class:`~kornia.geometry.liegroup.So3`; for the same
+ conversions on plain tensors, the functions of ``kornia.geometry.conversions``.
The general definition of a quaternion is given by:
@@ -50,8 +79,8 @@ class Quaternion(nn.Module):
Q = a + b \cdot \mathbf{i} + c \cdot \mathbf{j} + d \cdot \mathbf{k}
- Thus, we represent a rotation quaternion as a contiguous torch.Tensor structure to
- perform rigid bodies transformations:
+ A unit quaternion represents a rotation in 3D; see https://en.wikipedia.org/wiki/Quaternion. The last axis of
+ ``data`` holds:
.. math::
@@ -63,6 +92,7 @@ class Quaternion(nn.Module):
:ref:`Rotations and rigid motions <rotation-conventions>`.
- ``*`` between quaternions is the Hamilton product, so ``(q1 * q2).matrix()`` is
``q1.matrix() @ q2.matrix()``, ``q1 / q2`` is ``q1 * q2.inv()``, and ``q`` and ``-q`` are the same rotation.
+ A batch multiplies with a batch of the same size, a batch of one or an unbatched quaternion.
A float or tensor operand of ``+``, ``-``, ``*`` or ``/`` is the real quaternion ``[s, 0, 0, 0]``, and a
tensor of the shape of ``q.w`` holds one such scalar per quaternion of the batch.
- Nothing normalises the stored data or the results of ``*``, ``**`` and ``inv()``. ``matrix()``,
@@ -78,6 +108,20 @@ class Quaternion(nn.Module):
:func:`~kornia.geometry.conversions.normalize_quaternion`, so they do not give the rotation of its
direction, and the zero quaternion's ``matrix()`` is the identity.
+ See Also:
+ - :class:`~kornia.geometry.liegroup.So3`: the rotation group stored as a quaternion, which rotates points
+ and adds ``exp``, ``log`` and Jacobians.
+ - :func:`average_quaternions`: the mean rotation of many quaternions, instead of interpolating two.
+ - :func:`~kornia.geometry.conversions.quaternion_to_rotation_matrix`,
+ :func:`~kornia.geometry.conversions.rotation_matrix_to_quaternion`: :meth:`matrix` and :meth:`from_matrix`
+ on plain ``(*, 4)`` and ``(*, 3, 3)`` tensors.
+ - :func:`~kornia.geometry.conversions.quaternion_to_axis_angle`,
+ :func:`~kornia.geometry.conversions.axis_angle_to_quaternion`: :meth:`to_axis_angle` and
+ :meth:`from_axis_angle` on plain ``(*, 4)`` and ``(*, 3)`` tensors.
+ - :func:`~kornia.geometry.conversions.euler_from_quaternion`,
+ :func:`~kornia.geometry.conversions.quaternion_from_euler`: :meth:`to_euler` and :meth:`from_euler` on plain
+ tensors, with the quaternion as four separate ``w``, ``x``, ``y``, ``z`` tensors.
+
Example:
>>> q = Quaternion.identity(batch_size=4)
>>> q.data
@@ -335,7 +379,7 @@ class Quaternion(nn.Module):
@property
def data(self) -> torch.Tensor:
- """Return the underlying data with shape ``(B, 4)``."""
+ """Return the underlying data with shape ``(*, 4)``."""
return self._data
@property
@@ -345,7 +389,7 @@ class Quaternion(nn.Module):
@property
def real(self) -> torch.Tensor:
- """Return the real part with shape ``(B,)``.
+ """Return the real part with shape ``(*,)``.
Alias for :attr:`~kornia.geometry.quaternion.Quaternion.w`.
"""
@@ -353,20 +397,20 @@ class Quaternion(nn.Module):
@property
def vec(self) -> torch.Tensor:
- """Return the vector with the imaginary part with shape ``(B, 3)``."""
+ """Return the vector with the imaginary part with shape ``(*, 3)``."""
return self.data[..., 1:]
@property
def q(self) -> torch.Tensor:
- """Return the underlying data with shape ``(B, 4)``.
+ """Return the underlying data with shape ``(*, 4)``.
- Alias for :func:`~kornia.geometry.quaternion.Quaternion.data`
+ Alias for :attr:`~kornia.geometry.quaternion.Quaternion.data`.
"""
return self.data
@property
def scalar(self) -> torch.Tensor:
- """Return a scalar with the real with shape ``(B,)``.
+ """Return a scalar with the real with shape ``(*,)``.
Alias for :attr:`~kornia.geometry.quaternion.Quaternion.w`.
"""
@@ -374,32 +418,32 @@ class Quaternion(nn.Module):
@property
def w(self) -> torch.Tensor:
- """Return the :math:`q_w` with shape ``(B,)``."""
+ """Return the :math:`q_w` with shape ``(*,)``."""
return self.data[..., 0]
@property
def x(self) -> torch.Tensor:
- """Return the :math:`q_x` with shape ``(B,)``."""
+ """Return the :math:`q_x` with shape ``(*,)``."""
return self.data[..., 1]
@property
def y(self) -> torch.Tensor:
- """Return the :math:`q_y` with shape ``(B,)``."""
+ """Return the :math:`q_y` with shape ``(*,)``."""
return self.data[..., 2]
@property
def z(self) -> torch.Tensor:
- """Return the :math:`q_z` with shape ``(B,)``."""
+ """Return the :math:`q_z` with shape ``(*,)``."""
return self.data[..., 3]
@property
def shape(self) -> Tuple[int, ...]:
- """Return the shape of the underlying data with shape ``(B, 4)``."""
+ """Return the shape of the underlying data, ``(*, 4)``."""
return tuple(self.data.shape)
@property
def polar_angle(self) -> torch.Tensor:
- r"""Return the polar angle :math:`\arccos(w / |q|)` in :math:`[0, \pi]`, with shape ``(B,)``.
+ r"""Return the polar angle :math:`\arccos(w / |q|)` in :math:`[0, \pi]`, with shape ``(*,)``.
``q`` rotates by twice this angle about the axis ``vec``.
@@ -414,7 +458,7 @@ class Quaternion(nn.Module):
return torch.atan2(self.vec.norm(dim=-1), self.scalar)
def matrix(self) -> torch.Tensor:
- """Convert the quaternion to a rotation matrix of shape ``(B, 3, 3)``.
+ """Convert the quaternion to a rotation matrix of shape ``(*, 3, 3)``.
Example:
>>> q = Quaternion.identity()
@@ -432,7 +476,7 @@ class Quaternion(nn.Module):
"""Create a quaternion from a rotation matrix.
Args:
- matrix: the rotation matrix to convert of shape ``(B, 3, 3)``.
+ matrix: the rotation matrix to convert of shape ``(*, 3, 3)``.
check_rotation: if ``True``, raise ``ValueError`` unless every input
is a rotation matrix. The default ``False`` keeps the unchecked
behaviour, under which a reflection such as
@@ -489,7 +533,7 @@ class Quaternion(nn.Module):
"""Create a quaternion from axis-angle representation.
Args:
- axis_angle: rotation vector of shape ``(B, 3)``.
+ axis_angle: rotation vector of shape ``(*, 3)``.
Example:
>>> axis_angle = torch.tensor([[1., 0., 0.]])
@@ -564,7 +608,7 @@ class Quaternion(nn.Module):
device: Union[str, torch.device, None] = None,
dtype: Union[torch.dtype, None] = None,
) -> "Quaternion":
- """Create a random unit quaternion of shape ``(B, 4)``.
+ """Create a random unit quaternion of shape ``(B, 4)``, or ``(4,)`` without a ``batch_size``.
Uniformly distributed across the rotation space as per: http://planning.cs.uiuc.edu/node198.html
@@ -602,8 +646,8 @@ class Quaternion(nn.Module):
Args:
q1: second quaternion to be interpolated between.
t: interpolation ratio, ``0`` at ``self`` and ``1`` at ``q1``. It is not validated: values outside
- ``[0, 1]`` extrapolate along the same arc. A per-batch ratio has shape ``(B,)``, like
- ``self.w``, or ``(B, 1)``.
+ ``[0, 1]`` extrapolate along the same arc. A per-quaternion ratio has shape ``(*,)``, like
+ ``self.w``, or ``(*, 1)``.
Example:
>>> q0 = Quaternion.identity()
@@ -717,24 +761,44 @@ def _hamilton_product(left: "Quaternion", right: "Quaternion") -> "Quaternion":
def average_quaternions(Q: "Quaternion", w: Optional[torch.Tensor] = None) -> "Quaternion":
- r"""Compute (weighted) average of multiple quaternions.
+ r"""Average quaternions as their weighted chordal mean rotation, whatever their signs.
+
+ ``q_mean = average_quaternions(Q, w)`` takes a :class:`Quaternion` of ``M`` quaternions, ``(M, 4)``, and optional
+ non-negative weights ``w`` of shape ``(M,)``, uniform by default, and returns their mean rotation as a unit
+ :class:`Quaternion` of shape ``(1, 4)``. To interpolate between two rotations instead, use
+ :meth:`Quaternion.slerp`.
Convention:
- The chordal mean of scipy's ``Rotation.mean``: the eigenvector of
:math:`\sum_i w_i q_i q_i^\top / \sum_i w_i` with the largest eigenvalue. ``q_i`` and ``-q_i`` count the
same, and the sign of the result is arbitrary.
- - ``w`` need not sum to one: scaling it by a positive factor does not change the result. A negative weight,
- or weights that are all zero, raise ``ValueError``.
+ - ``w`` need not sum to one: scaling it by a positive factor does not change the result.
- The members are normalised first, so only their directions count: a member stored as ``3 * q`` weighs
the same as ``q``.
Args:
- Q (Quaternion): quaternion object containing data of shape (M, 4).
- w (torch.Tensor, optional): Weights of shape (M,). If None, uniform weights are used.
-
+ Q: quaternions of shape ``(M, 4)``.
+ w: weights of shape ``(M,)``. If None, uniform weights are used. ``(M, 1)`` weights raise a
+ ``RuntimeError`` (`#6482 <https://github.com/kornia/kornia/issues/6482>`_).
Returns:
- Quaternion: averaged quaternion of shape (1, 4), wrapped back in the Quaternion class.
+ the averaged quaternion, of unit norm and shape ``(1, 4)``.
+
+ Raises:
+ TypeCheckError: if ``Q`` is not a :class:`Quaternion`.
+ ValueError: if ``w`` does not have ``M`` elements, has a negative entry, or sums to zero.
+
+ See Also:
+ - :class:`Quaternion`: holds ``Q`` and the result; it composes, inverts and converts quaternions one at a
+ time rather than averaging them.
+ - :meth:`Quaternion.slerp`: interpolates between two rotations along the shorter arc instead of averaging
+ many.
+
+ Example:
+ >>> Q = Quaternion(torch.tensor([[1., 0., 0., 0.], [-1., 0., 0., 0.]]))
+ >>> average_quaternions(Q).data.abs()
+ tensor([[1., 0., 0., 0.]])
+
"""
KORNIA_CHECK_TYPE(Q, Quaternion)
# the chordal mean is the top eigenvector of sum_i w_i q_i q_i^T, which is sign invariant but weights each member
diff --git a/kornia/geometry/ransac.py b/kornia/geometry/ransac.py
index eadf8fc4d..0508f99ca 100644
--- a/kornia/geometry/ransac.py
+++ b/kornia/geometry/ransac.py
@@ -575,7 +575,7 @@ class RANSAC(nn.Module):
"""Return the batch size of a ``local_optimization="dlt"`` call: configured or ``"auto"``.
On CUDA and MPS a homography or affine batch costs about the same from a few hundred up to 8192 hypotheses
- (the four- and three-point solves are launch-bound), so the whole :attr:`sample_budget` is drawn in batches
+ (the four- and three-point solves are launch-bound), so the whole ``sample_budget`` is drawn in batches
of up to 8192; the epipolar solvers are compute-bound past 2048 hypotheses, so their batches stop there
and early stopping is checked in between. The verification holds a ``batch x N`` residual matrix and a
few temporaries of that size, so past ``2**27`` entries (about 1 GiB at peak in float32) the batch
diff --git a/kornia/geometry/subpix/__init__.py b/kornia/geometry/subpix/__init__.py
index b6e74c946..5c7b57db3 100644
--- a/kornia/geometry/subpix/__init__.py
+++ b/kornia/geometry/subpix/__init__.py
@@ -15,9 +15,26 @@
# limitations under the License.
#
-"""Kornia Geometry Subpix — Subpixel operations for Kornia.
+"""Sub-pixel locations from heatmaps and scale-space volumes: soft-argmax, strict NMS and quadratic refinement.
-This subpackage provides subpixel localization and softmax utilities for geometry tasks.
+2D coordinates are ``(x, y)`` = (column, row) and 3D ones ``(d, x, y)``, depth first (the scale level of a
+scale-space volume), with integer pixel centres; see :ref:`Coordinates and sizes <coordinate-conventions>`. A
+``normalized_coordinates=True`` result is in :math:`[-1, 1]`, corner-aligned; the default differs between functions.
+A class in parentheses is the module form of the function before it: it stores the arguments.
+
+Which function to use:
+
+- One location per heatmap channel, the softmax-weighted mean over the whole map: :func:`spatial_soft_argmax2d`
+ (:class:`SpatialSoftArgmax2d`), or its two steps, :func:`spatial_softmax2d` and :func:`spatial_expectation2d`.
+ The other direction, a Gaussian heatmap around a given mean: :func:`render_gaussian2d`.
+- One location per sliding window, for maps with several peaks: :func:`conv_soft_argmax2d`
+ (:class:`ConvSoftArgmax2d`), and for ``(B, C, D, H, W)`` volumes :func:`conv_soft_argmax3d`
+ (:class:`ConvSoftArgmax3d`).
+- Strict local maxima: of a map, :func:`nms2d` (:class:`NonMaximaSuppression2d`); of a volume, :func:`nms3d`
+ (:class:`NonMaximaSuppression3d`); the maxima and minima of a volume in one call, :func:`nms3d_minmax`.
+- Sub-voxel refinement of the maxima of a scale-space volume by a quadratic fit: :class:`AdaptiveQuadInterp3d`, which
+ by default runs :func:`conv_quad_interp3d` (:class:`ConvQuadInterp3d`) on CUDA and :func:`iterative_quad_interp3d`
+ (:class:`IterativeQuadInterp3d`) elsewhere.
"""
from __future__ import annotations
diff --git a/kornia/geometry/subpix/dsnt.py b/kornia/geometry/subpix/dsnt.py
index 98d72bc28..8f7f6fc98 100644
--- a/kornia/geometry/subpix/dsnt.py
+++ b/kornia/geometry/subpix/dsnt.py
@@ -15,9 +15,10 @@
# limitations under the License.
#
-r"""Implementation of "differentiable spatial to numerical" (soft-argmax) operations.
+r"""Spatial softmax, the expected coordinate of a probability map, and Gaussian heatmaps (DSNT).
-As described in the paper "Numerical Coordinate Regression with Convolutional Neural Networks" by Nibali et al.
+The "differentiable spatial to numerical" (soft-argmax) operations, as described in the paper "Numerical Coordinate
+Regression with Convolutional Neural Networks" by Nibali et al.
"""
from __future__ import annotations
@@ -52,19 +53,35 @@ def _check_positive_temperature(temperature: torch.Tensor | float) -> None:
def spatial_softmax2d(input: torch.Tensor, temperature: Optional[torch.Tensor | float] = None) -> torch.Tensor:
- r"""Apply the Softmax function over features in each image channel.
+ r"""Turn each heatmap channel into a probability map by a softmax over all its pixels.
- Note that this function behaves differently to :py:class:`torch.nn.Softmax2d`, which
- instead applies Softmax over features at each spatial location.
+ ``probs = spatial_softmax2d(heatmap, temperature)`` takes a ``(B, N, H, W)`` heatmap and returns, for each of its
+ ``B x N`` channels, the softmax of ``heatmap / temperature`` over all ``H x W`` pixels, ``(B, N, H, W)``: each map
+ sums to one. This differs from :py:class:`torch.nn.Softmax2d`, which applies the softmax across channels at each
+ pixel. An integer heatmap returns torch's default floating dtype, with a float or a tensor ``temperature``. Pass the
+ result to
+ :func:`spatial_expectation2d` for its expected ``(x, y)``, or use :func:`spatial_soft_argmax2d` for both steps in
+ one call.
Args:
input: the input torch.Tensor with shape ``(B, N, H, W)``.
temperature: softmax temperature: the input is divided by it; smaller is sharper. Must be positive.
- ``None`` means ``1.0``. A tensor temperature is not checked under ``torch.compile`` or export.
+ ``None`` means ``1.0``. A tensor temperature is not checked under ``torch.compile`` or export; it broadcasts
+ against the flattened ``(B, N, H * W)`` logits, so a ``(1, N, 1)`` tensor sets one per channel.
Returns:
a 2D probability distribution per image channel with shape ``(B, N, H, W)``.
+ Raises:
+ ValueError: if ``temperature``, or any element of a tensor ``temperature``, is not positive (``NaN``
+ included).
+
+ See Also:
+ - :func:`spatial_expectation2d`: the expected ``(x, y)`` of the probability maps this returns.
+ - :func:`spatial_soft_argmax2d`: this softmax and the expectation in one call.
+ - :func:`conv_soft_argmax2d`: a softmax over each sliding window instead of the whole map.
+ - :func:`render_gaussian2d`: a probability map from a mean and a standard deviation instead of a heatmap.
+
Examples:
>>> heatmaps = torch.tensor([[[
... [0., 0., 0.],
@@ -104,7 +121,12 @@ def spatial_softmax2d(input: torch.Tensor, temperature: Optional[torch.Tensor |
def spatial_expectation2d(input: torch.Tensor, normalized_coordinates: bool = True) -> torch.Tensor:
- r"""Compute the expectation of coordinate values using spatial probabilities.
+ r"""Compute the expected (x, y) coordinate of each channel of a probability map.
+
+ ``coords = spatial_expectation2d(probs)`` takes ``(B, N, H, W)`` maps, each summing to one, and returns the
+ probability-weighted mean position of each map, ``(B, N, 2)`` in ``(x, y)`` order: normalized to :math:`[-1, 1]`,
+ corner-aligned, by default, or in pixels with ``normalized_coordinates=False``. Make the maps from a heatmap with
+ :func:`spatial_softmax2d`, or use :func:`spatial_soft_argmax2d` for both steps in one call.
Convention:
- Coordinates follow :ref:`Coordinates and sizes <coordinate-conventions>`. The input is used as given, not
@@ -119,6 +141,13 @@ def spatial_expectation2d(input: torch.Tensor, normalized_coordinates: bool = Tr
Returns:
expected value of the 2D coordinates with shape ``(B, N, 2)``. Output order of the coordinates is (x, y).
+ See Also:
+ - :func:`spatial_softmax2d`: makes the probability maps this function expects from a heatmap.
+ - :func:`spatial_soft_argmax2d`: the softmax and this expectation in one call.
+ - :func:`render_gaussian2d`: the other direction, a probability map around a given ``(x, y)`` mean.
+ - :func:`~kornia.geometry.grid.create_meshgrid`: the coordinate grid the expectation weighs, in the same
+ convention.
+
Examples:
>>> heatmaps = torch.tensor([[[
... [0., 0., 0.],
@@ -155,10 +184,17 @@ def spatial_expectation2d(input: torch.Tensor, normalized_coordinates: bool = Tr
def render_gaussian2d(
mean: torch.Tensor, std: torch.Tensor, size: tuple[int, int], normalized_coordinates: bool = True
) -> torch.Tensor:
- r"""Render the PDF of a 2D Gaussian distribution.
+ r"""Render an axis-aligned 2D Gaussian heatmap that sums to one over an (H, W) grid.
- Each axis is normalised over the grid, so the heatmap sums to one. A mean outside the grid renders the part of
- the Gaussian that falls on the grid, rescaled to sum to one; far outside, that mass sits on the nearest border.
+ ``heatmap = render_gaussian2d(mean, std, (H, W))`` takes means ``(*, 2)`` and standard deviations ``(*, 2)``,
+ both in ``(x, y)`` order, and returns one ``(*, H, W)`` map per Gaussian: the product of a 1D Gaussian along ``x``
+ and one along ``y``, each normalised over the grid, so the map sums to one instead of holding the values of the
+ continuous density. ``mean`` and ``std`` are in normalized :math:`[-1, 1]` units by default, or in pixels with
+ ``normalized_coordinates=False``. For the other direction, the expected coordinate of a probability map, use
+ :func:`spatial_expectation2d`.
+
+ A mean outside the grid renders the part of the Gaussian that falls on the grid, rescaled to sum to one; far
+ outside, that mass sits on the nearest border.
Convention:
- ``mean`` and ``std`` follow :ref:`Coordinates and sizes <coordinate-conventions>`; ``size`` is
@@ -168,7 +204,12 @@ def render_gaussian2d(
Args:
mean: the mean location of the Gaussian to render, :math:`(\mu_x, \mu_y)`. Shape: ``(*, 2)``.
std: the standard deviation of the Gaussian to render, :math:`(\sigma_x, \sigma_y)`.
- Shape ``(*, 2)``. Should be able to be broadcast with `mean`. Every entry must be nonzero.
+ Shape ``(*, 2)``. Should be able to be broadcast with `mean`. Its sign is ignored. Every entry must be
+ nonzero: a zero raises :class:`~kornia.core.exceptions.AsyncCheckError`, a :exc:`RuntimeError`, during
+ the call on the CPU and on MPS before torch 2.13. On CUDA and on MPS from torch 2.13, torch raises a
+ ``RuntimeError`` of its own by the next synchronization. A compiled call raises torch's ``RuntimeError``
+ on the CPU, on CUDA and on MPS from torch 2.13, with Inductor from torch 2.9. With checks disabled
+ nothing is raised, and the heatmap of a Gaussian whose ``std`` has a zero is NaN.
size: the (height, width) of the output image.
normalized_coordinates: whether ``mean`` and ``std`` are assumed to use coordinates normalized
in the range of :math:`[-1, 1]`. Otherwise, coordinates are assumed to be in the range of the output shape.
@@ -176,6 +217,16 @@ def render_gaussian2d(
Returns:
torch.Tensor including rendered points with shape ``(*, H, W)``.
+ Raises:
+ TypeError: if ``mean`` and ``std`` differ in dtype or device.
+
+ See Also:
+ - :func:`spatial_expectation2d`: the other direction, the expected ``(x, y)`` of a probability map.
+ - :func:`spatial_softmax2d`: a probability map from a heatmap instead of a mean and a standard deviation.
+ - :func:`spatial_soft_argmax2d`: the other direction from a heatmap of logits: softmax, then expectation.
+ - :func:`~kornia.filters.get_gaussian_kernel2d`: a Gaussian filter kernel centred in its window, for
+ filtering rather than a heatmap at a given position.
+
"""
if not (std.dtype == mean.dtype and std.device == mean.device):
raise TypeError("Expected inputs to have the same dtype and device")
diff --git a/kornia/geometry/subpix/nms.py b/kornia/geometry/subpix/nms.py
index 9da884229..66e2ff867 100644
--- a/kornia/geometry/subpix/nms.py
+++ b/kornia/geometry/subpix/nms.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
+"""Strict non-maxima suppression of 2D response maps and 3D response volumes."""
+
from __future__ import annotations
from typing import Any, List, Tuple
@@ -65,7 +67,7 @@ def _neighbourhood_max2d(x: torch.Tensor, ky: int, kx: int) -> torch.Tensor:
``O(ky + kx)`` taps per position rather than the ``ky * kx - 1`` a literal neighbourhood costs.
Only positions whose full window lies inside the image are computed, so the result has shape
- :math:`(B, C, H - k_y + 1, W - k_x + 1)`; entry ``(i, j)`` belongs to the centre
+ ``(B, C, H - ky + 1, W - kx + 1)``; entry ``(i, j)`` belongs to the centre
``x[..., i + (k_y - 1) // 2, j + (k_x - 1) // 2]``.
"""
cy, by = _split_window(ky)
@@ -119,7 +121,12 @@ def _neighbourhood_max3d(x: torch.Tensor, kd: int, ky: int, kx: int) -> torch.Te
class NonMaximaSuppression2d(nn.Module):
- r"""Keep only the strict local maxima of a (B, C, H, W) map in a 2D window; the module form of nms2d.
+ r"""Keep the strict local maxima of a (B, C, H, W) map; the module form of nms2d.
+
+ ``nms = NonMaximaSuppression2d((ky, kx))`` stores the window size, and ``out = nms(response)`` returns what
+ ``nms2d(response, (ky, kx))`` returns: the ``(B, C, H, W)`` map with every position that is not a strict local
+ maximum set to ``0``, or with ``nms(response, mask_only=True)`` the boolean mask. The module holds no parameters
+ or buffers; a state dict saved by an older kornia release with a ``kernel`` buffer still loads strictly.
Convention:
- See the convention block of :func:`~kornia.geometry.subpix.nms2d`.
@@ -127,6 +134,16 @@ class NonMaximaSuppression2d(nn.Module):
.. warning::
On MPS, torch earlier than 2.16 gives wrong masks for some windows when the input holds nan or -inf; see the
warning of :func:`~kornia.geometry.subpix.nms2d`.
+
+ Args:
+ kernel_size: the window size ``(ky, kx)``, a tuple of two ints; a list or another length raises
+ ``AssertionError``.
+
+ See Also:
+ - :func:`nms2d`: the function form, with the window size passed on each call.
+ - :class:`NonMaximaSuppression3d`: the module for ``(B, C, D, H, W)`` volumes.
+ - :func:`nms3d_minmax`: strict maxima and minima of a volume in one call.
+
"""
def __init__(self, kernel_size: tuple[int, int]) -> None:
@@ -314,7 +331,11 @@ class NonMaximaSuppression2d(nn.Module):
class NonMaximaSuppression3d(nn.Module):
- r"""Keep only the strict local maxima of a (B, C, D, H, W) volume in a 3D window; the module form of nms3d.
+ r"""Keep the strict local maxima of a (B, C, D, H, W) volume; the module form of nms3d.
+
+ ``nms = NonMaximaSuppression3d((kd, ky, kx))`` stores the window size, and ``out = nms(volume)`` returns what
+ ``nms3d(volume, (kd, ky, kx))`` returns: the ``(B, C, D, H, W)`` volume with every voxel that is not a strict
+ local maximum set to ``0``, or with ``nms(volume, mask_only=True)`` the boolean mask.
Convention:
- See the convention block of :func:`~kornia.geometry.subpix.nms2d`.
@@ -322,6 +343,16 @@ class NonMaximaSuppression3d(nn.Module):
.. warning::
On MPS, torch earlier than 2.16 gives wrong masks for some windows when the input holds nan; see the warning
of :func:`~kornia.geometry.subpix.nms3d`.
+
+ Args:
+ kernel_size: the window size ``(kd, ky, kx)``, a tuple of three ints; a list or another length raises
+ ``AssertionError``.
+
+ See Also:
+ - :func:`nms3d`: the function form, with the window size passed on each call.
+ - :class:`NonMaximaSuppression2d`: the module for ``(B, C, H, W)`` maps.
+ - :func:`nms3d_minmax`: strict maxima and minima in a ``3 x 3 x 3`` window in one call.
+
"""
def __init__(self, kernel_size: tuple[int, int, int]) -> None:
@@ -414,7 +445,13 @@ class NonMaximaSuppression3d(nn.Module):
def nms2d(input: torch.Tensor, kernel_size: tuple[int, int], mask_only: bool = False) -> torch.Tensor:
- r"""Keep only the strict local maxima of a (B, C, H, W) map in a 2D ``kernel_size`` window, zeroing the rest.
+ r"""Keep the strict local maxima of a (B, C, H, W) map in a 2D window, zeroing the rest.
+
+ ``out = nms2d(response, (ky, kx))`` takes a ``(B, C, H, W)`` response map and returns it with every position that
+ is not a strict local maximum of its ``ky x kx`` window set to ``0``, ``(B, C, H, W)`` in the input dtype; with
+ ``mask_only=True`` it returns the boolean maxima mask instead. Each channel is suppressed on its own. For
+ ``(B, C, D, H, W)`` volumes, use :func:`nms3d`, or :func:`nms3d_minmax` for the maxima and minima together; to
+ suppress overlapping boxes by their IoU, use :func:`~kornia.geometry.bbox.nms`.
Convention:
- Only strict maxima survive: a position must be greater than every other value in its ``kernel_size``
@@ -435,12 +472,34 @@ def nms2d(input: torch.Tensor, kernel_size: tuple[int, int], mask_only: bool = F
`pytorch/pytorch#200789 <https://github.com/pytorch/pytorch/issues/200789>`__). A position next to a nan can
then be kept, and a negative maximum among -inf neighbours suppressed. Those four windows compare the
neighbours directly and are not affected, and neither are other devices or torch 2.16 or later.
+
+ Args:
+ input: response map with shape ``(B, C, H, W)``; another number of dimensions raises ``AssertionError``.
+ kernel_size: the window size ``(ky, kx)``, a tuple of two ints. ``(1, 1)`` keeps every position, and a
+ window larger than the map keeps none.
+ mask_only: return the boolean maxima mask instead of the suppressed map.
+
+ Returns:
+ the suppressed map ``(B, C, H, W)`` in the dtype of ``input``, or with ``mask_only=True`` a boolean mask of
+ the same shape.
+
+ See Also:
+ - :func:`nms3d`: the same rule over the depth, height and width of a ``(B, C, D, H, W)`` volume.
+ - :func:`nms3d_minmax`: strict maxima and minima of a volume in a fixed ``3 x 3 x 3`` window, in one call.
+ - :class:`NonMaximaSuppression2d`: the module form.
+ - :func:`~kornia.geometry.bbox.nms`: suppresses overlapping boxes by their IoU, not positions of a map.
+
"""
return NonMaximaSuppression2d(kernel_size)(input, mask_only)
def nms3d(input: torch.Tensor, kernel_size: tuple[int, int, int], mask_only: bool = False) -> torch.Tensor:
- r"""Keep only the strict local maxima of a (B, C, D, H, W) volume in a 3D ``kernel_size`` window, zeroing the rest.
+ r"""Keep the strict local maxima of a (B, C, D, H, W) volume in a 3D window, zeroing the rest.
+
+ ``out = nms3d(volume, (kd, ky, kx))`` takes a ``(B, C, D, H, W)`` volume, such as a scale-space response with the
+ scale levels along ``D``, and returns it with every voxel that is not a strict local maximum of its
+ ``kd x ky x kx`` window set to ``0``; with ``mask_only=True`` it returns the boolean maxima mask instead. For the
+ minima as well, in a ``3 x 3 x 3`` window, use :func:`nms3d_minmax`; for a ``(B, C, H, W)`` map, :func:`nms2d`.
Convention:
- See the convention block of :func:`nms2d`, applied over depth, height and width.
@@ -451,27 +510,47 @@ def nms3d(input: torch.Tensor, kernel_size: tuple[int, int, int], mask_only: boo
window (`pytorch/pytorch#200910 <https://github.com/pytorch/pytorch/issues/200910>`__), so a position next to
a nan can be kept. Those two windows compare the neighbours directly and are not affected, and neither are
other devices or torch 2.16 or later.
+
+ Args:
+ input: response volume with shape ``(B, C, D, H, W)``; another number of dimensions raises
+ ``AssertionError``.
+ kernel_size: the window size ``(kd, ky, kx)``, a tuple of three ints. ``(1, 1, 1)`` keeps every voxel, and a
+ window larger than the volume keeps none.
+ mask_only: return the boolean maxima mask instead of the suppressed volume.
+
+ Returns:
+ the suppressed volume ``(B, C, D, H, W)`` in the dtype of ``input``, or with ``mask_only=True`` a boolean
+ mask of the same shape.
+
+ See Also:
+ - :func:`nms2d`: the same rule over the height and width of a ``(B, C, H, W)`` map.
+ - :func:`nms3d_minmax`: the maxima and the minima in a fixed ``3 x 3 x 3`` window, in one call.
+ - :class:`NonMaximaSuppression3d`: the module form.
+ - :func:`iterative_quad_interp3d`: refines the maxima of a scale-space volume to sub-voxel positions; it
+ takes this function's mask as ``precomputed_nms_mask``.
+
"""
return NonMaximaSuppression3d(kernel_size)(input, mask_only)
def nms3d_minmax(input: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
- r"""Compute both local-maxima and local-minima NMS masks for a 3-D scale-space tensor in one pass.
+ r"""Mask the strict local maxima and minima of a (B, C, D, H, W) volume in a 3x3x3 window.
- Equivalent to calling ``nms3d(input, (3,3,3), mask_only=True)`` and
- ``nms3d(-input, (3,3,3), mask_only=True)`` separately, but only traverses
- the 26-neighbour comparisons once, halving the NMS cost.
+ ``max_mask, min_mask = nms3d_minmax(volume)`` takes a ``(B, C, D, H, W)`` volume and returns two boolean masks of
+ the same shape: the voxels strictly greater than all 26 neighbours, and those strictly smaller. For a floating
+ input they equal ``nms3d(volume, (3, 3, 3), mask_only=True)`` and
+ ``nms3d(-volume, (3, 3, 3), mask_only=True)``. :class:`~kornia.feature.ScaleSpaceDetector` uses it on every
+ octave; for another window size, call :func:`nms3d`.
- Uses integer slice literals (not Python loops or slice objects) so the 52
- comparison-and-reduction ops are visible to the compiler at trace time,
- allowing full fusion into a minimal number of kernels.
+ Under ``torch.compile`` the comparisons can fuse; in eager mode on the CPU and MPS the call costs about as much as
+ the two :func:`nms3d` calls.
Convention:
- The strict and border rules of :func:`nms2d` with a :math:`3 \times 3 \times 3` window, for maxima and
minima alike.
Args:
- input: 5-D tensor of shape ``(B, C, D, H, W)``.
+ input: 5-D tensor of shape ``(B, C, D, H, W)``; another number of dimensions raises ``AssertionError``.
Returns:
A pair ``(max_mask, min_mask)`` of bool tensors with the same shape as
@@ -479,6 +558,11 @@ def nms3d_minmax(input: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
strictly greater than all 26 neighbours; ``min_mask`` is the same for
strict local minima.
+ See Also:
+ - :func:`nms3d`: the maxima only, in a window of any size.
+ - :func:`nms2d`: the maxima of a ``(B, C, H, W)`` map.
+ - :class:`NonMaximaSuppression3d`: the maxima only, as a module.
+
Example:
>>> x = torch.randn(1, 1, 5, 10, 10)
>>> max_mask, min_mask = nms3d_minmax(x)
diff --git a/kornia/geometry/subpix/spatial_soft_argmax.py b/kornia/geometry/subpix/spatial_soft_argmax.py
index 4fdb36dbb..f4a9948b4 100644
--- a/kornia/geometry/subpix/spatial_soft_argmax.py
+++ b/kornia/geometry/subpix/spatial_soft_argmax.py
@@ -15,6 +15,8 @@
# limitations under the License.
#
+"""Soft-argmax over whole heatmaps and sliding windows, and quadratic refinement of 3D scale-space maxima."""
+
from __future__ import annotations
import math
@@ -177,10 +179,22 @@ def _get_window_grid_kernel3d(d: int, h: int, w: int, device: Optional[torch.dev
class ConvSoftArgmax2d(nn.Module):
- r"""nn.Module that calculates soft argmax 2d per window.
+ r"""Compute a soft-argmax (x, y) per window; the module form of conv_soft_argmax2d.
+
+ ``op = ConvSoftArgmax2d(kernel_size, stride, padding)`` stores the arguments of :func:`conv_soft_argmax2d`, and
+ ``coords = op(heatmap)`` returns what that function returns for them: the ``(B, C, 2, H_out, W_out)`` window
+ coordinates in ``(x, y)`` order, normalized by default, and with ``output_value=True`` the pair
+ ``(coords, values)``.
Convention:
- See the convention block of :func:`~kornia.geometry.subpix.conv_soft_argmax2d`.
+
+ See Also:
+ - :func:`conv_soft_argmax2d`: the function form.
+ - :class:`ConvSoftArgmax3d`: the module for ``(B, C, D, H, W)`` volumes, in voxels and with values by
+ default.
+ - :class:`SpatialSoftArgmax2d`: one soft-argmax per channel over the whole map.
+
"""
def __init__(
@@ -244,10 +258,21 @@ class ConvSoftArgmax2d(nn.Module):
class ConvSoftArgmax3d(nn.Module):
- r"""nn.Module that calculates soft argmax 3d per window.
+ r"""Compute a soft-argmax (d, x, y) per window; the module form of conv_soft_argmax3d.
+
+ ``op = ConvSoftArgmax3d(kernel_size, stride, padding)`` stores the arguments of :func:`conv_soft_argmax3d`, and
+ ``coords, values = op(volume)`` returns what that function returns for them: the ``(B, C, 3, D_out, H_out, W_out)``
+ window coordinates in ``(d, x, y)`` order, in voxels by default, and the ``(B, C, D_out, H_out, W_out)``
+ softmax-pooled values; with ``output_value=False``, the coordinates alone. Passing ``strict_maxima_bonus`` emits a
+ :class:`DeprecationWarning` and has no effect.
Convention:
- See the convention block of :func:`~kornia.geometry.subpix.conv_soft_argmax3d`.
+
+ See Also:
+ - :func:`conv_soft_argmax3d`: the function form.
+ - :class:`ConvSoftArgmax2d`: the module for ``(B, C, H, W)`` maps, normalized and without values by default.
+
"""
def __init__(
@@ -342,7 +367,15 @@ def conv_soft_argmax2d(
eps: float = 1e-8,
output_value: bool = False,
) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
- r"""Compute the convolutional spatial Soft-Argmax 2D over the windows of a given heatmap.
+ r"""Compute a soft-argmax (x, y) in every sliding window of a (B, C, H, W) heatmap.
+
+ ``coords = conv_soft_argmax2d(heatmap)`` takes a ``(B, C, H, W)`` heatmap and returns, for every ``kernel_size``
+ window at every ``stride``, the mean position of its pixels under the softmax of the window's values divided by
+ ``temperature``: ``(B, C, 2, H_out, W_out)`` with ``(x, y)`` along dimension 2, normalized to :math:`[-1, 1]` by
+ default or in pixels with ``normalized_coordinates=False``. With ``output_value=True`` it also returns the
+ softmax-weighted mean value of each window, ``(B, C, H_out, W_out)``. The default ``(3, 3)`` window with stride 1
+ and padding 1 gives ``H_out, W_out = H, W``. Each window locates its own peak; for one location per channel over
+ the whole map, use :func:`spatial_soft_argmax2d`, and for ``(B, C, D, H, W)`` volumes, :func:`conv_soft_argmax3d`.
.. math::
xy(X) = \frac{\sum_{p \in X} (x_p, y_p) \exp(v_p / T)}{\sum_{p \in X} \exp(v_p / T)}
@@ -364,8 +397,8 @@ def conv_soft_argmax2d(
accepted and has no effect (`#5020 <https://github.com/kornia/kornia/issues/5020>`_).
Args:
- input: the given heatmap with shape :math:`(N, C, H_{in}, W_{in})`.
- kernel_size: the size of the window.
+ input: the given heatmap with shape ``(B, C, H, W)``.
+ kernel_size: the size ``(ky, kx)`` of the window.
stride: the stride of the window.
padding: the padding added to each side of the input, at most half of ``kernel_size`` on every axis.
temperature: softmax temperature: the input is divided by it; smaller is sharper. Must be positive.
@@ -373,23 +406,34 @@ def conv_soft_argmax2d(
normalized_coordinates: whether to return the coordinates normalized in the range of :math:`[-1, 1]`.
Otherwise, it will return the coordinates in the range of the input shape.
eps: accepted for backward compatibility and ignored; every window's softmax denominator is at least 1.
- output_value: if True, val is output, if False, only ij.
+ output_value: whether to also return the softmax-pooled value of each window.
Returns:
- Function has two outputs - argmax coordinates and the softmaxpooled heatmap values themselves.
- On each window, the function computed returns with shapes :math:`(N, C, 2, H_{out},
- W_{out})`, :math:`(N, C, H_{out}, W_{out})`,
+ the coordinates of each window, ``(B, C, 2, H_out, W_out)`` with ``(x, y)`` along dimension 2, and with
+ ``output_value=True`` also the softmax-pooled heatmap values, ``(B, C, H_out, W_out)``,
where
.. math::
- H_{out} = \left\lfloor\frac{H_{in} + 2 \times \text{padding}[0] -
+ H_{out} = \left\lfloor\frac{H + 2 \times \text{padding}[0] -
(\text{kernel\_size}[0] - 1) - 1}{\text{stride}[0]} + 1\right\rfloor
.. math::
- W_{out} = \left\lfloor\frac{W_{in} + 2 \times \text{padding}[1] -
+ W_{out} = \left\lfloor\frac{W + 2 \times \text{padding}[1] -
(\text{kernel\_size}[1] - 1) - 1}{\text{stride}[1]} + 1\right\rfloor
+ Raises:
+ TypeError: if ``input`` is not a tensor.
+ ValueError: if ``input`` is not 4D, if ``padding`` is negative or more than half of ``kernel_size`` on an
+ axis, or if ``temperature`` is not positive.
+
+ See Also:
+ - :func:`conv_soft_argmax3d`: the same per-window soft-argmax over a ``(B, C, D, H, W)`` volume, returning
+ ``(d, x, y)``, in voxels and with values by default.
+ - :func:`spatial_soft_argmax2d`: one soft-argmax per channel, over the whole map.
+ - :class:`ConvSoftArgmax2d`: the module form.
+ - :func:`nms2d`: the strict local maxima of a window, at integer positions and without a softmax.
+
Examples:
>>> input = torch.randn(20, 16, 50, 32)
>>> nms_coords, nms_val = conv_soft_argmax2d(input, (3,3), (2,2), (1,1), output_value=True)
@@ -475,7 +519,16 @@ def conv_soft_argmax3d(
output_value: bool = True,
strict_maxima_bonus: Optional[float] = None,
) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
- r"""Compute the convolutional spatial Soft-Argmax 3D over the windows of a given heatmap.
+ r"""Compute a soft-argmax (d, x, y) in every sliding window of a (B, C, D, H, W) volume.
+
+ ``coords, values = conv_soft_argmax3d(volume)`` takes a ``(B, C, D, H, W)`` volume and returns, for every
+ ``kernel_size`` window at every ``stride``, the mean position of its voxels under the softmax of the window's
+ values divided by ``temperature``, ``(B, C, 3, D_out, H_out, W_out)`` with ``(d, x, y)`` along dimension 2, and
+ the softmax-weighted mean value of each window, ``(B, C, D_out, H_out, W_out)``. Unlike
+ :func:`conv_soft_argmax2d`, it defaults to voxel coordinates and to returning the values; pass
+ ``normalized_coordinates=True`` for :math:`[-1, 1]` coordinates and ``output_value=False`` for the coordinates
+ alone. To refine only the strict maxima of a scale-space volume by a quadratic fit, use
+ :func:`conv_quad_interp3d` instead.
.. math::
dxy(X) = \frac{\sum_{p \in X} (d_p, x_p, y_p) \exp(v_p / T)}{\sum_{p \in X} \exp(v_p / T)}
@@ -491,38 +544,48 @@ def conv_soft_argmax3d(
:ref:`Coordinates and sizes <coordinate-conventions>`; this function defaults to pixel coordinates.
Args:
- input: the given heatmap with shape :math:`(N, C, D_{in}, H_{in}, W_{in})`.
- kernel_size: size of the window.
- stride: stride of the window.
+ input: the given heatmap with shape ``(B, C, D, H, W)``.
+ kernel_size: the size ``(kd, ky, kx)`` of the window.
+ stride: stride of the window, an ``int`` for every axis or a tuple ``(sd, sy, sx)``.
padding: the padding added to each side of the input, at most half of ``kernel_size`` on every axis.
temperature: softmax temperature: the input is divided by it; smaller is sharper. Must be positive.
A tensor temperature is not checked under ``torch.compile`` or export.
normalized_coordinates: whether to return the coordinates normalized in the range of :math:`[-1, 1]`.
Otherwise, it will return the coordinates in the range of the input shape.
eps: accepted for backward compatibility and ignored; every window's softmax denominator is at least 1.
- output_value: if True, val is output, if False, only ij.
+ output_value: whether to also return the softmax-pooled value of each window.
strict_maxima_bonus: deprecated since kornia 0.9.0 and ignored; passing it emits a
:class:`DeprecationWarning`. Select strict maxima with :func:`~kornia.geometry.subpix.nms3d` instead.
Returns:
- Function has two outputs - argmax coordinates and the softmaxpooled heatmap values themselves.
- On each window, the function computed returns with shapes :math:`(N, C, 3, D_{out}, H_{out}, W_{out})`,
- :math:`(N, C, D_{out}, H_{out}, W_{out})`,
+ the coordinates of each window, ``(B, C, 3, D_out, H_out, W_out)`` with ``(d, x, y)`` along dimension 2, and
+ with ``output_value=True`` (the default) also the softmax-pooled values, ``(B, C, D_out, H_out, W_out)``,
where
.. math::
- D_{out} = \left\lfloor\frac{D_{in} + 2 \times \text{padding}[0] -
+ D_{out} = \left\lfloor\frac{D + 2 \times \text{padding}[0] -
(\text{kernel\_size}[0] - 1) - 1}{\text{stride}[0]} + 1\right\rfloor
.. math::
- H_{out} = \left\lfloor\frac{H_{in} + 2 \times \text{padding}[1] -
+ H_{out} = \left\lfloor\frac{H + 2 \times \text{padding}[1] -
(\text{kernel\_size}[1] - 1) - 1}{\text{stride}[1]} + 1\right\rfloor
.. math::
- W_{out} = \left\lfloor\frac{W_{in} + 2 \times \text{padding}[2] -
+ W_{out} = \left\lfloor\frac{W + 2 \times \text{padding}[2] -
(\text{kernel\_size}[2] - 1) - 1}{\text{stride}[2]} + 1\right\rfloor
+ Raises:
+ TypeError: if ``input`` is not a tensor.
+ ValueError: if ``input`` is not 5D, if ``padding`` is negative or more than half of ``kernel_size`` on an
+ axis, or if ``temperature`` is not positive.
+
+ See Also:
+ - :func:`conv_soft_argmax2d`: the same per-window soft-argmax over a ``(B, C, H, W)`` map, returning
+ ``(x, y)``, normalized and without values by default.
+ - :func:`conv_quad_interp3d`: refines only the strict maxima of a volume, by a quadratic fit.
+ - :class:`ConvSoftArgmax3d`: the module form.
+
Examples:
>>> input = torch.randn(20, 16, 3, 50, 32)
>>> nms_coords, nms_val = conv_soft_argmax3d(input, (3, 3, 3), (1, 2, 2), (0, 1, 1))
@@ -601,7 +664,14 @@ def conv_soft_argmax3d(
def spatial_soft_argmax2d(
input: torch.Tensor, temperature: Optional[torch.Tensor | float] = None, normalized_coordinates: bool = True
) -> torch.Tensor:
- r"""Compute the Spatial Soft-Argmax 2D of a given input heatmap.
+ r"""Compute one soft-argmax (x, y) per heatmap channel over the whole map.
+
+ ``coords = spatial_soft_argmax2d(heatmap, temperature)`` takes a ``(B, N, H, W)`` heatmap and returns one ``(x, y)``
+ per channel, ``(B, N, 2)``: the mean pixel position under the softmax of ``heatmap / temperature`` over all
+ ``H x W`` pixels, normalized to :math:`[-1, 1]` by default or in pixels with ``normalized_coordinates=False``. It
+ is a differentiable mean, not a maximum: it lands on the peak only when the peak dominates the map at the given
+ temperature, and the rest of the map pulls it away otherwise. For one location per sliding window, use
+ :func:`conv_soft_argmax2d`.
Convention:
- Coordinates follow :ref:`Coordinates and sizes <coordinate-conventions>`; this function defaults to
@@ -620,6 +690,13 @@ def spatial_soft_argmax2d(
Returns:
the expected coordinates of each map with shape ``(B, N, 2)``.
+ See Also:
+ - :func:`spatial_softmax2d`: the softmax step alone, to inspect or reuse the probability maps.
+ - :func:`spatial_expectation2d`: the expectation step alone, for maps that already sum to one.
+ - :func:`conv_soft_argmax2d`: one soft-argmax per sliding window, for maps with several peaks.
+ - :class:`SpatialSoftArgmax2d`: the module form.
+ - :func:`render_gaussian2d`: the other direction, a Gaussian probability map around a given ``(x, y)``.
+
Examples:
>>> input = torch.tensor([[[
... [0., 0., 0.],
@@ -635,10 +712,20 @@ def spatial_soft_argmax2d(
class SpatialSoftArgmax2d(nn.Module):
- r"""Compute the Spatial Soft-Argmax 2D of a given heatmap.
+ r"""Compute a soft-argmax (x, y) per channel; the module form of spatial_soft_argmax2d.
+
+ ``op = SpatialSoftArgmax2d(temperature)`` stores the temperature, ``torch.tensor(1.0)`` for ``None``, and
+ ``normalized_coordinates``, and ``coords = op(heatmap)`` returns
+ ``spatial_soft_argmax2d(heatmap, temperature, normalized_coordinates)``: one ``(x, y)`` per channel of a
+ ``(B, N, H, W)`` heatmap, ``(B, N, 2)``, normalized by default.
Convention:
- See the convention block of :func:`~kornia.geometry.subpix.spatial_soft_argmax2d`.
+
+ See Also:
+ - :func:`spatial_soft_argmax2d`: the function form.
+ - :class:`ConvSoftArgmax2d`: one soft-argmax per sliding window instead of per channel.
+
"""
def __init__(self, temperature: Optional[torch.Tensor | float] = None, normalized_coordinates: bool = True) -> None:
@@ -864,7 +951,15 @@ def conv_quad_interp3d(
dilation_radius: int = 1,
allow_scale_steps: bool = True,
) -> tuple[torch.Tensor, torch.Tensor]:
- r"""Subpixel localization of 3D scale-space extrema via quadratic interpolation.
+ r"""Refine 3D scale-space maxima by quadratic fit, precomputing the solve near each maximum.
+
+ ``coords, values = conv_quad_interp3d(response)`` takes a ``(B, C, D, H, W)`` scale-space response, with the scale
+ levels along ``D``, finds its strict ``3 x 3 x 3`` maxima with :func:`nms3d` (or reads ``precomputed_nms_mask``),
+ and returns for every voxel its ``(d, x, y)`` coordinates, ``(B, C, 3, D, H, W)``, and its value,
+ ``(B, C, D, H, W)``; only the refined maxima move off their grid index and change value. It solves the fit once
+ for every voxel within ``dilation_radius`` of a maximum and then follows the moves by table lookup, which suits
+ CUDA; :func:`iterative_quad_interp3d` solves again at each maximum on every iteration with no radius limit, which
+ suits the CPU, and :class:`AdaptiveQuadInterp3d` picks between the two by device.
For each NMS maximum the function fits a 3-D quadratic to the local
:math:`3 \times 3 \times 3` neighbourhood and solves for the sub-voxel
@@ -873,17 +968,21 @@ def conv_quad_interp3d(
and the solve is repeated — up to ``n_iters`` times.
Unlike a naive iterative approach, all Hessian solves are precomputed once at
- the start for every voxel that any keypoint could possibly visit (the
- **dilated NMS neighbourhood**, an L\ :math:`\infty` ball of radius
- ``dilation_radius`` around each maximum). The subsequent iteration loop contains
- **no data-dependent Python control flow** and no GPU→CPU synchronisation,
- making the function fully compatible with ``torch.compile`` / CUDA graphs.
+ the start for every voxel of the **dilated NMS neighbourhood**, an
+ L\ :math:`\infty` ball of radius ``dilation_radius`` around each maximum. The
+ iteration loop that follows contains **no data-dependent Python control flow**
+ and no GPU→CPU synchronisation. Finding the maxima and their neighbourhoods,
+ with ``torch.where`` on the mask and boolean indexing, does depend on the data:
+ on CUDA it synchronises with the host, and a default ``torch.compile`` breaks
+ the graph there; with ``torch._dynamo.config.capture_dynamic_output_shape_ops``
+ enabled, as :class:`~kornia.feature.ScaleSpaceDetector` does when it compiles
+ its subpix module, it is captured in one graph.
The ``dilation_radius`` controls the precompute footprint and should be set
to the maximum number of integer-centre moves expected per keypoint. With the
default ``max_subpixel_shift=0.6`` almost all keypoints converge within 1 move,
so the default ``dilation_radius=1`` (i.e. :math:`3^3 = 27` positions per
- maximum) is sufficient. Use ``dilation_radius=2`` (:math:`5^3 = 125`) for
+ maximum) covers almost all of them. Use ``dilation_radius=2`` (:math:`5^3 = 125`) for
extra safety. Setting it to ``n_iters`` gives the result of
:func:`iterative_quad_interp3d`, but is much slower on large images.
@@ -912,10 +1011,12 @@ def conv_quad_interp3d(
:func:`~kornia.geometry.subpix.nms3d` to skip the internal NMS call.
dilation_radius: L\ :math:`\infty` radius (in voxels) of the neighbourhood
around each NMS maximum where the Hessian solve is precomputed.
- Keypoints that attempt to move farther than this are marked invalid.
- allow_scale_steps: if ``True`` (default), the iterative shift is also
- applied along the scale (depth) axis; set to ``False`` to keep the
- keypoint on its original scale level.
+ Keypoints that move farther are rejected, keeping their grid coordinates
+ and input value, unless the voxels they reach were precomputed for
+ another maximum.
+ allow_scale_steps: if ``True`` (default), the integer centre may also move
+ along the scale (depth) axis; set to ``False`` to keep the integer centre
+ on its original scale level. The sub-voxel scale shift is applied either way.
Returns:
Tuple ``(coords_max, y_max)``:
@@ -926,6 +1027,20 @@ def conv_quad_interp3d(
* ``y_max`` — shape ``(B, C, D, H, W)``, quadratically corrected
response at each NMS maximum; other positions keep the input value.
+ Raises:
+ TypeError: if ``input`` is not a tensor.
+ ValueError: if ``input`` is not 5D.
+
+ See Also:
+ - :func:`iterative_quad_interp3d`: solves again at each maximum on every iteration, with no radius limit and
+ an optional ``max_candidates`` cap; faster on the CPU.
+ - :class:`AdaptiveQuadInterp3d`: by default runs this function on CUDA and :func:`iterative_quad_interp3d`
+ elsewhere.
+ - :class:`ConvQuadInterp3d`: the module form.
+ - :func:`nms3d`: the strict maxima this function refines; pass its mask as ``precomputed_nms_mask``.
+ - :func:`conv_soft_argmax3d`: a softmax-weighted position in every window instead of a quadratic fit at the
+ maxima.
+
Example:
>>> input = torch.randn(2, 3, 5, 64, 64)
>>> coords, vals = conv_quad_interp3d(input, n_iters=5)
@@ -1141,13 +1256,18 @@ def conv_quad_interp3d(
class ConvQuadInterp3d(nn.Module):
- r"""Subpixel localization of 3D scale-space extrema via quadratic interpolation.
+ r"""Refine scale-space maxima by quadratic fit; the module form of conv_quad_interp3d.
+
+ ``op = ConvQuadInterp3d(n_iters, max_subpixel_shift=0.6, dilation_radius=1)`` stores the arguments, and
+ ``coords, values = op(response, precomputed_nms_mask)`` returns what :func:`conv_quad_interp3d` returns for them:
+ the ``(B, C, 3, D, H, W)`` refined ``(d, x, y)`` coordinates and the ``(B, C, D, H, W)`` values of a
+ ``(B, C, D, H, W)`` response.
Wraps :func:`~kornia.geometry.subpix.conv_quad_interp3d`. The Hessian system
is solved once for each voxel in the dilated NMS neighbourhood (no dense
precomputation over the whole volume), then the shift chain is followed by
- table lookup with no GPU→CPU synchronisation — making the module compatible
- with ``torch.compile`` and CUDA graphs.
+ table lookup with no GPU→CPU synchronisation; finding the maxima still
+ synchronises, as :func:`~kornia.geometry.subpix.conv_quad_interp3d` describes.
Convention:
- See the convention block of :func:`~kornia.geometry.subpix.conv_quad_interp3d`.
@@ -1157,6 +1277,16 @@ class ConvQuadInterp3d(nn.Module):
strict_maxima_bonus: deprecated since kornia 0.9.0 and ignored; passing it emits a
:class:`DeprecationWarning`.
max_subpixel_shift: shift threshold that triggers integer centre move.
+ dilation_radius: radius of the precomputed neighbourhood, as in
+ :func:`~kornia.geometry.subpix.conv_quad_interp3d`.
+ allow_scale_steps: whether the integer centre may move along the scale axis, as in
+ :func:`~kornia.geometry.subpix.conv_quad_interp3d`.
+
+ See Also:
+ - :func:`conv_quad_interp3d`: the function form.
+ - :class:`IterativeQuadInterp3d`: the module that solves again at each maximum on every iteration.
+ - :class:`AdaptiveQuadInterp3d`: the module that picks this backend or the iterative one by device.
+
"""
def __init__(
@@ -1187,7 +1317,7 @@ class ConvQuadInterp3d(nn.Module):
def forward(
self, x: torch.Tensor, precomputed_nms_mask: Optional[torch.Tensor] = None
) -> tuple[torch.Tensor, torch.Tensor]:
- """Refine 3D extrema with convolution-based quadratic interpolation.
+ """Refine 3D extrema by quadratic interpolation, with the solve precomputed near each maximum.
The method starts from local maxima in a scale-space response volume
and fits a local quadratic model around each candidate. The fitted
@@ -1223,10 +1353,17 @@ def iterative_quad_interp3d(
precomputed_nms_mask: Optional[torch.Tensor] = None,
max_candidates: Optional[int] = None,
) -> tuple[torch.Tensor, torch.Tensor]:
- r"""Iterative subpixel localization of 3D extrema via quadratic interpolation.
+ r"""Refine 3D scale-space maxima by quadratic fit, re-solving at each maximum per iteration.
+
+ ``coords, values = iterative_quad_interp3d(response)`` takes a ``(B, C, D, H, W)`` scale-space response and returns
+ what :func:`conv_quad_interp3d` returns: for every voxel its ``(d, x, y)`` coordinates, ``(B, C, 3, D, H, W)``, and
+ its value, ``(B, C, D, H, W)``, moved and corrected only at the refined strict ``3 x 3 x 3`` maxima. It works on
+ the maxima alone, with no precomputed neighbourhood, so a maximum is not limited to a radius around its start, and
+ ``max_candidates`` caps the work per ``(b, c)`` map; it is the faster choice on the CPU, and
+ :func:`conv_quad_interp3d` on CUDA.
- Unlike :func:`conv_quad_interp3d`, which pre-computes the Hessian solve for all
- voxels reachable from NMS maxima and then follows shifts by table lookup, this
+ Unlike :func:`conv_quad_interp3d`, which pre-computes the Hessian solve for the
+ voxels near NMS maxima and then follows shifts by table lookup, this
function explicitly re-extracts the :math:`3 \times 3 \times 3` patch at each
NMS maximum and iterates up to ``n_iters`` times. When the estimated subpixel
shift along any spatial or scale axis exceeds ``max_subpixel_shift`` the integer
@@ -1245,13 +1382,13 @@ def iterative_quad_interp3d(
:func:`~kornia.geometry.subpix.nms3d` instead.
max_subpixel_shift: if the estimated shift along any axis is larger than this
threshold the integer center is displaced and another iteration is run.
- allow_scale_steps: if ``True`` (default), the iterative shift is also
- applied along the scale (depth) axis; set to ``False`` to keep the
- keypoint on its original scale level.
+ allow_scale_steps: if ``True`` (default), the integer centre may also move
+ along the scale (depth) axis; set to ``False`` to keep the integer centre
+ on its original scale level. The sub-voxel scale shift is applied either way.
precomputed_nms_mask: optional bool tensor of shape
``(B, C, D, H, W)`` — pass the result of
:func:`~kornia.geometry.subpix.nms3d` to skip the internal NMS call.
- max_candidates: if given, only the top-``max_candidates`` NMS maxima (ranked by
+ max_candidates: if given, only the top-``max_candidates`` NMS maxima of each ``(b, c)`` map (ranked by
pre-refinement response) are processed. The rest keep their grid-coordinate
values. This is a **CPU speed-up knob**: for large images the number of 3-D
NMS maxima can be 10x-100x larger than the desired number of keypoints,
@@ -1269,6 +1406,17 @@ def iterative_quad_interp3d(
* ``y_max`` has shape ``(B, C, D, H, W)`` and stores the quadratically
corrected response values.
+ Raises:
+ TypeError: if ``input`` is not a tensor.
+ ValueError: if ``input`` is not 5D or ``max_candidates`` is negative.
+
+ See Also:
+ - :func:`conv_quad_interp3d`: precomputes the solve within ``dilation_radius`` of each maximum and follows the
+ moves by lookup; faster on CUDA.
+ - :class:`AdaptiveQuadInterp3d`: by default runs this function except on CUDA.
+ - :class:`IterativeQuadInterp3d`: the module form.
+ - :func:`nms3d`: the strict maxima this function refines; pass its mask as ``precomputed_nms_mask``.
+
Example:
>>> input = torch.randn(2, 3, 3, 8, 8)
>>> coords, vals = iterative_quad_interp3d(input, n_iters=5)
@@ -1460,10 +1608,22 @@ def iterative_quad_interp3d(
class IterativeQuadInterp3d(nn.Module):
- r"""Iterative subpixel localization of 3D extrema via quadratic interpolation.
+ r"""Refine scale-space maxima by quadratic fit; the module form of iterative_quad_interp3d.
+
+ ``op = IterativeQuadInterp3d(n_iters, max_subpixel_shift=0.6, max_candidates=None)`` stores the arguments, and
+ ``coords, values = op(response, precomputed_nms_mask)`` returns what :func:`iterative_quad_interp3d` returns for
+ them: the ``(B, C, 3, D, H, W)`` refined ``(d, x, y)`` coordinates and the ``(B, C, D, H, W)`` values of a
+ ``(B, C, D, H, W)`` response. Passing ``strict_maxima_bonus`` emits a :class:`DeprecationWarning` and has no
+ effect.
Convention:
- See the convention block of :func:`~kornia.geometry.subpix.iterative_quad_interp3d`.
+
+ See Also:
+ - :func:`iterative_quad_interp3d`: the function form.
+ - :class:`ConvQuadInterp3d`: the module that precomputes the solve near each maximum.
+ - :class:`AdaptiveQuadInterp3d`: the module that picks this backend or the precomputing one by device.
+
"""
def __init__(
@@ -1519,19 +1679,21 @@ class IterativeQuadInterp3d(nn.Module):
class AdaptiveQuadInterp3d(nn.Module):
- r"""Subpixel localization of 3D scale-space extrema with automatic backend selection.
+ r"""Refine 3D scale-space maxima by quadratic fit with the faster backend for the device.
+
+ ``op = AdaptiveQuadInterp3d(mode="auto")`` stores the arguments of both backends, and
+ ``coords, values = op(response, precomputed_nms_mask)`` calls :func:`conv_quad_interp3d` with ``mode="conv"``, or
+ with ``mode="auto"`` for an input on CUDA, and :func:`iterative_quad_interp3d` otherwise (``mode="patch"``, or
+ ``"auto"`` on any other device). Both return the ``(B, C, 3, D, H, W)`` refined ``(d, x, y)`` coordinates and the
+ ``(B, C, D, H, W)`` values of a ``(B, C, D, H, W)`` response. With ``max_candidates`` unset they agree on every
+ maximum whose centre stays within ``dilation_radius`` voxels of its start; beyond that the conv backend may
+ reject a maximum the iterative one refines, so CPU and CUDA keypoints can differ, and ``dilation_radius=n_iters``
+ makes the two backends agree. It is the default ``subpix_module`` of :class:`~kornia.feature.ScaleSpaceDetector`.
Wraps :func:`~kornia.geometry.subpix.conv_quad_interp3d` and
:func:`~kornia.geometry.subpix.iterative_quad_interp3d`, choosing the faster
backend based on the input device and the requested ``mode``.
- Benchmarks show:
-
- * **GPU** — :func:`conv_quad_interp3d` is 1.5-2x faster due to better
- parallelism on the batched gather+solve.
- * **CPU** — :func:`iterative_quad_interp3d` is faster for large images because
- it processes only the NMS maxima directly without any dilation/dedup overhead.
-
Convention:
- See the convention block of :func:`~kornia.geometry.subpix.conv_quad_interp3d`.
@@ -1549,11 +1711,21 @@ class AdaptiveQuadInterp3d(nn.Module):
max_subpixel_shift: integer-centre move threshold.
dilation_radius: L\ :math:`\infty` precompute radius for ``"conv"`` mode
(ignored in ``"patch"`` mode).
+ allow_scale_steps: whether the integer centre may move along the scale axis, in both modes.
max_candidates: if set, only the top-``max_candidates`` NMS maxima by
pre-refinement response are processed in ``"patch"`` mode. Has no effect
in ``"conv"`` mode. Useful on CPU when the number of 3-D NMS maxima greatly
exceeds the desired number of keypoints (see :func:`iterative_quad_interp3d`).
+ Raises:
+ ValueError: if ``mode`` is not ``"patch"``, ``"conv"`` or ``"auto"``.
+
+ See Also:
+ - :func:`conv_quad_interp3d`: the backend on CUDA, with a ``dilation_radius`` limit.
+ - :func:`iterative_quad_interp3d`: the backend elsewhere, with an optional ``max_candidates`` cap.
+ - :class:`ConvQuadInterp3d`: the conv backend alone, as a module.
+ - :class:`IterativeQuadInterp3d`: the iterative backend alone, as a module.
+
Example:
>>> inp = torch.randn(1, 1, 3, 64, 64)
>>> subpix = AdaptiveQuadInterp3d(mode="auto")
diff --git a/kornia/geometry/vector.py b/kornia/geometry/vector.py
index 8cf37f0a0..942bce444 100644
--- a/kornia/geometry/vector.py
+++ b/kornia/geometry/vector.py
@@ -15,6 +15,26 @@
# limitations under the License.
#
+"""2D and 3D vector and scalar wrappers that planes, Lie groups and cameras take.
+
+:class:`Vector3`, :class:`Vector2` and :class:`Scalar` wrap a tensor without copying it, of shape ``(*, 3)``,
+``(*, 2)`` and any shape. They are not ``torch.Tensor`` subclasses: ``v.data`` is the wrapped tensor. Torch
+functions, tensor methods and operators accept a wrapper and wrap their result by the rule of
+:class:`~kornia.core.TensorWrapper`, while kornia functions that check for a tensor raise on one.
+
+Which class to use:
+
+- 3D points, directions and normals: :class:`Vector3`. :class:`~kornia.geometry.plane.Hyperplane` takes its normal
+ as one, and :class:`~kornia.geometry.liegroup.So3` and :class:`~kornia.geometry.liegroup.Se3` map a ``Vector3`` to
+ a ``Vector3``.
+- 2D points and directions: :class:`Vector2`. :class:`~kornia.geometry.liegroup.So2` and
+ :class:`~kornia.geometry.liegroup.Se2` map a ``Vector2`` to a ``Vector2``, and the camera models of
+ ``kornia.sensors.camera`` (:class:`~kornia.sensors.camera.CameraModelBase`) project a ``Vector3`` to one.
+- One value per vector, such as a dot product or the offset of a plane: :class:`Scalar`.
+- Points for kornia's tensor functions, such as :func:`~kornia.geometry.linalg.transform_points`: plain ``(*, 3)``
+ or ``(*, 2)`` tensors, or ``v.data``.
+"""
+
from typing import Optional, Tuple, Union, cast
import torch
@@ -29,10 +49,20 @@ __all__ = ["Scalar", "Vector2", "Vector3"]
# TODO: implement more functionality to validate
class Scalar(TensorWrapper):
- """Wrap a tensor of scalars of any shape, such as the per-vector result of :meth:`Vector3.dot`.
-
- The tensor is wrapped without a copy. An operation between a ``Scalar`` and a :class:`Vector3` follows the
- rule of :class:`~kornia.core.TensorWrapper`, so ``s * v`` is a ``Vector3`` like ``v * s``.
+ """Wrap a tensor of scalars of any shape, such as dot products and plane offsets.
+
+ ``s = Scalar(values)`` wraps a tensor of any shape without copying it; it adds no methods, and ``s.data`` is the
+ tensor. :meth:`Vector3.dot` and :meth:`Vector3.squared_norm` return one, and
+ :class:`~kornia.geometry.plane.Hyperplane` takes its offset as one and returns its distances as one. An operation
+ between a ``Scalar`` and a :class:`Vector3` follows the rule of :class:`~kornia.core.TensorWrapper`, so ``s * v``
+ is a ``Vector3`` like ``v * s``. Broadcasting follows the wrapped tensors and adds no coordinate axis: the
+ ``(B,)`` result of :meth:`Vector3.dot` meets the last axis of ``(B, 3)`` vectors, so ``s * v`` scales
+ coordinates for ``B == 3``, scales the one vector for ``B == 1`` and raises for any other ``B``. Scale vector by
+ vector with ``s.data[..., None] * v``.
+
+ See Also:
+ - :class:`Vector3`: 3D vectors ``(*, 3)``, whose dot products and squared norms come back as a ``Scalar``.
+ - :class:`Vector2`: 2D vectors ``(*, 2)``, likewise.
"""
def __init__(self, data: torch.Tensor) -> None:
@@ -40,12 +70,21 @@ class Scalar(TensorWrapper):
class Vector3(TensorWrapper):
- r"""Wrap a tensor of 3D vectors, shape ``(*, 3)``.
+ r"""Wrap a ``(*, 3)`` tensor of 3D vectors, with coordinates, dot products and normalisation.
+
+ ``v = Vector3(points)`` wraps a ``(*, 3)`` tensor without copying it, and ``Vector3.from_coords(x, y, z)`` and
+ ``Vector3.random(shape)`` build one; ``v.x``, ``v.y`` and ``v.z`` are plain ``(*)`` tensors, ``v.dot(w)`` and
+ ``v.squared_norm()`` return a :class:`Scalar` ``(*)``, and ``v.normalized()`` the unit vectors, with the norm
+ floored at ``1e-12``. A ``Vector3`` is not a ``torch.Tensor``: torch functions, tensor methods and operators
+ accept it (``torch.linalg.cross(v, w)`` is a ``Vector3``), but a kornia function that checks for a tensor, such
+ as :func:`~kornia.geometry.linalg.transform_points`, raises, so pass it ``v.data``. For 2D vectors, use
+ :class:`Vector2` instead.
Convention:
- - The tensor is wrapped without a copy; any leading shape and dtype are accepted. :attr:`x`, :attr:`y` and
- :attr:`z` are plain tensors of the leading shape ``(*)``, and :meth:`dot` and :meth:`squared_norm`
- return a :class:`Scalar` of that shape (for :meth:`dot`, the two operands' broadcast leading shape).
+ - The tensor is wrapped without a copy; any leading shape and dtype are accepted, and a last axis other than
+ 3 raises ``BaseError``. :attr:`x`, :attr:`y` and :attr:`z` are plain tensors of the leading shape ``(*)``,
+ and :meth:`dot` and :meth:`squared_norm` return a :class:`Scalar` of that shape (for :meth:`dot`, the two
+ operands' broadcast leading shape).
- :meth:`random` draws vectors uniformly in the unit cube from torch's global generator, so every vector
lies in the first octant: it is not a random direction.
- ``copy.deepcopy``, ``copy.copy`` and pickle return a ``Vector3``. An in-place operator such as ``v += 1``
@@ -59,6 +98,18 @@ class Vector3(TensorWrapper):
- Indexing the leading axes returns a ``Vector3`` (``v[0]``, ``v[1:]``, ``v[mask]``). An index into the
coordinate axis returns a plain tensor, like the same index into the wrapped tensor: ``v[..., 0]`` and
``v[:, 0]`` have the leading shape, and an unbatched ``v[0]`` is 0-d.
+
+ See Also:
+ - :class:`Vector2`: the 2D vector ``(*, 2)``, with ``x`` and ``y`` only.
+ - :class:`Scalar`: the wrapper for one value per vector, which :meth:`dot` and :meth:`squared_norm` return.
+ - :class:`~kornia.core.TensorWrapper`: the base class, whose rule decides which results stay a ``Vector3``.
+
+ Example:
+ >>> v = Vector3.from_coords(3.0, 0.0, 4.0)
+ >>> v.normalized().data
+ tensor([0.6000, 0.0000, 0.8000])
+ >>> v.dot(Vector3(torch.tensor([1.0, 1.0, 1.0]))).data
+ tensor(7.)
"""
_WIDTH = 3
@@ -175,15 +226,15 @@ class Vector3(TensorWrapper):
"""Construct a 3D vector wrapper from x, y, and z coordinates.
All coordinate inputs must share the same Python type (all floats or
- all tensors). For tensor inputs, coordinates are stacked along the last
- dimension to produce ``(..., 3)`` output.
+ all tensors); Python ints raise, so write ``1.0``. For tensor inputs, coordinates are
+ stacked along the last dimension to produce ``(..., 3)`` output.
Args:
x: X-coordinate value or tensor.
y: Y-coordinate value or tensor.
z: Z-coordinate value or tensor.
- device: Device used when scalar inputs are converted to tensor.
- dtype: Dtype used when scalar inputs are converted to tensor.
+ device: Device used when scalar inputs are converted to tensor; ignored for tensors.
+ dtype: Dtype used when scalar inputs are converted to tensor; ignored for tensors.
Returns:
:class:`Vector3` containing the assembled coordinates.
@@ -198,10 +249,19 @@ class Vector3(TensorWrapper):
class Vector2(TensorWrapper):
- r"""Wrap a tensor of 2D vectors, shape ``(*, 2)``.
+ r"""Wrap a ``(*, 2)`` tensor of 2D vectors, with coordinates, dot products and normalisation.
+
+ ``v = Vector2(points)`` wraps a ``(*, 2)`` tensor without copying it, and ``Vector2.from_coords(x, y)`` and
+ ``Vector2.random(shape)`` build one; ``v.x`` and ``v.y`` are plain ``(*)`` tensors, ``v.dot(w)`` and
+ ``v.squared_norm()`` return a :class:`Scalar` ``(*)``, and ``v.normalized()`` the unit vectors, with the norm
+ floored at ``1e-12``. For 3D vectors, use :class:`Vector3` instead.
See the Convention block on :class:`Vector3`, which applies to ``(x, y)`` vectors; :meth:`random` fills the
unit square.
+
+ See Also:
+ - :class:`Vector3`: the 3D vector ``(*, 3)``, whose conventions apply here.
+ - :class:`Scalar`: the wrapper for one value per vector, which :meth:`dot` and :meth:`squared_norm` return.
"""
_WIDTH = 2
@@ -289,14 +349,14 @@ class Vector2(TensorWrapper):
"""Construct a 2D vector wrapper from x and y coordinates.
Both coordinates must share the same Python type (both floats or both
- tensors). For tensor inputs, coordinates are stacked along the last
- dimension to produce ``(..., 2)`` output.
+ tensors); Python ints raise, so write ``1.0``. For tensor inputs, coordinates are
+ stacked along the last dimension to produce ``(..., 2)`` output.
Args:
x: X-coordinate value or tensor.
y: Y-coordinate value or tensor.
- device: Device used when scalar inputs are converted to tensor.
- dtype: Dtype used when scalar inputs are converted to tensor.
+ device: Device used when scalar inputs are converted to tensor; ignored for tensors.
+ dtype: Dtype used when scalar inputs are converted to tensor; ignored for tensors.
Returns:
:class:`Vector2` containing the assembled coordinates.
diff --git a/testing/docstring_rollout.py b/testing/docstring_rollout.py
index 057289874..04bc91310 100644
--- a/testing/docstring_rollout.py
+++ b/testing/docstring_rollout.py
@@ -53,13 +53,26 @@ EXCLUDED_PACKAGES: dict[str, str] = {
"""Top-level packages left out of the rollout, with the reason."""
CATALOG_NAMESPACES: tuple[str, ...] = (
+ "kornia.geometry.bbox",
+ "kornia.geometry.boxes",
+ "kornia.geometry.calibration",
+ "kornia.geometry.camera",
+ "kornia.geometry.camera.stereo",
"kornia.geometry.conversions",
"kornia.geometry.depth",
"kornia.geometry.epipolar",
+ "kornia.geometry.grid",
"kornia.geometry.homography",
+ "kornia.geometry.keypoints",
+ "kornia.geometry.liegroup",
+ "kornia.geometry.linalg",
+ "kornia.geometry.line",
+ "kornia.geometry.plane",
+ "kornia.geometry.pointcloud",
"kornia.geometry.pose",
"kornia.geometry.ransac",
"kornia.geometry.solvers",
+ "kornia.geometry.subpix",
"kornia.geometry.transform.affwarp",
"kornia.geometry.transform.crop2d",
"kornia.geometry.transform.crop3d",
@@ -70,18 +83,33 @@ CATALOG_NAMESPACES: tuple[str, ...] = (
"kornia.geometry.transform.imgwarp",
"kornia.geometry.transform.pyramid",
"kornia.geometry.transform.thin_plate_spline",
+ "kornia.geometry.vector",
)
"""Namespaces whose module docstring must name every export in ``__all__``."""
DOCSTRING_MODULES: tuple[str, ...] = (
+ "kornia.geometry.bbox",
+ "kornia.geometry.boxes",
+ "kornia.geometry.calibration",
+ "kornia.geometry.camera",
"kornia.geometry.conversions",
"kornia.geometry.depth",
"kornia.geometry.epipolar",
+ "kornia.geometry.grid",
"kornia.geometry.homography",
+ "kornia.geometry.keypoints",
+ "kornia.geometry.liegroup",
+ "kornia.geometry.linalg",
+ "kornia.geometry.line",
+ "kornia.geometry.plane",
+ "kornia.geometry.pointcloud",
"kornia.geometry.pose",
+ "kornia.geometry.quaternion",
"kornia.geometry.ransac",
"kornia.geometry.solvers",
+ "kornia.geometry.subpix",
"kornia.geometry.transform",
+ "kornia.geometry.vector",
)
"""Prefixes of the defining modules whose public objects carry the converted docstring style."""
diff --git a/testing/docstring_style.py b/testing/docstring_style.py
index e00250390..02fca6650 100644
--- a/testing/docstring_style.py
+++ b/testing/docstring_style.py
@@ -17,9 +17,10 @@
"""Helpers shared by the tests that check kornia's docstring style.
-They find the public API, read the structure of a docstring (``See Also`` entries, cross-reference roles, shape
-tuples) and resolve a cross-reference the way Sphinx does, through the names that the ``.rst`` pages under
-``docs/source`` register and, for torch, Python and NumPy objects, the cached intersphinx inventories.
+They find the public API (exported by an ``__all__`` or documented on a docs page), read the structure of a
+docstring (``See Also`` entries, cross-reference roles, shape tuples) and resolve a cross-reference the way Sphinx
+does, through the names that the ``.rst`` pages under ``docs/source`` register and, for torch, Python and NumPy
+objects, the cached intersphinx inventories.
"""
from __future__ import annotations
@@ -100,6 +101,10 @@ def _is_kornia_function_or_class(obj: object) -> bool:
return module_of_obj == "kornia" or module_of_obj.startswith("kornia.")
+def _qualified_name(obj: object) -> str:
+ return f"{obj.__module__}.{obj.__qualname__}"
+
+
@cache
def _exported_objects() -> tuple[tuple[object, str], ...]:
"""List ``(obj, "module.qualname")`` for every exported kornia function and class, deduplicated by identity."""
@@ -114,7 +119,7 @@ def _exported_objects() -> tuple[tuple[object, str], ...]:
exported = [getattr(module, name, None) for name in module.__all__ if not name.startswith("_")]
for obj in exported:
if _is_kornia_function_or_class(obj):
- collected.setdefault(id(obj), (obj, f"{obj.__module__}.{obj.__qualname__}"))
+ collected.setdefault(id(obj), (obj, _qualified_name(obj)))
return tuple(collected.values())
@@ -139,6 +144,33 @@ def undocumented_public_objects() -> dict[int, tuple[object, str]]:
return {id(obj): (obj, qualified) for obj, qualified in _exported_objects() if not own_docstring(obj)}
+def documented_objects(registry: Mapping[str, object]) -> tuple[tuple[object, str], ...]:
+ """List ``(obj, "module.qualname")`` for every kornia function and class that a docs page registers at top level.
+
+ These are the registry entries that are kornia functions or classes and are not members of a registered class
+ (the methods and nested classes autodoc renders inside a class page are the member check's business). Together
+ with ``_exported_objects()`` they make up what the rollout checks; an object that is both exported and documented
+ is the same object in both and counted once by ``rollout_objects``.
+ """
+ collected: dict[int, tuple[object, str]] = {}
+ for name, obj in registry.items():
+ if not _is_kornia_function_or_class(obj) or inspect.isclass(registry.get(name.rpartition(".")[0])):
+ continue
+ collected.setdefault(id(obj), (obj, _qualified_name(obj)))
+ return tuple(collected.values())
+
+
+def rollout_objects(registry: Mapping[str, object], with_docstring: bool = True) -> dict[int, tuple[object, str]]:
+ """Map ``id(obj)`` to ``(obj, "module.qualname")`` for the objects the docstring checks select.
+
+ The union of the exported kornia functions and classes (``public_objects()``'s walk) and the ones a docs page
+ registers (``documented_objects(registry)``), deduplicated by identity. ``with_docstring=True`` keeps those with a
+ docstring of their own, ``with_docstring=False`` those without one. Not cached: it depends on ``registry``.
+ """
+ union = {id(obj): (obj, qualified) for obj, qualified in (*_exported_objects(), *documented_objects(registry))}
+ return {key: pair for key, pair in union.items() if bool(own_docstring(pair[0])) == with_docstring}
+
+
def covered(module_name: str, prefixes: Iterable[str]) -> bool:
"""Whether ``module_name`` equals one of ``prefixes`` or lies below it (``prefix + "."``)."""
return any(module_name == prefix or module_name.startswith(prefix + ".") for prefix in prefixes)
@@ -495,7 +527,9 @@ _ATTRIBUTE_COMMENT = re.compile(r"^\s*#: ?(.*)$")
class _AttributeDocPicker(ast.NodeVisitor):
- """Collect ``(namespace, name)`` of the documented variables of a module, as ``VariableCommentPicker`` does.
+ """Collect the documented variables of a module and their text, as ``VariableCommentPicker`` does.
+
+ ``found`` maps ``(namespace, name)`` to the documentation text: the ``#:`` comment, or the string literal.
An assignment documents its targets with a ``#:`` comment after it, with ``#:`` lines right above it, or with
a string literal right after it. The namespace is the enclosing class path, ``""`` at module level. Inside a
@@ -503,14 +537,14 @@ class _AttributeDocPicker(ast.NodeVisitor):
``except`` and ``finally`` blocks are not read.
"""
- def __init__(self, lines: list[str], comment_columns: dict[int, int]) -> None:
+ def __init__(self, lines: list[str], comments: dict[int, tuple[int, str]]) -> None:
self.lines = lines
- self.comment_columns = comment_columns
+ self.comments = comments
self.context: list[str] = []
self.classes = 0
self.function: ast.FunctionDef | ast.AsyncFunctionDef | None = None
self.previous: ast.AST | None = None
- self.found: set[tuple[str, str]] = set()
+ self.found: dict[tuple[str, str], str] = {}
def visit(self, node: ast.AST) -> None:
super().visit(node)
@@ -548,10 +582,10 @@ class _AttributeDocPicker(ast.NodeVisitor):
return ".".join(self.context[:-1])
return None
- def _record(self, names: list[str]) -> None:
+ def _record(self, names: list[str], text: str) -> None:
namespace = self._namespace()
if namespace is not None:
- self.found.update((namespace, name) for name in names)
+ self.found.update(((namespace, name), text) for name in names)
def visit_Assign(self, node: ast.Assign | ast.AnnAssign) -> None:
targets = node.targets if isinstance(node, ast.Assign) else [node.target]
@@ -559,13 +593,17 @@ class _AttributeDocPicker(ast.NodeVisitor):
names = [name for target in targets for name in self._names(target)]
except TypeError:
return
- column = self.comment_columns.get(node.end_lineno or node.lineno)
- if (column is not None and column >= (node.end_col_offset or 0)) or (
- node.lineno > 1
- and not self.lines[node.lineno - 1][: node.col_offset].strip()
- and _ATTRIBUTE_COMMENT.match(self.lines[node.lineno - 2])
- ):
- self._record(names)
+ trailing = self.comments.get(node.end_lineno or node.lineno)
+ above = []
+ if node.lineno > 1 and not self.lines[node.lineno - 1][: node.col_offset].strip():
+ index = node.lineno - 2
+ while index >= 0 and (comment := _ATTRIBUTE_COMMENT.match(self.lines[index])):
+ above.insert(0, comment.group(1))
+ index -= 1
+ if trailing is not None and trailing[0] >= (node.end_col_offset or 0):
+ self._record(names, trailing[1])
+ elif above:
+ self._record(names, "\n".join(above))
visit_AnnAssign = visit_Assign
@@ -578,7 +616,7 @@ class _AttributeDocPicker(ast.NodeVisitor):
):
target = previous.targets[0] if isinstance(previous, ast.Assign) else previous.target
with contextlib.suppress(TypeError):
- self._record(self._names(target))
+ self._record(self._names(target), inspect.cleandoc(node.value.value))
def visit_Try(self, node: ast.Try) -> None:
for child in [*node.body, *node.orelse]:
@@ -607,20 +645,23 @@ class _AttributeDocPicker(ast.NodeVisitor):
@cache
-def _attribute_docs(module_name: str) -> frozenset[tuple[str, str]]:
- """``(namespace, name)`` of the variables the source of ``module_name`` documents (see ``_AttributeDocPicker``)."""
+def _attribute_docs(module_name: str) -> Mapping[tuple[str, str], str]:
+ """``(namespace, name)`` to documentation text of the variables the source of ``module_name`` documents.
+
+ See ``_AttributeDocPicker``.
+ """
path = getattr(sys.modules.get(module_name), "__file__", None)
if not path or not path.endswith(".py"):
- return frozenset()
+ return {}
source = Path(path).read_text(encoding="utf-8")
- comment_columns = {
- token.start[0]: token.start[1]
+ comments = {
+ token.start[0]: (token.start[1], _ATTRIBUTE_COMMENT.match(token.string).group(1))
for token in tokenize.generate_tokens(io.StringIO(source).readline)
if token.type == tokenize.COMMENT and _ATTRIBUTE_COMMENT.match(token.string)
}
- picker = _AttributeDocPicker(source.splitlines(), comment_columns)
+ picker = _AttributeDocPicker(source.splitlines(), comments)
picker.visit(ast.parse(source))
- return frozenset(picker.found)
+ return picker.found
def _is_data(value: object) -> bool:
@@ -881,12 +922,20 @@ def intersphinx_names(docs_source: Path) -> dict[str, frozenset[str]]:
return {name: frozenset(kinds) for name, kinds in types.items()}
-def _reference_scopes(obj: object, registry: dict[str, object]) -> list[str]:
- """The prefixes Sphinx tries a reference in the docstring of ``obj`` under: its module, then its class."""
+def _reference_scopes(obj: object, registry: dict[str, object], registered_as: str | None = None) -> list[str]:
+ """The prefixes Sphinx tries a reference in the docstring of ``obj`` under: its module, then its class.
+
+ With ``registered_as``, a name ``registry`` gives ``obj``, only the scopes of that one rendered entry; without it,
+ those of every name ``registry`` gives ``obj``.
+ """
+ if registered_as is None:
+ names = [name for name, registered in registry.items() if registered is obj]
+ elif registry.get(registered_as) is obj:
+ names = [registered_as]
+ else:
+ raise ValueError(f"the registry does not register the object as {registered_as!r}")
scopes: list[str] = []
- for name, registered in registry.items():
- if registered is not obj:
- continue
+ for name in names:
parent = name.rpartition(".")[0]
if inspect.isclass(obj):
scopes += [parent, name]
@@ -897,6 +946,36 @@ def _reference_scopes(obj: object, registry: dict[str, object]) -> list[str]:
return [scope for scope in scopes if scope]
+def documented_members(registry: Mapping[str, object], classes: Iterable[object]) -> dict[str, tuple[object, str]]:
+ """Map each member autodoc documents under one of ``classes`` to its registered name, as ``(member, docstring)``.
+
+ These are the methods, properties and data attributes Sphinx renders inside a class page, found as the registry
+ entries whose parent entry is one of ``classes`` (by identity). The docstring is the one autodoc shows, so an
+ override without one has its base's; for a data attribute it is the text of its ``#:`` comment or string. A member
+ without a docstring is left out, as there is nothing to check in it. A method registered under several of the
+ classes appears once per registered name; check each under that name's scopes alone by passing it as
+ ``registered_as`` to ``reference_resolves``.
+ """
+ wanted = list(classes)
+ members: dict[str, tuple[object, str]] = {}
+ for name, member in registry.items():
+ parent = registry.get(name.rpartition(".")[0])
+ if parent is None or not any(parent is cls for cls in wanted):
+ continue
+ if isinstance(member, _Attribute):
+ doc = _attribute_docs(member.owner.__module__).get((member.owner.__qualname__, member.name))
+ else:
+ doc = inspect.getdoc(member)
+ if doc is None:
+ # A class method is registered as its ``__func__``, on which ``inspect.getdoc`` does not find the
+ # docstring an override without one inherits; autodoc shows that one, so read it through the class.
+ bound = getattr(parent, name.rpartition(".")[2], None)
+ doc = None if bound is None else inspect.getdoc(bound)
+ if doc:
+ members[name] = (member, doc)
+ return members
+
+
def is_documented(obj: object, registry: dict[str, object]) -> bool:
"""Whether a docs page registers ``obj`` itself, so that short names in its docstring have a scope."""
return any(registered is obj for registered in registry.values())
@@ -907,7 +986,13 @@ def _role_target(target: str, role: str | None) -> str:
return target.removesuffix("()") if role in (None, "func", "meth") else target
-def resolve_reference(obj: object, target: str, registry: dict[str, object], role: str | None = None) -> object | None:
+def resolve_reference(
+ obj: object,
+ target: str,
+ registry: dict[str, object],
+ role: str | None = None,
+ registered_as: str | None = None,
+) -> object | None:
"""Return the registered object that ``target``, written in the docstring of ``obj``, resolves to, or ``None``.
As Sphinx does for a Python-domain role: a trailing ``()`` is dropped for ``:func:`` and ``:meth:`` (and when
@@ -917,9 +1002,15 @@ def resolve_reference(obj: object, target: str, registry: dict[str, object], rol
its type. A short import alias is not registered, so it does not resolve; neither does a short name in the
docstring of an object that no docs page registers. Only kornia objects resolve here; see
``intersphinx_names`` for the rest.
+
+ Sphinx resolves each rendered entry under its own class and module. An object registered under several names,
+ such as a method inherited by several documented classes, gets the scopes of every one of them unless
+ ``registered_as`` names the entry to check: then only that entry's scopes count, so a short name registered
+ under one class does not resolve on another class's page. ``registered_as`` must be a name ``registry`` gives
+ ``obj``; otherwise ``ValueError`` is raised.
"""
target = _role_target(target, role)
- candidates = [target, *(f"{scope}.{target}" for scope in _reference_scopes(obj, registry))]
+ candidates = [target, *(f"{scope}.{target}" for scope in _reference_scopes(obj, registry, registered_as))]
for candidate in candidates:
if candidate in registry:
return registry[candidate]
@@ -932,15 +1023,16 @@ def reference_resolves(
registry: dict[str, object],
external: Mapping[str, frozenset[str]] | None = None,
role: str | None = None,
+ registered_as: str | None = None,
) -> bool:
"""Whether ``target``, in a ``role`` role in the docstring of ``obj``, renders as a link.
- True when it resolves to a registered kornia object (see ``resolve_reference``) or is an ``external`` name, such
- as one from ``intersphinx_names(docs_source)``, of an object type that intersphinx links ``role`` to. Unlike the
- local lookup, intersphinx matches the type: ``:meth:`torch.svd``` (a function) and every ``:const:`` stay plain
- text. A ``role`` of ``None`` accepts any type.
+ True when it resolves to a registered kornia object (see ``resolve_reference``, which also explains
+ ``registered_as``) or is an ``external`` name, such as one from ``intersphinx_names(docs_source)``, of an object
+ type that intersphinx links ``role`` to. Unlike the local lookup, intersphinx matches the type:
+ ``:meth:`torch.svd``` (a function) and every ``:const:`` stay plain text. A ``role`` of ``None`` accepts any type.
"""
- if resolve_reference(obj, target, registry, role) is not None:
+ if resolve_reference(obj, target, registry, role, registered_as) is not None:
return True
types = (external or {}).get(_role_target(target, role), frozenset())
return bool(types if role is None else types & _ROLE_OBJECT_TYPES.get(role, frozenset()))
diff --git a/tests/geometry/test_bbox.py b/tests/geometry/test_bbox.py
index 7b9fca172..56e9e9585 100644
--- a/tests/geometry/test_bbox.py
+++ b/tests/geometry/test_bbox.py
@@ -798,9 +798,10 @@ class TestBbox3D(BaseTester):
def test_wart_validate_bbox3d_checks_equal_edge_extents_not_cuboids_4013(self, device, dtype, monkeypatch):
# Wart pin for kornia#4013 (the 3D validator raises where the 2D one returns False, and its one
# False result, a non-finite coordinate, warns that it will raise too): the check compares
- # the inclusive extents of the four edges parallel to each axis, so a sheared parallelepiped
- # and a zero-extent box pass, and only unequal edges raise. Under graph capture the extent
- # check is skipped and the shape check alone returns True.
+ # the inclusive x extents of the four left-to-right edges, the y extents of the two diagonals
+ # of the front face and of the back face, and the z extents of the four front-to-back edges,
+ # so a sheared parallelepiped and a zero-extent box pass, and only disagreeing extents raise.
+ # Under graph capture the extent check is skipped and the shape check alone returns True.
cuboid = self._unit_cuboid(device, dtype)
assert validate_bbox3d(cuboid) is True
assert validate_bbox3d(cuboid[None]) is True
diff --git a/tests/test_docstring_style.py b/tests/test_docstring_style.py
index 396739798..49f85f4b8 100644
--- a/tests/test_docstring_style.py
+++ b/tests/test_docstring_style.py
@@ -20,8 +20,9 @@
``testing/docstring_rollout.py`` lists the modules that are converted. For those, this file enforces what a reader,
an IDE or a coding agent relies on and what a refactor silently breaks; the shape rule applies everywhere:
-* every public function and class has a docstring, so none escapes the checks below;
-* every public function and class has a ``See Also:`` section whose entries name a sibling and say how it differs,
+* every public function and class, exported by an ``__all__`` or documented on a docs page, has a docstring, so none
+ escapes the checks below;
+* every such function and class has a ``See Also:`` section whose entries name a sibling and say how it differs,
unless its title line is a status title, ``Alias of :func:`target`.`` or ``Deprecated: use :func:`target`.``,
which points to another public kornia object and needs none (``Deprecated: use`` only on an object that
``kornia.core._compat.deprecated`` wraps, ``Alias of`` only on one that forwards to its target unchanged, as
@@ -29,7 +30,8 @@ an IDE or a coding agent relies on and what a refactor silently breaks; the shap
* every Python-domain object role (``:func:``, ``:class:``, ``:meth:``, ``:attr:``, ``:data:``, ``:const:``,
``:exc:``, ``:obj:``) resolves to a name that Sphinx registers from ``docs/source``, or to a torch, Python or NumPy
object of a type that role links in the cached intersphinx inventories, so it renders as a link and not as plain
- text (a short import alias is importable but is not a target); See Also entries name kornia objects only;
+ text (a short import alias is importable but is not a target), also in the docstrings of the methods, properties
+ and attributes autodoc documents under a converted class; See Also entries name kornia objects only;
* every ``kornia/**/*.py`` file and every page Sphinx builds from ``docs/source`` writes a tensor shape as a literal
such as ``(B, 1, H, W)``, not as ``:math:``, except for the values listed in ``MATH_NON_SHAPES``;
* the module docstring of a namespace names every export of its ``__all__`` in a role or a double-backtick literal.
@@ -62,6 +64,7 @@ from testing.docstring_style import (
catalog_unimportable,
covered,
docstring_roles,
+ documented_members,
forwards_to,
intersphinx_names,
is_deprecated_wrapper,
@@ -70,14 +73,13 @@ from testing.docstring_style import (
math_shape_roles,
modules_declaring_all,
own_docstring,
- public_objects,
reference_resolves,
resolve_reference,
+ rollout_objects,
rst_pages,
see_also_entries,
sphinx_registry,
status_target,
- undocumented_public_objects,
)
KORNIA_ROOT = Path(kornia.__file__).resolve().parent
@@ -87,17 +89,24 @@ REGISTRY = sphinx_registry(DOCS)
INTERSPHINX = intersphinx_names(DOCS)
CONVERTED = sorted(
- ((qualified, obj) for obj, qualified in public_objects().values() if covered(obj.__module__, DOCSTRING_MODULES)),
+ (
+ (qualified, obj)
+ for obj, qualified in rollout_objects(REGISTRY).values()
+ if covered(obj.__module__, DOCSTRING_MODULES)
+ ),
key=lambda pair: pair[0],
)
UNDOCUMENTED = sorted(
(
qualified
- for obj, qualified in undocumented_public_objects().values()
+ for obj, qualified in rollout_objects(REGISTRY, with_docstring=False).values()
if covered(obj.__module__, DOCSTRING_MODULES)
),
)
-
+MEMBERS = sorted(
+ (qualified, member, doc)
+ for qualified, (member, doc) in documented_members(REGISTRY, [obj for _, obj in CONVERTED]).items()
+)
SHAPE_CHECK_FILES = sorted([*KORNIA_ROOT.rglob("*.py"), *rst_pages(DOCS)])
@@ -218,17 +227,38 @@ def test_converted_object_has_see_also(qualified: str, obj: object) -> None:
assert not problems, f"{qualified} See Also: " + "; ".join(problems)
[email protected](("qualified", "obj"), CONVERTED, ids=[qualified for qualified, _ in CONVERTED])
-def test_converted_object_references_resolve(qualified: str, obj: object) -> None:
+def _unresolved_references(qualified: str, obj: object, doc: str | None, registered_as: str | None = None) -> str:
+ """Describe the roles in ``doc``, the docstring of ``obj`` shown as ``qualified``, that Sphinx cannot link.
+
+ With ``registered_as``, the roles resolve in the scopes of that registered entry only, as Sphinx resolves each
+ rendered entry; without it, in those of every name the registry gives ``obj``.
+ """
unresolved = [
f":{role}:`{target}`{_suggest(target)}{_wrong_role(role, target)}"
- for role, target in docstring_roles(own_docstring(obj))
- if not reference_resolves(obj, target, REGISTRY, INTERSPHINX, role)
+ for role, target in docstring_roles(doc)
+ if not reference_resolves(obj, target, REGISTRY, INTERSPHINX, role, registered_as)
]
+ if not unresolved:
+ return ""
hint = "write the full name the docs register"
if not is_documented(obj, REGISTRY):
hint = f"{qualified} is not documented on any docs/source page, so short names have no scope; {hint}"
- assert not unresolved, f"{qualified} has references Sphinx cannot link ({hint}): " + ", ".join(unresolved)
+ return f"{qualified} has references Sphinx cannot link ({hint}): " + ", ".join(unresolved)
+
+
[email protected](("qualified", "obj"), CONVERTED, ids=[qualified for qualified, _ in CONVERTED])
+def test_converted_object_references_resolve(qualified: str, obj: object) -> None:
+ assert not (message := _unresolved_references(qualified, obj, own_docstring(obj))), message
+
+
[email protected](("qualified", "member", "doc"), MEMBERS, ids=[qualified for qualified, _, _ in MEMBERS])
+def test_converted_class_member_references_resolve(qualified: str, member: object, doc: str) -> None:
+ """The methods, properties and attributes autodoc renders under a converted class link their references too.
+
+ Each is checked under its own registered name, so a method inherited by several documented classes has to link
+ its references on every class page that renders it.
+ """
+ assert not (message := _unresolved_references(qualified, member, doc, registered_as=qualified)), message
@pytest.mark.parametrize(
@@ -270,7 +300,7 @@ def test_catalog_names_every_export(namespace: str) -> None:
def test_rollout_progress() -> None:
"""Print, per in-scope package, the converted objects and catalogued namespaces (read it with ``-s``)."""
- objects = [obj for obj, _ in public_objects().values()]
+ objects = [obj for obj, _ in rollout_objects(REGISTRY).values()]
namespaces = modules_declaring_all()
print("\npackage converted objects catalogued namespaces")
for package in IN_SCOPE_PACKAGES:
diff --git a/tests/test_docstring_style_helpers.py b/tests/test_docstring_style_helpers.py
index d0f8fe556..fa6006204 100644
--- a/tests/test_docstring_style_helpers.py
+++ b/tests/test_docstring_style_helpers.py
@@ -31,11 +31,13 @@ from kornia.geometry.depth import DepthWarper, depth_to_3d, depth_to_3d_v2
from testing import docstring_style
from testing.docstring_style import (
+ _Attribute,
catalog_missing,
catalog_required,
catalog_unimportable,
covered,
docstring_roles,
+ documented_members,
forwards_to,
intersphinx_names,
is_deprecated_wrapper,
@@ -47,6 +49,7 @@ from testing.docstring_style import (
public_objects,
reference_resolves,
resolve_reference,
+ rollout_objects,
see_also_entries,
sphinx_registry,
status_target,
@@ -661,6 +664,139 @@ def test_public_and_undocumented_objects_split_the_export_walk(monkeypatch):
assert [qualified for _, qualified in public_objects.__wrapped__().values()] == ["m.documented"]
+class _Converted:
+ """A converted class."""
+
+ #: The size, see :class:`_Converted`.
+ size = 1
+ bare = 2
+ #: First line, see :class:`_Converted`.
+ #: Second line.
+ block = 3
+ trailing = 4 #: Trailing, see :attr:`size`.
+ literal = 5
+ """A string after the assignment,
+ on two lines."""
+
+ def method(self):
+ """Do it, like :meth:`other`."""
+
+ @property
+ def prop(self):
+ """Read it."""
+
+ def silent(self):
+ pass
+
+
+class _Unconverted:
+ """Another class."""
+
+ def method(self):
+ """Not wanted."""
+
+
+def test_documented_members_returns_the_registered_members_of_the_classes():
+ registry = {
+ "m.Converted": _Converted,
+ "m.Converted.method": _Converted.method,
+ "m.Converted.prop": _Converted.prop,
+ "m.Converted.size": _Attribute(_Converted, "size"),
+ "m.Converted.block": _Attribute(_Converted, "block"),
+ "m.Converted.trailing": _Attribute(_Converted, "trailing"),
+ "m.Converted.literal": _Attribute(_Converted, "literal"),
+ "m.Converted.silent": _Converted.silent,
+ "m.Converted.bare": _Attribute(_Converted, "bare"),
+ "m.Unconverted": _Unconverted,
+ "m.Unconverted.method": _Unconverted.method,
+ "m.function": _alias_target,
+ }
+ members = documented_members(registry, [_Converted])
+ assert members == {
+ "m.Converted.method": (_Converted.method, "Do it, like :meth:`other`."),
+ "m.Converted.prop": (_Converted.prop, "Read it."),
+ "m.Converted.size": (registry["m.Converted.size"], "The size, see :class:`_Converted`."),
+ "m.Converted.block": (
+ registry["m.Converted.block"],
+ "First line, see :class:`_Converted`.\nSecond line.",
+ ),
+ "m.Converted.trailing": (registry["m.Converted.trailing"], "Trailing, see :attr:`size`."),
+ "m.Converted.literal": (registry["m.Converted.literal"], "A string after the assignment,\non two lines."),
+ }
+ assert documented_members(registry, []) == {}
+
+
+def test_documented_members_reads_a_member_under_every_class_that_registers_it():
+ registry = {"a.Converted": _Converted, "b.Converted": _Converted}
+ registry |= {"a.Converted.prop": _Converted.prop, "b.Converted.prop": _Converted.prop}
+ assert set(documented_members(registry, [_Converted])) == {"a.Converted.prop", "b.Converted.prop"}
+
+
+class _ClassMethodBase:
+ @classmethod
+ def build(cls):
+ """Build one, see :meth:`other`."""
+
+
+class _ClassMethodOverride(_ClassMethodBase):
+ @classmethod
+ def build(cls):
+ return cls()
+
+
+def test_documented_members_reads_the_inherited_docstring_of_a_class_method_override():
+ # The registry holds a class method as its ``__func__``, on which ``inspect.getdoc`` finds no inherited docstring.
+ # autodoc renders the base's, so its references must be checked as well.
+ build = vars(_ClassMethodOverride)["build"].__func__
+ registry = {"m.Override": _ClassMethodOverride, "m.Override.build": build}
+ assert documented_members(registry, [_ClassMethodOverride]) == {
+ "m.Override.build": (build, "Build one, see :meth:`other`.")
+ }
+
+
+class _SharedBase:
+ def method(self):
+ """Run it, then :meth:`other`, as :func:`helper` does."""
+
+
+class _SharedA(_SharedBase):
+ def other(self):
+ """The other one."""
+
+
+class _SharedB(_SharedBase):
+ pass
+
+
+def _helper():
+ """Help."""
+
+
+def test_member_registered_under_several_classes_resolves_in_each_registration_only():
+ # One method rendered on two class pages, as an inherited ``forward`` is: Sphinx resolves its references once per
+ # rendered entry, under that entry's class and module, so a short name registered only for ``a.A`` links there
+ # and stays plain text on the ``b.B`` page.
+ registry = {
+ "a.A": _SharedA,
+ "a.A.method": _SharedBase.method,
+ "a.A.other": _SharedA.other,
+ "a.helper": _helper,
+ "b.B": _SharedB,
+ "b.B.method": _SharedBase.method,
+ }
+ method = _SharedBase.method
+ for target, expected in (("other", _SharedA.other), ("helper", _helper)):
+ assert resolve_reference(method, target, registry, registered_as="a.A.method") is expected, target
+ assert reference_resolves(method, target, registry, registered_as="a.A.method"), target
+ assert resolve_reference(method, target, registry, registered_as="b.B.method") is None, target
+ assert not reference_resolves(method, target, registry, registered_as="b.B.method"), target
+ # Without a registered name the scopes of every registration count, as for a top-level object.
+ assert resolve_reference(method, target, registry) is expected, target
+ assert resolve_reference(method, "a.A.other", registry, registered_as="b.B.method") is _SharedA.other
+ with pytest.raises(ValueError, match="does not register"):
+ resolve_reference(method, "other", registry, registered_as="a.A.other")
+
+
def _alias_target(a, b=1, *args, c=2, **kwargs):
return a
@@ -814,3 +950,41 @@ def test_forwards_to_rejects_a_live_function_titled_alias_of_another():
title = "Alias of :func:`~kornia.geometry.conversions.rotation_matrix_to_quaternion`."
assert status_target(title) == "kornia.geometry.conversions.rotation_matrix_to_quaternion"
assert not forwards_to(axis_angle_to_quaternion, rotation_matrix_to_quaternion)
+
+
+class _OnlyDocumented:
+ """A kornia class that a docs page registers and no ``__all__`` exports."""
+
+ def method(self):
+ """A registered method, not a top-level object."""
+
+
+def _undocumented_only():
+ pass
+
+
+def test_rollout_objects_union_exported_and_documented_objects(monkeypatch):
+ # Only kornia-defined classes count, so give the synthetic class a kornia ``__module__``.
+ monkeypatch.setattr(_OnlyDocumented, "__module__", "kornia.synthetic", raising=False)
+ monkeypatch.setattr(_OnlyDocumented.method, "__module__", "kornia.synthetic")
+ monkeypatch.setattr(_undocumented_only, "__module__", "kornia.synthetic")
+ exported = ((depth_to_3d, "kornia.geometry.depth.depth_to_3d"),)
+ monkeypatch.setattr(docstring_style, "_exported_objects", lambda: exported)
+ registry = {
+ "kornia.synthetic": docstring_style,
+ "kornia.synthetic.OnlyDocumented": _OnlyDocumented,
+ "kornia.synthetic.OnlyDocumented.method": _OnlyDocumented.method,
+ "kornia.geometry.depth.depth_to_3d": depth_to_3d,
+ "kornia.synthetic.alias_of_depth_to_3d": depth_to_3d,
+ "kornia.synthetic.not_kornia": inspect.getdoc,
+ "kornia.synthetic._undocumented_only": _undocumented_only,
+ }
+ selected = rollout_objects(registry)
+ names = sorted(qualified for _, qualified in selected.values())
+ assert names == ["kornia.geometry.depth.depth_to_3d", "kornia.synthetic._OnlyDocumented"]
+ assert id(_OnlyDocumented.method) not in selected
+ assert [q for _, q in rollout_objects(registry, with_docstring=False).values()] == [
+ "kornia.synthetic._undocumented_only"
+ ]
+ # One object registered under two names is one entry of the documented walk.
+ assert [q for _, q in docstring_style.documented_objects(registry)].count("kornia.geometry.depth.depth_to_3d") == 1