Skip to content

🐍 Geometric Computer Vision Library for Spatial AI

feat(geometry): affine fundamental matrix in find_fundamental and RANSAC (#6569)

Dmytro Mishkin2cdf08e96868
diff --git a/changelog.d/6569.added.md b/changelog.d/6569.added.md
new file mode 100644
index 000000000..2e3b3b1df
--- /dev/null
+++ b/changelog.d/6569.added.md
@@ -0,0 +1 @@
+`find_fundamental(..., method="AFFINE")` estimates the affine fundamental matrix of two affine cameras (weak-perspective or other parallel-projection views, such as aerial images taken with a long focal length) from four or more correspondences, by the Gold Standard orthogonal regression of Hartley and Zisserman's algorithm 14.1, and `RANSAC("fundamental_affine")` estimates it robustly from four-point samples, with the exact geometric distance as its residual.
diff --git a/docs/source/geometry.epipolar.rst b/docs/source/geometry.epipolar.rst
index 0cb26334f..997a40be9 100644
--- a/docs/source/geometry.epipolar.rst
+++ b/docs/source/geometry.epipolar.rst
@@ -42,12 +42,12 @@ two known poses (:func:`relative_camera_motion`).
Fundamental
-----------
-Estimate the fundamental matrix from pixel correspondences with the 8-point or 7-point algorithm
-(:func:`find_fundamental`), convert from an essential matrix or two projection matrices
-(:func:`fundamental_from_essential`, :func:`fundamental_from_projections`), compute epipolar lines and the nearest
-points on them (:func:`compute_correspond_epilines`, :func:`get_closest_point_on_epipolar_line`,
-:func:`get_perpendicular`), and normalize points and matrices for the estimators (:func:`normalize_points`,
-:func:`normalize_transformation`).
+Estimate the fundamental matrix from pixel correspondences with the 8-point or 7-point algorithm, or the affine
+fundamental matrix of two affine cameras from four or more (:func:`find_fundamental`), convert from an essential
+matrix or two projection matrices (:func:`fundamental_from_essential`, :func:`fundamental_from_projections`),
+compute epipolar lines and the nearest points on them (:func:`compute_correspond_epilines`,
+:func:`get_closest_point_on_epipolar_line`, :func:`get_perpendicular`), and normalize points and matrices for the
+estimators (:func:`normalize_points`, :func:`normalize_transformation`).
.. autofunction:: find_fundamental
.. autofunction:: fundamental_from_essential
diff --git a/docs/source/geometry.ransac.rst b/docs/source/geometry.ransac.rst
index 00c53e710..decb26c80 100644
--- a/docs/source/geometry.ransac.rst
+++ b/docs/source/geometry.ransac.rst
@@ -48,6 +48,52 @@ AdaLAM shares the linear fitting primitive but retains its specialized loop:
its local 2x2 models are constrained by a matched seed, and its variable-size
neighborhoods, sampling and confidence-based inlier selection are unchanged.
+Affine fundamental matrices
+---------------------------
+
+``model_type="fundamental_affine"`` estimates the fundamental matrix of two affine
+cameras, ``[[0, 0, a], [0, 0, b], [c, d, e]]``: views in parallel projection,
+such as weak-perspective views of a distant scene (aerial or satellite images
+taken with a long focal length) or images through a telecentric lens. Its epipolar
+lines are parallel in each image, and it has four degrees of freedom against seven
+for a general fundamental matrix. Each four-point sample is solved, and each new
+best model refitted on its inliers, by
+:func:`~kornia.geometry.epipolar.find_fundamental` with ``method="AFFINE"``:
+orthogonal regression on the stacked correspondences ``(x2, y2, x1, y1)``, the
+Gold Standard estimate of Hartley and Zisserman's algorithm 14.1. A
+correspondence's Sampson distance is then exact: the smallest total displacement
+of its two points that makes them correspond, the distance ``inl_th`` bounds.
+Like any fundamental matrix, it is not determined by correspondences on one plane,
+which a single 2D affine map relates (``model_type="affine"`` tests for that).
+A matrix with ``(a, b)`` or ``(c, d)`` near zero ignores one image: it is a line in
+the other, which every mismatch along that line fits, so a line of mismatches
+(along an edge, say) could outvote the true geometry. Such a model gets no
+inliers: each image's term of the distance, ``(a x2 + b y2)`` and
+``(c x1 + d y1)`` over ``|(a, b, c, d)|``, must vary over the keypoints by at
+least two inlier thresholds (a standard deviation). In 1000 px images with 30
+true correspondences, 40 mismatches within 0.3 px of a line and 30 random ones
+(ten scenes, the line in either image, ``inl_th`` of 1 or 2), it keeps 27 to 30
+of the true ones and 0 to 2 of the line; without the check, 0 to 1 and all 40.
+A true geometry whose keypoints spread less than that across its epipolar lines
+is rejected too, and RANSAC then returns zeros. With 200 correspondences, 0.5 px
+noise and ``inl_th=2.0``, that happens to a 15 px patch and to a second view at
+1/100 of the first one's scale, but not to a 30 px patch or a scale of 1/20.
+It suits views that are close to parallel projection. On synthetic pinhole views of
+a 60 m scene from 45 degrees off nadir (1000 px images, 0.5 px noise,
+``inl_th=2.0``), it kept 99% of the true correspondences from 3000 m away,
+89% from 1500 m and 49% from 500 m, where ``"fundamental"`` kept all of them;
+on exactly affine views both kept all of them, ``"fundamental_affine"`` with a more
+accurate matrix, several times faster.
+It uses ``local_optimization="dlt"``; ``"lm"``, ``compile=True`` and DEGENSAC
+are not supported.
+
+.. code-block:: python
+
+ from kornia.geometry import RANSAC
+
+ estimator = RANSAC("fundamental_affine", inl_th=2.0, seed=0)
+ F, inliers = estimator(points1, points2)
+
Batches, scores, and refinement
-------------------------------
@@ -102,8 +148,8 @@ endpoints to the target segment's line; zero-length target segments are outliers
Their local optimization re-weights segments by the same distance.
A model is accepted only with more inliers than its minimal sample (three
-correspondences for affine transforms, four for homographies, five for essential and seven or eight for
-fundamental matrices): a model fitted to a sample always fits that sample, so
+correspondences for affine transforms, four for homographies and affine fundamental matrices, five for essential
+and seven or eight for fundamental matrices): a model fitted to a sample always fits that sample, so
only further inliers show a consensus. Without one, the estimator returns the
all-zero matrix and an empty mask.
diff --git a/kornia/geometry/epipolar/__init__.py b/kornia/geometry/epipolar/__init__.py
index ecf4ba603..31344837d 100644
--- a/kornia/geometry/epipolar/__init__.py
+++ b/kornia/geometry/epipolar/__init__.py
@@ -27,7 +27,8 @@ motion. :ref:`Two-view geometry <two-view-conventions>` compares these conventio
Which function to use:
-- A model from correspondences known to be inliers: :func:`find_fundamental` from pixels (8 or 7 points),
+- A model from correspondences known to be inliers: :func:`find_fundamental` from pixels (8 or 7 points, or the
+ affine fundamental matrix of two affine cameras from 4 or more),
:func:`find_essential` from normalized camera coordinates (Nister's 5 points, up to ten candidates). For a plane
or a pure rotation, :func:`~kornia.geometry.homography.find_homography_dlt`; with outliers,
:class:`~kornia.geometry.ransac.RANSAC`.
diff --git a/kornia/geometry/epipolar/fundamental.py b/kornia/geometry/epipolar/fundamental.py
index 396ab0f12..da42fcc1e 100644
--- a/kornia/geometry/epipolar/fundamental.py
+++ b/kornia/geometry/epipolar/fundamental.py
@@ -24,7 +24,14 @@ import torch
from kornia.core.check import KORNIA_CHECK_SAME_SHAPE, KORNIA_CHECK_SHAPE
from kornia.core.exceptions import ShapeError
-from kornia.core.utils import _autocast_disabled, _torch_svd_cast, safe_inverse_with_mask
+from kornia.core.utils import (
+ _autocast_disabled,
+ _svd_work_dtype,
+ _symmetric_eigenvector,
+ _torch_svd_cast,
+ _torch_svd_in_work_dtype,
+ safe_inverse_with_mask,
+)
from kornia.geometry.conversions import convert_points_from_homogeneous, convert_points_to_homogeneous
from kornia.geometry.solvers.homogeneous import _det3, _null_space_lu
from kornia.geometry.solvers.polynomial_solver import _solve_cubic_real
@@ -623,19 +630,112 @@ def run_8point(
return normalize_transformation(F)
+def run_affine(points1: torch.Tensor, points2: torch.Tensor, weights: Optional[torch.Tensor] = None) -> torch.Tensor:
+ r"""Compute the affine fundamental matrix of two affine cameras by orthogonal regression.
+
+ Two affine cameras (parallel projection, such as weak perspective) have a fundamental matrix of the form
+ ``[[0, 0, a], [0, 0, b], [c, d, e]]``: the epipolar constraint :math:`a x_2 + b y_2 + c x_1 + d y_1 + e = 0` is a
+ hyperplane in the space of the stacked correspondences :math:`(x_2, y_2, x_1, y_1)`, and the distance of a
+ correspondence to it is the geometric error, the smallest total displacement of its two points that makes them
+ correspond. The hyperplane that minimizes the sum of the squared (weighted) distances passes through the
+ centroid, and its normal is the eigenvector of the smallest eigenvalue of the centred scatter matrix. This is
+ the Gold Standard (maximum-likelihood) estimate under Gaussian noise, exact for four correspondences. When the
+ points of one image lie on a line, the minimizer is that line, a rank-one matrix with ``(a, b) = 0`` or
+ ``(c, d) = 0``, which no pair of affine cameras has; ``RANSAC("fundamental_affine")`` rejects such models.
+
+ Reference: Hartley and Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., section 14.2, algorithm 14.1.
+
+ Args:
+ points1: points in the first image ``(B, N, 2)``, :math:`N \ge 4`.
+ points2: points in the second image ``(B, N, 2)``.
+ weights: optional nonnegative weights ``(B, N)`` of the correspondences' squared distances.
+
+ Returns:
+ The affine fundamental matrices ``(B, 3, 3)``, scaled so that ``F[2, 2] = 1``.
+
+ """
+ KORNIA_CHECK_SHAPE(points1, ["B", "N", "2"])
+ KORNIA_CHECK_SHAPE(points2, ["B", "N", "2"])
+ KORNIA_CHECK_SAME_SHAPE(points1, points2)
+ if points1.shape[1] < 4:
+ raise ShapeError(
+ f"points1 and points2 must have shape (B, N, 2) with N >= 4, got {list(points1.shape)}.",
+ actual_shape=list(points1.shape),
+ expected_shape=["B", "N >= 4", "2"],
+ )
+ if weights is not None and weights.shape != points1.shape[:2]:
+ raise ShapeError(
+ f"weights must have shape (B, N) = {list(points1.shape[:2])}, got {list(weights.shape)}.",
+ actual_shape=list(weights.shape),
+ expected_shape=["B", "N"],
+ )
+ dtype = points1.dtype
+ # Pixel coordinates in the hundreds overflow half precision, and the SVD has no half kernel. float32 points are
+ # fitted in float64 (float32 on MPS), so that far from the origin, as in a large image, the fit adds no rounding
+ # of its own to theirs.
+ work = _svd_work_dtype(points1)
+ with _autocast_disabled(points1.device.type):
+ X = torch.cat([points2, points1], dim=-1).to(work)
+ if weights is None:
+ w = torch.ones(X.shape[:2], device=X.device, dtype=work)
+ else:
+ # A negative weight counts as zero; ``where`` keeps the derivative at a zero weight one-sided (#4229).
+ w = weights.to(work)
+ w = torch.where(w < 0, 0.0, w)
+ # A batch element with a non-finite point or weight, whose sum is then not finite, has no matrix. It is fitted
+ # as zeros and its F set to NaN, so the other elements keep theirs and every gradient stays finite, as in
+ # _svd_null_vector (#4770).
+ finite = (X.sum((-2, -1)) + w.sum(-1)).isfinite()
+ X = torch.where(finite.view(-1, 1, 1), X, 0.0)
+ w = torch.where(finite.view(-1, 1), w, 1.0)
+ mean = (w[..., None] * X).sum(1) / w.sum(1, keepdim=True).clamp_min(torch.finfo(work).tiny)
+ centred = X - mean[:, None]
+ # Linear in w, so a zero weight has a finite gradient, unlike row scaling by sqrt(w) (#4912).
+ scatter = (centred * w[..., None]).transpose(-2, -1) @ centred
+ # The derivatives come from the scatter: _symmetric_eigenvector divides only by the gaps to the smallest
+ # eigenvalue, so they stay finite when two other eigenvalues are equal, as for a grid seen from the front and
+ # the side (#6214).
+ if work == torch.float64:
+ eigenvalues, eigenvectors = torch.linalg.eigh(scatter.detach())
+ normal = _symmetric_eigenvector(scatter, eigenvalues, eigenvectors, 0)
+ else:
+ # The scatter squares the condition number: in float32 (on MPS, or for a half input), its eigh mixes the two
+ # smallest directions of a sample near a line and a plane, and torch 2.5's MPS has no eigh. The right
+ # singular vectors of the weighted centred points keep them apart; their squared singular values are the
+ # scatter's eigenvalues. In float64, eigh resolves them and is several times faster than an SVD on CUDA.
+ _, S, V = _torch_svd_in_work_dtype((centred * w[..., None].sqrt()).detach(), full_matrices=False)
+ normal = _symmetric_eigenvector(scatter, S.to(work).square(), V.to(work), -1)
+ zeros = torch.zeros_like(normal[:, 0])
+ last = -(normal * mean).sum(-1)
+ F = torch.stack(
+ [zeros, zeros, normal[:, 0], zeros, zeros, normal[:, 1], normal[:, 2], normal[:, 3], last], dim=-1
+ ).view(-1, 3, 3)
+ F = normalize_transformation(F)
+ F = torch.where(finite.view(-1, 1, 1), F, float("nan"))
+ if dtype == torch.float16:
+ # Points centred on the origin put e near 0, and F[2, 2] = 1 then puts the other entries above float16's
+ # 65504: such an F is scaled to its largest entry 1 instead.
+ scale = F.abs().amax(dim=(-2, -1), keepdim=True)
+ F = torch.where(scale > torch.finfo(torch.float16).max, F / scale, F)
+ return F.to(dtype)
+
+
def find_fundamental(
points1: torch.Tensor,
points2: torch.Tensor,
weights: Optional[torch.Tensor] = None,
- method: Literal["8POINT", "7POINT"] = "8POINT",
+ method: Literal["8POINT", "7POINT", "AFFINE"] = "8POINT",
) -> torch.Tensor:
- r"""Estimate the fundamental matrix from pixel correspondences with the 8- or 7-point algorithm.
+ r"""Estimate the fundamental matrix from pixel correspondences with the 8-, 7-point or affine algorithm.
``F = find_fundamental(points1, points2)`` takes correspondences ``(B, N, 2)``, ``N >= 8``, of the first and the
second image, in pixels or any other image coordinates, and returns the rank-2 matrix ``(B, 3, 3)`` with
:math:`x_2^\top F x_1 = 0` (the second image's point on the left) of the Hartley-normalized, optionally weighted,
8-point least-squares fit. ``method="7POINT"`` takes exactly seven correspondences and returns the three
- candidates ``(B, 3, 3, 3)`` of the 7-point algorithm instead. For calibrated cameras and normalized camera
+ candidates ``(B, 3, 3, 3)`` of the 7-point algorithm instead. ``method="AFFINE"`` takes ``N >= 4``
+ correspondences of two affine cameras (parallel projection, such as weak-perspective views of a distant scene)
+ and returns their affine fundamental matrix ``[[0, 0, a], [0, 0, b], [c, d, e]]``, the Gold Standard
+ (maximum-likelihood) fit of Hartley and Zisserman's algorithm 14.1. For calibrated cameras and normalized camera
coordinates, use :func:`find_essential`; for a planar scene or a camera that only rotates, where ``F`` is not
determined, :func:`~kornia.geometry.homography.find_homography_dlt`; with outliers,
:class:`~kornia.geometry.ransac.RANSAC`.
@@ -645,20 +745,24 @@ def find_fundamental(
- The result is scaled so that ``F[2, 2] = 1`` by :func:`normalize_transformation`, which leaves it at its
unnormalised scale when ``F[2, 2]`` is numerically zero, as for exactly rectified stereo.
``method="7POINT"`` returns three candidates in no particular order.
- - ``weights`` weight each correspondence's equation in the linear system: only their ratios matter, a
- negative weight counts as zero, and ``method="7POINT"`` ignores them.
+ - ``weights`` weight each correspondence's equation in the linear system (for ``"AFFINE"``, its squared
+ distance): only their ratios matter, a negative weight counts as zero, and ``method="7POINT"`` ignores them.
+ - ``"AFFINE"`` fits and returns the affine form exactly; it computes in float64 (float32 on MPS and for
+ half-precision input). A float16 result whose entries ``F[2, 2] = 1`` would put above float16's maximum
+ (``e`` near 0, as for points centred on the origin) is scaled to its largest entry 1 instead. A batch
+ element with a non-finite point or weight gets a NaN matrix, and the others are unaffected.
- When the 7-point cubic has one real root, the two extra ``"7POINT"`` candidates are zero matrices.
- A caller's autocast region does not change the fit.
Args:
points1: A set of points in the first image with a tensor shape ``(B, N, 2)``: :math:`N \ge 8` for
- ``"8POINT"``, exactly 7 for ``"7POINT"``.
+ ``"8POINT"``, exactly 7 for ``"7POINT"``, :math:`N \ge 4` for ``"AFFINE"``.
points2: A set of points in the second image with the shape of ``points1``.
weights: Tensor containing the weights per point correspondence with a shape of ``(B, N)``.
- method: The method to use for computing the fundamental matrix. Supported methods are "7POINT" and "8POINT".
+ method: The method to use for computing the fundamental matrix: "8POINT", "7POINT" or "AFFINE".
Returns:
- the computed fundamental matrix with shape ``(B, 3, 3)`` for ``"8POINT"`` and
+ the computed fundamental matrix with shape ``(B, 3, 3)`` for ``"8POINT"`` and ``"AFFINE"``, and
``(B, 3, 3, 3)`` for ``"7POINT"``.
Raises:
@@ -677,8 +781,10 @@ def find_fundamental(
result = run_7point(points1, points2)
elif method.upper() == "8POINT":
result = run_8point(points1, points2, weights)
+ elif method.upper() == "AFFINE":
+ result = run_affine(points1, points2, weights)
else:
- raise ValueError(f"Invalid method: {method}. Supported methods are '7POINT' and '8POINT'.")
+ raise ValueError(f"Invalid method: {method}. Supported methods are '7POINT', '8POINT' and 'AFFINE'.")
return result
diff --git a/kornia/geometry/ransac.py b/kornia/geometry/ransac.py
index 0508f99ca..f7accbe9e 100644
--- a/kornia/geometry/ransac.py
+++ b/kornia/geometry/ransac.py
@@ -19,15 +19,16 @@ r"""Estimate geometric models robustly from correspondences that contain outlier
:class:`RANSAC` fits one model to one set of ``(N, 2)`` correspondences (``(N, 2, 2)`` segments for
``"homography_from_linesegments"``), with the first image's points first: a homography or an affine map from the first
-image to the second, a fundamental matrix with :math:`x_2^\top F x_1 = 0` from pixels, or an essential matrix from
-normalized camera coordinates. Its threshold ``inl_th`` is a distance in the units of the correspondences.
+image to the second, a fundamental matrix with :math:`x_2^\top F x_1 = 0` from pixels (a general one, or the affine
+fundamental matrix of two affine cameras), or an essential matrix from normalized camera coordinates. Its threshold
+``inl_th`` is a distance in the units of the correspondences.
:ref:`Two-view geometry <two-view-conventions>` compares this with OpenCV.
Which class to use:
- One model from correspondences with outliers: :class:`RANSAC`, with ``model_type`` one of ``"homography"``,
``"homography_from_linesegments"``, ``"affine"``, ``"fundamental"`` (seven-point samples),
- ``"fundamental_7pt"``, ``"fundamental_8pt"`` or ``"essential"``.
+ ``"fundamental_7pt"``, ``"fundamental_8pt"``, ``"fundamental_affine"`` or ``"essential"``.
- A model from correspondences known to be inliers, without sampling: the estimators it wraps,
:func:`~kornia.geometry.homography.find_homography_dlt`, :func:`~kornia.geometry.solvers.find_affine`,
:func:`~kornia.geometry.epipolar.find_fundamental` and :func:`~kornia.geometry.epipolar.find_essential`.
@@ -65,6 +66,7 @@ from kornia.geometry.epipolar.fundamental import (
_seven_point_candidates,
normalize_points,
normalize_transformation,
+ run_affine,
)
from kornia.geometry.homography import (
_four_point_homography,
@@ -97,6 +99,43 @@ _DEGENSAC_MODELS = ("fundamental", "fundamental_7pt")
# tuned batch sizes; large explicit batches are split without changing the minimal-sample budget.
_COMPILED_CPU_SCORE_RESIDUALS = 1 << 22
+# An affine fundamental matrix must tie the two images together: as a correspondence's point in one image moves over
+# the keypoints, the geometric distance must vary by at least this many inlier thresholds (a standard deviation). The
+# tests bound it on both sides: at 1, twenty mismatches along a line outvote twelve true correspondences, and at 4, a
+# true geometry whose keypoints spread 3.7 thresholds across its epipolar lines is rejected.
+_AFFINE_FUNDAMENTAL_MIN_SPREAD = 2.0
+
+
+def _affine_fundamental_ties_both_images(
+ kp1: torch.Tensor, kp2: torch.Tensor, models: torch.Tensor, inl_th: float
+) -> torch.Tensor:
+ """Return which affine fundamental matrices ``(B, 3, 3)`` depend on both images' points, ``(B,)``.
+
+ ``F = [[0, 0, a], [0, 0, b], [c, d, e]]`` puts a correspondence at the geometric distance
+ ``(a x2 + b y2 + c x1 + d y1 + e) / |(a, b, c, d)|``. With ``(c, d) = 0`` that is the distance of ``x2`` to one
+ line, whatever ``x1`` is, so mismatches whose second points lie along a line all fit it and can outvote the true
+ geometry; with ``(a, b) = 0`` the same holds in the first image. Points near a line give small terms rather than
+ zero ones, and what counts as small depends on the images' extent and the threshold. So each image's term,
+ ``(c x1 + d y1)`` over ``kp1`` and ``(a x2 + b y2)`` over ``kp2``, divided by ``|(a, b, c, d)|``, must have a
+ standard deviation of at least ``_AFFINE_FUNDAMENTAL_MIN_SPREAD`` times the inlier threshold ``inl_th`` (not
+ squared). For the true geometry the first image's term is the spread of the matched points across the epipolar
+ lines, seen in the second image. Correspondences with a non-finite point, which no model fits, are left out.
+ """
+ with _autocast_disabled(kp1.device.type):
+ work = torch.promote_types(torch.promote_types(kp1.dtype, models.dtype), torch.float32)
+ normal = models[:, [0, 1, 2, 2], [2, 2, 0, 1]].to(work) # (a, b, c, d)
+ # The correspondences as (x2, y2, x1, y1), the order of (a, b, c, d); a row whose sum is not finite is left out.
+ points = torch.cat([kp2.reshape(-1, 2), kp1.reshape(-1, 2)], -1).to(work)
+ finite = points.sum(-1, keepdim=True).isfinite()
+ count = finite.sum().clamp_min(1)
+ points = torch.where(finite, points, 0.0)
+ centred = torch.where(finite, points - points.sum(0) / count, 0.0)
+ covariance = centred.mT @ centred / count
+ # Each image's term has the variance n^T C n over its own 2 x 2 block of the covariance; compared squared.
+ floor = (_AFFINE_FUNDAMENTAL_MIN_SPREAD * inl_th) ** 2 * normal.square().sum(-1)
+ n2, n1 = normal[:, :2], normal[:, 2:]
+ return (((n2 @ covariance[:2, :2]) * n2).sum(-1) >= floor) & (((n1 @ covariance[2:, 2:]) * n1).sum(-1) >= floor)
+
def _affine_transfer_error(pts1: torch.Tensor, pts2: torch.Tensor, models: torch.Tensor) -> torch.Tensor:
"""Squared one-way transfer errors ``(B, N)`` of affine models ``(B, 3, 3)`` on one set of correspondences.
@@ -203,7 +242,8 @@ class RANSAC(nn.Module):
when its error under the model is at most ``inl_th``, in the units of the keypoints. The model follows the
estimator of its type: a homography or affine map from ``kp1`` to ``kp2``
(:func:`~kornia.geometry.homography.find_homography_dlt`, :func:`~kornia.geometry.solvers.find_affine`), a
- fundamental matrix with :math:`x_2^\top F x_1 = 0` (:func:`~kornia.geometry.epipolar.find_fundamental`), or an
+ fundamental matrix with :math:`x_2^\top F x_1 = 0` (:func:`~kornia.geometry.epipolar.find_fundamental`; for
+ ``"fundamental_affine"`` its ``method="AFFINE"``, the matrix of two affine cameras), or an
essential matrix for camera-normalized keypoints (:func:`~kornia.geometry.epipolar.find_essential`). Each call
estimates one model, since ``batch_size`` batches hypotheses rather than problems; for correspondences known to be
inliers, call those estimators directly (see also https://en.wikipedia.org/wiki/Random_sample_consensus).
@@ -211,16 +251,17 @@ class RANSAC(nn.Module):
Convention:
- ``kp1`` and ``kp2`` are passed as the first- and second-image arguments of the estimator selected by
``model_type``. ``forward`` returns an all-zero model and no inliers when no candidate has more inliers
- than its minimal sample (three correspondences for affine transforms, four for homographies,
- five for ``"essential"``, seven for ``"fundamental"`` and ``"fundamental_7pt"``,
+ than its minimal sample (three correspondences for affine transforms, four for homographies and
+ ``"fundamental_affine"``, five for ``"essential"``, seven for ``"fundamental"`` and ``"fundamental_7pt"``,
eight for ``"fundamental_8pt"``).
- ``"fundamental"`` draws seven-point samples, like ``"fundamental_7pt"``; ``"fundamental_8pt"`` draws
eight-point ones.
- ``inl_th`` is in the keypoints' own units (pixels, or calibrated units for ``"essential"``): a
correspondence is an inlier when its one-way transfer error for ``"homography"`` or ``"affine"``,
- its Sampson distance for the fundamental models and ``"essential"``, or the mean distance
- of its transferred endpoints from the image-2 segment's line for ``"homography_from_linesegments"``
- is at most ``inl_th``.
+ its Sampson distance for the fundamental models and ``"essential"`` (for ``"fundamental_affine"`` the
+ exact geometric distance: the smallest total displacement of the two points that makes them correspond),
+ or the mean distance of its transferred endpoints from the image-2 segment's line for
+ ``"homography_from_linesegments"`` is at most ``inl_th``.
:ref:`two-view-conventions` compares this with OpenCV.
- ``score_type="msac"`` (the default) ranks candidates by ``sum(1 - min(e / inl_th**2, 1))`` over the
squared errors ``e``; acceptance and early stopping count inliers for either score.
@@ -236,12 +277,18 @@ class RANSAC(nn.Module):
fundamental matrices are estimated on Hartley-normalized correspondences; essential matrices in the
caller's calibrated coordinates, which a normalization would take off the essential manifold, refined on
that manifold and returned with unit Frobenius norm.
- - ``local_optimization="dlt"``, the only choice for ``"affine"`` and line segments, refits each new best
- model from its inliers: with the default ``lo_sample_size=32``, ``max_lo_iters`` randomized refits on
- 32-inlier subsets followed by one full-inlier refit; ``lo_sample_size=None`` refits all inliers
- iteratively. Affine models use a linear least-squares refit with free translation and return
- a homogeneous matrix with bottom row ``[0, 0, 1]``; their minimal samples need three
- non-collinear source points.
+ - ``local_optimization="dlt"``, the only choice for ``"affine"``, ``"fundamental_affine"`` and line
+ segments, refits each new best model from its inliers: with the default ``lo_sample_size=32``,
+ ``max_lo_iters`` randomized refits on 32-inlier subsets followed by one full-inlier refit;
+ ``lo_sample_size=None`` refits all inliers iteratively. Affine models use a linear least-squares refit
+ with free translation and return a homogeneous matrix with bottom row ``[0, 0, 1]``; their minimal
+ samples need three non-collinear source points. ``"fundamental_affine"`` solves its four-point samples
+ and refits its inliers by orthogonal regression, the Gold Standard estimate (Hartley and Zisserman,
+ algorithm 14.1). It counts no inliers for a matrix that (nearly) ignores one image, ``(a, b)`` or
+ ``(c, d)`` near 0: that matrix is a line in the other image, which mismatches along a line fit. Each
+ image's term of the distance must vary over the keypoints by at least two inlier thresholds (a standard
+ deviation). A true geometry whose keypoints spread less across its epipolar lines, such as a 15 px patch
+ at ``inl_th=2``, is rejected too, and RANSAC returns zeros.
- ``degensac=True``, an opt-in for ``"fundamental"`` and ``"fundamental_7pt"`` with
``local_optimization="lm"``, runs DEGENSAC (Chum, Werner and Matas, CVPR 2005). Every seven-point model that
sets a new record among the raw scores is tested for an H-degenerate sample: five or more of its seven
@@ -277,8 +324,8 @@ class RANSAC(nn.Module):
- A caller's autocast region does not change the estimate.
Args:
- model_type: "affine", "homography", "fundamental", "fundamental_7pt", "fundamental_8pt", "essential", or
- "homography_from_linesegments".
+ model_type: "affine", "homography", "fundamental", "fundamental_7pt", "fundamental_8pt",
+ "fundamental_affine", "essential", or "homography_from_linesegments".
inl_th: positive inlier threshold, in the units given above.
batch_size: hypotheses generated and verified at once, or ``"auto"`` to pick the batches per call from
the device and the number of correspondences (see :meth:`resolve_batch_size`).
@@ -343,7 +390,7 @@ class RANSAC(nn.Module):
Args:
model_type: type of model to estimate: "affine", "homography", "fundamental", "fundamental_7pt",
- "fundamental_8pt", "essential", "homography_from_linesegments".
+ "fundamental_8pt", "fundamental_affine", "essential", "homography_from_linesegments".
inl_th: inlier threshold; the class docstring gives its unit per ``model_type``.
batch_size: number of generated samples at once, or ``"auto"`` (see :meth:`resolve_batch_size`).
max_iter: maximum batches to generate. At most ``batch_size * max_iter`` minimal samples are drawn
@@ -378,6 +425,7 @@ class RANSAC(nn.Module):
"fundamental",
"fundamental_7pt",
"fundamental_8pt",
+ "fundamental_affine",
"homography_from_linesegments",
"essential",
]
@@ -472,6 +520,12 @@ class RANSAC(nn.Module):
self.minimal_sample_size = 7
self.polisher_solver = find_fundamental
self.polisher_sample_size = 8
+ elif model_type == "fundamental_affine":
+ self.error_fn = sampson_epipolar_distance
+ self.minimal_solver = run_affine
+ self.minimal_sample_size = 4
+ self.polisher_solver = run_affine
+ self.polisher_sample_size = 4
elif model_type == "essential":
self.error_fn = sampson_epipolar_distance
self.minimal_solver = find_essential
@@ -775,6 +829,9 @@ class RANSAC(nn.Module):
inf = float("inf")
errors = errors.nan_to_num(nan=inf, posinf=inf, neginf=inf)
inl_mask = errors <= inl_th
+ if self.model_type == "fundamental_affine":
+ # A matrix that (nearly) ignores one image is a line in the other, which a line of mismatches fits.
+ inl_mask = inl_mask & _affine_fundamental_ties_both_images(kp1, kp2, models, math.sqrt(inl_th))[:, None]
score_ransac = inl_mask.sum(dim=1)
if self.score_type == "msac":
# Equivalent to minimizing the truncated squared loss (MSAC), normalized
diff --git a/testing/two_view.py b/testing/two_view.py
index c6f1eac38..04ee2298a 100644
--- a/testing/two_view.py
+++ b/testing/two_view.py
@@ -92,3 +92,88 @@ def calibrated_two_view_scene(
)
E = tx @ R
return {"x1": x1, "x2": x2, "E": E / E.norm(), "R": R, "t": t}
+
+
+def affine_two_view_scene(device: torch.device, dtype: torch.dtype) -> Dict[str, torch.Tensor]:
+ """Two affine cameras and twelve correspondences, built in ``dtype`` from literals.
+
+ The cameras ``x = M X + t`` (``M1 = [[1.2, 0.1, -0.3], [0.05, -1.1, 0.4]]``, ``t1 = [320, 240]``,
+ ``M2 = [[0.9, 0.5, 0.2], [-0.3, -0.8, 0.7]]``, ``t2 = [300, 260]``) see twelve non-coplanar points spread over
+ about 190 units. ``x1``/``x2`` are their exact pixel projections. ``F`` is the affine fundamental matrix of the
+ cameras, with ``[a, b] M2 + [c, d] M1 = 0`` and ``e = -([a, b] t2 + [c, d] t1)`` for
+ ``F = [[0, 0, a], [0, 0, b], [c, d, e]]``, scaled so that ``F[2, 2] = 1``. ``noise`` holds fixed offsets
+ ``[dx1, dy1, dx2, dy2]`` and ``F_gold`` the minimizer of the summed squared geometric distances of the noisy
+ correspondences, found by a general-purpose optimizer (``scipy.optimize.minimize`` on
+ ``sum(r**2) / (a**2 + b**2 + c**2 + d**2)`` with ``r = a x2 + b y2 + c x1 + d y1 + e``). Generated by::
+
+ X = np.array([[-80, 35, 12], [64, -71, 40], [-23, 90, -55], [47, 18, 73], [-95, -42, -8], [12, 58, 27],
+ [81, -9, -66], [-56, -77, 51], [30, 44, -90], [-7, -60, 15], [93, 71, 5], [-38, 6, 84]], float)
+ x1, x2 = X @ M1.T + t1, X @ M2.T + t2
+ n = np.linalg.svd(np.c_[M2.T, M1.T])[2][-1] # [a, b, c, d]
+ F = np.zeros((3, 3))
+ F[0, 2], F[1, 2], F[2, 0], F[2, 1], F[2, 2] = *n, -(n[:2] @ t2 + n[2:] @ t1)
+ F /= F[2, 2]
+ x1n, x2n = x1 + noise[:, :2], x2 + noise[:, 2:]
+ cost = lambda p: np.sum((p[0] * x2n[:, 0] + p[1] * x2n[:, 1] + p[2] * x1n[:, 0] + p[3] * x1n[:, 1] + p[4])
+ ** 2) / (p[:4] ** 2).sum()
+ p = minimize(cost, np.array([F[0, 2], F[1, 2], F[2, 0], F[2, 1], 1.0]) * 1.03, method="Nelder-Mead",
+ options=dict(xatol=1e-14, fatol=1e-16, maxiter=200000, maxfev=200000)).x
+ p = minimize(cost, p, method="BFGS", options=dict(gtol=1e-14)).x
+ F_gold = np.zeros((3, 3))
+ F_gold[0, 2], F_gold[1, 2], F_gold[2, 0], F_gold[2, 1], F_gold[2, 2] = p / p[4]
+ """
+ x1 = torch.tensor(
+ [
+ [
+ [223.9, 202.3], [377.7, 337.3], [317.9, 117.85], [356.3, 251.75], [204.2, 278.25], [332.1, 187.6],
+ [436.1, 227.55], [229.8, 342.3], [387.4, 157.1], [301.1, 311.65], [437.2, 168.55], [249.8, 265.1],
+ ]
+ ],
+ device=device,
+ dtype=dtype,
+ ) # fmt: skip
+ x2 = torch.tensor(
+ [
+ [
+ [247.9, 264.4], [330.1, 325.6], [313.3, 156.4], [365.9, 282.6], [191.9, 316.5], [345.2, 228.9],
+ [355.2, 196.7], [221.3, 374.1], [331.0, 152.8], [266.7, 320.6], [420.2, 178.8], [285.6, 325.4],
+ ]
+ ],
+ device=device,
+ dtype=dtype,
+ ) # fmt: skip
+ noise = torch.tensor(
+ [
+ [
+ [0.31, -0.12, -0.25, 0.4], [-0.44, 0.07, 0.18, -0.29], [0.05, 0.36, -0.41, 0.12],
+ [0.22, -0.33, 0.09, 0.27], [-0.15, 0.24, 0.38, -0.06], [0.42, 0.11, -0.2, -0.37],
+ [-0.3, -0.27, 0.14, 0.33], [0.08, 0.45, -0.1, -0.21], [-0.36, -0.04, 0.29, 0.16],
+ [0.19, -0.4, -0.32, 0.03], [-0.07, 0.2, 0.44, -0.39], [0.35, -0.18, -0.03, 0.25],
+ ]
+ ],
+ device=device,
+ dtype=dtype,
+ ) # fmt: skip
+ F = torch.tensor(
+ [
+ [
+ [0.0, 0.0, 0.005968045112781947],
+ [0.0, 0.0, -0.010385338345864652],
+ [-0.007471804511278188, 0.009586466165413523, 1.0],
+ ]
+ ],
+ device=device,
+ dtype=dtype,
+ )
+ F_gold = torch.tensor(
+ [
+ [
+ [0.0, 0.0, 0.0059939486403256526],
+ [0.0, 0.0, -0.010334260027148275],
+ [-0.007497428822287553, 0.009532615627438538, 1.0],
+ ]
+ ],
+ device=device,
+ dtype=dtype,
+ )
+ return {"x1": x1, "x2": x2, "noise": noise, "F": F, "F_gold": F_gold}
diff --git a/tests/geometry/epipolar/test_fundamental.py b/tests/geometry/epipolar/test_fundamental.py
index d5d3dd4c7..ace7ba7f7 100644
--- a/tests/geometry/epipolar/test_fundamental.py
+++ b/tests/geometry/epipolar/test_fundamental.py
@@ -15,7 +15,7 @@
# limitations under the License.
#
-from typing import Dict
+from typing import Dict, Tuple
import pytest
import torch
@@ -37,7 +37,7 @@ from kornia.geometry.solvers.homogeneous import _null_space_lu
from testing.base import DYNAMO_UNAVAILABLE_REASON, BaseTester, dynamo_is_available, script_uncached
from testing.geometry.create import create_random_fundamental_matrix, generate_two_view_random_scene
-from testing.two_view import two_view_scene
+from testing.two_view import affine_two_view_scene, two_view_scene
class TestNormalizePoints(BaseTester):
@@ -397,6 +397,205 @@ class TestFindFundamental(BaseTester):
fundamental_module.run_8point(points, points, weights)
+class TestFindFundamentalAffine(BaseTester):
+ """``method="AFFINE"``: the affine fundamental matrix, Hartley and Zisserman's algorithm 14.1."""
+
+ @staticmethod
+ def _tol(dtype: torch.dtype) -> Dict[str, float]:
+ return {"rtol": 1e-7, "atol": 1e-9} if dtype == torch.float64 else {"rtol": 5e-5, "atol": 1e-7}
+
+ @staticmethod
+ def _noisy(scene: Dict[str, torch.Tensor]) -> Tuple[torch.Tensor, torch.Tensor]:
+ return scene["x1"] + scene["noise"][..., :2], scene["x2"] + scene["noise"][..., 2:]
+
+ def test_exact_correspondences_give_the_cameras_matrix(self, device, dtype):
+ _skip_half(dtype, _HALF_PIXELS)
+ scene = affine_two_view_scene(device, dtype)
+ F = epi.find_fundamental(scene["x1"], scene["x2"], method="AFFINE")
+ assert F.shape == (1, 3, 3)
+ assert F.dtype == dtype
+ assert not F[:, :2, :2].any()
+ self.assert_close(F, scene["F"], **self._tol(dtype))
+
+ def test_four_correspondences_determine_it(self, device, dtype):
+ _skip_half(dtype, _HALF_PIXELS)
+ scene = affine_two_view_scene(device, dtype)
+ F = epi.find_fundamental(scene["x1"][:, :4], scene["x2"][:, :4], method="affine")
+ self.assert_close(F, scene["F"], **self._tol(dtype))
+
+ def test_noisy_correspondences_give_the_gold_standard(self, device, dtype):
+ _skip_half(dtype, _HALF_PIXELS)
+ scene = affine_two_view_scene(device, dtype)
+ x1, x2 = self._noisy(scene)
+ F = epi.find_fundamental(x1, x2, method="AFFINE")
+ self.assert_close(F, scene["F_gold"], **self._tol(dtype))
+ # Control: the noise moves the minimizer away from the cameras' matrix by more than the tolerance.
+ assert (scene["F_gold"] - scene["F"]).abs().max() > 1e-5
+
+ def test_swapping_the_images_transposes_it(self, device, dtype):
+ _skip_half(dtype, _HALF_PIXELS)
+ x1, x2 = self._noisy(affine_two_view_scene(device, dtype))
+ F = epi.find_fundamental(x1, x2, method="AFFINE")
+ self.assert_close(epi.find_fundamental(x2, x1, method="AFFINE"), F.transpose(-2, -1), **self._tol(dtype))
+
+ def test_weights_semantics(self, device, dtype):
+ _skip_half(dtype, _HALF_PIXELS)
+ x1, x2 = self._noisy(affine_two_view_scene(device, dtype))
+ x2[0, 5] += torch.tensor([40.0, -25.0], device=device, dtype=dtype)
+ keep = torch.arange(12, device=device) != 5
+ w = torch.tensor([[0.5, 1.0, 2.0, 1.5, 0.75, 0.0, 3.0, 0.25, 1.0, 2.5, 0.6, 1.8]], device=device, dtype=dtype)
+ F_w = epi.find_fundamental(x1, x2, w, method="AFFINE")
+ # A weight of 0 drops the gross mismatch: the fit is the weighted fit of the other eleven.
+ self.assert_close(F_w, epi.find_fundamental(x1[:, keep], x2[:, keep], w[:, keep], method="AFFINE"))
+ # Only the ratios of the weights matter.
+ self.assert_close(epi.find_fundamental(x1, x2, 7.0 * w, method="AFFINE"), F_w, **self._tol(dtype))
+ # A negative weight counts as zero, to the bit.
+ w_neg = w.clone()
+ w_neg[0, 5] = -5.0
+ assert torch.equal(epi.find_fundamental(x1, x2, w_neg, method="AFFINE"), F_w)
+ # Control: equal weights keep the mismatch and give another matrix.
+ assert (epi.find_fundamental(x1, x2, method="AFFINE") - F_w).abs().max() > 1e-4
+
+ def test_half_precision_points_fit_in_float32(self, device, dtype):
+ if dtype not in (torch.float16, torch.bfloat16):
+ pytest.skip("a half-precision input check")
+ scene = affine_two_view_scene(device, dtype)
+ # Squared pixel coordinates in the hundreds overflow float16, so the fit runs in float32.
+ F = epi.find_fundamental(scene["x1"], scene["x2"], method="AFFINE")
+ assert F.dtype == dtype
+ assert torch.isfinite(F).all()
+ expected = epi.find_fundamental(scene["x1"].float(), scene["x2"].float(), method="AFFINE").to(dtype)
+ assert torch.equal(F, expected)
+
+ def test_float32_points_fit_in_float64(self, device):
+ if device.type == "mps":
+ pytest.skip("MPS has no float64; the fit runs in float32 there")
+ x1, x2 = self._noisy(affine_two_view_scene(device, torch.float64))
+ # Far from the origin, as in a large image: the float32 points round to 2e-3 px, and a float32 scatter matrix
+ # of their centred coordinates would move F by more than that.
+ x1, x2 = (x1 + 15000.0).float(), (x2 + 15000.0).float()
+ F = epi.find_fundamental(x1, x2, method="AFFINE")
+ assert F.dtype == torch.float32
+ expected = epi.find_fundamental(x1.double(), x2.double(), method="AFFINE").float()
+ self.assert_close(F, expected, rtol=1e-6, atol=0.0)
+
+ @staticmethod
+ def _cube_seen_from_front_and_side(device: torch.device, dtype: torch.dtype) -> Tuple[torch.Tensor, torch.Tensor]:
+ # A 3 x 3 x 3 grid with spacing 100 seen by two orthographic cameras 90 degrees apart: x1 = (X, Y) and
+ # x2 = (Z, Y + 10). The stacked correspondences (Z, Y + 10, X, Y) have scatter eigenvalues 0, v, v and 2v:
+ # F = [[0, 0, 0], [0, 0, -0.1], [0, 0.1, 1]] is well defined, its eigenvalue 0 being simple, but eigh's
+ # backward divides by the gap between the two equal ones.
+ grid = torch.arange(3, device=device, dtype=dtype) * 100.0
+ X = torch.cartesian_prod(grid, grid, grid)
+ return X[None, :, :2], torch.stack([X[:, 2], X[:, 1] + 10.0], dim=-1)[None]
+
+ def test_gradient_is_finite_when_two_other_eigenvalues_are_equal(self, device, dtype):
+ _skip_half(dtype, _HALF_PIXELS)
+ x1, x2 = self._cube_seen_from_front_and_side(device, dtype)
+ weights = torch.ones(1, 27, device=device, dtype=dtype, requires_grad=True)
+ x1.requires_grad_()
+ x2.requires_grad_()
+ F = epi.find_fundamental(x1, x2, weights, method="AFFINE")
+ expected = torch.tensor([[[0.0, 0.0, 0.0], [0.0, 0.0, -0.1], [0.0, 0.1, 1.0]]], device=device, dtype=dtype)
+ self.assert_close(F, expected, **self._tol(dtype))
+ (F * torch.arange(9, device=device, dtype=dtype).view(3, 3)).sum().backward()
+ for grad in (x1.grad, x2.grad, weights.grad):
+ assert torch.isfinite(grad).all()
+
+ def test_gradcheck_when_two_other_eigenvalues_are_equal(self, device):
+ x1, x2 = self._cube_seen_from_front_and_side(device, torch.float64)
+ weights = torch.ones(1, 27, device=device, dtype=torch.float64)
+
+ def fit(p1: torch.Tensor, p2: torch.Tensor, w: torch.Tensor) -> torch.Tensor:
+ return epi.find_fundamental(p1, p2, w, method="AFFINE")
+
+ self.gradcheck(fit, (x1.requires_grad_(), x2.requires_grad_(), weights.requires_grad_()))
+
+ def test_float16_result_is_finite_for_centred_points(self, device):
+ # Points centred on the origin and scaled to unit size, as normalize_points leaves them, put the epipolar plane
+ # near the origin: e is near 0, and scaling F to F[2, 2] = 1 put the other entries above float16's 65504.
+ # Fifty points seen by the fixture's two affine cameras, with 0.3 px noise.
+ generator = torch.Generator().manual_seed(4)
+ X = torch.rand(50, 3, generator=generator) * 200.0 - 100.0
+ M1 = torch.tensor([[1.2, 0.1, -0.3], [0.05, -1.1, 0.4]])
+ M2 = torch.tensor([[0.9, 0.5, 0.2], [-0.3, -0.8, 0.7]])
+ points1 = (X @ M1.T)[None] + 0.3 * torch.randn(1, 50, 2, generator=generator)
+ points2 = (X @ M2.T)[None] + 0.3 * torch.randn(1, 50, 2, generator=generator)
+ points1, points2 = ((x - x.mean(1, keepdim=True)) for x in (points1, points2))
+ points1, points2 = (x / x.abs().max() for x in (points1, points2))
+ points1, points2 = points1.half().to(device), points2.half().to(device)
+ F = epi.find_fundamental(points1, points2, method="AFFINE")
+ assert F.dtype == torch.float16
+ assert torch.isfinite(F).all()
+ # The same matrix up to scale: the largest entry is 1 instead of F[2, 2].
+ expected = epi.find_fundamental(points1.float(), points2.float(), method="AFFINE")
+ self.assert_close(F.float(), expected / expected.abs().amax(), rtol=0.0, atol=1e-3)
+ assert F.abs().amax() == 1.0
+
+ @pytest.mark.parametrize("image", [1, 2])
+ def test_points_of_one_image_on_a_line_give_that_line(self, device, dtype, image):
+ # Every four of the fixture's twelve correspondences, with one image's points moved onto the line y = 50. The
+ # stacked correspondences then vary along neither that y nor, for some samples, a second direction whose
+ # scatter eigenvalue is below 1e-7 of the largest. A float32 eigh of the scatter matrix (a half input's, or a
+ # float32 input's on MPS) mixed the two; the singular vectors of the centred correspondences keep them apart.
+ scene = affine_two_view_scene(device, dtype)
+ x1, x2 = scene["x1"][0], scene["x2"][0]
+ (x1 if image == 1 else x2)[:, 1] = 50.0
+ samples = torch.combinations(torch.arange(12, device=device), 4)
+ F = epi.find_fundamental(x1[samples], x2[samples], method="AFFINE")
+ expected = torch.zeros(3, 3, device=device, dtype=dtype)
+ expected[(2, 1) if image == 1 else (1, 2)] = -0.02
+ expected[2, 2] = 1.0
+ self.assert_close(F, expected.expand_as(F), rtol=0.0, atol=1e-4)
+
+ def test_nonfinite_batch_element_gives_nan(self, device, dtype):
+ _skip_half(dtype, _HALF_PIXELS)
+ x1, x2 = self._noisy(affine_two_view_scene(device, dtype))
+ x1, x2 = x1.repeat(3, 1, 1), x2.repeat(3, 1, 1)
+ x1[1, 4] = float("nan")
+ x2[2, 7, 0] = float("inf")
+ x1.requires_grad_()
+ F = epi.find_fundamental(x1, x2, method="AFFINE")
+ # The finite element keeps its matrix and the others are NaN, rather than the whole batch raising.
+ self.assert_close(F[0], epi.find_fundamental(x1[:1].detach(), x2[:1], method="AFFINE")[0])
+ assert torch.isnan(F[1:]).all()
+ F[0].sum().backward()
+ assert torch.isfinite(x1.grad).all()
+
+ def test_too_few_correspondences_raise(self, device, dtype):
+ points = torch.rand(1, 3, 2, device=device, dtype=dtype)
+ with pytest.raises(ShapeError, match=r"points1 and points2 must have shape \(B, N, 2\) with N >= 4, got"):
+ epi.find_fundamental(points, points, method="AFFINE")
+ points = torch.rand(1, 4, 2, device=device, dtype=dtype)
+ with pytest.raises(ShapeError, match=r"weights must have shape \(B, N\) = \[1, 4\], got \["):
+ epi.find_fundamental(points, points, torch.ones(1, 5, device=device, dtype=dtype), method="AFFINE")
+
+ def test_gradcheck(self, device):
+ x1, x2 = self._noisy(affine_two_view_scene(device, torch.float64))
+ weights = torch.ones(1, 12, device=device, dtype=torch.float64)
+
+ def fit(p1: torch.Tensor, p2: torch.Tensor, w: torch.Tensor) -> torch.Tensor:
+ return epi.find_fundamental(p1, p2, w, method="AFFINE")
+
+ self.gradcheck(fit, (x1.requires_grad_(), x2.requires_grad_(), weights.requires_grad_()))
+
+ def test_zero_weight_backward_is_the_one_sided_derivative(self, device, dtype):
+ _skip_half(dtype, _HALF_PIXELS)
+ x1, x2 = self._noisy(affine_two_view_scene(device, dtype))
+ grads = []
+ for w3 in (0.0, 1e-6):
+ weights = torch.ones(1, 12, device=device, dtype=dtype)
+ weights[0, 3] = w3
+ weights.requires_grad_()
+ F_mat = epi.find_fundamental(x1, x2, weights, method="AFFINE")
+ (F_mat * F_mat.detach().sign()).sum().backward()
+ grads.append(weights.grad)
+ assert torch.isfinite(grads[0]).all()
+ assert grads[1][0, 3].abs() > 1e-6 # control: the dropped match moves F
+ tol = {"rtol": 1e-3, "atol": 1e-8} if dtype == torch.float64 else {"rtol": 5e-2, "atol": 1e-6}
+ self.assert_close(grads[0], grads[1], **tol)
+
+
class TestFundamentalAutocast(BaseTester):
@pytest.mark.parametrize(
"method, num_points, weighted",
@@ -802,6 +1001,7 @@ class TestGetClosestPointOnEpipolarLine(BaseTester):
_NO_HALF_EIGH = "find_fundamental calls torch.linalg.eigh, which has no float16/bfloat16 kernel"
+_HALF_PIXELS = "half precision rounds the fixture's pixel coordinates, 120 to 440, to steps of 0.125 to 0.25"
_HALF_PIXEL_F = (
"a pixel-unit F spans eight decades (entries down to ~1e-8): float16 flushes the small entries to zero and "
"bfloat16's 8-bit mantissa cannot resolve the epipolar residual, which kornia evaluates in the input dtype"
diff --git a/tests/geometry/test_ransac_fundamental_affine.py b/tests/geometry/test_ransac_fundamental_affine.py
new file mode 100644
index 000000000..2fe15314c
--- /dev/null
+++ b/tests/geometry/test_ransac_fundamental_affine.py
@@ -0,0 +1,201 @@
+# LICENSE HEADER MANAGED BY add-license-header
+#
+# Copyright 2018 Kornia Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import annotations
+
+from typing import Dict, Tuple
+
+import pytest
+import torch
+
+from kornia.geometry import RANSAC
+from kornia.geometry.epipolar import sampson_epipolar_distance
+
+from testing.base import BaseTester
+from testing.two_view import affine_two_view_scene
+
+# Mismatches 29 to 156 px (geometric distance) away from the fixture's affine epipolar geometry.
+_OUTLIERS1 = [
+ [220.6, 256.8], [222.6, 304.0], [376.3, 227.3], [268.2, 355.7],
+ [200.4, 433.6], [414.0, 340.4], [207.3, 369.7], [358.5, 423.6],
+] # fmt: skip
+_OUTLIERS2 = [
+ [334.3, 283.9], [260.2, 186.7], [240.0, 268.9], [310.1, 217.3],
+ [218.6, 222.2], [258.4, 327.9], [236.1, 170.9], [197.7, 294.9],
+] # fmt: skip
+
+# Twenty mismatches whose points in one image lie within 0.23 px of the line y = 0.5 x + 60, and in the other image are
+# scattered: a rank-one matrix, with (c, d) = 0 or (a, b) = 0, takes all of them as inliers. Each is at least 8.9 px
+# (geometric distance) from the fixture's epipolar geometry. Generated by
+# rng = np.random.default_rng(4)
+# on_line = rng.uniform(190, 430, 20)
+# on_line = np.round(np.c_[on_line, 0.5 * on_line + 60 + rng.uniform(-0.25, 0.25, 20)], 1)
+# scattered = np.round(rng.uniform([200, 150], [440, 380], (20, 2)), 1)
+_ON_A_LINE = [
+ [416.3, 268.3], [312.7, 216.6], [424.3, 272.2], [209.4, 164.5], [335.8, 227.9],
+ [280.4, 200.2], [382.5, 251.2], [231.9, 176.2], [399.2, 259.5], [320.5, 220.1],
+ [406.5, 263.3], [304.5, 212.3], [293.3, 206.9], [379.3, 249.7], [426.2, 273.0],
+ [278.7, 199.6], [422.5, 271.3], [413.0, 266.6], [232.6, 176.3], [336.1, 227.9],
+] # fmt: skip
+_SCATTERED = [
+ [366.2, 327.2], [245.8, 255.8], [286.8, 189.3], [253.1, 371.4], [412.2, 237.9],
+ [377.5, 159.7], [419.6, 273.5], [396.9, 213.5], [290.3, 230.1], [433.4, 248.8],
+ [319.9, 369.9], [416.8, 240.9], [273.8, 335.6], [225.8, 240.2], [410.3, 361.1],
+ [256.0, 168.1], [389.1, 293.0], [257.0, 266.4], [217.1, 311.0], [326.4, 312.1],
+] # fmt: skip
+
+
+def _with_outliers(device: torch.device, dtype: torch.dtype) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """The twelve exact correspondences of the affine fixture followed by eight mismatches, and the true F."""
+ scene = affine_two_view_scene(device, dtype)
+ x1 = torch.cat([scene["x1"][0], torch.tensor(_OUTLIERS1, device=device, dtype=dtype)])
+ x2 = torch.cat([scene["x2"][0], torch.tensor(_OUTLIERS2, device=device, dtype=dtype)])
+ return x1, x2, scene["F"][0]
+
+
+def _tol(dtype: torch.dtype) -> Dict[str, float]:
+ return {"rtol": 1e-7, "atol": 1e-9} if dtype == torch.float64 else {"rtol": 5e-5, "atol": 1e-7}
+
+
+def _skip_half(dtype: torch.dtype) -> None:
+ if dtype in (torch.float16, torch.bfloat16):
+ pytest.skip("half precision rounds the fixture's pixel coordinates to steps of 0.125 to 0.25")
+
+
+class TestRANSACFundamentalAffine(BaseTester):
+ @pytest.mark.parametrize("score_type", ["ransac", "msac"])
+ @pytest.mark.parametrize("lo_sample_size", [None, 8])
+ def test_inliers_and_model_with_outliers(self, device, dtype, score_type, lo_sample_size):
+ _skip_half(dtype)
+ x1, x2, expected = _with_outliers(device, dtype)
+ estimator = RANSAC(
+ "fundamental_affine",
+ inl_th=1.0,
+ batch_size=32,
+ max_samples=256,
+ score_type=score_type,
+ lo_sample_size=lo_sample_size,
+ seed=0,
+ )
+ model, inliers = estimator(x1, x2)
+ assert model.dtype == dtype
+ assert model.device == device
+ assert torch.equal(inliers, torch.arange(20, device=device) < 12)
+ assert not model[:2, :2].any()
+ self.assert_close(model, expected, **_tol(dtype))
+
+ @pytest.mark.parametrize("lo_sample_size", [None, 8])
+ def test_refit_gives_the_gold_standard_of_the_inliers(self, device, dtype, lo_sample_size):
+ _skip_half(dtype)
+ scene = affine_two_view_scene(device, dtype)
+ x1, x2, _ = _with_outliers(device, dtype)
+ x1[:12] += scene["noise"][0, :, :2]
+ x2[:12] += scene["noise"][0, :, 2:]
+ estimator = RANSAC(
+ "fundamental_affine", inl_th=1.0, batch_size=32, max_samples=256, lo_sample_size=lo_sample_size, seed=0
+ )
+ model, inliers = estimator(x1, x2)
+ assert torch.equal(inliers, torch.arange(20, device=device) < 12)
+ # The refit on the twelve noisy inliers is their Gold Standard fit; a four-point model is 5e-4 away from it.
+ self.assert_close(model, scene["F_gold"][0], **_tol(dtype))
+
+ @pytest.mark.parametrize("image", [1, 2])
+ def test_points_on_a_line_in_one_image_give_no_model(self, device, dtype, image):
+ x1, x2, _ = _with_outliers(device, dtype)
+ # With one image's points all on the line y = 0, the rank-one matrix of that line, (a, b) = (0, 1) or
+ # (c, d) = (0, 1), fits all twenty correspondences exactly: no epipolar geometry. A sample that is nearly
+ # coplanar besides can be fitted by another matrix, which a few correspondences fit.
+ (x1 if image == 1 else x2)[:, 1] = 0.0
+ _, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=256, seed=0)(x1, x2)
+ assert int(inliers.sum()) < 20
+
+ @pytest.mark.parametrize("image", [1, 2])
+ def test_mismatches_along_a_line_do_not_outvote_the_true_geometry(self, device, dtype, image):
+ _skip_half(dtype)
+ x1, x2, expected = _with_outliers(device, dtype)
+ on_a_line = torch.tensor(_ON_A_LINE, device=device, dtype=dtype)
+ scattered = torch.tensor(_SCATTERED, device=device, dtype=dtype)
+ x1 = torch.cat([x1, on_a_line if image == 1 else scattered])
+ x2 = torch.cat([x2, scattered if image == 1 else on_a_line])
+ # The twenty line mismatches fit a near-rank-one matrix (their noise off the line makes (c, d) or (a, b) small,
+ # not zero) better than the twelve true correspondences fit theirs. A sample of four true ones has probability
+ # (12 / 40)^4 = 0.8%: 2048 samples miss it with probability 6e-8.
+ model, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=2048, seed=0)(x1, x2)
+ assert torch.equal(inliers, torch.arange(40, device=device) < 12)
+ self.assert_close(model, expected, **_tol(dtype))
+
+ def test_true_geometry_that_spreads_little_is_kept(self, device, dtype):
+ _skip_half(dtype)
+ x1, x2, F = _with_outliers(device, dtype)
+ # View 2 shrunk 20 times about (300, 260), as a far smaller-scale image would be. Over the keypoints, the
+ # true matrix's two terms of the distance vary by 3.7 and 4.9 px (standard deviations): a check at four
+ # thresholds or more would reject it. The mismatches stay at least 2.1 px from it.
+ scale, cx, cy = 0.05, 300.0, 260.0
+ centre = torch.tensor([cx, cy], device=device, dtype=dtype)
+ x2 = centre + scale * (x2 - centre)
+ # The fixture's F applies to the unshrunk points, T x2.
+ T = torch.tensor(
+ [[1 / scale, 0.0, cx * (1 - 1 / scale)], [0.0, 1 / scale, cy * (1 - 1 / scale)], [0.0, 0.0, 1.0]],
+ device=device,
+ dtype=dtype,
+ )
+ expected = T.mT @ F
+ expected = expected / expected[2, 2]
+ model, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=256, seed=0)(x1, x2)
+ assert torch.equal(inliers, torch.arange(20, device=device) < 12)
+ self.assert_close(model, expected, **_tol(dtype))
+
+ def test_nonfinite_correspondences_are_outliers(self, device, dtype):
+ _skip_half(dtype)
+ x1, x2, expected = _with_outliers(device, dtype)
+ # Copies of five true correspondences, three with a NaN point in image 1 and two with an infinite coordinate
+ # in image 2. Most samples draw one of them, and the keypoints' spread includes them.
+ x1, x2 = torch.cat([x1, x1[:5]]), torch.cat([x2, x2[:5]])
+ x1[20:23] = float("nan")
+ x2[23:, 0] = float("inf")
+ model, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=256, seed=0)(x1, x2)
+ assert torch.equal(inliers, torch.arange(25, device=device) < 12)
+ self.assert_close(model, expected, **_tol(dtype))
+
+ def test_threshold_is_the_geometric_distance(self, device, dtype):
+ _skip_half(dtype)
+ x1, x2, _ = _with_outliers(device, dtype)
+ # Moves of x2 along the epipolar lines' normal (a, b) by which the correspondence is 0.75 px and 1.5 px from
+ # the nearest pair of corresponding points (the four-dimensional distance, which the Sampson distance of an
+ # affine F equals): r = a dx + b dy = d * |(a, b, c, d)|.
+ x1, x2 = torch.cat([x1, x1[:2]]), torch.cat([x2, x2[:2]])
+ x2[-2:] += torch.tensor([[0.532378098326968, -0.9264217301595274], [1.064756196653936, -1.852843460319055]],
+ device=device, dtype=dtype) # fmt: skip
+ model, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=256, seed=0)(x1, x2)
+ assert inliers[-2]
+ assert not inliers[-1]
+ classified = sampson_epipolar_distance(x1[None], x2[None], model[None])[0] <= 1.0
+ assert torch.equal(inliers, classified)
+
+ def test_minimal_sample_is_four(self, device, dtype):
+ x1, x2, _ = _with_outliers(device, dtype)
+ with pytest.raises(ValueError, match=r"at least \[4, 2\]"):
+ RANSAC("fundamental_affine").validate_inputs(x1[:3], x2[:3])
+ # Four correspondences fit a model exactly, which is no evidence of a consensus.
+ model, inliers = RANSAC("fundamental_affine", max_samples=16, seed=0)(x1[:4], x2[:4])
+ assert not model.any()
+ assert not inliers.any()
+
+ @pytest.mark.parametrize("kwargs", [{"local_optimization": "lm"}, {"degensac": True}, {"compile": True}])
+ def test_unsupported_options_raise(self, kwargs):
+ with pytest.raises(ValueError):
+ RANSAC("fundamental_affine", **kwargs)