π Geometric Computer Vision Library for Spatial AI
Python artificial-intelligencecomputer-visiondeep-learninghacktoberfestimage-processingmachine-learningneural-networkpythonpytorchroboticsspatial-ai
feat(geometry): affine fundamental matrix in find_fundamental and RANSAC (#6569)
Dmytro Mishkin2cdf08e96868
diff --git a/changelog.d/6569.added.md b/changelog.d/6569.added.mdnew file mode 100644index 000000000..2e3b3b1df--- /dev/null+++ b/changelog.d/6569.added.md@@ -0,0 +1 @@+`find_fundamental(..., method="AFFINE")` estimates the affine fundamental matrix of two affine cameras (weak-perspective or other parallel-projection views, such as aerial images taken with a long focal length) from four or more correspondences, by the Gold Standard orthogonal regression of Hartley and Zisserman's algorithm 14.1, and `RANSAC("fundamental_affine")` estimates it robustly from four-point samples, with the exact geometric distance as its residual.diff --git a/docs/source/geometry.epipolar.rst b/docs/source/geometry.epipolar.rstindex 0cb26334f..997a40be9 100644--- a/docs/source/geometry.epipolar.rst+++ b/docs/source/geometry.epipolar.rst@@ -42,12 +42,12 @@ two known poses (:func:`relative_camera_motion`).Fundamental------------Estimate the fundamental matrix from pixel correspondences with the 8-point or 7-point algorithm-(:func:`find_fundamental`), convert from an essential matrix or two projection matrices-(:func:`fundamental_from_essential`, :func:`fundamental_from_projections`), compute epipolar lines and the nearest-points on them (:func:`compute_correspond_epilines`, :func:`get_closest_point_on_epipolar_line`,-:func:`get_perpendicular`), and normalize points and matrices for the estimators (:func:`normalize_points`,-:func:`normalize_transformation`).+Estimate the fundamental matrix from pixel correspondences with the 8-point or 7-point algorithm, or the affine+fundamental matrix of two affine cameras from four or more (:func:`find_fundamental`), convert from an essential+matrix or two projection matrices (:func:`fundamental_from_essential`, :func:`fundamental_from_projections`),+compute epipolar lines and the nearest points on them (:func:`compute_correspond_epilines`,+:func:`get_closest_point_on_epipolar_line`, :func:`get_perpendicular`), and normalize points and matrices for the+estimators (:func:`normalize_points`, :func:`normalize_transformation`)... autofunction:: find_fundamental.. autofunction:: fundamental_from_essentialdiff --git a/docs/source/geometry.ransac.rst b/docs/source/geometry.ransac.rstindex 00c53e710..decb26c80 100644--- a/docs/source/geometry.ransac.rst+++ b/docs/source/geometry.ransac.rst@@ -48,6 +48,52 @@ AdaLAM shares the linear fitting primitive but retains its specialized loop:its local 2x2 models are constrained by a matched seed, and its variable-sizeneighborhoods, sampling and confidence-based inlier selection are unchanged.+Affine fundamental matrices+---------------------------++``model_type="fundamental_affine"`` estimates the fundamental matrix of two affine+cameras, ``[[0, 0, a], [0, 0, b], [c, d, e]]``: views in parallel projection,+such as weak-perspective views of a distant scene (aerial or satellite images+taken with a long focal length) or images through a telecentric lens. Its epipolar+lines are parallel in each image, and it has four degrees of freedom against seven+for a general fundamental matrix. Each four-point sample is solved, and each new+best model refitted on its inliers, by+:func:`~kornia.geometry.epipolar.find_fundamental` with ``method="AFFINE"``:+orthogonal regression on the stacked correspondences ``(x2, y2, x1, y1)``, the+Gold Standard estimate of Hartley and Zisserman's algorithm 14.1. A+correspondence's Sampson distance is then exact: the smallest total displacement+of its two points that makes them correspond, the distance ``inl_th`` bounds.+Like any fundamental matrix, it is not determined by correspondences on one plane,+which a single 2D affine map relates (``model_type="affine"`` tests for that).+A matrix with ``(a, b)`` or ``(c, d)`` near zero ignores one image: it is a line in+the other, which every mismatch along that line fits, so a line of mismatches+(along an edge, say) could outvote the true geometry. Such a model gets no+inliers: each image's term of the distance, ``(a x2 + b y2)`` and+``(c x1 + d y1)`` over ``|(a, b, c, d)|``, must vary over the keypoints by at+least two inlier thresholds (a standard deviation). In 1000 px images with 30+true correspondences, 40 mismatches within 0.3 px of a line and 30 random ones+(ten scenes, the line in either image, ``inl_th`` of 1 or 2), it keeps 27 to 30+of the true ones and 0 to 2 of the line; without the check, 0 to 1 and all 40.+A true geometry whose keypoints spread less than that across its epipolar lines+is rejected too, and RANSAC then returns zeros. With 200 correspondences, 0.5 px+noise and ``inl_th=2.0``, that happens to a 15 px patch and to a second view at+1/100 of the first one's scale, but not to a 30 px patch or a scale of 1/20.+It suits views that are close to parallel projection. On synthetic pinhole views of+a 60 m scene from 45 degrees off nadir (1000 px images, 0.5 px noise,+``inl_th=2.0``), it kept 99% of the true correspondences from 3000 m away,+89% from 1500 m and 49% from 500 m, where ``"fundamental"`` kept all of them;+on exactly affine views both kept all of them, ``"fundamental_affine"`` with a more+accurate matrix, several times faster.+It uses ``local_optimization="dlt"``; ``"lm"``, ``compile=True`` and DEGENSAC+are not supported.++.. code-block:: python++ from kornia.geometry import RANSAC++ estimator = RANSAC("fundamental_affine", inl_th=2.0, seed=0)+ F, inliers = estimator(points1, points2)+Batches, scores, and refinement-------------------------------@@ -102,8 +148,8 @@ endpoints to the target segment's line; zero-length target segments are outliersTheir local optimization re-weights segments by the same distance.A model is accepted only with more inliers than its minimal sample (three-correspondences for affine transforms, four for homographies, five for essential and seven or eight for-fundamental matrices): a model fitted to a sample always fits that sample, so+correspondences for affine transforms, four for homographies and affine fundamental matrices, five for essential+and seven or eight for fundamental matrices): a model fitted to a sample always fits that sample, soonly further inliers show a consensus. Without one, the estimator returns theall-zero matrix and an empty mask.diff --git a/kornia/geometry/epipolar/__init__.py b/kornia/geometry/epipolar/__init__.pyindex ecf4ba603..31344837d 100644--- a/kornia/geometry/epipolar/__init__.py+++ b/kornia/geometry/epipolar/__init__.py@@ -27,7 +27,8 @@ motion. :ref:`Two-view geometry <two-view-conventions>` compares these conventioWhich function to use:-- A model from correspondences known to be inliers: :func:`find_fundamental` from pixels (8 or 7 points),+- A model from correspondences known to be inliers: :func:`find_fundamental` from pixels (8 or 7 points, or the+ affine fundamental matrix of two affine cameras from 4 or more),:func:`find_essential` from normalized camera coordinates (Nister's 5 points, up to ten candidates). For a planeor a pure rotation, :func:`~kornia.geometry.homography.find_homography_dlt`; with outliers,:class:`~kornia.geometry.ransac.RANSAC`.diff --git a/kornia/geometry/epipolar/fundamental.py b/kornia/geometry/epipolar/fundamental.pyindex 396ab0f12..da42fcc1e 100644--- a/kornia/geometry/epipolar/fundamental.py+++ b/kornia/geometry/epipolar/fundamental.py@@ -24,7 +24,14 @@ import torchfrom kornia.core.check import KORNIA_CHECK_SAME_SHAPE, KORNIA_CHECK_SHAPEfrom kornia.core.exceptions import ShapeError-from kornia.core.utils import _autocast_disabled, _torch_svd_cast, safe_inverse_with_mask+from kornia.core.utils import (+ _autocast_disabled,+ _svd_work_dtype,+ _symmetric_eigenvector,+ _torch_svd_cast,+ _torch_svd_in_work_dtype,+ safe_inverse_with_mask,+)from kornia.geometry.conversions import convert_points_from_homogeneous, convert_points_to_homogeneousfrom kornia.geometry.solvers.homogeneous import _det3, _null_space_lufrom kornia.geometry.solvers.polynomial_solver import _solve_cubic_real@@ -623,19 +630,112 @@ def run_8point(return normalize_transformation(F)+def run_affine(points1: torch.Tensor, points2: torch.Tensor, weights: Optional[torch.Tensor] = None) -> torch.Tensor:+ r"""Compute the affine fundamental matrix of two affine cameras by orthogonal regression.++ Two affine cameras (parallel projection, such as weak perspective) have a fundamental matrix of the form+ ``[[0, 0, a], [0, 0, b], [c, d, e]]``: the epipolar constraint :math:`a x_2 + b y_2 + c x_1 + d y_1 + e = 0` is a+ hyperplane in the space of the stacked correspondences :math:`(x_2, y_2, x_1, y_1)`, and the distance of a+ correspondence to it is the geometric error, the smallest total displacement of its two points that makes them+ correspond. The hyperplane that minimizes the sum of the squared (weighted) distances passes through the+ centroid, and its normal is the eigenvector of the smallest eigenvalue of the centred scatter matrix. This is+ the Gold Standard (maximum-likelihood) estimate under Gaussian noise, exact for four correspondences. When the+ points of one image lie on a line, the minimizer is that line, a rank-one matrix with ``(a, b) = 0`` or+ ``(c, d) = 0``, which no pair of affine cameras has; ``RANSAC("fundamental_affine")`` rejects such models.++ Reference: Hartley and Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., section 14.2, algorithm 14.1.++ Args:+ points1: points in the first image ``(B, N, 2)``, :math:`N \ge 4`.+ points2: points in the second image ``(B, N, 2)``.+ weights: optional nonnegative weights ``(B, N)`` of the correspondences' squared distances.++ Returns:+ The affine fundamental matrices ``(B, 3, 3)``, scaled so that ``F[2, 2] = 1``.++ """+ KORNIA_CHECK_SHAPE(points1, ["B", "N", "2"])+ KORNIA_CHECK_SHAPE(points2, ["B", "N", "2"])+ KORNIA_CHECK_SAME_SHAPE(points1, points2)+ if points1.shape[1] < 4:+ raise ShapeError(+ f"points1 and points2 must have shape (B, N, 2) with N >= 4, got {list(points1.shape)}.",+ actual_shape=list(points1.shape),+ expected_shape=["B", "N >= 4", "2"],+ )+ if weights is not None and weights.shape != points1.shape[:2]:+ raise ShapeError(+ f"weights must have shape (B, N) = {list(points1.shape[:2])}, got {list(weights.shape)}.",+ actual_shape=list(weights.shape),+ expected_shape=["B", "N"],+ )+ dtype = points1.dtype+ # Pixel coordinates in the hundreds overflow half precision, and the SVD has no half kernel. float32 points are+ # fitted in float64 (float32 on MPS), so that far from the origin, as in a large image, the fit adds no rounding+ # of its own to theirs.+ work = _svd_work_dtype(points1)+ with _autocast_disabled(points1.device.type):+ X = torch.cat([points2, points1], dim=-1).to(work)+ if weights is None:+ w = torch.ones(X.shape[:2], device=X.device, dtype=work)+ else:+ # A negative weight counts as zero; ``where`` keeps the derivative at a zero weight one-sided (#4229).+ w = weights.to(work)+ w = torch.where(w < 0, 0.0, w)+ # A batch element with a non-finite point or weight, whose sum is then not finite, has no matrix. It is fitted+ # as zeros and its F set to NaN, so the other elements keep theirs and every gradient stays finite, as in+ # _svd_null_vector (#4770).+ finite = (X.sum((-2, -1)) + w.sum(-1)).isfinite()+ X = torch.where(finite.view(-1, 1, 1), X, 0.0)+ w = torch.where(finite.view(-1, 1), w, 1.0)+ mean = (w[..., None] * X).sum(1) / w.sum(1, keepdim=True).clamp_min(torch.finfo(work).tiny)+ centred = X - mean[:, None]+ # Linear in w, so a zero weight has a finite gradient, unlike row scaling by sqrt(w) (#4912).+ scatter = (centred * w[..., None]).transpose(-2, -1) @ centred+ # The derivatives come from the scatter: _symmetric_eigenvector divides only by the gaps to the smallest+ # eigenvalue, so they stay finite when two other eigenvalues are equal, as for a grid seen from the front and+ # the side (#6214).+ if work == torch.float64:+ eigenvalues, eigenvectors = torch.linalg.eigh(scatter.detach())+ normal = _symmetric_eigenvector(scatter, eigenvalues, eigenvectors, 0)+ else:+ # The scatter squares the condition number: in float32 (on MPS, or for a half input), its eigh mixes the two+ # smallest directions of a sample near a line and a plane, and torch 2.5's MPS has no eigh. The right+ # singular vectors of the weighted centred points keep them apart; their squared singular values are the+ # scatter's eigenvalues. In float64, eigh resolves them and is several times faster than an SVD on CUDA.+ _, S, V = _torch_svd_in_work_dtype((centred * w[..., None].sqrt()).detach(), full_matrices=False)+ normal = _symmetric_eigenvector(scatter, S.to(work).square(), V.to(work), -1)+ zeros = torch.zeros_like(normal[:, 0])+ last = -(normal * mean).sum(-1)+ F = torch.stack(+ [zeros, zeros, normal[:, 0], zeros, zeros, normal[:, 1], normal[:, 2], normal[:, 3], last], dim=-1+ ).view(-1, 3, 3)+ F = normalize_transformation(F)+ F = torch.where(finite.view(-1, 1, 1), F, float("nan"))+ if dtype == torch.float16:+ # Points centred on the origin put e near 0, and F[2, 2] = 1 then puts the other entries above float16's+ # 65504: such an F is scaled to its largest entry 1 instead.+ scale = F.abs().amax(dim=(-2, -1), keepdim=True)+ F = torch.where(scale > torch.finfo(torch.float16).max, F / scale, F)+ return F.to(dtype)++def find_fundamental(points1: torch.Tensor,points2: torch.Tensor,weights: Optional[torch.Tensor] = None,- method: Literal["8POINT", "7POINT"] = "8POINT",+ method: Literal["8POINT", "7POINT", "AFFINE"] = "8POINT",) -> torch.Tensor:- r"""Estimate the fundamental matrix from pixel correspondences with the 8- or 7-point algorithm.+ r"""Estimate the fundamental matrix from pixel correspondences with the 8-, 7-point or affine algorithm.``F = find_fundamental(points1, points2)`` takes correspondences ``(B, N, 2)``, ``N >= 8``, of the first and thesecond image, in pixels or any other image coordinates, and returns the rank-2 matrix ``(B, 3, 3)`` with:math:`x_2^\top F x_1 = 0` (the second image's point on the left) of the Hartley-normalized, optionally weighted,8-point least-squares fit. ``method="7POINT"`` takes exactly seven correspondences and returns the three- candidates ``(B, 3, 3, 3)`` of the 7-point algorithm instead. For calibrated cameras and normalized camera+ candidates ``(B, 3, 3, 3)`` of the 7-point algorithm instead. ``method="AFFINE"`` takes ``N >= 4``+ correspondences of two affine cameras (parallel projection, such as weak-perspective views of a distant scene)+ and returns their affine fundamental matrix ``[[0, 0, a], [0, 0, b], [c, d, e]]``, the Gold Standard+ (maximum-likelihood) fit of Hartley and Zisserman's algorithm 14.1. For calibrated cameras and normalized cameracoordinates, use :func:`find_essential`; for a planar scene or a camera that only rotates, where ``F`` is notdetermined, :func:`~kornia.geometry.homography.find_homography_dlt`; with outliers,:class:`~kornia.geometry.ransac.RANSAC`.@@ -645,20 +745,24 @@ def find_fundamental(- The result is scaled so that ``F[2, 2] = 1`` by :func:`normalize_transformation`, which leaves it at itsunnormalised scale when ``F[2, 2]`` is numerically zero, as for exactly rectified stereo.``method="7POINT"`` returns three candidates in no particular order.- - ``weights`` weight each correspondence's equation in the linear system: only their ratios matter, a- negative weight counts as zero, and ``method="7POINT"`` ignores them.+ - ``weights`` weight each correspondence's equation in the linear system (for ``"AFFINE"``, its squared+ distance): only their ratios matter, a negative weight counts as zero, and ``method="7POINT"`` ignores them.+ - ``"AFFINE"`` fits and returns the affine form exactly; it computes in float64 (float32 on MPS and for+ half-precision input). A float16 result whose entries ``F[2, 2] = 1`` would put above float16's maximum+ (``e`` near 0, as for points centred on the origin) is scaled to its largest entry 1 instead. A batch+ element with a non-finite point or weight gets a NaN matrix, and the others are unaffected.- When the 7-point cubic has one real root, the two extra ``"7POINT"`` candidates are zero matrices.- A caller's autocast region does not change the fit.Args:points1: A set of points in the first image with a tensor shape ``(B, N, 2)``: :math:`N \ge 8` for- ``"8POINT"``, exactly 7 for ``"7POINT"``.+ ``"8POINT"``, exactly 7 for ``"7POINT"``, :math:`N \ge 4` for ``"AFFINE"``.points2: A set of points in the second image with the shape of ``points1``.weights: Tensor containing the weights per point correspondence with a shape of ``(B, N)``.- method: The method to use for computing the fundamental matrix. Supported methods are "7POINT" and "8POINT".+ method: The method to use for computing the fundamental matrix: "8POINT", "7POINT" or "AFFINE".Returns:- the computed fundamental matrix with shape ``(B, 3, 3)`` for ``"8POINT"`` and+ the computed fundamental matrix with shape ``(B, 3, 3)`` for ``"8POINT"`` and ``"AFFINE"``, and``(B, 3, 3, 3)`` for ``"7POINT"``.Raises:@@ -677,8 +781,10 @@ def find_fundamental(result = run_7point(points1, points2)elif method.upper() == "8POINT":result = run_8point(points1, points2, weights)+ elif method.upper() == "AFFINE":+ result = run_affine(points1, points2, weights)else:- raise ValueError(f"Invalid method: {method}. Supported methods are '7POINT' and '8POINT'.")+ raise ValueError(f"Invalid method: {method}. Supported methods are '7POINT', '8POINT' and 'AFFINE'.")return resultdiff --git a/kornia/geometry/ransac.py b/kornia/geometry/ransac.pyindex 0508f99ca..f7accbe9e 100644--- a/kornia/geometry/ransac.py+++ b/kornia/geometry/ransac.py@@ -19,15 +19,16 @@ r"""Estimate geometric models robustly from correspondences that contain outlier:class:`RANSAC` fits one model to one set of ``(N, 2)`` correspondences (``(N, 2, 2)`` segments for``"homography_from_linesegments"``), with the first image's points first: a homography or an affine map from the first-image to the second, a fundamental matrix with :math:`x_2^\top F x_1 = 0` from pixels, or an essential matrix from-normalized camera coordinates. Its threshold ``inl_th`` is a distance in the units of the correspondences.+image to the second, a fundamental matrix with :math:`x_2^\top F x_1 = 0` from pixels (a general one, or the affine+fundamental matrix of two affine cameras), or an essential matrix from normalized camera coordinates. Its threshold+``inl_th`` is a distance in the units of the correspondences.:ref:`Two-view geometry <two-view-conventions>` compares this with OpenCV.Which class to use:- One model from correspondences with outliers: :class:`RANSAC`, with ``model_type`` one of ``"homography"``,``"homography_from_linesegments"``, ``"affine"``, ``"fundamental"`` (seven-point samples),- ``"fundamental_7pt"``, ``"fundamental_8pt"`` or ``"essential"``.+ ``"fundamental_7pt"``, ``"fundamental_8pt"``, ``"fundamental_affine"`` or ``"essential"``.- A model from correspondences known to be inliers, without sampling: the estimators it wraps,:func:`~kornia.geometry.homography.find_homography_dlt`, :func:`~kornia.geometry.solvers.find_affine`,:func:`~kornia.geometry.epipolar.find_fundamental` and :func:`~kornia.geometry.epipolar.find_essential`.@@ -65,6 +66,7 @@ from kornia.geometry.epipolar.fundamental import (_seven_point_candidates,normalize_points,normalize_transformation,+ run_affine,)from kornia.geometry.homography import (_four_point_homography,@@ -97,6 +99,43 @@ _DEGENSAC_MODELS = ("fundamental", "fundamental_7pt")# tuned batch sizes; large explicit batches are split without changing the minimal-sample budget._COMPILED_CPU_SCORE_RESIDUALS = 1 << 22+# An affine fundamental matrix must tie the two images together: as a correspondence's point in one image moves over+# the keypoints, the geometric distance must vary by at least this many inlier thresholds (a standard deviation). The+# tests bound it on both sides: at 1, twenty mismatches along a line outvote twelve true correspondences, and at 4, a+# true geometry whose keypoints spread 3.7 thresholds across its epipolar lines is rejected.+_AFFINE_FUNDAMENTAL_MIN_SPREAD = 2.0+++def _affine_fundamental_ties_both_images(+ kp1: torch.Tensor, kp2: torch.Tensor, models: torch.Tensor, inl_th: float+) -> torch.Tensor:+ """Return which affine fundamental matrices ``(B, 3, 3)`` depend on both images' points, ``(B,)``.++ ``F = [[0, 0, a], [0, 0, b], [c, d, e]]`` puts a correspondence at the geometric distance+ ``(a x2 + b y2 + c x1 + d y1 + e) / |(a, b, c, d)|``. With ``(c, d) = 0`` that is the distance of ``x2`` to one+ line, whatever ``x1`` is, so mismatches whose second points lie along a line all fit it and can outvote the true+ geometry; with ``(a, b) = 0`` the same holds in the first image. Points near a line give small terms rather than+ zero ones, and what counts as small depends on the images' extent and the threshold. So each image's term,+ ``(c x1 + d y1)`` over ``kp1`` and ``(a x2 + b y2)`` over ``kp2``, divided by ``|(a, b, c, d)|``, must have a+ standard deviation of at least ``_AFFINE_FUNDAMENTAL_MIN_SPREAD`` times the inlier threshold ``inl_th`` (not+ squared). For the true geometry the first image's term is the spread of the matched points across the epipolar+ lines, seen in the second image. Correspondences with a non-finite point, which no model fits, are left out.+ """+ with _autocast_disabled(kp1.device.type):+ work = torch.promote_types(torch.promote_types(kp1.dtype, models.dtype), torch.float32)+ normal = models[:, [0, 1, 2, 2], [2, 2, 0, 1]].to(work) # (a, b, c, d)+ # The correspondences as (x2, y2, x1, y1), the order of (a, b, c, d); a row whose sum is not finite is left out.+ points = torch.cat([kp2.reshape(-1, 2), kp1.reshape(-1, 2)], -1).to(work)+ finite = points.sum(-1, keepdim=True).isfinite()+ count = finite.sum().clamp_min(1)+ points = torch.where(finite, points, 0.0)+ centred = torch.where(finite, points - points.sum(0) / count, 0.0)+ covariance = centred.mT @ centred / count+ # Each image's term has the variance n^T C n over its own 2 x 2 block of the covariance; compared squared.+ floor = (_AFFINE_FUNDAMENTAL_MIN_SPREAD * inl_th) ** 2 * normal.square().sum(-1)+ n2, n1 = normal[:, :2], normal[:, 2:]+ return (((n2 @ covariance[:2, :2]) * n2).sum(-1) >= floor) & (((n1 @ covariance[2:, 2:]) * n1).sum(-1) >= floor)+def _affine_transfer_error(pts1: torch.Tensor, pts2: torch.Tensor, models: torch.Tensor) -> torch.Tensor:"""Squared one-way transfer errors ``(B, N)`` of affine models ``(B, 3, 3)`` on one set of correspondences.@@ -203,7 +242,8 @@ class RANSAC(nn.Module):when its error under the model is at most ``inl_th``, in the units of the keypoints. The model follows theestimator of its type: a homography or affine map from ``kp1`` to ``kp2``(:func:`~kornia.geometry.homography.find_homography_dlt`, :func:`~kornia.geometry.solvers.find_affine`), a- fundamental matrix with :math:`x_2^\top F x_1 = 0` (:func:`~kornia.geometry.epipolar.find_fundamental`), or an+ fundamental matrix with :math:`x_2^\top F x_1 = 0` (:func:`~kornia.geometry.epipolar.find_fundamental`; for+ ``"fundamental_affine"`` its ``method="AFFINE"``, the matrix of two affine cameras), or anessential matrix for camera-normalized keypoints (:func:`~kornia.geometry.epipolar.find_essential`). Each callestimates one model, since ``batch_size`` batches hypotheses rather than problems; for correspondences known to beinliers, call those estimators directly (see also https://en.wikipedia.org/wiki/Random_sample_consensus).@@ -211,16 +251,17 @@ class RANSAC(nn.Module):Convention:- ``kp1`` and ``kp2`` are passed as the first- and second-image arguments of the estimator selected by``model_type``. ``forward`` returns an all-zero model and no inliers when no candidate has more inliers- than its minimal sample (three correspondences for affine transforms, four for homographies,- five for ``"essential"``, seven for ``"fundamental"`` and ``"fundamental_7pt"``,+ than its minimal sample (three correspondences for affine transforms, four for homographies and+ ``"fundamental_affine"``, five for ``"essential"``, seven for ``"fundamental"`` and ``"fundamental_7pt"``,eight for ``"fundamental_8pt"``).- ``"fundamental"`` draws seven-point samples, like ``"fundamental_7pt"``; ``"fundamental_8pt"`` drawseight-point ones.- ``inl_th`` is in the keypoints' own units (pixels, or calibrated units for ``"essential"``): acorrespondence is an inlier when its one-way transfer error for ``"homography"`` or ``"affine"``,- its Sampson distance for the fundamental models and ``"essential"``, or the mean distance- of its transferred endpoints from the image-2 segment's line for ``"homography_from_linesegments"``- is at most ``inl_th``.+ its Sampson distance for the fundamental models and ``"essential"`` (for ``"fundamental_affine"`` the+ exact geometric distance: the smallest total displacement of the two points that makes them correspond),+ or the mean distance of its transferred endpoints from the image-2 segment's line for+ ``"homography_from_linesegments"`` is at most ``inl_th``.:ref:`two-view-conventions` compares this with OpenCV.- ``score_type="msac"`` (the default) ranks candidates by ``sum(1 - min(e / inl_th**2, 1))`` over thesquared errors ``e``; acceptance and early stopping count inliers for either score.@@ -236,12 +277,18 @@ class RANSAC(nn.Module):fundamental matrices are estimated on Hartley-normalized correspondences; essential matrices in thecaller's calibrated coordinates, which a normalization would take off the essential manifold, refined onthat manifold and returned with unit Frobenius norm.- - ``local_optimization="dlt"``, the only choice for ``"affine"`` and line segments, refits each new best- model from its inliers: with the default ``lo_sample_size=32``, ``max_lo_iters`` randomized refits on- 32-inlier subsets followed by one full-inlier refit; ``lo_sample_size=None`` refits all inliers- iteratively. Affine models use a linear least-squares refit with free translation and return- a homogeneous matrix with bottom row ``[0, 0, 1]``; their minimal samples need three- non-collinear source points.+ - ``local_optimization="dlt"``, the only choice for ``"affine"``, ``"fundamental_affine"`` and line+ segments, refits each new best model from its inliers: with the default ``lo_sample_size=32``,+ ``max_lo_iters`` randomized refits on 32-inlier subsets followed by one full-inlier refit;+ ``lo_sample_size=None`` refits all inliers iteratively. Affine models use a linear least-squares refit+ with free translation and return a homogeneous matrix with bottom row ``[0, 0, 1]``; their minimal+ samples need three non-collinear source points. ``"fundamental_affine"`` solves its four-point samples+ and refits its inliers by orthogonal regression, the Gold Standard estimate (Hartley and Zisserman,+ algorithm 14.1). It counts no inliers for a matrix that (nearly) ignores one image, ``(a, b)`` or+ ``(c, d)`` near 0: that matrix is a line in the other image, which mismatches along a line fit. Each+ image's term of the distance must vary over the keypoints by at least two inlier thresholds (a standard+ deviation). A true geometry whose keypoints spread less across its epipolar lines, such as a 15 px patch+ at ``inl_th=2``, is rejected too, and RANSAC returns zeros.- ``degensac=True``, an opt-in for ``"fundamental"`` and ``"fundamental_7pt"`` with``local_optimization="lm"``, runs DEGENSAC (Chum, Werner and Matas, CVPR 2005). Every seven-point model thatsets a new record among the raw scores is tested for an H-degenerate sample: five or more of its seven@@ -277,8 +324,8 @@ class RANSAC(nn.Module):- A caller's autocast region does not change the estimate.Args:- model_type: "affine", "homography", "fundamental", "fundamental_7pt", "fundamental_8pt", "essential", or- "homography_from_linesegments".+ model_type: "affine", "homography", "fundamental", "fundamental_7pt", "fundamental_8pt",+ "fundamental_affine", "essential", or "homography_from_linesegments".inl_th: positive inlier threshold, in the units given above.batch_size: hypotheses generated and verified at once, or ``"auto"`` to pick the batches per call fromthe device and the number of correspondences (see :meth:`resolve_batch_size`).@@ -343,7 +390,7 @@ class RANSAC(nn.Module):Args:model_type: type of model to estimate: "affine", "homography", "fundamental", "fundamental_7pt",- "fundamental_8pt", "essential", "homography_from_linesegments".+ "fundamental_8pt", "fundamental_affine", "essential", "homography_from_linesegments".inl_th: inlier threshold; the class docstring gives its unit per ``model_type``.batch_size: number of generated samples at once, or ``"auto"`` (see :meth:`resolve_batch_size`).max_iter: maximum batches to generate. At most ``batch_size * max_iter`` minimal samples are drawn@@ -378,6 +425,7 @@ class RANSAC(nn.Module):"fundamental","fundamental_7pt","fundamental_8pt",+ "fundamental_affine","homography_from_linesegments","essential",]@@ -472,6 +520,12 @@ class RANSAC(nn.Module):self.minimal_sample_size = 7self.polisher_solver = find_fundamentalself.polisher_sample_size = 8+ elif model_type == "fundamental_affine":+ self.error_fn = sampson_epipolar_distance+ self.minimal_solver = run_affine+ self.minimal_sample_size = 4+ self.polisher_solver = run_affine+ self.polisher_sample_size = 4elif model_type == "essential":self.error_fn = sampson_epipolar_distanceself.minimal_solver = find_essential@@ -775,6 +829,9 @@ class RANSAC(nn.Module):inf = float("inf")errors = errors.nan_to_num(nan=inf, posinf=inf, neginf=inf)inl_mask = errors <= inl_th+ if self.model_type == "fundamental_affine":+ # A matrix that (nearly) ignores one image is a line in the other, which a line of mismatches fits.+ inl_mask = inl_mask & _affine_fundamental_ties_both_images(kp1, kp2, models, math.sqrt(inl_th))[:, None]score_ransac = inl_mask.sum(dim=1)if self.score_type == "msac":# Equivalent to minimizing the truncated squared loss (MSAC), normalizeddiff --git a/testing/two_view.py b/testing/two_view.pyindex c6f1eac38..04ee2298a 100644--- a/testing/two_view.py+++ b/testing/two_view.py@@ -92,3 +92,88 @@ def calibrated_two_view_scene()E = tx @ Rreturn {"x1": x1, "x2": x2, "E": E / E.norm(), "R": R, "t": t}+++def affine_two_view_scene(device: torch.device, dtype: torch.dtype) -> Dict[str, torch.Tensor]:+ """Two affine cameras and twelve correspondences, built in ``dtype`` from literals.++ The cameras ``x = M X + t`` (``M1 = [[1.2, 0.1, -0.3], [0.05, -1.1, 0.4]]``, ``t1 = [320, 240]``,+ ``M2 = [[0.9, 0.5, 0.2], [-0.3, -0.8, 0.7]]``, ``t2 = [300, 260]``) see twelve non-coplanar points spread over+ about 190 units. ``x1``/``x2`` are their exact pixel projections. ``F`` is the affine fundamental matrix of the+ cameras, with ``[a, b] M2 + [c, d] M1 = 0`` and ``e = -([a, b] t2 + [c, d] t1)`` for+ ``F = [[0, 0, a], [0, 0, b], [c, d, e]]``, scaled so that ``F[2, 2] = 1``. ``noise`` holds fixed offsets+ ``[dx1, dy1, dx2, dy2]`` and ``F_gold`` the minimizer of the summed squared geometric distances of the noisy+ correspondences, found by a general-purpose optimizer (``scipy.optimize.minimize`` on+ ``sum(r**2) / (a**2 + b**2 + c**2 + d**2)`` with ``r = a x2 + b y2 + c x1 + d y1 + e``). Generated by::++ X = np.array([[-80, 35, 12], [64, -71, 40], [-23, 90, -55], [47, 18, 73], [-95, -42, -8], [12, 58, 27],+ [81, -9, -66], [-56, -77, 51], [30, 44, -90], [-7, -60, 15], [93, 71, 5], [-38, 6, 84]], float)+ x1, x2 = X @ M1.T + t1, X @ M2.T + t2+ n = np.linalg.svd(np.c_[M2.T, M1.T])[2][-1] # [a, b, c, d]+ F = np.zeros((3, 3))+ F[0, 2], F[1, 2], F[2, 0], F[2, 1], F[2, 2] = *n, -(n[:2] @ t2 + n[2:] @ t1)+ F /= F[2, 2]+ x1n, x2n = x1 + noise[:, :2], x2 + noise[:, 2:]+ cost = lambda p: np.sum((p[0] * x2n[:, 0] + p[1] * x2n[:, 1] + p[2] * x1n[:, 0] + p[3] * x1n[:, 1] + p[4])+ ** 2) / (p[:4] ** 2).sum()+ p = minimize(cost, np.array([F[0, 2], F[1, 2], F[2, 0], F[2, 1], 1.0]) * 1.03, method="Nelder-Mead",+ options=dict(xatol=1e-14, fatol=1e-16, maxiter=200000, maxfev=200000)).x+ p = minimize(cost, p, method="BFGS", options=dict(gtol=1e-14)).x+ F_gold = np.zeros((3, 3))+ F_gold[0, 2], F_gold[1, 2], F_gold[2, 0], F_gold[2, 1], F_gold[2, 2] = p / p[4]+ """+ x1 = torch.tensor(+ [+ [+ [223.9, 202.3], [377.7, 337.3], [317.9, 117.85], [356.3, 251.75], [204.2, 278.25], [332.1, 187.6],+ [436.1, 227.55], [229.8, 342.3], [387.4, 157.1], [301.1, 311.65], [437.2, 168.55], [249.8, 265.1],+ ]+ ],+ device=device,+ dtype=dtype,+ ) # fmt: skip+ x2 = torch.tensor(+ [+ [+ [247.9, 264.4], [330.1, 325.6], [313.3, 156.4], [365.9, 282.6], [191.9, 316.5], [345.2, 228.9],+ [355.2, 196.7], [221.3, 374.1], [331.0, 152.8], [266.7, 320.6], [420.2, 178.8], [285.6, 325.4],+ ]+ ],+ device=device,+ dtype=dtype,+ ) # fmt: skip+ noise = torch.tensor(+ [+ [+ [0.31, -0.12, -0.25, 0.4], [-0.44, 0.07, 0.18, -0.29], [0.05, 0.36, -0.41, 0.12],+ [0.22, -0.33, 0.09, 0.27], [-0.15, 0.24, 0.38, -0.06], [0.42, 0.11, -0.2, -0.37],+ [-0.3, -0.27, 0.14, 0.33], [0.08, 0.45, -0.1, -0.21], [-0.36, -0.04, 0.29, 0.16],+ [0.19, -0.4, -0.32, 0.03], [-0.07, 0.2, 0.44, -0.39], [0.35, -0.18, -0.03, 0.25],+ ]+ ],+ device=device,+ dtype=dtype,+ ) # fmt: skip+ F = torch.tensor(+ [+ [+ [0.0, 0.0, 0.005968045112781947],+ [0.0, 0.0, -0.010385338345864652],+ [-0.007471804511278188, 0.009586466165413523, 1.0],+ ]+ ],+ device=device,+ dtype=dtype,+ )+ F_gold = torch.tensor(+ [+ [+ [0.0, 0.0, 0.0059939486403256526],+ [0.0, 0.0, -0.010334260027148275],+ [-0.007497428822287553, 0.009532615627438538, 1.0],+ ]+ ],+ device=device,+ dtype=dtype,+ )+ return {"x1": x1, "x2": x2, "noise": noise, "F": F, "F_gold": F_gold}diff --git a/tests/geometry/epipolar/test_fundamental.py b/tests/geometry/epipolar/test_fundamental.pyindex d5d3dd4c7..ace7ba7f7 100644--- a/tests/geometry/epipolar/test_fundamental.py+++ b/tests/geometry/epipolar/test_fundamental.py@@ -15,7 +15,7 @@# limitations under the License.#-from typing import Dict+from typing import Dict, Tupleimport pytestimport torch@@ -37,7 +37,7 @@ from kornia.geometry.solvers.homogeneous import _null_space_lufrom testing.base import DYNAMO_UNAVAILABLE_REASON, BaseTester, dynamo_is_available, script_uncachedfrom testing.geometry.create import create_random_fundamental_matrix, generate_two_view_random_scene-from testing.two_view import two_view_scene+from testing.two_view import affine_two_view_scene, two_view_sceneclass TestNormalizePoints(BaseTester):@@ -397,6 +397,205 @@ class TestFindFundamental(BaseTester):fundamental_module.run_8point(points, points, weights)+class TestFindFundamentalAffine(BaseTester):+ """``method="AFFINE"``: the affine fundamental matrix, Hartley and Zisserman's algorithm 14.1."""++ @staticmethod+ def _tol(dtype: torch.dtype) -> Dict[str, float]:+ return {"rtol": 1e-7, "atol": 1e-9} if dtype == torch.float64 else {"rtol": 5e-5, "atol": 1e-7}++ @staticmethod+ def _noisy(scene: Dict[str, torch.Tensor]) -> Tuple[torch.Tensor, torch.Tensor]:+ return scene["x1"] + scene["noise"][..., :2], scene["x2"] + scene["noise"][..., 2:]++ def test_exact_correspondences_give_the_cameras_matrix(self, device, dtype):+ _skip_half(dtype, _HALF_PIXELS)+ scene = affine_two_view_scene(device, dtype)+ F = epi.find_fundamental(scene["x1"], scene["x2"], method="AFFINE")+ assert F.shape == (1, 3, 3)+ assert F.dtype == dtype+ assert not F[:, :2, :2].any()+ self.assert_close(F, scene["F"], **self._tol(dtype))++ def test_four_correspondences_determine_it(self, device, dtype):+ _skip_half(dtype, _HALF_PIXELS)+ scene = affine_two_view_scene(device, dtype)+ F = epi.find_fundamental(scene["x1"][:, :4], scene["x2"][:, :4], method="affine")+ self.assert_close(F, scene["F"], **self._tol(dtype))++ def test_noisy_correspondences_give_the_gold_standard(self, device, dtype):+ _skip_half(dtype, _HALF_PIXELS)+ scene = affine_two_view_scene(device, dtype)+ x1, x2 = self._noisy(scene)+ F = epi.find_fundamental(x1, x2, method="AFFINE")+ self.assert_close(F, scene["F_gold"], **self._tol(dtype))+ # Control: the noise moves the minimizer away from the cameras' matrix by more than the tolerance.+ assert (scene["F_gold"] - scene["F"]).abs().max() > 1e-5++ def test_swapping_the_images_transposes_it(self, device, dtype):+ _skip_half(dtype, _HALF_PIXELS)+ x1, x2 = self._noisy(affine_two_view_scene(device, dtype))+ F = epi.find_fundamental(x1, x2, method="AFFINE")+ self.assert_close(epi.find_fundamental(x2, x1, method="AFFINE"), F.transpose(-2, -1), **self._tol(dtype))++ def test_weights_semantics(self, device, dtype):+ _skip_half(dtype, _HALF_PIXELS)+ x1, x2 = self._noisy(affine_two_view_scene(device, dtype))+ x2[0, 5] += torch.tensor([40.0, -25.0], device=device, dtype=dtype)+ keep = torch.arange(12, device=device) != 5+ w = torch.tensor([[0.5, 1.0, 2.0, 1.5, 0.75, 0.0, 3.0, 0.25, 1.0, 2.5, 0.6, 1.8]], device=device, dtype=dtype)+ F_w = epi.find_fundamental(x1, x2, w, method="AFFINE")+ # A weight of 0 drops the gross mismatch: the fit is the weighted fit of the other eleven.+ self.assert_close(F_w, epi.find_fundamental(x1[:, keep], x2[:, keep], w[:, keep], method="AFFINE"))+ # Only the ratios of the weights matter.+ self.assert_close(epi.find_fundamental(x1, x2, 7.0 * w, method="AFFINE"), F_w, **self._tol(dtype))+ # A negative weight counts as zero, to the bit.+ w_neg = w.clone()+ w_neg[0, 5] = -5.0+ assert torch.equal(epi.find_fundamental(x1, x2, w_neg, method="AFFINE"), F_w)+ # Control: equal weights keep the mismatch and give another matrix.+ assert (epi.find_fundamental(x1, x2, method="AFFINE") - F_w).abs().max() > 1e-4++ def test_half_precision_points_fit_in_float32(self, device, dtype):+ if dtype not in (torch.float16, torch.bfloat16):+ pytest.skip("a half-precision input check")+ scene = affine_two_view_scene(device, dtype)+ # Squared pixel coordinates in the hundreds overflow float16, so the fit runs in float32.+ F = epi.find_fundamental(scene["x1"], scene["x2"], method="AFFINE")+ assert F.dtype == dtype+ assert torch.isfinite(F).all()+ expected = epi.find_fundamental(scene["x1"].float(), scene["x2"].float(), method="AFFINE").to(dtype)+ assert torch.equal(F, expected)++ def test_float32_points_fit_in_float64(self, device):+ if device.type == "mps":+ pytest.skip("MPS has no float64; the fit runs in float32 there")+ x1, x2 = self._noisy(affine_two_view_scene(device, torch.float64))+ # Far from the origin, as in a large image: the float32 points round to 2e-3 px, and a float32 scatter matrix+ # of their centred coordinates would move F by more than that.+ x1, x2 = (x1 + 15000.0).float(), (x2 + 15000.0).float()+ F = epi.find_fundamental(x1, x2, method="AFFINE")+ assert F.dtype == torch.float32+ expected = epi.find_fundamental(x1.double(), x2.double(), method="AFFINE").float()+ self.assert_close(F, expected, rtol=1e-6, atol=0.0)++ @staticmethod+ def _cube_seen_from_front_and_side(device: torch.device, dtype: torch.dtype) -> Tuple[torch.Tensor, torch.Tensor]:+ # A 3 x 3 x 3 grid with spacing 100 seen by two orthographic cameras 90 degrees apart: x1 = (X, Y) and+ # x2 = (Z, Y + 10). The stacked correspondences (Z, Y + 10, X, Y) have scatter eigenvalues 0, v, v and 2v:+ # F = [[0, 0, 0], [0, 0, -0.1], [0, 0.1, 1]] is well defined, its eigenvalue 0 being simple, but eigh's+ # backward divides by the gap between the two equal ones.+ grid = torch.arange(3, device=device, dtype=dtype) * 100.0+ X = torch.cartesian_prod(grid, grid, grid)+ return X[None, :, :2], torch.stack([X[:, 2], X[:, 1] + 10.0], dim=-1)[None]++ def test_gradient_is_finite_when_two_other_eigenvalues_are_equal(self, device, dtype):+ _skip_half(dtype, _HALF_PIXELS)+ x1, x2 = self._cube_seen_from_front_and_side(device, dtype)+ weights = torch.ones(1, 27, device=device, dtype=dtype, requires_grad=True)+ x1.requires_grad_()+ x2.requires_grad_()+ F = epi.find_fundamental(x1, x2, weights, method="AFFINE")+ expected = torch.tensor([[[0.0, 0.0, 0.0], [0.0, 0.0, -0.1], [0.0, 0.1, 1.0]]], device=device, dtype=dtype)+ self.assert_close(F, expected, **self._tol(dtype))+ (F * torch.arange(9, device=device, dtype=dtype).view(3, 3)).sum().backward()+ for grad in (x1.grad, x2.grad, weights.grad):+ assert torch.isfinite(grad).all()++ def test_gradcheck_when_two_other_eigenvalues_are_equal(self, device):+ x1, x2 = self._cube_seen_from_front_and_side(device, torch.float64)+ weights = torch.ones(1, 27, device=device, dtype=torch.float64)++ def fit(p1: torch.Tensor, p2: torch.Tensor, w: torch.Tensor) -> torch.Tensor:+ return epi.find_fundamental(p1, p2, w, method="AFFINE")++ self.gradcheck(fit, (x1.requires_grad_(), x2.requires_grad_(), weights.requires_grad_()))++ def test_float16_result_is_finite_for_centred_points(self, device):+ # Points centred on the origin and scaled to unit size, as normalize_points leaves them, put the epipolar plane+ # near the origin: e is near 0, and scaling F to F[2, 2] = 1 put the other entries above float16's 65504.+ # Fifty points seen by the fixture's two affine cameras, with 0.3 px noise.+ generator = torch.Generator().manual_seed(4)+ X = torch.rand(50, 3, generator=generator) * 200.0 - 100.0+ M1 = torch.tensor([[1.2, 0.1, -0.3], [0.05, -1.1, 0.4]])+ M2 = torch.tensor([[0.9, 0.5, 0.2], [-0.3, -0.8, 0.7]])+ points1 = (X @ M1.T)[None] + 0.3 * torch.randn(1, 50, 2, generator=generator)+ points2 = (X @ M2.T)[None] + 0.3 * torch.randn(1, 50, 2, generator=generator)+ points1, points2 = ((x - x.mean(1, keepdim=True)) for x in (points1, points2))+ points1, points2 = (x / x.abs().max() for x in (points1, points2))+ points1, points2 = points1.half().to(device), points2.half().to(device)+ F = epi.find_fundamental(points1, points2, method="AFFINE")+ assert F.dtype == torch.float16+ assert torch.isfinite(F).all()+ # The same matrix up to scale: the largest entry is 1 instead of F[2, 2].+ expected = epi.find_fundamental(points1.float(), points2.float(), method="AFFINE")+ self.assert_close(F.float(), expected / expected.abs().amax(), rtol=0.0, atol=1e-3)+ assert F.abs().amax() == 1.0++ @pytest.mark.parametrize("image", [1, 2])+ def test_points_of_one_image_on_a_line_give_that_line(self, device, dtype, image):+ # Every four of the fixture's twelve correspondences, with one image's points moved onto the line y = 50. The+ # stacked correspondences then vary along neither that y nor, for some samples, a second direction whose+ # scatter eigenvalue is below 1e-7 of the largest. A float32 eigh of the scatter matrix (a half input's, or a+ # float32 input's on MPS) mixed the two; the singular vectors of the centred correspondences keep them apart.+ scene = affine_two_view_scene(device, dtype)+ x1, x2 = scene["x1"][0], scene["x2"][0]+ (x1 if image == 1 else x2)[:, 1] = 50.0+ samples = torch.combinations(torch.arange(12, device=device), 4)+ F = epi.find_fundamental(x1[samples], x2[samples], method="AFFINE")+ expected = torch.zeros(3, 3, device=device, dtype=dtype)+ expected[(2, 1) if image == 1 else (1, 2)] = -0.02+ expected[2, 2] = 1.0+ self.assert_close(F, expected.expand_as(F), rtol=0.0, atol=1e-4)++ def test_nonfinite_batch_element_gives_nan(self, device, dtype):+ _skip_half(dtype, _HALF_PIXELS)+ x1, x2 = self._noisy(affine_two_view_scene(device, dtype))+ x1, x2 = x1.repeat(3, 1, 1), x2.repeat(3, 1, 1)+ x1[1, 4] = float("nan")+ x2[2, 7, 0] = float("inf")+ x1.requires_grad_()+ F = epi.find_fundamental(x1, x2, method="AFFINE")+ # The finite element keeps its matrix and the others are NaN, rather than the whole batch raising.+ self.assert_close(F[0], epi.find_fundamental(x1[:1].detach(), x2[:1], method="AFFINE")[0])+ assert torch.isnan(F[1:]).all()+ F[0].sum().backward()+ assert torch.isfinite(x1.grad).all()++ def test_too_few_correspondences_raise(self, device, dtype):+ points = torch.rand(1, 3, 2, device=device, dtype=dtype)+ with pytest.raises(ShapeError, match=r"points1 and points2 must have shape \(B, N, 2\) with N >= 4, got"):+ epi.find_fundamental(points, points, method="AFFINE")+ points = torch.rand(1, 4, 2, device=device, dtype=dtype)+ with pytest.raises(ShapeError, match=r"weights must have shape \(B, N\) = \[1, 4\], got \["):+ epi.find_fundamental(points, points, torch.ones(1, 5, device=device, dtype=dtype), method="AFFINE")++ def test_gradcheck(self, device):+ x1, x2 = self._noisy(affine_two_view_scene(device, torch.float64))+ weights = torch.ones(1, 12, device=device, dtype=torch.float64)++ def fit(p1: torch.Tensor, p2: torch.Tensor, w: torch.Tensor) -> torch.Tensor:+ return epi.find_fundamental(p1, p2, w, method="AFFINE")++ self.gradcheck(fit, (x1.requires_grad_(), x2.requires_grad_(), weights.requires_grad_()))++ def test_zero_weight_backward_is_the_one_sided_derivative(self, device, dtype):+ _skip_half(dtype, _HALF_PIXELS)+ x1, x2 = self._noisy(affine_two_view_scene(device, dtype))+ grads = []+ for w3 in (0.0, 1e-6):+ weights = torch.ones(1, 12, device=device, dtype=dtype)+ weights[0, 3] = w3+ weights.requires_grad_()+ F_mat = epi.find_fundamental(x1, x2, weights, method="AFFINE")+ (F_mat * F_mat.detach().sign()).sum().backward()+ grads.append(weights.grad)+ assert torch.isfinite(grads[0]).all()+ assert grads[1][0, 3].abs() > 1e-6 # control: the dropped match moves F+ tol = {"rtol": 1e-3, "atol": 1e-8} if dtype == torch.float64 else {"rtol": 5e-2, "atol": 1e-6}+ self.assert_close(grads[0], grads[1], **tol)++class TestFundamentalAutocast(BaseTester):@pytest.mark.parametrize("method, num_points, weighted",@@ -802,6 +1001,7 @@ class TestGetClosestPointOnEpipolarLine(BaseTester):_NO_HALF_EIGH = "find_fundamental calls torch.linalg.eigh, which has no float16/bfloat16 kernel"+_HALF_PIXELS = "half precision rounds the fixture's pixel coordinates, 120 to 440, to steps of 0.125 to 0.25"_HALF_PIXEL_F = ("a pixel-unit F spans eight decades (entries down to ~1e-8): float16 flushes the small entries to zero and ""bfloat16's 8-bit mantissa cannot resolve the epipolar residual, which kornia evaluates in the input dtype"diff --git a/tests/geometry/test_ransac_fundamental_affine.py b/tests/geometry/test_ransac_fundamental_affine.pynew file mode 100644index 000000000..2fe15314c--- /dev/null+++ b/tests/geometry/test_ransac_fundamental_affine.py@@ -0,0 +1,201 @@+# LICENSE HEADER MANAGED BY add-license-header+#+# Copyright 2018 Kornia Team+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+#++from __future__ import annotations++from typing import Dict, Tuple++import pytest+import torch++from kornia.geometry import RANSAC+from kornia.geometry.epipolar import sampson_epipolar_distance++from testing.base import BaseTester+from testing.two_view import affine_two_view_scene++# Mismatches 29 to 156 px (geometric distance) away from the fixture's affine epipolar geometry.+_OUTLIERS1 = [+ [220.6, 256.8], [222.6, 304.0], [376.3, 227.3], [268.2, 355.7],+ [200.4, 433.6], [414.0, 340.4], [207.3, 369.7], [358.5, 423.6],+] # fmt: skip+_OUTLIERS2 = [+ [334.3, 283.9], [260.2, 186.7], [240.0, 268.9], [310.1, 217.3],+ [218.6, 222.2], [258.4, 327.9], [236.1, 170.9], [197.7, 294.9],+] # fmt: skip++# Twenty mismatches whose points in one image lie within 0.23 px of the line y = 0.5 x + 60, and in the other image are+# scattered: a rank-one matrix, with (c, d) = 0 or (a, b) = 0, takes all of them as inliers. Each is at least 8.9 px+# (geometric distance) from the fixture's epipolar geometry. Generated by+# rng = np.random.default_rng(4)+# on_line = rng.uniform(190, 430, 20)+# on_line = np.round(np.c_[on_line, 0.5 * on_line + 60 + rng.uniform(-0.25, 0.25, 20)], 1)+# scattered = np.round(rng.uniform([200, 150], [440, 380], (20, 2)), 1)+_ON_A_LINE = [+ [416.3, 268.3], [312.7, 216.6], [424.3, 272.2], [209.4, 164.5], [335.8, 227.9],+ [280.4, 200.2], [382.5, 251.2], [231.9, 176.2], [399.2, 259.5], [320.5, 220.1],+ [406.5, 263.3], [304.5, 212.3], [293.3, 206.9], [379.3, 249.7], [426.2, 273.0],+ [278.7, 199.6], [422.5, 271.3], [413.0, 266.6], [232.6, 176.3], [336.1, 227.9],+] # fmt: skip+_SCATTERED = [+ [366.2, 327.2], [245.8, 255.8], [286.8, 189.3], [253.1, 371.4], [412.2, 237.9],+ [377.5, 159.7], [419.6, 273.5], [396.9, 213.5], [290.3, 230.1], [433.4, 248.8],+ [319.9, 369.9], [416.8, 240.9], [273.8, 335.6], [225.8, 240.2], [410.3, 361.1],+ [256.0, 168.1], [389.1, 293.0], [257.0, 266.4], [217.1, 311.0], [326.4, 312.1],+] # fmt: skip+++def _with_outliers(device: torch.device, dtype: torch.dtype) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:+ """The twelve exact correspondences of the affine fixture followed by eight mismatches, and the true F."""+ scene = affine_two_view_scene(device, dtype)+ x1 = torch.cat([scene["x1"][0], torch.tensor(_OUTLIERS1, device=device, dtype=dtype)])+ x2 = torch.cat([scene["x2"][0], torch.tensor(_OUTLIERS2, device=device, dtype=dtype)])+ return x1, x2, scene["F"][0]+++def _tol(dtype: torch.dtype) -> Dict[str, float]:+ return {"rtol": 1e-7, "atol": 1e-9} if dtype == torch.float64 else {"rtol": 5e-5, "atol": 1e-7}+++def _skip_half(dtype: torch.dtype) -> None:+ if dtype in (torch.float16, torch.bfloat16):+ pytest.skip("half precision rounds the fixture's pixel coordinates to steps of 0.125 to 0.25")+++class TestRANSACFundamentalAffine(BaseTester):+ @pytest.mark.parametrize("score_type", ["ransac", "msac"])+ @pytest.mark.parametrize("lo_sample_size", [None, 8])+ def test_inliers_and_model_with_outliers(self, device, dtype, score_type, lo_sample_size):+ _skip_half(dtype)+ x1, x2, expected = _with_outliers(device, dtype)+ estimator = RANSAC(+ "fundamental_affine",+ inl_th=1.0,+ batch_size=32,+ max_samples=256,+ score_type=score_type,+ lo_sample_size=lo_sample_size,+ seed=0,+ )+ model, inliers = estimator(x1, x2)+ assert model.dtype == dtype+ assert model.device == device+ assert torch.equal(inliers, torch.arange(20, device=device) < 12)+ assert not model[:2, :2].any()+ self.assert_close(model, expected, **_tol(dtype))++ @pytest.mark.parametrize("lo_sample_size", [None, 8])+ def test_refit_gives_the_gold_standard_of_the_inliers(self, device, dtype, lo_sample_size):+ _skip_half(dtype)+ scene = affine_two_view_scene(device, dtype)+ x1, x2, _ = _with_outliers(device, dtype)+ x1[:12] += scene["noise"][0, :, :2]+ x2[:12] += scene["noise"][0, :, 2:]+ estimator = RANSAC(+ "fundamental_affine", inl_th=1.0, batch_size=32, max_samples=256, lo_sample_size=lo_sample_size, seed=0+ )+ model, inliers = estimator(x1, x2)+ assert torch.equal(inliers, torch.arange(20, device=device) < 12)+ # The refit on the twelve noisy inliers is their Gold Standard fit; a four-point model is 5e-4 away from it.+ self.assert_close(model, scene["F_gold"][0], **_tol(dtype))++ @pytest.mark.parametrize("image", [1, 2])+ def test_points_on_a_line_in_one_image_give_no_model(self, device, dtype, image):+ x1, x2, _ = _with_outliers(device, dtype)+ # With one image's points all on the line y = 0, the rank-one matrix of that line, (a, b) = (0, 1) or+ # (c, d) = (0, 1), fits all twenty correspondences exactly: no epipolar geometry. A sample that is nearly+ # coplanar besides can be fitted by another matrix, which a few correspondences fit.+ (x1 if image == 1 else x2)[:, 1] = 0.0+ _, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=256, seed=0)(x1, x2)+ assert int(inliers.sum()) < 20++ @pytest.mark.parametrize("image", [1, 2])+ def test_mismatches_along_a_line_do_not_outvote_the_true_geometry(self, device, dtype, image):+ _skip_half(dtype)+ x1, x2, expected = _with_outliers(device, dtype)+ on_a_line = torch.tensor(_ON_A_LINE, device=device, dtype=dtype)+ scattered = torch.tensor(_SCATTERED, device=device, dtype=dtype)+ x1 = torch.cat([x1, on_a_line if image == 1 else scattered])+ x2 = torch.cat([x2, scattered if image == 1 else on_a_line])+ # The twenty line mismatches fit a near-rank-one matrix (their noise off the line makes (c, d) or (a, b) small,+ # not zero) better than the twelve true correspondences fit theirs. A sample of four true ones has probability+ # (12 / 40)^4 = 0.8%: 2048 samples miss it with probability 6e-8.+ model, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=2048, seed=0)(x1, x2)+ assert torch.equal(inliers, torch.arange(40, device=device) < 12)+ self.assert_close(model, expected, **_tol(dtype))++ def test_true_geometry_that_spreads_little_is_kept(self, device, dtype):+ _skip_half(dtype)+ x1, x2, F = _with_outliers(device, dtype)+ # View 2 shrunk 20 times about (300, 260), as a far smaller-scale image would be. Over the keypoints, the+ # true matrix's two terms of the distance vary by 3.7 and 4.9 px (standard deviations): a check at four+ # thresholds or more would reject it. The mismatches stay at least 2.1 px from it.+ scale, cx, cy = 0.05, 300.0, 260.0+ centre = torch.tensor([cx, cy], device=device, dtype=dtype)+ x2 = centre + scale * (x2 - centre)+ # The fixture's F applies to the unshrunk points, T x2.+ T = torch.tensor(+ [[1 / scale, 0.0, cx * (1 - 1 / scale)], [0.0, 1 / scale, cy * (1 - 1 / scale)], [0.0, 0.0, 1.0]],+ device=device,+ dtype=dtype,+ )+ expected = T.mT @ F+ expected = expected / expected[2, 2]+ model, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=256, seed=0)(x1, x2)+ assert torch.equal(inliers, torch.arange(20, device=device) < 12)+ self.assert_close(model, expected, **_tol(dtype))++ def test_nonfinite_correspondences_are_outliers(self, device, dtype):+ _skip_half(dtype)+ x1, x2, expected = _with_outliers(device, dtype)+ # Copies of five true correspondences, three with a NaN point in image 1 and two with an infinite coordinate+ # in image 2. Most samples draw one of them, and the keypoints' spread includes them.+ x1, x2 = torch.cat([x1, x1[:5]]), torch.cat([x2, x2[:5]])+ x1[20:23] = float("nan")+ x2[23:, 0] = float("inf")+ model, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=256, seed=0)(x1, x2)+ assert torch.equal(inliers, torch.arange(25, device=device) < 12)+ self.assert_close(model, expected, **_tol(dtype))++ def test_threshold_is_the_geometric_distance(self, device, dtype):+ _skip_half(dtype)+ x1, x2, _ = _with_outliers(device, dtype)+ # Moves of x2 along the epipolar lines' normal (a, b) by which the correspondence is 0.75 px and 1.5 px from+ # the nearest pair of corresponding points (the four-dimensional distance, which the Sampson distance of an+ # affine F equals): r = a dx + b dy = d * |(a, b, c, d)|.+ x1, x2 = torch.cat([x1, x1[:2]]), torch.cat([x2, x2[:2]])+ x2[-2:] += torch.tensor([[0.532378098326968, -0.9264217301595274], [1.064756196653936, -1.852843460319055]],+ device=device, dtype=dtype) # fmt: skip+ model, inliers = RANSAC("fundamental_affine", inl_th=1.0, max_samples=256, seed=0)(x1, x2)+ assert inliers[-2]+ assert not inliers[-1]+ classified = sampson_epipolar_distance(x1[None], x2[None], model[None])[0] <= 1.0+ assert torch.equal(inliers, classified)++ def test_minimal_sample_is_four(self, device, dtype):+ x1, x2, _ = _with_outliers(device, dtype)+ with pytest.raises(ValueError, match=r"at least \[4, 2\]"):+ RANSAC("fundamental_affine").validate_inputs(x1[:3], x2[:3])+ # Four correspondences fit a model exactly, which is no evidence of a consensus.+ model, inliers = RANSAC("fundamental_affine", max_samples=16, seed=0)(x1[:4], x2[:4])+ assert not model.any()+ assert not inliers.any()++ @pytest.mark.parametrize("kwargs", [{"local_optimization": "lm"}, {"degensac": True}, {"compile": True}])+ def test_unsupported_options_raise(self, kwargs):+ with pytest.raises(ValueError):+ RANSAC("fundamental_affine", **kwargs)