π Geometric Computer Vision Library for Spatial AI
Python artificial-intelligencecomputer-visiondeep-learninghacktoberfestimage-processingmachine-learningneural-networkpythonpytorchroboticsspatial-ai
kornia.models
=============
.. meta::
:description: The Kornia models overview provides detailed information about key built-in models for computer vision tasks, including real-time object detection (RT-DETR), edge detection (DexiNed), super-resolution (Real-ESRGAN), semantic segmentation (a wrapper for segmentation_models_pytorch networks) and the Kimi-VL and Qwen2.5-VL vision encoders. It offers comprehensive documentation on each model, including methods, parameters, and example usage to streamline the integration of these models into computer vision workflows.
Builders for Kornia's ready-to-use models (object detection, edge detection, super-resolution, semantic segmentation
and Kimi-VL), each returning a regular ``nn.Module``. The object detection, edge detection and super-resolution
builders load pretrained weights by default and ``KimiVLBuilder.from_pretrained_hf`` loads them always, downloading
them on first use. ``SegmentationModelsBuilder`` downloads nothing: it wraps a network you built yourself. The page
also covers the vision tower of Qwen2.5-VL, whose weights kornia does not host: ``Qwen25VisionTransformer.from_pretrained``
reads a checkpoint you downloaded yourself (see :ref:`Qwen25VisionTransformer`). For the papers behind the models, see
the :doc:`Models </models/index>` section. Every model here accepts a batched ``(B, 3, H, W)`` float image in
``[0, 1]``, except Kimi-VL, Qwen2.5-VL and a ``SegmentationModelsBuilder`` model built without ``preproc_params``,
which passes its input to the network unchanged. Kimi-VL takes that image normalized with mean = std = 0.5 per channel
(``2 * x - 1``), as its checkpoint's ``preprocessor_config.json`` specifies, with ``H`` and ``W`` multiples of 28 (see
``kornia.models.kimi_vl.KimiVLModel.forward``). The Qwen2.5-VL vision tower takes it normalized with the CLIP mean and
std. :func:`kornia.io.get_sample_images` provides a couple of sample images for quick experiments.
.. _RTDETRDetectorBuilder:
RTDETRDetectorBuilder
---------------------
The `RTDETRDetectorBuilder` class is a builder for constructing a detection model based on the RT-DETR architecture, which is designed for real-time object detection. It is capable of detecting multiple objects within an image and provides efficient inference suitable for real-world applications.
**Key Methods:**
- `build`: Constructs and returns an instance of the RTDETR detection model.
- `visualize`: Draws the detected boxes on the input images.
.. autoclass:: kornia.contrib.object_detection.RTDETRDetectorBuilder
:members:
:undoc-members:
:show-inheritance:
.. rubric:: Example
The following code demonstrates how to use `RTDETRDetectorBuilder` to detect objects in an image:
.. code-block:: python
import kornia
from kornia.contrib.object_detection import RTDETRDetectorBuilder
image = kornia.io.get_sample_images()[0][None]
model = RTDETRDetectorBuilder.build()
detections = model(image) # list of (D, 6) tensors: class id, score, x, y, w, h
drawn = model.visualize(image, detections) # the boxes drawn on the image
.. _EdgeDetectorBuilder:
EdgeDetectorBuilder
-------------------
The `EdgeDetectorBuilder` class implements a state-of-the-art edge detection model based on DexiNed, which excels at detecting fine-grained edges in images. This model is well-suited for tasks like medical imaging, object contour detection, and more.
**Key Methods:**
- `build`: Builds and returns an instance of the DexiNed edge detection model.
- `visualize`: Returns the edge maps as images for further processing or display.
.. autoclass:: kornia.contrib.edge_detection.EdgeDetectorBuilder
:members:
:undoc-members:
:show-inheritance:
.. rubric:: Example
The following code shows how to use the `EdgeDetectorBuilder` to detect edges in an image:
.. code-block:: python
import kornia
from kornia.contrib.edge_detection import EdgeDetectorBuilder
image = kornia.io.get_sample_images()[0][None]
model = EdgeDetectorBuilder.build()
edges = model(image) # list with one (1, 1, H, W) edge map per image
.. _RRDBNet:
RRDBNet
-------
The `RRDBNet` class is the Residual-in-Residual Dense Block generator behind ESRGAN and Real-ESRGAN.
It is a plain ``nn.Module`` that upsamples a batched ``(B, 3, H, W)`` image by a factor of 1, 2 or 4,
and its module and parameter names match the reference implementation, so the published Real-ESRGAN
checkpoints load with ``strict=True``. ``kornia.contrib.super_resolution.RRDBNetBuilder`` configures
it for the released Real-ESRGAN variants and downloads their weights, and returns a
``SuperResolution`` wrapper ready for inference::
from kornia.contrib.super_resolution import RRDBNetBuilder
model = RRDBNetBuilder.build("RealESRNet_x4plus")
upscaled = model(images) # (B, 3, H, W) -> (B, 3, 4H, 4W)
The architecture is vendored from `BasicSR <https://github.com/XPixelGroup/BasicSR>`_ (Apache-2.0,
Copyright 2018-2022 BasicSR Authors); no extra package is required to use it.
.. autoclass:: kornia.models.RRDBNet
:members:
:undoc-members:
:show-inheritance:
.. rubric:: Example
The following code upsamples an image by a factor of 4 with a randomly initialized generator:
.. code-block:: python
import torch
from kornia.models import RRDBNet
model = RRDBNet(num_in_ch=3, num_out_ch=3, scale=4, num_feat=64, num_block=23).eval()
upsampled = model(torch.rand(1, 3, 32, 32)) # (1, 3, 128, 128)
.. _SegmentationModels:
SegmentationModelsBuilder
-------------------------
The `SegmentationModelsBuilder` class wraps a segmentation network you have already built -- typically one from
`segmentation_models_pytorch <https://github.com/qubvel-org/segmentation_models.pytorch>`_ (smp), but any
``nn.Module`` mapping ``(B, 3, H, W)`` to ``(B, C, H, W)`` works -- in a
:class:`~kornia.models.segmentation.SemanticSegmentation` container, prepending the ONNX-friendly preprocessing
(BGR-to-RGB, range rescaling, mean/std normalization) that the encoder's pretrained weights expect. Kornia does not
import smp; you build the network and fetch its preprocessing parameters yourself.
**Key Methods:**
- `build`: Wraps a constructed segmentation network and its encoder's preprocessing parameters.
- `get_preprocessing_pipeline`: Turns a preprocessing-parameter dictionary into an
:class:`~kornia.augmentation.container.ImageSequential`.
**Main parameters of** `build`:
- `model`: (nn.Module) The segmentation network.
- `preproc_params`: (dict | None) The encoder's preprocessing parameters, in the shape returned by
``smp.encoders.get_preprocessing_params(encoder_name)``: ``input_space``, ``input_range``, ``mean`` and ``std``.
``None`` means the input is fed to the network unchanged.
- `name`: (str) The name of the wrapped model, used by ``save``.
.. autoclass:: kornia.models.segmentation.segmentation_models.SegmentationModelsBuilder
:members:
:undoc-members:
:show-inheritance:
.. rubric:: Example
Here's an example of how to use `SegmentationModelsBuilder` with an smp UNet for two-class segmentation:
.. code-block:: python
import kornia
import segmentation_models_pytorch as smp
from kornia.models.segmentation import SegmentationModelsBuilder
net = smp.Unet(encoder_name="resnet34", encoder_weights="imagenet", classes=2, activation="softmax2d")
params = smp.encoders.get_preprocessing_params("resnet34")
model = SegmentationModelsBuilder.build(net, params, name="Unet_resnet34")
input_tensor = kornia.io.get_sample_images()[0][None]
segmented_output = model(input_tensor)
print(segmented_output.shape) # (1, 2, H, W)
The softmax head is what :meth:`~kornia.models.segmentation.SemanticSegmentation.visualize` needs: it
colours each pixel by its most probable class and raises on raw logits.
.. autoclass:: kornia.models.segmentation.SemanticSegmentation
:members: forward, visualize
.. _KimiVLBuilder:
KimiVLBuilder
-------------
The `KimiVLBuilder` class constructs Kimi-VL models from a configuration or downloads pretrained weights. Pretrained
loading currently supports only the converted Kimi-VL-A3B-Instruct vision encoder and projector checkpoint.
**Key Methods:**
- `from_config`: Constructs a randomly initialized Kimi-VL model from a `KimiVLConfig`.
- `from_pretrained_hf`: Downloads and strictly loads the supported pretrained checkpoint.
.. autoclass:: kornia.models.kimi_vl.KimiVLBuilder
:members:
:undoc-members:
:show-inheritance:
.. rubric:: Example
The following code loads the supported pretrained Kimi-VL vision model:
.. code-block:: python
from kornia.models.kimi_vl import KimiVLBuilder
model = KimiVLBuilder.from_pretrained_hf().eval()
.. _Qwen25VisionTransformer:
Qwen25VisionTransformer
-----------------------
The `Qwen25VisionTransformer` class is the vision tower of Qwen2.5-VL. It takes a batched ``(B, 3, H, W)`` image,
with ``H`` and ``W`` multiples of 28, already normalized with the CLIP mean and std the checkpoint's
``preprocessor_config.json`` names, and returns the merged vision tokens. kornia does not host the weights:
`from_pretrained` reads the ``visual.*`` tensors of a Qwen2.5-VL checkpoint you downloaded yourself.
.. autoclass:: kornia.models.qwen25.Qwen25VisionTransformer
:members: forward, from_pretrained
.. rubric:: Example
.. code-block:: python
from kornia.models.qwen25 import Qwen25VisionTransformer
model = Qwen25VisionTransformer.from_pretrained("path/to/Qwen2.5-VL-3B-Instruct")
.. autoclass:: kornia.models.qwen25.Qwen25VisionConfig
----
.. note::
This documentation provides detailed information about each model class, its methods, and usage examples. For further details on individual methods and arguments, refer to the respective code documentation.