π Geometric Computer Vision Library for Spatial AI
Python artificial-intelligencecomputer-visiondeep-learninghacktoberfestimage-processingmachine-learningneural-networkpythonpytorchroboticsspatial-ai
ONNX support
============
.. meta::
:description: How to export Kornia operators and models to ONNX and chain them with ONNXSequential: installation, combining models, input/output mapping, export to a single file, and running on CUDA with ONNX Runtime execution providers.
:class:`~kornia.onnx.sequential.ONNXSequential` combines several ONNX models into a single computational graph and
runs it with ONNX Runtime. It is useful when you have pre-exported operators or models, from Kornia's
``kornia/ONNX_models`` Hugging Face repository or your own, that should run one after another as a pipeline.
- **Model chaining**: combine multiple ONNX models into one graph, with control over how the outputs of one model
feed the inputs of the next.
- **Export**: save the combined model as a single ONNX file.
- **PyTorch-like interface**: call the sequence directly, like an ``nn.Sequential``, on numpy arrays or CPU torch
tensors of a dtype numpy has (not bfloat16 or float8). It returns numpy arrays.
- **Execution providers**: run on CPU, CUDA, TensorRT or OpenVINO through ONNX Runtime, with control over the
session options.
Which operators Kornia promises to export, and how continuous integration checks it, is stated in
:ref:`What Kornia promises <export-support-promise>`. The survey on the same page records whether each operator, model
and augmentation exported in one dated measurement, with the cause of each failure: for anything the promise does not
cover, export your own pipeline and compare it with eager before relying on it.
Quickstart
----------
1. **Install ONNX and ONNX Runtime**
.. code-block:: bash
pip install "kornia[onnx]"
2. **Port your own kornia module**
Any kornia operator or ``nn.Sequential`` of operators is a regular ``nn.Module``, so
``torch.onnx.export`` turns it into an ONNX file that :class:`~kornia.onnx.sequential.ONNXSequential`
can load, chain and re-export. Use the ``torch.export``-based exporter (``dynamo=True``). It is the
default of ``torch.onnx.export`` from PyTorch 2.9, which deprecates the TorchScript-based one, and it is
the exporter :ref:`Kornia's ONNX promise <export-support-promise>` covers. Passing ``dynamo=True`` explicitly
makes the same call work on PyTorch 2.5 to 2.8, where the TorchScript-based exporter is still the default:
.. code-block:: python
import torch
import kornia
batch, height, width = torch.export.Dim("B"), torch.export.Dim("H"), torch.export.Dim("W")
torch.onnx.export(
kornia.color.RgbToGrayscale(),
(torch.rand(2, 3, 256, 512),),
"gray.onnx",
input_names=["input"],
output_names=["output"],
dynamic_shapes=({0: batch, 2: height, 3: width},),
dynamo=True,
)
``dynamic_shapes`` keeps the batch, height and width dynamic in the exported graph. Give the example
input a batch of at least 2: PyTorch 2.5 fixes a dimension of size 1 to that size.
The file plugs into a sequence like any other model. A fresh export has a newer ONNX IR version and
opset than the pre-exported Hub operators, so pass ``auto_ir_version_conversion=True`` to bring every
graph to one version before they are merged; without it, the merge fails with ``IR version mismatch``:
.. code-block:: python
from kornia.onnx import ONNXSequential
onnx_seq = ONNXSequential(
"gray.onnx",
"hf://operators/kornia.geometry.transform.affwarp.Resize_512x512",
auto_ir_version_conversion=True,
)
3. **Combine ONNX models**
Initialize :class:`~kornia.onnx.sequential.ONNXSequential` with ONNX models or file paths. The models are chained
and an optimized inference session is created.
.. code-block:: python
import numpy as np
from kornia.onnx import ONNXSequential
# Two operators from the kornia/ONNX_models Hugging Face repo
onnx_seq = ONNXSequential(
"hf://operators/kornia.color.gray.RgbToGrayscale",
"hf://operators/kornia.geometry.transform.affwarp.Resize_512x512"
)
input_data = np.random.randn(1, 3, 256, 512).astype(np.float32)
outputs = onnx_seq(input_data)
print(outputs)
.. note::
By default each ONNX model is assumed to have one input node named ``"input"`` and one output node named
``"output"``. For other models, pass an ``io_maps`` argument.
4. **Map inputs and outputs between models**
``io_maps`` holds one list of ``(output, input)`` name pairs per joint between consecutive models:
``io_maps[0]`` connects the first model to the second, ``io_maps[1]`` the second to the third, and so on.
.. code-block:: python
# one joint, which feeds two outputs of model1 to two inputs of model2
io_maps = [[("model1_output_0", "model2_input_0"), ("model1_output_1", "model2_input_1")]]
onnx_seq = ONNXSequential("model1.onnx", "model2.onnx", io_maps=io_maps)
The merged graph does not check a later model's declared input types: ``ONNXSequential`` warns when the tensor
element types of an output and the input it feeds differ at a joint.
5. **Export the combined model**
.. code-block:: python
onnx_seq.export("combined_model.onnx")
6. **Pick an execution provider**
.. code-block:: python
onnx_seq = ONNXSequential(
"hf://operators/kornia.geometry.transform.flips.Hflip",
# a local model works too: "YOUR_OWN_MODEL.onnx", or a loaded onnx.ModelProto
"hf://models/kornia.models.detection.rtdetr_r18vd_640x640",
providers=['CUDAExecutionProvider']
)
outputs = onnx_seq(input_data)
Running on CUDA
---------------
CUDA execution needs ``onnxruntime-gpu``; for other CUDA versions see
https://github.com/microsoft/onnxruntime/issues/21769#issuecomment-2295342211. For example, to install
``onnxruntime-gpu==1.19.2`` for CUDA 11.x:
.. code-block:: console
pip install onnxruntime-gpu==1.19.2 --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-11/pypi/simple/
Then move a sequence between devices with ``as_cuda()`` / ``as_cpu()``:
.. code-block:: python
import time
import kornia
from kornia.onnx import ONNXSequential
onnx_seq = ONNXSequential(
"hf://operators/kornia.geometry.transform.flips.Hflip",
"hf://models/kornia.models.detection.rtdetr_r18vd_640x640", # Or you may use "YOUR_OWN_MODEL.onnx"
)
inp = kornia.io.get_sample_images()[0].numpy()[None]
onnx_seq.as_cuda()
onnx_seq(inp) # GPU warm-up
start_time = time.time()
onnx_seq(inp)
print("--- GPU %s seconds ---" % (time.time() - start_time))
onnx_seq.as_cpu()
start_time = time.time()
onnx_seq(inp)
print("--- CPU %s seconds ---" % (time.time() - start_time))
On an onnxruntime build without CUDA, ``as_cuda()`` only warns and the sequence keeps running on the CPU;
``get_session().get_providers()`` shows which provider runs.
A typical result:
.. code-block:: console
--- GPU 0.014804363250732422 seconds ---
--- CPU 0.17681646347045898 seconds ---
Frequently asked questions
--------------------------
**Can I chain models from different sources?**
Yes. Models can come from ONNX files, ``onnx.ModelProto`` objects or ``hf://`` references;
:class:`~kornia.onnx.sequential.ONNXSequential` merges their graphs. When their ONNX IR versions or opsets
differ, as a fresh export and a Hub operator do, pass ``auto_ir_version_conversion=True``.
**What if the input/output names of two models do not match?**
Use the ``io_maps`` argument to say which output feeds which input: one list of ``(output, input)`` name pairs per
joint, as in step 4.
**Can I use custom ONNX Runtime session options?**
Yes. Pass them as ``session_options=`` to the constructor of
:class:`~kornia.onnx.sequential.ONNXSequential`, or replace the session of an existing object with
``seq.set_session(seq.create_session(session_options=...))``. ``create_session`` alone returns a new session and leaves
the object on the one it already has.
**Which operators export?**
The promised ones: :ref:`What Kornia promises <export-support-promise>` counts them per package, and the survey on the
same page marks them *promised*. The survey lists every surveyed operator with its ONNX export, ``torch.export`` and
``torch.compile`` outcome in one dated measurement, searchable by name and filterable by result; treat an entry that is
not promised as a starting point, not a guarantee.
**Where are the pre-exported Kornia operators?**
In the `kornia/ONNX_models <https://huggingface.co/kornia/ONNX_models>`_ Hugging Face repository; load them with
``"hf://operators/<name>"`` and ``"hf://models/<name>"`` or with :class:`~kornia.onnx.utils.ONNXLoader`.
See the :doc:`kornia.onnx </onnx>` API reference for the full class documentation.