Skip to content

armgit / nota

Public Imported from GitHub
0

Nota — a free DAW for macOS, Windows and Linux. No subscription, no account — just open it and make music.

(docs) AI models: changelog, feature list and third-party notices

Egor Khindikaynen4508ce557442
diff --git a/CHANGELOG.md b/CHANGELOG.md
index eab27208..ee6f8b85 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -40,6 +40,24 @@ Entry categories: `Added`, `Changed`, `Fixed`, `Removed`.
checks there is room for a pack before downloading it. On both sides the filters and
search stay at the top while the list scrolls.
+- **Separate Stems:** right-click an audio clip → **Separate Stems…** splits it into
+ **Drums, Bass, Other and Vocals** — a group of four tracks right under the clip's track,
+ each clip lined up with the original (warp, reverse, fades and ADSR included). The original
+ clip is switched off, not deleted, and the whole thing undoes in one step. It runs on this
+ computer with Meta's Demucs model: a three-minute song takes well under a minute, with
+ Cancel.
+- **Better Convert Melody and Convert Harmony:** with Spotify's basic-pitch model installed
+ they hear every note, chords included, with velocities that follow the playing. Without it
+ they work as before, and the Convert menu offers the model.
+- **AI Models** in **Settings → Downloads**, beside Plug-ins and Sample Packs: install or
+ remove the two models. The first one also brings the AI runtime they share (ONNX Runtime);
+ removing the last one removes it. Everything is downloaded from its authors, checked
+ against a pinned checksum, and runs offline — no audio leaves the computer. Separate Stems
+ offers the download the first time you use it.
+- **Over MCP**, an AI can list the installed models, separate a clip into stems and
+ transcribe a clip into a MIDI track (`list_ai_models`, `separate_stems`,
+ `convert_audio_to_midi`). Installing a model stays with you.
+
### Changed
- **Faster saves of projects with audio:** a save now writes only new recordings and
imports instead of every sample again, and the same audio used in several places is
diff --git a/FEATURES.md b/FEATURES.md
index fdc9c3d3..89617415 100644
--- a/FEATURES.md
+++ b/FEATURES.md
@@ -206,7 +206,10 @@ the mixer opens as a separate floating window (**View → Mixer**, ⌘M).
rest centred (a deadzone keeps a worn stick from drifting a mapped parameter);
triggers are unipolar and rest at zero.
- **Highlighting** of the pressed key on the roll's keyboard and as a bar along its row.
-- **Audio→MIDI** (right-click an audio clip → Convert):
+- **Audio→MIDI** (right-click an audio clip → Convert). With the **basic-pitch** AI model
+ installed (Settings → Downloads → AI Models), Melody and Harmony use it instead: every note,
+ chords included, with velocities from the playing; Melody keeps the strongest note where notes
+ overlap. Without it they use the DSP below, and the Convert menu offers the model.
- **Convert Melody** — monophonic pitch detection (YIN) → a new Nota Synth track.
- **Convert Harmony** — polyphonic (STFT + spectral peak picking) → chords: overtones,
chorus shimmer and slight detune don't add stray notes, held chords stay whole, and
@@ -243,6 +246,13 @@ the mixer opens as a separate floating window (**View → Mixer**, ⌘M).
Editing in either place updates the other, each edit is one undo step, split keeps the
attack on the left piece and the release on the right, and the shape is saved with the
project.
+- **Separate Stems** (right-click an audio clip): the **htdemucs** AI model (Demucs v4)
+ splits the clip into Drums, Bass, Other and Vocals — a group of four audio tracks under the
+ clip's track. Each stem clip copies the original's shape exactly (region, warp markers, trim,
+ reverse, envelopes, ADSR) over a stem file as long as the source, so it lines up sample for
+ sample; the original clip is switched off. One undo step; runs locally (about a third of the
+ clip's length on Apple Silicon) behind a progress dialog with Cancel; clips up to 10 minutes.
+ The first use offers to download the model.
- **Warp / time-stretch**: **Complex** and **Complex Pro** modes (the latter with formant
preservation, correct even when the sample rate and the device rate differ), plus
transient detection. A grid-snap toggle governs trimming a warped clip.
@@ -664,6 +674,10 @@ and user presets, automation, persistence and cloning.
an install runs. A project that needs a missing registry plugin offers to install it and
reopens with its saved settings. Plugins are found even when installed in a different
folder than on the computer that saved the project.
+- **AI Models** (Settings → Downloads, third side): **htdemucs** (Separate Stems, MIT) and
+ **basic-pitch** (Convert to MIDI, Apache-2.0), plus the **ONNX Runtime** 1.23.2 library they
+ run on, which comes with the first model and goes with the last. Every asset is pinned to a
+ URL, size and sha256 in the app; models install into Nota's data folder and run offline.
- **Plugin device card** in two sizes: the small card shows vendor, format, parameter count
and latency with Open editor and Save preset; the full card adds every parameter as a knob
(with filter and pages, the plugin's own value text, double-click for its default) and a
@@ -800,7 +814,7 @@ and user presets, automation, persistence and cloning.
list updates on hot-plug, with a live activity indicator beside each pad that names the
note played or the control driven.
- **Plug-ins**: scan paths, Rescan.
-- **Downloads**: the plugin registry installer (see [Plugin hosting](#plugin-hosting)).
+- **Downloads**: Plug-ins, Sample Packs and AI Models (see [Plugin hosting](#plugin-hosting)).
- **Library**: library folders.
- **Appearance**: **Theme** — Ember Graphite (dark), Ember Paper (light) or System, which
follows the OS appearance and switches with it; the choice applies live and is
@@ -850,6 +864,9 @@ Coverage:
- **Audio Effect Rack**: snapshot, Parallel/Series/Select mode, dry/wet, volume,
chain-select, add and remove chains, parameters, 8 macros.
- **Export** the master or stems to WAV (pcm16/pcm24/float32).
+- **AI models**: `list_ai_models` (installed or not), `separate_stems` (a clip → a group of
+ Drums / Bass / Other / Vocals tracks), `convert_audio_to_midi` (melody or harmony with
+ basic-pitch → a new MIDI track). Installing models stays with the user.
- **MIDI devices and MIDI Learn over MCP**: list connected MIDI inputs, toggle listening,
survey the controller's CCs and notes, view and edit bindings (range, inversion,
deletion), and enter learn mode.
diff --git a/LICENSES/THIRD-PARTY-NOTICES.md b/LICENSES/THIRD-PARTY-NOTICES.md
index 32e41974..c192f619 100644
--- a/LICENSES/THIRD-PARTY-NOTICES.md
+++ b/LICENSES/THIRD-PARTY-NOTICES.md
@@ -71,6 +71,22 @@ Public-domain reverb algorithm by Jezar at Dreampoint. Nota ships its **own
implementation** of the algorithm (`src/native/nota.engine/src/Reverb.h`); no original
code is vendored.
+## ONNX Runtime
+Copyright © Microsoft Corporation.
+Licensed under the **MIT License**. The managed API ships with Nota; the native library is
+downloaded with the first AI model, together with its own license and third-party
+notices. <https://github.com/microsoft/onnxruntime>
+
+## AI models (downloaded on request)
+- **htdemucs** (Demucs v4) — Copyright © Meta Platforms, Inc. and affiliates. Licensed
+ under the **MIT License**. <https://github.com/facebookresearch/demucs>
+- **basic-pitch** — Copyright © Spotify AB. Licensed under the **Apache License 2.0**.
+ <https://github.com/spotify/basic-pitch>
+
+Neither model is part of the Nota download; Settings → Downloads → AI Models fetches them
+when asked. Nota's C# code around them ports the projects' reference Python (spectrogram,
+segmenting, note decoding), under the same licenses' terms.
+
## Apple system frameworks
Portions © Apple Inc. On macOS, Nota calls the CoreAudio, CoreMIDI, AudioToolbox,
AudioUnit, CoreAudioKit and GameController frameworks provided by the operating system.
diff --git a/LICENSES/third-party.md b/LICENSES/third-party.md
index b179f5a4..7c4d3ece 100644
--- a/LICENSES/third-party.md
+++ b/LICENSES/third-party.md
@@ -22,6 +22,10 @@ public domain, or AGPL-compatible copyleft.
| HIIR (Laurent de Soras) | 1.40 | Oversampling (polyphase halfband IIR) | WTFPL | ✅ in use — see note 4 |
| Apple CoreAudio / CoreMIDI / AudioToolbox | system | Audio + MIDI I/O (macOS) | system SDK | ✅ in use |
| Geist / Geist Mono (Vercel) | 1.7.2 | UI and numeric typefaces (`assets/fonts`) | **SIL OFL 1.1** | ✅ in use — see note 5 |
+| ONNX Runtime — managed API (`Microsoft.ML.OnnxRuntime.Managed`) | 1.23.2 | Runs the AI models (Separate Stems, Convert to MIDI) | MIT | ✅ in use — see note 6 |
+| ONNX Runtime — native library | 1.23.2 | The same, downloaded with the first AI model | MIT | ✅ downloaded, not bundled — see note 6 |
+| htdemucs (Demucs v4, Meta) | v4 weights, ONNX export | Separate Stems model | MIT | ✅ downloaded, not bundled — see note 6 |
+| basic-pitch (Spotify) | 0.4.0 | Convert Melody / Harmony model | Apache-2.0 | ✅ downloaded, not bundled — see note 6 |
| Freeverb (algorithm only) | — | Built-in reverb | public domain | ✅ in use — own implementation of the Jezar algorithm; no vendored code (`src/native/nota.engine/src/Reverb.h`) |
Status key: ✅ cleared · ⚠️ cleared with a condition · ⏳ planned · ❌ rejected (incompatible).
@@ -57,6 +61,15 @@ SemiBold, Bold of each family) are copied unmodified from the `geist` npm packag
"Geist" is a Reserved Font Name, so a subset or other modification must be renamed.
They replace Inter, which came in through the `Avalonia.Fonts.Inter` package.
+**6 — The AI models and their runtime are downloaded, not shipped.**
+Only ONNX Runtime's managed API (MIT) is in the installer. The native ONNX Runtime library
+(from Microsoft's own release archives, with its `LICENSE` and `ThirdPartyNotices.txt`),
+htdemucs (MIT; exported to ONNX by `nota-daw/nota-models`) and basic-pitch (Apache-2.0,
+from Spotify's repository) are fetched by Settings → Downloads → AI Models into Nota's data
+folder, each pinned to a size and sha256 in `ModelCatalog.cs`. All three licenses permit
+use and redistribution with AGPL software. Nota's STFT, segmenting and note-decoding code
+around the models are its own ports of the reference Python (`src/managed/Nota.Infrastructure/Ai`).
+
## Not shipped
The JUCE submodule (`src/native/nota.engine/vendor/JUCE`) is fetched at build time and